# Introduction (view-source!) TL;DR <p style="margin-top: 25px;">Hi, this is my daily work journal. Although you're certainly invited to read it, there's really no reason unless you like wading through the random thoughts of another person. In other words, there's a really terrible signal-to-noise ratio here -- kind of like a random compilation of notes for a book. Twist is, I'll be "extracting" the book through a series of data transformations and queries. Often, the two are the same. I will also be refining this file, adding meta data and other things to slice and dice output and results in different ways for different uses, devices, etcetera. Tools matter, Linux, Python, vim, git, yay, yay, yay! SEO blah, blah, this page is a one-page website experiment, plus a sincere journal and exercise in Cialdini's principle of commitment and consistency. Ex Amiga fanboy. More within.</p> ## Probably Better Places to Visit Than Here: - [Subscribe to my YouTube Channel](https://www.youtube.com/mikelevin) - [Follow me on Twitter](https://twitter.com/miklevin) - [Connect on LinkedIn](https://www.linkedin.com/in/miklevin) - [My WordPress Site](http://mikelev.in/) - [My Instagram Pics](https://www.instagram.com/miklevin/) - [My Github Profile](https://github.com/miklevin) - [My Lightweight Linux VM Distro](https://github.com/miklevin/Levinux) - [My Free & Open Source SEO Software](https://github.com/miklevin/Pipulate) ### I begin my day by asking myself these questions: 1. What's most broken? 2. Where do you get the biggest bang for the buck? 3. What plates need to be spun? BE WHERE MANY OTHERS WANT TO BE, AND HELP LEAD THEM THERE. ### In "Striking Distance" Projects I Try Advancing Every Day: - **1stDoseFree** - Give people "free value" to build trust with audience - YouTube Channel, organize your playlists - Journal, "zip" all your media links & embeds into Journal chronology - Cross-channel fertilization, extract-and-publish to Medium.com - Start developing your subscriber-list with Google Forms / MailChimp - **MailMinion** - Email digestion, posting here, dispatching instructions, etc. - **AntiCloud** - Defend against loss using services and a personal data-centers ### To-Do List - PERSONAL - Finalize D. - Sell co-op - Taxes - Dentist - Eye Exam - CAREER - SERP Archiver - Daily GSC Data-pulls # Beginning of Journal -------------------------------------------------------------------------------- ## Thu Dec 8 16:03:50 EST 2016 Ugh, okay, the meeting occurred, and I had way more data than I had actionable insights or findings. I need to get this data easier to work with than Excel. I also need to finish automating the DAILY data-pulls. This is just about the most useful thing you can do for the organization. What if I were to just keep on that track that I had started on Tuesday and Wednesday, having to do with the cloud server data-pulls. Everything went so well up until the Excel work. Even the segmentation work today was really inspired. I'm sort of on a wild ride right now. Lots of questions right now, and I need to provide answers. If I don't have the actual answers, I at least need good data on which to base decisions. Focus is the hardest thing to achieve and to maintain. It can be your secret weapon if you can and do. But what about RIGHT NOW? What about your next thing? Go for thoroughness. Fill in some pieces that you rushed past on Tuesday and Wednesday. It feels like Jupyter Notebook date experimentation stuff locally that you can then move over to the cloud server. Just start out with little tasks, and document them. Maybe consider making the little non-proprietary tasks into public YouTube videos. Okay, I'm going home tonight with the intention of: 1. Thinking through my current project and really preparing to nail it in a huge way that makes me really happy and powerful. 2. Getting my home in order enough to feel really good about Adi being there this weekend. 3. Getting enough sleep. -------------------------------------------------------------------------------- ## Thu Dec 8 13:11:45 EST 2016 Next! Now, I need to look for whatever URLs were part of campaigns. Okay, I separated SuperNavActual from SuperNavRecos's. And I got in SaaS. Now, let's checklist the rest: - NovGaps - OctGaps - FrequentlyUpdated Okay, I've got enough in there to make a killer deliverable. Let's do it! - Change Position to 1 decimal place - Change CTR to 2 decimal places - Save as Excel Workbook - Bold & Center the head - Freeze Top Row - Insert PivotTable - Check URL - Check Position - Change Position Value Field Settings to Average - Change Position's decimal places to 1 (again) - Check Date - Move Date to Column Labels - Go into PivotTable Tools / Design and turn Grand Totals off for Rows & Cols -------------------------------------------------------------------------------- ## Thu Dec 8 09:14:51 EST 2016 ### Awesome Code to Segment URLs by Campaign In the office again. Not bad, 9:15 AM. Was expecting closer to 6 or 7 AM to hit this latest round of work hard, but I'm still getting used to being just a normal human being, only pushing myself as hard as normal human beings do in not-save-your-life time. Because truth be told, my life isn't at stake here. Only my reputation and earning capacity. Oh shit, hop to it! I got out the YouTube video AND the Podcast from this morning's commute ALREADY! And it was worth doing, because it helped me think through this morning's round of work. I have to hit this thing fast, and get the benefit of my approach sooner, rather than later! Do not go hunting for some other office to work in this morning -- just work (focus) like you did! Probably no need to shoot the videos. Even those will slow you down. But blast your way through this work, maybe just documenting your thought processes here instead. Okay, go clear your calendar. New flash! The reports are not even running this morning. This is either going to be a case of Murphy's Law, and everything collapsing, or this will be you pulling off some pretty amazing things of the precise sort the company needs most. Let's make sure it's the later. You can let it go for one day. Your prep-work for the meeting is much more important. Still, do your work here on this laptop. It's a MacBook Air (13-inch, Mid 2011) with macOS Sierra. Even this 5 year-old computer is more comfortable to work with than a modern one running Windows 7. I don't need Excel in the picture until this Python work is done. But focus, and get the Python work done. Confirm that a single column in Excel can be broken into columns. Confirmed! It's a simple based-on-delimiters scheme, so I'm going to have to be very careful about HOW I field-stuff. It's suddenly looking a lot like binary. I could use fixed-position with each digit standing for a campaign. I only think this because it occurred to me to use Yes/No's in the column, and then True/False's... and then 0's and 1's. Hmmmm. Keep it simple and readable. It will be most readable with the CSV like follows: col1,col2,col3,seg1seg2seg3seg4 1,2,3,"True;False;False;False" 1,2,3,"True;True;False;False" This just adds one more column to the already large file. It can use the Col1 value (URL) as the key to a dict. That dict contains URLs as keys, and lists as values. But how does that list get built? I of course have to spin through a much flatter data-structure that supports multiple URLs to multiple Segments. What's THAT data-structure? It should probably end up a NamedTuple, but I'm not going that direction right now. I think I just need an easily parsed format... oh! The campaigns become the key and the value is just a plain old list of URLs! So, I need to make my campaign-mapping look like this: {"CampaignName" : ['url1','url2','url3']} Ugh! I'm not going to be able to get all this coding done in the time-frame we're looking at! Especially not in a timeframe necessary to be prepared for the 3:00 PM meeting. Look at your algorithm! Oh, it's still best IF you get into it NOW! 1, 2, 3... 1? Get Jupyter Notebook running locally again... on your Windows laptop! I had quit out. So remember... activate py35 jupyter notebook Okay, this helps your flow continuity, switching between your thoughts here and your work there. You don't need to swap screens or anything! Make a new folder in your cygwin home... hmmm... gsc! Create a new ipynb in that folder. Call it segmenter! Okay, now move the CSV files from dropbox into this folder. Done. Okay, what's the best way to iterate through these? Oooh, great fast progress! I think I've got this. THIS is the power that Python's supposed to start yielding quickly. THIS is the reason for using it. Okay, think through next step! I'm using the csv.reader to spin through every row of every csv in a folder, and that itself is big. Okay, shoot an email to the boss telling him you'll be focusing very intently until about 1:00 PM. Now, stack those CSV files! Rename the folder to a better identity for this project, and then put it in github. Okay, it's in github. Okay, so now it's time to write the one big CSV file! Okay, I got the great big CSV being generated with a new last row-title and last row-data placeholder being inserted... woot! Next, I have to populate the contents of that last column based on what campaigns segments it is deemed to be a part of. Hmmm, okay, I definitely need the value of a dictionary getting inserted in that last location, using the URL as the key. Okay, it's already 11:30. I have an hour and a half to essentially nail this. By 1:00 PM, you want to be combing over the data in it's segmented and ideal form. Okay, I've got a list coming in from an external file, but I need to do better. Think! Thu Dec 8 12:54:15 EST 2016 Okay, I've got the technique of creating the segments. It's almost 1:00 PM, and that gives me 2-hours, including with the boss being here, to nail the actual URLs belonging to each campaign. But I sure organized this well. Let's get the SaaS URLs in there right now! Thu Dec 8 13:02:38 EST 2016 Okay, wow! After what felt like some desperate straits, I think I'm in good condition again! And I've done some of my finest (fast) work. An amazing little piece of reusable Python magic. This is a time to do a journal-cut, and totally switch my focus now into building those URL segment lists! Woot! -------------------------------------------------------------------------------- ## Wed Dec 7 19:21:35 EST 2016 ### Almost Ready to Eat Wow, here I am at home, at barely 7:20 PM. I've fed the cats, visited the concierge (Meow Mix), relieved myself, and am ready to receive Adi's phone-call on my main living room Mac, which is right in the middle of the upper kitchen, dining area where the Internet drop from the co-op is. So, most things are wired into the network ports on the back of the directly connected router. Woot! So, this is not even WiFi bottleneck. FaceTime will always work best now on this PC (Mac) in this location. Hmmm, circa Early 2009. Macs are sure built to last, aren't they? This is 2-years older than the already old 2011 Macbook Air I use every day at the office to shoot my screencam talking-head video. And still, it's more elegant along with VMWare Fusion and the way full-screen mode and virtual screen-swapping work than anything I've encountered, and I'm on a Microsoft Surface Pro 4 a lot of the time at home. I guess I sort of like it as a highly "compatible" Tablet mostly web-surfing device. For all my good intentions regarding writing and drawing Khan Academy-style animated white board videos on both the Samsung Note and Microsoft Surface Pro 4, I have very little to show for it. But Adi is interested in the concept of tele-drawing. I should really research that. I have to do a lot of work tonight. I expect I'll do it here on this circa 2009 Mac. I might even need to run Excel... hmmmm. Nah, I think I may want to roll my work back to the data-resides-in-Python state for processing, and not as an Excel file for further processing. I know there are Python libraries that let you read and write Excel files directly, and sure, that's an option. But too much overhead. Now remember lessons you've learned from ages past. Don't get stuck in either an over-fitting OR under-fitting situation. Size your solutions to fit! Put the amount of effort into it proportionate to the win that's up-for-grabs at stake here. And this is a doozie. Would you be willing to go straight through the night and be on your second wind at work over this? Wouldn't that one extra night of vampire-focus make an disproportionate about of benefit in your life for the twang of bad-health feeling it may cost? Why not "go angry" like in your Scala days? You need a conceptual enemy and conceptual mission. Conceptual? Ha, ha, okay, your mission is to boost your professional standing by doing some extraordinarily cool things, and prove yourself to be hot shit to your daughter. Let her see me becoming profoundly successful. Let her see my smarts paying off. Self determination! I am a human. I even understand the neurotransmitters and... Wed Dec 7 20:05:05 EST 2016 A very emotional call with Adi just wrapped up. She got in trouble for misbehaving, and I automatically side with her mother. Adi absolutely has to learn about consequences. She can make her decisions, but she has to live with the consequences. Life isn't without them, and it's a hell of a lot better that she learn these lessons from her mother and me now than at some much worse time in the future under much less parental and forgiving circumstances. We have to get her to understand the wisdom of often playing by the rules, if only to get them to relax for later on when you really need our guard to be down. She's still got a lot of key learnings to do. Even at 6, she's not really fully developed. Wow, she sure knows how to negotiate. That's coded way down deep in there, haha! Everything becomes a negotiation, as like Adi's starting point for coming back from a crying fit. She immediately starts looking for concessions that'll make her start not having a hissy fit, haha! Plenty of actual self-control there, and just liking the power of the hissy-fit situation. I've got to teach her that shit don't work with daddy. I understand those ability-to-control-tantrum strategies. By the time our call was done tonight, she was much more relaxed than when we started. I'm trying to get to the point in my story where the water-globule whale-beings who talked so slow a hello took a whole day got to talk with the tap-dancing yellow poofballs who's chitchat is rapid energetic tappity-tap's. The poofballs can talk to the spacewhales through squirrels. Almost there! -------------------------------------------------------------------------------- ## Wed Dec 7 09:01:25 EST 2016 ### Lots Accomplished Today, 3 More Videos Ah, Snickers for breakfast, with the corporate coffee. Get a taste of that old hyper-sugar college student commodore scala feeling again. Shit, was I on one crazy route! How did that happen? Wed Dec 7 12:38:39 EST 2016 Okay, it's already coming up on 1, and I'll want to be on the phone with my boss by 4:00 PM, going over the deliverables for tomorrow. That leaves both making the deliverables, and reporting on the effectiveness of prior projects, all in the 2 o'clock and 3 o'clock hours. That's really just 2-hours to work with, so everything that does not directly contribute to what I need to have ready for the 4:00 PM call and tomorrow's meeting goes on the back-burner. I have a chance to feel ready and prepared and NOT behind the 8-ball, and already, I'm pulling data from GSC. Wed Dec 7 14:49:14 EST 2016 Okay, it's already 2:50 PM, but on the other hand, I am making great progress. I'm doing the weekly data-pulls now of GSC and dropping them into the repo folder on the cloud server. I'm going to have to shuttle the files to where I can work with them. I will be served best by having dropbox working here, similarly to how you're using it on the other AWS instance that's running the pulse reports. Okay! Wed Dec 7 15:51:40 EST 2016 Okay, wow. I've got Dropbox working on the cloud server, and now shuttling files that get automatically generated is going to be really easy. However, the work I need to do now is much more Excel and Windows-oriented than Mac, and so the screencam videos stop here as I bake the datapulls into the deliverable for tomorrow. I'm about to go into pivot-table and filter land. I wonder if I can build slices based on discreet URL-sets. It's been a fairly intense day thus-far, and I'm against the wire now. -------------------------------------------------------------------------------- ## Wed Dec 7 07:09:42 EST 2016 ### This Morning's Thoughts... in Fours? We all have a need to work; to contribute to a collective, to further our greatness and stability and familiarity for future generations, so that we and our descendants may be the ones there to enjoy it. This self-organizing, compelled behavior and implied ingrained bias towards behavior that allows it, and things like it, to continue to exist is the essence of life, and what makes living collections of matter different from less-animate and self-determined forms of matter. So, things are really rather relative, you see. I fucked up a few things in my life, and the "expected" form of response is regret. But I feel no regrets. I feel like is a forever-forward road, and we are still not 100% sure that we're not in a deterministic universe. I think quantum randomness is nice and all that, but perhaps there are initial conditions to be made aware of at another scale or dimension, which could certainly be known or knowable, thus eliminating apparent random. Who's to say, really? Wish I could speak the language of Mathematics better. Wish I could spell mathematics without a spell-checker. My brain is sub-optimal in certain areas, but that's fine. - We can't all excel at all things. - So, excel at manipulating text. - Make text meaningful to machines. - Control machines at a system level. That sounds about right. Structure my thoughts in fours, to be a similar looking data structure to the 4-items I want to get done each day. So what are those four items? I don't have to know all four now, but 2 are: - Get the data-pull script done - Use the output-data from one property to prepare for a meeting tomorrow. -------------------------------------------------------------------------------- ## Tue Dec 6 20:21:35 EST 2016 ### Warnings From My Latest Audio Book: Alogorithms to Live By Beware over-fitting and training scars. I'm listening to Algorithms to Live By by Brian Christian and Tom Griffiths. This will likely be in the category of the Quantum Guide For the Perplexed book, as a great always useful filling in down-time refresher course. Be sure to select it as my last selection one month. If you can't explain it simply, you don't understand it well enough. -------------------------------------------------------------------------------- ## Tue Dec 6 16:26:08 EST 2016 ### Amazon Cloud Instance virtualenv Reminder Amazon Cloud server requires the following in order to create the virtualenv for the scripts: source ~/py35/bin/activate -------------------------------------------------------------------------------- ## Tue Dec 6 09:43:55 EST 2016 ### Huge Accomplishment Day, New Cloud Server Finally Launched In the office. Have 2 items to push out: YouTube video and Podcast. Been awhile on both fronts. Also, absorb the message of what you're pushing out. I'm about to start on a wonderful project now, which is going to optimize my time here at ZD. In other words, I'm taking steps now to make the most of my experience here, by making sure that I get what I need for myself, along with what the organization needs most. Jeff Bezos used a regret minimization algorithm early enough in his life so as to really guide it and take advantage of his particular timing. I'm already 46 years old, and I have thus-far failed to really do anything that reads on the upper echelons of the power distribution curve that is the noteworthy accomplishments of people around today. I'm barely even a blip, and I think I'd like to change that. And the beginning of the process is the here-and-now with my professional projects at-hand, which actually do end up being interesting enough. Plan those 4-things you really want to get done each day. Even if it's a mix of personal and professional, that's fine. Don't necessarily make them 4 ambitious things, either. Make it realistic, so that you can feel a sense of accomplishment and not disappointment every day. I hate to-do lists where the items just keep rolling over from day-to-day, nothing really being accomplished. Today, it's pretty clear: - Find a good place to sit and shoot some screencam video - Set up AWS cloud server instance provided to me by DevOps - Isolate script that does daily Search Console data-pull - Prepare for Thursday's "Major Stakeholder" Meeting Tue Dec 6 12:53:21 EST 2016 Okay, I got items #1 and #2 done... woot! And I have videos for both, wherein I actually confirmed the whole theory of being able to publish to only people with ziffdavis.com email addresses. -------------------------------------------------------------------------------- ## Tue Dec 6 07:48:57 EST 2016 ### Figuring Out My Retirement Gig / Python Steam Take advantage of the environment that is sure to exist over the next 4 to 8 years. Plan on Trump getting re-elected. If he doesn't totally screw up, there's a chance he totally nails it. I've been protectionist in my heart, since I learned about such matters in lower schools, not understanding why we wouldn't put higher taxes on imported goods if that's what other countries did to protect THEIR economies. We have the raw materials, workforce and markets. It just doesn't make sense to ship everything overseas, just to ship finished goods back for sale or final assembly and sale. Does seem like the United States is getting the short end of the deal, especially now that the argument that the TYPE of jobs that we're shipping overseas are of the type that nobody in the USA wants. That's an urban view. Apparently out in the boondocks, we're just like developing nations, with poverty and folks who will take any sort of factory opportunity over their other option of being a fast food cashier or something. We have cheap (smart?) labor. Anyhow, I think my role in all of this is teaching people how to bootstrap themselves from ONLY qualifying for "stupid" assembly-line jobs or soft-science bullshit into qualifying for more steam (a for arts) opportunities. Python is STEAM. Gain STEAM with Python. Python Steam LOTS of opportunity there. Use your toe-hold of "Python Advantages" and widen the opening and work towards a strongly visualized end-goal that has a revenue-generating component that could be your retirement gig. -------------------------------------------------------------------------------- ## Mon Dec 5 20:34:24 EST 2016 ### Proceed Intelligently I'll never stop getting in trouble for good intentions and trying my best. I had a wonderful weekend with Adi, and after she gets back to her Staten Island house, even such perfect weekends somehow managed to be tarnished after-the-fact, and I have to fight off attempts to somehow color them bad. If I get stressed out over such stupid maneuvers, then I lose. Let it roll off my back. It's just 8:30 PM, and I'm finished my chat with Adi. Got home about 7:30 and had a good half-hour chat with Adi. She cries a lot, and sometimes I feel it is just hunger or exhaustion, so I just have to be that much more skillful in my interactions. It breaks my heart when she cries. I used to cry a lot as a kid, and I didn't know why. She explained those same crying feelings to me, and I understand. My plan is to make the most of today's state of technology with video chat, and her iPad. I want her to realize she can talk to me anytime she wants (and I'm not in meetings at work). I so look forward to her first time calling me. I try to have stories ready to imprint on her and make her feel good, but still drama happens. I can't reduce the drama levels to low levels, it seems. Just keep doing the right thing/things, and the rightness will be born out over time. Sleep is one of the great pleasures of life, and now with this new arrangement of talking early, I will be able to get to sleep earlier, wake up earlier, and do stuff in the morning. Maybe I can even wake up at 4 or 5 AM and really get some stuff done when my energy levels are better, and it doesn't feel like suffering. I don't know how I made it over those past 10 years -- especially the dog. Oh my god, what had I gotten myself into. It's amazing how such a wonderful person as Adi could have come from such a situation. Adi got mad at me for talking her out of training wheels, and almost got hysterical when I tried explaining to her that it was the right way to do it. I had to agree to get training wheels put on the bike, or she would have totally broken down on the phone-call. I'm cornered into letting her get her way, because if I go the disciplinarian route, it looks even worse with me making her cry. I have such little leeway given how I'm always in danger of looking like "the bad guy". When Adi is that way with her mother, there are of course no ramifications. But with me, it's always potentially bad daddy. Ha ha, my best hope is Adi just maturing, and helping her to become a better and better person as she does. I will equip her to be a happy person. During all this time, I will walk simply walk that very thin line. What I can not do is blindly sign papers. Even just the suggestion that we include a definition of what "liberal visitation rights" actually is set her off in the typical fashion in the way that has been a constant corrosive influence on the marriage. If we can't even have a civil discussion as to what my rights are supposed to be after the divorce, how can I ever expect to actually have them? If it's this bad with just typical discussion, imagine how bad it would be after we disagreed on something after the papers are signed. I would be stupid on another level if I let access to Adi get cut off. I may be better off with a strictly defined schedule. I'm pretty sure what I want is joint legal and joint physical custody, with Adi with me every weekend, and legally up to 2 or 3 mutually agreed upon additional days of the week, as my schedule and life situation and Adi's educational situation allow. That's pretty reasonable, I think. I need guarantees against access to Adi being used as a constant threat against me, which she actually already started doing for just daring to bring up adding a definition of liberal visitation rights. Bad sign. Listen to the signs. This is a place where I should hold out for assurances that the law will be on my side before I sign anything. Navigate dangerous waters well. -------------------------------------------------------------------------------- ## Mon Dec 5 14:20:18 EST 2016 ### One Sheeter Done I got the 1st draft of the 2017 one-sheeter done, but I need to focus more on mobile. Find quotes and such. Okay, I've collected the materials. Think through the next one-sheeter. Mon Dec 5 18:24:38 EST 2016 Done. -------------------------------------------------------------------------------- ## Mon Dec 5 10:47:35 EST 2016 ### Reduce Expensive Context-Switching Okay, be super-focused. It is of paramount importance that you do not keep switching state. Switching state is the most costly thing you do, and doing so too often causes thrashing, and can completely prevent you from being organized on a daily basis. Your path to actual effectiveness depends on NOT switching context too often, and thereby allowing yourself to actually bear down completely upon and finish some task. -------------------------------------------------------------------------------- ## Mon Dec 5 09:44:20 EST 2016 ### Prune & Shape Don't let anything erode your ability to rejuvenate yourself and get back your mental and physical prowess. My work/life balance got terribly out-of-whack, counter-intuitively it was actually my "life" that destroyed work. I was not strong enough for a needy relationship on top of having a kid on top of New York living expenses on top of being the primary housekeeper in a house full of pets on top of a fairly ambitious job in which I have to stay at the peek of my game. All-in on the family-front eroded what I could do on the professional front, and I suffered and I collapsed. I addition to simply not being able to do the upkeep required, I simply collapsed from exhaustion. Any time-off I would ever get would be "generously" given, and then be cut in half by unexpected circumstances, and cut in half again, until I was getting only a quarter of the time that I was relying upon for recharging my reserves -- oh, and did I mention the money? Shit, I can't believe I went for ten years. Anyway, the trick now is to avoid entanglement, and all costs. Cause decoherence at every turn. Another target has been chosen to take the brunt, and a new artificial crisis-drama has been manufactured to take away peoples' choice in the matter of denying an anger-feeder reasons to be angry. There are now countless and unlimited reasons for the righteous fury of the expecting to be let loose upon the innocent bystanders. In current matters, I am an innocent bystander, and my duty to myself, and even to Adi, is to stay very clear of the anger fallout. Sometimes life is full of shit that you can't avoid, but you can employ strategies to minimize damage, impact, exposure, and even the need to think about such dangers raging just outside those closed doors that make your personal space safe. Make your safe personal space. This weekend with Adi and the bicycle was an enormous win, on the order of the sort that you could not pull off when you were a subject under the reign of anger. The power of putter, and the awesomeness of opportunity meeting preparedness! Life can be just as good as it can be shitty. Focus on the good. Prune and shape. Prune and shape. -------------------------------------------------------------------------------- ## Sun Dec 4 23:25:58 EST 2016 ### I Am Such A Proud Father Well, another fine, indeed perfect, weekend with Adi. But as with so many things in life, notating so perfect can go untarnished, and so I was pinged tonight by Adi's mother over me telling Adi to not tell her mother that she was hogtied. And that is, because she wasn't. Hogtied is an awful, degrading position of hands and legs tied behind back. I did not explain this to Adi when she asked for such nonsense, because I did not want her to, now recently being so-informed, demand true hog tying. It is not something I would do to my daughter in a million years. And so, I did a simple leaning forward wrists-to-legs shrink-wrapping using the very same spools that Adi's mother bought. I did a silly few-wraps of this and then undid it, getting it out of Adi's system. And I told her to not tell her mother that she was hogtied... what I neglected to tell Adi is that it is because she wasn't. And I got texts that Adi was in tears that I told her not to tell her mother she was hogtied, and she didn't want to like... hahaha! Well, that's typical of my life, so I responded after rapid algorithmic extrapolation as to what good could possibly ever come from engagement and entanglement, and the answer is none. I have nothing to gain from informing Adi's mother as to the subtlety of the situation, and how I am actually in the innocent position of momentarily entertaining Adi so that she doesn't enter a screaming hissy-fit for being denned her "hog tying". And so, I am happily and willingly the "bad guy", potentially in need of threatened therapy for navigating myself gracefully out of a silly damn situation of modern-day living. The father in such situations as I will always be cast as the bad guy, for friends of the other will always have to assume a sympathetic position as a friend. And so, I am surely a villain to many... hahaha! I do believe that my ability to read these situations as the transparent veils of what they are so easily is a source of great anger. Simply respond with "Okay, understood." No good can come from entanglement. Your happiness actually depends on avoiding entanglements with your ex except insofar as completely left-brained practical matters of Adi's education and child support. However, for this other stuff, it is much better to avoid engagement, and thus inextricable collapsed-state entanglements, and instead let it fizzle. Letting it fizzle is optimal. Engaging with actual discussion with me is now contingent upon intense left-brain examination of the engagement. Only sort what makes sense, and make sense of how you're naturally organizing things. You are optimizing, and not making a cartoon parody of yourself. Think like a human being, and behave like a human being. Take advantage of the fact that you happen to be a human being. You are able to do a bunch of things that almost no other type of matter can do, such as daily journal entries such as this. You are desiring to be a simple yet happy person. Avoid guilt. Don't let yourself be "forced" or "made" to feel guilty about who you are or what you have done. I have co-made Adi, and I will launch her out upon this world as my explanation of who I am. She's totally awesome. I am privileged to be her father, and I thank whatever forces that may be. I want to help her deal with her sudden unexplained sadness that she experiences sometimes. I totally get her. She is one variation on me. Optimization-time is upon us, and I will need to switch soon from exploration to exploitation behavior. If you don't write it down, it didn't happen. Be your own greatest inspiration and motivation. You are you. You did face this question and this problem, and your answer was vim and github. Your answer COULD be that, because you are you. This is not something most people can do, I am starting to see. Think about teaching Adi a list. Teach her the things most likely to be needed. I let her know I don't "believe" in time. And how time is just a side-effect of a bunch of other stuff that is going on. Now I have to decide whether it is okay to publish this. Yes! Of course it is. It is evidence of my mind-state at the times things happened. I was harshly scolded, and have the screen shots. I will add to those these thoughts that were in my mind at the time it happened. My cards are not so close to my chest, because this. I am keenly aware this is all public. And so, I make the writing so dense and intimidating, that only the most deducted parsers of my brain will dive in and stay so deep for so long, hahaha! I'll probably delete this post after I've regained my senses and realize this is nobody's business. But so what? This is so good as invisible, because people at large are so plagued with short attention-spans. I revealed to Adi this weekend that I do not believe in time. As she gets older, I will direct her towards Slaugherhouse 5 to explain. In the meanwhile, I'm trying to explain it in a way that will not imply that I believe in a deterministic universe. That's the trick. How do you state your belief that what we are is the cross-section of an object created by a membrane rippling forward in 4 spacial-dimension space through 4-spacial dimension stuff to a 6 year old? We are all an eternal beautiful snapshot of who we always were, who we are at this moment, and who we will be in the future. It's a fair view of all that is, because in the end, everything is some sort of composite object with interactions with other similar composite objects. We will probably have no idea when we reach true fundamental building blocks, even if we do. There's a problem of induction here, whatever we evolve to become. Even if it's our machine children, there will be a certain meme, counter-meme contest going on, so long as there is not some sort of cohesive unity. I'd imagine some sort of force, counter-force. A.I.'s will probably be separate from each other, I think. Person of Interest got some funny details correct, I think. But the disembodied shit is worrying. I rather would like to thing that any real A.I. schemes, beyond smart algorithm machine learning shit which is still not really life. Shit like scaled-stuff that could exist under today's Von Neuman computer architecture could theoretically be extended to. And that's really an awful lot. It could be all the A.I. tech we needed to improve human lives. The stuff created could never really make you uncomfortable about extinguishing it, because it still somehow just felt like a running program state. Digital is digital, so snapshot, reboot! Not so with true Androids. Or at least, not if you want results -- a practical human-sympathetic artificial intelligence. An artificial intelligence that truly understands humans as its parents, and thus give those parents some fighting chance of survival. We need something like Iain M. Bank's The Culture Series, which feels like probably one of the best case scenarios for existence, but for all that virtual hell stuff. Everything else, I'm a pretty big fan of. Can't wait until Adi's old enough for me to introduce Player of Games to... OMG, I got the chills. If she stays geek and even partially interested in what her old dad has to say, then I could really plant some wonderfully interesting seeds... on the meaning of existence thing, which I am led to believe she is already pondering. I am such a proud father. -------------------------------------------------------------------------------- ## Sat Dec 3 10:57:24 EST 2016 Almost heading out. The purpose of life is to bring order to matter. -------------------------------------------------------------------------------- ## Sat Dec 3 09:37:40 EST 2016 ### Making Progress I almost can't believe how disorganized I was. It's stunning that I was even functioning at all on the home-front. Oh, I wasn't. A place to sleep at night and barely get dressed and off to work the next day. My replenishment and rejuvenation and infusion of new motivation had to come from exterior sources, and not from my own home. My home went from source of strength to a little bit frightening to paralyzingly rehabilitating. I was not blind to it all. I just suffered silently. When I tried to fix things, the marriage would start to go south (in my mind) but the truth is it had gone south long ago, and my inability to actually tackle these things (after midnight after I got the child to sleep) was just a punch-line in the entire situation, because it caused me to lose even more sleep, get even more stressed, barely make a dent, and possibly make things even worse because once things get that bad, you sometimes have to tear things apart in order to re-order them, and I couldn't leave things in half transition-states during the day for many reasons. Flashbacks of trying to save the plecos when ick set into the tank, and they were just overfed and overfed and overfed. I was cycling most of the water in the tank nightly, just to keep the poor thing alive, suffering. We lost the big one, but the smaller one survived. The smaller one was one given to us with the tank from the in-laws, and the big one was one I had from my old 10-gallon tank that I got with the neon danios. Symbolism in everything. And so... and so... enough complaining. Actually, I'm making fabulous progress now, keeping everything alive. Now to focus on happiness! Happiness with Adi first. -------------------------------------------------------------------------------- ## Sat Dec 3 07:47:00 EST 2016 ### Adi's Desk into Living Room Awake and having my first coffee. Woke up at just about 7:00 AM thanks to the cats, but I got enough sleep. Went to bed listening to my new audio book, Algorithms to Live By. A new morning process is often finding the last thing I remember listening to. That book will be helpful in just my normal everyday life and day-to-day. It's the practical version of Quantum: A Guide For the Perplexed, which has been my while-traveling go-to book, so I think I will have to re-purchase it. It unfortunately was one I chose to use Audible's unlimited exchange program on, and I regret it. No matter, I'll re-purchase it as my end-of-the-month choice that ends up permanently in my library. But not this month, because I want to keep the Algorithms book. Hoping it will help keep me on track. I need to be at the in-laws by Noon today, and it's almost 8:00 AM. I should leave just before 11:00 AM and call the car by 10:00 AM. That leaves me a precisely 2-hour nearly uninterrupted window of weekend morning caffeinated bliss. Don't waste it! Okay, move Adi's desk out into the living room next to where I put the standing corner desk that I brought out to where the home security system is, so I could plug right into the net-drop and combine techie wire stuff. Also, just a great location to do daily journaling -- gets me out of the bedroom. This will create the feeling of Adi and I "working" side-by-side. And when it comes time to watch something, we can sit side-by-side here by the last remaining big(ish) screen in the house (24-inch). Glad I didn't buy that TV from Target last week. Those are the little types of things that direct life's course so greatly. Much better to go TV-less for a bit longer. -------------------------------------------------------------------------------- ## Fri Dec 2 19:32:22 EST 2016 ### The Day I Got Organized Sometimes a single day can change your entire life. Yesterday was one such day. Wow, if I could have had just one single day like that once a year for the past 10 years, I bet I could have been a much better husband. As a disorganized husband, I guess I was pretty shitty. I couldn't keep the monthly bills paid unless they were automated, and as fast as papers hit a table-surface, they were buried under the next thing and the next, and then little paper-piles had to be recovered and re-collected into random, miscellaneous mixtures of worthless junk mail and important tax documents. Dug myself out yesterday, I did. And I'm financially much worse off for being disorganized, but it's my bad. I have no excuses, and no one to blame but myself. It was a personality deficiency on my part that let me come so close to ruin, just by sheer virtue of disorganization on the home-front. Things could have been much different if I had a slightly different situation, where help would be shall we say a little more forthcoming... -------------------------------------------------------------------------------- ## Fri Dec 2 09:35:27 EST 2016 ### Paperwork & Reports Friday Plan the day. I didn't get my last edits form home committed and pushed, because I did an entry this morning. I have to put the "optimization" of my field of SEO into my life. I need to start optimizing things for myself. I know that's a vary vague notion, but generally I want to do things in better, smarter ways, and occasionally put in significantly more effort than I currently do, but only because the payoff will be big enough and I'm pretty confident of that fact. Okay, think through today: - Greg's Request(s) - Benefits PDF on Phone - Weekly SEO Report What else? Hmmm, okay, I had better slam out that weekly SEO report. Fri Dec 2 18:12:41 EST 2016 Outta here! -------------------------------------------------------------------------------- ## Fri Dec 2 07:45:16 EST 2016 ### Woke Up On Time! I didn't oversleep. Going to be heading out of here at a reasonable time too. Already finished my first coffee. Amazing how much more "clean" I feel now that I've tackled the mountains of accumulated papers. I'll have another window from when I get home tonight after work (not picking Adi up from homeschooling classes) to when I leave for the in-laws tomorrow morning. Me actually organized and on top of things in TODAY'S world is almost a scary idea. The things I'm good at and WANT to talk/write/shoot videos about are so on-point for today's world that it's not even funny. -------------------------------------------------------------------------------- ## Thu Dec 1 10:15:13 EST 2016 ### Dug Through Mountains of Accumulated Paper & Mail Put on a show for Adi. Make her visits here almost like witnessing time-lapse photography (like the NG big/small, slow/fast documentary I had her watch last week). Be much like a gathering storm, going from disorganized loser to organized force-of-nature. Gather your winds. Thu Dec 1 11:01:19 EST 2016 Okay, just set up my computer working area. Moved it over to the Internet drop into the apartment, and consolidated with the home security system in the tall-standing corner desk. Also put together the garment rack, and now all clothes are off the floor. Looking into good bureau/dresser system for Adi here, and kid's hangers. Her and I can get into this organizing stuff together! Thu Dec 1 11:42:39 EST 2016 The requests are piling on at work, and I have to do the gathering storm, force-of-nature thing there too. But that takes time and putting the extra push into it of the type that I'm spending right now on the home-front. Getting into the zone in order to get all your systems serving you well, so that the feedback loops start to turn positive and work for you day-in and day-out. Yes! Keep on this track at home. It's a VERY SMALL window, and watch for intimidation techniques being used on you which pose the danger, intentionally or not, of derailing you by pulling you out of the zone with sufficient frequency to keep automatic productive muscle-memory from kicking in. Start by muting your computer so you don't hear any alerts, and put your phone on DND. Also just don't carry your phone in your pocket today. Put it somewhere out of sight, out of mind, but still easy to find. Keep your coffee in sight. Okay, don't take excessively stupid risks. Always measure what's best by what you believe is best for Adi and your sanity. A successful game is played by Adi reaching adult-hood and being relatively happy with her Mom and Dad and to be funneling her inevitable angst, rebelliousness and anger (whatever that quality is that a lot of people have that drive them to restlessness) into generally productive and positive pursuits and outlets. She should know me as a person, an individual, and a human being pretty well by then. Those good parts of my personality, attributes and habits should have been instilled into her by then, by example. And the thing she's seeing me do now is get my shit together. And I pretty friggin successfully plotted and navigated a return-course after my fall following switching out of SEO at 360i. My gut told me to do it, and my gut was right. I don't know where I would be now if I didn't have my exposure to Jeff Scott Ward and Brian Housman, who indoctrinated me to how different things were really becoming with the all the available "web stacks". Thu Dec 1 15:53:04 EST 2016 Okay, the day is slipping away from me. I've got a lot done, and am starting to feel more myself than I have in years, but there's still plenty to do tonight. Don't let any wrenches get thrown into the works. Keep the pressure on. Focus on priorities. Haul out some trash. Things are starting to feel a little more doable. I am no longer crammed into immobility. My situation was so opposite to who I am and what I need that it's no wonder that I was reduced to just mere survival. Above and beyond... having EXCESS capacity... was completely out of the question. Maybe by tonight, that will not be the case anymore. Work very hard now so you can feel good about yourself. Thu Dec 1 23:12:12 EST 2016 Got a lot done, but not everything. Will once again have to get SOME sleep, and then come back to it. Maybe set alarm for 4:00 AM. Another good stretch should do it for me. Fri Dec 2 00:14:39 EST 2016 Still going at it. Faster. More of a sense of desperation to get it done. Saving all the miscellaneous birthday cards I encounter. Now I'm starting to understand why Dad kept all those cards... proof. Proof that that old family once existed. I had an interesting 10-year run. Bitter-sweet. Encountering all the paperwork of the complaints lodged against us by the idiot busybody. Shit, I can't believe what I endured. I can't believe what a toll it can take on you when you find yourself surrounded by genuinely awful people. My experience here at Park Terrace Gardens with the neighbor is up there with dealing with my mother's schizophrenia and my father's sudden death and me having to take over the check cashing and having to shoot a guy and go to jail. This neighbor was up in that category of awful. Anyway, I got through most of the paperwork. It's awakening just how asleep I've been over these past bunch of years. The impossibility of sitting down and addressing paperwork in my life had made me give up even trying. I wonder how that other guy... anyway... probably just bad chemistry between her and I, because it's almost like a KT-boundary -- there is a time before which I actually was organized and did everything correctly and on time. And then, there was a time after. The thing that paralyzed me from doing the things I needed to must have been... oh, screw it. Not even worth ruminating about this late. I hope I can get to sleep now. Part of the reason I hated trying to do this. It's a whole day I put into it, and now past midnight. Imagine if I STARTED at midnight, which really has been just about my only option for the past 6 years, minus the 7 months we've been separated now, where I was in a sort of paralyzed stupor and just trying to dig myself out enough to function like a normal human being again. Okay, anyway I can feel myself pulling myself on-track again, and that's actually a promising feeling. I can imagine now that there's an end in sight to all of this... an end of the stress and jammed-up crammed-up feeling, and a beginning to being the daddy that I know I can be for Adi, and which I desperately want now more than anything else. Fri Dec 2 01:29:55 EST 2016 Good night. -------------------------------------------------------------------------------- ## Thu Dec 1 08:46:55 EST 2016 ### Artists At The Helm of The Future Okay, I'm taking a personal day today, and I'm going to make the most of it, including actually getting started working before 9:00 AM and after a good night's sleep. I even spoke with Adi early last night, so I got to sleep relatively early, so besides from the cats waking me up for food at sunrise, I probably got a good 8-hours or even more. It still feels like a rare and precious thing to get enough sleep, and to have a full day to work on personal affairs after a good night's sleep is almost like a decade-long dream fulfilled. Don't squander this. I've had potential days like this in the past that I didn't make the most of because of my broken emotional state and inability to work efficiently and near my potential. Today, I can. So, today I will, and will no longer be behind the 8-ball or have my head under the surface, and all that. Watch your money like a hawk now, and see where all of it is going. A thousand dollars got transferred out this morning after my deposit yesterday, which puts me in danger of not being able to meet this month's expenses. It's probably time to cash out my IRA and pay off my credit cards, just for starters. But focus on your priorities for the next few hours, and that involves setting things up for a very efficient day without overdoing it. It's also your window to get laundry done. Thu Dec 1 09:38:34 EST 2016 Should you (me) and could you ask about the what-if's of life? Maybe I'm actually still on the perfect and most well-optimized experience for my particular being in this world. Maybe Adi was simply meant to be, and I was meant to be the un-distracted and purposeful daddy to her, helping launch her into her turn at it all in the most optimized and best-for-her way possible. Maybe all those measures society and culture uses aren't the right ones, particularly for me. And even so, I'm not doing THAT BAD by those. I could be stuck in the unexceptional dregs of the suburbs where I was born. There is simple happiness there, but... but what? Isn't simple happiness exactly what I want? No, there's something about being connected to something more exceptional and magnet-like for a lot of like-minded people, which is what pulled me out of my suburban burrow in the first place. Back then, it was the Amiga computer. Today, it's most likely Python, with Jupyter Notebook and vim playing a close second runner-up. I can see it in diagrams in my head. That really should be my online content and persona, helping people deal with all those same issues and have a well-balanced life, and connect well with things worth connecting with, in a real capability-expanding, life-bettering, and generally entertaining way, like the role the Amiga played in my life, and the way robots and miracle-material products are going to play in Adi's life. We can't (read: shouldn't) all just become mindless consumers of that stuff. There should be artists at the helm, and Adi should be one of those artists, I think. -------------------------------------------------------------------------------- ## Wed Nov 30 18:49:31 EST 2016 ### Keeping The Pressure On And here I am back home... on the early side. Ugh, okay, tomorrow I'm taking as a personal day. And I'm going to make it through it. I am going to be the rare gentleman. I will not make anything any harder for anyone, but I do know what I want. Tonight will be interesting. Both the garment rack and the fold-out table. Woot! Wow, little things like this can move life so significantly forward, getting shit off the floor and shit. It occurred to me that a joy this generation's children will never know is the combination of Saturday morning cartoons and a new box of cereal with a prize in the box. Both sides of that Venn diagram are no more. Tried getting a box of Fruit Loops with glow in the dark stickers inside. They were outside the sealed cereal bag loose in the cardboard box. Sigh. But theirs is a world of tablets and apps. How to use this generation's knowledge to make her the master of the next? And that's only if she wants to go that sort of route. Yes, Adi. Your dad is a nerd. Guilty as charged! Making ramen noodles again. Get them e't' and get on with some garment rack building and some table unfolding and paperwork pruning. -------------------------------------------------------------------------------- ## Wed Nov 30 10:20:16 EST 2016 ### Making Room on Macbook Air So I Can Shoot Screencasts Okay, merged yesterday's work in. Interesting having to do a git pull from work before I get started in the morning. That's a change from my usual norm. I should be careful typing. I'm currently copying almost 30GB of old iPhone backup from this laptop to an external drive. Life moves forward, but sometimes you can keep a captured state from the past to reminisce some day... that is IF there's some way to get an old iphone backup actually running on anything in the future, and that's a big if due to the Apple-proprietary nature of the stuff. That's part of the cost of being locked in a beautiful prison. They even control a little piece of your past-state digital memories. Slick! Once those files are moved (hopefully soon, because I'd like to use this laptop for an 11:00 AM meeting... oh no way, won't make it. I just control-C'd out of it to take the pressure off of me, and even the stress of worrying about not typing too hard as the file-copy is taking place. But an important thing to remember is that I had to make it recursive and verbose, so the command was something like: cp -vcn . "/Volumes/[blah blah blah]/" V for verbose, c for clobber, n for recursive And when you're on a Mac, especially an older one like this, when you're in directories that contain a vast quantity of files without using hierarchy to organize, a drag-and-drop, especially one with an Apple modifier key to move instead of copy, is a system-killer. Wait and wait, and do I let go of the mouse-click? What will happen? Is it dependent on where my pointer is when I let go? Sheesh! Totally worth investigating the Unix/Linux way of doing a recursive file-copy from the Terminal, which I did. But it's way too risky to keep working on that laptop during such a large (fragile) file-copy, then I'd have to deal with how-far-it-got and copy vs. move issues. Shit, okay. Let me think. Do that later, but I really NEED the space that's going to free up to do my videos on this laptop that I'm dying to do for a soon-to-exist ZD internal YouTube channel. It's so weird the feeling of live-TV broadcasting coming back into my life via Facebook. I know YouTube is also into live broadcasts, but because of the social network in Facebook, it just feels so much more... I don't know, sincere or interesting to go tune into as it's happening. Also, I guess that's due to live-TV notifications being tied into the "normal" Facebook notification system which everyone keeps turned on with their whatever filters set, least they not be informed as some important news is breaking. So Facebook Live gets through notification-filters. Our gmail addresses that drive much of our Google presence (search histories and such) are far less social (and in fact, would be a little bit scary if they were) and are therefore not allowed quite the leeway of letting notifications get through your shield. I subscribe to a lot of things on YouTube, none of which would make me want to be interrupted at work when something new was published on one of them. No, that can wait until my personal time when I'm at home. But a Facebook notification... well, that's like a friend, even if it does happen to be Random Access. It's feeding a hunger that I haven't really fed since the net in general broke my addiction to the non-stop cable TV feed. I cut the cord, and get my Internet through my building. Feels good to not pay a cable company. Okay, but now given the time between now and my first meeting of the day should be spent fleshing out that one-pager outline I have to deliver ASAP. It's already Wednesday, and I wanted to deliver it Monday, and I'm going to be asking for the next 2 days off, even though the pressure is suddenly even more intense here than usual. Shit, okay... think! This is life. Success and failure is determined by how we deal with moments like this, and rise to the occasion ON ALL FRONTS. Okay, my less-than-effectiveness right now can be attributed to no single front receiving the full force of my loving daily attention. I am always dropping bone-1 that I'm burying in order to go dig a hole for bone-2, and pretty soon you've got a yard full of holes and no bones buried. Thanks for that colorful analogy, Marc Rifkin. Never forgot it. Also like it because of how it's tied to our suburban roots. Guess that's something Lizzie used to do? Anyhoo, I like focus and I like getting into the zone and I like being able to do the miraculous by being able to focus on and tackle a problem for longer periods of time than others are willing to put into it, and create solutions that are just slightly more generalized than the particular use-case you were just tackling, so that it can be re-molded and shaped to address other similar such problems that you are going to surely be facing again in the not-to-distant future. FINALLY now, some of that strategic, high-impact thinking that I've used throughout my life to bootstrap me up from out of sticky situations in the past. This current round of stress I'm undergoing holds nothing to that of my Dad dying and taking over the fucking Check Cashing operation and having to shoot a guy and my Mom going crazy and trying to rescue her in Mexico and the stress and isolating effect of my always-angry sister and generally attention-starved state that left me a little more than vulnerable to yearning for a sense of attachment and belonging that got mis-translated in almost every conceivable way, in job loyalties and relationships... sigh... yeah, I "get it". Just finally becoming a degree more self-aware in that whole -------------------------------------------------------------------------------- ## Wed Nov 30 05:53:45 EST 2016 ### Breaking Out of Negative Feedback Loop Okay, I'm back awake. I did quite a bit of organizing last night after all. I did a first-pass on the paperwork, finally, sorting out those things that could not possibly have the sorts of documents that I need. This morning, I will complete that, and make a grab for all the documents that I need. That's the goal for this morning. Try not to be late for the office. I've been doing good getting in on the earlier side a lot lately, and that's good. Also not cutting out early on "holiday" hours either. Put in for Thursday and Friday with my 2 remaining personal days (don't have to pull out the sick days yet). Pumping some coffee into me, and trying to invoke the fight-or-flight response (minimizing stress), so that I can get a little adrenaline working on my side. It was pretty clear all my discretionary time, especially that zone wherein I'd I'd have to find my motivation and make the big efforts, all landed in the time-windows I was using for Adi, such as my evening phone-calls with her, and every weekend from Friday after walking out of work until Sunday night after dropping her off and getting back home exhausted. Wed Nov 30 07:52:50 EST 2016 Okay, not as much done as I'd like, but some progress. Making more room for things so that I can have room for things. Do other people have problems like this? I'm sure, but I'm somewhere on the spectrum. My natural proclivities and propensities made it way too easy for me to just shut up and pay, without doing all those other things you have to do so that you continue to have money to pay. Drained. Emotionally, physically and environmentally -- complete collapse of motivation, with only enough umph to keep myself professionally viable, which I have done. Now that pathways are clearing, it's time to lay down new ones that work better. Arrange your environment constantly to work FOR you and not AGAINST you. Shit, was I blind. Definitely a negative feedback loop. -------------------------------------------------------------------------------- ## Tue Nov 29 19:09:12 EST 2016 ### Intergalactic Squirrels Well, I did a whole day of work and journal notes and forgot to commit, and got home and realized on a git pull. It's amazing with how much more confidence I'm typing stuff even like this now, knowing I'll be able to merge and at worst, move some blocks of text around to be in correct chronological order, and delete a few merge markers. Usually I just search on a bunch of equals signs to find them. I highly recommend using git to keep your personal journal, without using any fancy software or anything, you can just load a text file into a text editor and start typing. There's your blog. The thing that makes it more than that is that it is also a git repo -- or at least a local one. If you don't mind making it public, or trusting it in a private-but-online (by definition, not REALLY private) Github account, then you can participate in this wonderful world of Github. You could also use BitBucket. I don't discriminate, but Github changed the world a little bit. BitBucket reversed their business model and me-too'd. Yeah, I know, Mercurial (hg), but one's becoming a consistent requirement in the plumbing of the Internet, and the other has a few interesting projects on it. But if you want to go where Metcaffe's Law is working at its finest in the Free and Open Source Software (FOSS) world, then look to Github. Tue Nov 29 19:24:00 EST 2016 Just had an awesome quick chat and good night with Adi. I'm spinning a yarn involving intergalactic rainbow shooting squirrels who got their powers from a rare Macadamium Nut mined from the asteroid belt of an intelligent squirrel world who raided Earth for worker squirrels whom were recruited by being promised glamorous space adventures, but it was really just asteroid-mining, where all the squirrels were motivated by the competition to find the rumored rare nut-elements, of which Macadamium us the most rare. It bestows upon the gobbler the ability to fly through rainbow wormholes between planets all around the Galaxy... and maybe more. Adi just found out that the squirrels have used this to colonize all known life-bearing planets in the Galaxy, so that no matter where you go and no matter how wacky lifeforms may get, there are also squirrels. us the most rare. It bestows upon the gobbler the ability to fly through rainbow wormholes between planets all around the galaxy... and maybe more. Adi just found out that the squirrels have used this to colonize all known life-bearing planets in the Galaxy, so that no matter where you go and no matter how wacky lifeforms get, there are also squirrels. And squirrels use this vast intergalactic network to play games that involve hopping from planet to planet and tap-dancing, and one other thing that I forget, but believe it involved combat. And this is more or less where I left her, with the explanation that she needed time to anticipate what was coming, and I needed time to make it up in my mind, as like my homework. She was impressed by the fact that I had homework. Wow, I really connect with that kid. Now, off for some ramen noodles. And try to do again tonight what you did last night, early-to-bed and early to rise. I did SO MUCH BETTER this morning after a good night's sleep than I have in months, trying to psyche myself up after coming home from work, pretty much drained. Tue Nov 29 20:19:30 EST 2016 I just got the notification that Amazon tried to deliver something and couldn't, but I didn't hear the buzzer ring. Damn, that was the hanging wardrobe thingie. Not a humongous wardrobe, but something that looks a lot like baker's rack material. Was looking forward to using it tonight. Oh well, maybe tomorrow. Not going to do much tonight. Finish eating and go to bed. Wake up, and generate an extra 2 or 3 productive hours during the day than you would have otherwise, or could possibly be able to put in now. And continue reading The Master Key System, and energize yourself. Make your inner world in-tune with and positively project onto the outer world. It's all the same stuff as every other positive mental attitude book, but with thinly layered on cosmic mysticism. It's not so much really mysticism as much as trying to nail the whole Gaia thing. Everything is interconnected in one giant system, and we are a part of that system just like every other conscious living thing. Interesting that I realized the appeal of reliving the chronicles of that asshole Thomas Covenant. I get it. A lot of my self-taught mental discipline came in being able to get through that 6-book (at the time) series of a sort of Tolkien anti-Fellowship. A guy who has to live with the reality that one false move could kill him (leprosy) and "the land" and it's healing power puts him always in danger of dropping his guard and getting out of his habits, right as he knows he'll need it most to keep himself alive. So, he's radically mentally disciplined -- not necessarily in the make-yourself-a-genius way that I think many of us would like to accomplish, but more in a make yourself behave a certain way, because the reality you accept necessitates that you must for your survival. Yep, my reading material will be rich nowabouts. Now, finish those noodles and go to bed! -------------------------------------------------------------------------------- ## Tue Nov 29 09:41:31 EST 2016 ### Debugging Cron Job Okay, not to shabby for having woken up at 5:00 AM to get organized. Starting a journal entry by 9:40. Recorded great audio for a podcast, but may be too out there to actually publish. Going to save it on my phone and listen to in entirety to decide later. Right now, I'm generating the morning SEO Pulse Report data feeds, because it stopped working automatically. I have to go check /etc/cron.daily... first make sure the file is actually there. My suspicion is that an apt-get update and upgrade blew away files or file permissions or something. No, the file is there and the permissions are correct. I'm already generating the CSVs, so I might as well wait until that's done, because I want them all completely generated in time for the meeting today that will touch on the SEO dashboards. Okay... next. -------------------------------------------------------------------------------- ## Tue Nov 29 06:28:18 EST 2016 ### Do It For Adi The last piece I need to get my bedroom in workable order is a bureau. I put one in my Amazon shopping cart, but I don't have the money this month, so I'll just wait until next month. I have to start budgeting like a hawk. I may even start writing about that here, if it isn't already boring enough for those nutjobs who actually think this is worth reading. It's just the trials and tribulations of a regular person working his way through life. Some of the basics have become challenging to me of late, and it shouldn't be so. It's like I have to self-repair and self-heal after some sort of crazy ride... which is exactly what's going on. That crazy ride resulted in Adi, and now I have a better reason than ever to become my old self. In the short-term, everything is more difficult than it should be, but the more I do, the easier everything gets (again). Okay, a plan is forming. Lots of table space is what's needed, and the kitchen table hasn't been clear since "we" got it. I almost have it clear, and will transfer the piles and piles of paperwork there. Tue Nov 29 07:15:00 EST 2016 Take pride from this. Find strength in progress. Do it for Adi. Each time she comes over, impress her anew with how much someone can take control of and master their environment, in order to give themselves a better life. There's something now almost surreal and mystical about being able to clean up unimpeded, without a sort of interference being constantly run. The kitchen table is completely clear, and all the puffy stickers scraped off. Now, I'm gathering the piles of paper and putting the file cabinet and shredder into position. I'm almost crying as I start taking out file-folders and labeling them. How could I have drifted so far from who I am, and what I need to do. Now, I finally realize why I'm being drawn back to reading Thomas Covenant, The Unbeliever after all these years. I NEED to be that diligent against the onslaught of disorganization. It's nothing in the neighborhood of what lepers need to do in their forever-diligence routine of not getting an injury, but I do indeed need a little touch of that always-awareness and forever-vigilance. The world is made of the forces of chaos and order at war with each other, and despite all my talk, I'm not one of those Steve Jobs simplicity aesthetic freaks, but I do need a does of that to keep me from falling off the other end. Sheesh! I can see that the difference between a happy and not-happy marriage could have come from me just somehow getting the time it takes to get and stay organized. Shit, live and learn. Well, my happiness will not be wasted on Adi. She will feel the rays of happy sunshine upon her. Shit, it's almost 8:00 AM. Wrap this up, and even get into work early! Wow, it's 8:00 AM already! Okay, wrap up and head out. You won't feel poor anymore when this is all over. Every minute that you're not commuting or talking to Adi on the phone or at the office working should be dedicated to this purpose now -- for Adi and me. That's the way to think of it. -------------------------------------------------------------------------------- ## Tue Nov 29 05:43:34 EST 2016 ### Do it For Adi Woke up early. Facing impossible odds, but rise to the occasion. You can't undo 10 years of disorganization and clutter with 2 extra hours you gave yourself in the morning (should talk to Adi early every night), but you can make some really good progress, and have everything ready to sort into the table when it arrives when I get home from work today. Both the Staples-style folding table and the garment rack are arriving by 8:00 PM tonight. Do progressive sweeps and don't relent. Focus on paperwork, that is the priority. It took 10 years to get to this point, and it's going to take some time to dig out entirely, but it's only going to happen with steady, even pressure and diligence. Oh, there's that steady even pressure concept that was in maybe a good 10-year stretch of my journaling, up until about 10 years ago. Sheesh! Okay, lots of artificial outside pressure is being put on you now, but don't do it for that reason. Do this for yourself. Wrap up everything into a tidy little package, and start the new chapter in your life, where you're the most awesome dad that ever lived, and give Adi a great boost and advantage in life, most finely tuned to what are her own actual interests (not just mine). Steady, even pressure. Do this for Adi. -------------------------------------------------------------------------------- ## Mon Nov 28 09:24:09 EST 2016 ### Thanksgiving Breather Okay, made it through Thanksgiving Weekend. Things are going to get a bit strange, as if they weren't already. Or perhaps in fact, things are going to gradually settle down to a better new normal than they've ever been in my life, because I now have Adi in my life. Never forget how important these impressionable years are to her, and how what's going on with you and how you handle it is so important to her. I'm not doing everything as well as I should these days, but I never have. Striving for optimization. People are themselves guilty of what they talk about and even criticize most, and in my case it's definitely true. I lack optimization in my everyday life, and it takes me so much work, and I value it so much. It's not even optimization that I seek, but rather basic, competent organization. That's what's lacking in so many things, and THAT should go into the one-sheeter. - Basic, Competent Organization - What you MUST do - What you SHOULD do That's it! Transition your thought process to work. Throw yourself totally into work, and put in a kick-ass day. I need to be inspired. This is when you have to rise to your finest. Again, flash-backs of difficult prior times in my life. I got through my dad dying and having to take over the fucking check cashing place. I survived my mom going crazy... repeatedly, and the internal struggle that created. I survived my first round of marriage so-far, and I'm made to be the super-awesome Dad that wonderful girl is going to remember all her life fondly. I'm doing the best I can, but I have to do a little better. Turn your attention to weekly reports, and of course use it to: - Plan the DAY - Plan the WEEK You have, above everything else, got to feel good about yourself. Take pride in your work. Do things that you would be excited to show Adi and that she would be interested to see. Maybe take up your picture-drawing again. At it's very simplest, you could draw line drawings on paper and take pictures. It's probably better to start out with computer-based digital stuff, and have it automatically look good and not have that rough edge that would need to be refined out. And getting your automations rock-solid. That's soooo important right now. Okay, shit. And the parts of Pipulate that need to be stabilized more, mostly having to do with reliably 24x7 reliability. But I think I'll be doing .py-file extractions. So many interesting things! But you need to start making a more serious impact fast! It's not like things are going to go quiet. More and more is going to pile up, and you should expect that. Keep your cool, and don't get stuck in negative feedback loops. I've been thinking about positive feedback loops a lot lately, but what about the negative ones? I'm stuck in one. In many aspects of my life, I've been stuck in one for awhile. And it's all about breaking cycles. Apparently, this journal doesn't help you that much, because you've been keeping it for awhile and you haven't broken out. But maybe that's not true, as I've only bee keeping it since during Flying Point Digital. In fact, it was on December 6, 2014. So it's just about been 2 years, and really that's just about right in getting my life back on track. It HAS been helping, I think! Of course. Okay, now today. 1, 2, 3... 1? Weekly Report! Done. Next: Feed Check! Oops, sent a message regarding one of the property's GSC verification to one of the DevOps guys. Got to get that fixed pronto. But dig into your feed check. Get into the zone. Mon Nov 28 16:21:34 EST 2016 Okay, the day is almost behind me (office-wise), and I have one more thing to deliver today. Focus on it and get it done. Okay, I just sent it out as-is. It'll move the project forward. I can't make better data selections right now for the menu. We made some really good ones together. And so next... next... oh, of course! It's time to do that beautiful one-sheeter. Ugh! I can't get my head into the visual stuff right now. Think Fast! Flow from panel to panel and tell a story. What is that story? It goes something like: Think Fast! The need for speed in 2017... yes! Okay, I need to start laying this out on a page. Make it so you can push and move stuff around. Choose your page design tool. Okay, it actually looks like Google Docs may be up to it. -------------------------------------------------------------------------------- ## Sun Nov 27 23:13:31 EST 2016 ### Get Yourself More Disciplined! Back from dropping Adi off. Tired and want to go to sleep. So much to do! I have to find ways to get more done. I really give my whole weekends to Adi, and I feel it's only fair to her to do so. But my energy levels after getting home from work are so low, that there's no way that I can give myself a second wind to get done the formidable amounts of stuff I need to get done, and all this, and I barely even feel like I've got my head up over the surface again. I was truly under water and drowning. Now, I'm catching my breath and waving my waterlogged and frozen clothing trying to turn myself into the champion swimmer I will need to be to get to shore. And so, I will do my best. But I will not drive my health into the ground. I will get sleep. I will have to call in some sick days. It can't be tomorrow, because tomorrow is Cyber Monday. I need to be a hand on-deck. But I will take at least one day this week to meet accountant. And that means tomorrow night after work (I will be more up to it than I am tonight / exhausted). I need to find my motivation. Like that feeling Adi said she got from the Moana movie, which made her want to do the rope challenge at the Palisades Mall. And knitting those memories with her now at her most impressionable age is my priority for these weekends, during which I have her. I may need at least one weekend without her to get myself organized. I could of course do it right now. Ugh, I feel canker sores forming just at the thought of that loss of sleep and stress. Tomorrow night. Paperwork organization tomorrow night. Leave work promptly at 6:00 PM. That means get to work promptly at 9:00 AM (a problem you've been having recently). Work efficiently. You have the weekly report tomorrow. You will have a few things to report. And you have to get out that one overdue deliverable. And this and that and this and that. It's ALSO going to be a busy and intense week at work. Shit, okay. The trick now is to not waste the time in the margins. Make use of everything. Make INTELLIGENT use of everything -- including THIS. Is this intelligent? Yes, I have a plan. Discipline! Sleep now. Hit the ground running bright and early tomorrow morning. Stay up with the first cat wake-up call. Make sure the cats are well fed tonight. Don't go back to sleep after a first feeding. Make coffee. Shower, shave and shoot out that door! Slam out that weekly report, deliver that deliverable, do that other deliverable, and that other and that other! No down-time in-between. Shit, I may not be able to perform at these levels. I may have to have a heart to heart with my boss about expectations, and what it's going to take to get some of this stuff done. Or is it my own inability to focus? Do I actually HAVE those uninterrupted time-blocks that I need, and all I need is discipline. Hmmmm, I may be onto something. Okay so then it's easy: be disciplined. Go to bed! Yes, sir! More soon. -------------------------------------------------------------------------------- ## Thu Nov 24 11:53:15 EST 2016 ### Thanksgiving Breather Well, here I am alone on Thanksgiving, after having a wonderful daughter who is now 6 years old, and probably at about her most impressionable. It's the old out-of-sight, out-of-mind problem. She's with her mother and her mother's boyfriend for 5 out of the 7 days of the week, homeschooling, so the psychological shaping is mostly a product of her mother. I have very small windows, which means many things. These times are both very intense one-on-one with my daughter, and I can't screw up. Or if I do screw up, I have to correct course quickly. But I'm perpetually unprepared and behind the eight ball... because I'm a fucking over-extended wage slave with an oppressive mortgage and shitty commute with too little discretionary time to recuperate, rethink and pro actively shape. I need faster feedback loops between thinking and doing on the personal level, just like I advocate professionally. Faster iteration and better feedback loops help everything. Strong mental images to rally around. Mental templates. Places for things, and automatic decision-making helpers. It's fucking tough, as I feel my brain and my surroundings and my personal affairs have been horribly scrambled over the past number of years, and I have effectively lost who I am. This is most scary because it is right as my daughter is most impressionable, and I have to drag myself through sheer force of will back up to where I am something like an admirable force-of-nature who's got my shit together, and proceeds through life like an unstoppable force of nature, never wanting for resources or time to do the things I want to do. Shit, I've got some fucking climbing to do, and it's such hard work, that I find myself always looking for those siren's songs of distraction. And then responsibility layers upon responsibility, setting me back further. Shit. I think I'm getting either more lazy or tired as I get older. I'm approaching 50, and I still think about my life I think in the terms I was when I was 16 or 20, before my Dad died, and I felt like I was preparing to make my mark on the world. And I've been preparing ever since, and what's worse, been what effectively feels like derailed from those preparations, so that I'm going to die old and greatly unaccomplished. At least my legacy will be partially in my daughter. But I even have to be careful abut pinning hopes on that. I'm right back to not knowing what's important and why. Of course HER experiences in life are of the utmost importance, but what I'm going to be doing in the meanwhile, frequently helping her on her journey, is also quite important, and I'd like to have a little more freedom than I do currently. This writing is not doing. This writing is only thinking and lamenting, which is even worse. I made a commitment to being with friends tonight, and bringing some potluck food tonight, wine, pie and some pre-cooked rotisserie chickens. Get as much done before then as possible. Take the enormous symbolic pressure that you're building up here on yourself off of yourself, and work in those progressive 80/20-rule sweeps. You've done it to a fairly large degree already, just for walking around your place and stuff. Now, you can do it to the next degree and get your paperwork (maybe) properly together. I get stress even thinking about it. Shit, okay. Get those chickens while you still can, and then report back. -------------------------------------------------------------------------------- ## Tue Nov 22 10:06:32 EST 2016 ### Planning I just had an access token refresh error that stopped my overnight scheduled Jupyter Notebook job that uses the built-in sched module from the Python core library, along with the Google-provided Python client libraries for OAuth2 authentication, referred to often throughout this work journal. I have decided this morning to turn my daily process into a sort of internal-network YouTube channel and maybe podcasts and such, not sure yet. - Get an Internal Ziff Davis-only YouTube Channel going - Work through my issues and daily deliverables completely openly there - Talk through that stuff in a sanitized fashion here, so outsiders can follow along, just about. Nothing proprietary given out, but SEO as a field is evolving and I am free to talk about that. - Tidy up Levinux and Pipulate - Continue broadcasting on all my own "personal" channels when appropriate - Create a lot of cross-fertilization of "big picture" ah-ha moments that are only really accessible when you're consuming both - Don't expect a viwership on any of your channels. Continue in your daily work journal style, and to hell with whether this in itself is really even an audience development ploy. It's daily idea-capture and processing, mostly for my own sake. Consumable content of maybe some possible interest to some folks is just a byproduct. Pshwew! Yeah, that's about it. I am embarking on the "big thing" that I've somehow envisioned for myself ever since my days at the Commodore computer spin-off, Scala Multimedia, back before even their focus on the Digital Signage industry (chosen, so they could be all high and mighty and not have to deal with the much more finicky and intolerant consumer market), when there was a special halo or glow around insider tech knowledge and know-how, most notably how to use the Amiga computer in those early days. Only massive disappointment, punctuated with small flashes of hope, followed. And those Amiga days were my rosy reminiscence days, which I am led to believe is the new psychological term referring to those days in your 18 to 28 year-old range where everything you do is the best of all possible experiences you could be having, soon to become the good ol' days for which you forever yearn. Well, a big ol' key to happiness is to constantly have new days of future rosy reminiscence. That's the trick. That's the code that needs to be cracked for eternal youth of the spirit. In particular, try to make sure that the thing you are working on right now is just about the most interesting, engaging and important, with the important qualification of not actually being excessively emotionally draining, that you could be working on right now. It should be something that you would gladly "go into the zone" working on, achieving that feeling of wellness and craftiness and achieving a thing experienced by master craftsmen and musicians and athletes and coders and artists of all sorts who find artistry and beauty and worthiness in their endeavors. I saw all that in the Amiga computer, and felt truly privileged and honored to be a part of that movement in computing. After having read Steven Levy's Hackers book, I understand the lineage and legacy of computing technology that again in a few minutes, but if no luck, use my boss' office. Think! The script! -------------------------------------------------------------------------------- ## Thu Nov 17 09:59:57 EST 2016 ### SEO 101 Okay, at least I got the new journal entry started before 10:00 AM... by 3 seconds, haha! I have a few important checklist items to make sure I do... Preferably, today. My followers are going up all around. It's at a very small rate, but it's definitely detectable on both Twitter and YouTube. And my daily listens on my podcasts reaching the 400 downloads level, well, maybe I'm striking into something people've been waiting for. Reality Podcasting. Of course! Hardly knew that's what I was doing. But it's like reality TV without the artificial drama nonsense. In fact, I have almost like an allergic reaction to drama. And artificial drama is my least favorite kind, and so follow your instincts and guts for a few moments in free association writing, then flick over to the rationale. Hmmm, I think the first order of business is getting an SEO 101 Deck started. -------------------------------------------------------------------------------- ## Wed Nov 16 13:07:44 EST 2016 ### Google Search Console I more and more often appreciate the value of a live SQL connection to a very standard SQL environment. In this case, I'm hitting Amazon RedShift, which is essentially PostgreSQL for the Cloud. It's cool. I'm not an inserter of data yet, and am mostly a data-selector for reporting purposes, but that's about to change. I'm about to be carrying out the one project that all Webmasters and SEO should consider it beholden upon themselves to carry out. It's storing the impression that Google has of your site, as reflected by the Google Search Console services, previously known as Google Webmasters Tools, in such a way that you can do trending and analysis with it beyond the 90-day rolling window for which they store the data, or at least make it retrievable through the API. Any Webmaster or SEO worth their salt should at least be doing that task. It's baseline stuff, retrieving what Google's trying to give you, keeping it long-term in a nice efficient and easily-actionable way, so that it's not lost to you forever once it slips off the end of that 90-day window. Okay, one more RankLayer. Crank it out ASAP. It's getting into my muscle memory now. Wed Nov 16 17:42:30 EST 2016 Another RankLayer deliverable complete. -------------------------------------------------------------------------------- ## Wed Nov 16 10:39:24 EST 2016 ### 24x7 Stability Next step... hmmm. So much is nearly working. It's time to hit a few things home. Work like you like to work, and draw the big picture, then zoom quickly into stunningly specific details that serve as working examples, then zoom smoothly back to the big picture. Rinse and repeat. Get stuck in an infinite loop that is stable and well thought out enough to just keep running. A place to run code 24/7. Fortunately, the world these days provides us plenty of opportunities and variations and flavors. A few meetings this morning and a few more I'll be in on regularly from now on. Think through the rest of the day. It's got to be the scheduled data-pull of last-full-week of data... hmmm. Last available full week, then we control what day we pull it? Probably, yes. This is all relative and directional. I want to combine the "test suite" work that I want to do for Pipulate with the weekly directional Google Search Console stuff. It's getting urgent, as I have promised something to the PCMag staff "early" this week, and it's already Wed. I actually have a good system for pulling the data. The only thing I'm somewhat unhappy with is the new functions for monitoring and processing a feed. I have to get to that FAST, and that's also the portion that needs testing. The monitoring of the feed can BE the testing. Hmmm, but the way the configuration values get set, I'm also a bit unhappy with. -------------------------------------------------------------------------------- ## Tue Nov 15 09:48:57 EST 2016 ### Rank Layer Okay, focus on the RankLayer until it's done! report.py wasn't running for a few days. Kicking it off manually, but it is now critically important to get that off onto the AWS instance that DevOps has provided me, at whatever personal cost! I'm already making my decisions about what's important by not spinning by the management office to drop off the maintenance check today, because I over-slept (desperately needed the sleep) and wanted to get in on time-ish, fighting the rain and getting done a few urgent deliverables that I have to get done. I'm in the critical path of a number of things. I need to switch into "high impact player" mode. Shit, don't over-promise and under-deliver. That would be a disaster. Tap capabilities you know you have in you, but rarely tap. But that sort of resource-tapping takes sacrifice, and it's almost always free time, relaxation, and even a sort of necessary-for-sanity recovery. Going back to sleep at 6:00 AM after already fighting off the cats from a 3-AM wake-up, and not waking up until 8:00 AM really hit the point home. I LOVE to get enough sleep. I feel the canker sore in my mouth finally healing after a week, given the stress of the election and other personal things going on in my life, and the intensity of the weekends with Adi. I have to take care of my just mere ability to function as my first priority, my livelihood and ability to provide for Adi second, and then finally, all the little amenities of life that inch it from misery towards pleasant. Is this depression? Nope, just life. Okay, onto the RankLayer. Do I do it any different than the traditional approach? Maybe this is my chance to do it all with Pipulate and do it in a great, scalable, shareable way. Or maybe I just cobble it together again in Excel like any other. Either way, get your latest edits committed to Github. Done. Avoid distractions. The largest distractions come from your phone. Okay, don't try to innovate here. Just plow through the work! -------------------------------------------------------------------------------- ## Mon Nov 14 09:58:39 EST 2016 ### The Diamond Age Influencing Me I love clean, smooth merges. Read about The Spy Who Couldn't Spell from Apple News this morning, promoting the book of the same name that'll be published soon. It's Monday Morning, and I have to make sure my automated tasks are still running, and if not, why. Unpinned Outlook from the Taskbar! BAM! Switching to the gmail UI as much as I can now. Reserving rooms is easier. I have to get my Monday Morning report written and done. Things are getting interesting. Recorded both a video for YouTube (even with the zit) and audio for my podcasting channel. The former is about 6 minutes, and the later about 20 minutes. They each sort of document the length of my commute. Sometimes I stand a few moments at my destination before I walk into work or down into a subway, but that's offset by the fact that I often don't start recording until a few minutes after I started walking and realized I had something to get down. Finished listening to The Diamond Age, and I'll have to re-listen to it again and again. Mon Nov 14 12:57:29 EST 2016 Do a "latest" data pull against the property you're going to do the SuperNav for. Mon Nov 14 17:28:09 EST 2016 All over the place today, but not totally unproductive. Moving endeavors forward. -------------------------------------------------------------------------------- ## Sun Nov 13 10:19:16 EST 2016 ### Adi is 6. Under-promise, Over-deliver Pshwew! I really thought that having a daughter, especially as she reaches 6 years old, would really catalyze and clarify my thoughts, but it's as cloudy and scattered as ever. These weekends with her are so nice, but they work entirely diametrically opposite to my need to get and keep organized, even just enough to be a responsible adult. But that's all on me. That's my own fault in every way imaginable, and it is entirely up to me to fix that, whether it's allowing Adi to participate in my getting organized or having to only have half-weekends with her occasionally, so I can get my things done. I do not have the energy I used to for doing all-nighter after all-nighter to get stuff done. It's really amazing I was able to keep that sort of approach to productivity going so strong so late into my thirties and early forties. Now, I'm entering my late forties, and I feel that afterburner energy reserve getting smaller and smaller, and my reliance on it being revealed as more and more dangerous. Under-promise and over-deliver. Don't let the expectations of the people around you get out of whack. And more than anything else, don't let females -- your daughter or otherwise -- walk all over you. That was your big mistake of previous years. I should do the pet cages soon. I also have a baker's shelf I want to put together. Going to watch Spirited Away a second time with Adi -- good movie for her to absorb the messages from, all things consider. It's the parents who are gluttonous and can't control themselves, and it's up to the young female role to save the day, in a very Alice or Dorothy-like adventure. Solid choice. -------------------------------------------------------------------------------- ## Fri Nov 11 08:26:31 EST 2016 ### Positive Feedback Loops for Adi Okay, you need to keep determinedly on your path. It's a good path -- at least, most of it. And make it as totally awesome as you can for Adi. She needs to face some challenges, and always know that the total love and support and encouragement is there for her. Adversity can shape great personalities, but then so can nurturing and a guiding hand to make sure that the feedback loops are generally positive, in both the accelerating sense, and the warm and fuzzy sense. 1, 2, 3... 1? Fri Nov 11 12:30:56 EST 2016 Okay, that meeting went really well. The question is: what now? Look at the mobile deck again. Okay, came up with the concept of "Pruning and Training" which may work better than decrufting for analogy purposes. Think! What do you REALLY want to do here? Fri Nov 11 16:04:03 EST 2016 I have just thoroughly and completely ran out of steam today. I am going to have to make sure I get orange juice, milk, ramen noodles and other essentials on the way home with Adi today. I want for us to be able to eat without having to order every time. I am going to try using PowerPivot. Downloading and installing now. I believe it is in-memory SQL Server 2012 tricks. Downloading all the data for this new mega RankLayer project, and going to try using PowerPivot to make it nice and snappy. -------------------------------------------------------------------------------- ## Thu Nov 10 10:33:29 EST 2016 ### RankLayer Deliverable, Don't Forget! Pushed out one podcast. Looks like both my YouTube channel and Podcast are both becoming gradually more popular. I have to keep the tension in the machinery now. I have stopped those talking-head coding videos lately. That always seems to be the way once a project approaches a certain level of complexity. But that's nonsense. It shouldn't be that way. I should be able to turn just about any endeavor I sit down to do into a video -- or at least, after one or two test-runs, I should be able to do that. Ah, there's the rub. It doubles or triples or quadruples how long it takes to do a thing, thanks to documenting it. Hmmm, okay, for today... ? It's all about prepping for my major stakeholders meetings tomorrow. I have something on the calender with my boss for 2:30, and by that time, I should be able to run down a number of selection choices. I basically have under 4 hours to work with, so I had better get hopping! Two approaches: - Trending from Search Console that also answers other questions I need to answer for this stakeholder - Gap analysis from the Darryl/Jenn method that shows us traffic-gaps to attack. Hmm, this could also lead to the first round of SuperNav adjustments. Okay, 1, 2, 3... 1? First step is to run a Search Console query that does the trick that I do for the SuperNav's in the first place. I'm pretty sure I have it easily accessible in Pipulate. I have to clean up the process around that. Thu Nov 10 12:54:25 EST 2016 Ugh, I had to fix some code to make the Google Search Console data pull work again, and it's already 1:00 PM, and I have my meeting at 2:30. It's going to take some time to finish this data pull. It's at Range 19 of 318, filling the GSC data pull into Google Sheets. I will be able to correlate that with the data from the previous sheet that informed the SuperNav project. Okay, I have that file loaded as well. This will take some time to finish, so use your time to reproduce the gap analysis process of the other side of the equation. Thu Nov 10 13:20:50 EST 2016 Ugh, the long data-write to Google Sheets failed (~15K rows in 50-row chunks), and so I'm re-executing with a local CSV file output safeguard. Perhaps I should have done that from the start, and that should probably be designed into Pipulate as just a standard thing, so that even when a big chunky update to Google Sheets fails, you're still sitting on top of the data cached locally in the repo directory. Yes, that seems like a good idea. Now onto the other type of deliverable! ...oops, not before I change the csv writer command to "w" instead of "wb" and explicitly use utf-8 encoding, etc. Ugh, once that CSV-outputting is done -- especially on Populate() functions -- it makes sense to make actually trying to do a chunky-fill into Google Sheets optional! Avoid a lot of unnecessary latency-pain, if you don't really need that extra convenience. Of course as an alternative, I'm going to be uploading a ~15K row spreadsheet. And finally, I've got the ~15K rows of data into Google Sheets. Next! Find example of last months deliverable. Okay, got it. Now plow through it. ahrefs / Site Explorer / Organic keywords (on left) / Export (link in upper-right) Export all the Organic keyword CSVs as you want on which you want to do the analysis. Inside Excel after setting up the Pivot Table, there's a few important touches: Design / Report Layout / Show in Tablular Form Design / Subtotals / Do Not Show Subtotals Highlight row below where you want to freeze rows, and: View / Freeze Panes / Freeze Panes Okay, I've got the documents ready for both. Ugh! Had to re-process them from a mistake I made. A bit time-crunched, but it's turning out well. Great data and great findings. -------------------------------------------------------------------------------- ## Tue Nov 8 09:29:02 EST 2016 ### I Can Push Out Podcasts From Commutes Okay, I already pushed out 2 podcasts, and am journaling already by just 9:30 AM... AND I voted this morning. Pshwew! Okay, let me think. Today? Don't limit your thinking, but DO limit your distractions. You have some important deliverables to deliver, and no time to dawdle. Start the process of having little else of importance in your life, but for your way of generating income, because that falling apart would be disastrous. Use this realization to focus and catalyze your behavior, so that you can recount every single day like an interesting chapter of a story. Yes! -------------------------------------------------------------------------------- ## Sun Nov 6 09:05:07 EST 2016 ### The Diamond Age, Lots of Lessons A great Saturday with Adi, not doing much. Rachel called to talk with Adi and was quite surprised about how well her play-date had gone with Lily. Okay, let me think. Oh yes, it's been coded into me that I should read Diamond Age again, as Adi was reaching an old enough age for it to make a strong difference in my actions and behaviors. Neal Stephenson (both things spelled differently than I'm used to) projected his thoughts into matter, and went beyond to reach down into the soul of... what? Probably tens of thousands have actually read Diamond Age. Hypocrites! Mr. Hackworth... yep. You've become the very thing you're looking for, and spend far too much time looking outside yourself, and not inside yourself for who you really are. And distance and separation can simply be a set of creative constraints to tease out inspired work. Be King Coyote! Think of yourself more that way. Don't forget humans have our evolved-up-from histories. Our missteps along the way is what make it interesting. Nobody ever said that it was easy to stick to a code of conduct. It is how we conduct ourselves in that struggle that may affect how we may be judged by a higher power. I love teasing out the extra levels of meaning and awesomeness from this great literary creation. It's also noteworthy that I encountered this book through explaining what I'm trying to do with Adi to my friend and fellow professional SEO and actor, Boris Zilberman essentially insisted that I read this book, Diamond Age. I was already familiar with Neal Stephenson, even if I can never quite remember how to spell it. Hmmmm. Yes, my timing is good. Adi is 6 years old tomorrow. Some time in my younger years, I got it into my head that a person is basically fully shaped into who they are going to be by the time they are 6 years old. Meaning that the time from 5 to 6 years old is incredibly important, and what happened with Adi and I essentially amounts to a greater separation than anything that has occurred between Rachel and myself, for this is the very time when Adi is being shaped into who she is. And what she's been made to feel -- probably as much due to a result of my own behavior's as my soon-to-be Ex's. And this makes me choke up and almost cry nearly every time I let this thought sink in. And so, I spring into action the best I can by giving her my every and entire weekend, from as early as I can get her on Friday to as late as I can return her on Sunday night, and spend excellent one-on-one time with her through the weekend. She stays up late into the night, insisting I do so with her, even when I almost physically can't stay awake anymore, and then I wake up fairly early by the cats trying to wake me up. And all I have to do this weekend with Adi's 6 year old birthday coming up at Chuck E. Cheese in Staten Island is get her out for about 10 gift bags for her guests. Adi has various groups of friends. It's interesting. Mapping challenges. I need to help Adi deal with those mapping issues the best I can, as seeing and keeping these relationships available as modeling-tools when engaged in thoughts concerning them is one of life's great survival tools. Hmmm, so is reading. Yes, it is an enormous responsibility of mine to help Adi read. And maybe today I do some more quality reading with her. I both have the book Ghosts (written by the same writer as The Babysitter Club) which I got as one of the gifts for her 6 year old birthday (along with the Yo Kai Model-Zero Watch), and the Dick and Jane book that Lily's mom gave me. I need to work on these experiences with Adi, as a number of these are a big part of my responsibility. Hmmm, okay. The wrestling thing. I have to deal with that. Teach Adi Martial Arts! How can I be so dumb. I have to give her an opportunity to do some real practical hitting. Yes! I think I've got it. -------------------------------------------------------------------------------- ## Sat Nov 5 13:43:31 EDT 2016 ### Be That Strategic Thinker Again At home (Inwood) and had a good Friday night with Adi and her homeschooling friend. I was going to go to the Catskills with her today, but then decided no. Also, I'm tying this right now on one of the old Macbook Air's, which are so superior for typing than say the Microsoft Surface Pro that it's not even funny. The keys feel better. I can actually use it on my lap, which despite what many people say, is something that's actually done quite a bit -- especially on these really ultralight formats that the Air popularized. Anyhoo, I'm going to try to get some stuff done around the apartment today without actually fighting Adi over it. And I have to get a number of things going to get into that zone while still being present for Adi. She's playing Perplexus, which is just perfect. Now, she's doing Lego's. Basically, whatever I start to clean now is what she's interested in, and "locks me out" of the real possibility of cleaning and organizing that thing. So either, it's confrontation and explanation and coercion, or it's shifting unpredictable strategies, like Bruce Lee's water. And of course, I choose the later. I have to use shifting and revised strategies, but that in itself is draining and not feeling very optimized or highly-effective. I've got some good core abstract rules, like: - The 80/20 Rule - Using Your Environment AS Your Checklist - Forever Improving Places for Things -------------------------------------------------------------------------------- ## Fri Nov 4 10:03:21 EDT 2016 ### Just a Work Entry Ugh, okay. Lot of time lately in "PowerPoint" decks, although really, it's Google Sheets. Google is using variations of Microsoft's strategies on Microsoft to release its strangle-hold on the Office community, and it feels good. Okay, you have to unify what you're doing for your employer now with what you're doing for yourself and your self-image and your long-term professional value. Forge your optimal path through the conditions put before you. Fri Nov 4 14:28:48 EDT 2016 Okay, a fairly intensive day so far on the presentation. I can see why people say "deck"... presentation is too many syllables. Hmmm, next step? Gotta do some tracking pretty urgently. Shift gears a bit to coding just to clear your mind. Oops, no, make that the weekly report. Done. Next? Fri Nov 4 16:27:18 EDT 2016 Okay, going to pick up Adi early. It's the weekend before her birthday, and I want to make the most of it. >>>>>>> 57e0c6f455c04858a821e1c8f82e94caf630374f -------------------------------------------------------------------------------- ## Thu Nov 3 11:39:18 EDT 2016 ### Tales Of The Self-Determinant cd ~/ cd reponame git pull commit -am "Oh, I left some stuff uncommitted." git pull vim filewithconflict.txt [move some stuff around] :w :sh git commit -am "Two paths shall merge into one." exit And back to editing the journal. Haha, things are interesting these days. We're at the cusp of a technology acceleration curve. Some folks who are akin to the James Bond super-criminal type, but for good (let's call them Tony Stark types) emerge, like Steve Jobs and Elon Musk. The right people at the right time to drive us forward as a society, striking the metal while it was hot, in order to figuratively shape the human global societal organism that is us primitives. I have trouble focusing and getting stuff done. It is a problem, but I think I have a way to deal with it. I have to empty out all queue's and use my surroundings and environment every day as my checklist and cue, insofar as what to do next and why, and this has been a huge failing on my part of late when it comes to certain adult responsibilities. And so, I ratchet-up my intent to be an excellent Daddy, and likewise to be a star SEO employee at a for a print publisher successfully adapted to the online world. And also, I need to take care of back taxes, divorce, selling an apartment, and keeping pets alive, and maybe perhaps some chance of a social life too. God, imagine me getting back onto my feet on all these different fronts. And that's a matter of clearing out the old, gummy queues that are holding you back. You're living in another age, just like my dad did. Ugh, okay, your selection of the field of SEO has forced you to be in the mental state of always re-thinking the rules and conditions that govern the game... and life, for that matter. Optimization is core and essential to life. Things happen by chance first, and they are rarely born optimized. Life was that way, and the development of the digital nervous system infrastructure of the world is that way. Apps built on local player platforms sitting on globally interconnected TCP/IP networks, sending streams of packets that are dynamically routed point-to-point, according to algorithms. And on top of that, you can build streaming video apps like Skype and FaceTime. Ever so slowly, and not optimized at first. But feature-creep burdens an organism, and eventually various forms of competition kick in, and those most suited for survival tend to propagate across the network, displacing the old with the new, more suitable for survival for the current conditions at this place and time that happen to exist, such as whether you're at the front or tail-end of an ice-age, and such. We're at the tail-end, and still have our polar icecaps, as tenuous as they might be. However, the original point still stands. Radicalism introduces new features, but the slow-burn of competitive pressures both optimizes and tempers those features into the average population of on-average most suitable for survival variations on the other recent life-forms. And that's life. That's us. That's why Homo Sapiens over the seemingly more robust and just as intelligent Neanderthals. And how we're actually also the Neanderthals, bred back into us, because we really weren't so different. And I bet there's a lot of interesting lost-to-history stories back in those days, wow. Lineages break off as small populations, separate, grow big, and then re-encounter each other generations later, as an almost different species. Those must be some interesting family reunions, if the simple differences between us today can be so catalyzing, can you imagine back then before science, and superstition had little opposing forces to keep it in balance. Literally, anything you could get away with could go, if you can convince the tribe that yours is the word of the divine mysterious powers that are behind everything. And so, here we are today. Richard Dawkins and Stephen Hawking seem to be on some sort of hate-mail campaign to the divine, and scientists who really approach the question scientifically have to come to the conclusion that because the scientific method itself is within a certain arguable frame-of-reference, that in the end a line of belief called solipsism might indeed be the correct thinking, and that there may indeed be no objective time and space in which we all actually co-habitat, and a completely subjective and separate reality is playing out, almost as if in a virtual-reality nature, in each of our heads -- that is, if anyone other than me really exists. This all stems from the problem of induction. The problem of induction is that we only ever know what we know through a series of sensory and motor apparatus, which for all we know, could be lying to us. And as we learn about how things work on the subatomic scale, it almost starts to look that way. The probabilities that just collapsed that resulted in you, most unlikely might I tell you, ending up here reading this right now are so astronomically unlikely, as to be effectively impossible. Yet, here you are. And in all that impossibility, how much more likely is it that we're all occupying some collectively "real" reality than it is that we each occupy our own? We are effectively trying to understand and pass judgement on something that lives outside a system that we live within. It's the Flatland argument. Be careful with your secrets, because you're visible from every angle if you go only one more dimension out. The privacy concerns of media companies today would not even register on the meter compared to how naked and exposed you are before 4 spacial-dimensional beings, where 'dere be tesseracts. You yourself are a programmable robot. We are all made up of many routines. Routines layer upon routines upon routines, until we are an average of all those running routines, which together cross-validate and bring greater meaning to each other. And our rules as complex as they might seem to be are nothing compared to the everything-cubed complexities that the possibility of another spacial dimension actually existed. But back to the reality of the here-and-now. Nature forged one way of doing things. Here we are now in the position to coerce nature into having another way of doing things. - The rules of the universe make us - We make tools - We make tools that makes making better tools easier - Robot overlords I am SO GLAD that I read Iain M. Banks' The Culture series. It is such food for the soul. We are an interesting lot, us humanoids. Oh, I'm also glad that I finally read (listened to) A Wrinkle in Time again... time to be a truly excellent human being. No excuses. Just do the hard work it takes to get and stay on track. Everything should be love-worthy, and when it's not, take those steps you know often lead to them becoming love-worthy after all. Look around, and use the real physical world and your very environment AS your check-list. And know the bad kind of distractions when they intrude in on the edge of this, because getting back into that "zone" state, is one of the most important skills of the smart and self-determinant. -------------------------------------------------------------------------------- ## Wed Nov 2 20:18:01 EDT 2016 ### Back to Diamond Age! Back at home, and doing a portion of this equation that I've been missing a lot lately -- making journal entries from home. Hut! Listening to Neal Stepenson's Diamond Age on Audible... what a brilliant branding move, naming that company Audible. It's perfect. I could have just as easily said audio book. Anyway, it's about damn time I got back to this with Adi about to turn 6 years old... damn! I almost screwed up imprinting on her... but absence makes the heart grow fonder, so maybe my separation actually plays into imprinting effectively on her. Listing to the accounting of Bud getting his headgun. It's hard to follow the establishing of these radically different future-worlds with books like this. I actually couldn't do it with Dhalgren. That was a book that defeated me. I may try taking it up again, but wow. I can see how that sorta influenced the cyberpunk movement. And the fear of nuclear annihilation again are quite disturbing as new warhead distribution strategies are added to a new generation of missiles. Ugh, but so many more reasons to be positive about the future. Many people don't see the pictures being drawn if it actually all working out, in one of many infinite possible ways ahead of us. Oh good, the book is narrated by a female voice, and all the explicatives are in place, and all the grittiness is actually kinda sexy. But I need to immerse myself into something motivational, high tech, connective with my daughter, and somehow also helpful to my career in SEO (search engine optimization), which I am currently blending together with Python programming, now that I'm actually capable of doing a thing or two. But right now? Next thing? Be a bit more productive at home than you usually are. Try to get SOMETHING done on the home-front. Don't be such a lazy-ass. This is a book in a rare class, such as The Bohr Maker by Linda Nagata that I listened to way back when. My rosy reminiscence days included quite a bit of SciFi reading, and preeminent among them was this odd book that first indoctrinated me into all that probably not far in the future clone stuff. And so, next step? Don't go to sleep right away... or maybe do and wake up VERY early, like on the first cat awakening. The fact that I can even entertain this possibility should indicate to you that I am currently NOT in the zone. But I want to get myself into that zone, where the need for sleep almost starts not mattering. But getting there is hard. Some dots are hard to connect, and their interconnecting lines intricate and energy-draining, even just to peer into how they're supposed to connect. I am at that point currently with my system. And I'm listening to the part near the beginning of the book where a nano-coral fantasy-land island grows from out of the bay. I know it's getting close to where that royalty guy meets the architect guy. I myself would like to cast myself as that Coyote King, very akin to what I'm planning on doing with telepresence with Adi. I desire for her and I to be at the bleeding edge of what's possible, creating that fantasy-land using tech in new ways for the first time. -------------------------------------------------------------------------------- ## Wed Nov 2 15:51:57 EDT 2016 ### Desperately Need a Clean-Up Okay, I got my "manual" spot-checking done today. I have to really deliver on the tracking for the menu project. And so... and so... it's the positions of a bunch of particular URLs over time. This is all like SERP-tracking, but pulled from directly out of Google Search Console. Keep this stuff simple. Think it through. Make it a fairly manual process at first, before you try to automate it. Work off of some other work that is already incredibly similar. Get compounding returns! Okay, I promised a very major stakeholder a report today that I really want to get going. So... First, I need to just address some usability and simplification issues here. I'm going to tear out every workflow behavior that's not dead-simple default. I want my system to have a somewhat pure applies-to-everyone center to it, with the customizations easily chopped-off (even if in the repo) appendages, and the biggest violator right now is extra workflow stuff. 1, 2, 3... 1? Create a new notebook called workflow.ipynb. Ugh, I undid all that. Scope and dependencies. Yuck! Think this through. Do a nice pass of housecleaning that sets this up for sharing. Speaking of housecleaning, you VERY MUCH have to do ACTUAL housecleaning when you get home, no exceptions. I took my discretionary time yesterday to make sure the lizard got fed his crickets by overshooting on the subway to the Bronx and walking back. But time is running out this week, and before Adi's birthday. Make sure you have it off! Okay, updated request in HR system. What now? What will be MOST USEFUL moving you towards most of your goals? I still want to externalize workflows. See if you can just do the whole get_workflow function and handle it as an imported module, so you don't have to deftly separate internal dependencies. Yeah okay, it's gotten to the point where I want to make pipulate.ipynb something you no longer have to edit, or very rarely edit only when moving the whole system forward. This is part of the separation -------------------------------------------------------------------------------- ## Wed Nov 2 09:56:48 EDT 2016 ### Did Today Count? Make today count. It's already November 2nd, and I have to dig out the more true positives on my feed monitoring project. I had barely gotten all the moving parts working together correctly, and now I'm dealing with issues of timing that violate the top-down, left-right sweeping motion of Pipulate -- at first glance. This is one of those cases where you have to look carefully at your primitives before you dismiss possibilities and go start doing more complex and convoluted code than you originally intended. Okay, how to describe what just happened. In accommodating all the various needs of the project, caching, proxy'ing, constructing different types of Google queries, parsing out the results, comparing those results to datasets, spot-checking the validity of the dataset matches... pshwew! I got it all working... almost, but for the timing of the test and the accuracy of the test. Shit, I've got some clean-up work to do in process, but in the meanwhile, I need to do the work manually, and I want to do it in a SMART manual way, to set the stage for much cleaned-up and much organized SERP-monitoring process. 1, 2, 3... 1? Hmmmm. Well, start the manual checking, of course. But wait, first clear the pipes for doing architectural improvements as you go. Embolden yourself! It's a very small cost to get potentially big pay-offs. Well, let's make pipulate.py almost unnecessary to edit, first of all, except maybe for expanding how the system works and squashing bugs. Besides that (and I mean adding new workflows), it shouldn't need to be edited. Yet, looking at it directly should show you how workflows work, and where the customizations are being loaded from -- a file right in the same repo, of course, and probably even part of the repo, because still customized workflow EXAMPLES should still be part of the repo, even though that's the file that users are themselves expected to customize. And they always could change what "default" is in pipulate.py, if they ever want to slam something out quick-and-dirty. No! No! No! Distraction. Chasing the rabbit. Don't. Not now. Get answers, immediately. But smartly. Immediate and smart means constructing things in multiple columns of process-exposing steps: 1, 2, 3... 1! Okay, make a function that constructs the query given input provided by a column... the column's name should be the same thing as how it will manifest in the query... ah-ha! Now, use your kwargs capability to fill in the rest of the parameters! I need a way to generically build strings from what's coming in from the kwargs parameter. I think I want to create a series of proto functions that do some really basic stuff that's super-useful for testing and development, but which get hidden or isolated later on once you're happy with the end output. But until you get to that point... Okay, it's working quite well. I made the simple function kwargs, which does: def kwargs(**kwargs): return Response(ok=True, status_code='200', text=kwargs) This is as primitive as it gets with my system. I'm really following the wisdom of the massively popular Requests package, in choosing this response protocol. -------------------------------------------------------------------------------- ## Thu Nov 3 20:31:26 EDT 2016 ### Google Slides Well, I put in a pretty good day of PowerPoint'ing, but ironically, this time it was in Google Slides. My Pipulate deck came in great handy, mostly in the sense that I simply knew how to use it, and have become quite adept at it. Sheesh, I really should polish my gimp skills. I forgot how friggin' powerful The Diamond Age was, and how wonderfully perfect it is for me listen to it again right about now. I really need it. And now, to clean my place in order to do a litter better for my daughter. Let me take advantage of my situation, and navigate out of it and above it that I will forever be able to use as an inspirational story for my daughter. That's how much I love her -- enough to get my case cosmically together. -------------------------------------------------------------------------------- ## Thu Nov 3 12:34:20 EDT 2016 ### 80/20 Rule - Time for an 80/20-rule pass over things. - Multiple 80/20-rule passes is all that's ever really possible. - That's because there's only 24 hours in a day, and we all have to sleep. - And this methodology can be brought to all types of work. -------------------------------------------------------------------------------- ## Tue Nov 1 14:55:26 EDT 2016 ### Make The Ad Hoc Stuff Easier Newsflash! Credibility on the line. My Feed-monitor is not coming up with enough true positives! I have to have additional Q/A and ferreting out of more positives. There's issues here involving: - Time-delay before performing the test - The method of the test / - verbatim headline - from site-X linking to site-Y I have the requests queuing up, and I have to be able to dispatch with quickly, and track quickly! Go, go, go! Tue Nov 1 17:05:54 EDT 2016 Okay, I'm getting certain notions clear in my head about making ad hoc work easier. Even with as much as I built, I'm still running a kind of desperate last mile to meet ad hoc requests, and it should be almost mindlessly easy at this point. Keep things easy to read. Do not obfuscate, if you can help it. Okay take it home. Be serous about doing some stuff at home to clean up and make as easy as possible the more advanced ad hoc stuff. -------------------------------------------------------------------------------- ## Tue Nov 1 10:14:00 EDT 2016 ### It's Not What's Important. It's What Next and Why? What to do next, and why? I used to always ask myself what's important, and why. These days, it's a bit different, I think because I'm a dad, and the answer to that is self-evident. Certain things instill a sense of finality, and having helped bring another human being into this world is one of those things, and so the question "What's important and why?" has morphed into "What to do next, and why?"... with the "why" question being already half-answered implicitly, making the question much more tactical, and too uselessly high-level for everyday use. And so I repeat, what to do next and why? Well, the next step is practicing your mad PowerPoint skillz, yo! -------------------------------------------------------------------------------- ## Mon Oct 31 09:52:29 EDT 2016 ### Happy Halloween Wow, not a single journal entry from home. We did do Forbidden Planet at the public library, but not much else this weekend. Today is Halloween. Adi is getting plenty of Halloween this year, and the important thing is that I spent some really good time with her. It's getting better. I'm helping her mature through this phase where she really wants to rough house with me like she's still a 4 year old. It's hard to let go of, but it's absolutely essential... for my health, primarily. I don't have her energy-level. But I try to make sure she gets some way to work out her energy. I've got to get my weekly report out. I need to think through and clarify things in my mind. I'm feeling a bit out of sync at the moment. Haven't conducted all the dots like I had wanted. Zeroing in once again, but still not as hyper-effective feeling as my Scala days. -------------------------------------------------------------------------------- ## Fri Oct 28 16:57:18 EDT 2016 ### Forbidden Planet Tomorrow, Maybe Okay, maybe I should just free-form write for a bit. See what I can extract out of a bit of free association. I'm hardly even free associating, as I'm not starting with a core idea... oh wait, yes I am. It's October 28th. Halloween is Monday, and there may be activities this weekend, and I should be a little more prepared, such as TimeOut New York for kids, or maybe Facebook Events. There's that thing October 29th, which is tomorrow, which is tomorrow 2:30 to 4:30 at the Inwood branch of the New York Public Library at 4790 Broadway. Donna KS is suggesting I do it with Adi. Oh, Forbidden Planet! Yeah, that's an option to get us out of the house tomorrow. -------------------------------------------------------------------------------- ## Fri Oct 28 09:23:30 EDT 2016 ### Surprise System Reboot Overnight & Lost VirtualBox My system restarted overnight, and my VirtualBox was marked as Inaccessible, by simply not being found. So, I went into that directory and found a temp and a previous version of the virtual box... uh, definition file? Too small to be the actual virtual disk image, so I did a simpy file copy and started that up, and it appears to have restarted just right. I have to generate the reports now, and also get Jupyter Notebook running with the scheduled tasks. I'm dealing now with both virtualenv for Python 2.7 with the scheduled reports, and conda, with Pyton 3.5. It's pretty easy to get the reports going again, as the virtualenv process is super-well documented on the net. Miniconda without paths set, not so much. But well documented here. Maybe I'll set that path, one day. source ~/miniconda3/bin/activate py35 ~/miniconda3/bin/jupyter notebook ...and then just surf-to, load & run pipulate.py with correct parameters. Oh nice, the job I want to run was sitting there in a pipulate directory named after the particular job. That's a VERY STRONG thing about the system. One job, one pipulate directory. It actually quit out on a permission problem. I edited in a try/except to get past it, and re-started it where it left off. It just goes to show that I should probably occasionally manually watch these jobs run anyway, so this may have been a good thing this morning. Maybe today is the day that I log into the AWS instance I was given and start getting some redundancy in the system, at very least. Fri Oct 28 15:53:33 EDT 2016 Okay, most of today went into massaging a presentation into better form for a 1:00 PM discussion. It's almost 4:00 now, and I'm almost no further along. I need to clear my head and focus, because I only really have today and Monday to finish this thing, before its ready to present to its first round of intended audience. Hmmm, sent out one email to a property-manager I don't often interact with. Fri Oct 28 16:52:47 EDT 2016 Okay, I'm totally running out of steam. I need to go meet Adi. I also need to think through next steps. -------------------------------------------------------------------------------- ## Wed Oct 26 16:19:24 EDT 2016 ### Next Steps I got a good chunk of work done in the full-time feed monitor & processing, but I still hold hope of doing something that applies a bit more generically to the world. Wed Oct 26 17:04:27 EDT 2016 Okay, I'm actually starting to use the "Gist" feature in Github. Seems a pretty reasonable alternative to SimpleNote for things I don't mind being public. Ugh! JupyterLab is another one of those rabbit holes I'm being tempted towards, but I will resist. I connected a few dots today, and I think I want to continue on that path, taking the next steps. Hmmm. Okay, I need to give this some more thought, perhaps on the way home. Just think through the TYPE of experience you want folks to have with Pipulate, primarily your co-workers, but then also in-house SEOs, and then finally, the great masses on Github and other public sites who could get tuned-in. The trick is to make it something worth tuning-into as a side-effect of meeting the needs around me wonderfully. For the latest work I'm working on, you need: - A bank of fairly exclusive web proxies (if doing SERP-work) - A place to store a fairly sizable amount of screenshots (if doing SERP-work) - Google Sheets (for most jobs) - An alternative to Google Sheets (for larger data) Okay, wrap it up there and go home and think. Make sure you talk to Adi tonight, and show her the moon-sand, and maybe read her some comics. Generally, make her feel good and glad when I call. Project feel-good, and make HER feel good. Don't be a dumbass. -------------------------------------------------------------------------------- ## Wed Oct 26 10:14:29 EDT 2016 ### Magic Numbers of The Last Update Chunk Pshwew! The A-train was terrible this morning. Okay, I put a good deal of time into the "what to update" request from a major stakeholder yesterday, at the expense of the SERP machinery. And I need to get to the serp machinery stuff urgently, breaking down the tasks remaining into discreet parts, each of which I hit out of the ballpark on implementation. Rule #1: Condition your mind to focus. Even the word distractions is distracting. Find the love means zeroing in on the task and hand, and finding why you'd rather be doing this one particular thing than anything else in life -- except of course at this time, being with my daughter Adi, which I would take in a heartbeat over SEO, focusing on science and strategy and happiness. And so, there is SOME overlap here. I'm going to need to instill into her everything it takes to funnel that primal animal force of reptilian automatic behavior into real, solid accomplishment stuff, and not the wheel-spinning feeding the primal animal needs. You can feed those needs and improve your place in life -- and by place in life, I don't mean class-shit, but rather just being happing with yourself, and your place in it all. Okay, so with that out of the way (for now / re-visit the thought soon), onto the work of the day. Okay, let's state the goal I want for end-of-day. Have jobs running that: Monitor a data feed (already done), and always targets its data to the correct "tab" in Google Sheets. This could just be a clever way the workflow is set up, since the targeted tab is a variable. There's going to be 2 superimposed jobs targeting the same worksheet in the same spreadsheet, which will be perfectly fine because they can be serial items in the same workflow -- a case in which NOT using concurrency works for me very well. This is not designed to be industrial-strength and scalable, but is designed to keep this one particular task running on modest resources indefinitely. It is also very worth pointing out that right now that all this runs in Jupyter Notebooks, but there is nothing inherently about it that mandates that this should always be so. Once a job is reliable and stable, it can most likely be moved out to a much more traditional filename.py and cron model, and in fact SHOULD be done so, so that there can be a nice centralized headless server location that you can bump jobs over to, and more-or-less forget about them. But then, nothing really ever gets forgotten about in this system. You have to keep checking in on the data, which is generally flowing into Google Sheets, where you have special concerns regarding not making a single spreadsheet too bogged down with voluminous data violating both the 100K-cells rule AND excessive field-stuffing (not rules, but just good general wisdom). Okay, so organizing your work RIGHT NOW for TODAY? I've had a couple of setbacks last week that were like a punch in the stomach, then I went off for a marathon weekend with Adi, and came back into work to different priorities, and lost tons of momentum. Wind got taken out of my sails, and I have to gather the tinder, infuse the fuel, and ignite the fire. I need to make one of my "tests" the monitoring of a data feed, appending its findings into a sheet... Accelerating the process that presumably takes place over months into just minutes. The obvious data-feed choice for such a task is Twitter -- BUT I don't want to sit on a constantly open connection. Okay, so accept that with the volume that goes through Twitter, you will NOT be going for 100% data capture on whatever hashtag or username (well, maybe username), but rather enough to perform the test. Okay, so identify the Twitter feed that would be appropriate for this use. BEWARE the rabbit hole! Screen scrape if the Twitter API requires too much overhead. Okay, probably their "Today Moments": https://twitter.com/i/moments This should be a general clean-up as well, making everything clear and easy to understand. Gone should be many of my unusually named or special-use functions. Here should be clear-to-understand, mildly parameterized general-use functions. 1, 2, 3... 1? Simulate that separate stand-alone function. The first step of a little tour de force test that really shows off the capabilities of the system. Load pipulate, functions, goodsheet and tests. This should be your pattern and rhythm for getting started. It's all always in these 4 files -- unless somethings being hidden in private.py, which will always be called out. The big temptation right now is to try out vim8 on macOS Sierra, now that it appears to be straight forward under Homebrew, but that could be a huge rabbit hole distraction right now, so put it aside. I don't need that async notification stuff. Later, later. Another distraction to avoid is upgrading my iPhone to iOS 10.0.2. Later, later. Design each day like a page in a story, and don't fucking flub it. Goddamn it, make today great! For this test, I'm actually going to need a Twitter page that updates much more frequently than that top news one: https://twitter.com/search?f=news&vertical=news&q=trump&src=typd Yeah, that'll do it, haha! Okay, once again, 1, 2, 3... 1? Okay, work in the very same repo as is your main Pipulate HEAD. But don't mess around with the functions that are making the current jobs run as well as they are. Instead, make a new set of functions, and move the old job over to the new one. For example, I can't use populate_from_atom for this job, but I CAN reproduce it in a more generic fashion. Scrape Anything! Okay, this is where my thinking can SIMPLIFY all future tasks, instead of creating a jumbled mess of redundant-functionality functions. I have these 3 wonderful behaviors: - Pipulate() - Populate() - Append() It's a very solid model for almost everything I'll need to do with this system, and is the first front on which to defend simplicity! We're designing "job music" that gets expressed as workflow compositions. Workflow Compositions... very powerful! Think like that! What we're talking about with monitoring from an active Twitter feed is just throwing a recurring list-of-lists at the Append() function. Simple as that. Whatever generates the list-of-lists is inconsequential, and there should be isolate-able implementation details, as to whether it's an ATOM XML feed using BeautifulSoup4 to parse it, or an HTML page using RegEx. Atom/BS4 vs HTML/RegEx should be of absolutely no consequence. By the time the list-of-lists hits whatever is primarily called by Append(), the details of what generated that list-of-lists is forgotten. And so... and so... we need another strong nickname in the system for what precisely accomplishes this separation. I already have it. It's the list_of_rows parameter of the Append function. currently, I have: atom_feed = "functions.populate_from_atom('%s', 'Feed', '%s')" % (feed_sheet, feed) list_of_rows = atom_feed And so, all I have to do is write a function that returns a list-of-lists to get started here. Just jump in head-first. Stop lallygagging! Okay, the gist here is to design something with NOT A LOT OF MOVING PARTS, but parts which are moving very quickly and reliably and repeatedly, so that we can hammer out bugs and stuff. Highly exposing of the Pipulate process. No! As much as I want the high-volume part of this, when the work is done, I want the very thing that NEEDS to be working well, working well -- not an abstracted case of it with a higher update frequency. Just re-work what you have in a parallel set of functions -- same plan as before, just no new feed. Work through this systematically. Stop the jobs that are now constantly processing the current 2 sheets. When next they get updated, they will be NOT from the VM, but instead from my current code. Done. Okay, now make a copy of the historical sheet where I have been doing the post-data-collection processing. Delete the hidden columns, and prepare this to be THE PLACE where the data is also collected. Done. Okay, now switch the name of the feed_sheet in private.py to be the one that the boss is already shared into and regularly monitoring... done. Okay, do a list comprehension that adds the empty cells for each row that are needed. Done. Okay, now find the job that's normally done separately -- oh, no special job. It's just standard Pipulate with a particular table/tab name given. So add a Pipulate event to the feed... done. So far, so good. Okay, things become easier if you scale-down the ambition and refactoring urge, and take things just one tiny baby step at a time. Okay, in doing this, I've ferreted out a bug that I've seen before -- pipulating a calculated range that's longer than the existing spreadsheet. I should be able to catch that condition and adjust the range to something legal. Wed Oct 26 13:47:20 EDT 2016 Okay, I have most things working correctly now, having combined these two jobs into one, using Pipulate workflows the way they were imagined, combining appending and pipulating rows. But it does look forward I'm getting a false positive on finding the first pipulate-able row now... pinpoint it! Okay, pinpointed, committed, pushed, and pulled over on the VM, and now run on a scheduled loop. Woot! This is big. It's always just going to be updated, and I've like completely broken off the combined job. This is a big round of simplification. I need to clean up my home directories a bit. This one should be in a folder named pipulate-feed... okay, done. Now, delete all old directories except the core and report ones. Okay, done. No reason to delete anything that's not distracting, so it's only the old seonotebook folders to go, and they're gone. Now, I have this job continuously running in the background. Every time my boss looks at it, it will be updated to the latest. Okay... next! -------------------------------------------------------------------------------- ## Tue Oct 25 13:39:47 EDT 2016 ### Today & Life Okay, I'm having a very hard time getting my mind into things right now. Deja Vu. Sit, need to use all sorts of mental tricks to focus -- but none of which are actually better than FINDING THE LOVE for the work, and figure out why you'd rather be doing this particular thing more than anything else in the world -- except, you wouldn't -- not with missing Adi, and living with Adi, so much. Shit, this is the first time I think it's really poking up to the surface how deeply I miss having Adi in my life during the week, but I have to accept that that's as much my fault as anyone else's. I'm choosing my own personal sanity over being deeply daily connected to my own daughter's life during these most important formative years -- and THAT'S what's really getting me. Maybe I should have waited until after Adi was a bit older before worrying about my own happiness. Adi says she misses the old times, and I choke up even now thinking about it, and it's been putting me on emotional edge lately -- not to mention the latest news right as our 10-year anniversary rolled past. Shit, I'm a strong person and I can take a hell of a lot of shit that life throws at a person. I'm still part of the privileged of people who are alive today, and probably who have ever been alive. No cause for complaints -- but DO get your mind back into the things that result in a secure income. THAT'S very important too. And you have to LOVE it, or it will be eclipsed and undermined by the other things competing for space and time and attention in your consciousness. Okay, I'm having trouble with a simple SQL statement and join. The dots are all around me, just taunting me to connect them. I'm doing the current project in an ad hoc fashion, typing the SQL into Workbench/J just to get the work done. Okay, the thing about these requests is they shouldn't be difficult. Tue Oct 25 17:39:24 EDT 2016 Done that work, and combed over quite a bit. This is my most important stakeholder. I've got a number of other things to do very soon, but... but I'll think about them on my walk home. Maybe another podcast. I may have to switch my database plans from shove (stuck on Python 2.7) to maybe TinyDB. -------------------------------------------------------------------------------- ## Mon Oct 24 09:43:53 EDT 2016 ### Monday Morning Wow, journal entries from home this weekend. Surprising. Here we are on October 24th, 2016 at last. I'm a shit-load behind where I wanted to be by this time, but I spent my time focusing on my new job, catching up on sleep, and digging myself out of the crushing accumulation of strata. I only just barely avoided becoming an artifact myself. Okay, pick yourself up, dust yourself off and start all over again. Okay, your priorities actually must continue to remain similar, until you get your living expenses down. You can't cut off the fuel supply while the fucking airplane is still so big. And there's likely going to be hostile anti-aircraft fire shot at me, and I'm without much maneuverability right now. Know your strengths, and play into them. Stabilize. Be happy with what you've accomplished so far, and look even more forward to what's still to come. I need to get my Monday morning weekly report out. I should do my best to stay focused without even having to necessarily try to hard today. That will be a good sign of things actually still going well. It's funny, on the commute, I'm being drawn back now to reading book 2 of the Chronicles of Thomas Covenant instead of Python or listening to audio books or whatever. It feels comforting, like getting back to my roots. Focus and sharpen and become expert. - Familiarity with "the way" - Tighter and better feedback-loops - Achieving expertise and mastery Glad I encountered The Phoenix Project when I did. Consistently keep connecting the dots. Keep your own greater forest-and-trees ideas in mind, and know how you're forging your path through it all -- more or less. Don't let yourself get bullied by life or ex or neighbors or employers or offspring or any of the other numerous "nodes" on the same life/existence network on which you yourself are a node. This is where all those stupid thoughts of "selfishness" come into play. Given my belief that people most often accuse other people of what they fear most and hold as their greatest insecurity in themselves (smelly neighbors), be keenly tuned into people who might tell me that I'M being selfish in MY decisions -- I'm not. I'm just trying to get through it all with my sanity and ability to take care of myself -- and now my offspring -- intact. Anybody who says otherwise can go fuck themselves, and anybody who thinks otherwise can just keep it to themselves, while I gradually edit them out of my life, because I'll still be tuned into the vibe, and don't need that shit. Okay, I feel better now. I have to do just enough ranting in order to get it out of my system and focus on what's important RIGHT NOW -- which is satisfying my employer -- living up to my end of the deal for the little slice of economic produce that gets allocated into allowing me to live in around that same-said economic machine (which is New York, Media, etc.). Mon Oct 24 11:44:06 EDT 2016 Okay, there is nothing more important right now than finishing out the new Pipulate's core set of capabilities. I need to walk and think and clear my head. I'm sort of being crushed under some weight right now, and my internal feeling is that I should shut up and sit down and code, but this is actually connecting the wrong dots. I need a clearer idea of what I'm implementing and why. Resiliency! Mon Oct 24 14:32:03 EDT 2016 Having some real trouble getting myself to focus today. No shit, Sherlock. All things considered, I'm doing pretty good, I think. I'm going to enjoy getting my sleep tonight, though. I need to just be able to throw Pipulate into a "mode" that proves its ability to do all this stuff. I need a tight non-leaky loop that just keeps working so long as you: 1. Still have Internet 2. Have a connection to Google 3. Can visit some consistently high reliably feed I think I didn't get enough sleep last night, plus the emotionally draining weekend. I can barely think right now. The podcast and YouTube video went well in and of themselves, but didn't have the desired effect of ramping me up and putting me on a sold, ready track. And I have 2 consecutive meetings coming up, so I hardly can even get into the work right now, because anything I do I'll have to stop in the middle of. Shit. Size to fit. 1, 2, 3... 1? Recap! Mon Oct 24 15:40:18 EDT 2016 First meeting over, into 2nd. Invited into wireframe meeting for a major property. That's a change. Great to work with people concerned with baking SEO into the design. -------------------------------------------------------------------------------- ## Fri Oct 21 07:16:31 EDT 2016 ### Early Start Okay, here I am in the office. That commute is really a killer. It goes from just before walking out the door ~6:00 AM to just walking into the office ~7:20 AM. It's an hour & 20-minute commute each way, and that's nearly 3 hours a day squandered away. It's a good thing I at least record my thoughts during that time for both publishing, and just processing my thoughts. This time you made for yourself now is super-precious. Feeling this totally uninterrupted time makes me realize how I'm sleeping the most valuable part of the day away. The magic numbers of flying under the radar are so important. If we really aren't gobbling up to much resources, then there's no reason it will trigger off the limit. Get the project running conservatively, and then look for ways to speed things up without risk. It's a classic puzzle, and if you get this right, it can actually be most rewarding. What principles are most important here? I need persistent memory of what proxies have been used, and how much. I think my whole random mix them up and make an random pattern approach didn't pan out. I need another approach that's more systematic and record-keeping like. It's all in the algorithm of how use actually use these proxies, and I'm leaving too much to chance right now. Okay, there's an optimization formula here on the use of 10 proxies. What are the constraints? - Each proxy should run up to 9 consecutive queries. - There should be from from 1 to 5 seconds between each query. - After that's done, a proxy should be given 11 minutes to rest. I can use threads or multiple processes here. Be cautious, because EVEN WITHOUT this, I 503'd, so I don't want to go all concurrent or parallel until there's a solid reason. Think through the optimized linear approach. Out of the 10 proxies, put 5 aside to fully rest them. This will be a 5/5 split initially, because why not? It's a nice 50/50 duty-cycle model. The advice is no more than 500 requests per 24 hour period per IP. And so... with 10 proxies, that should equate to 5000 requests possible in a 24 hour period, but that's flying too close to the sun. It should be completely half that. So, 250 requests per proxy per 24 hour period. 24/250 = .096 per hour. That's 1%, or 25 per hour. Over 5 proxies (because half are resting), that's about 125/hour. 60 minutes per hour divided by 25 is one request every 2.4 minutes... hmmm. Let's say 2 minutes. So, 2 per minute x 5 proxies is 10 per minute, or one entry every 6 seconds. That would be very satisfying and fairly easy to ballpark calculate how long a job will take. There's definitely going to be housekeeping here. Think it through! First, we take our 10 proxies. Every 24 hour period, we will go through half of them. That means we're in the world of persistent counters. This is a standard dictionary task. There's a chance I could use the Counter object from the collections library, but I'm thinking that looks like performing counting tasks against existing sets -- like running histograms on data that already exists. When you're doing your counting AS you're processing, there's little reason for that, and it's easier just to think of a shelved dictionary as a bunch of persistent counters. Yes! Okay, think! There's a Pipulate instance starting where the housekeeping has never been done before, and it has to initialize the counters. But then there's a Pipulate instance starting after previous jobs have run, and housekeeping records already exist. The logic should pretty much be the same all the time. The 24 hour periods need to be either relative to start-time or absolute. Absolute is much easier, because... well, same answer every time. So... so, this might be a great case for the keys in a dictionary actually being a tuple in order to use the proxy's IP:port PLUS the date. The simplest data-structure would go: {('127.0.0.1:8080', '2016-10-21'): 123} This is the simplest possible schema, because for any given day/proxy combo, you can just do a less-than evaluation to see if you should be allowed to continue using it. Hmmmm. I can't really let the same set of 5 proxies be used continuously in the 24 hour period. Maybe I should switch between them within the 24 hour period. That would be using the first 5 at half-capacity, and then the second 5 at half-capacity. That feels better, because they would each get a rest each day. But then they all get used every day. Too much of a pattern? Hmmm. Okay, on firs-run: You shove 10 proxies into the system. They get divided straight in half, no shuffling (for predictability). No wait! 2 proxies should always be held in reserve. And the 2 in reserve should cycle. So, that's another bit of house-keeping! That's a modulus 10 thing. Hmmm. Skip that complexity for now. Just assume you'll hold 2 proxies in reserve all the time, so you shove 8 proxies into the system, and they get split into 4 and 4. Let's call them bank1 and bank2. Bank1 immediately goes into service, allowing each proxy to be used... hmm, the numbers change with 4 instead of 5 proxies, but not much. 500 in a 24 hour period, cut by half to 250. Each can still be allowed to run about every 2 minutes. 12 hours (because each will be in service for a half-day) is 720 minutes. So, that's each proxy being used 360 times in a 12-hour period. And each proxy should be able to keep itself from being used any sooner since 2 minutes since its last use, so we're going to need "last used" as part of the housekeeping data structure, so it will look like this (military time). {('127.0.0.1:8080', '2016-10-21'): (123, '18:42'} Hmmm. Getting very close to an utterly viable scenario that maximizes proxy-usage while still flying nice and low. So, we cycle through proxies 0, 1, 2, 3, 0, 1, 2, 3... and each proxy knows how soon it can run again, based on the house-keeping. So on the first run, all 4 can go immediately, but then when it goes around the cycle, it calculates a time difference between "now" and "next allowed to run" times, and throws a sleep for that many seconds. Consequently, we now have some fairly good predictability, but at the expense of randomness. Worry about the randomizing later. For now, it's all about usage and counts and putting proxies on vacation from time to time. What's the most logical way to make sure each proxy is used in its appropriate 12-hour time-slot? It's to make it not even an issue to any of the system past the "proxy list pull". The system should be built so that when you do a proxy list pull, you get back N-proxies, and the system proceeds with the identical behavior regardless. It's built to handle N-proxies, running potentially at full-throttle, but being throttled by a global variable that determines the minimum amount of time that must pass before the proxy can be used again. Yes! So the function that chooses the proxy starts out by grabbing all 10, then with some sort of inspection of the time will choose either the first half of the proxy list, or the second half. It's got to be an even number of proxies. That way, you can put proxies on vacations in intervals of 2. With 10 in the mix, we'll do 2 on vacation, with the remainder of 8 being split 4 & 4. Let's start cobbling together this process. This will switch from False to True as it passes noon: import datetime now = datetime.datetime.now() late = now.hour > 12 Okay, now that we have a True/False for whether it's afternoon, the only thing left is to divide a list into a first-half or second-half based on it: r = range(8) if late: bank = r[:len(r)//2] else: bank = r[len(r)//2:] Wow, Python makes this so easy. And I'm ready to implement, so now I actually have to move the existing job onto my virtual machine... okay, done. My main desktop instance is once again available for dev work. Interesting experience, all this run-repos-anywhere stuff. Okay, now that I've updated my proxy_throttler function, I need to create a test around it that DOESN'T hit Google and interfere with currently running jobs that are actually relying on the non-503'd state. Think! What does a test around this look like? Ugh! Currently, all the proxy intelligence is built into the verbatim function, which is not good. Decorator? Helper functions? Hmmm. I just need a plain old "proxy" function, so you can just get a request's response object back that you know used a proxy according to the scheme. A goal here should be to externalize much of what's in verbatim for reuse. Ugh. Okay, didn't get as far as I would like today, but have made some good progress. Some deeper understanding necessary. Learned that I can't add https as the scheme in Requests proxy API, even though https is probably being used, tunneling through... gotta research it! Okay, hmmm. I need to actually confirm that the proxy is being used! Okay, it does break if it's a bad proxy number. I have some testing to do. -------------------------------------------------------------------------------- ## Fri Oct 21 05:49:08 EDT 2016 ### Stable and Sustainable Woke up early, going into work early. Going to tackle this thing very well and get things stabilized. What's going on at work has parallels with what's going on in my life. Simple premisses turning out to be more difficult in practice than I had expected. Either back out and lose credibility, or double-down and make it work. Pros and cons each way, but really want to live up to the tasks that I have taken on. It's just a matter of living within parameters. Budgets. Constraints. Being hyper-aware of them, and flying well beneath their levels -- simultaneously holding back resources that you could employ later in a pinch. Always hold something in reserve that you could pull out in a pinch. A rainy day fund. That's just for starters. There's a number of such strategies and tactics that will make things more achievable without burning through everything you've got to spend on the problem, while reaching a forever more stable and sustainable place. -------------------------------------------------------------------------------- ## Thu Oct 20 20:37:53 EDT 2016 ### And What Now? Wow, okay I'm on a laptop I have not been on since Sept 3, and even had journal edits here I never committed. And so now, committed they are. Hmmmm. I'm at home, and it's coming up on time to call Adi. 8:38... now is the perfect time! So strange to be on nearly all Apple equipment, again. From my phone, through my 2 laptops, through my old aluminium iMac, it's all still a very nice experience, and elegant. And this is from an old Amiga-hand. I'm texting back and forth, getting ready for the call. My Apple headphones are on. I just know it's all going to work. This was an important step in staying in touch with Adi while her and I don't live together for most of the week. That tugs at my heartstrings, like nothing else. I think this age-range is sooo formative, and another man is spending the majority of the days with her. Ouch. But our weekends and time together is very special -- all the more so. And, I need it. I am still... decompressing? Not sure, but... Thu Oct 20 22:06:42 EDT 2016 Ugh, just had a tough phone call with Adi, and equally difficult follow-up text convo with her mother, and now emotionally drained. And still have stuff I want to do for work, so that I don't start to lose credibility by not delivering, even after we paid for dedicated proxies. I think I'm starting to see what's going on, and I think I'm starting to get a feel for the levers and bullies going on behind the scenes. I have to write in part to keep myself nice and centered and sane. Life can load a lot of burdens upon a person, and people adapt rapidly to new normals. This can be a double-whammy when you load up more responsibility upon yourself than you are capable of managing. Family and mortgage and ambitious job. Having grabbed for the brass ring, but instead of achieving it, finding you only get our of it as much as you put into it, and it's exhausting. I have to switch to a more efficient engine technique. I have no weekends just to myself, and the commute is a killer. The job is stimulating and right up your alley, but the demands are great, and you find that you may have already over-promised, and feel the earliest days of what could become a pattern of under-delivering, if you don't pull yourself up by your bootstraps... now! No time to spare. Maintaining your earning capacity is the most important thing, while you navigate through these waters, because without a steady income, a lot of stuff comes crumbling down and becomes a hell of a lot more stressful than it is today. So, get and keep your shit together. Don't let things balloon and escalate to anything bigger than it is -- either in your mind, nor in interaction with others. And find the love in it. And yet, physical and emotional exhaustion kick in. And what now? -------------------------------------------------------------------------------- ## Thu Oct 20 09:42:00 EDT 2016 ### Captcha Challenges Time to systematically tackle and defeat this captcha gotcha of screen scraping SERP results. Do NOT take the most difficult path on this. See if you can't crank of the speed. I thought my contingency plan was to use HMA VPN desktop proxy, but I'm getting flaky results even there. I need to design a nice, simple task to see if we're getting 503 rejected even with an HMA IP, and if so, to see if I can't actually clear it out. Okay, I should be able to go into incognito mode and do a simple search and reliably see the Captcha... yep. Okay, test satisfying the captcha and seeing if the current error stopping processing goes away, at least temporarily. I'm running out of time to satisfy my boss while maintaining credibility. I have to admit that I was lulled into a false sense of how easy this was going to be, based on how many of the published HMA anonymous web proxies were coming up good. They were however NOT good, because of the 503's Google itself was returning. Re-thinking required. Too many moving parts! Complexity is becoming your enemy. All this server-side proxy stuff if where you went wrong, I think. Use the advantage of being on a "normal" desktop computer, where you can run software like HMA Proxy. The key thing there is how slowwwww doing a single SERP-check can take. I should create a decorator that generically times how long a function took to execute -- perfect use for a Python decorator. Thu Oct 20 14:42:05 EDT 2016 Okay, did my homework, and found a good web proxy service for SEO, and I'll be refactoring my code to use that. All this difficult experimentation stuff magically just goes away, but I will still need to live under the magic number radar. But I think I'll still document some of my findings here. After the Google captcha is thrown up and completed by the user, the user now carries around the cookie: GOOGLE_ABUSE_EXEMPTION. If you actually set that, then you can continue to do Google searches, but this only works because of the unique platform that is Jupyter Notebook, because the place your code is running from is the same actual machine you're sitting at -- a true 2-edged sword situation. Don't waste any more time on red herrings. Now, implement against this solid new model. I have 10 anonymous web proxies to work with. Step 1: put the 10 IPs into private.py (perfect use for this location). Okay, now do this in a testing notebook, so you're not burdened by these enormous files. And ONLY the serp-request should use a proxy-server. I want the speeds of the local network for everything else, which should speed things up significantly over how it was being done through the desktop HMA VPN software. -------------------------------------------------------------------------------- ## Wed Oct 19 09:55:42 EDT 2016 ### Fighting Web Proxy Issues Yesterday, I just got totally crammed-up and jammed-up and couldn't drive home this super-important next-step and dot-connecting. I made mistakes on several accounts, and even had the IT team come to me and ask about the automatic Google queries... haha, okay, have to make double-sure I don't hit against Google from here. There are also another set of important criteria that I need. Wed Oct 19 12:35:31 EDT 2016 Ugh, okay I completely hate this floundering feeling. I am using the proxy list from HMA that I got the lifetime membership of, and it seems they all are burned by the time I get to a project where I would use them by noon. This is not a tenable solution. I need to take another tact. Let's do some diagnostics. A 503 response code will probably become okay again. I'm using free anonymous web proxies that anyone and everyone else can use -- although the $25 lifetime membership to the HMA mailing list MIGHT be slightly more exclusive than ones that are just scrape-able off sites. Here's the basic Google querystring parameters in incognito mode: https://www.google.com/search? q=test &oq=test &aqs=chrome..[deleted] &sourceid=chrome&ie=UTF-8 Interesting. I just checked in non-incognito mode, and it's the exact same parameters. I think my new tact has to be sending all the standard parameters that usually get sent, plus setting all the request headers more accurately like what a real web browser is sending. I will stop short of doing anything with JavaScript that might be going on for validation. Basically, this is going to be a checklist of things to do to look like a human. Operation "Look Human". The key values to set in headers I believe are: - useragent - referrer What else? Well, when the Captcha DOES occur, it would be nice to actually clear it. The proxies that are actually good, but for which Google is no longer serving results. Also consider all possibilities: - Buy exclusive web proxies from a provider - Get generic cloud instances and turn them into proxies ourselves - Getting around the captchas on the public anonymous proxies - Use desktop VPN software Wed Oct 19 16:54:00 EDT 2016 Okay, I'm solving this in the short-term by using the HMA desktop proxy software, and turning off the in-function proxy features. While this is good for ad hoc manual runs, it does not feel right for long-term automated tasks, which really should have working web proxies. But this can be a place where I really differentiate my system, by allowing a manual "clean-up" of good proxies, but which are getting 503 responses on Google queries. The solution might even be tied to the use of Selenium or Mechanize, which I'll have to start doing soon in order to generate those thumbnail images. -------------------------------------------------------------------------------- ## Tue Oct 18 10:30:47 EDT 2016 ### Figuring Out SERP Scraping Well there I go again, distracting myself! Ugh. Okay, okay, let's connect those final mile dots that need connecting (yes, I mix metaphors freely). I need to do the 1, 2, 3... 1? Procedure for the thing I've been putting off. Now is a good time. But first! But first, I need to get a job re-processing that I said I would get going yesterday, but constant meetings derailed me. Okay, that's the incorporation of the endless proxy system to a SERP system. And THIS is fundamental to the system. Like every task I do almost could benefit by this. And I went so fast-and-furious getting the job running even just at all the first time. Re-process everything in as simple a way possible. Okay, the way to do this is simply to re-process the "big" job which was previously throttled by query-limits. Tue Oct 18 12:04:02 EDT 2016 Okay, try to not break new things. Had to fix the "break" out of the looping segment if a chunk-limit value was set. That threw me off. But now... now... 1, 2, 3.. 1? Go have lunch... Tue Oct 18 13:02:17 EDT 2016 Ran into my developer-side coworkers and joined them for lunch and ate on the 11th floor -- so glad I did! I need to hang out there more often, but I did finally get a place and setup where I can actually focus, which I've been basically waiting for for 10 years. Now I have it, and not even camaraderie can get me to give up. Anyhoo, back at my desk. Think! Run that big job, incorporating in the endless proxy generator! Of course, just wedge it in at the appropriate place and run the big job. 1, 2, 3... 1? Find the function that does the SERP http call... It's what I'm calling verbatim100 currently. But it calls url_cacher, which I wrote to generically cache (pickle) http Requests library response objects onto the hard drive or other persistent storage device through Python data-picking and the shelve API. I'm getting hung-up because my function names are too_similar and not_strong_enough. Sure, they're verbose and descriptive, but what's the good in that if you can't refer back to them easily whenever you need? We're building a very strong key in a key/value pair. We're giving a nickname that will stick. Go for generic SERP that: - Can run against any keyword column - Can run with any results per page - Can run in verbatim-mode when desired - Can put double-quotes around the keywords when desired Think through the API from the Google Sheets side of the equation. What should the column names be, and how should it be set into different "modes"? Hmmm, I need the same core function able to be turned into different parameter variations merely by nature of the way its invoked. Forget about using optional parameters and arguments, but instead make very strong different nicknamed functions that all call the same common function, but with different optional arguments being set. Simplify! Ugh, defeated. Going home to sleep on it. Also going to leave the endless proxy generator running to see if I can come up with any hits overnight. -------------------------------------------------------------------------------- ## Tue Oct 18 10:12:33 EDT 2016 ### Assistant, Siri, Alexa, Cortana, Watson, Viv, and... Jeeves? So sad. There's a bunch of projects here that have been getting put off and deferred forever. I'm going to force myself to do them today, but I'm also going to make it as interesting as possible, and connect a bunch of dots I've been resistant or hesitant or procrastinating or some other time-delay tactic from doing for quite some time now. Okay, I need to stay on top of Google Assistant. It's where Google's personalized machine learning stuff is going to be attempting to "help" us. It's a re-branding of Google Now and Hello Google, clearly. It's also looking a lot like Siri. Stay tuned-in and on top of all this. It's likely my future-career. Okay, speaking of that future career possibility of wrangling and corralling AI's, let's do the usual suspects line-up. I should probably promote this to somewhere more prevalent in this journal. In fact, this really highlights the need for some other meta-information or markdown in my journal to take things to the next level, organization-wise. - Google Assistant - Apple Siri - Amazon Alexa - Microsoft Cortana - IBM Watson? - Samsung Viv? - Ask Jeeves (RIP) -------------------------------------------------------------------------------- ## Mon Oct 17 12:46:18 EDT 2016 ### Weekly Report Okay, got my weekly report out, had my 1st meeting of today and the next one is minutes away. Okay, in next meeting. Interesting one to be invited into. And now, it's 2:30 PM... ugh! Almost as little writing as when I'm in the zone. Interesting. Extremes. Fascinating that Google announced its intent to get off of Intel hardware. 3:00 PM phone-call rapidly approaching, and another meeting at 4:00 PM. How do people plagued with meetings all the time get anything done? Mon Oct 17 16:41:34 EDT 2016 Cool, I just got my own official virtual server here, and I can begin planning the retirement of the free-tier Amazon micro instance. Woot! -------------------------------------------------------------------------------- ## Mon Oct 17 09:46:12 EDT 2016 ### Week Begins I do believe I have even more journal entries queued up at home, waiting to get pushed out. Lots to think about and process, lately. This is when life-navigation is really most important -- Adi is expressing that her bouts of sadness come from her not being able to see both her Mommy and her Daddy every day. This is heart breaking. Is Adi's happiness the price of my own? That would be terrible. No, it's not my happiness that's at stake, it's my sanity, and being any good to Adi AT ALL as a father. The self-destructive path I was on was no good for anybody, and I'm better off this way. I have to rebuild my strength, re-center and refortify before I take any big next steps that stand the chance of jeopardizing it all. Just pushed up my first podcast since The Python Experience was accepted in the iTunes store. This is another solid experiment I need to keep running. I should really list the "Mike Levin SEO Experiments". ### Mike Levin SEO Experiments - A YouTube Channel (>5000 Subscribers) - A Facebook Page (>500 Likes) - A 1-textfile Journal (>2-years of my life) - WordPress-as-a-CMS Site (>10 years continuity) - Pipulate Free & Open Source SEO Software on Github (36 Stars) - Levinux lightweight Linux virtual server for education (90 Stars) Wow, 4 meetings today. Don't even try to dip down into the zone. -------------------------------------------------------------------------------- ## Sun Oct 16 12:03:09 EDT 2016 ### There are Disciplinarians, and I Am Not One... Good! Odd place for me to cut a journal entry, but I'd like to put that business behind me. I am not a disciplinarian. I don't think my own dad was either. What to do with today? Try to clean up around the house more? That is the sore temptation. I could try to get her out for Halloween stuff, like at Sleepy Hollow. -------------------------------------------------------------------------------- ## Sun Oct 16 11:54:44 EDT 2016 ### Perhaps Happiness is Purpose -- Help Adi Plot a Course Wow, hmmm. Okay, had a great day with Adi yesterday at the Yo Kai Watch movie, and then after at Lego Land, but she started to get really really angry over increasingly small things towards the end of the day, and there was no way she would yield to the sleep she so desperately needed, instead opting for getting angrier and angrier, to the point of raging in to kick me when I went in to brush my teeth, resigned to the fact that she just wouldn't -- out of nothing more than stubborn principle, it seems. This is worrisome -- particularly concerning my future happiness. I get too entangled with angry women. That seems to be my lot in life. I'm wishing (but not praying) that this is not the case with my very own daughter, who I will be bound to in ways that... well, let's just say, I don't want to end up like my dad. Shit, repeating patterns. I need to navigate these waters intelligently, and with purpose. Help Adi navigate her ship with equal or greater intelligence and purpose. Don't let her descend into that angry place where everything is blame. It's okay to feel sadness, but always try to live in "the now". No matter how we got here, there is nothing more important than the moment for each of us, and plotting our courses into a happier tomorrow. If there is any such thing as purpose, I would presume that it would be happiness. -------------------------------------------------------------------------------- ## Fri Oct 14 08:10:38 EDT 2016 ### A Couple of Too-busy For Journaling Days Wow, I was so busy yesterday, I didn't even have time for thought-work. I guess that's the difference beteween coding & Excel work. I'm at the office a bit early, and I have about an hour until I review a deck with the boss... 1, 2, 3... 1? Fri Oct 14 17:05:57 EDT 2016 Another day of almost no notes. -------------------------------------------------------------------------------- ## Thu Oct 13 09:46:15 EDT 2016 ### What Just Happened? Another day, another bunch of information spurt out through a variety of channels, most recently, podcasting. -------------------------------------------------------------------------------- ## Wed Oct 12 09:47:09 EDT 2016 ### New Deliverable Based on ahrefs I'm going to try to break myself of my Microsoft Office habit, starting today. How long can I go without loading any Microsoft software (aside from the OS, which I have to use)? Don't let it negatively impact productivity -- but THAT'S exactly the point. My hypothesis is that it will IMPROVE productivity by having all that extra memory on my machine by not having that hog loaded all the time (Outlook on virtuawin #1). VirtuaWin #1 becomes Chrome, and that's a big deal in my way of thinking. Chrome on Win2 is almost engrained. What would be better than Outlook on 1 without throwing off my way? I'm actually enjoying watching my employer's daily video broadcasts. I'm like 10 years late to the game. I was totally aware that tech personalities from the G4 cable network days -- the days about which I have rosy reminiscence of tech TV -- have moved over to the Net, but it's time I rekindle this affinity. Can't believe how much better this situation is than "agency-life". Don't squander it. Knock it out of the ballpark. It's what you owe them. Going to attend the sprint meeting today. Told there won't be much SEO stuff, but still good to sit in. But use the time to figure things out regarding the rank-layer deliverable. - ahrefs.com - site explorer - organic search / organic keywords - export link in upper-right - Full Export (50K) / For Microsoft Excel It's working fine for most of the sites I need to do the data-pull for, but not so much for one particular property (one of the "big" ones), and I'm texting ahrefs through their in-site chat program to ask why. -------------------------------------------------------------------------------- ## Wed Oct 12 09:29:19 EDT 2016 ### Getting Reports Running Again (Windows re-started my machine) I'm at the point where I actually have to drop out of my journal in the morning and do a git pull, because I just might have made an entry the prior day from home and not remembered. So: :w[Enter]:sh[Enter]git pull[Enter] ...is part of my muscle memory. ...as is checking the daily scheduled scripts, which... gulp... failed to run, because Windows installed updates and restarted my machine overnight. Quick, get them all running again. Now, there's more than just starting the VM. I have to start the whole Jupyter Notebook environment, and one of the scripts therein. Hmmm, I think I'm going to look at converting .ipynb files to their .py counterparts for the actual scheduled stuff. There's no reason at this point why things developed as .ipynb files can't be "promoted" to a standard Python 3.5 cron job living in (referred-to from) /etc/cron.daily/ or even hourly. And always remember to start your virtualbox sessions headlessly, because you can always open them later. Ugh, there's only one place I'm using venv anymore, and that's on the daily pulse reports, which require: cd ~/repo-name source venv/bin/activate venv-name python report-name.py On the same exact (virtual) machine, I have to do the following command to get my "other" scheduled task to run: source ~/miniconda3/bin/activate py35 ~/miniconda3/bin/jupyter notebook ...and then just surf-to, load & run pipulate.py with correct parameters. ...and just did my @c macro to trim trailing spaces... 111 edits... haha! Always enjoy how that looks in the repo. Commit this odd entry, then think about the day (time so-far was just getting reports running again), but that's done. -------------------------------------------------------------------------------- ## Tue Oct 11 19:01:39 EDT 2016 ### Finally Advancing Levinux And here I am at home, again. It'll be time to call Adi soon. I think it's important that I capture my state of mind from time to time in these little gem moments between responsibilities and exhaustion. They do exist, fleeting as they are. It's time to finally make my move, too. I have to start throwing some punches to land some bigger successes. I'm totally primed for it, too. Been practicing for awhile. I'm a daily vim user, in a big way... not like I'm actively seeking out new key combo's every day, but I do have some favorite basics that I really do like, which I can type fast vimgq. I sometimes think in vim, and sometimes think it should be embedded in my brain. I sorta resent the hand/keyboard hardware interface requirement between humans and machines. Voice control would be great, but for manipulating pure symbolic information, little beats being able to write a bit... to think, and write... in words. Typed words. I should take these little scraps of time to advance Levinux, too. I have some questions I want to answer, like - Where does pip install it's stuff under Tiny Core Linux? - Can I make that location persistent, the way /tce and /home are? - Can I install MiniConda on Levinux? Which Python does it use? - Does the new Python3.5 from the TCL repo include pip? - And of not, is the somewhat slick hack I'm doing to install it still the best way to be handling pip? - Can Jupyter Notebook use just any CPython 3.5? Or has Continuum packed its own "known compatible" version of Python in with Anaconda? Okay, am I going to try to get self-promoting at Reddit? No, I need to just actually BE Reddit-worthy, and let myself become discovered. It's got to be a particular thing... a "shtick". -------------------------------------------------------------------------------- ## Tue Oct 11 09:59:48 EDT 2016 ### Getting Back to Making Stakeholders Happy At the office. Getting a late start, but look at what I did with my "vacation day"... waking up at 3:30 AM in the night, and working until about 9:00 AM working, finishing up a Python function to do the final yes/no determination on a project we've been working on for months -- plus, my weekly report. It was very intense, and I had to build a lot of the Pipulate system to support it. This is the flame in which my current system is being tempered. I can't go as far as to say "forged", because I've forged the thoughts and the notions twice before. This is most clearly iteration THREE of this particular system; I've had 3 iterations of a prior different type of system too. In total, I'm up to six then. Six time's the charm, I guess, because I'm going to try to achieve break-out success on this one -- at least, notoriety-wise if not financially. I'm starting to watch things like Random Access now in the morning. The last time I was into those types of videos, I believe it was cable television with Leo Laporte shows. It became Attack of The Show, and I tuned out. But the super-geeks moved to online, and I work for one of the companies now that does live daily broadcasts. I should probably go watch this stuff in person. The TV studio is right near the main entrance. Ah, it's a Facebook Live stream. Things are really getting funny when I don't specifically recognize which tech is delivering the phone-based streaming video to me daily. Well, I guess if it's daily and effectively notifying me and actually compelling me to get it playing, then it's probably Facebook video... haha! YouTube/Hangout never reached out to me properly yet. Imagine that -- with my YouTubing. Should I be moving that over to Facebook? At any rate, my Pipulate system came a long way this weekend branding-wise. This is my git pull of the repo from my main office work machine: $ git pull remote: Counting objects: 86, done. remote: Compressing objects: 100% (57/57), done. remote: Total 86 (delta 60), reused 55 (delta 29), pack-reused 0 Unpacking objects: 100% (86/86), done. From github.com:miklevin/pipulate e130d93..38fa656 master -> origin/master Updating e130d93..38fa656 Fast-forward README.md | 124 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------ functions.ipynb | 169 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------------------------- goodsheet.ipynb | 15 ++++++++-- logo.svg | 136 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ pipulate.ipynb | 167 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------------------------ 5 files changed, 482 insertions(+), 129 deletions(-) create mode 100644 logo.svg So, now there's the Pipulate logo! I ow need to start cross-linking EVERYTHING. I need to do this more intelligently. My new podcast URL location is: http://python.madewithopinion.com Defer all the things you want to do surrounding this for later. You have to throw yourself into your at-hand work with some force -- and NOT JUST the things that most interest you, but the day-to-day promised and expected deliverables. Don't get into the hated agency vibe, but do do some stepping-up and knock people's socks off. Tue Oct 11 12:48:32 EDT 2016 Okay, eating lunch. Spicy Udon noodles from Essen. Mmmm. Tue Oct 11 17:04:59 EDT 2016 Most of the rest of the day was meetings and answering questions and such. -------------------------------------------------------------------------------- ## Tue Oct 11 07:44:48 EDT 2016 ### Start Moving Finally getting up for the day (fed cats about 45 minutes ago). Get your ass in gear. Hit the day in such a way as to craft another fine moving-your-life-forward sort of day. Give Adi a wakeup call when you get out on the street. Start moving. Saw Storks at theater on Staten Island on Forrest Avenue exit right after Gothel's bridge. Great theater to stop at on way to drop her off at end of weekend. Gave them the 27-inch Mac as planned. Began podcasting with the "Opinion" app all from iPhone. -------------------------------------------------------------------------------- ## Mon Oct 10 08:46:28 EDT 2016 ### Had a Great Day With Adi Okay, the sun's up and Adi's still asleep, and I'm clear of the... actually, just wrote a whole follow-up email to my weekly report with increased detail on all the stuff I have written of late. Hmmmm. Time to just let it absorb into me over the day the significance of what I just built. Don't try to work on it more, except insofar as how the introductory videos might go. -------------------------------------------------------------------------------- ## Mon Oct 10 04:41:04 EDT 2016 ### Wish I Could ALWAYS be that effective Good, good. Now hit it home. Visit each URL in the set... Mon Oct 10 06:05:13 EDT 2016 Chased the rabbit a little but, but with good reason. Wanted to properly support optional arguments coming in from a configuration tab. It was highly worth it, because now I don't have to put private argument values in any of the files in the repo, even temporarily. It's like a per-job version of private.py. So now finally, visit each URL and look for the exact string match in the response.text of each. Mon Oct 10 08:29:43 EDT 2016 Pshwew! Got my weekly SEO report off to my boss AND finished the function that looks for the final positive hits on the citation monitor AND fixed how optional arguments are passed in from other tabs in Google Sheets. Damn, that was an effective few hours spent since I woke up at about 3:30 AM this morning, realizing I fell asleep with Adi around 12:30 AM. Now if I could ALWAYS be that effective. So much more free to think about the day with Adi today. Make it just as effective at being a great dad. Now, continue your general clean-up and organizing until she wakes up. And maybe have your movie options lined up with Storks and stuff. -------------------------------------------------------------------------------- ## Mon Oct 10 04:00:57 EDT 2016 ### Waking up at 4:00 AM to Hit This Thing Home Adi finally fell asleep somewhere around 12:30 AM. I fell asleep with her, and just woke up. I'm going to try to get the last function done, along with my weekly report and feel really good about the whole day with Adi today. So, I have to work very quickly. Make some coffee. 1, 2, 3... 1? First, I change the filename value in goodsheet, so I'm going to be working with the work gmail login, rather than my personal one, which also helps to to make the point that goodsheet.py can be run independently of pipulate to take care of the oauth stuff ahead of time. Done. Meanwhile, I turn workflow back to default for developing this function. I'm working against Sheet2, since I extracted the ones that have unconfirmed positives already, and am working against a much smaller set. And I need the dummy function in place for testing: def confirmedhit(**kwargs): return Response(ok=True, status_code='200', "bar") Okay, confirm that I'm working with set data every time: def confirmedhit(**kwargs): if "extracthit" in kwargs: extracthits = kwargs["extracthit"] else: Response(ok=True, status_code='200', text=set()) if not isinstance("extracthits", set): extracthits = eval(extracthits) return Response(ok=True, status_code='200', text=type(extracthits)) I love that I can return any datatype using this technique, and I always get the string representation of the function object in the GSheet cell... very slick fork quick tests like this. 1, 2, 3... 1? Well, visiting each url found in the set of course, constructing an object (maybe the byte size of response.text) and sending it back as a likewise set. And now, return back the same data set that you read in: def confirmedhit(**kwargs): if "extracthit" in kwargs: extracthits = kwargs["extracthit"] else: return Response(ok=True, status_code='200', text=set()) if not isinstance("extracthits", set): try: extracthits = eval(extracthits) except: return Response(ok=True, status_code='500', text=set()) a_set = set() for an_item in extracthits: a_set.add(an_item) return Response(ok=True, status_code='200', text=a_set) I feel a decorator in my future that handles all this pre-processing argument stuff. Make a mental note that I can use things like @clean_set or some other conventional way to make sure we're handing the decorated function a "clean set" as its argument data. -------------------------------------------------------------------------------- ## Sun Oct 9 12:49:16 EDT 2016 ### Getting As Organized As I Can Create and use an algorithm for simply cleaning your apartment and getting organized. Put the same thought into getting your life together as you put recently into creating the (brilliant if I do say so myself) endless proxy generator. I hit all the key points on that thing. I'm starting to think much more clearly in Pythonic terms, and there must be the equivalent in everyday life, simply to help you live better, according to some rough equivalent of thinking pythonically. Crazy, the words I'm teaching my vim spell checker. Sun Oct 9 14:23:34 EDT 2016 I spent a bit more time on things like my online presence than I had planned, but I so improved it, fixed a Levinux bug (and updated it), and made a ton of ways to reach me from my website. I have to put the optimizing into optimization -- but ALSO in my life, so make GREAT use of this last hour or so, before you head out to pick up Adi. I do believe that I can be fooled twice, before deciding to not be fooled again. I do believe I have a high tolerance for getting fooled. I feel quite a bit of stress mounting for having squandered by some measures some of the time that I carved out for myself this weekend to get shit done. So, get shit done. Very reflective while I do, and that's good for me. Sun Oct 9 16:54:56 EDT 2016 Heading out to pick up Adi. Sun Oct 9 20:45:37 EDT 2016 Okay, have Adi. Back home. Unpack the groceries. She's bursting with energy. I still have to get my Monday morning weekly SEO report in, and I really have to finish that last question for the definitive answer concerning feed "hits"/. It's actually a very easy regex (most likely) check on an http get for a final yes/no call for whether it's a true or false positive. I think I'll be able to do it after Adi goes to bed... wish I did it earlier. Ugh, but that's life. Also the weekly report itself. I'm so glad the endless proxies project went so well. Doesn't look like a lot's being accomplished, but it's such a critical piece that applies to so many other things. Let's finally commit this thing. Let's keep Adi's iPad charged, so I can actually use this computer. Let me also get the other (27-inch iMac, even newer model) together for sending back with Adi tomorrow. Just don't sleep. Adi and I should invent ladders for old cats. Let's see what the Internet says about this. Sun Oct 9 23:20:43 EDT 2016 The processing of the feed extracts is actually going pretty well without the endless proxy system incorporated into the process. I'm going to see if I can process the rest of the new headlines. But get that last confirmedhit function done. Think it through out loud. I have to rally my best strength to get this function down and also not disappoint my daughter. Sheesh, the situations I get myself into. But it's all for the better. The is the fire in which we forge our codes. It's too bad sourceforge became so not github. The concept of source forge is so spot-on for a collaborative public free and open source collaborative repository. It's just a git repo, with a fancy Web user interface and a few decorators applied on top. I love it. Github really hit on something. It's the GNU spirit, or something. Sun Oct 9 23:44:13 EDT 2016 Think through the function. First, get my Jupyter Notebook environment back. It's time to really document the heck out of this stuff. The input to the function is a set containing URLs. That alone causes me to think through the interesting boilerplate logic that I'll be using all over the place here for processing strings that represent data objects. First, is a question of the function's expectations. I'm not going to do string parsing this time around, looking for curly braces and making determinations, because that's a sort of guess-work arms race I'm not ready to accept, especially now that I'm using data objects like OrderedDict's, and I'm not going to be parsing every case. A function must know the data-type it's expecting. -------------------------------------------------------------------------------- ## Sat Oct 8 17:17:43 EDT 2016 ### Polishing My Online Presense a Bit Added the Pipulate logo to the Github repo, and now have fully eradicated the name of SEO Notebook. Also did a good pass at cross-linking my higher-volume / higher-exposure pages to get people in one channel to maybe hop over to another channel. And so... and so... what I need to do is continue on the physical-world path a bit more intensely for the next couple of hours. Stop sabotaging yourself. Go into a panicky sort of mode. But be smart and look around you. What we're talking about is the fight-or-flight reaction, but with slowed-down-time "The Flash" style intelligent observation, thought, and quick decisions about next actions. Work fast. Develop that skill of working fast. Commit it to habit, so it stops feeling like work. As I listen to my podcasts as I do my organizing, I am compelled to make note that it may Beeee important to know http://pybee.org/ Wow, this guy is a pleasure to listen to. He's got deeply felt anti-emacs feelings that are so echos my never-spoken sentiment that emacs, despite it's intellectual appeal, just can't make that step. The onboarding process is toxic. But isn't that the definition of terminal-based command-line Unix, after which his, Russel Keith McGee, (check spelling) BeeWare project is based. So, it's like the Unix command set, in functionality organizationl principles, but with very different APIs. Vim is nice. What he's describing as one of his future BeeWare text editor unitilities would be like a Python implementation of vim with well designed APIs to replace the bafflingly terrible part of vim which is VimScript. That'll all come. Think about my role in all this -- I am clearly a tool evaluator, tester, and user whose capabilities are expanding in proportion to how much better the APIs are becoming "for humans"... yeah, it is all sort of beginning to fit together. Don't miss the boat. Continue organizing. Listening to Risky.biz, the Australian security blog, and they're talking about the recent busting of Yahoo for yielding to the requests of the government to scan incoming mail to look for some sort of text signature pattern matching. There seems to be some debate about what really happened, but the Yahoo top security guy apparently stepped down over it, so it was something. Something was being scanned-for in all Yahoo user's incoming email. Wow, that's something no matter what the particular detials. But it's a lot like Google's email Advertising system. -------------------------------------------------------------------------------- ## Sat Oct 8 10:25:35 EDT 2016 ### Do A Little Sculpting Of Your Live Okay, build on my prior work and don't despair. I still have to go get coffee, and I let my whole waking up and showering take way too long, but it's part of getting the first good night sleep I have in awhile, and decompressing, and taking that first deep breath before an effort. But don't think of it as an intimidating effort. Instead, think of it like digesting. So, building on prior work is picking up on all the things you would have LOVED to have gotten to the last time you did this, but you couldn't even get to the paper organization for all the piles of laundry, which you've now done. Positive attitude is everything now, and so keep your mind properly conditioned as you go. Focus on how awesome you would be in a world that doesn't hobble you with paperwork and housework on top of all the other duties. I could be great. I am great, and am going to reveal the greatness held within the block of marble by chipping away all the surrounding material keeping it hidden. Strange metaphor. Am I the sculptor or the sculpture? -------------------------------------------------------------------------------- ## Sat Oct 8 09:34:32 EDT 2016 ### My Plan For Today Okay, today is my day to get stuff done. Saw Adi last night, and I'll pick her up again on Sunday evening to sleep over, and I'll spend Monday with her which is Columbus day, and which I put in as a vacation day. One thing's for sure, I'm going to be using up my vacation days all the time now that I have Adi in my life. My habit from before then of sometimes never using my vacation days in a year are over, as is the whole work-is-more-important-than-me attitude. While I do enjoy my work, I have new priorities now with Adi, but I do have to remain a good provider, so I can't allow certain traps I fell into in the recent past to happen again, either. It's about striking that balance. But I'm so buried from being disorganized, I need a good solid block of time to work on it, and it's going to have to be during the day, and it's going to have to be today. It's also Helana's birthday party tonight, which I'm going to make an appearance at, but I really can't let that throw me off my organization groove either. Perhaps the most important thing is to do things for reasons I have carefully thought through and deem important, and not merely living up to others' expectations of me. I come first now. Adi comes second. My means of generating an income comes third, and the rest of the world can take a leap. I don't have any coffee here, and that's a problem. So take a shower and use this journal to get your head in the game, and then come back here, and start some determined and focused organizing. Re-listen to an audio book you've already heard on the non-thinking physical stuff organizing phase of today's work (de-cluttering), and then change to silence or music for the thinking phase (sorting paper-work and figuring out what's what). -------------------------------------------------------------------------------- ## Fri Oct 7 16:57:32 EDT 2016 ### Getting Ready to Leave / Get Scheduling Going Ugh, I'm being forced to restart my Windows machine for the first time in weeks, after that locked-on control key symptom started happening. I've never found any other way around it. It'll be good to have fresh memory again, but there's so many scheduled things now I'm running, it's always a bit scary to do it. In addition to just starting up the Kubuntu VBox, I have to start Jupyter Notebook with it's virtual environment, load 2 notebooks and start them executing... sheesh! And I should be walking out the door RIGHT NOW. Shit, go fast. Get those 2 jobs running, and then commit and push your latest endless generator work, so you can do a bit of it over the weekend... so close! Ugh, so now I have to wait for 2 bloated systems to load. But I can work in parallel. Get Jupyter Notebook running on your Windows side, while Kubuntu is loading. Okay, all scheduled tasks running, and latest work committed and pushed. Now run to Adi! -------------------------------------------------------------------------------- ## Fri Oct 7 12:31:19 EDT 2016 ### The Endless Good Web Proxy Generator Now? Now! Finish the proxy thing. Do it as part of a battery of tests. Fri Oct 7 13:22:22 EDT 2016 Ugh, distractions. The SEO phone-call, and realizing Columbus day is not a day off from work, so putting in the vacation day request. Recalibrating consciousness to the task at hand is friggn' difficult. That's why I do the 1, 2, 3... 1? Thing. Forget about keywords for a second. Just focus on the proxy thing, even though the proxies are mostly for keywords. Isolate stuff. Make a list of the things your battery of tests is going to do. Fri Oct 7 15:07:08 EDT 2016 Okay, I've begun my battery of tests. I've started by filling in a list of sites from the Alexa Top Sites list. I think it just dawned on me that I'm going to need a strong visual way to activate and deactivate Pipulate functions in the spreadsheet. Previously, it was through the presence of a question mark in every cell that should have a value plugged in, but I think now that I want to have a special character in the column labels. Don't worry about it now, but do put it into the README to-do list. Okay, now I want to use a proxy to fetch a URL and record what proxy was used. It's essentially what I'm already doing to process the monitored URLs, but instead of the word "cached", I actually put the proxy address. Okay, here's the pattern for when a generator runs out: def gen(): for i in range(5): yield i def use_gen(): global a_gen for x in range(10): try: y = next(a_gen) except StopIteration: a_gen = gen() y = next(a_gen) print(y) a_gen = gen() use_gen() The generator only goes up to 5, but when it hits the end, it's re-created, which is exactly what I need. Oops, it gets a little more complicated. Because a generator does not appear to be able to recognize its own StopIteration (detectable from a "try" that calls it), I need an "endless generator" wrapper function so that the all the various functions that might have to call a proxy don't have the responsibility for recreating the generator. And that would be this: def proxy_generator(num_pages): import hma_scraper from random import shuffle from collections import namedtuple from urllib.parse import urlparse import requests from requests.packages.urllib3.exceptions import InsecureRequestWarning requests.packages.urllib3.disable_warnings(InsecureRequestWarning) Proxy = namedtuple('Proxy', 'good scheme address seconds') proxy_list = hma_scraper.scrape_hma(num_pages) shuffle(proxy_list) for a_proxy in proxy_list: url_parts = urlparse(a_proxy) requests_arg = {url_parts.scheme: a_proxy} try: result = requests.get('https://www.google.com', proxies=requests_arg, verify=False, timeout=1) yield Proxy(True, url_parts.scheme, a_proxy, result.elapsed.total_seconds()) except: pass def proxy_cache(**kwargs): resolved_url = kwargs['resolved_url'] import requests from requests.packages.urllib3.exceptions import InsecureRequestWarning requests.packages.urllib3.disable_warnings(InsecureRequestWarning) proxy = next(endless_proxies()) proxy_arg = {proxy.scheme: proxy.address} for i in range(100): try: response = requests.get(resolved_url, proxies=proxy_arg) break except: pass else: print("It seems no proxies are working.") raise SystemExit() #return_tuple = proxy.address, response.status_code, len(response.text) return Response(ok=True, status_code='200', text=proxy.address) def endless_proxies(): global proxies if not proxies: proxies = proxy_generator(2) try: proxy = next(proxies) except StopIteration: print("Ran out of good proxies. Re-creating generator.") proxies = proxy_generator(2) proxy = next(proxies) yield proxy proxies = None Okay, this is getting very solid. -------------------------------------------------------------------------------- ## Fri Oct 7 10:32:39 EDT 2016 ### Pipulate Readme Much Revised Okay, start using the README.md in Pipulate again to keep track of your to-do list AND the most important things that I am going to have to get across in the documentation and "indoctrination" to Pipulate. So glad to have dumped the SEO Notebook moniker. Also takes care of a smoother eventual transition to Jupyter Lab. 1, 2, 3... 1? Load the README.md into vim on your Windows machine... Okay, did some really good updates on the readme. I think it will serve to get people into it a bit more. I definitely have to update Levinux to support Jupyter Notebook. That new readme is going to be SOOOO important. Don't dilute it. Keep it focused, and everything need-to-know or need-to-act-upon. None of the fluff. There's plenty of other places you can put fluff. The Github crowd doesn't want to have anything to do with that kind of fluff stuff. Just "what does it do" and "how do I install it". Okay, do the battery-of-tests stuff that you've had in mind forever -- really, since Tiger at 360i, but this new system is particularly suitable for it. See if you can't actually work in pytest at this point. Can't you use pytest as an import in functions, and just make one of the workflows test every function? Yes! I certainly can, and that feels like closing the loop now between: - Testing - Debugging - Workflows - Documentation -------------------------------------------------------------------------------- ## Fri, Oct 07, 2016 9:29:28 AM ### Learn a bit about PyTest (and Selenium) Continuing my journal on Windows because I want to watch a video on my Mac laptop because it's easier to get the audio working on my Mac with headphones than on my Lenovo laptop. I'm listening to Andreas Pelme from EuroPython 2014. I'm choosing PyTest because I hate being forced into object oriented design, so I don't want to use the built-in UnitTest. PyTest, from everything I've heard, including Brian Okken's opinion, PyTest is for people who want less code for more testing. I do believe this video is from before PyTest 3, but still it's the most popular video on YouTube on the topic. Tests with less boilerplate code is how they promote it. First, he's showing how to do a test with UnitTest. PyTest can run existing unittests just as they are. As I see him demonstrating, I see it's what I had hoped. It's basically stripping out all the class stuff, leaving only the def's. It uses the normal Python assert, and you don't need to subclass unittest. Okay, HOW it's run is actually important. Tests are found and collected with the py.test command. That's the part I need to understand. Just noticed I lost read-access to one of the reporting tables that I use for the pulse reports. Just shot out email to ask for it back. Ugh! Another bug popped up in getting the Feed Monitor processing to run. Apparently, the top-row of the range for Pipulating has shifted down by one row. Shit, how does that happen? Look for which value is wrong... No, it's not a generally wrong value. It's a condition thing. I think some cells are looking like they have values when they don't. Yep, confirmed. If just a "blank space" gets into a pipulate column on the first "blank" row, then the whole first-range to Pipulate gets shifted down... by how much and according to what rules, I don't quite know yet, but I have seen it both slip down by just 1 (in the Monitor sheet) and by the whole range (in the Testing sheet). I have to wrangle these exceptions under control. How to actually run the tests? Just by invoking py.test, it will run all the tests, recursively searching the test directory to find all the tests. But you may want to just define the test you want to run. This stuff appears to run mostly on the command-line in these demos, but I'm thinking I could just have the tests as a workflow in pipulate. Decorators are used in pytest, so that you can "mark" functions, so that you can skip all the slow tests. You can implement your own logic with markers, like skipping a test if it's on Mac OS. Okay, now he's getting up to fixtures. He says we probably have no idea what he's talking about, but it's one of my top questions -- how you get to an "initial state" required to do a test. I'm doing less that uses global values now (almost nothing), so my fixtures should be pretty small, but I definitely need them. - You create fixtures USING the decorator @pytest.fixture. - The fixture function should get named what you want to pass into test functions. - You then write test functions that use that function-name as it's argument. - This trick of actually mapping function names-to-arguments is dependency injection. - You will now have an object inside your test function (goodsheet). - When the test is over, an automatic tear-down/clean-up will occur. You move the dependency construction out of the test case itself. That's the point there. Hmmm. Okay, okay. I'm actually pumped over Andreas using selenium in the demos, clearing the way for TWO important things I need to do. -------------------------------------------------------------------------------- ## Fri Oct 7 09:21:23 EDT 2016 ### Testing & Stability Today is about testing and stability and getting out early to have some time with Adi, after she's done with her homeschooling group in Manhattan today. Focus to an extreme. Go into that nearly fugue state where you really get shit done. Make a check-list of things that have to be ensured correct: - Make sure the Append() behavior adds rows when it needs them. - Make sure functions that depend on prior column dependencies work. - Handle data-type conversions when Python/json is in a cell. - Settle on an explicit Empty or None representation in cells. - Output a Timestamp while scheduling. - Defer generating a list of proxies until it is known a proxy is needed. - Somehow report-on/record which proxy was used, when used. - Add Geolocation to the proxy discovery process. I need an introduction to PyTest... watch a video. -------------------------------------------------------------------------------- ## Thu Oct 6 22:20:23 EDT 2016 ### Progress What a friggin' day. I only just walked in the door. That's 10:20 PM, and I didn't really nail it like I had hoped. There's some good stuff, though. -------------------------------------------------------------------------------- ## Thu Oct 6 18:27:20 EDT 2016 ### Infinite Proxy Generator Improved Better still: from collections import namedtuple from urllib.parse import urlparse import requests def score_proxies(num_pages): import hma_scraper from random import shuffle from requests.packages.urllib3.exceptions import InsecureRequestWarning requests.packages.urllib3.disable_warnings(InsecureRequestWarning) Proxy = namedtuple('Proxy', 'good address seconds') proxy_list = hma_scraper.scrape_hma(num_pages) shuffle(proxy_list) for a_proxy in proxy_list: url_parts = urlparse(a_proxy) requests_arg = {url_parts.scheme: a_proxy} try: result = requests.get('https://www.google.com', proxies=requests_arg, verify=False, timeout=1) yield Proxy(True, a_proxy, result.elapsed.total_seconds()) except: pass proxy_generator = score_proxies(10) next(proxy_generator) -------------------------------------------------------------------------------- ## Thu Oct 6 14:35:57 EDT 2016 ### Proxy Time! Pshwew! Okay, now I can stop referring to SEO Notebook entirely. It's now just Pipulate, and I kept my repo commit histories for both projects when I merged them, right as I "buried" the old Pipulate project. Okay, now it's time to deal with that pesky 503. There's some different approaches to it, but it's probably the biggest payback for my investment, right now. 1, 2, 3... 1? Close & Halt the old SEO Notebook scheduled job that was doing the scheduled Citation Monitor work. Reel it back into your development (native Windows) side of operations. Implement an anonymous web proxy pool. Let's look at what's out there. It appears more emphasis is being put on the harvesting than on just the testing component, which is what I actually need because I get the list emailed to me every day from HMA, for the $25 lifetime membership. Anyhoo, the ones I'm finding are: - https://pypi.python.org/pypi/grey_harvest/0.1.3.5 - https://github.com/IDSninja/hma-scraper Okay, I think I have a fairly viable solution. One of the main things will be to figure out how to work the files into the Pipulate project. Okay, did my research. I like being able to dig into the proxy lists deeper than most people are willing to go, using in this case hma-scraper. That will produce a list which I can then run a verify on to pick off the best ones. Another aspect of this is that I ONLY want to use these proxy addresses at the moment I perform a serp-check. I DO NOT want to throw the entire system into using the proxy. Most traffic will continue using the default, and proxy-use will be local to functions that might need proxying. Decorator? Hmmm, not sure. But for now, lift the code (check licenses) and work it in. Think! Hmmm, definitely write my own proxy-checker. Nothing out there is so much better that I should deny myself the experience. I'll almost definitely be using the proxy capabilities in requests: http://docs.python-requests.org/en/latest/user/advanced/#proxies Okay, making great progress. No reason to lift code except from hma-scraper, which thankfully is CC0 licensed. I'm tempted to do geoip stuff now, but I think I have to defer that to later. https://ushamim.wordpress.com/2016/01/24/ip-address-geolocation-with-python/ Yup, I got my solution: import hma_scraper from collections import namedtuple from urllib.parse import urlparse from random import shuffle import requests from requests.packages.urllib3.exceptions import InsecureRequestWarning requests.packages.urllib3.disable_warnings(InsecureRequestWarning) proxy_candidates = hma_scraper.scrape_hma(30) shuffle(proxy_candidates) def score_proxies(proxy_list): Proxy = namedtuple('Proxy', 'good address seconds') for a_proxy in proxy_list: url_parts = urlparse(a_proxy) requests_arg = {url_parts.scheme: a_proxy} try: result = requests.get('https://www.google.com', proxies=requests_arg, verify=False, timeout=1) yield Proxy(True, a_proxy, result.elapsed.total_seconds()) except: pass proxy_generator = score_proxies(proxy_candidates) next(proxy_generator) ...yup. -------------------------------------------------------------------------------- ## Thu Oct 6 10:24:25 EDT 2016 ### Merged Pipulate and SEO Notebook (into Pipulate) Okay, I need to have TWO instances of Jupyter Notebook running my scheduled tasks. I see no reason not to do it using my main Github Repo as the "bridge" to my Kubuntu VirtualBox, and just use a SECOND tab (and hence, 2nd Python VM) to run the second scheduled task. Process concurrency... haha! Okay, 1, 2, 3... 1? Make sure the job is committed to Github, with everything with the exception of what's in private.py. Make it work out of a newly cloned github repo... okay. Now, move the files outside the repo over: - private.py ...wow, that's all. Okay, as part of my introducing people to Jupyter Notebook and Pipulate|SEO Notebook, I'll have to walk them through setting up a Python virtual environment under conda, and getting the dependencies installed. Okay, let's finally get to this. I'm going to lose a bunch of stars in Github that gives me a lot of credibility, but so be it. I have to eradicate SEO Notebook, and elevate and redefine Pipulate... NO! I love those Github stars... it's so hard for someone like me to re-acquire such an asset, so figure it out. MOVE the new seonotebook repo into the old Pipulate repo. Yes! Then, move the old repo to private, which only has one star... okay... let's see. Thu Oct 6 12:59:39 EDT 2016 Okay, ate lunch. Feeling pretty good. Getting an actual good night sleep last night helped, I think. Check your 2 scheduled tasks... Thu Oct 6 13:33:49 EDT 2016 Okay, I did it. I merged the old Pipulate repo and the new SEO Notebook repo, and made seonotebook a private repo. That pretty much does it, I think. Now, I need to create a brand new Pipulate instance locally, and do a git pull there to see it go in. This is how I keep all my stars. Thu Oct 6 14:05:56 EDT 2016 Done, and also actually made my vim repo private, so I'm just really promoting Levinux and Pipulate, both of which have a respectable number of stars. I'm on the right track here. I'm very close to making the introductory videos. Shit, think what else for today. The proxy work? I'm (predictably) getting hit by the 503s. I should have flown lower under the radar. Okay but first before I do that, I should just re-settle myself into the pipulate directory, by closing out of all my current Jupyter Notebook tabs, clone the new repo, move private.py over and base myself out of that new location. Move the caches over too! Okay, done and confirmed. Keep this in mind for the scheduled jobs. You'll have to switch them over soon too. -------------------------------------------------------------------------------- ## Thu Oct 6 10:21:27 EDT 2016 ### Python SEO With Jupyter Notebook Working on an introductory script that I'm very happy with: SCRIPT #1 Hello World. I'm Mike Levin, an SEO in New York City. That is, I help people connect their website content with their intended audience — the sort of invisible matchmaker of search. And I'm here to help you do the same, by revealing my tools and techniques in a computer programming video series called Python SEO with Jupyter Notebook. I dabbled with making website-based tools in the past, probably with my still-going-strong HitTail from 2006 being the best known. But I've always been drawn to locally-running tools, for the sake of flexibility and control. But until now, such tools were not very shareable, due to platform dependency issues. In other words, it's tough to share reliable programming environments in a multi-platform and aging-resistant fashion. Yet, it is very advantageous to be able to do so in order to always keep yourself at the forefront. -------------------------------------------------------------------------------- ## Wed Oct 5 20:25:40 EDT 2016 ### Tonight Listening to the "Not Alone" audio book from Audible. Just got home. Popping some Ramen noodles in the microwave, and I'll relax a few minutes and eat before calling Adi. Life is kicking my ass, and I'm gathering my resources and energy to kick its ass right back. Hmmm, okay, think. My self-image at this moment comes from: 1. Still being a good daddy to Adi, even during separation 2. My programming work and ability reaching an all-time awesomeness I want a third regular ongoing instance of Pipulate. I think in fact running Jupyter Notebook in a browser tab is going to be my most common homepage, from now on. How could I NOT want a Python code execution environment, there to do my on-the-fly bidding? I don't even need to do big software builds to get awesome results fast. Okay, I did an update of my local seonotebook folder, and I do have Jupyter Notebook installed on this (circa 2009) 24-inch aluminum iMac, which won't let Sierra macOS install, haha! So, I've been an Apple fanboy that has one Mac (a smaller white plastic version) stuck at OS 10.7 or something, and this now stuck at 10.11. But not big deal, either one would run Jupyter Notebook, I think, which makes Python and a web browser even more of a universal platform than even Unix with the Mac Cocoa proprietary graphical user interface. Yeah, I think my message is coming together. What I want to do now is really start to document this stuff, and pave the way. Hello World! I'd like to write a hundred different books, on everything ranging from philosophy to technology to being a good daddy, even in light of separation. Wed Oct 5 20:52:33 EDT 2016 Now, on the phone with Adi. Wed Oct 5 21:56:20 EDT 2016 Now, off phone with Adi. I think I'll go to sleep. I think I'm letting the stress get to me, and my immune system is suffering. Stay strong. Stay well fed, and stay rested. I am going to need my energy for this weekend. -------------------------------------------------------------------------------- ## Wed Oct 5 12:37:14 EDT 2016 ### Today Back from lunch. Had lunch with Chris R. at the street food trucks. I should be doing that much more often -- both Chris and the trucks. Just sent a LinkedIn invite to him, finally. Okay, it's coming up on 1:00 PM and my next meeting starts at 3. Focus everything you've got now on the particular application of your system that you need to have running LIVE ASAP! 1, 2, 3... 1? Find where you were. Filename, function name. Wed Oct 5 13:05:49 EDT 2016 I'm just noticing that there are presentation capabilities built into Jupyter Notebook -- wow, I have to take advantage of that. Okay, I'm going to have some things to work out, such as controlling WHICH columns are going to process on each run, without renaming columns. There's a column dependency thing going on within the same row. Since processing the columns is order independent... hmmm. Yeah, there's a real order independence issue going on here. I should really consider enforcing the processing of rows left-to-right, so that after a value is returned for the row, it's in the new row_dict available to the next function that required the return-value from the prior pipulate-column on the same row. Hmmm. You don't have to work out EVERYTHING right now. Instead, just get to the finish-line of this particular project, which my boss is waiting for and has to make decisions based upon. So, what you need to do is... work on new function. You have to make this function work in several passes. This is probably going to have to be a multi-column function. Focus on SMALL TASKS. Next small task is simply to read the set data from a prior column and see if you can treat it like an original Python set data object. Okay, I need like a 1, 2, 3 step procedure for cranking out a new Pipulate function. Fill in the barebones boilerplate: def comparesets(**kwargs): return Response(ok=True, status_code='200', text="Change Me") Set workflow to "throttled", and actually check running Pipulate with the boilerplate function, confirming it works... done. Okay, now make sure we can return the value of another column on the row. def comparesets(**kwargs): a_set = kwargs['serpinspector'] return Response(ok=True, status_code='200', text=a_set) And we can! Now, can we treat it as an actual set datatype? def comparesets(**kwargs): a_set = kwargs['serpinspector'] a_count = len(a_set) return Response(ok=True, status_code='200', text=a_count) Nope, the numbers are too big. So, we need to add an evil eval. The world will explode! Good thing I control everything end-to-end. def comparesets(**kwargs): a_set = eval(kwargs['serpinspector']) a_count = len(a_set) return Response(ok=True, status_code='200', text=a_count) Yep, I got the right size numbers back. Very good, so now I'm sitting on a list. Now, I have to compare the contents of one list to another. Okay, I have the list compare going, and actually have 2 hits. Now, I have to confirm positives, and get actual URLs! Hmmm. Okay, repeat the boilerplate process... done. Now, retrieve the locally stored SERP. Wed Oct 5 16:24:29 EDT 2016 Out of my meetings. I need to clarify my thoughts. Right now, I have a series of functions that depend on each other. Not good without order sensitivity on the per-row pipulation. Wed Oct 5 17:43:15 EDT 2016 Ugh, there were a few fire-drills. This is exactly the sort of day I feel drained after. Shit, I have to see this thing through to the end. I CAN'T let this go into next week. Bear down. Wed Oct 5 18:40:00 EDT 2016 Okay, got a critical piece done. Pipulate columns can now use prior pipulate columns as a dependency, even if that column is as-yet still blank, assuming it will be filled-in soon enough (left-to-right processing) to satisfy the dependency. Final steps? Finish the extracthit function and make a truepositive function. These names will clearly change as Pipulate gets ready for prime time. A lot of stuff will be pulled out into a private area not in the main Pipulate repo (have to turn SEO Notebook into Pipulate). Okay, I reached a success assured moment. Get the SERP-pulling going, and then come in tomorrow fresh to finish that last function. Okay, process running. Let's see how far it gets overnight. -------------------------------------------------------------------------------- ## Wed Oct 5 08:27:52 EDT 2016 ### Pipulate is getting VERY clean and exciting Back at the office, and actually here almost an hour earlier than usual. Don't squander it. Get coffee and go right into your "descent" ritual before the day really gets underway. I just realized Bing was the default search engine on Siri since 2013 -- the amount of time I've been on Android -- haha! Serves me right. It was good to have the Android experience, and now I don't talk about it with any edge of mystery or wondering. I'm a hands-on person now with both iOS (in it's latest rendition) and Android, at least up to Marshmallow. I'll have to keep abreast of Nougat now too. Guess they couldn't call it Milky Way. Descending. Look at your work in Github... okay, looks great! Quick round of cleanup on the imports. - If no sheet or tab name are provided, prompt user... done. - Pickle last provided sheet and tab name. Okay, pickling defaults takes a bit of thought. There's: - No pickle file exists yet and values MUST be provided - A check must take place for existence of pickle file - Attempts must not be made to insert default values in prompt - A pickle file does exist Okay, done. This is a BIG step forward, making the whole system MUCH more ready for prime time. This looks like a good time for a commit, because now I need to shift gears. But a lovely thing I can do is promote this over to production, and make sure I didn't break everything. Production in this barely deployed status is a form of testing. Okay, ducking out of the sprint meeting today. Hmmm, okay. Think. Get food inside you again, and push hard on the next round of this work, which is the actual thing you need to delver on! Enough framework, which could be a bottomless pit of rabbit hole work, but I'm very happy with what I just achieved. I think it is my best starting-point for a reinvigorated career in SEO I've ever created (and I've made quite a few good ones). -------------------------------------------------------------------------------- ## Tue Oct 4 19:04:22 EDT 2016 ### Some Stuff Done Pshwew! I'm getting a journal entry begun barely after 7:00 PM. This gives me a good amount of tonight to get stuff done. STATE THAT on a public journal, and use Cialdini's principles of commitment and consistency to kick your ass into gear. It's phenomenal how much one can get done even in a single night if you're smart and efficient about it. Okay, okay, do a quick pass to make walking paths again. I'm still in the aftermath of an Adi visit. It's like my head is under the water and each gulp at breath fills my lungs with more water on my way to drowning. What I'm doing is a lot like rescuing myself from a drowning. Balloons first. Ugh, keep your strange self and displaced anger under control. Could I ever accept the possibility that I'm mad at my own daugher, who I hardly ever see except on weekends. I think she knows that and she plays it, hitting and biting me in ways no nearly six year-old could really still get away with. That's horrid behavior, of the Charlie and the Chocolate Factory type. Tue Oct 4 19:50:44 EDT 2016 I'm doing it, I'm doing it. Cleaning up and getting organized. Holy shit, yes, it's just like digesting. It's just like having a little self-awareness and self-respect to actually look around me and give a shit and do something about it. It's coming up on 8:00 PM, but I've made a tiny dent by starting promptly when I walk in the door after work. One journal entry to lock-in the time-slot for something productive and quasi-transparently out there. Hmmm. What next? A little food maybe I will grant myself, before the upcoming Adi phone-call. I will try to keep it to a reasonably short time. I can't be what she does for the whole night, or I'll get my ass in trouble (more than I already am). But seriously, people have been in far worse than me, and I am equipped to dig myself out. I just need to be able to hold firm against the onslaught of time-demands, even post-separation. Shit. Just power through it. More focus, less distraction. Some food. Tue Oct 4 21:25:09 EDT 2016 The vice presidential debate is going on. Wow, these two should be the presidential candidates. We do live in a reality TV world, shit. Okay anyway. It's so easy for motivation to get busted. I was on the phone with Adi for almost 40 minutes just now, and that was with me winding it down, and making it feel like I was cutting it short. But there's emotional maturity in there for both of us, I think. -------------------------------------------------------------------------------- ## Tue Oct 4 15:45:36 EDT 2016 ### Development versus Production instance of SEO Notebook / Pipulate Well, here I am back on my Mac with Sierra macOS running. Goodbye OS X. No more ambiguity around with or without a space between OS and X. Hello ambiguity surrounding whether to capitalize M in MacOS or not. Difficult to get the font size in the default terminal window exactly right. I want it somewhere between 24 and 36, and it doesn't appear to let you type it in. No matter, a few seconds of setup for a wonderful full-screen Mac Unix terminal is no big deal. And I'm back in business. Do a commit and push from here, to make sure your ssh credentials aren't blown away... yep, all good. Okay now... let's see. Back to what I've been trying to do for the past 2 days: hits against the feed... or possibly using proxies to bounce SERP checks off of. I'd like to avoid doing that if I can. 1, 2, 3... 1? There's still something about these configuration variables. I want... hmmm. There's an organization and management and privacy and boilerplate example issue here. I want to leave the very best examples in place for the Github repo, but I also want my main work to now be in an external include file. Think! It could possibly even go into private.py, since that's already in play now. Everything you want to keep private goes into private.py, including workflows. Think! Hmmm, okay. I want to start getting more into workflow. Some of the configuration values really do belong in a per-Pipulate or per-Populate basis. Shit, okay. Let's make that happen. Okay, obliterating the Configuration named tuple approach. The only "configuration" variable left not being handled through the Pipulate and Populate APIs is repeat_seconds, which can't really be done on a per-item basis in workflows. It applies to the entire workflow. Tue Oct 4 17:36:59 EDT 2016 Hmmm, okay. I just did a wonderful round of clean-up. It's all about clean and readable APIs now. Do I want to actually try to get anything done RIGHT NOW? I think I'm pretty well burned out for today. Okay, the big breakthrough today is that I now have what is effectively a testing and development instance of Pipulate versus a production instance. Wow, and it feels fabulous. -------------------------------------------------------------------------------- ## Tue, Oct 04, 2016 3:40:01 PM ### Mac Sierra macOS Upgrade Wiped out Unix Terminal Settings Correction. One more journal update on Windows. The Terminal settings got all messed up, back to the little-white-box. Got it to be black and Courier New again, but lost git. Gotta reinstall xcode developer tools: xcode-select --install Okay, got my new serial number and activated the new ScreenFlow while I was waiting. Okay, xcode-select installed. Take Two! -------------------------------------------------------------------------------- ## Tue, Oct 04, 2016 2:41:48 PM ### Sierra macOS Installed - ScreenFlow Broke Okay, journaling again on my work-provided Windows 7 laptop while I install Sierra macOS on my personal 2011 Macbook Air laptop. I didn't need that much room free. I'll have to check ScreenFlow and VMWare Fusion immediately after the install is done. Those tend to be a bit sensitive to OS upgrades, and they're both relatively old versions -- the oldest I could stay at after my Maverick update a few years back. There's about 11 minutes later. I'm trying to not be distracted, but it's hard. Today is also the big Oct 4 Google announcement, with their: (https://madeby.google.com/) - Pixel Phone - Google Home Assistant (Alexa-like) - VR Headset (Daydream View) - Google WiFi Router - Streaming (Chromecast / ambiguous naming) What shitty naming conventions again. Sheesh, can't they make just 5 clean easy to list bullet points. I guess it should really go something like: - Google Phone - Google Assistant - Google VR - Google WiFi - Google TV Okay, Sierra is installed. VMWare Fusion 7.1.3 is working on Sierra, but not ScreenFlow... don't even know what version I was on (3.x, I think -- resisted upgrading for a long time), but it stopped working. I got the crossed-out icon, and am downloading the trial version of 6.1 now, which will require a new serial number, and looks like a $34 upgrade. Okay, upgraded. Few pieces of software could get me to do a paid upgrade so easily, but Screenflow has really proven itself over the years. Now, I'm just waiting for the new serial number to arrive. Okay, commit this and pull your journal up over on your new Sierra macOS-enhanced Mac. -------------------------------------------------------------------------------- ## Tue Oct 4 09:40:53 EDT 2016 ### Scheduling under Jupyter Notebook Success Okay, time to submerge for a few hours. Yeah, talk about the process of descending into the flow/zone as part of what you do in your journal here. If anybody ever really reads this stuff, then the self-discipline and mental conditioning to "go there" is one of the things that will probably be of more interest. Altered states, all that. Not quite fugue, but something like that is what you're going for. The stimulation coming into all your senses is quite important. In the modern office environment (often, open cubicles), you can't be a prima donna about what you require to get into the zone, and so being able to actually do so, AND PRODUCE correspondingly inspired work from a "place" inaccessible to most people, is one of the great abilities and measures of an extraordinary employee. Hmmm. I want that to be me. So, hold your nose and dive! The opening music in a playlist is very important. It's the slow controlled descent, getting your body to achieve neutral buoyancy. I use bouncy rhythmic high energy stuff. Now, for the work. I am still actually jugging a lot that threatens to pull me out of the zone -- in a lot of ways, this journal itself. But it's a worthwhile risk, because I'm not actually hired as a full-time dedicated programmer. I'm in a marketing department, and I'm spinning a lot of plates. In particular, I'm my own sysadmin and devops, so I deal with crazy stuff like keeping my own scheduled execution environment running to back-end reports. Just did a wake-up call for Aid... case in point! Schedules and schedulers! That is my life now... ugh! Okay, the more automation the better. Gradually rise to force-of-nature likeness. Not too hard to do. Just requires a consistent build... oh, and in such a way that you're not building a house of cards. Resiliency, robustness, flexibility, and general not-brittleness are essential and urgent. Jupyter Notebook WORKING as an OS-independent scheduling environment contributes to that goal considerably, I think. Test still in process to that effect, actually. What does closing the tab that a Jupyter Notebook script is running in ACTUALLY DO? Cancel it? Keep it running in the Python VM in the background? Okay, it is urgent that I clean up all the Job Configuration stuff. I have the workflow dict of dicts, which is pretty good, but there's all these other variables that are tied to the workflows that are external to the workflow dict of dicts. Maybe incorporate them in. What are they? The objective here is just to switch between fairly dramatically different behaviors just by changing the workflow_name. I shouldn't have to remember to adjust this-and-that... hmmm. Okay, inventory those "environment" variables: - tab_name - sheet_name - start_date - end_date - site - feed - gaid - repeat_loop - rows_to_batch - chunk_limit Okay, make a bit of consistency: - tab_name - sheet_name - start_date - end_date - site_url - feed_url - google_gaid - repeat_seconds - rows_to_batch - chunk_limit Yup, these will all be in a second dictionary, that can work in lock-step OR independently from workflow. Hmmm. No, it's going to get mixed into workflows. You can/must be able to change these configuration values during workflow processing. One entry in a workflow uses one set of configuration values, and another uses another. Is that viable? Because workflows themselves can get scheduled... and you shouldn't be able to change scheduling details from within a schedule, or you get mindblowing complexity and unpredictability. Okay, many of those values actually belong in workflow and not in configuration. They "leaked out" because I was making abbreviations for their variable names, so the workflows won't look ugly. Hmmm. Okay, the only things surviving into the configurations dictionary are: - oauth_file - repeat_seconds - rows_to_batch - chunk_limit ...everything else I was talking about goes into a set of abbreviations that can be used in any of the workflows. I think I subdivided things just right for day-to-day usability. Newsflash! I just looked at the spreadsheet for the Jupyter Notebook job that had its tab closed and opened, and new rows have been inserted! BAM! I've got at least semi-reliable Jupyter Notebook scheduling. Those Python VMs running in the background are not really so fragile that closing their tab stop the running tasks (unless you're explicit about stopping it). Okay, confirmed again! Scheduling with the sched module from a Jupyter Notebook browser tab continues running after the tab is closed. I just saw another bunch of rows inserted, so now if I keep that VM running for for the SEO Pulse reports, I'm also keeping it running for... hmm. Pipulate! Woot! I've got to change the name of this thing back to Pipulate sometime soon. I should time it with the introduction-to-the-system videos that are coming down the pike. -------------------------------------------------------------------------------- ## Tue Oct 4 09:10:26 EDT 2016 ### OAuth2 Refresh Success Okay, the prolonged OAuth2 sustained login from Jupyter Notebook on the Kubuntu VirtualBox machine was a total success. It ran since I started it yesterday, and is still running now. Here is it's output: You are logged in as [gmail address] WORKFLOW BEGINNING: monitor Append request being processed... Checking for "Feed" in "Monitor"... Appending to "Feed" in "Monitor"... List appended! WORKFLOW COMPLETE! Setting a 3600 second loop for next workflow. WORKFLOW BEGINNING: monitor Append request being processed... Checking for "Feed" in "Monitor"... Appending to "Feed" in "Monitor"... List appended! WORKFLOW COMPLETE! Setting a 3600 second loop for next workflow. WORKFLOW BEGINNING: monitor Append request being processed... Checking for "Feed" in "Monitor"... Appending to "Feed" in "Monitor"... List appended! WORKFLOW COMPLETE! Setting a 3600 second loop for next workflow. WORKFLOW BEGINNING: monitor Append request being processed... Checking for "Feed" in "Monitor"... No rows to append. WORKFLOW COMPLETE! Setting a 3600 second loop for next workflow. WORKFLOW BEGINNING: monitor Append request being processed... Checking for "Feed" in "Monitor"... No rows to append. WORKFLOW COMPLETE! Setting a 3600 second loop for next workflow. WORKFLOW BEGINNING: monitor Append request being processed... Checking for "Feed" in "Monitor"... No rows to append. WORKFLOW COMPLETE! Setting a 3600 second loop for next workflow. WORKFLOW BEGINNING: monitor Append request being processed... Checking for "Feed" in "Monitor"... No rows to append. WORKFLOW COMPLETE! Setting a 3600 second loop for next workflow. WORKFLOW BEGINNING: monitor Append request being processed... Checking for "Feed" in "Monitor"... No rows to append. WORKFLOW COMPLETE! Setting a 3600 second loop for next workflow. WORKFLOW BEGINNING: monitor Append request being processed... Checking for "Feed" in "Monitor"... Appending to "Feed" in "Monitor"... List appended! WORKFLOW COMPLETE! Setting a 3600 second loop for next workflow. WORKFLOW BEGINNING: monitor Append request being processed... Checking for "Feed" in "Monitor"... No rows to append. WORKFLOW COMPLETE! Setting a 3600 second loop for next workflow. WORKFLOW BEGINNING: monitor Append request being processed... Checking for "Feed" in "Monitor"... No rows to append. WORKFLOW COMPLETE! Setting a 3600 second loop for next workflow. WORKFLOW BEGINNING: monitor Append request being processed... Checking for "Feed" in "Monitor"... No rows to append. WORKFLOW COMPLETE! Setting a 3600 second loop for next workflow. WORKFLOW BEGINNING: monitor Append request being processed... Checking for "Feed" in "Monitor"... No rows to append. WORKFLOW COMPLETE! Setting a 3600 second loop for next workflow. WORKFLOW BEGINNING: monitor Append request being processed... Checking for "Feed" in "Monitor"... No rows to append. WORKFLOW COMPLETE! Setting a 3600 second loop for next workflow. WORKFLOW BEGINNING: monitor Append request being processed... Checking for "Feed" in "Monitor"... Appending to "Feed" in "Monitor"... List appended! WORKFLOW COMPLETE! Setting a 3600 second loop for next workflow. WORKFLOW BEGINNING: monitor Append request being processed... Checking for "Feed" in "Monitor"... No rows to append. WORKFLOW COMPLETE! Setting a 3600 second loop for next workflow. WORKFLOW BEGINNING: monitor Append request being processed... Checking for "Feed" in "Monitor"... No rows to append. WORKFLOW COMPLETE! Setting a 3600 second loop for next workflow. WORKFLOW BEGINNING: monitor Append request being processed... Checking for "Feed" in "Monitor"... No rows to append. WORKFLOW COMPLETE! Setting a 3600 second loop for next workflow. This is fairly huge. Now to test whether a still-running task like this is tied to the open tab in the browser, or the virtual machine task. Close Firefox and re-open it. Done. Now, don't rush it. Leave it running. Mark the last row in the sheet with a highlight color. And now, just leave it. See if new rows get inserted after an hour-window goes passes. Count those entries... okay, it ran for 18 hours. That's 18 loops. And I see JUST NOW that the reports just ran too. This is huge! I've got 2 different Python execution environments with scheduling running: - A Python 2.7 file.py in a Debian-like /etc/cron.daily/ location - A Python 3.5 jupyter.ipynb Notebook using the standard library's sched module I've got a 2-hour block of meetings today that could totally throw me out of the flow. Focus the best you can on getting the Citation Monitor done. You never really got into the flow yesterday on new features, but that's okay because you can't overestimate the importance of getting smooth OAuth2 token renewal working -- FINALLY, like over 2 iterations of Pipulate, almost. Version #1 worked because ultimately it was the cron scheduling model, and that forced complete exit and re-entry of the script, which did the re-initialization magic. However now that I've learned to use a callable() instead of a property, I'm getting the token refreshed whenever I need it. -------------------------------------------------------------------------------- ## Mon Oct 3 09:54:50 EDT 2016 ### Running Jupyter Notebook Scheduled Tasks on Virtual Linux Desktop Another fine weekend, by some measures. Fairly crappy by others. Total wash on the organization front. But I did get Adi to the Maker Faire. However, she was particularly into punching and biting this weekend, justifying it as its her only time she's able to let her "animal side" out with me, and that she cannot help it. I started to discipline her, and it's the same particular freak-out in me standing up for myself in any way, shape or form that has to a great extent been the hallmark of the past 10 years. I cannot tolerate it from my own daughter now, or there will be an increasing wedge driven between her and me. I suppose it's aggravated by being under tremendous stress right now as well. Jewish new years, my ass. This is precisely the type of time where I'm going to be forcing myself back-on-track instead of being knocked off the fucking wagon by out-of-whack expectations. Bear down and do great! Okay, it's Monday morning. You're going to have to do very well on multiple simultaneous fronts! It's already October 3rd, and that is significant in several ways. Focus on solutions. Provide them fast, now. Learn your lessons of ages past. Climb back on the wagon, grab the reins, and get this wild ride back under control. 1, 2, 3... 1? Well, it's Monday Morning, so step #1 is the weekly report. Hop to it! Get some coffee first. Mon Oct 3 11:09:00 EDT 2016 Okay, weekly report delivered and an email of the SEO Dashboard changes delivered to one of the stakeholders. Think through this week like performance art. Go for major credibility gains. Don't fuck up. Think things through logically, and go for the biggest wins satisfying the largest number of stakeholders in the shortest period of time. ALSO go for knocking things off of deliverable and to-do lists. Fix the one feed that's reported to not be working. Mon Oct 3 11:25:14 EDT 2016 Okay, there was actually nothing wrong with the feed. Just crossed wires regarding the naming convention. Pshwew! Things going right. Okay, that virtual machine running on my laptop is actually one of the things going very right. It's a bit scary that it's my intended-to-carry-around machine, but I almost never un-dock it. It's time to let my DevOps counterpart know about the change in the reporting system. Okay, also did some spot-checking & Q/A work on the menu project. It's getting ready to go live. Okay, actually go get some lunch. Got it. Sushi. Eating it now. Listing to "Not Alone" on Audible. Okay, done and now getting into the coding mood. Interesting, the function of my "high energy focus" music playlist. It almost feels like I'm a scuba diver submerging slowly to avoid a mental sort of dysbarism. I don't know why the issues of zone and flow are not much more mainstream and commonly discussed. Get your system to the point where it can reliably do its thing. I'm still fighting OAuth2 timeouts, it seems. Let's do some housecleaning before getting ambitious this morning. Finally, I have logging implemented, though I'm not using it to great effect yet. But first, it's figuring out why: "Reset & Clear Output" ...from Jupyter Notebook refreshes the oauth token correctly, while putting: goodsheet.refreshcreds() ...elsewhere in the code doesn't. Hmmm. Okay, adjusted how goodsheet connections work, so that it goes through full "initialization", such as it were. There's no longer goodsheet.connect, but rather there's goodsheet.oauth(), which doesn't try to recycle the original connection over and over, but instead goes through the entire oauth verify and authorize on every connection attempt. A bit expensive, but worth it. Okay, I stripped all the unnecessary functions out of the SEO Notebook instance of goodsheet, and am sorely tempted to try to wrap in the few remaining goodsheet functions into either pipulate.py or functions.py, but it's premature to do so, and risks too much for too little return, and so I'm just going to get rid of the duplicate instance of create_google_service, and purge all references to goodsheet.connect and replace with goodsheet.oauth(). Okay, I just committed and ran Pipulate at 1:30 PM. Around 2:30, I should see if the OAuth... ugh, no! Just assume for now you nipped this one in the bud for now, and test it at night. I can't do something that creates a check-every-hour and "keep the tab open" sub-task through today. I have too much other fully focused work to do. Just remember to test scheduling again tonight! The problem is that I'm using the default 1-hour Google oauth2 default timeout, and I'm setting scheduling for every hour (not fighting it), so my test cycle is 1-hour, and that's too long for during the day on my main work machine. Very close to moving this stuff over to the VM. Sure why not? It's under-utilized right now with the reports every night, and incurs no paid cloud service liability. Hmmm. Maybe install miniconda only. Yes, this is a good plan. A Jupyter Notebook "miniconda" server for scheduled Pipulate tasks. I can pull repo snapshots and set them running, giving the same attention to keeping that VM running as I do with report.py. Oh yeah, finish my email to the devops counterpart that I started this morning... Mon Oct 3 14:06:21 EDT 2016 Okay, progress doesn't always happen smoothly. It happens in fits and spurts. And getting Miniconda installed on my Kubuntu VM for a Jupyter Notebook 3.5 pseudo-server is one of those things. Hmmm. Okay, got it installed. Didn't add it to my .bashrc path, for fear of messing up the Python 2.7 reports. Here was the final message: Do you wish the installer to prepend the Miniconda3 install location to PATH in your /home/miklevin/.bashrc ? [yes|no] [no] >>> You may wish to edit your .bashrc or prepend the Miniconda3 install location: $ export PATH=/home/miklevin/miniconda3/bin:$PATH Thank you for installing Miniconda3! Share your notebooks and packages on Anaconda Cloud! Sign up for free: https://anaconda.org Okay, and now typing jupyter notebook from a terminal... oops, not found. Okay, this is what conda install is for. Ugh, okay. I'm about to install a ton of stuff on this VM, but it's worth it. Double-check that the reports are running correctly afterwards. miklevin@kubuntuvbox:~/miniconda3/bin$ ./conda install jupyter Fetching package metadata ....... Solving package specifications: .......... Package plan for installation in environment /home/miklevin/miniconda3: The following packages will be downloaded: package | build ---------------------------|----------------- conda-env-2.6.0 | 0 502 B expat-2.1.0 | 0 365 KB icu-54.1 | 0 11.3 MB jpeg-8d | 2 806 KB libffi-3.2.1 | 0 36 KB libgcc-5.2.0 | 0 1.1 MB libsodium-1.0.10 | 0 1.2 MB libxcb-1.12 | 0 1.5 MB dbus-1.10.10 | 0 2.4 MB glib-2.43.0 | 1 5.4 MB libpng-1.6.22 | 0 214 KB libxml2-2.9.2 | 0 4.2 MB zeromq-4.1.4 | 0 4.1 MB decorator-4.0.10 | py35_0 12 KB entrypoints-0.2.2 | py35_0 6 KB freetype-2.5.5 | 1 2.5 MB gstreamer-1.8.0 | 0 2.6 MB ipython_genutils-0.1.0 | py35_0 33 KB jsonschema-2.5.1 | py35_0 56 KB markupsafe-0.23 | py35_2 32 KB mistune-0.7.3 | py35_0 653 KB path.py-8.2.1 | py35_0 47 KB ptyprocess-0.5.1 | py35_0 19 KB pygments-2.1.3 | py35_0 1.2 MB pytz-2016.7 | py35_0 176 KB pyzmq-15.4.0 | py35_0 800 KB simplegeneric-0.8.1 | py35_1 7 KB sip-4.18 | py35_0 240 KB six-1.10.0 | py35_0 17 KB tornado-4.4.2 | py35_0 597 KB wcwidth-0.1.7 | py35_0 21 KB clyent-1.2.2 | py35_0 15 KB conda-4.2.9 | py35_0 374 KB fontconfig-2.11.1 | 6 405 KB gst-plugins-base-1.8.0 | 0 3.1 MB jinja2-2.8 | py35_1 302 KB pexpect-4.0.1 | py35_0 66 KB pickleshare-0.7.4 | py35_0 9 KB prompt_toolkit-1.0.3 | py35_0 308 KB python-dateutil-2.5.3 | py35_0 237 KB terminado-0.6 | py35_0 18 KB traitlets-4.3.1 | py35_0 124 KB anaconda-client-1.5.1 | py35_0 112 KB ipython-5.1.0 | py35_0 991 KB jupyter_core-4.2.0 | py35_0 52 KB qt-5.6.0 | 0 43.9 MB jupyter_client-4.4.0 | py35_0 99 KB nbformat-4.1.0 | py35_0 117 KB pyqt-5.6.0 | py35_0 5.4 MB ipykernel-4.5.0 | py35_0 126 KB nbconvert-4.2.0 | py35_0 315 KB jupyter_console-5.0.0 | py35_0 29 KB notebook-4.2.3 | py35_0 5.0 MB qtconsole-4.2.1 | py35_1 152 KB widgetsnbextension-1.2.6 | py35_0 1.1 MB ipywidgets-5.2.2 | py35_0 59 KB jupyter-1.0.0 | py35_3 3 KB nb_anacondacloud-1.2.0 | py35_0 23 KB nb_conda_kernels-2.0.0 | py35_0 30 KB nb_conda-2.0.0 | py35_0 28 KB _nb_ext_conf-0.3.0 | py35_0 966 B nbpresent-3.0.2 | py35_0 463 KB ------------------------------------------------------------ Total: 104.4 MB The following NEW packages will be INSTALLED: _nb_ext_conf: 0.3.0-py35_0 anaconda-client: 1.5.1-py35_0 clyent: 1.2.2-py35_0 dbus: 1.10.10-0 decorator: 4.0.10-py35_0 entrypoints: 0.2.2-py35_0 expat: 2.1.0-0 fontconfig: 2.11.1-6 freetype: 2.5.5-1 glib: 2.43.0-1 gst-plugins-base: 1.8.0-0 gstreamer: 1.8.0-0 icu: 54.1-0 ipykernel: 4.5.0-py35_0 ipython: 5.1.0-py35_0 ipython_genutils: 0.1.0-py35_0 ipywidgets: 5.2.2-py35_0 jinja2: 2.8-py35_1 jpeg: 8d-2 jsonschema: 2.5.1-py35_0 jupyter: 1.0.0-py35_3 jupyter_client: 4.4.0-py35_0 jupyter_console: 5.0.0-py35_0 jupyter_core: 4.2.0-py35_0 libffi: 3.2.1-0 libgcc: 5.2.0-0 libpng: 1.6.22-0 libsodium: 1.0.10-0 libxcb: 1.12-0 libxml2: 2.9.2-0 markupsafe: 0.23-py35_2 mistune: 0.7.3-py35_0 nb_anacondacloud: 1.2.0-py35_0 nb_conda: 2.0.0-py35_0 nb_conda_kernels: 2.0.0-py35_0 nbconvert: 4.2.0-py35_0 nbformat: 4.1.0-py35_0 nbpresent: 3.0.2-py35_0 notebook: 4.2.3-py35_0 path.py: 8.2.1-py35_0 pexpect: 4.0.1-py35_0 pickleshare: 0.7.4-py35_0 prompt_toolkit: 1.0.3-py35_0 ptyprocess: 0.5.1-py35_0 pygments: 2.1.3-py35_0 pyqt: 5.6.0-py35_0 python-dateutil: 2.5.3-py35_0 pytz: 2016.7-py35_0 pyzmq: 15.4.0-py35_0 qt: 5.6.0-0 qtconsole: 4.2.1-py35_1 simplegeneric: 0.8.1-py35_1 sip: 4.18-py35_0 six: 1.10.0-py35_0 terminado: 0.6-py35_0 tornado: 4.4.2-py35_0 traitlets: 4.3.1-py35_0 wcwidth: 0.1.7-py35_0 widgetsnbextension: 1.2.6-py35_0 zeromq: 4.1.4-0 The following packages will be UPDATED: conda: 4.1.11-py35_0 --> 4.2.9-py35_0 conda-env: 2.5.2-py35_0 --> 2.6.0-0 Proceed ([y]/n)? Okay, it's installed. Because I didn't alter my path, I have to go into: ~/miniconda3/bin/jupyter notebook Okay, now that it's running (yay!) do a pull of SEONotebook on this VM, and get your latest job running. It can help remind me of the things that need to be moved into place too. Hmm. Okay, do an apt-get update and upgrade and reboot this VM. Get it into a nice patched up to the moment state. Reboot, and confirmed, the reports still run, even though I didn't wait for them which is fine, because it's default Python executable and paths that I'm testing, and they're all set to the old 2.7 standard. ~/miniconda3/bin/conda create -n py35 python=3.5 jupyter source ~/miniconda3/bin/activate py35 pip install gspread pip install httplib2 pip install google-api-python-client pip install bs4 pip install lxml private.py And there it is. All dependencies met, and scheduling of the Jupyter Notebook stuff is occurring now on the 24x7 virtual machine side of operations. Hmmm. I also got a phone-call about the tax stuff finally hitting Rachel. Now shit gets real, and I have to clean this stuff up pronto. Time will be made now. Well, at any rate, this is a good stopping point. Maybe it's time to go home and continue on the organization front, before all my energy for the day is spent. I don't have a second to breath sideways, except for my commute. Shit, even now that I'm separated I have the time issues. How can that be? It's because after a full day of work, it's really hard to do this shit, and I have exactly zero weekends otherwise, or have to call in sick days and trying to cram it all in there. It isn't right. Okay, good. Crisis will help me fix all this. Let me get going, and if there's any questions, I just got that critical Jupyter Notebook scheduling piece working, and I need to let a few hours pass for the test. -------------------------------------------------------------------------------- ## Sun Oct 2 16:22:20 EDT 2016 ### My Phoenix Project Today is as lazy and around the apartment as yesterday was active at the Maker Faire at the New York Hall of Science. It's our 3rd time three, though of course Adi doesn't remember the 1st 2. Those times were mostly for me and Rachel, and just setting the right tone for Adi moving forward. Missing it last year was a sign of how much I didn't have my act together and had my head underwater this time last year. As much worse of as I am in several ways this time around, I have my head on much straighter. I have to listen to myself more often and others less. Nobody has my own interests at heart as much as I do, and that's not a selfish statement either. If I don't have my act together for myself first and for my own sanity first, then what chance do I have at being a good father to Adi? None. Her seeing me slowly disintegrate, because I care about the feelings of others more than my own is an even worse proposition, and so I'm separated and gradually digging my way out of -- well, realistically even MORE than 10 years worth of accumulation and baggage. It really all began to spin out of control -- when? In my childhood when I was choked off for all affection and happiness by a worse-than-jealous sibling. I wanted to just disappear, and that pretty much still always hangs over me. My high-profile-ness on the Internet these days is almost contrary to my nature, except for that magician deep inside me that still hungers for approval and affirmation. Shit, I'm still that little kid, and barely any more responsible than that too. Stuff needs fixing, and I can only realistically do that in solitude-mode. Deep, defective solitude-mode, which I still do not have, really. And this is really not even my first time in a mess like this. My Dad dying and me having to take over the fucking check cashing was a major contributor too. Yet, it's not external things. All my shortcomings and failures to adapt gracefully to the rules is my own fault. I have to look inward to fix things outward. I have to alter my own long-lived and comfy but ultimately self-destructive behaviors. I have to do that every day. If I were a factory floor, the 3 ways would be: 1. A deep familiarity with and appreciation for the system. Know the why's. 2. Intrepidly trying small incremental continuously deployed improvements. 3. Become an expert at understanding and doing more of what's working well. Knowledge, action and expertise. Hmmm. Don't sit here typing too long. Enter phases 2 & 3 by getting up and doing stuff. Small, incremental improvements in your life -- until you are who and what you want to be, and an expert at living life, and able to help instil into Adi that same attitude and love for life... fake it until you make it. Gulp! -------------------------------------------------------------------------------- ## Sat Oct 1 08:22:42 EDT 2016 ### The Makings of a Tech Manifesto, haha. Let's Call It a Book Plan Well, here I am the morning of the Maker Faire. Was work getting Adi to bed last night, and didn't happen until around midnight. I'm going to have to wake her up soon. I should read over the schedule too, before she wakes up so we can make good use of our time today. Turn on the lights. Done. The lesson of the Amiga computer to me should be: don't tie yourself excessively to a particular hardware implementation, unless you want a quasi-religious experience, and then eventual heartbreak. Instead, focus on the underlying bits that glue it all together just high enough in abstraction layers that you don't need to write your own hardware drivers. But you do need to go through an API to talk to everything. There's pretty much no "hitting the hardware", and therefore, you are probably not going to be much like a mobile app developer, unless you adopt a browser-is-the-hardware mindset. If so, you're heartbreak is just deferred another 20-years or so, because don't fool yourself, things WILL change again drastically, like base user interface assumption-wise. Yup, floating augmented reality holograms and stuff. Yet somehow, I still suspect when you look under the hood of all that nanodatacenter tech, you'll find some sort of Unix/Linux boot kernel, and then something a lot like BusyBox providing the basic Unix command-set, and probably the gcc compiler and just enough to do the Aboriginal Linux trick -- define itself and build itself. That's all quite doable under today's tech -- minus the floating holograms, we're at the edge of a breakthrough where truly viable, helpful, economical and seemingly smart robots will be possible. It's just a matter of optimizing and achieving economies of scale with a bunch of stuff we already figured out, like computer-vision and fairly smooth, albeit not Turing Test passing, conversation. - Think in Python, and think in vim. - To be a true wizard, think in LISP and emacs. - Some were meant to go the Samurai way of vim. - Others were meant to fold steel into million-layer blades. - Kooky stuff wins for a few, while solidly-reliable wins for the masses. - Few Samurai's made their own blades. - There's not enough time in life for everything. - We each choose our own path, but it should land on one side or the other of the browser wars. - Unless you go the Java/C way, and only live in IDE's and text editors du jour. - Or you've succumbed to JavaScript as the one language to rule them all. - And you probably use Atom -- which reveals a cool piece of the future. - Unless you still use. Sublime. Or maybe TextMate. Edit Plus? pfe? ced? - There are many good paths, but some should just be sub-paths. - JavaScript for UI/UX is a sub-path, like PostScript before it, and curses before that. - I'm stuck in curses. There's something future-friendly about text terimals. - Text terminals seem more future-friendly than pointer-on-screen desktops. - Unless it's a system meant to reboot society with an illterate population who might appreciate a point-and-click UI. - But let's be optimists, and so within our contest, state the challenge. - Pick a path. Pick any path, and I will predict the success-distribution probability-curve of that path (not that I've done THAT well for myself, mind you). - Your prevailing path should be the one that lets you: - Address the broadest array of problems. - Solve those problems in elegant ways you can be proud of. - Make a living by doing it. - Share and improve. - Keep your secrets secret, while sharing plenty to build noteriety. - Stay connected and grounded in your loves, and not lose your way. - Rapidly adjust to new opportunites that come alog. - Support happy schizms, because you can't satisfy everyone. - Nor should you try. That's what Github is for. -------------------------------------------------------------------------------- ## Thu Sep 29 20:10:45 EDT 2016 ### Robert Ringer is the only good thing to come out of the real estate industry Journaling AGAIN now that I'm home. I suppose there's far worse stuff I could be addicted to. In fact, I could do a lot better for myself with MORE of this and not less. The time that's not reflected here goes directly into Adi, work, the kitties, or completely non-productive stress relieving pastimes. I have to just provide easier solutions to things. Divest myself of burdens. If someone or something makes stuff shitty and difficult for me, bet that something or someone out of your life. I need to pay more attention to Robert Ringer, just about the only good thing to ever come out of the real estate industry. Though I probably am guilty of ego violations here. Is this about ego? I guess yeah in part, for sure, but I more so just think of it as thinking out loud and improving my writing, vim and coding skills. ======= ## Fri Sep 30 17:17:25 EDT 2016 ### Scheduling Okay, back on the Mac. Don't want to desensitize my fingers, such as it were. Doing journal entries on the Mac is completely the correct things. The low-profile but high quality keys are still just right. Okay, heading out now. Ugh! Setting only ONE sceduled Pipulate job and one scheduled Kubuntu cron job. -------------------------------------------------------------------------------- ## Fri, Sep 30, 2016 4:33:05 PM ### Switching to Windows for Journal for Experiment I'm pulling my journal over to my Windows machine for a bit as a funny sort of experiment. I have some ideas that I may want to act on if this works out, but it's part of a sort of breather and experiment I'm giving myself right now after a full day of fairly intensive work. I can type really fast, and I'm on the dasKeyboard pro, and I have some theories about it that need testing. Yes, I think I might be correct. Okay then. Monitor the job you're doing and think about next steps. Keep yourself typing as much as you possibly can as you switch back and forth between screens. I'm on about row 80 of 1500+ in processing the headline serps, and it's going to take quite awhile to get through them. And while it's running, I want to do a number of things to check over the durability of the code, along with getting the new SEO Pulse Report feed swapped out finally in the scheduled cron job on my Kubuntu VM. Okay, 1, 2, 3... 1? Let's do those feeds first and get it out of the way. Just keep checking on the Pipulate SERP job to make sure it keeps running. Give just enough of a breather between typing sessions to... well, give a breather, but then crank it back up with my confident and intelligent typing sounds. I need a second terminal window on Windows now that I'm using my main one for this, but I won't be going the tmux route again. Full-screen text windows is really the only way to go on Windows 7, switching between screens with VirtuaWin. Okay, I can git clone into sequential seonb[x] named folders. Just made seonb1. I will have to copy various files around. To avoid having to authenticate again, I can copy over oauth.dat, but all the other stuff I can as well. It's important to note that the caches are always local to these locations, and I always CAN (but don't have to) copy the caches around. The important files are: - private.py - google_ids.py Okay, I interrupted the every 41-seconds SERP job to do some post-processing. Very nervous about that captcha quota kicking in. I have some theories to check, like manually passing the captcha check on a browser. Check that now... Yep, the job had stopped because one of my functions was returning a None object where it was not allowed to. I fixed that, and also put in a 5-minute timeout if the SERP http fetching returns anything other than a 200. That should take care of the Captcha problem. If not, I can up that to a half-hour or so, and let it run through the weekend. In the meanwhile... hmm. Okay, this is your chance to finally swap out the Pulse Report feed-generating script. Take a deep breath... ugh. Okay. This is on your Kubuntu VM. Go look at it first. Refresh yourself on what's what. Okay, it's referring to report.py in the actual git repo location! All I have to do is change the path to the new location. That's an easy undo redo. Do a stash of the current status. Okay, I got the new script running. I'm doing a test run now. Switch back to the Mac. Test completed. -------------------------------------------------------------------------------- ## Fri Sep 30 11:32:35 EDT 2016 ### Marathon Entry on Beginning of SERP Monitoring Okay, going for broke, as they say. Uploaded the 3 queued-up videos from my walks, plus one of the still unpublished journal recordings from yesterday. And so... I'm at a really interesting point in my code. I need to get something done quick! Early in the day would have been best, but doing it by the 1:00 PM phone-call will be good too. At least have it running. Process each headline only once from the Atom feed, and really only do it in the traditional Pipulate() method way. Even within your system, there is the manual versus the automated way of doing things. The non-scheduled process IS the manual process. Doing it against ONE row with ONE function is the same as doing it against ALL rows, which is the beauty. Pipulate handles the "outer loop" such as it were. So, get to it! 1, 2, 3... 1? Make the function. Nickname it! Citations. It's all about citations. So, the function should just be... hmmm. Wait. It's a 2-pass thing, if you want to keep it simple. One pass grabs the SERPs per-row. DON'T stuff the SERP results into the Google Sheet. That's precisely the overweight file bloat problem I'm trying to avoid, especially with 100 rows per result, so... I already have a lovely caching system. It's only in the Pipulate() process. Maybe externalize the caching piece from there, and make it reusable for this other purpose! Yes. You will be able to be able to create a [file].dat in the local repo folder for any function that uses it, where the filename is the same name as the function. I already intuitively did that with the URL column... which is actually an input parameter that makes an html argument (actually a Requests response object) available automatically (pre-fetched HTML) to any other Pipulate functions on that row, through the **kwargs mechanism. And so, what I'm building now is... tearing out that caching functionality. Did I mention all those response objects were being cached locally as urls.dat? Well, they are. And I'm tearing out that bit of logic into an external function that can do it generically with any function. Any function can support caching whatever it likes in a shelved object that goes by the same name as the function itself. Mmmm. Brilliant. Get to it. Okay, this is NOT a time to "work in reverse" tearing out code I know I want to externalize... because maybe I don't. Looking at it in context, I realize the Python code is so minimal, and tied to the use-case, it is not ripe for externalizing, but rather is an argument for just repeating yourself -- when what you said was already terse, an exactly right granularity application of an API expression, and easily re-stated in a slightly different way for a different use case. So there, DRY crowd. You're all WET. I've got a really strong identity for this function: def verbatim100(**kwargs): # stuff ...woot! This is going to be very strong. I'm going to go back to the conventional **kwargs for arguments. This won't need to be decorated, because I'm going to do all the processing inside the function for now. I want to control all the details of the caching. Also, I should probably change the URL column label in the Google Sheet, so I don't unnecessarily cache all those article URLs. So, go make a copy of the Monitor's Feed tab... Okay, done, and I see how the whole purpose of the verbatim100 function is to get those SERP results locally cached for later processing! Yes! Do this. Fri Sep 30 12:20:41 EDT 2016 Nope! I need to create a caching service that services verbatim100 first. That's the reuse-able bit: def url_catcher(url, dbname): # do the with open stuff And here it is: def url_cacher(url, dbname): with shelve.open(dbname) as db: if url in db.keys(): response = db[url] else: try: response = requests.get(url) db[url] = response #The moment of pickling except requests.exceptions.RequestException as e: print("Bad URL") return None return response url_cacher(url='http://mikelev.in/', dbname="miketest") That should just about do it. Wow. Okay, now all I have to do is construct the outer function that calls THIS as a resource. How useful! Fri Sep 30 14:03:19 EDT 2016 Okay, had my 1:00 PM call, and have a... well, I will have to edit this video after all -- just cutting out the dead-time. I didn't successfully pause recording when I hopped up. Okay, now... think! These meetings and touching-bases are important, but boy how do they throw off flow! def verbatim100(**kwargs): title = kwargs['title'] return Response(ok=True, status_code='200', text=title) Here is me returning back just the exact headline that was fed to the argument. Now, I have to construct the Google query parameters... And the non-cached version: def verbatim100(**kwargs): title = '"%s"' % kwargs['title'] url_endpoint = 'https://www.google.com/search' arg_dict = {'num': '100', 'tbs': 'li:1', 'q': title} response = requests.get(url_endpoint, params=arg_dict) return Response(ok=True, status_code='200', text=response) Okay, now to add the newly created external cache checker! And I got rid of the notion of using dictionaries to encode parameters: def verbatim100(**kwargs): from urllib.parse import quote_plus title = '"%s"' % kwargs['title'] endpoint = 'https://www.google.com/search' search = '%s?num=100&tbs=li%%3A1&q=%s' % (endpoint, quote_plus(title)) #response = requests.get(url_endpoint, params=arg_dict) return Response(ok=True, status_code='200', text=search) Very nice. Now, I can introduce the cache, because I made the keys of this system URLs, and I needed the URLs in order to use it. I can't feed them as separate data parts like I'm doing with the requests package. Okay, got the cache working. Now to load the data out of the cache and parse the URLs from the SERP results. Each 100-result SERP page probably has 100... not URLs, no. I need... well yes, I need URLs, but I also need the titles too - the undecorated titles. Okay, I have this code somewhere... oh yeah. Another similar project to this I did not long ago. Go find it! Okay, found it. I am able to pull out the response.text from the archives as well, so this bodes very well. Okay... I have a list of landing page URLs from the SERPs, but I need to "equalize" it with a prior list so I can do an "in" compare between 2 lists (or sets) to find positive hits. Okay, time is running out! It's quarter of four, and I still have a bit to go. I have basic reporting on "hits" happening, but they're all coming up negative on my small sample set, as one might expect. So, I need to collect the SERPs on a much longer list of headlines. Fri Sep 30 16:18:33 EDT 2016 Wow, this is shaping up to be quite a marathon session. I have to hit this home in a good way that will be satisfying going home for the weekend. It's time to spin several plates. It's time to take a few moments for yourself finally and think about where you are. - Get multiple Pipulate jobs going... wow! - Make sure they're working as multiple independent instances. - Get the new SEO Pulse Report feed in location. -------------------------------------------------------------------------------- ## Fri Sep 30 09:38:12 EDT 2016 ### Dissecting SERP parameter arguments for a citation monitor Okay, get right into it. Upload your YouTube videos, but get right into it. First of all, every moment today is precious, on all counts: work, personal finance and time with Adi. Shit! I just realized I forgot to do a wake-up call for Adi. Well, make it up to her. Shit, shit, shit! Okay, be your badass self, and don't be a sissy weakling. Don't beat yourself up over it, and certainly don't let others lunge at an advertised weakness -- so don't go advertising it. Call to talk to Adi during lunch or something. It's a ROC day, so your timing may have to be pretty good. Hmmm. What's the best next move? But also, be sure to apply that next-best-move thinking to your professional day here at work. You urgently now need to switch your scheduled feed on your Kubuntu VirtualBox to the new feed script. Yes! You need an easy go-back too. Just rename files in-location. Trace the scheduling thing again. I believe it abides by the cron.hourly Debian-like system. Also, the scheduled thing didn't happen correctly overnight because of an expired OAuth token, of course! Here was the error output: HTTPError: 401: b'<HTML>\n<HEAD>\n<TITLE>Token invalid - Invalid token: Stateless token expired</TITLE>\n</HEAD>\n<BODY BGCOLOR="#FFFFFF" TEXT="#000000">\n<H1>Token invalid - Invalid token: Stateless token expired</H1>\n<H2>Error 401</H2>\n</BODY>\n</HTML>\n' So, that's pretty self-explanatory. I know that tokens get refreshed whenever I reset and restart the whole Jupyter Notebook virtual machine environment for that tab, but I need to isolate the part that actually checks and refreshes the token, and ensure that that precedes any attempt to do anything that needs that authentication completed. Okay, I re-ran without a system reset and got it to work, so I moved the (repeated) goodsheet import into the scheduled loop. I'll have to do another test. I should get that working right away. I think I'm going to have to deal with the double-instance of SEO Notebook sooner rather than later. Clone it into a folder called scheduledtasks. Make that a very special folder where all the live (and mission critical -- gulp!) Jupyter Notebook-based scheduled tasks live -- probably all under one single giant workflow. I also liked having Jupyter Notebook in the background here while I journaled. If I am video-casting my journaling session, which yesterday's experiment showed was actually a viable alternative to my more carefully crafted coding session videos. MUCH easier to do. I actually have 2 more I did yesterday, which I'm sitting on top of. Just how much CAN I push out through my channels while I do my day-to-day work and get it done efficiently, without any excessive overhead? Why hold back? Why pull your punches. If this is what you want to do, and are good at doing, pump everything (that you're not embarrassed of) out as the background white noise of your main publishing channels. If people want anything more, or anything better prepared with higher production value and actual effort required, then make some sort of Python / general tech training class, and charge them money. All this stream-of-conciousness stuff is the first does free. If you want the REAL products of my labor, you have to pay for it. It's a fair bargain, because it keeps me doing what I do. Gee, I should be recording writing this. I could embed the videos of me writing any particular journal piece near the very text of being written in the journal in that video. In fact, that would be the natural side-effect of me "combing together" YouTube embeds of my videos into this journal, according to timestamps. Duhhh, that is so of course one of my next projects. This is precisely the sort of thing my system is shaping up to do. Scheduled customized Python tasks that are bast run-from, monitored and modified from Jupyter Notebook (soon, Jupyter Lab). Fri Sep 30 10:41:24 EDT 2016 Okay now ask yourself what the process is for each of these article line entries? I need to look for hits for a particular headline. Once I nail that as a single standard Pipulate function sort of thing, a side-effect can be adding an entry to another table. Some of the attributes of this system are: For every headline, an incognito-mode search is performed with 100 results-per-page. Oh, it should also be in verbatim-mode. Oh, ALSO in with double-quotes around the headline. Yes, it needs both verbatim AND double-quotes. And even then, it's not 100% exact-match. So be it. Here is an example of an exact-match search: https://www.google.com/search?num=100&biw=1242&bih=616&tbs=li%3A1& q=%22Penguin+is+now+part+of+our+core+algorithm%22&oq=%22Penguin+is+now+part+of+our+core+algorithm%22& gs_l=serp... It's interesting how Google uses "serp" as part of their own value parameters. I chopped off most of what was in the gs_l parameter value because who knows what it's giving away. Other parameters are more obvious like num & q. The others, not so much: biw bih tbs Oh yeah, browser inner width, browser inner hight and advance search. I need to keep: tbs=li:1 ...for Verbatim results. Oh yeah, gs_l is geolocation information. Ah, what a good way to cut down on geo-customized results. Okay, good, so when building the proper search parameter, the things I want are: num=100 tbs=li%3A1 q=%22encoded+search+query%22 Okay, I already have a good caching system in place for caching the Requests response object per URL. The same thing's going to apply here. I'm going to want to perform just one search per headline in the system. That is standard Pipulate behavior. I will add one more column to the Monitor tab... uh no wait, this scheduling stuff is running on that and working great. Peel off a copy and Pipulate against the copy, EVEN IF it's in the same spreadsheet. That'll be TWO Pipulate sessions against the same Google Sheet, each working on a different tab! I wonder if that would cause problems. Try to be conservative in these decisions. Separate Google Sheets is such a good isolation layer between tasks, you should try to take advantage of it. Don't deliberately make things more difficult for yourself. Using your own system should gradually begin to make all tasks EASIER! Just drop each thing into the right location, and peel off the correct Pipulate instances to support the peeled off bits. Think of Pipulate as a bunch of (mostly) VM instances that bud off of Github, using whatever hardware you happen to have available that can run Jupyter Notebook (I need a dedicated Jupyter Notebook server now). Hmmm. -------------------------------------------------------------------------------- ## Thu Sep 29 17:26:56 EDT 2016 ### Scheduling Added to SEO Notebook Okay, that was some flying by the seat of my pants figuring out of sched. There's some basics to fill in, and some good education about APIs. It's always best to use the optional argument labels in Python (precious horiz space permitting) so that you can know what the heck you're looking at. I have to remember there's all this parameter passing options available, but don't let it clutter things up for now. I'm facing some interesting choices now. I'll probably need to make a parallel cloned instance of the SEO Notebook repo from Github, so that I can leave a scheduled instance doing its thing ALL THE TIME. Whenever I interrupt it, I'll have to make sure I restart it every time, and also I will need to deal with not creating duplicate actions. Housekeeping. Reliable non-brittle housekeeping. Next step? Hmmm. Well, to really understand what's going on, we have to carry through the argument labels to the scheduler object itself! This is important, so that we can really understand... I guess it's a relative vs. absolute timing issue. You can have x-units of time since the last time you ran, or you can have "every day at x-o'clock". There's significantly different approaches to scheduling, and with a low-resource-consuming process, you can schedule it to run more often (avoiding overlap), and make sure your housekeeping process prevents duplicate actions, like causing duplicate records. Now is the creative or inspired part. SEO Notebook so far has been written to run when you hit the run button. It's an "attended" app, of the sort where you say Go! And it does its thing then finishes. It's like an unscheduled batch job. We need to support OPTIONAL scheduling without introducing befuddling complexity. Hmm, Google Apps now known as G Suite. Interesting. I want to leave today with the feed monitor running in a loop, even if I have to clean it up and alter it later. Make SEO Notebook in close to its current form simply support scheduling as an alternative mode in which you can run it. Put it into a loop similar to the type you would simply use the sleep or wait look with in the past without a specialized scheduling module. 1, 2, 3... 1? Import the sched module, dumbass... and I figured it all out and have it on an every hour loop. That's a good stopping point if I ever saw one. Let it run overnight and see how it did. BAM! Running every 3600 seconds. Leave it go. Even lock my Windows machine (I should theoretically even be able to close the tab, but I won't). And see how well it ran tomorrow morning. Diminishing returns. Running out of steam. But you've got the idea. This is how I work, and this is how ideal days go when I can get into the flow. The only downside is the whopping 1-hour commute each way that's total downtime, as far as real productivity is concerned. I could add 2 more hours a day to this sort of work if I could telecommute every once in awhile. That will be nice. Step #1: become a hero. Step #2: ask for stuff. -------------------------------------------------------------------------------- ## Thu Sep 29 16:53:04 EDT 2016 ### Figuring Out the sched Standard Python Library Module Okay, you get to see the sort of relentless pressing towards a goal that I use these journals to accomplish. It's coming up on 5:00 PM, and the normal work day is running out, but I cleared tonight to work late, because I really have to drive this project home. It's the recurring infrastructure tools I'll be using for many other projects in the future. It's got a monitoring component and a housekeeping component, and I'm about to add both a Google search monitoring (SERP) component, in addition to a scheduling component. At this point, the scheduling component is actually more important, I think. Okay, 1, 2, 3... first we decide what scheduler component to use. We want platform independence, insofar as we can get it, and that means neither: - Linux cron - Windows Scheduler What then? Easy, the Python standard library package called sched. Let's take a look... Okay, hmmm. This will be possible within Jupyter Notebook, but that notebook tab is going to have to be left (probably)... no wait, those Python virtual machines are actually left running even when you close a tab... woot! This is going to be very nice. 1, 2, 3... 1? Next step is going to be nailing what every minute scheduling is, then I'll switch it to hourly, then (likely) daily. But for testing, I want every minute testing. Okay, I've got the barebones scheduling framework. I deleted out a bunch of stuff having to do with default parameters and such, but this is enough to put Python (even under Jupyter Notebook) an every 5 seconds loop: import sched, time s = sched.scheduler(time.time, time.sleep) def print_time(a='default'): print("From print_time", time.time(), a) s.enter(5, 1, print_time) s.run() -------------------------------------------------------------------------------- ## Thu Sep 29 14:48:21 EDT 2016 ### Finished Atom Feed Monitoring & Housekeeping into Google Sheets Time for a new journal entry and a new video. Pushed up my first journalcasting experiment. Okay, populate_from_atom() is returning a generic list of lists for testing, and the Append() behavior is working very well. Next! Next, I need to make the function return a list of lists that's aware of the table it's inserting into, avoiding duplicates. Interesting! I need the same GData connection available in functions as I do in the main body of the program that does the orchestrating. That was a problem in my older systems, but it should not be so now. I sort of wish I could show you my coding screen as I go, to show you the development of this stuff instead of just my thoughts, but I'm not going to throw off my workflow, of my Mac being used for journaling, which also happens to have ScreenFlow, making this version of the process very easy. Windows can't even capture the keys I'm pressing with Camtasia Studio. I switched back to ScreenFlow on the Mac almost exclusively for screencasting, because it's soooo much better for my purposes than anything I found on Windows. Okay, this function needs to select the last-X rows from a table for comparison's sake. And so, I need a new worksheet instance created by GSpread. Done. With that, I need to get the last row of the sheet... and for that, I need to get worksheet.row_count and then worksheet.get_addr for it, plus the number of columns. That gives me the end of my range I'll be grabbing a cell list of. Then, I need to subtract a bunch from rows and do it again on column 1, and that gives me a nice rectangular cell range for comparison's sake against the feed I'm about to be processing. It's housekeeping, you see. Okay, confirmed. Functions have all the same capability with GSpread as the main program, and that's excellent. No handing around GDoc objects or anything. I'm so glad I have such a better grasp on namespaces and scopes and how imports work. Okay, I'm introducing the concept of first_row. If rows < 100, then first_row is just row 2. If rows > 100, then first_row is rows - 100. And that's how I get my start_range. Now, I have a new start_range and end_range. That's good. Hmmm. Grab that cell range. Interesting thing is that even though I'm grabbing a cell_list in this case (almost by habit at this point) which is what I do when I'm planning a batch update back into the spreadsheet, I'm not going to need to do that this time. And so, I have the chance to transform this cell object into something more familiar, like a list of lists. Then, I can do list comprehensions against it. Maybe I can do those without a preprocessing transform. Must test. That's to grab out columns as lists and such. Yep, confirmed. To grab the values out of a cell_list with list comprehensions is simply: [x.value for x in cell_range] Actually, that produces a flat list. It contains in this case title, url and timestamp over and over. So, I can use a stride. No, strides are a slice thing. Oh, I could slice the list, or use a filter directly in the list comprehension. Oh, but if a filter, I'd need an index value for modulus. Easier to slap a striding slicer on at the end... [x.value for x in cell_range][1::3] And this is why Python is so friggin' awesome! Just look at that. It's hard to imagine things can be this simple, cool, readable and all around excellent. Okay, I have the list of URLs I need to check against -- basically making sure I don't double-record anything that already has appeared in the feed, which is what this list of last-100 URLs from feed will represent. Okay, now for the next part: pulling all the Titles, URLs and publishing dates from an atom feed. Okay, I already wrote the logic to grab the feed and use BeautifulSoup to yank out the headlines. That gives me a list of the headlines. I need to keep everything in sync. I'm currently using soup.find_all("title") for example and that takes things out of their original correlation -- not good. Look to see if BSoup can keep multiple elements tied together on a select. If not? Hmmm. Okay, the answer with atom feeds is to select the element called entries, and then pick the items from each entry. Thu Sep 29 16:42:56 EDT 2016 I friggin' did it. So much transpired (in the code) since the last entry! Going to commit it. Hmm. I'll give you a before and after... see? This is how I work day-to-day, as much of it as possible like this. This is being in the zone, or more recently, "the flow", and it's really awesome for getting work done. And I guess I'll cut this entry here. My next step is a big shift to SERP stuff. Very different type of work, but I urgently have to get it done too. -------------------------------------------------------------------------------- ## Thu Sep 29 12:06:29 EDT 2016 ### Journalcasting Experiment Hello World. This is an experiment. I'm going to endeavor to talk to "you", my audience, right at work while I type and think, to create a novel one-of-a-kind type of YouTube experience, where you see someone working in vim "all day". Reality is, I'll be doing a lot of pausing of the video recording while I stop and think, but this is another fine example of Cialdini's commitment and consistency principles, just taken to a next logical level -- killing multiple birds with one stone, if you'll excuse the analogy. I have some things to urgently get done, and pretty small time-windows to work with. I am going to try to instill a touch of the feeling that I got from reading (listening to, really) The Phoenix Project from Audible.com, as the one I'm probably going to end up "keeping" on my $15/mo Audible audiobook club that I joined a few months back. Last one I kept was A Wrinkle In Time (for Adi), even though I was sorely tempted to keep Ready Player One. Okay, don't lose focus. Hmmm. System building. But small easily readable and maintainable systems. Take inspiration from bottle.py, even though even they ended up using Werkzeug as a dependency. That doesn't make it any less one-file-y. If it weren't werkzeug, it would be something from the standard library. At some point, you've got to import something. Occasionally, it's 3rd party. I'm getting the pausing the video stuff down. I thought I had to switch screens to check if the video was paused, but I don't. I have to become confident with the custom keyboard shortcut I made for ScreenFlow to pause and resume: Command, Shift, Option Z. I can use uppercase Z because that's what the shift turns it into. Don't forget or drop the "option" as my fingers sometimes want to do. Just keep in mind how very 4-finger special a ScreenFlow pause is. It's like Ctrl+Alt+Delete in how you must commit it to muscle memory. It's already 12:15 PM, and I have a meeting coming up at 1 that lasts to 2, and under these time-crunches, that's very disruptive to flow. I must establish and keep flow going, transporting myself metaphorically to a different place and just becoming one with the task and work at hand. I thought it through very well home "sick" yesterday. Let's call it a mental health day. I did all the car stuff I needed to do. Got less overall organized than I planned, but I'm set up semi-well for the Maker Faire this weekend with Adi, which is the important thing. But for now, 1, 2, 3... 1? Step one is refreshing my mental state from last I left off on Tuesday evening when I implemented the Append() behavior in Pipulate / SEO Notebook. Apple just killed my last attempt at a YouTube upload. Those videos serve as the glue between one work-session and the next, mostly on daily increments. Continuity of thought is a hugely under-addressed and under-thought-about issue. You can be totally inspired one day and lose it the next. The trick is to keep that inspiration-level primed up at a highly efficient and "electric" level. It makes you (me) feel good, and makes things move forward at a sufficient rate. I only have a half-hour until my next meeting, so strengthen "state" for when you come out of that meeting. How? Hmmm. Where you left on on Tuesday was adding Append(), which is awesome. Now, I can: - Populate: Create tables from scratch, given a Doc name, name name and list of lists data object (and optional list of column names to use as headers). - Pipulate: Replace empty cells on a per-row basis of a pre-populated table, given the name of the Doc and tab we're working with, and some options regarding static parameters drawn from another tab or on-hand data object. - Append: Add new rows to the bottom of a pre-existing table, given a Doc name, tab name and list of lists data object. And THAT is the recipe for a shitload of inspired next generation SEO deliverables. Is this good "television"? I doubt it. If you're watching this, I am both oddly surprised and flattered. Onward! Okay, I need to work on the popuate_from_atom function. I use "Restart and Run All" from Jupyter Notebook all the time. I wish that were a stand-alone button. Maybe it is, and I just need to understand it better. Running it now. Yup: AttributeError: module 'functions' has no attribute 'populate_from_atom' And so, I make it! Funny, the Python language "AttributeError". Designing the API: def populate_from_atom(sheet_name, tab_name, feed_url): # return a list of lists I have a really good generic system going here. Pipulate, Populate and Append all deal with list of lists being returned from queries. It's the easiest data manipulating model to wrap my head around for spreadsheets. While it's totally possible, and even in some cases desirable to use dict object to define spreadsheet data to keep column-data in forced alignment (using the key/value nature of dicts to put the column name in the key and cell value in the dict), that becomes overweight and burdensome after a short while. And so, this consistent API device makes Pipulate, Populate and Append very similar to each other. Wish I reused their code more, but the granularity and terseness of Python makes it rather not necessary, and in some ways actually undesirable (tying things excessively together that don't need to be). Okay, how to make a quick programmatic list of lists for experimentation and development purposes? Ranges, probably. At it's simplest, it's: [list(range(3))]*3 Just accidentally stopped the ScreenFlow recording, and one thing I want to avoid at all costs is video editing. This session actually got up to a half-hour, so I'm going to just experimentally upload it and see what sort of feedback I get on it so far. It show some basic day-to-day vim skills, and the sorts of things I'll continue to do in the terminal and with vim, even given my new recent "re-platforming" onto Jupyter Notebook. Okay, now return that test list from populate_from_atom(). I should unconditionally always get 3 new rows, populated with 0, 1 and 2. Which I do... woot! Okay, the next phase is to make the list of lists that gets returned from populate_from_atom() not dumb. It should actually LOOK AT the pre-existing list, and make sure it doesn't insert anything new that's already been recently recording. Pulling off that sort of automated system housekeeping using Google Sheets as the database layer is actually a pretty big deal. Okay, this'll be my first function that is aware of the table it's going to be returning data for before it returns the data. -------------------------------------------------------------------------------- ## Thu Sep 29 11:45:39 EDT 2016 ### What I'm Trying To Do Here Okay, had a nice long talk with the boss about what I'm building, reasons and expectations, and delivery time-lines. I'm in nothing different than anyone else ever is dealing with in this industry. It's only a tiny bit different because it's me, and my particular circumstances. But the generalities abound, and the opportunities to take advantage of the wisdom on generic work-flow improvements and improvements to bottom-line profitability imparted by both the Goldratt The Goal Theory of Constraints book, and the Behr, Kim and Spafford book The Phoenix Project. It's all about the 3 ways: - Deep familiarity with the systems - Short deployment cycles with good feedback loops for continuous improvements - Habits and the general samurai ninja muscle memory mastery thing Those are my philosophies, too. I am taking them up now the way I took up Drucker and Demmings after hearing Marc Rifkin talk about his TOC-based college major in the late 80s. It all comes around again, just tweaking whose filter and lens we're looking at the world through. Before, it was a TQM-style lens, and now, it's an agile lens -- minus the hype, and applied to what single free-agent (even if you're pulling a pay-check) individuals can do. Keep a powerful and always-applicable toolbox. -------------------------------------------------------------------------------- ## Thu Sep 29 09:43:02 EDT 2016 ### Jupyter Notebook is a Game Changer for SEO It's going to be a lot of conflict resolution in my journal for awhile, because I'm updating from changes here that weren't committed and pushed for a couple of days knowing that when I go back home, I'll have the same issue with updates sitting there that aren't committed or saved, either. Jupyter Notebook is a game-changer for the field of SEO. It's a perfect storm thing, and I have to push hard to make sure that I'm positioned at the forefront of that revolution. Get to work NOW on that. Emphasis on the talking head videos. Do them even more comfortably and frequently. Don't allow it to be a big thing. Just do it as a natural part of what you're doing for your job anyway. All the things I have to do for my employer are the "big versions" of all the things I ever imagined with Pipulate, and now with all the power of today's typical "local host" environments (the hardware of your own laptops, etc. thanks to Jupyter Notebook), my SEO tools can be a bit less inhibited on the capabilities-front (keeping it modest for Levinux) and the ambitious front (lots of dependencies and big-ish data storage requirements). At some point today, I have to change the scheduled task on my local virtual machine to be the new data feed script. That's like your most important to-do item for today, so don't forget it! Think through your other responsibilities as well today before you dive too deep -- but the thing you WANT to dive deep on actually is the Citation Monitor, which is the same thing you HAVE to finish up pronto, per schedules communicated to your boss. -------------------------------------------------------------------------------- ## Wed Sep 28 19:57:22 EDT 2016 ### Re-Invention Ha! There! I almost never add headlines to my entries until AFTER I've written my say, so that I may tease out of the flow what the thematic focus really is. I'm not a believer in putting the headline ahead of the writing. It's to predisposing and too writer's block imposing. Rather, instead, write what you feel. But what I feel is that in order to generate another highly impressionable time of rosy reminiscence for myself, I need to bootstrap myself (is that a horrible word), and culminate everything it is I've been leading up to all these years, early enough to make a difference in Adi's life, while she's still so impressionable, and it still matters so much. Make good, and do it for Adi (if not yourself). I'm also all about doing it for myself, because hey, through one lens, many of the things I'm doing and the way I'm behaving is very selfish. Through another lens, I'm still catching my breath after barely my head popped up out of the water -- but certainly not yet, financially. Now that I can breath, take those steps to insure that now you can file paperwork and do taxes and such. But the reinvention -- the face of the reinvention that I put on to the public -- THAT, I have to think through. Gee, I hope that double hyphen renders as proper markdown by strapdown.js. I tend to err conservatively, and re-flow my text so I don't have that situation. However, as an experiment, I will not this time. For you see, I am not trying to get things perfect. I am just trying to get them out there and under observation so that I can watch it (and watch you -- Google HotJar) and start to thinking about slicking and dicing and mining and transforming this shit into some shit. That's a big part of my re-invention. If I could do what I did with XSLT circa 2006, knit together with VBScript, imagine what I could do today with EVERYTHING knit together by Python, and transformation tech like XSLT generally not even needed for most situations in this era of sufficient caches to allow programmatic page-builds, and them have no functional disadvantage performance, SEO-wise or otherwise to more static technologies, like just serving HTML kept as text files on the webserver's local hard drive. There's even caching technology in that scenario, at the application-level. It's always caching. It's just a matter of where. And so, it doesn't always have to be Web tech. In fact, lots of the most interesting stuff is not. O'Reilly's good at nicknames, and they did (I believe) both IoT and Maker (versus distributed embedded computing and hacker) and both those things they nicknamed play a huge role in who I am and my developing identity. Many of the most interesting things in programming and tech are going to be embedding code into and controlling devices all around us in our lives BESIDES computers and phones. Wed Sep 28 20:42:33 EDT 2016 Okay, time to talk with Adi. Let's see if we'll do video call or voice. Wed Sep 28 22:10:12 EDT 2016 The call's over. Wow, we talk for a long time and could just keep going. -------------------------------------------------------------------------------- ## Wed Sep 28 17:07:38 EDT 2016 ### Mike Levin: Python-programming Daddy and SEO in New York Ugh! It's already 5:00 PM. Why haven't I been journaling like this to myself for more of the day to keep me on track? The truth is, little gets accomplished in a day, but a day can move you incrementally and inexorably forward towards your goals. Hmmmm. Plus, the stuff I want to do/finish for work. Everything must become the same. Organization in all things and on all fronts is pretty much the same. Hmmm? How so? Godel Escher Bach springs to mind. There's something about just exercising what it is to be intelligent matter that is in itself all the reward that's required for being an instance of such matter. Compose, store, retrieve the best you can. Apply. Send. Do little dances and tributes to make everyone happy. Perchance have offspring. The privilege of just our mere state as self-aware lumps of matter should be all the reward we should ever expect for our time in this world. This is it. It's both heaven and hell. That's the punchline of it all. All that time spent trying to get into one and avoid the other is in itself a system-stabilizing force being exerted over the here-and-now, so that the here-and-now gradually starts to look more like heaven and not like hell. Those who are trying to send us to hell probably do not see that. They likely see fault in others. Blame. Scape-goating and self-righteousness. Damn, I hope the country does not succumb to demagoguery. Yet somehow, I think it is. The cynic in me says the world is much more ready for a demagogue than it is a woman president. Damn, I hope I can put "down with the patriarchy" in contrast to a woman holding the highest office of the land. Wow, for that to happen just as my only child and daughter is turning 6 and at her most impressionable would be one of the greatest gifts possible. Next time someone shouts "Down with the patriarchy", I'll be able to say "Okay, I'll bring that up with President Clinton next time I talk to her". Ha, that would be fun. Oh yeah, again, since we're given that brief blip of heaven-on-earth reward for whatever cosmic lottery we won to be cast into the lot of the most minority and rarified matter in all the known universe -- life-on-Earth. We are the tiny dots on the surface of the foam that's on the shore of the dark matter rippling ocean surface over the dark energy depths. We can't even feel the ripping and churning of the foam, so tiny and brief and boundary-ridden are we. So, what then? Perchance create a self-perpetuating memetic contribution to society. Want to make a difference? Make some waves. Make those waves ripple forward through time with somehow enough internal self-renewing energy to not continuously lose cohesion over time. Keep the wave energetic and keep the wave in-sync. God, I need someone to just take care of things on the domestic front. I need first and foremost, an accountant. My energies are spent where I love spending my energies, making those ripples -- or at least, trying to. I've got some funny pieces, like Levinux, Pipulate and SEO Notebook. Sheesh! I've got to resolve that "Is it SEO Notebook, or is it Pipulate" thing. And at the end of the day, it's just using Python in Jupyter Notebook, but also doing a bunch of stuff in vim, because ultimately things get scheduled, and so those become .py files and not .ipynb files. I am really enjoying how imported modules can actually be other iPython Notebook files. But when things stabilize, you want to be able to have "normal" Github repos in which the Python code will just run with CPython (python.exe) in almost any system without all of Anaconda and Jupyter Notebook installed. While those are the environments I most like to work in (for now), I do not want to make all of Anaconda and Jupyter Notebook a dependency. Thing can, should and WILL be broken out as more formal mainstream libraries that you'll someday be able to pip install. That should really be my goal here. The timing is right. It's been awhile since the last, say, Ruby on Rails. The thing I'm building can't look like its too ambitious. It should be just a tiny little bit of code that enables a whole philosophy and workflow around information publishing, monitoring, and performance-measure evaluations. That's what I'm doing now. And that's how I'm trying to make my brief little blip of existence meaningful. I'm designing my branded waveform that I'm going to try to start ripping forward through time. I need to make something that makes an impression. I know I have it in me. I know I'm capable of that. I can do an extraction of all my thoughts and work and methodologies of my many years in tech and regurgitate it out into a thing of beauty. Oh! How can I forget to mention meeting Vishal Berry at the Conductor Dinner at Uncle Jack's Steakhouse last night. It was very funny how he kept asking why Stephan Bajaio hadn't introduced me to him yet. Very flattering. I think a lot of people are really interested in SEO, and where it's going. I should really lay out my shtick and my thing. Tell the story. Tell my thing. Make it a series. A Web video series -- probably YouTube. Yeah, of course YouTube. Hello Ladies and Gentlemen, and I hope a few impressionable children. I am Mike Levin, and I want to teach you how to seize super-powers for yourself in the impending world of robot-awesomeness. What we are, are the immediate parents of the generation who will either harness or be harvested by robots. We shape the generation that shapes the robots, and we had better do a good job. Yeah, that's my thing. But practical. Teaching everybody Python, because we all need to be able to communicate with machines through an API that's closer to the metal than voice recognition. Because once you have super-competent voice recognition, you're already in deep shit if you didn't prepare. There are things to be discussed and pursued, like a really solid education that includes languages and coding of all types. Human spoken language really requires something of near human intelligence to fully process and make meaning of. There's a human experience component to decoding the encoded thoughts of other thinking humans. Language is the API for reading books and decoding the patterns encoded into the sound waves all around us when someone speaks. So, I need to be taking better care of myself. So, we need to identify the patterns that keep us most happy and stable. Stability is happiness and happiness is stability... except when it's not because excessive stability, I believe, becomes uniquely vulnerable to disruption. You always want a little something going on that threatens to upset the stability -- a challenge that can only be met by collectively rising to. WWII was in its own way that. The whole purpose in the end could be seen as putting us to the Atomic Bomb test. Will developing the ability to annihilate ourselves necessarily mean that we will annihilate ourselves? Some say, human nature, of course it does. If not by the superpowers today, then by the terrorists who get their hands on them tomorrow. Those always-miserable folks are just the nattering nabobs of negativism who will always have a role to play in an otherwise not-miserable existence. We can't solve every problem on the planet by waving the magic altruism / technology / post-scarcity magic wand at everything, because then you just have other problems, like the afore-mentioned homogenized-in disruption vulnerabilities that you're not aware you have until devastation hits. No, it's better that moral relativism keep things in flux, with some unfortunate but necessary misery. I'm sure all these notions are old as notions themselves. The danger in that thinking is that it becomes an excuse in itself to justify any behavior. Why not be one of the assholes, if there's got to be assholes? Well, because it's better to be on the majority side, so long as the majority side themselves are not just homogenized assholes too, which is always a danger. And so, diversity. Diversity in all things. It's time that the United States had a woman president. God, I hope Hillary wins. Is that it? Am I cashing one in? Is that my greatest hope? Gender equality? It would not be a bad one. Okay, get back to something more productive than spouting out onto the Internet in a place where nobody's going to ever read... haha! And there it goes full circle. This information is voluminous on the Internet, but also meaningless. But I LOVE my way of working here in vim, git and Github on what amounts to a single text-file, reverse chronological just like a blog, and with deleted bits that still exist in the private Github repo, which I will eventually bring back here, just as soon as I'm confident I can keep things on an even keel as I navigate my way through this next phase of life. <pre> What do I do? What is SEO? Well me personally, I Navigate and Narrate. I Narrate while I Navigate. In doing so, I am also trying to lead. I am trying to lead people places to do things. Mostly, I do this on behalf of employers. But if you're not doing it for yourself, you don't "get it". There's something that divides SEO-hacks from deep-feelers. Maybe it's just the experience of having skin in the game. I am of the opinion that to be truly best, you need to do SEO for yourself. And to that end, you have something else going on. You need some thing that differentiates you, besides being yet another SEO. Where are your passions? How does SEO help them. In need not be purely profit-incentive. It can just be a passion on the side. For me, it's Linux, Python vim and git... ...and now, Jupyter Notebook too. Oh, mix-in being a Daddy in NYC and tech geek. And so, one must become known for ONE thing. This is my essential problem. I want my Mike Levin SEO identity to do it all. But I have to put a finer edge on things. A person may be striving advancement in many domains In that pursuit, the temptation is to talk about all of them But that doesn't compute. People need simple nicknames and labels. Association-making needs to be quick and strong. Take 2 unlikely things and stick them together. Damn it, okay, my thing is making it all work: Mike Levin, Daddy, SEO, New Yorker, Occasional programming inventor. Big advocate of Python. Mike Levin, Python SEO Yeah, that's it. Connect the dots that were there just waiting to be connected. Be the first in this synthesis of Jupyter Notebook with SEO. That alone is going to be big. Lead that charge. You see it. It's your thing. That's the razor-edge on your identity. Let go of Linux, vim and git. Focus on SEO, Python and Jupyter Notebook. Have your mad vim skillz always lurking there in the background. Have Adi (or just my daughter) as someone you refer to a lot. But it's not a "Daddy Blog" Just being a Daddy isn't my thing. Being a Daddy as we CHARGE our way into the dawn of robots, is. And SEO is a very compatible sub-plot. Mike Levin: Python-programming Daddy and SEO in New York SEOs must be a little bit branding strategists. SEOs must see patterns in online crowd wayfaring habits. SEOs must be the creative artists in the era of data science. We must be able to read the future in the tea leaves of data. But it must be less intuition and fortune telling, and more intelligence. That means good traditional matrix analysis, plus machine learning Data comes in two predominant arrangements: grids and trees. Both are important, so we must know both pandas and scikitlearn. Oh yeah, TensorFlow and other things that transcends fad to become fabric. These technologies are to be used to tell the future the best we might. The purpose is to sort of divine the direction the herd is going. Thought the purpose does vary, depending on your personal goals. Pure publishers that resell access to their readers may want large numbers. However, providers of limited, expensive products may want super-targeting. The more you target, the smaller your numbers. Unless of course, your target is the always-present big wins. The public's group conciousness is always clustered up around a few topics. These are the ones a publisher typically wants. And I currently work for a publisher, so this is what I want now too. It's going to be fun to be that sort of SEO. I get to really practice my craft. I get to carry out on all the project I've only ever dreamed of. I'm going to be able to do that on office time. A certain amount of it will be able to be shared back out to you. You on Github, and following me on YouTube. You, connected to me by Twitter and LinkedIn. You, who I will occasionally connect with over other channels I experiment These are the ones a publisher typically wants. And I currently work for a publisher, so this is what I want now too. It's going to be fun to be that sort of SEO. I get to really practice my craft. I get to carry out on all the project I've only ever dreamed of. I'm going to be able to do that on office time. A certain amount of it will be able to be shared back out to you. You on Github, and following me on YouTube. You, connected to me by Twitter and LinkedIn. You, who I'll occasionally connect with over other channels I check out Like, uhhhh, Snapchat? Oh please, no. But those Spectacles are cool. But I have prescription glasses. And no, I'm not going to change to contacts for that. Oh, but it's getting so close to my world. I see it, I feel it, I'm jumping the gun a bit. But I get it. Life-casting IS our product, when our endeavors are worthy and cool Yeah, that's it. I have to be cool to follow. Yeah, be one of those. Partake in Google's offerings, but don't be blind. Keep an open mind to game-changers coming from outside Google. Hedge your bets. Ride the Google wave while it lasts. It's an awfully good wave. Oh yeah, waves. Movement of the herd. Predicting herd movement. And then positioning ourselves right where the herd charges through. Having a piece of spot-on relevant, interesting and engaging content there. Content that was already baking, whose URL might even be repurposed. And so, it's always a sort of trolling. Sigh, SEOs are still trolls. And yes, we must continue to exercise invisible hand influence. We manipulate things just-so to serve our, and our employer's purposes. How can it not be so? Somebody's got to do it. Let's at least make it a little more noble by sprinkling Python education. Yeah, that's the ticket. Python and good-daddy stuff is my redemption. My redemption for making my programming-challenges for Python SEO ones. Yup. We must divide our purpose between inevitabilities and desires. We apply force ages before anyone else realizes to. We invest in content yesterday for dominance tomorrow. We make a stink about consistent, long-lived URLs. We are in the real estate business. We are building valuable content at valuable addresses. We keep our absent-minded professor tenants at those addresses well. We make sure links to these locations never break. We make sure the business that's for-grabs here is always grabbed. We ensure everyone in the organization knows the value of these addresses. We make sure such assets are not burned, squandered or piddled away. We raise sufficient warnings at site migrations. We devise future-proofing publishing strategies. Some people find it annoying. I find their annoyance at my trade not bothering me. (only bothering me enough to mention) And indeed, I've tried to get out of it a few times. But whenever I think I'm out they pull me back in. It's what I'm known for, and what I can earn the best living at. And so, in order to make it sufferable, I have to make it love-worthy. And to make it love-worthy, I have to share my enthusiasm about Python. And I need to make some measurable goal for myself online. And that goal should quite definitely be tied to monetary gain. And being in a position to pursue that must be what I try to visualize. I'm close, and zeroing in. Surely, I'm not taking an optimized route (ironic). But I'm navigating. And narrating. And that gets me back to my original point. I'm just generating data now that is my stream of conciousness. Because that's our times. We express ourselves online Into the digital landscape Into the digital void You listen to what you want And you filter everything else out Self-selecting and self With crawlers as the only visitors At least, for a while. Whether, when and how large your second wave of visitors will be Pshwew! Yeah. I'm going to have to mine this shit. There's something in here. See the relationships. Connect the dots. Don't be a dumbass. </pre> -------------------------------------------------------------------------------- ## Wed Sep 28 13:47:46 EDT 2016 ### Get Digesting Okay, don't squander this day that you took off to get organized. Time is already running out. The best thing to do is to get yourself thinking, and then try to instil the fight-or-flight adrenaline rush response to kick my ass into gear. I already got my car's State Inspection, so that's good. I also did the oil change the car's electronic system has been prodding me to do for awhile. And I'll be at the Maker Faire this weekend with Adi, and I'm planning to drive there, so it's good to be legal. But I'm almost entirely without money now, which feels pretty shitty. Use the rest of the day to know what's what. Even if you don't actually do any of the things to solve it directly, knock out all those dependencies like knowing what's what and what's where, which is something I lost track of long ago. If I'm true to my self-image, I can drag my ass out of this situation. I just need the motivation for mobilization, and that usually comes from instilling the fear of God into myself with the image of the penalties of not doing a thing. I just noticed the last entry below is Monday, September 26. The Sept 27 entries must not have gotten pushed out from work yesterday. No problem. The Git merges don't phase me anymore. Avoid distraction games that you play on yourself to run out the clock. I have a fear of facing up to my demons. I guess everyone does, but I have to get over it. I'm no worse off than tons of other people, and quite a bit better by actually being gainfully employed. I can't do too many days like today either where I call in sick for the purpose of getting my personal life on-track, but if I don't, I'm fucked. I guess that's my motivation right there. Now go organize your paperwork shit. Think of it as digesting. Throw out as much as possible, and what's left over is what's important and "part of your body". -------------------------------------------------------------------------------- ## Tue Sep 27 10:30:25 EDT 2016 ### Append Implemented I was at YouTube Space New York last night to watch the 1st of the Presidential Debates. Wow, what a train wreck, and how little it's going to matter to those suckered into demagoguery through fear. It's the perfect strategy for a megalomaniac in the era of hypermedia to employ. Hillary has really got to turn out the vote. Okay, double-checked that I'm registered at my current address and where the voting location is. I'm all set. Now, what about today? The Citation Monitor is the most important thing, and I should put stuff in place to make sure it's firing off perfectly reliably. I got the backfill list just now, so the onus is on me to archive all headlines moving forward. I really need to keep it in a database, like forever. Shit. Okay, think! Google Sheets is still the best approach, believe it or not. It's way more reliable and likely to be there when I need it than most other approaches, but I have to make sure to not make the spreadsheet to heavyweight, or slower as it gets used more. It's going to all be about partitioning. Google Sheets has massive amounts of data in it -- just not too much in any one single spreadsheet or tab. That shouldn't be a problem. Okay, switch from the print() function to logging. Hmmm, no because of the streaming output I'm doing, it's better to stick with print. However... hmmm. It appears modules regularly write to logging. It's way too noisy for regular output. Leave logging in there so you can experiment with it over time. I'm as close to "in-the-zone" trance as ever I was here at ZD. There is always that edge of guilt, but that's fucking shit from my childhood that I have to leave behind. I am on such the right track here, so long as I don't chase rabbits too far down holes. The danger here is always rabbit holes, losing track of how precious time really is, and how much better everything is if you deliver on expectations sooner rather than later. Very few things are over your head, so long as you tackle them correctly. This phase of SEO Notebook / Pipulate development is all about tackling things correctly. Okay, I've got the logger stuff in there. Now, for the next round of citation work. It's going to be a function to generically connect to RSS feeds, grab things out according to patterns, and update an existing table. This is going to be a highly useful recurring capability. It really is right to use Google Sheets as a primary database for an app, particularly in the world of marketing. It is however a trick to not let the collected data get unwieldy in a Google Sheet. This next phase is going to address... hmmm. Okay, list all the rest of the things that you have to address: - Monitoring an RSS feed, and grabbing URL, Headline and Publish Date - Checking which of these is not yet in the storage table - Appending those not yet in table to bottom of list And THIS is just the housekeeping component of this project. But without meticulous housekeeping, everything else becomes irrelevant. I already learned the hard way what happens if you don't keep on top of this. Okay, okay. Go get some food. Stop starving yourself, but go light because tonight is the Conductor dinner. Tue Sep 27 14:23:17 EDT 2016 Okay, I have to make the Append() behavioral mode pronto. But I have to do it systematically, too. I hate the feeling of just sitting and staring at the screen and not really know what to do next. I have to go in steps. Make plenty of print statements to force my thought process through the steps. I can always tear them out later. Tue Sep 27 16:43:47 EDT 2016 Okay, I added the Append behavior and it works just like Pipulate and Populate, API-wise. It's very strong, and moves responsibility for jobs into the correct place in the system. Now, I need to write a function that checks the feed and pulls out everything it needs. It should also have some logic to make sure it generally works WITHOUT housekeeping, but housekeeping will be required. It's actually quite important now to keep tabs on the feed and not let things scroll past that I'll have to ask developers for the archive again. I have up until today, and my system HAS TO WORK, but I have 15 minutes until my next meeting, and then I'm going to the Conductor dinner tonight at Jake's Steakhouse that starts at 6:30, and I don't want to be too late. I may actually end up doing some of this at home tonight. I'm getting to the point where I really LOVE my work again. Thank God. Next? What can I do in these last 15 minutes? I can set up another function just like populate_from_gsc but for a feed. I can also keep the arguments hidden so you can't really see the URL I'm hitting. I can just throw the URL value into private.py. Nice. Do that first! -------------------------------------------------------------------------------- ## Mon Sep 26 18:13:50 EDT 2016 ### Failed on 1st Pass Integrating MOZ Features NEXT! Okay, It's actually after 6:00 PM. The kitties will be lonely tonight, but I'm going to bear down hard on the next Pipulate feature. I made great progress today filling-in a very missing part that will pay back enormously: chunking the Populate() behavior. I'm regularly populating Google Sheet tabs with tens of thousands of rows, 50-row batched updates at a time through GSpread. Okay, this is going really well. Now, let's get those Moz features done that you thought you were going to do first today. Ugh, too many issues with just making the Moz functions active. I made a good attempt, but my work for today was good. Now, get to YouTube Space NYC. -------------------------------------------------------------------------------- ## Mon Sep 26 11:44:37 EDT 2016 ### Chunking Populate() Okay, this is your time to really do stuff. This is the sort of time you dream of getting. This is the sort of time with which you can advance yourself permanently and irrevocably (baring death) forward. Carpe diem, even if it is just geek stuff. 1, 2, 3... 1? Document the stuff you want to do to move your system AWESOMELY forward: - Logging (built-in library) - Generic SERP-monitoring with thumbnail-archiving and proxy-ing ability - Sched (built-in library) - Most popular MOZ functions https://moz.com/help/guides/moz-api/mozscape - retry (retry vs retrying vs retry.it vs backoff / make selection) - Pytest 3 https://pypi.python.org/pypi/pytest Okay okay. Check one thing off your list. Start with Moz. You already have a version of it written, and it should be a breeze. Of course, it's under Python 2.6 and handled byte data differently, but that's the obstacle. 1, 2, 3... 1? Re-wire you oauth login to an actual ZD property, so you're also doing the other stuff you need to be doing. I'm facing a sort of puzzle, balancing: - This all being in a public Github Repo - Using it on sites that I don't want to name Okay, I made a private.py whose contents should be obvious to anyone capable of using this product. I may include a example template private.py maybe as private_rename_me.py. But I'm already running into the problem with lack-of-chunking in the Populate capability. Ugh! I'm going to have to revisit that sooner rather than later. Apparently, my function was able to pull the data because it knew how many rows and columns to create, but it couldn't actually insert them, because the hunk was too big. It tells me that 'An existing connection was forcibly closed by remote host', None, 10054, None. Ha ha, interesting! Not very informative, but I pretty much anticipated this. Okay, the right way to do this is to turn the chunkulating into a shared function by both Pipulate and Populate... uh, no. Not really. What they do is actually very different -- that's why they're 2 separate things. If you can find efficiencies, great. But tackle Populate as its own separate process, lifting what you can from Pipulate. Okay, deep breath... go! We don't need chunks_to_go because it will always be the total number of chunk_ranges, because it is a table initialization process. Mon Sep 26 15:15:11 EDT 2016 I'm at a mental-block impasse. I know it's a problem I solved before having to do with the chunking challenge. The absolute index of a cell is always the number of row-number multiplied by the number of columns in every row, minus the inverse of the column number you're on... ugh! Okay... Mon Sep 26 16:34:49 EDT 2016 Okay, the SEOInit workflow looks awesome. I've tested populating from Search Console with date-ranges of a day, a week and now the full 90-day period in Search Console. Almost all of today went into chunking the Populate method. It's interesting deciding whether to talk in object oriented terms when I'm using named tuples. It was harder than I thought to blank sheets that already have thousands of rows -- especially making it actually look good to the user (not leaving row 2 intact). I made it so that it will work well with a frozen first row, which means row 2 needs to remain populated on a resize. -------------------------------------------------------------------------------- ## Mon Sep 26 09:42:17 EDT 2016 ### Monday Morning, Improve Performance No journal entries from home over the weekend... hmmm, maybe there should have been. My big objective over the weekend was to spend as little money as possible and to come back to work with my energy reserves recharged. I can't go spending all my energy over the weekend, and come back to work fatigued right when I need peek performance. And once again, I pay for that out of Adi's coffers. Shit. Anyway, all I need is a fucking big exertion for a few weeks in October, then all will be good. The winding-down of this shitty disorganized phase of my life will be fully underway, and the winding-up of the concentrated awesome phase that prepares Adi to be almost super-powered in the days ahead begins. That's pretty much my goal. Let her choose her own path, but be the tool-provider and skills-trainer for when she latches onto something that's of interest to her. Yeah, that's a good plan and is what will let me keep feeling good about myself through days like today on the personal front. On the work front, well, that's another matter entirely. Oh shit, It's Monday and I owe the weekly update. Make this weekly update slightly different from ones that came before. Put a brief description of your new tech in it. Give it a very strong nickname that can be used around ZD. SEO Notebook may be the way to go for that, since that's what's used in the public repo, and Pipulate stinks too much of personal product. SEO Notebook is just generic and catchy enough to serve the purpose. Okay, let's see... -------------------------------------------------------------------------------- ## Fri Sep 23 14:19:31 EDT 2016 ### Good Work! Okay, I got the Google Analytics portion of the seoinit workflow going. What's next? Look at that table below... oh, the Moz stuff! Huge application of the 80/20 rule, because everyone wants PageAuthority and DomainAuthority at quick first glance. Do the free ones first. -------------------------------------------------------------------------------- ## Fri Sep 23 09:36:36 EDT 2016 ### Now, I've got a plan Interesting doing a git pull in the morning at work, actually pretty much knowing that there will be no updates, because this machine is the last machine I did any journal entries on -- in vim/git. I'm now using other journaling devices again, now that I'm back on a one-handed iPhone again after 3 years on Samsung Galaxy Notes. I was on the Note 3 and 5. Guess I switched back to Apple just in time... hahaha! The moment in time I bought was halfway between the Note 7 coming out and the iPhone 7 coming out. Didn't go with the Note 7, so I avoided exploding phones, and didn't go with the iPhone 7, so avoided that extra potential lightning connector charger-cord anger (yes, I hate this proprietary port shit) that would have come with the removal of the earphone jack, no matter how ancient it may be. Certain old-school things have great advantages, I know. Some enduring things are that way for a reason... the power to endure means the 80/20 rule criteria is superbly balancing costs and benefits. So, the iPhone 6SE it will be for awhile. I have another wallet designed for it (really, the 5-models) coming in from a company that saw my YouTube video of the other one and asked me to review theirs. I sent them my address and they told me it's shipped. I'll have to do a follow-up on the Twelve South BookBook as well, now that I've been using it for awhile. Speaking of the power to endure, vim has finally had an integer-version increment, and vim 8 just came out earlier this month. I'm not really plugged into the vim community, I guess. I still never took up ANY plug-in's like the fuzzy finders as I had intended and planned. I guess I haven't really needed them, and my .vimrc minor customizations have always been enough. But I think I would like to jump on the vim 8 bandwagon sooner rather than later. Hmmmm. Learn a little more about it first, but take the plunge very soon to make a video. It's exactly the sort of thing I should be making videos about. My video this morning was about Pipulate under SEO Notebook. Hmmm, it's probably time to get the Pipulate logo displaying from within the Jupyter Notebook interface. I could probably link to the svg file from localhost:8888. Worth a try anyway, sometime very soon. Inching closer to an announce-able and release-able form. Make a check-list of the things that need to be supported. | Data Source | Pipulate | Populate | | ------------| --------- | ----------| | Search Console | Not yet | Done | | Google Analytics | Done | Not yet | | Site Crawl | Only Title | Only URL | | SERP Monitor | Not Yet | Not Yet | | SERP Thumbnails | Not Yet | Not Yet | | Moz Free | Not Yet | Not Yet | | Redshift SQL | Not Yet | Not Yet | And the projects I need to work on that USE these features: - Citation Monitor - Breakout Monitor - SERPchiver +thumbnails It seems pretty obvious that I need a generic and persistent key/data store that works a lot like the HTML cache that I built for URL fields, but with generic key/data stores, exactly per the power and flexibility that the dictionary-oriented shelve API provides. Coupled with shove for generic enterprise and other database backend swap-outs, and it's a brilliant scalable solution that can start out with NO database backend but for Python native pickle files. Woot! Now, I've got a plan. -------------------------------------------------------------------------------- ## Thu Sep 22 16:15:34 EDT 2016 ### seoinit Wow, I got seoinit working in Pipulate as a Populate workflow item. Shit, this is going to be big. I want to do different timeranges to get directional predictive stuff, but first its important to get some of the moz functions in here, as popular as they are. But I don't want to require downloading a whole client library just for that. Ugh! Thu Sep 22 17:41:09 EDT 2016 Okay, I have workflows really working well, with multiple populating events occurring, creating or blanking the tabs as necessary. Now, I have to add pipulating events to that same sample workflow, and maybe the free Moz Domain and Page Authority. -------------------------------------------------------------------------------- ## Thu Sep 22 10:52:00 EDT 2016 ### Streamlining Pipulate Okay, today again. I need to get my system more ready. I have a bunch of stuff I need to do on the current Citation project, but I also need to pay down some technical debt. By that, I need to make my system more generically capable. I need to put some functions in there that come up every day. I also want the killer instantiation workflow for the system that ties into Search Console. This is all sort of like lost queue time, waiting for inspiration and insight to hit me. That's shit. Shake the inspiration out of the tree. Do that through full engagement, and allowing yourself to slip into the zone. Be like you were in your days at Scala. Can you imagine the impact you would have here if you REALLY took it all personally. You know the most important work you need to do. DO IT! 1, 2, 3... 1? Okay, I need the simplest possible solution to API keys and stuff. I want a file that people can feel free to open and edit and get really familiar with. I want to avoid adding argument-passing complexity to Pipulate functions, which is exactly what happened when I tried to port over some of my Python 2.7 Pipulate code for Moz functions. And Moz functions are going to have to come soon, for things like PageAuthority and DomainAuthority. However, that comes AFTER a big initializing pull from Search Console. For the Search Console initialization, it's going to require the site's domain name and of course the OAuth permission to access that site's Search Console API. So... And so, I see the mistake I almost made by adding Process when Pipulate and Populate are going to be enough -- and enough to explain! Don't go building new complex interfaces. Just two! Keep it to two more flexible ones for as long as reasonably possible. Just populate the list of lists with the output of a function! Put a string command defining the command you want to execute, then eval it later on and thumb your nose at everyone who screams like a fragile little flower at the concept of eval. Thu Sep 22 12:19:40 EDT 2016 Making great non-progress progress. Paying down technical debt. Making my Jupyter Notebook Pipulate infectiously cool by being very clear and usable. Actually cutting down surface area. Taking out Process (or any off-the-rails customization) makes everything so much simpler. -------------------------------------------------------------------------------- ## Thu Sep 22 10:32:39 EDT 2016 ### The Goal: A Process of Ongoing Improvement by Eliyahu M. Goldratt and Jeff Cox Today, today, today. Get your dumbass head out of your ass and start being super-productive. Know your systems, shorten feedback loops and practice your way to zen-like mastery. Simple as that. The 3-ways of The Phoenix Project book. It's time I heard the seminal continuous deployment talk. It's time I even knew which talk they're referring to. And I should probably read about Goldratt's theory of constraints. This is the TQM stuff that Marc Rifkin was telling me about when we were both still in college IN THE 80s!!! Late 80s though it may have been. TOC. TQM. OOE. I remember him telling me about them all. Amazing I didn't take that stuff more to heart. I guess I sort of did by listening to all that Peter Drucker stuff on my commutes back and forth to Scala on the Schuylkill Expressway and Route 202 out to Exton (my old Commodore commute, almost). I was just in a different place... in many ways. I didn't know enough (or wasn't smart enough) to PUT myself in the right place. I'm sure I was doing as well as my dad or many people like him would have hoped for for their kids in those days. It was before the dot-boom-times. The Goal (non-30th-anniversary edition) is only $2.75 on Audible. Crazy to not make it my next listen EVEN while I'm trying to work. May just be the stimulation I need to take the shot-in-the-arm I got from The Phoenix Project to the next level. Just do it. Okay, and now it's downloaded on my phone. Shit, Amazon and Audible sure took to heart the lessons of these books, I think. Or more accurately, the wrote the fucking book. Ugh! It's only like a cliffnotes of The Goal. Shit, I'll have to wait until next month to apply a credit, unless I return another book. I'll keep/listen to this key takeaways thing. Maybe that's really all I need right now. I don't need another fictional account of a company on dire straights, if it's almost the same story as The Phoenix Project. Just absorb what's worth absorbing. Let it be background noise as you work today. -------------------------------------------------------------------------------- ## Thu Sep 22 09:55:49 EDT 2016 ### iPhone Still Only Task Switches In Some Important Cases Merging coming so much more naturally, along with understanding how chronological order is going to need to be fixed. Uploading to YouTube after I get into the office is interesting, because I just have to let the phone sit with auto-lock turned off for as long as it takes, and there's certainly no risking going over to start music playing in the background and coming back to the upload. It's not really that fragile, but I don't recall having this sort of precautions with task switching since I started printing jobs from WordPerfect on the Amgia, and even though it COULD multi-task, the rule of the day was hands-off, so you didn't have to restart the print job over and over after every crash. Yes, Apple sure learned its lessons from Commodore and the Amiga, but in avoiding the foibles of multitasking fragility, they created task-switching fragility, which from a usability standpoint is no better. Of course, in the big picture these decisions give Apple amazingly longer battery life than Android, with all those background tasks being discouraged and always having to go through tightly controlled and rule-regulated OS mechanisms. Uhg I have auto-lock set on 5-minutes, and it did end up locking the phone before the YouTube video was "processed" and it seems to be stuck at 95%, but I think that's a result of server-side issues and not the phone. I think the upload finished, and I'm just waiting for some queue or process to finish at YouTube, maybe thumbnail selection. They appear to be trying to cut down the time between when the video is available for playing and the thumbnail for the video is actually available. It has to be quick, because people want to share these things on social media as soon as they're playable, but if the thumbnail isn't available yet for the video, the share doesn't look nearly as good. I've also been leaving off descriptions on the videos lately, just in an attempt to lower publishing friction. It should almost be besides-the-fact in the morning, these videos. -------------------------------------------------------------------------------- ## Wed Sep 21 19:32:00 EDT 2016 ### Proprietary Thoughts Today's edits from work, I forgot to commit and push before I left. No big deal. Transitions and transformations. Finally listening to The Phoenix Project. Wow, wish I was more tech-mature (even more than I was) when I was at Scala Multimedia. You are who you are because of your journey... and your genes. Your genes and your journey are who you are. What else is there if everything but what you're born with is part of your journey? What we've pulled off as a species is remarkable. Sure, some may think that of not much use to the planet, but let's face it, we're better at taking control of the very world we live in better than any other species... well, uh... in history? As far as we know it. Doubt things came before us, but this is the sort of thing on my mind when I'm evading committing to a course for tonight. I'm actually exhausted feeling -- and I'm even considering coffee. I have to consider breaking this caffeine addiction, because I fear it's not working for my cause. If I want to be productive at night in any capacity, I need caffeine. Those 5-hour energy drinks do actually do it for me, but that's terrible. Developing muscle memory in a way that only comes from long-term commitment and practice. How may times is Microsoft Word going to change it's UI, or force you to upgrade, or try to make you use their cloud services? Constantly, is the answer. An in one way or another, always in service to corporate profits. Don't tie your daily journal to proprietary file formats accessible only from proprietary software running from proprietary cloud servers, displaying on terminals built from proprietary hardware using proprietary processors. As actual circuit board fabrication comes down to the level of home kits like today's 3D printers, the "products" we will be able to "print" (fabricate really by that time) will astound. It may not be down to the nanometer scale like tic toc billion dollar corporations can produce. But rudimentary quad-core (for some modest parallelism) I bet could be at home-fab tech levels to manufacture in another 25 to 50 years. Maybe I'm radically over-estimating, and it could be here in 10 years. Why not? What are the insurmountable obstacles? Isn't the more logical inflection point, the ability to print products that can print more products? Who needs intelligence or creativity when you've got grey goo? Anyhoo. Evaded the question of how to make tonight productive for long enough. Give yourself that adrenaline jolt somehow that you need to keep moving your life forward. Remember, just think of it as digesting, and not as work. Getting insight into the origins of the concept of devops was fascinating. It was from 2013. Wasn't devops already a thing back then? Wow, I'm forgetting when I started to become aware of particular buzzwords. Or am I? Hmmm. Anyhoo, it's interesting to think of things through the sort of lens that book strives to teach people to peer through. Business and traditionally separate, but in-service-to business Information Technology. It used to be just mere competency, and a sort of copy-and-paste creativity was enough. But not anymore. As things get competitive, the game changes, and those who grasp on too tightly to the old ways are just weighing down an increasingly decrepit system that can't let you be as nimble, responsive and generally high-quality as your competitors. A sort of Harley Davidson pull back from the brink to stellar success again story. Emphasis on infotec and even infosec and the developers and QA people (quality assurance). I can see how it was contributing lots and lots of fuel to the agile and continuous deployment nearly religious movement. And it fits into the whole Sun Tzu strategy of one domain is applicable to strategy everywhere sorts of books. There were some different types of work and some different types of ways of behaving that all intermixed, and if only you understood them. But you had to earn it. You had to identify for yourself the through observing the manufacturing factory floor the types of work and their respective requirements and bottlenecks, and the ways of being: appreciative of your systems, able to create tight feedback loops and iterative cycles, and able to form the good automatic habits and mastery surrounding these disciplines to support them. This is what was going on in normal old-fashioned IT-departments when the outsourcing movement, cloud services, and variable-cost tech infrastructure all started coming onto the scene, thanks to managed partition-able datacenters, agile software development movements, and the Internet and the maturing web browsers of the Web, and people beginning to shop from home and use Amazon (though Amazon was not specifically mentioned, it may as well have been the enemy without -- until it became their bursty little friend). Pshwew! Well anyway, I really enjoyed that book, and it was one I enjoyed in audio format, and I think it will be my new commit-it-to-memory on the commute book that Drucker's Managing in Times of Great Change was during my Scala days in the late 90s and early 2000s. I need new mantras. New business mantras, that is. My one from those days is "The mission of every company or organization is to get and keep customers", for without customers, how can you be a company? It is actually all about the customer, and their needs and desires and indeed right to be serviced by whoever can service them best. Plug in some crazy technology accelerators we're seeing today, and we're in for one wild ride. And now it's time to go be the best Daddy I can be. Sent the text to Rachel that I'll be ready to video-chat with Adi at any time. I have to list the projects her and I want to do together. We're going to the Maker Faire this Saturday. I have to make sure I actually have enough money to do anything interesting there. I'm going tight this month. Pshwew! I really have to start getting expenses under control. Go over everything. Consider a big changing of accounts to force things to clean up. So that auto-switching auto-pay systems doesn't kick in. Be sure to cancel things that need to be cancelled. Soon! Now? Ugh. -------------------------------------------------------------------------------- ## Wed Sep 21 09:28:42 EDT 2016 ### Recalibrating, Refactoring, Re-have-to-get-my-act-together Gonna chat with my boss this morning. Just go over issues. Time-lines, scheduling and process. The chat is a bit open ended, and really probably amounts to a directional adjustment. Purging the unknowns. Making all projects fairly similar and standardized. My plans versus what I "want" to do to meet schedules. Different approaches to work, me being the bottleneck. Making myself more available for help-desk like work, while being able to produce at levels that appears incredible, without actually killing me. Okay, meeting held. Very good. Break citation monitor down into phases. There are easy phases, and there are harder phases. Do the easy ones first. Do the work manually, initially. Don't get bogged down. Keep track of what you're doing. Always tackle things manually first. Shape your system (SEO Notebook / Pipulate) to accommodate and expedite and ultimately automate the sorts of things you're doing manually, but don't get bogged down with system-shaping issues WHILE you're doing it manually. Okay, updated my high-energy playlist. Even with playing the same songs over and over to stay in the zone uninterrupted, it still becomes unbearably monotonous. Gotta balance these two things. Added Pharrell's Despicable Me to the list, hahaha! I think I'll be able to relate to Adi even over my music choices. She DID express her favorite music to me as the Garnet and Jasper fight song Stronger Than You from Steven Universe and the Crystal Gems show... hahaha, awesome! She's really got my taste in music. And then of course there's the Beach Boys surfing songs... oh, add those to my high energy list for the day just to keep Adi on your mind. I have to remember there's 2 of them that I always get confused: - Surfin' Safari - Surfin U.S.A. Okay okay, get the query together to get a list of domains to use in a SQL like statement. Hmmm. Like. In. Contains. Pattern matching, for sure. PostgreSQL-specific? Gotta google. Make it simple. Hmmm... From the documentation, there are three separate approaches to pattern matching proviced by PostgreSQL: - Traditional SQL LIKE operator - The SIMILAR operator - "POSIX-style" regular expressions It seems to me if there's no reason to use SIMILAR or RegEx, then I should reduce complexity and probably execution-time too. So, like it is. Construct that query! -------------------------------------------------------------------------------- ## Tue Sep 20 19:18:20 EDT 2016 ### Not Alert Back at home, and the trick now is to keep my new habits going. I feel better after getting a bunch of stuff done, and even more urgency pushing me forward for the next day's round. A little bit of habitual behavior every day is necessary at this point. Tue Sep 20 21:49:56 EDT 2016 Okay, just finished up with Adi on the phone. Exhausted. Shit, no 5-hour energy booster tonight. And could hardly get anything substantive done over past 2 days. Connection? I don't know, but be on the... uh... alert, haha! Shit, I'm worried. Tomorrow morning. Definitely, tomorrow morning. Wake up early enough and rested enough to face the day effectively. -------------------------------------------------------------------------------- ## Tue Sep 20 13:15:02 EDT 2016 ### Another Less Than Productive Day Well, I ended up helping a co-worker with some SQL for some campaign monitoring and arbitrage he wanted/needed to do. It's precisely the sort of thing we need to be doing more of, and have come totally naturally to us, so I was glad to help him. Hoping to infect him with the SQL bug and get one more person who prefers to do stuff with a decent querying language over the Google Web user interface. But I lost momentum. I am going to use the waning momentum as an opportunity to install iOS 10, because why TF not? I'm doing it connected from a Mac through iTunes after a backup, so there's no harm. I think I'd like to see it, and at least be on the leading edge with my OS if not with the phone itself. Interesting the iPhone 5 appearance was chosen over the iPhone 4 (more durable) for the 6SE. Tue Sep 20 15:16:48 EDT 2016 Okay, it's already 3:15 PM, and another day is slipping away. Fuck! Okay, these dots are waiting there to be connected, and it's aligned with one of your greatest strengths and one of the most valuable things you have to offer to your employer and the world. Just connect those fucking dots. You'll feel a lot better tonight when you go home to face all that other shit. 1, 2, 3... 1? Again! Okay, I broke out the site monitor project into its own github repo, and reset the history of the "external" nested github repo that's designed to hold custom workflows and such. Tue Sep 20 17:00:39 EDT 2016 Ugh! I'm learning to just not fight the import conventions of Python. I successfully created an "external" directory inside a git repo directory which is in turn its own separate repo, and the outer repo can import from the inner repo, but the inner repo cannot import from the outer one, which is actually quite important. Why not just lump it all together? Git submodules are no better because still they are subdirectories. What I need is easy 2-way imports. File A needs to be able to import from file B and file B from A, which is completely fine with Python. Okay, I am going to tear out the concept of external, and put in the concept of a processes file. The processes file is where you keep custom processes. Why can't I drive this one thing home? Ugh! I'm missing something here. I think that the functions file should probably just be used for workflows as well for now, and I should strip out the external stuff (for now). Okay, done. Now, think! -------------------------------------------------------------------------------- ## Tue Sep 20 09:42:10 EDT 2016 ### New Pipulate Workflows Imagined Well, at least I can sort of document the ship going down, hahaha! Okay, no really I have to keep this ship from going down, and apply as much force as I can into making it seaworthy and shipshape. Every day is both bailing and patching. Patch and bail and bail and patch until you're buoyant. Don't know why it took me so long to read (listen to) The Phoenix Project, which I just got from Audible yesterday after chatting with one of their reps about the exchange program. I returned both Redshirts and Ready Player One, because I value getting onto the next book more than "owning" something in my digital library. There's a lot about being honest with yourself here. Yesterday was pretty much a giant snafu, with the data feeds from the reports looking incorrect, but it's really just the default sort orders that got messed up. They're still sorted descending alphabetically. I actually even pre-sort the feeds, so all he has to do is use the default feed order. Speaking of which, I'm going to need to get a list of all article headlines that went out over our feeds to do my latest project. Fire off that email right away. Okay, done. Even commented on the Jira case. Okay, good. Next! Oh, just want to capture the fact that I talked with Adi on my way into the office today -- that walk between Penn Station and Madison Ave (long walk), and the Apple headphones really make it work well due to their hands-free headset functionality. Wow, what a simple but important thing. I think I may try to standardize my daily headphone-time on Apple headphones in order to always have that hands-free benefit when talking with Adi. It will encourage me to call and talk with her on my walks more often. That's each direction, twice-a-day which is FOUR opportunities to talk with her every day JUST DURING MY COMMUTE! Wow, why didn't I think of that earlier? Mission critical stuff: utilize my commute time to talk with Adi. Nickname it: Walkie Talkies with Daddy. Are you ready for a "Walkie Talkie" with Daddy? Okay, good. Now focus. One more distraction. I have about $1000 to make it until October 1st, and I don't think I'm going to make it, and even when I do, the 1st of the month bills are going to wipe almost everything out. There's nowhere else its going to come from now at this point, except for my IRA, and that takes a few days for it to take place. I can't even call my guy to do that until I fully know what my situation is... which I'll make sure I know TONIGHT. Pshwew, wow even putting that down in a location like this is a pretty BFD for me. Now, follow through and make everything work. Be as "on your game" as you've ever been in your life. You can't even be a provider if it all gets pissed away in a SITCOM, and I don't even have 2 kids. Just an oppressive mortgage and single income. Shit, okay. Get ass in gear. Do some here to protect your (future?) power as a provider. 1, 2, 3... 1? Get your head into the game and find the love by a quick Pipulate refinement or two. Your playlist disappeared from Music on your iPhone... ugh! Which made me think about iOS 10, which I just decided to download but not install via iTunes from my Macbook Air... won't do the over-the-air (OTA) install given the fiascos I've been hearing about. If you're running desktop Sierra, you can have synced desktop content from your Mac... hmmm. Interesting. I'm still on El Capitan. Don't rush stuff. Go slow and be in the stable, less ouchie second wave. Reading about iOS 10... no way! Not today, anyway. Same for Sierra. I'm not a bleeding edge Apple fanboy anymore. I'm a pragmatic Apple user, because their proprietary shit is actually quite good, and I like quite good. But I'm not over-enthusiastic anymore, except maybe keeping things patched from a security standpoint. Again, focus! There are some dots right at the edge of being connected here. Workflows are collections of named tuples. The tuple names that I'm going with right now are: - Pipulate - Populate There is an infectious elegance here that leads right into the discussion of the difference between what it is to Pipulate and Populate. Populate is populating an entire table from scratch like an initialization process, or CREATE TABLE. Pipulate is doing a function per-row, just updating a pre-existing but not-yet-populated column. When it's just on the columns on already existing tables, it's pipulating. But am I to introduce the dreaded THIRD named tuple type to be supported by workflows? It's probably a good idea function-wise, but it's kinda scary from a semantic and elegance standpoint. Pipulate, Populate and... Process! Of course! A process will obviously be a named thing. The arguments for Process will be: - nickname - reference - lambda This implies a few things. These are always references to functions in one way or the other. A nickname is a string possessing the same name as a function in global space, so that you can just "nameit" as such. Reference is probably going to be just the actual name of an existing function so that it is a true reference to the function that can be called() at any time by just adding the parenthesis later. It would be something like this: reference = function_name reference() # Invokes function_name There is a question of arguments here. Should arguments be supported, and if so, how? Certainly on the nicknames you could use arguments, because inevitably some sort of eval is going to be used to turn it into code (as opposed to a text string). This is about having the flexibility that's really required in this type of work. Most common work can be achieved with a combination of Populate and Pipulate, but on occasion you will have to create a more complex Process, which can be a simple (cute) and common "nickname" like "seoinit"... yeahhhhh, I like that. It would look something like: Process(nickname="seoinit", reference=None, lambda=None) Named tuples means everything needs to be explicit, and it does appear very wasteful to have the extra reference=None and lambda=None on every line that doesn't really need it, except for the purpose of explicitly stating that those other two optional values are indeed set to None. But it really does give Process its own distinct personality from Pipulate and Populate, which are all about table names and such. Should a process also know about sheet and tab names? Am I leaving off important parameters? Okay, just focus on getting ONE special process implemented. -------------------------------------------------------------------------------- ## Mon Sep 19 22:59:18 EDT 2016 ### Got Some Stuff Done Yep, thinking about it as digesting helps a lot. Thinking about my apartment as part of my body, and what I'm doing the work of the... Mon Sep 19 23:49:14 EDT 2016 Yep, gotta wrap up here, but really made another decent step forward. I have a place to sit down and also a tiny bit of table-space now to work with. -------------------------------------------------------------------------------- ## Mon Sep 19 21:55:43 EDT 2016 ### Enthalpy Please Okay, the correct answer is, keep your momentum going, you dumb-ass. Use the rest of that 5-hour energy-drink you downed, because any sleep you could really possibly get now wouldn't be quality & restful, unlike what you will get when you let it burn off and have the satisfaction of actually and permanently moved your life a few more units-of-measure towards greater enthalpy. Enthalpy -- now, there's a good one to work into Adi's education. Entropy vs. Enthalpy. Learned that in elementary school, I do believe, and it forever stuck with me, and nobody knew what the heck I was talking about during later years, until I could prove once again that there was such a thing as enthalpy once the Internet got into common everyday use for storing and searching public data. BAM! Not everything is greater order, however. This paragraph as a case in point. It is just information noise that wastes everybody's time who's reading it. See! There, I'm avoiding doing what I know I need to be doing. Off to bring more order to my life. -------------------------------------------------------------------------------- ## Mon Sep 19 18:58:03 EDT 2016 ### Ass-saving Urgency Now Required It's never too late to get your act together, so long as you're not thrown into jail and kept from being able to generate new money with your earning power. It is also important to maintain your mental condition, and ALSO to not do excessive evasion of the issues -- something I've been doing for far too long. It's been at the point where I could not even get my act together -- a place to even just sit down and do paperwork. It's been ridiculous to the point of comical tragedy. Now, you have the time and no excuses, so jump to it! It's 7:00 PM, and I just downed an extra-strength 5-hour energy boost drink and have about an hour before Adi's going to expect hearing from me. It's September 19th and I have to keep time very much in-mind. Shit, okay, make today the last of your "path clearing". Definitely follow the 80/20-rule, and constantly keep yourself from diving into asinine "special projects" that will swallow-up all your time self-justifying itself by feeling worthwhile. Keep this going to record your thoughts, but it's mostly for the purpose of moving yourself forward and NOT to have yet another off-point time-sink. I have to get my ship tight and my ducks in a row. Focus all your effort tonight into making attacking paper-work with the attention it requires at this point, full with a in-use and active filing-cabinet, and prioritized sorting bins. I'm listening to The Phoenix Project on Audibles, which in itself can be a source of stress, but it also helps give me a sense of perspective. I'm going to pull through this okay, and all that's at risk really is money -- I hope. Okay, focus. Even acquiring the ability to answer those sorts of questions is my priority. Actually label things. I've got about an hour before Adi's going to be expecting to hear from me, so build some momentum in this less-than-an-hour window, and when you switch to video-calling with Adi, do it on your phone where you can carry it around. Find/use those original Apple microphone-headphones. If this is too bad over your head, consider pulling in your in-laws. They might be able to sympathize and help. "The never-ending hamster-wheel of pain." What do I have to do to get my ship in order? What do I need to do to get all my ducks in a row? I hauled out another couple of wagon-loads of crap out. It's just about time for my phone-call with Adi. If I call at 8:20 PM, I can go to 9:20 guilt-free. Wow, becoming guilty by talking to my own daughter because I'm in so deep. Fuck! I'm almost choking-up crying. Mon Sep 19 20:22:11 EDT 2016 Okay, just texted Rachel that I'm ready any time Adi is. They need a few minutes to get home, and Rachel asked to switch Tuesday night with Thursday night for getting together with Adi. I said no problem. I really have to start keeping a calendar now... actually, for real. Shit, I'm so disorganized, it's a crime. Anyhoo, the world and your environment is your checklist -- especially when you're at the level of disorganization that I'm actually at. But I am zeroing in on getting the place Roomba-able again. In addition to the fear of letting myself get into as much trouble as I have let myself get into is the exhilaration of knowing the culmination of much of my professional Kung Fu is finally coming together. And the lessons are: Lesson #1: Life is actually pretty short. Whatever it's true nature, and whether we're all just an infinitely nested virtual hologram or "The Real", the one thing that we know for sure is that our individual experiences are very real, indeed. It doesn't matter that they're abstracted side effects of impossible to define consciousness You can have it all. But "it all" isn't always what you think. If everything is ultimately subjective anyway, which I am convinced that the problem of induction proves to us that it is, then ultimately, one must decide for one's self what is and isn't worth having, and the definition of possession, and reconciling the fact that they're part of a generation of humans that are unlikely to preserve themselves in much meaningful way beyond death. And death maybe is the one absolute in all things, because we're never really aware in a human-conciousness way that we mostly think about it as adults of coming into existence. No, we went through a miniature fast-forwarded version of evolution itself, being momentarily a one-celled organism, momentarily invertebrate. Being momentarily fish and salamander and furry rodent-thing and scared superstitious hominid... stop. Yup, we're just the "latest state" in a game that may or may not be a video game. And we are all really quite lucky and privileged to be thinking-matter at all, given the odds of these particular elements that you're comprised of... Mon Sep 19 21:45:04 EDT 2016 Okay, just finished up with Adi. Ate dinner while on phone with her, the remains of the crab cake sandwich from the weekend, and another home made round-roll pizza. Now, if I just go to sleep, I can make effective use of the time. Or if I stay up I could make effective use of the time. But... but... lost motivation. Either pay for it tonight or pay tomorrow morning. If tomorrow morning, don't cheat. Go to sleep pronto. -------------------------------------------------------------------------------- ## Mon Sep 19 11:41:49 EDT 2016 ### Note To Self: Be Smarter! Okay, we all need something to be excited about, either on the work or the professional front. On the work-front, it's now definitely shaping up to be replacing the default capability-testing in my GoodSheet Python module with the basic data-pulls one would always want to do with their own website, via Search Console. It's an interesting way to introduce this system, but a worthy and currently highly relevant one. Mon Sep 19 17:25:00 EDT 2016 Lost almost all day chasing optical illusions, and discovering one unrelated bug. Note to self: be smarter! -------------------------------------------------------------------------------- ## Mon Sep 19 09:46:29 EDT 2016 ### Tradition Transplants Things like Facebook HAD to occur, lest we become hyper-effective, equipped by all the amazing new tools of technology, as we could be. I have 2 videos I very much want to push out ASAP, and then focus on my work today -- the citation-monitor. Think through other approaches. There are many, given my end-of-day Friday discussion with my boss about the objectives. Get the uploads occurring ASAP (or the first, then the second, thank-you iPhone), and start working on your weekly status report. I also have the brainstorm of making the stuff in GoodSheet specifically do a full weekly SERP-pull from Search Console as the way of "initializing" SEO Notebook (soon to become Pipulate). The last time I "deeply cared" was Scala, where I was slapped around as thanks for my effort, and then I became skeptical about anything ever being for my own good every again when someone else WANTS me to believe that. Aligning goals for sums-greater-than-parts is something akin to a Unicorn in my mind. What's worth deeply caring about again after Commodore and the Amiga, and then Scala Digital Signage (still around, but not "mine")? The answer is things purely of my own creation... but true freedom from dependencies is an illusion... so then based on the most logical and basic dependencies. We all stand on the shoulders of giants, so we must choose the best combination of giant shoulders so we don't fall off and land on our asses. Okay, the first video is uploaded and the second is in-progress. And I checked on the daily SEO reports (that are still running off of my laptop) to make sure they're still firing off well, and they are. Gotta switch to the new feed method soon. I thought I forked a repo, but not really. It was just a clone with a different folder name, and so I've been altering the main branch every time I committed and pushed, which is just as well, as we're moving forward... always moving forward! Speaking of which, I need to make my list of things to revisit about the report, such as the decimal place control. Maybe today should be a BIG video day, given how I'm catching up with my workload, almost, sorta kinda. Well, the big brainstorm this morning is that the sample scripts that GoodSheet does should be made to work against Search Console, to do a full per-keyword, per-URL, per-weekly averages data-dump for the baseline sort of directional trending that almost everyone should be doing. Every search result that you do have active is like a directional tracking beacon when combined with some simple linear regression. Hmmm. I think I may here be having my Moz-moment -- the thing that only I could implement with this sort of style and built-up methodology and clever selection of tools to get it done while sort of incubating something that is so much bigger and so much more. Dear Adiella, I think I will begin to talk to you directly here, since I'm having my brainstorming ah-ha dot-connecting moments. Of course, I am writing to you. Why else would I be writing this. I'm submitting my vapor-deposited contributions to the human-made digital layers being piled up like stratigraphy. When you go mining for facts about your father, if you ever do, here I am laid bare. You just lost your first tooth. There was an explosion in New York on Saturday in the Chelsea area on the same day we did a downtown trip to the Financial District in order to see a viking ship. We were too late to see it docking because the trains were running crazily. Not sure what time the bomb went off, but it might have been responsible for those crazy trains. Had I been more tuned-in on Twitter or whatever, I might have just turned around and gone home. But as it was, we had a wonderful day out and visited 2 of the best toy stores left in New York, which is very important now that the Times Square Toys R Us is gone. We need to transplant traditions. Ah, transplanting traditions. There's going to be a lot of that going on. -------------------------------------------------------------------------------- ## Mon Sep 19 07:41:41 EDT 2016 ### I'm Digesting I am successfully digging myself out, bit by bit. The trick is like I said to keep the tension in the machinery -- just like so many things in life -- the moment you let down your guard, the days, weeks and months slip by with no progress. The trick is to move forward EVERY DAY. Don't get constipated. It's so funny how what I'm doing reminds me so much of finally digesting what I've been consuming over the past 10 years. It's all just a matter of the mechanisms of the body breaking down all the stuff that's been shoved into it into its useful component pieces to be incorporated into the body either for growth or healing injuries. There's protien in the mess, and I suppose there's also sugar energy. It's a strong analogy. I should stick with it. I'm digesting. -------------------------------------------------------------------------------- ## Sun Sep 18 16:07:24 EDT 2016 ### Make Adi an Adept Among Idiots Wow, Adi really will be growing up in the age of the rise of Robots. With me, it was the Internet and Web's whose rise I saw up-close and personal. Add to that, Personal Computers in general with an emphasis on that old kooky 8 and 16 bit stuff, and with an emphiasis there on the Amiga computer, that did so much first and is so under-credited -- most recently, by Ready Player One. Excellent book, but what a slight to the Amiga. The fact that it itself came out as a product of the 80's and not the 90s is in itself, remarkalbe. And to add insult to injury, Star Trek was so well covered, which was hardly an 80s thing, but for the premiere of TNG in the late 80s and two odd-numbered movies -- a time-period also marked by the Amiga. But I digress. Adi. Adi's own Amiga is coming up. I so wonder what will it be, and whether she will really ever find a passion at all. So wish I had someone to guide me while I was going through that, but I was basically discovering it on my own, with it colored by the opinions of two very pro-Commodore individals, Marc Rifkin and Ed Flocco -- two friends from high school, and even earlier. They saw this incredible local gem. I can't believe my Dad didn't, with Jack Tramiel being a holocost survivor, and all that. That's one of the things I would ask my dad if I ever had the chance again -- didn't you know the Commodore story in your back yard? Wow, my Dad would have done much better for himself as a computer scientest, than as a textile engineer. But he was just old enough that being Jewish was a huge disadvantage to getting into college -- and then, basically being poor on top of that. Yep, he had to go his uncles' enticing "family-business" textile engineering route. But it should serve as a lesson to me. Do better for Adi than your dad did for you in preparing you for the future. Make Adi an adept amongst idiots. -------------------------------------------------------------------------------- ## Fri Sep 16 20:25:02 EDT 2016 ### Githug Back home with Adi, wow what an awesome feeling. I look forward to this all week, yet here I am typing. Continuity of thought. Always let Adi know you're not actually working, and that you're just capturing your thoughts through encoding them into the English language, and then into symbolic glyph representation encoded into digital form and distributed out over many machines, including one (or several) that are making it effectively published to the world. Daddy is broadcasting his thoughts through this nifty extension to the place where he stores a lot of his code -- Github. Come here and give me a Githug. -------------------------------------------------------------------------------- ## Fri Sep 16 09:32:35 EDT 2016 ### Content Publishing Monitor Ahhhh, continuity. I love the fact that this physical-ish memory of my thoughts connects multiple points in time. I am using a new philosophical approach to life, in which every moment counts -- though relaxing isn't off the menu. But at the office, write that awesome, compelling story. Start doing that right now, when your ass hits the Aeron. 1, 2, 3... 1? Fri Sep 16 11:46:22 EDT 2016 Okay, enough tweaking the general framework. Time to crank out your specific task for today. I'm itching now to incorporate Search Console data better, as the iron is hot and with search volumes imminently going away again, it's the perfect time to show Pipulate to the world again, proposing an alternative to scraping search volumes from AdWords (even if indirectly through other peoples' products like KeywordFinder and SEMRush). 1, 2, 3... 1? The one-page "small crawl". Okay, this isn't really a can of worms, but is more like blasting out the pipes for getting something into the greater system than the RegEx-based text node extractor. In fact, I should have some old code to borrow. Look over my repositories for the last place where I implemented the one-page-crawl. Focus, and get this done. It's a very interesting one to start with! Think through the function, and don't over-build! The main reliable pages to visit are the homepages, but most sites will have a pretty reliable "tech" section. It's pretty easy to find these, and the static list should always start with these tech addresses, because it's silly to re-discover the tech pages on every re-run. But we should use a function specifically to grab pages that have a certain value in the URL. This is a perfect case for using the optional arguments tab in Pipulate now too, to avoid hard-wiring what value that should be. It's interesting, my main pipulate.py file is going to be getting modified in the main trunk reflecting my current jobs. I should probably find a way to avoid that in the future. But don't get paralyzed by the small details. Forge ahead. Have something to show by the 1:00 PM call! Fri Sep 16 15:05:26 EDT 2016 Definitely some things to get used to in the new system. Very powerful in its simplicity and much-less-code personality. Fri Sep 16 15:40:52 EDT 2016 Okay, I've cleaned the source list. I can visit every one of these URLs. Now, I just need to look for matches to our published headlines from our feeds. I can work against the entire contents of the HTML of each of these pages, which is good. Let's commit the work I've done so far to Github. Fri Sep 16 17:20:09 EDT 2016 Okay, just caught up with the boss on the project. The scope of this monitoring thing is larger than I thought. I have to be able to see what's been picked up by these organizations and published (of our content) by them in the past. Looking at Google results is probably the most logical way to see that, unless I crawl each of the sites directly. I could do a site: search to see if it's actually worth doing, or if the site is too big. I have to sleep on this. -------------------------------------------------------------------------------- ## Thu Sep 15 20:53:56 EDT 2016 ### I Was Weak. Now I Am Strong. I am on video chat with Adiella right now. I am on the computer with my hands and the phone with my face and voice as I read this out loud to my daughter as I type. And I think that's pretty cool because you get a general idea how fast you can type compared to how fast you can talk. We are fake burping to each other, and that is something worth remembering. Adi is getting good at fake burping. I see her tomorrow night, after work because we're probably doing the city house this weekend. Just checked with her, and that's what she wants to do. I already told her that the wardrobe is gone, so that won't be a surprise to her, AND we went over all of her nice clothes. And the typing has just become annoying, so I will stop. She's okay with me typing so long as she can see the keyboard, which I just set up that way. Let me turn off the air conditioner. What to talk about? Thu Sep 15 21:28:51 EDT 2016 Just got off the phone with Adi. I think this was good practice. We were on for an hour already, and we wrapped up together willingly. Well, almost willingly but we have talked for long enough I don't think she was feeling shortchanged. I got some lyrics for her from Charlie The Unicorn, both the main lyrics and the one with the Millipede. She was happy. I am happy too, though I can feel the tiredness starting to set back in. At least I got the broken wardrobe out of here. Okay... sigh. See if you can't get a second wind and build some momentum again. This sort of time is rare, and Adi will be here tomorrow and I'd like to impress her with what I can do when I finally set my mind to it. How weak am I, anyway? How many times must people have asked that ABOUT ME over the past 10 years? Yes, I was very weak. -------------------------------------------------------------------------------- ## Thu Sep 15 19:32:27 EDT 2016 ### 5-Hour Energy Drink & Bearing Down On This Tonight I was falling asleep on the train on the ride home, but I just had my ramen noodles, and downed a 5-hour energy drink. That should give me what I need to do the next round of work here tonight. Nights like this can make ALL the difference. Figure out how I'm not going to be locked in one place with Adi for an hour, and how maybe you can keep it to less than an hour, given I'm going to be seeing her tomorrow. Maybe I'll call in sick tomorrow to do the car inspection. Time's running out. I had to be in today for the paycheck (I really should put that on auto-deposit), but tomorrow is an option to get my head above the water. But no, really drive this thing home tonight, get a good night sleep after the 5-hour thing wears off, and bring Adi back here to the apartment tomorrow night. Impress Adi with the state of the apartment. That's like your goal now. Get that broken wardrobe out of here. Take as many trips as you need to to do it all low profile and without too much effort. That will clear a lot of space. Also, I will have to get Adi's room looking a bit nicer. Maybe I'll put one of those glass shelves in there to make it more beautiful. We can make them her special shelves, and start filling them with the things that are really special to her. Yeah, that's a plan. -------------------------------------------------------------------------------- ## Thu Sep 15 18:41:21 EDT 2016 ### Keep The Tension In The Machinery Right from journaling at work to journaling at home! Accountability to myself! Lighting the fire under my ass includes regular check-ins. Did I waste all my time since the last check-in, or did I actually get my life to move forward some minuscule little bit past keeping my head above the surface maintenance. Well, I got my maintenance check off to management today. Hopefully, my deposited paycheck today will hit fast enough to cover it, haha! Good to keep that positive mental outlook. Of course, always easier said than done, but at least I'm a lot less stressed over this stuff than I used to be. I was stressed and didn't do stuff about shit in the past (seized-up) and now, I'm not stressed about it (or, at least LESS so) and AM doing something about it -- almost, barely. Just keep at it. Don't let the tension out of the machinery. Keep tension in the machinery. Pump some more caffeine into yourself, and use the world as your check-list. You already got your big-ish meal of the day as lunch, and now just heat up some ramen noodles, finish the laundry putting away from yesterday, and then tackle the next few things with 80/20 rule passes. Get the place in nice, impressive order for Adi this weekend! I think she got her "obligatory" visit to the Catskills out of her system last weekend. I managed to avoid having to sell that place this season, but still I have to get some of these changes that I'm trying to keep (through laziness, really) from happing all at once to happen at least a little bit, at all. I'm still actually married, and that's not good. I have to wrap up other things first, and work my way around to the marriage. Rush yourself on the correct things -- the correct things for financial reasons, mostly. -------------------------------------------------------------------------------- ## Thu Sep 15 17:05:46 EDT 2016 ### Easier & Clearer Okay, I got a great expansion of Pipulate done to day called Populate, by which a list_of_lists and an optional list of column names can be used to populate a new or existing tab. VERY useful, and works into workflow very nicely. I should make it support the same chunking logic as the pipulate process, but that will be for another day, along with the ability to append rows or columns. Okay, going to head out a little early and keep the momentum going at home. Who knows, maybe I'll finish my work from home. Everything's gradually getting easier and clearer. -------------------------------------------------------------------------------- ## Thu Sep 15 10:13:25 EDT 2016 ### Always Start From ~/ (Home) One of your big "defenses" with this daily journal is sheer volume. Deliberately make it something that will require machine learning in order to properly digest, haha! Wow, okay, think! Book ideas! Always work out of ~/ if you can, because you will always know where you keep things -- especially in the world of Github and cloned repos all over the place. Fight for that first-level directory name under ~/[something]. My journal, for example is always ~/miklevin.github.io. Windows restarted my machine overnight, so I'm re-running the reports now just to be safe. Ugh, there's permission things. Seems I have to sudo su before I python report.py. AND as I now finally check the dropbox location, it does appear that the Sept 15th reports DID run. Wow, when did Windows do the reboot, between 8:30 AM and when I got into the office? Maybe. Okay yeah, the reports look fine. Do an apt-get update and upgrade, and then restart the virtual machine headlessly. -------------------------------------------------------------------------------- ## Thu Sep 15 07:29:41 EDT 2016 ### Get Shit Done, But Don't Lose Sleep (too much, at least) Okay, I went to bed around 2:00 AM (finished MOST folding/putting away) and woke up just before 7:00 AM, so I got close to 5 hours sleep. Not bad. Not great, but not bad. Okay, so now I have to do is get my maintenance check written and dropped in the mailbox this morning, throw out some trash and recyclables, put away A LITTLE more laundry this morning, and not forget to deposit my paycheck later today (desperately needs to be in the account). I have to start tackling paper-work with equal determination and vigor, perhaps starting tonight. I may insist that we spend this weekend at the city apartment again this weekend to do the adult things I have to do to keep my life moving forward, and indeed to prevent it from becoming totally derailed. Go back into that mode that you've had to from time to time in the past to keep your shit together, keep from being overwhelmed by life, and to just generally have a tight center (not going insane, for example). Okay, sanity check complete. Get your ass moving again. -------------------------------------------------------------------------------- ## Thu Sep 15 07:05:23 EDT 2016 ### Filter Out The Noise & Keep a Good Attitude Okay, allow a new sense of time and urgency about things set in. You can't let yourself still take guff, now that you're separated. Have a healthier attitude about things. No marriage or family life is at stake, that's already shot to shit. So, when I try to wrap up discussions with Adi after only 20 minutes or so and she puts up a fuss (hysterics one way or the other), I just wind it down and say goodnight. As it was just put to me "You're the parent..." and so, I will be firm but kind and gentle and loving about ending a discussion after 20 minutes or a half hour. I can absolutely spare a half-hour with my child every night -- that's not the issue at all. It's how much laundry I had to do and how that made a situation where I had to do something for 10 minutes every 20 minutes or so to keep things moving forward. I explained that very carefully, and maybe next time on "laundry night", I either just skip the video call with Adi entirely, or call during the day telling her I won't be able to talk later that night, or do a very short video call no matter the fuss of winding up before she feels ready, which is what "got me in trouble" insomuch as that sort of bullshit flack is trouble. I answered back to "End the conversation earlier, you're the parent" with "Okay, done". Not even going to check my phone again until I'm out the door on the way to the office, and then I'm going to take it with a grain of salt so it doesn't become one of the traditional, but now retired, morning grenades lobbed at me to ruin my day. Focus and filter out the noise what you totally have the right now to filter out. I much more want to talk/think about my work on Pipulate today, which is going to be AWESOME! I'm glad I'm re-branding SEO Notebook back to Pipulate. I need to do a video on the new Pipulate urgently, and reach out to the people who've starred and followed it, trying to move them over to an immediately-usable SEO Notebook-based Pipulate. At some point, I'll even throw the old Pipulate to private, but that loses its stars and watchers, so delay that for awhile, even as you change all the internal references within SEO Notebook into Pipulate. -------------------------------------------------------------------------------- ## Thu Sep 15 01:00:28 EDT 2016 ### Act Now Do not feel bad. Do not apologize. Do not feel guilty that you are still awake at 1:00 AM. You are almost done folding, and without starting at 8:00 AM, this would most likely be much worse. You did the right thing. You must be more firm with Adi when you say that you have to say goodnight now and keep it short. That is what I tried to do several times. Shit, there was no winning in that situation. I see that now. All probabilities wind up shit fit. So, fuck it. Don't let it be my problem as well as much as you do, or else you'll have much bigger problems. If there's one thing you can say for sure about me is that I make mistakes. But then hey, we all make mistakes. That's a pretty friggin part of what it is to be a human. But you know what's even cooler? Rolling with the punches. Learning from the bruises. Replaying the event in your mind, striving for a form of objective outsider's perspective, even though you're never going to truly achieve that. The exercise is humbling and enlightening. Hey, sorry every for however I wronged you! I get mine, don't you worry! But then again, there's a keener sense of purpose now. I've got a reason to keep my edge honed, and to get my standing in good order with as many people and organizations as matter to me. And there are a few. They're up there, whether they believe it or not, given my inconsistent behavior and bouts of moodiness. I think I'm entering a moody phase, but the funny thing is, I like myself best that way. I clean up. I rise to the occasion. I internalize and visualize those things that I must to motivate my ass into action. I am doing that again today. Or was that yesterday? Shit, just sweep the rest of the laundry into the clean/sort basket and clean/sort after a good night's sleep... almost, maybe, I hope. -------------------------------------------------------------------------------- ## Thu Sep 15 00:12:57 EDT 2016 ### Still Folding Come to think of it, I shouldn't be surprised that finally getting around to doing all that laundry wasn't going to trigger off some sort of rage, if even from afar. After all these months, and after all the seemingly monumental efforts already on my part, I look around and realize I haven't accomplished jack shit. Nothing's done. It all just amounts to clearing walk-ways so far. It's better than that, but I exaggerate to get the point across. I'm working my ass off just to keep my head above the water. Good Daddy and good earner consumes every ounce of my time and energy. There's nothing left over. And if I want to do more than just keep my head above the water, it gets "paid for" out of either good daddy or good earner. Shit, which should it be? Oh yeah, I can just pay for it out of sleep. I repeat, shit. It's funny that doing (and getting done some) laundry can be such a bitter sweet feeling. It caused the shit-grief that I thought I was through with when I tried to do this sort of thing, because I happened to have been smart enough to have started it early enough to get a good amount done early enough to not end crazy-late. And I repeat, shit. Why not just not let it fucking bother you? Why not just let it roll off your back like a duck? Can I be that callous person? Pretty much the typical dads out there. Yeah, I know how pretty fucking special I am that this sort of thing bothers me so much. Too good. Get that goodness out of your system. Be a litter meaner and get some shit done. I am sorry, Adi. I truly am. -------------------------------------------------------------------------------- ## Wed Sep 14 23:54:32 EDT 2016 ### Working A Thing Or Two Out / Manning Up So I said I was avoiding staying up late by doing my laundry early, yet here I am near midnight. Well guess what? I'm still folding and putting away. I haven't done laundry in a very long time, and there's tons of Adi's and Rachel's stuff in here, so some of it is at least 5 months old, and I separated from Rachel mid-April. So here we are, mid-Sept and that's 5-months. Shit, I have to pick up the pace. And so I did. And so I decided to do a laundry night. Get in in the machines as early as possible. Transfer them to drier likewise. But I have more than one full cycle, meaning staggered in half-hour increments. Plus, you're not starting every machine simultaneously. So, no matter when you start, you're talkig about an every-20-minutes check-in until it's all out of the dryer. Rachel and Adi kept asking me if I could just do it later, and I said no. Once you get this process going, you kept it going until it was done, and I have a right to do it starting early, and not wait until after these phone-calls are over, at like 10:00 PM just to get started. Starting at 8:00 PM when I got home, I'm still folding at midnight. Starting at 10:00 PM, I'd be folding until 2:00 AM, and that's if I wrap-up folding RIGHT NOW, which I'm sorely tempted to do. Only fold the stuff that wrinkles. Make a "put-away" pile, then grab some sleep (VERY IMPORTANT), then hit everything anew tomorrow morning. And get to taxes! Yes, I used to pay to get laundry done, but I'm at zero-cash right now, and even if I weren't, the cleaning service was the first thing to go after separation. It's trial by fire for me. But I have to man-up starting now. And the way I had to handle Adi tonight is part of manning up. I can't let her become so bossy as to force me to stay on video chat focusing all my attention at her for as long as she likes, and I can't let her simultaneously to bossy be so fragile as for any reasonable interruption, or expectation of being allowed for myself some time at a time during the night when I still have enough energy to handle such things. And for the fuckers who might say, well here you are spending all this time typing, well a guy has to be allowed some breather to stop and think, can't he? Shit, we can't be automatons. You know what? Let's compare the time I spend writing to the time people spend watching TV, on average. I watch almost no TV at all. Every once in awhile, I binge on a show, but actually quite rarely, and often I just stop upon realizing what a colossal wast of time that is, when I could be spending my time thinking out loud into my journal, working a thing or two out. -------------------------------------------------------------------------------- ## Wed Sep 14 23:15:20 EDT 2016 ### Interruptions, a shorter call, or no call at all Adi went into hysterics tonight because I was doing laundry tonight. I can't drive these things late into the night and lose sleep over, and Adi gets angry when I try to do a quick call too. Fuck, fuck, fuck! I just can't catch a break. And this taxes shit is about to catch up with me too. I simply need this time, and I need my sleep, and I need to focus. Shit. Okay, it's still relatively early. I tried moving the wardrobe, and it broke apart which doesn't make me feel any better. And now I have piles of laundry, and no room to put it away. Throw out as much as possible, and make bags of what goes to Rachel, and do something zero-cost ad hoc that I can make work in my room. But basically, SOLVE THE LAUNDRY PROBLEM TONIGHT. And do it soon, so you can both have made a dent in the shithole and make Adi's going hysterical over phone-call interruptions not be for nothing. Everything is going to collapse on me if I don't get and keep my act together. 1. Be a good father to Adi (and doing laundry doesn't disqualify me) 2. Protect my earning capacity, because being unemployed and in debt sucks. 3. Get myself in order with taxes and legally. Don't take shit I receive from other people -- even Adi and her mom -- too seriously, or you will never get 1, 2 and 3 done. You'll never climb out of your hole, because doing #1 according to OTHER PEOPLE'S unrealistic criteria just digs me deeper in the hole... even separated! Use every spare moment of time digging yourself out, and laundry is an excellent place to start. But finish up and get some sleep. Think about how (and whether) you're going to handle the video call tomorrow. I'm not going to keep taking shit for doing the right thing. Even now, I'm doing the folding into the night, trying to not make it too late. If I started all this AFTER a typical 1-hour+ video-call with Adi, I'd be up all night. And if I tried to wrap up earlier, it's the same hysterics. I don't know what I'm going to fucking do. It's finally time to let her just burn out on her hysterics, and learn that's not the answer, and on some nights when she's talking with Daddy, there's just going to have to be interruptions or no video-call at all. -------------------------------------------------------------------------------- ## Wed Sep 14 09:08:20 EDT 2016 ### Your Environment Is Your Checklist Just continue being yourself. Do more of what you do AND the stuff you know you need to be doing. Life isn't easy, but humans can rise to the occasion and do great things. Look at the energies you have put into things over the years, and look at putting that same energy into things now. Wrap things up and wind things down that need wrapping up and winding down. Don't delay, and USE YOUR VISIBLE ENVIRONMENT as your checklist. No moment exists for sure except for the now, so apply your efforts on what is most important RIGHT NOW and what you can do the most about RIGHT NOW. Use a series of iterative sweeps, deliberately making the next things up that much easier. Adi is actually WITH Rachel almost every minute of every day, on their end at Staten Island, and that gives ME most of the free, discretionary time no matter how tired I am at the end of every day. So, the letter of intent thing you have to do for Adi's homeschool today, you can do during lunch, stopping at a local post office to send it certified. I'll have to find an envelope, which I forgot to bring in from home today, but so what. I'll find it. Don't let the little things in life stop you anymore. The little things in life are what you sweep away through diligent preparation. That diligent preparation is the type of thing I used to do so much in my life, but almost don't do at all anymore. I was crammed up and jammed up for almost 10 years, but I'm not anymore, so don't let any nonsense force-of-habit things deter or delay you. Focus! Time to go into the High Energy Zone. I have a playlist of like 4 almost identical (beat-wise) songs on a loop. Use them to drown out all background, set the energy-level of your work, and not be TOO interesting to distract you. Now, think: 1, 2, 3... 1? Ugh! Email alerts on my phone are totally destructive to flow. Turn them off. Okay, now... All your work on Pipulate doesn't mean jack shit if you can't work FASTER than other people would on equivalent projects. It's time for The Pipulate Way. There's many ways to tackle problems put before you in the field of marketing or any other endeavor where you need to monitor data and be part of the feedback loop. Data comes in, you observe, digest and act on it, which in turn somehow alters future data coming in. Theories about cause/effect are posed, tested, and iterated upon. If you do a good job, things start going your way. If you don't, you question and adjust your suppositions. This is the feedback loop which is present in so many things in life. Here, we concern ourselves with one particular approach to all this, which I call The Pipulate Way. Create the docstring for the pipulate function. I have to start ignoring my emails. I can't stop right when I get started to explain hreflang to an editor because the production person is out this week. Highly disruptive, and should be in the hands of someone who handles hreflang regularly. Otherwise, it will be fubar. Okay, back to what's important. NO DISTRACTION! And I do have to go to the sprint meeting today, because the menu work that I've been working so long on is going into production (yay! but distracting again). Okay, do what you can as quick as you can. Re-establish flow. Fuck. Wed Sep 14 11:37:19 EDT 2016 Okay, sort of at the sprint meeting. Turing taken. Holding meeting out in the open room. Interesting. I'm just going to sit by and listen for when it gets up to the menu piece. Okay, I just made the pipulate function support all the most important values as local variable arguments. This is the key thing for enabling workflows. Workflow instructions can be as simple as stringing up Pipulate function invocations. What's the best way to tackle the citation monitor? Wed Sep 14 18:17:35 EDT 2016 Got workflows working. -------------------------------------------------------------------------------- ## Tue Sep 13 22:00:37 EDT 2016 ### Navigating My Way Through Life I should take advantage of this journal more often for personal navigating my life type issues. I'm a friggin' SEO in NYC who got in over his head with debt in a very short period of time. Now, I gotta dig my way out. Good thing is I earn well, and I have to protect that earning capacity. And I have to protect that earning capacity by remaining brilliant, and on top of all the really important things in life, in society, and as a father to a child that I must be, which I am not currently. So, there is course correction stuff involved here. I reserve my right to be vague. Suffice to say, continuing to do what I do online, precisely as I do it is a key part to me correcting course. With 5000+ YouTube subscribers, and my latest iteration of the Pipulate system under my belt, I think I'm locking in some credibility, potential, and quite possibly be reaching my greatest career heights by really having parented a noteworty (in these turbulent times of SEO) programming endeavor. I just have to keep at it, while still ensuring enough additional slices of time in my life to take care of everything else that's also equally and maybe even more important than protecting earning capacity. Keep that in mind. Keep that in mind tomorrow morning, and every morning and day following it. Be nice to your nighttime self, and move things forever little more in the correct direction every friggin day. Better and better and better should be my mantra. And I have to stay true to it. And I have to begin by getting a good night sleep when I can, and that means now. Good night. -------------------------------------------------------------------------------- ## Tue Sep 13 08:51:32 EDT 2016 ### Operation Awesome I'm going to start Operation Awesome on All Fronts! Currently, I'm not awesome on any front: - I'm only a part-time dad to Adi (shit, shit, shit!) - I'm not the innovating all-star at work (yet, but I soon will be) - I'm totally fubar'd on the home-front (several home fronts) At least I'm keeping the pets that are still with me alive. Oh well, also at least I'm self-aware enough to get my ass in gear and fix a few things... optimize! Shit, it's not even optimizing yet if you don't have baseline performance occurring. Optimization is part of ongoing maintenance, fine-tuning and tweaking. I have to get to the point where I can afford those sorts of luxuries. Yet indeed, it is critical, or I could get my ass in some real trouble. I have over-extended, and now I have to reel my life back in. Evaluate, systematize, reduce. ESR. Had the formula 20 years ago. Should I have stuck with it? Then, I would have never had Adi, and I have to believe, having Adi is the best of all possible possibilities. Ensuring your income is the top priority, but cannot be allowed to eclipse all else. Avoid the big colossal fuck-ups, but recover from the merely big fuck-ups that you can't back yourself out of anymore. -------------------------------------------------------------------------------- ## Mon Sep 12 09:46:13 EDT 2016 ### Monday I got one of my first talking head videos in awhile out just now. I guess Google taking away search volumes predicated it. Wow, yet another "Not Provided" event to drop on the heads of SEOs. Guess we had it coming. On the positive side, I just did my all time favorite Google Analytics integrations on Friday before I left for the weekend, with seconds to spare. I got caught up on commas in URLs in the filter field of GA API requests. Commas in particular have to be backslash escaped in URLs. Guess old Vignette URLs need a lot of escaping. Make sure the non-debug-mode of last week's report revamp works. Do this in time for Marat having a full set of refreshed datafeeds by the time he comes in this morning. Okay, running it now. Also, use the GA function you made on Friday to run a different date range to assure yourself its still working. Mon Sep 12 12:58:28 EDT 2016 Okay, I delivered the findings on the search friendly URLs, batch 2. I was able to do this with the SEO Notebook system, instead of pulling everything from Google Analytics directly. I'm going to have to do the same with Search Console as what I did with Google Analytics... AND start incorporating in SERPs. Ahhh! Okay, I need to do a sort of monitor against a known list of URLs, to see if there's a mention of of it on a list of provided pages. Mon Sep 12 16:24:39 EDT 2016 Pretty productive day. I need to think through the next move on SEO Notebook. Ponder it a bit and head home on the early side to make some progress on the home front as well. Full weekend with Adi in the Catskills. I have to start living like a full fledged human being who has little to so support on anything, who has over-extended myself on life, and who has almost no personal discretionary time where I'm not made exhausted by the commute or time on phone with the kid every night -- burning the candle at both ends AND the middle. It all has to be paid for out of somewhere, but I also need to start fixing the situation EVERY SINGLE DAY! No excuses. -------------------------------------------------------------------------------- ## Fri Sep 9 09:50:51 EDT 2016 ### Friday I'll be in the Catskills this weekend. I have to say, I'm enjoying the Catskills a lot less this season than prior, and Adi always wants to go to the city apartment and hang with the Kitties. Even in the Catskills, she often tries to opt for the cozy indoor stuff over the running around outside stuff. She is certainly my daughter. But the season is winding down, and we don't want to have regrets for hardly having gotten out there. I have that futility feeling lately. I had a very good Labor Day weekend with Adi at the apartment, FINALLY starting to clear away the paralyzing clutter. It was a 3-day weekend, and I stated right at the beginning what it was all about with Adi, and she was cool with it and let me work. I got so much done, like Adi's wardrobe and the giant pile in the main area. Still, a week has gone by and I've done nothing further. I remember I used to so beat myself up over this stuff. Now, I feel that way at work too. I took MUCH too long to get a project done that should have been a quick one, but the complexity combined with the shock of going back to Python 2.7 and NOT using Jupyter Notebook took me by surprise. The rapid testing I had gotten used to for just these types of situations went away. Maybe I should have installed Anaconda or Miniconda for Linux on the Kubuntu box, but I'm at the other side of this challenge. My counterpart still has to do the UI components of it, but I go into a support role now, only having to switch out which script is run under cron.daily once we're comfortable with the new data sources. However now, I have some very urgent work that has been getting delayed because I was too slow on the Report 2.0 update. I have to give more accurate time estimates, or just get faster. Getting faster is the better idea, because that aligns with my self-image and overarching goals of actually mastering and becoming adept at Python, and technology in general. So, can I apply that thinking to the job immediately at-hand? Yes! Get the SEO Notebook work done that allows you to pull numbers from Google Analytics on a per-URL basis! Extend it then to Search Console as well, and pretty soon, you'll have one of the most useful libraries out there, made specifically for use in Jupyter Notebook as the main UI... also preparing yourself for Jupyter Labs when the UI becomes much slicker. It's a very good plan. I need to figure out the prescribed approach to databases. But the most important thing right now is that GA/SC integration with SEONB. 1, 2, 3... 1? Hmmm. Finish a Pipulate function specifically for generic GA integration... right back to where I was with Adam Whippy YEARS ago... haha! Okay, deep breath... Okay, put some niceties on SEO Notebook. It'll get you into the way of Google for today's work. Fri Sep 9 12:26:34 EDT 2016 Okay, this morning was settling back into SEO Notebook. I really have to make this most useful to most people. This is a big dot connecting I'm about to do now. It's what Amie Nguyen asked for. If I do this round of work well, I could possibly ring her up and give her the demo. But first, I've got to make it exist! Okay, I've got the code written and scattered around. Gather up a remarkably clear instance, worked in Template-wise. Fri Sep 9 17:53:10 EDT 2016 Wow, took me long enough, but I finally got the GA function working. I had a long hitch on URLs in filters that had commas. They need to be backslash escaped, and not URL encoded. Sheesh! -------------------------------------------------------------------------------- ## Thu Sep 8 20:19:07 EDT 2016 ### Dear Amiga Computer, If you weren't planning on sticking around, why did you let me fall in love with you? Oh, you did stick around, you say? You just never grew older, while the world around you integrated and matured in a positive feedback loop with which you couldn't compete? Not with Commodore as the stewerds, such as it were, you say? Ahhh, I understand. You were one of those anomolous events that was before your time, and would hardly even have been recognized or believed for being what you were by the majority of folks interested in such things at the time. Turns out, innovation isn't always recognized nor rewarded for what it is when it first appears, so says Bezos: <blockquote>Invention requires a long-term willingness to be misunderstood. You do something that you genuinely believe in, that you have conviction about, but for a long period of time, well-meaning people may criticize that effort … if you really have conviction that they’re not right, you need to have that long-term willingness to be misunderstood. It’s a key part of invention.</blockquote> Read this: http://www.collaborativefund.com/blog/when-you-change-the-world-and-no-one-notices/ That. Yup. It doesn't pay finantially to be too far aheas of the flock, for a flock they are. A flocking flock of follow-the-leader flocks. Of course, many a premature innovator uses this as the excuse (including myself) that they could have been the next Bezos, if only their timing were right when they had that idea x-y-z, back in such-and-such a year. Well, bullshit! I call ballshit on that concept, even embedded into my own psyche as it is, for I'm coming to the realization that it's a complex game with lots of moving parts, including wonderful luck, good timining, and obsessive compulsive preparation habits. Perfect storms are found. For this reason and more, I've made Levinux. Levinux is a kooky little distro of Linux. -------------------------------------------------------------------------------- ## Thu Sep 8 10:47:28 EDT 2016 ### Planning Today Okay, check out every data feed to make sure SQL Workbench/J can do a select against every new CSV file foreign data wrapper under PostgreSQL. Okay, everything check out except for the one case where GSC became invalid. Three is a magic number. Still listening to Ready Player One. Wow! Okay, do I have to get into the flow today? Or is it time to pop my head up into help desk land? Probably. Let me knock off a couple of our maintenance projects. Stay on-message. Don't wander. Keep your eyes on the prize. Okay, think this through well. Don't be an idiot. Give yourself super-powers instead of housekeeping grief. Build your housekeeping robots instead of scrambling from one to the next to the next project. Hmmm. Back to Jupyter Notebook and Python 3, I think. Time to run easy reports against lists-of-URLs. Hmmm, don't have a list of the 2nd batch. Get an email out to co-worker who might know. Okay, he forwarded it to the developer who did the work and probably has the list. Get a bunch of plates spinning now. You can afford to switch between some different tasks now. Okay, now I need to fill in a few gaps now in my technology toolbox. Databases are still a weak point. I like Python SO MUCH, and I avoid more formal database work by using Python dictionaries with picking and shelves and such. It's time to... I'm not exactly sure, what. But it has to do with scaling the Darryl Excel pivot table tools. Hmmmm. Think. Thu Sep 8 12:02:23 EDT 2016 Got the list of new search friendly URLs to check. That'll be my #1 priority today. Hmmm, okay. Knock this item of the list very firmly today. Oh, also there is the taking Reports 2.0 out of debug mode, and making them a non-overlapping cron job with the old reports. These two things combined are plenty for today. Given the number of search friendly URLs to check, I think I'd like to do that through Jupyter Notebook and GoodSheet rather than manually through Google Analytics. See yourself from the outside, as an objective observer. Be the sort of person that your daughter Adi could and will be proud of. Don't be a dumbass. Thu Sep 8 13:47:09 EDT 2016 Okay, I have a meeting coming up at 3:00 PM to do things like Gap Analysis to figure out new topics for us to address. I have about an hour to do the GA lookups against a list of URLs. This is a classic sweet spot for my new SEO Notebook system. Okay, I need SEO Notebook to start to feel totally second nature. Hmmm. Thu Sep 8 14:36:09 EDT 2016 My head is still sort of spinning from my Report work of the last couple of weeks. -------------------------------------------------------------------------------- ## Wed Sep 7 09:51:47 EDT 2016 ### Reports 2.0, Take Ten! Okay, get this CSV massaging done. The first-full-data logic doesn't appear to be working. Solve that first. Transitions are so important. All this audio-listening stuff I'm doing lately... firstly, like wow, I never really enjoyed listening to music and shit much growing up -- always a distraction from my thoughts -- intrusive, and unwelcome. Mind-altering, when I liked the state of my mind just fine. Finally after all these years, getting in touch with my inner animal. Wed Sep 7 16:42:54 EDT 2016 Ugh, okay it all really became tricky in the reports. There's so many moving parts. Combing together multiple lists and dealing with all that entails is... well, complicated. Lots to keep in mind and to be sure goes precisely correct. And if the original data-pulling queries need to be re-executed. Hmmm. Okay, I feel confident I have it correct now. Test one property. Yep, it tests out good on a single property. Running all the properties now. There should be a bunch of good feeds there tomorrow. Pshwew! Manage the complexity of all this. Don't let future projects get like this. Try to shape them yourself for maximum flexibility and minimum complexity. Find that UI that's going to get the Darryl tools online easily. -------------------------------------------------------------------------------- ## Tue Sep 6 11:04:48 EDT 2016 ### Taking Advantage of Being in New York Okay get your weekly report out! Okay, done. Fix the CSV files to call the reports 2.0 project, at least from your end, done. Go get some food or something. Tue Sep 6 13:33:32 EDT 2016 Just took a walk to Grand Central Station and back. Good for mind-clearing. Got a chance to look at the "new" Apple Store -- only my 2nd time there! Wow, those Apple watches look pretty. It's almost amazing to me that I don't have one, given my interest in say like calculator and memo watches all growing up. Now that I really have such a real-life need for such organization, my switch onto Android basically took me out of the arena when the product was released. Okay, I had my fun. It was important in my continuing mission to map all the NYC space around me better than I do currently. I need to visualize and "feel" where things are better. I entered at the Madison & 42nd Street entrance. But it's onto debugging those CSV files. Time to break out my emergency Dr Pepper. This is the time I have it around for. NEED that caffeine. Didn't catch up on lost sleep from the weekend, but don't let that stop you. 1, 2, 3... 1? Okay, it looks like I needed na_rep=" " on Panda's to_csv method: http://stackoverflow.com/questions/20665141/writing-a-pandas-dataframe-into-a-csv-file-with-some-empty-rows Re-generating reports. Tue Sep 6 16:33:07 EDT 2016 And just met with Josh Hyman for a coffee. Much fun. Okay, very very close. But for some reason, the last day with collected data from GSC is coming up empty. Maybe the date I'm discovering isn't actually being used. Check that as the last thing before you head out today. -------------------------------------------------------------------------------- ## Tue Sep 6 10:20:22 EDT 2016 ### Producing Content For Multiple Channels According to Permissions I'm starting to come to realize just how important moving my journal over to vim, and more generically, electronic format has become in my life and my thinking. I've become a multi-channel individual with lots of publishing channels for different types of thoughts and content, or whatever you want to call it. My thinking-out-loud that I don't mind being public goes well here, which I type-into every day from home or work or wherever, using Github as the repository location, but because it's a distributed repository, a complete copy of this journal exists everywhere I edit it from. I may even end up adding my phone, if I can get an environment that isn't terrible for this sort of thing on mobile. But I sort of doubt it, because the mobile platform lends itself to something totally different, which I first saw done well with Apple Notes, but which lately I've moved over to SimpleNote. It's nearly real-time, reliable syncing of mostly unstructured notes across mobile and desktop platforms -- so long as there's an Internet connection. And THAT'S another channel -- a less public channel, where I can still capture thoughts, although also not so private, due to their cloud nature. Truly private (writing) thoughts go into yet different apps that have passwords and encryption and don't touch the cloud nearly at all, except insofar that they live on devices that are themselves Internet-capable and under the complete control of the hardware vendor (Apple), and so can't really be considered 100% private, either. But then what really is, except for not writing things down at all? And then, you lose all the thought-processing and archival advantages of writing-things-down, which is not really acceptable especially on those most private thoughts, which sometimes need the most "thinking out loud" type of processing. -------------------------------------------------------------------------------- ## Tue Sep 6 09:24:58 EDT 2016 ### Wow, I Knew How To Spell Careening Tuesday morning after Labor Day weekend. Gotta get my weekly report together for boss. Don't forget to do that -- not even for one week. It's a great exercise in clear thinking. I want to actually DO my first to-do item now, desperately to get the reports ready for their 2.0 phase. I also am using Adi's Boogie Board (given to her by Jason Minter) to help keep myself organized at the office. Like so many of her things, it was just relegated to the "too much to know what we actually own" pile. I did a lot to fix that this extended weekend, and it was one of the most important weekends of my life, as I begin to get my life back on track and "code up" Adi correctly, so that she knows how to tackle big life organization events. I'm grabbing the reins of my own life and steering the careening wagon into the direction it needs to be traveling. -------------------------------------------------------------------------------- ## Sun Sep 4 22:35:05 EDT 2016 ### Make My Moments Work For Me Adi's taking a bath. Pshwew! She went to sleep late last night and woke up late this morning. I'm in for it again, tonight. Try to wear her out. This wrestling stuff is killing me, but I brought it on myself. Wow, what an intense 2 days of cleaning and organizing. Well, not really cleaning much. But organizing, for sure. It's like taking the first deep breath of getting my life back on track. Steady, now. Get a feel for what's what and what's where. Neurally map all the locations of the important stuff, and start self-programming and modifying my own code and my own environment to get running a new, most excellent program. It's doable, and even quite a positive prospect, this digging myself out of everything I got myself buried under feeling. Playing A Wrinkle In Time for her while she's in the tub. That's like an ultimate pleasure -- a hot bath and a good story and no rush or hurry to be anywhere to do anything to live up to any responsibilities you got yourself into. It's total freedom. Freedom of spirit and thought and possibilities in life. I want her to see the possible outcomes of various probability paths. She could be a scientist or an explorer or an athelte or a musician. Or she could be all four or none at all, and still be perfectly happy and fulfilled in life. Possibilities. It has been a nearly herculean effort for me today to stay the course and get organized. This has been a fabulous first major pass. Just stopping to relax a bit while she's enthralled in A Wrinkle In Time. But I hear her talking to her toys. Even so, going over there to hover. Maybe... soon, after Billy gets his cuddle time. He's pulling his billy games. I'm enjoying typing on this Razer Chroma keyboard. I'm going to have to experiment with some other configurations and effects. Hmmmm. Yeah, it's not quite the professional typist feeling of the Das Keyboard Slient Pro that I use at the office, but also... just friggin awesome. Very promotional of getting into the zone. Hypnotizing almost, I think. I may take it into work just for fun. The C64x would probably be more fun. Ugh! Building that again and getting it all working will be highly symbolic of me getting my life back on track. The morning I tried and was knocked off my ass was just as symbolic in the other direction. Now is the time to make my moments work for me. -------------------------------------------------------------------------------- ## Sun Sep 4 10:02:22 EDT 2016 ### Zzzzz Went REALLY well last night, but Adi got didn't get to sleep until about 1:00 AM. -------------------------------------------------------------------------------- ## Sat Sep 3 14:12:01 EDT 2016 ### Six Months of No Progress Would Be Shitty Yesterday, Rachel sprung Hebrew School "plans" on me, which would be Sunday school with me having to drive Adi back to Staten Island early Sunday morning, and then "being the Sunday school parent" hanging around while she's in class, and then picking her up and spending the rest of the day in that area, because it wouldn't be worth coming back to Inwood for the rest of the day. It amounts to the drive back to Staten Island I'd have to do that night anyway occurring in the morning. I wish I was relocated in Staten Island already so I wasn't burning through my money so fast with this mortgage, like a dumbass. Get out of sitcom-mode. Shit, what a trap I fell into. Past ten years are only just starting to become clear to me, and no wonder it's taking me so many months since "true" separation (mid-April) to get my act together. I started at Ziff-Davis in March. March, April, May, June, July, August, September... ...and just beginning September. Okay, it was mid-April when Rachel moved out. Mid-April to mid-September will make 5 months -- six months would be too much and too symbolic to not have gotten any of the important stuff done. But I'm still too crammed-up and jammed-up to even do ANYTHING! I'm going to need some serious table-space when I get to the paper-work stage of unclusterfucking my life. But that has to be sooner rather than later, and if that means plopping Adi in front of movies all day, so be it. Shit, father-of-the-year award for sure. Well, at least it's Big Hero 6 right now. She's asking me questions abut the movie like "what does that say" when I can't even answer. Shit, these things bother me. Okay, okay, continue on the next 80/20-rule pass... -------------------------------------------------------------------------------- ## Sat Sep 3 13:27:26 EDT 2016 ### Trying to Get Organized I'm relying on the 3-day weekend at the apartment to make this work. No Catskills on Labor Day weekend. Fuck it, I'm pulling myself out of a downward spiral in my life that resulted from a time-of-weakness (while I was at 360i and all-in on baby-support) and got kicked in the stomach by being laid off... well not really laid off as such, but voluntarily taking the hint. I was no longer needed in the tech department, where I transferred to to sharpen the saw, and fortunately did so as I got my proper HTML5/JavaScript responsive design Bootstrap wake-up-call. Took it. Internalized it, right in time to be downsized and outsourced. Well then, I've been proving to myself that some scarred suburban kid who could climb so high could climb precisely back that high. It often comes from secret weapons, but my secret weapons take years to build (or get familiar with), and then tend to have fairly short shelf-lives -- like the generalized system I used at Scala, Prophet 21 and Connors Communications. That was a pretty good run, but now Python itself is that secret weapon, and my stuff is just very light packages on top. And so, I will take brief breaks to do this journaling to adjust my mental condition. The challenge is to always get me back into the right mindset to take on the next small 80/20-rule sweeps. Do the next sweep... -------------------------------------------------------------------------------- ## Sat Sep 3 18:38:51 EDT 2016 ### Adi and Audibles Been through Brave, a Star Butterfly, and now The Real Show. Bob's Burgers may be in our future. Oh, and a long stretch (1st 3 chapters or so) of A Wrinkle in Time, which Adi was getting pretty into, before it reached information overload. All new stuff in massive quantities can be too much, so we switched back to "TV" at that point. Funny thing, TV as I knew it is something that Adi is not growing up with. I don't even know if they subscribe to cable at Adi's Staten Island home. Anyway I'm glad Adi is into Audibles. I should switch to ME reading the story (one to buy on Amazon for the Kindle) just to jolt Adi into different perspectives. That's a job on the brain, alternating between Audio books and actual reading. I look forward to that tonight. Neat! -------------------------------------------------------------------------------- ## Sat Sep 3 15:41:26 EDT 2016 ### Calculating Whether I Am Really Time-strapped (yes!) Okay, this attitude and this mindset is perhaps the best thing I could do to get myself back on the right track. Making better progress than I have in the past 6 months, with the exception of the minor room re-arrangement soon after I still had the benefit of the cleaning people having recently done their thing. The piles were essentially tidy and pushed out of the way, so I had enough room to rearrange some furniture to have a bigger feeling living room. Next, I have to keep the pressure on. The constant, 100% all-in maintenance mode. Only relax and decompress enough for you to maintain your sanity, then go immediately back into the next 80/20 rule sweep, until it's hard to work in 80/20 mode anymore. At that time, get some table space and tackle the paperwork. Oh, and get another sport wagon so carrying stuff back and forth to the car becomes easy again. Rachel bought a replacement one, but it's in Staten Island now. Okay, wagon ordered along with contractor bags. I'll fill a contractor bag or two each week, and then wagon them over to Rachel's. That should clear things out pretty quickly. I guess I end up moving her after all, hahaha! Okay, still making some good progress. Don't let up. I hugged Adi, and she played the won't-let-go game, and I had to be very stern with her. These little things explode into yelling and crying in about 2 minutes if I don't squash them immediately. I have to use a lot of the survival mechanisms I developed for my mother/sister combo even with Adi. But do it in the most constructive way possible. Sometimes Daddy needs to do what Daddy needs to do, and that bullshit that was said to me once "If I'm just going to drop her off with you for you to ignore her..." can go to where it belongs. Just because I have so little time with Adi during the week doesn't mean I can let my overall life go to shit. I don't have the strength to do this work after an 9-hour day (9 to 6) and an hour-and-a-half commute on either side. That's 12 hours of the 24 hour day. Subtract 7 hours sleep for 4 hours. Adi likes to be with me on the phone for at least an our every day, and that's 3 discretionary hours. Then of course, there's eating and getting dressed and bathing. That's a total hour, leaving about 2 discretionary hours per day for me to... try to cram all this shit I need to do into -- while I'm exhausted. I have to pay for it either out of sleeping time, or... I don't know. Adi-time during the weekend. Shit. Do this in the most constructive and helpful-to-Adi way possible. -------------------------------------------------------------------------------- ## Sat Sep 3 15:15:58 EDT 2016 ### Following Her To The Living Room to see Blasts From The Past Okay, making tiny, incremental improvements. I think I'm getting through to Adi. This is part of personality development, taking her from.... Daaaad... Okay, had to light birthday candles to pretend like it's her birthday. Better to just give into those fancies than to try to fight them and play the role of the asshole dad. Be the great friend. Daaaaad... she calls me whenever she hears me typing... a huge downside of these clicky keyboards. Okay... shit, just plow through this striking the right combination of attentive and engaged father, and having to live up to my adult responsibilities father. I won't be any good to her if I get hauled off to jail or shit like that for tax stuff. Gotta literally dig my way up to being able to address the big things head-on and organized enough for it to not just be a boatload of stress based on the helplessness of being disorganized. So fuck it, sometimes you have to rip off the bandaid. And for all you fucks who might be reading this in a critical mindset in that I'm spending this time writing, remember I write very fast, and this amounts to just sitting down for a momentary breather for a few minutes. -------------------------------------------------------------------------------- ## Sat Sep 3 14:50:48 EDT 2016 ### Daaaaad! Dig your way out. Feel the Panic. Okay, even doing 80/20-rule passes, it's almost still like paralysis, the pile of crap is so large. Be merciless in throwing out. Anything that would make Adi cry if she found out, just throw into some "keep" container. It's all about the container units. Too small, and you're fucked. Too large, and you're fucked. Too hard to open and close, and you're fucked. Too easy to open (and never close), you're fucked. Strike that right balance. Get everything into a place. Have a place for things. Make decisions fast. Be clear with Adi about what you're doing and why. Don't get angry, but rather, make it par of her eduction. Let her see the transformation take place around her, no matter how (subconsciously or not) she tries to prevent me from it. Echoes of her mother, but I have a chance to shape my environment, and help shape structures in HER head by her seeing me shaping my environment. And around 5 months have slipped by with barely any progress, making me a dumbass who has to go into panic mode, so that I can have my act together enough to serve papers right after the 10-year anniversary. Daaaaad! -------------------------------------------------------------------------------- ## Sat Sep 3 13:50:59 EDT 2016 ### Off to Make Vanilla Pudding - Pushover vs Asshole Did another merge of an uncommitted journal instance. I'll clean up all the order and stray tags later. No urgency, as my data tricks against this index.html file will let me sort it all out. It'll be an interesting exercise. The cleaning up and organizing of my journal and my thoughts. The process of actually extracting a book out of this. Even in these few moments of escaping to switch journal locations (to save battery) and capture a few thoughts, I can't even get a moment. I have be more clear in how I communicate to Adi. Sometimes I have to just say no to her. I'm talking this through out loud with her, and she says, that's all I needed to say. I know from experience that's the opposite from the truth, but it's the delusion I need to embrace to not have this weekend degrade into a terrible time. Okay, let me go now make vanilla pudding with her, hoping not to make more clusterfuck mess than I've undone so far. She's walking up right behind me right now trying to give me two-part braids. I have to remind her again that I said no. I'm going to try my best to not deteriorate into being a shithead asshole dad in the process of simply trying to not become a pushover spoiling her dad. I already probably did. Shit. Repeating patterns. Okay, off to make that pudding. -------------------------------------------------------------------------------- ## Sun Aug 28 23:31:30 EDT 2016 ### Make Coding Look Cool These weekend sessions with Adi are really something. Typing on this rippling waving rainbow is really something. So begins a psychedelic era that might just be able to begin rivalling the Amiga. The Amiga was indeed psychedelic. But for now, I eat and try to get to sleep ASAP. Have to knock it out of the ballpark tomorrow. -------------------------------------------------------------------------------- ## Sun Aug 28 02:16:52 EDT 2016 ### Today I am 46 Years Old And here I am at 2:00 in the morning, typing by the radically strange light of a dim waving rainbow which radiates out from every keypress as a bright rippling rainbow. It's as awesome an experience as I can imagine, and on any other night, I would go on a long typing sprawl, but not tonight. Tonight I commit and push and get some zzzzzs for another awesome day tomorrow (today) with Adi. She didn't get to sleep until after midnight, and then when she did, I did my own things for an hour and a half or so. And now, I wrap up by testing the degree of awesomeness of this keyboard, and yes, I do see the difference to a typing keyboard connoisseur between this and a das keyboard. But the truth is, I'm in it for being able to easily make my own variations under Synapse as anything else -- so long as it was a clicky keyboard experience. Buckling springs are just really too loud, and I don't want to buy an illuminated Apple cordless keyboard. I want to have an illuminated keyboard with a really awesome typing experience. I would describe what I'm experiencing here as close to psychedelic. Wow, these designers of these responsive multi-chromatic LED keyboards are really onto something... wow. I don't need any room lights on to use it. The dim rainbow wave is perfect for identifying where the keys are, and then the bright rippling rainbow is just positive feedback typing satisfaction, just as the clicky key sounds and snappy tactile switch feeling on a keypress. It's only a bit odd that the alternative (shift) values of the keys is not also illuminated. I guess they had to make the choice and aesthetics won out over higher information resolution that would be ugly. I think they made the right choice. I will just have to commit the shift-values on the numbers row to memory finally, after 46 years. Woot! Another year closer to fifty. Get something done in your life, man! Lock up some legacy stuff. Be known for a few things. Being the sort that keeps this sort of journal, and then Python-modifies will be one. Can't wait to do the YouTube journal integrations. -------------------------------------------------------------------------------- ## Sat Aug 27 19:00:03 EDT 2016 ### All I Want For My Birthday is To Putter It's amazing how Adi's letting me get some stuff done at home, once I explained to her that wanting to do so was my birthday wish. It's not that I don't want to spend time with her. It's just that I want to spend time with her during this most valuable time when I am both organizing and expressing my thoughts and recording them for... well, who knows? This just may be the beginning of a common human group conciousness, for all we know. We here in Github today may very well be the pioneers of such a thing... or Wikipedia... or Fred Fish. Who knows. Anyway, I'm enjoying things. Sat Aug 27 19:48:00 EDT 2016 Just got back from the local supermarket at the bottom of the 215 steps -- Associated, I think. Got close to $100 worth of groceries, including steak, blue cheese, apple cider, bagel chips and whipped cream, that last thing being for the PieFace game Adi bought for me over Amazon. Today is going to be a fun night! Sat Aug 27 20:35:57 EDT 2016 It's my turn to get a pie in the face. Wow, what a wonderful 46th birthday choice. Do I detect Rachel's humor in it? Perhaps. I'll do some selfies, I think. Yeah, those Apple Live Picture 1.5 seconds of video plus megapixel still as you would have normally (only) had. Brilliant. Brilliantly, Apple. I'm under instructions from my daughter to whip cream the hand. In fact, the little nut is talking into her voice changer gift to me saying "Whipped Cream the hand..." over and over again. Hear this, Internet. I read this out loud to my child as she puts the megaphone up against my ear, and I slip my phone out to capture some video of her talking into my ear. Sat Aug 27 21:01:21 EDT 2016 Just enjoying things now. Wow, two big bowls of vanilla Turkey Hill ice cream. Hmmm. Think. This is a next phase of Adi growth. Set good examples. Take control of your environment, and assert yourself and set boundaries and limitations that she MUST observe. Do I use ALL CAPS for emphasis, or the more ***markdown friendly*** notation, using triple asterisks. But this is part of your process of self-optimization. I'm trying to demonstrate to myself that I can be as self-aware, and self-asserting as any of the most educated and privileged of the world. Except for the fact that it's a ridiculous term, I like to feel like I'm boostrapping my own life. Adi is so into Bob's Burgers, it's amazing, hilarious, and sort of heartening. I love that her sense of humor is so intuitively solid. She's going to do really well. Today, when making her iTunes Music Playlist, I had to think twice about adding Fuck the FCC song from Monty Python to her list (ultimately, decided against it), but I want her to be just anti-establishment and uninhibited language-wise as it is good for an optimally expressive individual can and should be. Free. Freedom of language and freedom of thought. And relativism to an extent, and some moral absolutes as anchors. Some deep and straight forward philosophies of how the whole world works -- my theories, such as it were. I have a few doozies, particularly souped-up of late by my quantum befuddlement and inquisitiveness. Gonna do A Wrinkle in Time with Adi. Seeking this shit out, couldn't be any better time to be tuned-into all this Alice in Wonderland genera crap. Young woman coming of age, balancing opposing views of the world, and inherent dislike of boring authoritative establishment. Nonsense! Possibilities. Adi will most likely (hopefully) be coming of age during the first age of women holding the most powerful political position in the world. Wow, I'm trying to imagine a world where Trump wins... ha ha, wow. Would it be awesome, or horrifying and terrible. Fiction could not possibly be stranger than reality. I mean don't you remember growing up wondering why such-and-such celebrity couldn't be president, and having it explained to you why there's a certain process... and then there was Reagan. And then there was... Trump? Wow, that would be amazing. Wonder what it would REALLY mean. Couldn't be worse than G.W., I suppose. Sat Aug 27 21:49:44 EDT 2016 Alternating between blue cheese and vanilla ice cream. Mmmm. One downside of my new iPhone leather BookBook wallet is Adi's propensity to steal all my money out of my wallet. Harumph! How are my vi skills. Killer awesome, dude. Why do you ask? Adi's telling me that all the monsters are out by this time of night and all the pets are going into tents. She has her bed fort set up using blankets hanging down from the loft bed to the lower bed. Classic. Drinking down the last of the vanilla ice cream. Mmmmmm. Typing on a very dim rainbow waves that has waves of bright rainbow ripples on it why I type... hypnotizing. >>>>>>> caf111ef8e0b22511d2f20f9d7e01b8ad46e40fa -------------------------------------------------------------------------------- ## Sat Sep 3 13:10:04 EDT 2016 ### It pains me to fight Adi just to do my responsibilities Okay, I have 75% charge in this Mac, and that's how much journaling I'll be doing on this machine -- the only Mac available as Adi monopolizes the other two on days like this with movies. I only have one Macbook Air charging cable right now, and that's at the office, so I journal while it lasts. I can just feel Adi's hunger for my head-to-head lock-in style of attention that she demands, and could fully occupy, for multiple days at a time. And so, I have to occasionally just ignore. I've seen other parents do this all the time, and I've been conditioned out of doing that because of the viewpoints of a woman, but with her out of my life, I will start to re-assert my own way. Some level of ignoring is approprite when she's been informed (repeatedly) of the fact it was going to happen and why... and the demands are asserted RIGHT IN those time periods (when going to the bathroom -- even #1 -- would be a perfect example). In a 20-second time-period, the cries for Dad can escalate from matter-of-fact to pure panic, just to show me something about a doll or imaginary friend. I guess that's part of her being a single child from a broken marriage. Yeah, I guess it could be that. I may have to start asserting myself likewise at work, to get some more days at home to take care of my responsibilities without having to fight Adi and stress out for every 5-minutes of focus. And not carrying through on my responsibilities is REALLY starting to catch up with me. Shit, okay back to the grind. Subdue any anger you feel towards Adi for her attempts at stopping me. Always find good ways to keep her occupied. -------------------------------------------------------------------------------- ## Sat Sep 3 12:32:01 EDT 2016 ### Don't Be a Spineless Dumbass Just merged in some old not yet published stuff from the rarely used Macbook Air. Okay, this is your window to get the apartment in order. No excuses. Only motivation. Get into the right mindset. FIND THE LOVE. This is getting to be one of those panicky situations. Adi is asking me to wrestle, and more and more often that can only end poorly, because she will never stop when I HAVE TO stop, and she will not take no for an answer, and somebody always ends up getting hurt in some dumbass way. Last night she kicked me in the face really hard. So, I'm just saying no -- daytime or nightime, giving her the choice of helping me clean up this shithole. I cancelled the cleaning service, and so now it's all on me. Great, one more responsibility. This will now be the story of me bootstrapping my shitty, situation after ten years of mounting fatigue. My writing is, and always has been, my secret weapon. Be able to think-through, explore, and take control and guide your own life, through the "what it is to be human" sort of cognition process. It's the stuff that I'm learning is hierarchal structures of pattern recognizers located in the prefrontal cortex of the brain, in Kurzwelian terms. Every bit of research and writing out there, both in the hard and soft sciences, will lead us to an understanding that all living things are layered-up modifications of everything that has come before it that has led up to it. Every living creature has one unique address of ancestral tree leading up to its own unique instance. The tree branch address is unique, because the instance is unique. Twins would be split branches off a parent node. It's almost your DNA's github of highly forked repos. Soooo... let's write. Writing is the fuzzy residual data record of that DNA molecule instance, of which the actual molecule will likely not outlast this media I'm committing to, I'm sure, several public data record archives -- the sort of dependents of The Wayback Machine Internet Archive, but during a time when entire snapshots of the public Internet, and perhaps even the darknet too, will be no more weighty and no less common than trading cards. Data is valuable, so all the world's past data -- the inheritance of the generations that follow us -- are those massive crawls & local-saves of all that is known and has been recorded and published in modern times. This can easily and naturally lead to the question of over exuberance in the production and publishing of new personal media. Sometimes personal inner thoughts just thought out loud, dumped out where everyone can see... well... yes, I'm generally comfortable with that. I do have my second thoughts, and have pulled back on published material and thrown some Github repos from public to private. But nothing there is that terrible. It's just the mind-numbingly boring (like this) trials and tribulations of life. If the love is real, the relationships will endure. What is marriage but a label, and do I not deserve a fresh start after ten years of a primary fantasy being a coffin-sized room I could escape to to be alone for half hour-or-so without being inundated by the relentless demands of pet, wife and kid -- with no help, except for the $180/week cleaning service that was required to keep things passable and livable. Now, I'm alone during the week and one-on-one intensely with Adi over the weekend. Today (this Labor Day weekend) I intend to get this place in order -- enough to at least have the realistic option of giving it to a real estate agent for listing. Shit, yes. Time is running out, and I have to have proper motivation, and all I have to do is not engage with the most destructive habit of wrestling with Adi. I never thought I'd be saying that, but it's getting bad. It brings out all her worst, and I don't have the skills to handle it without letting myself get hurt and guilt her out with the I-told-you-so's. Shit, I'm not going to do that any more, so simply don't engage in wrestling. No matter what. This is something that will really test my self-control today. Adi is testing me in every way she knows how -- all the things that she knows get me upset. I simply have to stop responding to those things. Use what you learned from growing up with Joyce and Mom. Animals. Unable to control themselves or their tempers... like ever. Not even once, and me... my childhood happiness and nurturing or whatever... was the casualty. Yup, that's a large part of who I am, because that is how I got coded up. It's a bit sad, but nice that I am finally now beginning to see it with my life about half-way over. Now, do something about it. Don't be a spineless dumbass, like you let yourself be for so long. Even now she's yelling Daaaaad! like her happiness is a direct feed off of MY behavior and how good/bad I do like I'm in some sort of make-her-happy contest. With me not wanting the marriage to break up for so long and going to such great lengths to keep it together, everyone's (Adi, included) has picked up on the "I'm a push-over" vibe. Well now that I'm on the way to divorce, that effect gets transformed into quite a bit of don't give a shit surprise. Adi's old enough to start getting these things. Daddy's acting a bit differently... I can't always get shoulders anymore, and when I wrestle past when he asks me to stop, a bump on the head or a getting dragged off the bed and falling to hit the floor when not letting go is a real possibility. If Daddy stops wrestling and I haven't, Daddy isn't going to change his behavior because I insist on continuing to fight. And that's ALWAYS the recipe for something bad happening -- out-of-sync, different intentions, and faults for the injury become your own because you were warned and have consistent proof of what the consequences will be. Consistency is important, along with the clarity of the message. -------------------------------------------------------------------------------- <<<<<<< HEAD ## Thu Aug 11 08:18:20 EDT 2016 In the Catskills with my daughter, at her grandparent's place. Happy August 11th, 2016. I'm typing on one of the 3 Macbook Airs in our "family". It's Rachel's 30th birthday, but we're now separated and on our way to divorce. I'm taking forever to do it, because I'm so wrapped on at work now, and all the rest of my time is going into Adi's marathon weekends, and recovery after work. I have basically zero energy left over for fixing up the Inwood co-op apartment getting it ready for sale, but that is my goal. Adi has expressed that she wants to keep it, as the place she grew up. Well of course, but she will adjust. That is where all my money is wrapped up, and it would be crazy at this point to try to hang onto that and have to live like I'm poor for the rest of my life. Just fucking sell it, and have my income far exceed my expenses once again. God, that was a good feeling. Lesson learned: never over-extend yourself. Shitty fucking feeling. Earn more / spend less -- JUST LIKE I always figured, but it got out of whack. Maxed out my credit cards and lost my fucking mind. Well, that phase in my life is over, and there's pretty much nothing short of making sure things are on-track for Adi that can alter this plan. And things are much better on-track for Adi if her father doesn't appear like such a weakling, with all my paycheck getting gobbled up within days of it hitting the account -- shit. I will have this weekend coming up to get back on-track with preparing the apartment. It's going to have to be a disciplined marathon session with the place. No Mr. Robot or TV programming shit like that. The Audible books are okay, because I can be listening to them on headphones, walking around being otherwise productive. I shot 2 videos on the way here from the city, but the 2nd got broken into 2 parts -- something I've seen before, which I think has to do with going over a certain size limit that Android or one of its subsystems can't exceed. I'm using a system, AirMore, to get the video pieces wirelessly off of my Note 5. Oddly, Android File Transfer isn't working anymore. I think Marshmallow is changing a lot of internals of Android and perhaps doing more Apple-style lock-down on the environment. This is probably for multitudinous reasons, and not just the paranoid ones that pop into my head. 80-column vim looks just right on a 13-inch screen. It's amazing that all these Macbook Airs we have are circa 2011 models. I guess we must have gotten them between 2011 and 2012 -- three in total. That's pretty amazing and an example of what was possible with such excess income before I was really paying that fucking double mortgage/maintenance for so long. Shit, it's painful now even thinking about it, but like so many things in life, just keeping your head above the surface daily is a distraction from the harder fundamental fixes that need to be carried out -- the really hard work that takes self-discipline, motivation, and a series of mental tricks to overcome old bad habits. I used ScreenFlow 3.0.5 to combine the videos and now I'm exporting to a YouTube uploadable format. That's the encoding step really, and takes the longest amount of time of anything in the process, so that'll go for awhile, considering it's over a 45 minute video this time. Oh hey, let me go answer that other person who posted at the end of my last long rambling video! Thu Aug 11 09:03:26 EDT 2016 I can't believe how early I'm waking up these days, mostly by habit I think, given how the cats consistently wake me up around 6:00 AM for feeding. I have to take advantage of that as my secret weapon to get stuff done. Thu Aug 11 11:00:37 EDT 2016 Started doing Dungeons & Dragons with Adi this morning. It's really SO MUCH nicer to do this typing on a "real" laptop, and not the Microsoft Surface Pro 4. I guess after a few months into the SP4 (almost a year, now), I'm starting to see the shortcomings of that platform. Really DOESN'T hit that 80/20 rule point, and because Camtasia Studio sucks (compared to ScreenFlow), it never took over my screencasting. And because ScreenFlow actually CAN show all the keys you press, PLUS it can use the Mac's own internal microphone (even though I generally prefer an external one these days), and because you can flip between full-screen virtual screens so seamlessly and comfortably, the old Macbook Airs have totally blocked the MSP4 from inserting itself into my daily habits. -------------------------------------------------------------------------------- ## Fri Sep 2 14:04:03 EDT 2016 ### Dropbox Feed Publishing Process Okay, the final things to get done today before I go, because the CSV files are getting generated properly... woot! Okay, thing is it's all admin tech now. It's about copying the correct files into location in Dropbox (not all that I generated) and having the correct foreign data wrappers set up in Postgres on the AWS instance. 1, 2, 3... 1? Hmm, waiting for the reports to run. Investigate executing individual copy commands, renaming the files in the process. That will eliminate one more set of redundant files hanging around. Construct another loop, based on the same criteria for all the rest of the loops in the report, namely site name and report type. Uh, nope. I don't want to copy onto a dropbox location and then rename the files, or else it will trigger all the file-change notifications twice. Instead, rename the files in location and add the supporting logic, just knowing that the one with no special modifier term is the final product. Done. Fri Sep 2 14:56:57 EDT 2016 Wow, okay. The rest is just the process of getting it live as SQL data feeds from the CSV files for Tableau. And to that end... to that end... hmmm. Find login credentials for the AWS instance. Okay... logged in. Let me constrain what syncs with Dropbox. That uses: ~/Dropbox/dropbox.py ...I want to put that in ~/bin to be consistent with Dropbox's official instructions. Okay, now I'm filtering directories out from syncing using: python ~/sbin/dropbox.py exclude add dir1 dir2 dir3 And then you can see which folders are excluded with: python ~/sbin/dropbox.py exclude list And to see what Dropbox is up to: python ~/sbin/dropbox.py exclude status Okay, I have it constrained to just the folder(s) I want, but I will need to remember that it's an issue whenever I make a new folder name on Dropbox. I should get in the habit of keeping folder names to a minimum on Dropbox and other cloud drive roots. They should be thought of as drive partitions, given the tremendous ramifications of making new ones, moving them around and renaming them. Okay, I'm in the home stretch on these reports. Get the datafeeds ready before you leave. It's just about 4:00 PM, and I need to leave about 5:45 to get to the ferry on time to meet Rachel and Adi at 6:30. NOW I go into my private Github repo where I maintain my SQL commands. There's a bunch of new foreign data wrapper tables to create. Let's just wrap this up and get out of here. Just got a phone-call that I need to pick Adi up from Staten Island tonight... something to do with Cecil. Okay, I can feel the stress already, but don't sweat it. 1, 2, 3... 1? Fri Sep 2 16:58:34 EDT 2016 Okay, I have all the new table definition code ready. Now I just have to remember how to execute it with the right user context. Oh yeah: sudo -u postgres -i psql -d template1 -a -f /home/ubuntu/repo/commands.sql And then we confirm with: -------------------------------------------------------------------------------- ## Fri Sep 2 09:43:40 EDT 2016 ### Great Friday Morning Start Thought-wise (book ideas & such) Just right... just right. I see where a lot of my love for Apple is coming from. Someone had to carry on the torch from the days of the Amiga kicking the Apple IIGS's ass, and every other dumbass entry into the computer market, even in many ways including the original Mac and on up to the switch-over to OS X and a Unix-based core OS. Then, there was iOS which addressed every dumbass issue the computer industry has ever faced, with such innovations as putting everything in jail (sandboxes) and not letting anything but the genuine OS have privileges to a lot of the system resources that can really screw things up. So, Apple had this duality of incredibly differently purposed OSes -- one for desktop designed to yield lots of mainstream advantages (getting onto Intel hardware, developer familiarity, etc.) and one for keeping phones free of a terrifying new breed of phone trojans and viruses. How people would love to bring down Apple, and Apple's greatest revenge is to become embraced and beloved by the non put-down videogames-only crew. Hmmm. Okay avoid distraction today, and hit home the rest of the Reports 2.0 project. You made amazing progress yesterday -- all the stuff that you realized was much bigger than you expected. And you only recently had to go into the Amazon AWS instance to do some PostgreSQL stuff, and that will allow me to have an easier time of it today. It's definitely weird using the vim command to add to the dictionary (zg) and see Dropbox on other machines tell me that files have been updated. Such a simple thing. I struggle with whether the Cloud is the greatest thing in computing, or whether it isn't just other people's computers serving the power-structure, and wouldn't just disposable little uber personal data centers in a shoebox be MUCH better for all kinds of reasons? Yes. The answer is yes, most certainly, but that's still 10 years out. I should really make part of MY thing being leading THAT charge. Maybe do some REAL projects. Massive parallelism on personal GPU card -- in a non-OS-dependent way. Python should have something to help. Was listening to that Microsoft evangelist guy on Talk Python to Me. Maybe something there. I have to remember z-Enter in vim as the corollary to my often-used zz (move line under cursor to middle). z-Enter is move line under cursor to top. VERY useful. Think of it often but often forget keystroke. Get it into your fingers, like you eventually did hjkl. Book ideas: hj kl backdown upright Mr. Tech Tour Hello, I'm Mike Levin, A.K.A. Mr. Tech Tour — here to give you a gentle and accessible introduction to the technologies I love, and which fill my day-to-day life. They are already diverse, and evolving as I adjust to our changing world and help prepare my daughter for the crazy future we're going into. First-off, we already live in that crazy future. It's true. Already, we each have supercomputers in our pockets that do video teleconferencing, simulate reality and is starting to knit together a sort of global human consciousness. We are all aware in real time what's going on around the world, like a digital nervous system. Already, that's SciFi crazy. Layer in the drones, batteries and power sources getting better, fabrication of materials at the molecular scale that have miracle properties like superconductivity, invisibility or virtual indestructibility. Clothing that can never stain or wrinkle. Cars that drive themselves. One illness after another being cured, or at least becoming treatable. This is not the distant future. These are all things that are in the labs today. And you my friend can be planning ahead to be the master of that future — or, at least really well prepared to prosper in it. And you can do it in such a way so that it does not overwhelm. And I do this for you now, free, because I have to do it for my homeschooled daughter, and I figure why not. Life is a journey, and the journey is the reward, and the now is the only moment that really exists, but doing nice things for future-you will be much appreciated by future-you. So we balance what's best for the in-the-moment you and future-you. Nobody gets 100% win, but rather we distribute up the awesome so to keep everyone optimally happy. You give up a little today so future you will have a little more, just like past-you gave up a little bit in the long-ago so that today-you can be where you are today. And if you're not happy with where you are today, try to do something nice for future-you, so you'll be a little more happy with where you are tomorrow. This is my site. It's a little bit about life. It's a little bit about tech. And it's a lot about personal capabilities. And from the subway this morning: Okay let's give this habit forming a try. It's funny, I always now journal nearly every day. Lots of good it's done me. I guess I'm not doing too bad. But I do need this place to not hold back. Gotta wait for the iPhone itself to fade into the background. The whole new wallet situation, at that. I'm liking it. It feels more "me" than anything before, even the Tumi's. Listening to A Wrinkle In Time. Rachel was very impressed I picked that one out on Audible. Hmmmm. I have to be a love-worthy dad for Adi. Listening to this book for myself in advance is, so, well me. Now let me think. Yup. Master Key System. Think it through. You're on a good track in many ways. Actually doing a thing is soooo important. Yet it's so easy to NOT do things — or at least those meaningful things that accumulate into abundance. Now maybe is the time to take up iOS development, finally. I won't need to use Objective C or any other hackish solutions. I can just use Swift, which is apparently supposed to be Python-like. I can see that many of my entries here are just fine for my public journal. Eroding the divide but with highly policed selective denies. This stuff will copy and paste into SimpleNote and then into a vim terminal. Get reacquainted with iPhone. Explore your old favorite apps. Get those apps you really need and want to learn from and share with Adi. Come to grips with your relationship with Apple. From Master Key chapter one: most loss comes from self-depreciation. Most fixing this right is a matter of checklist items and habit forming and thought exercises and tiny behavior adjustments. It's driven by desire. -------------------------------------------------------------------------------- ## Thu Sep 1 15:25:28 EDT 2016 ### Nesting Loops and Stuffing Fields Next? Get that field stuffing stuff done. 1, 2, 3... 1? Figure out WHERE to do it. Before the dictionary is created. WHILE the dictionary is created. Definitely where interactive Jupyter Notebook process would help, but I can... what? Pickle the data? Hmmm. Oh, not so! This happens to data that was loaded from CSV files... woot! Okay, I can handle this. Thu Sep 1 17:26:48 EDT 2016 Okay, the field stuffing is done. Was easier than I thought once I started implementing. I'm getting these type of Python patterns down. Shit, I wished I was using Python from when I was twelve or earlier. Shouldn't be learning this stuff at fourty fucking six years old... sheesh! Okay, lesson learned. Learn the RIGHT lessons, then help Adi learn them earlier. -------------------------------------------------------------------------------- ## Thu Sep 1 09:40:30 EDT 2016 ### Made Excellent Progress Today (And Still Going) Today is the hunker down and concentrate day! Get done Reports 2.0. Free up tomorrow for Search Friendly URLs 2.0 and the provided-site-list citation monitor. It's the really small things in life that make the big differences. Tiny decisions. It's not quite the butterfly effect, but it's along those lines. Distractions are just as significant of tiny things as anything else, because distractions themselves can conflafrate. Hmmm, distractions. That's why I love this full-screen mode on Macs, particularly vim in the Unix terminal. This is maybe the best thing I do to promote "flow" or being "in the zone" during the day. I'll definitely be using earbud headphones to help me stay in the zone today, but I'm thinking I may pick one song and put it on repeat to help with this, per this Lifehacker advice: http://lifehacker.com/listing-to-a-single-song-on-repeat-when-you-want-to-get-1690818056 <blockquote>The more we listen to music on repeat, we tend to dissolve into it. That's extremely useful for creative work, when tuning out the monkey mind is of utmost importance.</blockquote> My choices are actually quite ridiculous, but they work I think. They all share a similar beat, and the astoundingly cool lyrics that I am starting to believe activates my higher cognitive capabilities: - Bad, Bad Leroy Brown: Jim Croce - Candyman: Cristina Aguilera - Crazy Rap: Afroman - Another One Bites The Dust: Queen Four songs, just set it on repeat and forget about it. If three songs is too much, I'll reduce it to one. Even the transition from one to the next might be too much for me to stay in the flow, especially the beginning of Crazy Rap. I think I'd settle on Candyman if I had to choose. Okay, enough talk. Now time for total immersion. 1, 2, 3... 1? Okay, in the same sense as you're cutting off distractions, do ALL your work today in the Kubuntu VM without the benefit of Jupyter Notebook and Python 3.5. Not sure if either of them is really a benefit in this case. And don't even think about PyCharm at this point on the VM. You're sort of in a corner, and simply live with it. Perfect the debug mode you started to innovate the other day. You were on a VERY good track. Follow through! Think it out loud. If DEBUG, then only process one site, but process ALL reports for that site. Can we avoid re-executing the SQL every time? Yes, if the required files are already there. We will assume typical behavior for debug mode is to: 1. Run ALL the reports for a single property very similar to the way it is being done already. It creates the new-style keyword CSV file. I'm going to have to confirm that these new tables connect to PostgreSQL on the free tier AWS instance sooner rather than later. Don't leave that for last minute. Have something to report sooner rather than later! That's your motivation. THIS is your motivation: Having a job is important. Having a good job of the sort you can hold while you're near the top of your game is important too. Of course the argument goes that there's nowhere to go but down at that point, but that's not true. If you're at the top of your career, you can ALWAYS go higher. But go higher without ever becoming management. And when it comes to "my thing", I think it's going to be: ## Book Issues - Teach you to get around in a Unix terminal (starting on whatever you're on) - Teach you to use general platform management: VMs, cloud, containers and real hardware, and various software repositories - Teach you editing text files with vim - Teach you managing and protecting your code with git and Github - Teach you basic reference implementation CPython 3.x - Teach you other Python execution contexts, like Jupyter Notebook - Everything else: Robots, IoT, Security, world domination Okay, I figured out how to loop my playlist on iPhone. It's been awhile since I've been on it, but I'm glad now there's the "one true way" to do things now on my phone, like the "Find My Phone" feature and backups. Wow, now I have a fuller appreciation of just how fragmented things are on the Android platform, even when you're on the most mainstream Samsung phones. Okay, I'm working my way systematically through this process. I'm eliminating a few bugs I let get in mostly due to becoming immediately too reliant on Jupyter Notebook. Ahhh, the tyranny of the IDE. Users of PyCharm, Visual Studio, Xcode and the rest are going to expound upon the benefits of IDEs, particularly for intellisense-like auto-completion and object-explorer features. But you need to know how to get around without them as well. Basically TALK through the report process. Create a sort of logging summary of how everything went, as far as file creation, rows-per-file, and file renaming. Make sure the columns balance, such as it were. Now, if I could only do that in my own life! Exercise clear (remarkable) thinking at work, and it will come into place everything else through the forming of good habits in one domain that can carry over to the others. GET YOUR LIFE BACK ON TRACK! You're a special someone, and don't let daily shit get you down. Okay, now we're getting somewhere. Hmmm. Okay, I want to keep ALL the files of the process around. I want to make the ORIGINAL file named special, and each phase with its own keyword, and then the final file exactly as they are today. The filenames should go: 1. report-Property-original.csv 2. report-Property-titles.csv 3. report-Property-serps.csv 4. report-Property.csv Okay, tackle the problems in the order they "get generated". And the first one is to find the first day for which Search Console data is available. This is something I was doing before I refactored. Bring it back. I need to write a function that can return the first day that has data, and call it from the make_date_markers class. Ugh. Okay, think! Going very well. On an aside, I just discovered the censys.io search engine. Wow, a search engine for hackers! On another aside, I LOVE the text effect on this site: https://avicoder.me/2016/07/22/Twitter-Vine-Source-code-dump/ which I just heard about listening to the Risky Business podcast. Okay, I fixed having the correct sort technique on the table join... wow! Reversing the sort order was exactly the thing. Now, the stuffed field... hmmm. Maybe do that last. Okay, I fixed the ambiguity that sometimes pops up with semicolons in a csv file in certain apps, like LibreOffice Calc. Best to just filter semicolons out of csv files. Okay, finish the logic to find the most recent full day available in Search Console. Wow, that's going to require a lot of full report generating (time consuming). Think! Okay, done. Now, let's get rid of all those pesky extra decimals. Looks like I'm importing numpy after all. Couldn't do it natively with DataFrame methods -- ugh. Anyway, 4 decimals looks right everywhere. Next? Thu Sep 1 15:23:16 EDT 2016 I got most of the important stuff for these reports done today, but there's that little issue of the field stuffing still. Hmmm. -------------------------------------------------------------------------------- ## Wed Aug 31 10:55:26 EDT 2016 ### Lost Steam, Must Recharge The Battery Monday to Wednesday with no journal entry -- either a sign that I knew exactly what to do, or a sign I just couldn't get my head into the game yesterday. I think a little of the later. However, the things distracting me are right at the hart of the issues surrounding the industry today and in the near-term future. Everyone's tying to grab eyeballs -- but that's all backwards. Everyone should be trying to do the most valuable and contributing-to-the-world sort of thing they can do, and allow the world to find it by virtue of its sheer awesomeness. In such cases, all it takes is a little bit of content marketing to "release" the building-potential. But the potential needs to be built first, and the content entirely meritorious and deserving of the attention you're tying to focus on it. Otherwise, you come off like a bozo going "Hey, look at me! Look at me!" Don't be that bozo. Genuinely do interesting stuff. Get the timing right. I have to be proactive, and get out ahead of things. I still haven't found my REALLY FAVORITE tools in the SEO industry yet. Wed Aug 31 14:39:10 EDT 2016 Forgot to call Adi during lunch again. Set a couple of alarms for that (lunch and night time). This means that the rest of the time has to be spent that much more effectively. Like right now, for instance. How am I making sure I put in a good solid work-day? What are the to-do lists I'm working from? -------------------------------------------------------------------------------- ## Mon Aug 29 11:04:00 EDT 2016 ### Monday Okay, getting ready for today / this week. I know I have some git entries at home to still push out, so that'll be a merge from home. That keyboard at home is really freaky. What a birthday, too. I really love spending my time with Adi. Her and my interests are so totally aligned. But I do have to dramatically clean up on the home front. I should really get into the mental state at work preparing myself for when I get home -- hitting the ground running, and hitting it hard. Mon Aug 29 17:46:16 EDT 2016 Lots of talk today on the Google announcement about penalizing modal interstitials starting January 10, 2017. -------------------------------------------------------------------------------- ## Sat Aug 27 16:47:07 EDT 2016 ### Puttering With Rainbow Ripples on my Razer BlackWidow Chroma Clicky Keyboard This is turning out to be one of the most interesting and best "textured" days I've had in a good, long while. I'm looking forward to my Twelve South BookBook wallet getting distressed and broken in. I already dropped my phone fiddling with the fob to unlock a door until I realized I didn't even need to open my wallet to use the fob. Then, I discovered my lost keys. Good feeling. The time has come, the Walrus said, to get your ass organized. Use your time when you have it, and that means when you're with Adi, to get your dumb ass organized. One of your (and by your, I mean my) problems is that you feel too much like you "made it", no matter how hand-to-mouth you're actually living now, money-wise. Adi still managed to steal my chair in this room (not my computer), so I'll go get a kitchen chair. Wow, this is really an awesome experience. No matter it contributes to getting into the flowzone. Yeah, is it "the zone" or is it the "flow" or is it the groove? I hereby label it for my own use the flowzone to use a bit of redundency to eliminate any question of what it is I'm talking about. These keyboards contribute to getting into the flowzone by constantly rewearding the animal part of your brain while you type. Sat Aug 27 18:16:19 EDT 2016 Things still going pretty well. It's almost spiritual to have the keyboard rainbow ripping in response to typing... and I am actually puttering, doing an 80/20-rule first-pass organizing of the apartment. It's amazing. Whatever room Adi switches to, I find something to do there. I think I'm on the verge of a re-organizing, maybe using Adi's wardrobe for my own use, and then giving her the half-wardrobe, and maybe even the Rubbermaid $70 shelves from Amazon. That would be great for getting her stuff out of the way. I think she needs closable cabenits now more than she needs wardrobe space here at this apartment. I'm inching back to not being ashamed of the way I live. It is my birthday present to myself. The difference between winners and losers is finding a way to find the time, regardless. A lot of that is optimization, born of a higher-level abstracted understanding of experiences and improved general set of reusable solution patterns being at your disposal. -------------------------------------------------------------------------------- ## Sat Aug 27 13:55:58 EDT 2016 ### Happy 46th Birthday To Me - Razer BlackWidow Chroma Clicky Keyboard Extreme Happy Birthday to me! This is my first time typing into my journal on my new Razer BlackWidow Chroma Clicky keyboard. It all started when I realized I wanted illuminated keys on my keyboard at night when I typed, because I wanted to do more of that "vampire time" type concentration-time that by boss talks about. Problem is, I'm exhausted and not motivated to do that sort of thing when I get home at the end of the day, and knowing that it's so much about mental attitude that comes about by shaping one's own environment, and tricks to trigger off different happy chemical neurotransmitters. It is so irrationally tied to environment and stimuli. And so, the silly Razer keyboard. It's one more set of slightly strange and different key arrangement to get used to, but there's big reward. I can't wait to actually figure out the LED programming that helps me most late at night. Right now, I'm typing on a ripply rainbow. What I want for my birthday is to putter around the house all day and gradually get my place into not-objectionable status. That's merely a bunch of 80/20-rule sweeps, while having peek energy and without Adi pulling me away as soon as I start to do something that really starts to help. Wow, this is an awesome feeling. Being on Apple desktop hardware really isn't bad. They got to the superior UI like good full-screen, just like usual. Proprietary isn't always that bad if they're working their asses off to give you the best product imaginable, and sometimes do in fact know what's best for all of us. Someone at Apple remembers what it used to be like being on the Amiga computer, and is trying to deliver a semblance of that experience to use through OS X -- soon to be once again, MacOS. Good! You can feel the subtle compass-less cluelessness of Android and Windows, just aping Apple at every turn, often getting the so-important tiny details all wrong. Shame on me for throwing 3 years at Android in the vain hope that it was just the way I saw it, and not really inherent inferiority, due to the chronic pandertitis that plagues everyone who things they can out-Apple Apple. Maybe they should focus on out-Amiga'ing Apple instead, so they can generally get out ahead of where Apple seems to constantly be trying to go. Even to this day, to switch between full-screen apps, there's no hard-and-fast equivalent to Amiga+N. Some might say Alt-Tab, but that's only task-switching on Windows. Some might say the multi-finger swoosh on an Apple Trackpad, but that's only on Apples with trackpads. I see that Apple Key + arrows is doing it for me right now. Maybe I'll try to commit that to muscle memory for when I'm using this keyboard, because moving my hand away from the keyboard really does throw off the groove. It's amazing thinking how much lost productivity for advanced people came with gained productivity for beginners, thanks to the mouse. Okay, Adi followed me in here of course and demanded Gravity Fulls on the very system I'm typing on with my new keyboard, and I held firm with a "No" and that part of my birthday present to myself is to be able to type on this keyboard, and that I could set up another screen in this room for her to watch on, and she was fine with that. I set her up on the Microsoft Surface Pro 4, which now has a smashed corner, but which doesn't interfere with watching the screen okay. I may try to fix it myself at some point, or maybe just see if I can just pay to have somebody else fix it, or better yet, just be happy with it for a few years until I'm ready to buy something newer and better -- yes, that's probably the most likely. But I DO want to start doing some more do-it-yourself broken screen fixing, like of old iPhones and stuff. Okay, but back to the order of business for today -- puttering and organizing... ALL DAY! Wow, it's been like 10+ years. And in the end, days like this are what make all the difference. I think getting through to Adi is helping too. -------------------------------------------------------------------------------- ## Fri Aug 26 09:45:17 EDT 2016 ### Today It's definitely very odd and weirdly satisfying to have my phone as now an integral part of my wallet, with my new Twelve South BookBook wallet. It's going to take some getting used to, but very worth it. Adi will have some adjustment to make, me not being on Android anymore. I like Apple for my main phone, and I caused myself unnecessary pain and anxiety for long enough forcing myself to get familiar with (and used to) Android. Game over. Fri Aug 26 12:02:34 EDT 2016 I'm having a great deal of difficulty "finding the love" in my current project, mostly because of the alternating between Python 2.7 in a Kubuntu VirtualBox and Python 3.5 in Jupyter Notebook... well, anywhere else (Mac, Windows, etc.) Fri Aug 26 16:30:25 EDT 2016 Okay, heading out now. Did a big-ish deliverable for a stakeholder today separate from the report update. I think I need to get a bit of distance from the report for some clarity. The debug mode that I'm adding now is key. -------------------------------------------------------------------------------- ## Thu Aug 25 13:22:17 EDT 2016 ### Have to convert Python List to Dict with Win/Loss Scenarios Ugh! Okay, time disappearing again. Buckle down and FOCUS! Love what you're doing. I have a more usable DEBUG mode in the reports now which lets me set a whole set of values to keep file collisions from happening in the non-debug file locations. So now I can run this thing freely over and over, but each time takes a VERY log time, especially if I'm generating all the files from SQL every time. Okay, I can see where my logic has gone terribly awry. I need to create make the keyword_dict that uses URLs as key ACTUALLY grab the top-performers and not the last encountered in the loop, as I currently am. Sheesh! I totally knew that issue was coming, and I forgot in my initial implementation attempt--stupid! Maybe I just reverse the sort order of the original list before the loop! -------------------------------------------------------------------------------- ## Thu Aug 25 09:22:09 EDT 2016 ### Efficiency Lost Handle today much more efficiently. I was terribly inefficient on this last report update. Allow yourself a few thoughts before totally throwing yourself into focus-mode. I am falling behind on several fronts in life, but simultaneously pulling ahead in others. The trick now is to pull ahead on all fronts, and that's a bit tricky given how much I'm "giving my all" on both the professional (SEO, Python, 2+ hour commute every day etc.) and Adi/Daddy front (weekends, Catskills, nightly calls) that I basically have no time or energy left over for the rest of the essentials. But I'm taking steps to fix that now, going into my 46th birthday. Not a lot of professional years left, and I'm getting into those years that when I think back, I KNOW I remember with my dad. These were the most impression-making years I believe in forming that higher-level abstraction-thinking adult. This is where & when those adult-thinking abstractions are formed, and I'm going to be the coolest dad ever -- right as I do right by my employer and earn well and get myself financially responsible and back on track. But first, get a few things to get my head into the right place. There is a feedback loop between environment and mind. Get that feedback loop to be working in the positive direction -- and THAT means a Razer Chroma keyboard for myself! And a few other things (magnets, pocket microscopes, magnetic silly putty, solar cockroaches, buckyball magnets, etc.) -------------------------------------------------------------------------------- ## Wed Aug 24 14:26:09 EDT 2016 ### Paralysis... Un-Paralyze! There's so many utterly terrible way to go about this next step on the report updating that this may be the reason for my seeming paralysis. It seems like it should be easy, but everything as it exists is such a balanced creation, that to go in and start doing stuff is to risk collapse. Wed Aug 24 18:55:00 EDT 2016 I think I've got it. Running a test. Long-running Gonna head home now. -------------------------------------------------------------------------------- ## Wed Aug 24 09:31:12 EDT 2016 ### Switching from Android back to iPhone Love my podcasts. Have to think of more beyond the Python, and even the programming ones. It's also strange being back on the iPhone as my main platform. I think after 3 years with Android, I've drawn out of it most of what's valuable about it. I think it amounts to: - Multitasking - Knox (on Samsung platforms) ...and not much else. All that "freedom" stuff people talk about is patently false. It's amazing that anyone gets anything done on Android. It's subtly distracting and disjointed. Apple's iOS has its problems too, such as sandboxing apps to an inconvenient level, but I see why, and I appreciate the results of it. Okay difficulty focusing comes from difficulty focusing -- NOT environment. After a bunch of jobs now, I see that my focus really amounts to my LOVE for the work I'm doing, and my ability to filter everything else out that's going on, and sincerely just being "transported" to a different place -- and THAT'S not easy. The legitimate complaint is the number of things that jolt us OUT of that strange zone, and not the process of getting into it. - The responsibility for getting INTO the zone is on us. - The blame for jolting us OUT of the zone is on whatever allowed that jolt-vector to exist, be "turned-on" or whatever. And so... and so... yes! I got it. vim is a pathway into the vector, but one that comes equipped with its own warning of how difficult it is to get and stay there. I'm experimenting with music again. Now that I've listened to podcasts and audiobooks for awhile, I'm beginning to condition myself to "just listen" and "not listen" all at the same time -- something I've always had difficulty with. My hyper-literal conscious mind has had quite a difficult time with music all my life -- I think that's why I love the sort of things that are rife with lyrical brilliance, like Disney's Be Prepared. I just have to let the rhythm carry me on occasion. Adapting to different platforms rapidly, and living happily within their confines, is incredibly important. Don't balk at the restrictions. Embrace them, and make them yours. Make other people look in admiration at what you can do, no matter the adverse or strange conditions. Okay what you do now is shape reality moving forward. Learn your lessons from all that stuff you've been reading about the mind and quantum reality. I'm doing it one way as I type in vim, and that's why it's a transitional activity... a catalyst, and the reason that whole 1, 2, 3... 1? thing is so friggin' important. Wed Aug 24 13:28:03 EDT 2016 Still delaying... shit! Have an accumulating set of work to get done, but hit that invisible wall. Time to plow through it... because I'm a fucking human. I can control a lot of aspects of my behavior, and where I can't directly force myself into certain behaviors, I can do life-hacks to compel myself to at least start down the right paths. Small decisions can have huge consequences. -------------------------------------------------------------------------------- ## Tue Aug 23 16:33:34 EDT 2016 ### Productivity Issues Today NOW! 1, 2, 3... 1? I'm gradually losing my ability to focus today. I need a better night sleep than I got last night. I need to think through my challenge and do it cleanly and efficiently. I need to look at the problem anew, knowing that I have this big, wonderful tens-of-thousands-long keyword list to work with. I need to start getting my life into better order on the home-front to better accommodate things on the work front. Consider heading out early enough so that you have some energy still when you get home, and do a big, wonderful 80/20 rule pass over your place. Lock-in on the refreshed and renewed interest in organization and treating things well that this new iPhone 6SE is infusing back into you. It's been a long time since I've had this feeling. iPhone got SO MUCH right, and about all it seems that Android got right is multitasking and Knox, which is not even an Android innovation, but a Samsung one. I will have to unlearn a few Android habits, but it won't be hard. This was a strange 3 years, and the few things I used the Stylus for was totally not worth it -- except maybe insofar as making sure Adi is multi-lingual, such as it were. And she is now. I'll figure out some way to give her what she wants (her Panda games). Maybe I'll make that a car phone. That'll be a good idea. Gotta get that cracked screen fixed on the Note 5... maybe. -------------------------------------------------------------------------------- ## Tue Aug 23 10:04:26 DST 2016 ### Done With Platforms That Subtly Add Stress Wow, it seems like forever later. I took Monday off and spent it with Adi. I didn't do a single journal entry over the weekend, I think. I just switched back to iPhone, and am now on the iPhone 6se, interestingly enough. I just love this small form factor. I will not have to carry aroud 2 phones again, and I'm going to be on the platform which is my real preference (if I'm being honest with myself), anyway. And I just now ordered a leather wallet case for it that holds credit cards and money. Tue Aug 23 14:35:52 DST 2016 Had a good meeting with one of the awesome stakeholders here. Gotta think about the rest of tday. It's already after 2:30 PM and I'm slipping further behing. I believe I need to start exercising discipline. Get myself back into the flow and the zone. Just not... not... not in the correct mindset. Oh, also have some help to offer a department that sits next to me on preparation of content for an upcoming show... okay, touched base with them. Now... now... think! Clarity. Nah, clarity not required. It's the 1, 2, 3... 1? thing again. Okay, for starters, get the hell off of this Windows 10 virtual machine under VMWare Fusion that I'm using merely to test the Ubuntu Bash Shell under Windows. I've tested it. It's great. It's going to make Windows usable without Cygwin. But now, stop using it. It's a distraction. You are constantly replatforming, and its a source of stress, and the pure Apple stuff works just so transparently and without additional cogntative burden. You've switched from Android (Marshmallow) now back to iOS. So now embrace it. -------------------------------------------------------------------------------- ## Fri Aug 19 11:55:08 DST 2016 ### Refactoring SEO Pulse Reports I still have tons of work to do today, and a 1-PM call coming up. Go grab an early lunch. Be fueled-up and ready to hit this thing hard. Light meal and plenty of hydration and caffeine. Okay, got my food. This feels pretty good. Asides from the dissociated spell check file (which I'll fix), this totally could be my day-to-day journaling environment. Okay, focus! 1, 2, 3... 1? Just change the way the dates are being set in the forked copy of the reports repo. Okay, I got the date system ported over to the new way. That was a nice round of cleanup on the report.py file. Next! 1, 2, 3... 1? Ugh, I'm actually going to really clean up the keyword section of the SEO Pulse Report to be the super-set keyword list I need everywhere else. I mean because like, why not? It's the super-list. I just need to keep the database field names identical! Ugh, no! I'm looking at the logic, and I pulled off some really amazing stuff there with joining multiple tables and rearranging columns just-so. Ugh, okay let me think. I would really like to gut that area of the report and rebuild based on the new logic. I mean like, why not? It's the same list (essentially) that I need elsewhere and I have the logic essentially written AND MUCH SIMPLER! So, why not? Gut first is the rule. Don't try to figure out what you WERE doing, but rather just go for the new method. Use it here as the new Keyword table, and then join IT'S results to the other URL tables. Hmmm. Okay, remember these column names: - keyword - positiondy - positionwk - positionmo - position90 - clicks (for what time period?) Okay, I'm hopeful again. If I'm only displaying those few columns for the keywords table, then I can keep those column names exactly the same and in exactly the right order... ohhhh, I may need to execute a whole bunch of foreign table drop and create commands. -------------------------------------------------------------------------------- ## Sat Aug 20 15:14:22 EDT 2016 ### Found A Guide For The Perplexed The law of eventual return. If it was meant to happen for you, it will. It might take awhile, and you have to do more than your part of the work. But if you keep it up, and are intelligent enough about it, you'll probably get there by 50 or 60 years old. And if not, it was fun trying, huh? But it can't always be... tough luck, Einstein. No grand unified field theory for you. But who wouldn't want to be along for that ride... uh, yeah, right okay sorry. Anyhoo... solipsists are ultimately correct (not existentialists, after all) because you can't disprove them, due to that pesky problem of induction. And so, why not go for the gold, in whatever event you enjoy doing? Seems like as good a way as any other to pass your time. So glad I read Iain M. Bank's Culture series. Now, off to the Catskills to meet Adi. Oh, one Audible tape update. So burned out now on anchors and cognitive biases (Thinking Fast & Slow) that I need a dose of... Quantum, A Guide For The Perplexed -------------------------------------------------------------------------------- ## Sat Aug 20 09:15:11 EDT 2016 ### Ending Your Turn Fairly Satisfied Back in vim. I feel really comfortable typing in vim. I wish I had this skill starting from like 10 or 12 years old. Things to keep in mind when trying to take up the new skills that really make all the difference: - Yes, the ways of the new generation often ultimately replace (trump) the old... but... the ways of the old generation are that way for reasons worth knowing (like, how do those robots work and how might we turn them off?) - The ways of the old generation may be so lost by now that even their creators no longer see their connection to the modern world, such as with the creator of the vi text editor, Bill Joy. Efficiency always still gives advantage. - What comes around goes around -- always. It's not like society will have to reboot from total collapse, but it's nice to know how you could. Knowing things down deep helps when things go into flux, as they always do. - Things are so complexly built-up these days, that they have become, as Arthur C. Clarke once said, indistinguishable from magic. In general this is fine for improving human quality-of-life, but must come with a warning. - While inserting extra abstraction-layers is fine and necessary--nature itself favors this trick--that experiential trickery we (humans) have imposed on our own world will too easily become dangerous prisons, the moment we forget. - There's a few really important (fundamental) world-hacks to know. How transmitters and receivers are the same thing is one. The two-slit experiment of quantum mechanics is another. It is always important to know these things. - In life, in general, especially in your times, there is just too much to know. You can't know everything, so get the gist of what the latest in physics and grand unified theories tell us. Then, focus on whatever else you like. - Relays, vacuum tubes and transistors all help produce "digital logic" through a reduction in the actual richness and infinite depth of our physical world, and should not be over-valued as an apex of technological accomplishment. - The world's not digital. Nor is it even binary or bound by tight, confining rules but for a few, like the-speed-of-light. Everything else is made-up human mental constructs and should often be challenged. - Much of this reality-challenging is being done by particle physicists through mind-numbingly boring experiments of smashing atoms with large atom-smashing colliders to see what happens. Though important, it's just one route. - Big world-shifting changes do occur just as a result of thinking. People have these thoughts all the time, but do nothing about it. Some who have done something about it include Copernicus, Galileo, Newton, Plank and others. - If you think hard enough, are well enough educated, and have good communication skills, you can make it possible to blow up the world and end all human life. A man named Leó Szilárd did just that, and showed Einstein. - We survived people having those amazing world-changing thoughts, but just barely. It's like when Sammy clawed you near your eye. Humans are a lot like infants, Sammy like the world, and tail-pulling like chain reactions. - This should not scare us. It should just drive us to be cautious when we invent, and have good friends to look over our work, give us opinions, and chime in at the critical times (as with the Szilárd-Einstein letter to FDR). - Looking at the same things as everyone else, but coming up with different results is one step in the process. Persistence, tenacity, confidence, and exercising good communication skills are most of the other steps. - At this point, you should watch Richard Feynman, particularly his Caltech lecture series. Then, watch The Honeymooners. Now, ask your mother about old-time New Yorkers and your Great Grandpa Bernie. You're a New Yorker. - Occasionally, a step is really-really-really hard. If you find this to be the case, back off the problem and focus on something else (related) for awhile. Sleep on it. Come back. Hard problems often require a new perspective. - Taking a break is never a crime. Life itself seems to swing back and forth rhythmically like a pendulum. We go bizarrely unconscious every night to optimize and index our databases. Get a good night sleep... often. - Our bodies need a bunch of things to operate correctly. One is starchy carb-sugars to burn like slow-burning logs for energy (candy is tinder). The other is protein for growth and repair. Stay hydrated and eat diversely. - As beings, we are layered up, increasingly complex circles around an inner fish at our core, following the 4-limb amphibian design branch on the tree of nature. Rodents, social monkeys, us. "Us" is only a tiny veneer on top. - Most people are lazy and scared of new things. They are following closely in the foot-steps of those who have come before them, sort of running their lives on automatic, not taking full advantage of what it is to be human. - At this point, you should read the SciFi book, Dune. The movies are not good enough. Get the point of the gom jabbar. No matter how much you think you may be in control of yourself, you are only an animal, without crazy-discipline. - Because thoughts flow freely in an unlimited brain, but actions are inhibited by our bodies, circumstance, motivation and the physical world, there is nothing more important than deciding what to do next in the moment. - Many good thoughts that run through your head are momentarily considered and dismissed by the part of your brain that could do something about it. The big differences in life come down to these tiny "right now" moments. - But don't let that thought paralyze you. We don't (can't) live optimized and efficient lives. We are each just one in a long chain of billions and billions of similar humans. Just make good thoughtful decisions on average. - You chain-up these momentary decisions from the time you're aware of the fact that you're kinda-sorta in control of your own life, until the day you die. There's more, but that may be all that we as individuals may ever know of it. - It's all a constant building and refining and pruning and shaping of your own mind. Your brain holds a subjective copy of reality, like a virtual universe. I believe this is what makes humans most different from other animals. - And so, there is an often discussed "inner-world" in literature, stories, and much human thought. I once read one called The Master-Key system. All this self-help crap says the same thing, but it's true. - Imagine new things in your inner world. Don't just reproduce the boring old world as you know it, but experiment with new possibilities, plotting a course through life you think you might like. Take small steps towards it. - All this can give rise to notions of greater purpose, the meaning of life, and other unknowables. Don't go crazy over it. Someday, we may be in a position to know, but for today, just live well by your own standards. - Create your own standards, and dismiss others who try to judge you by their standards (control you). Much of what you think and believe comes from me and your mom, and are our own standards (not yours). Shed imposed limits. - Much of the point of all this stuff is to simultaneously feel good about yourself, your immediate circumstances and conditions around you (the things you can do something about), and to end your turn fairly satisfied. - Google belief systems like existentialism, solipsism and nihilism. Similar human experiences can make one person depressed, another pleasure-seeking, others mean and self-serving, and still others heroes tying to help everyone. - Not everybody has the same view of life as this, and many folks will try to impose their beliefs upon you (similar to what I'm doing here), but WITHOUT telling you to think for yourself. Beware of wolves in sheep's clothing. - It's not always easy to tell one type of person from another at first. This takes practice, and you should strive to surround yourself with people who bring out what you believe to be the best in yourself. - Ultimately, we humans (and most life) are self-contained, self-animating, temporary and reproducing units of energy/matter with lots of sensory equipment giving us empirical evidence of an objective common world. - The rest is up to you, and not up to all the bullies around you playing the exact same game as you. The only difference between you and them is a roll of the dice concerning genetics and initial circumstances in life. - I can't help but feel that despite the self-evident subjectivity of it all (google the problem of induction), the highest callings in life are those that on the whole help others without hurting. But that's subjective too. - The only other thing that I find self-evident in this whole "existing" mess is that the journey is probably the reward. We're beings bound-inside and limited by the very things we're trying to understand, so don't sweat it. - Stay in tune with your rhythms and the energy you're willing to put into things, then take on seemingly compatible challenges that push you a smidge beyond your limits. Rinse and repeat. At least, I find that very satisfying. - There is a beautiful, musical, dancing balancing of attractive and repulsive forces at play amongst the stuff that lumps together to form us and all the stuff around us. And at its most basic, we have very little idea what it is. - Watch Sagan's Cosmos, and take note of the cosmic calender. Also think how long a human life is end-to-end. That's how long ago WWII was. Super villains really did try to take over the world just that long ago. We're very young. - On occasion, there's persuasion that you ought to write insightful With an algorithmic rhythm we all learned from Doctor Geisel Intertwined communication that something something parameter To get across a notion that abiding our pentameter. -------------------------------------------------------------------------------- ## Fri Aug 19 11:37:31 DST 2016 ### My First Github Commit/Push from Bash on Ubuntu on Windows Okay, here I am making my first journal entry on vim-nox from a Windows 10 Ubuntu Bash Shell -- amazing! It wasn't quite smooth getting here, but also it wasn't difficult. It just took a long time, and has all the same issues as settling into any new hardware (plus some extra download, install, reboot steps), but it's working full-screen and quite competently under VMWare Fusion on my old Macbook Air. I have it showing in Courier New with a nice big font, and my .vimrc moved over. However, I can't bring myself to do Dropbox again on a VMWare session after all that drive-shrinking, so I won't be sharing my spellcheck file with this system, which means I'm looking at a lot of distracting maroon words, unless I ":set nospell" at the beginning of all my vim sessions. That's going to get tedious. I already broke off the .vimrc because of no dropbox, so just edit it to change your spellcheck mode default. You can always spellcheck on other machines, which you'll be on plenty. But so far, that's the only downside of this experiemnt, and it has nothing to do with the Windows platform, but rather with my feeble Windows hosting environment (a VMWare Fusion session). And this is my first commit on Bash on Ubuntu on Windows. -------------------------------------------------------------------------------- ## Fri, Aug 19, 2016 10:10:01 AM ### Last Entry From Windows 7 Cygwin Shell Okay, one more journal entry on my Windows 7 machine. Ugh, because I run the VirtualBox cron job on this machine every night, I don't want to mess around with the Windows 10 upgrade to get the Ubuntu BASH shell on my primary working machine -- one less reason to want to use my personal Macbook Air all the time. Sigh, maybe soon. I should bring in some "real hardware" on which to run these reports. Actually, I tried with an ARM device, but that didn't work out too well. Okay, in the middle of making a video showing how to shrink a Windows 10 VMWare Fusion image after a Windows upgrade that plops a 19GB recovery-point onto your drive. Or rather than "plops on", it's more like "leaves a copy" that's de-referenced, but still protected. It's like deallocated bad bytes on a hard drive. But my work today is really a matter of synthesizing my new-style Python 3.5 with heavy leveraging of Jupyter Notebook for code development to my old-style Python 2.7 with heavy leveraging of dedicated server-like Linux instances with just enough to have a conservative but workable development and code execution environment, which currently is a Kubuntu virtual machine under Oracle VirtualBox on my Windows 7 machine. It's a bit of a multi-dimensional schism I've created for myself here, and it's time for me to start re-consolidating my personalities here, and that will for the most part be under: - Any OS - Superb software repo system to compensate for "any OS" (pip, conda, etc.) - Python 3.5 - Jupyter Notebook - vim - git (and usually, Github) - The typical set of Linux/Unix commands in a BASH-like shell - Rapid application-switching between full-screen apps By the above criteria, Windows 7 with Cygwin, VirtuaWin and Anaconda qualifies as a good development system, and indeed I do most of my daily work (with the exception of the journaling) on a system much like that. I use a separate Mac sitting to the side for the daily journal so that I don't even have to switch full-screen apps to jot notes in my journal (type into vim). I think a lot about absolute and relative positions, and the use of things like "indexes" to access different information (experiences) through the same viewport without moving your frame of reference. But sometimes moving your whole frame of reference is exactly what's required. Doing these switches throws you out of the flow or the zone usually one way or the other, so why not make a nice clean zero ambiguity, immediately understood by your "system-1" lizard-brain self? It's way easier to turn to another machine and jot a note on a different keyboard than remembering what keyboard invocation switches you to the next full-screen to the right (for example) and back -- though lacking a whole second system for daily journaling purposes, rapid full-screen-app switching without lifting your hands from the keyboard is the next best thing. Macs accomplish this nicely with full-screen apps, and Windows 10 only just got there with their versions of virtual screens, but probably even more useful full-scree-apps than the now-available "actual" Windows virtual screens. I feel like somebody's trying to make a point with Windows 10's implementation of full-screen modes (apps vs screens). However so far, I prefer Windows 7 with VirtuaWin over Windows 10. -------------------------------------------------------------------------------- ## Thu Aug 18 20:12:28 EDT 2016 ### Cats Not Really Gone Missing (I Hope) Shit, shit, shit, shit. Adi's... as fast as I started typing that, I take it back. There's a medium-dilemma going on, of Adi's cats in Staten Island appearing to be missing, but it's a big house and small cats. The person taking care of them hasn't seen them in days, and found a window open. I got that call fairly early today, but I've been in touch with the person feeding the cats and getting more info. Sounds like there was a 2nd (of 3) cat sighting. A dark blur running across, and that sounds like the cupatrabrah herself... in other words, Dorothy. I think Dorothy's safe. But wow, what a wake-up call. Got my adrenaline going good. It's that snap back into reality that everyone talks about. It's a real feeling, that fight-or-flight response. I guess sometimes we maybe ought to manufacture it happening more often. Is that the high of working out? Is that the benefit of exercise? Hmmm. You can always put off until tomorrow what won't get you thrown in jail for ignoring today. I am paralyzed by just where to start at my apartment. I've put it off long enough, "catching up on my sleep", and "decompressing" long enough. Now, I've got to get my act together. No excuses, and no distractions. I'm reading currently the Thinking Fast and Slow book by Daniel Kahneman. Man, I could have been a scientist like him, if only I hadn't misplaced my passions into Commodore so young. There's other excuses I could use. My parents could have guided me better, but ultimately we all take control of our own fates. The excuses run out. The only question about who ultimately has the most control over you, the system-1 or the system-2, in your brain. Are you actually ever really in control? There are so many thought-illusions out there, wow! Anchoring effects. The role of associative coherence in anchoring... man. -------------------------------------------------------------------------------- ## Thu, Aug 18, 2016 9:41:04 AM ### There's another name for Pythonista: Dunderhead! According to Thinking Fast and Slow, which I am currently reading, the System 2-type thinking that handles exceptions from the System 1-type brain, is both hard and requires discipline, and is the source of great advantage in certain situations where working on automatic will get you in trouble. While you can't really be "always on", in that it uses up all your energy fast, research does support that infusions of sugar-energy can put you back on track. But I'm sure we're trying to reach an optimal and context-appropriate state to balance our energy burn-rate (in part, adjusted knowing our accessibility to new energy-sources and our time-to-digest and make it available), and the size of the tasks we take on, considering their cognitive stress level, work involved, and often-times high risk of failure (assessment that it will be a bad investment) -- especially when considering, the lazy-way so often works. We're built-up around the presumption and systems accommodating the fact that things we experience once can be useful again in the future, and should weigh towards forever-being-refined automatic rules -- the System 1, such as it is called. This is why I write so much here. It is becoming clear to me, why "vim mode" hits a fundamental in the human experience and state of being. It's a vocabulary that comes natural, and dove-tails perfectly with other vocabularies we have, such as the English (or other) language; both written and spoken, and Python; mostly written, but actually spoken by in increasing number of dunderheads. Isn't dunderhead the ultimate in self-deprecation? Isn't that the ultimate in expressing what it is to be in the Python community, and the inextricable tying together of the new learner with the old hat? Hmmm, nope, it's registered and on sale for $3,999. Nope, no thank-you. Dunderheadin.it? For an extra $19.99 per year over the registration fee, I could do it. But again, nope. I just gave up MikeLevin.me, as that sort of silliness just isn't worth it -- except, in the case of MikeLev.in, which took over Mike-Levin.com for me, from back in the day when I was stupid and let my registration expire. My continuous domain history could have gone back to 1998, or whenever it was that I went by that. Sighhhh, okay, back to the 1, 2, 3... 1? Today is your day to nail this report modification project. Nail it. Hit this thing home. Get ahead of your work back-log. That's where your extra time with Adi is going to come from. Knock it out of the park... repeatedly! Become the Babe Ruth of SEO Data Science... for better or for worse, haha! We are all flawed individuals, and sometimes it is most fun when exceptionalism surfaces in the most flawed ones; lots of personality! -------------------------------------------------------------------------------- ## Thu, Aug 18, 2016 9:35:20 AM ### How Useful Will Python Be After the Singularity? 1, 2, 3... 1? This is our perpetual state while conscious. We're stuck in a loop called "this moment". We're riding the read/write head of some 4-dimensional being who dropped their phonograph needle into our branes... or is that brains... or both... or neither. That just about covers how well we understand it all. For my money, Kurt Vonnegut got it right in Slaughterhouse 5. We're all exhibits in a zoo... portholes and infinitely improbable instances of what this universe can come up with, for better or for worse. God doesn't roll dice with the universe, but that is precisely what every human being is, among other lumpy units of matter/energy. Mattergy? Yup, it's the complete state of mattergy that would have to be observable and copy-able for Star Trek style transporters to really be possible. And the moral implications... Oy! Again, for my money I'd bet on wormhole-style portals being more viable and less of a disgusting mess. Avoid copying. We keep track of things often through references to instances of our images of the original thing that's files away somewhere on our drives, in our databases or in our brains. Our computer tech is a product of our own brains, after all. Why should we not be patterning them, consciously or not, after ourselves? References to original copies is efficient, but is not the whole story. Our sensory equipment, like our eyes, noses, skin, tongues and ears are scanners and the real-world application program interface (API) to our computing control center that deals with various autonomous systems, the sexiest of which is our thinking minds. But here, a lot of the similarities between our biology and our commonly constructed machines breaks down -- though we, with a shot in the arm by DARPA challenges, are trying to address that. Of how much use will it be to program Python, once a running program becomes smart enough to start improving its own code and append new resources to itself, indefinitely? What that moment is, of course, is new life -- and the singularity point of which Ray Kurzweil and Sarah Conner both speak. But it's unlikely to be some single freak instance in a lab. It's more likely going to be a Github project involving scikit-learn, tensorflow or their descendants running on Python 5 or something. So yeah, learning and knowing how to program Python will likely continue to be important. -------------------------------------------------------------------------------- ## Thu, Aug 18, 2016 8:56:03 AM ### Commit, Push, Paste, Commit, Push... And the Windows 10 Anniversary Bloat At work early, and back journaling using my Dell ThinkPad on Windows 7 under Cygwin on a VirtuaWin full-screen terminal window with a huge Courier New font, so that 80-column fills 2/3 of my wide-screen monitor. It's really easy on my increasingly feeble eyes. Reading Thinking Fast & Slow. I'm listening to the video I made on my phone on the walk to the subway station this morning. It's interesting switching between medias of expression, from videos on YouTube, to SimpleNote text notes on my phone, to typing into vim on one of various laptops and desktop Macs and PCs that I use. Hmmm, really want to get that Linux-first machine (no Windows/Mac VM host) into the picture. Resurrect my Commodore 64x? Very symbolic. Wow, I do have a vivid memory when I want to -- the C64x early morning debacle. Dropping convenient-lie F-bombs, in order to have the automatic and default winning position in any fight coming afterwards, or else I would be forever labelled as (and looking back to this instance as) the bad dad. Well, I reject that reality and replace it with my own, in which such tactics are met with a fizzle-inducing blank stare -- or not even the courtesy of answering the phone (or whatever communication method being used). Yup... yup. Had to delete Kubuntu 64 from my 2011 Macbook Air laptop's install of VMWare Fusion, because the Windows 10 virtual machine is demanding so much of the machine's hard drive space. I can't believe what a hog Windows 10 really is / must be. I had to leave the install going overnight just to get the Windows 10 Anniversary Edition installed, and it froze up in memory suspension mode overnight, so I'm still doing the install now, which is alternating between that blue screen with the spinning dot-wheel, whatever that's supposed to be, and a black screen with a simple Windows logo in the middle, and an entirely black screen that forces me to confirm that the whole thing didn't crash -- from which it returns to the blue screen with a new percentage-complete number being reported for Working on Updates. Sighhhh, the better Microsoft gets, it's still Microsoft trying to satisfy every last edge case of heterogeneous hardware -- and we all suffer. I'm making a video as I go... hahaha. Make sure you get the updated reports (at least, their data sources) done today. Ahhhh... 1, 2, 3... don't forget to paste in your writing from your subway ride. I really should find a way to start marking up the stuff that I want to publish as stories. Well, I could certainly just make it it's own journal entry instead of just pasting it below. And so, commit, push, paste, commit, push... -------------------------------------------------------------------------------- ## Wed Aug 17 10:53:11 EDT 2016 ### In The Melody of Fiddler On The Roof: Transsssition! Cut! I'm not going to stick to one journal entry per day. It's all about thought-transition. Forcing a thought-transition event is sometimes important in thinking about something fresh. Okay, my strategy has got to be to just program this as a Python 3, SEO Notebook integrated "thing"... a separate product, if you will. For pulling this list ALMOST IS the finished product. What CAN'T you tell from searching and sorting this list? WHAT LIST? That's precisely why I wanted to break this out into Python 3 and SEO Notebook. WHAT list is actually a very interesting question. I'm thinking this may be my first utilization of Pandas and DataFrames in an SEO Notebook project. SEONB itself doesn't need Pandas for the basic pipulation process, however... this preprocessing stuff can significantly benefit from Panda -- especially doing SQL-like data manipulations without SQL. -------------------------------------------------------------------------------- ## Wed, Aug 17, 2016 10:28:59 AM ### Tending Your Own Career Like a Business (even if you're a clock-puncher) Okay, I have my Mac back and am charging it now. Meanwhile, start my journal here. Look at the day. I think I'm going to bow out of the tech sprint meeting this morning as well. I got to work a bit late AND let myself dive into a couple of discussions this morning. I'm listening to Talk Python To Me #71 about Soft Skills, and Peter Drucker style concepts (though, they don't mention him so far). We are each our own business, and people don't think they need enough skills to treat themselves as their own little business, even if they're clock-punching employees. People skills are more important than increasingly commoditized programming skills. It's not just about coding. It's about communicating, and making the team work better. Most of your time is not spent writing code. It's spent writing emails, talking to co-workers, planning next steps. If you're not good at those, you're lacking in the majority of what you do -- however, I'm a bit different. My "people skills" are often expressed in the thought-vomit I exercise here. I'm processing my thoughts and ideas, is what I'm doing. It's a lot like talking to myself. I do write my weekly reports for my boss, and that's really been something that helps -- and even though I've TRIED to do this in the past in other situations, it's either been un-asked-for, or it's been annoying over-detailed time-tracking systems (for sox-compliance, at agencies). My weely report this week is due this Friday. It makes my work more highly visible. Have notes. Not just my own personal notes (this), but also my actual weekly reports that get sent out officially to my boss. Think like a business. What does my manager need? I think I had a talk with him this morning that LETS ME KNOW what he needs (whether he knows it or not) from me. It's the portability of processes via Jupyter Notebook... wow! How clear. Such moments of clarity in one's career I think are quite rare. It's a real shift in thinking for me. Less vim and more Jupyter Notebook. Okay, get today's journal-writing off the Windows laptop... okay, back on the Macbook Air. 1, 2, 3... 1? -------------------------------------------------------------------------------- ## Tue, Aug 16, 2016 12:13:20 PM ### ShelvItAll comes and ShelvItAll goes Okay, the trick here is that the things I code in shelvitall can work totally standalone, in other capacities than SEO Notebook integrations. I will be venturing into object oriented-land through the traditional Simula-inspired (or maybe Modula-3?) class API... ugh! Avoided for very many years, self, self, self. But now is time. Tue, Aug 16, 2016 4:36:48 PM Nope, changed my mind. No reason to go OO on this project. Went into it a little bit, and felt OO working against me and not for me, and so back to classical functions and procedural programming. Also, I feel the elegance of SEO Notebook slipping away as I introduce shelvitall... ugh. Okay, delete it and rip it out of the repository. Leave the pervasive_data object in location, and use the pipulate.ipynb file that. That's what it was starting to look like, anyway. -------------------------------------------------------------------------------- ## Tue, Aug 16, 2016 10:03:49 AM ### Adding persistent data module shoveitall to SEO Notebook Okay, I'm doing my journaling today on my Windows laptop. It's screen-swapping hell, today I guess. But I forgot to bring in my Mac charger, and don't want to go on a scavenger hunt. Maybe just hit up eBay for another (and Amazon for a bunch of regular stuff I need again). Don't forget about yourself -- although, you MUST be more productive when you're at home. It's almost like a feeling of helplessness that sets in at home. The trick today is to totally avoid distraction, and to get this latest work done. But just as important as the "get it done" mentality, is simply to make it interesting and loveworthy today. I am falling in love with both Jupyter Notebook AND the process of moving to Python 3.5. To simply execute this project under vim in Python 2.7 on a Kubuntu VM that doesn't have Anaconda installed... well, it's motivation-busting. Solve that. Okay, first (and last!) distraction. Couldn't resist my curiosity about SymPy, and I am soooo taking up Algebra again (in advance of having to help teach Adi) BECAUSE Python makes it so easy. Here's the video that sold me: https://www.youtube.com/watch?v=cvHyaE_bs8s I now want to make my own video of this. This guy is actually pretty good, but my series is a tour de force, producing videos every few days, covering nearly every topic. But don't force it. Be genuine. Let the wind fill under your wings and start to lift you over as long as it takes to lift. I'm a long runway type person. Give it a few years now. Do a few more interesting dot-connecting projects. Start referring people cross-channel in all the content in all the channels you publish on. Go find your audience, corral and rally them in. Do that mailing-list shit -- finally! You were so on top of all this at Scala, then just let it go when you stopped earning commission on the sales you created. Now, it's time to break-down and tackle this project. Do it the most modern way you WANT to do it, which is also easily transposable BACK to the Python 2.7 / vim / cron world that I live in back over on the Kubuntu virtual machine. So, do your 1, 2, 3 breakdown... 1? In my heart-of-hearts, I want to incrementally and cautiously advance the SEO Notebook project. I ACTUALLY want to switch over to SEO Lab, because I feel the "big payoff" of a systematic input/output widget system to incorporate JavaScript-driven browser interactivity just on the other side of one alpha-software install away from achieve-ability. Not that I'm going to do it, but when the time comes: - https://github.com/jupyter/jupyterla://github.com/jupyter/jupyterlab Okay, I'm convinced I DO NOT really want to switch to Jupyter Labs right away. It just confuses matters. And almost everything I do in Jupyter Notebook will be applicable to Labs, so long as I plan my APIs correctly, so I can rewire I/O rather easily -- no prob. Get on with 1, 2, 3... 1? Well in this light, step #1 really must be figuring out how to "wedge in" special jobs into SEO Notebook. Tasks such as the one I'm about to perform cannot possibly be external to SEO Notebook. Sure, we're going to generate 15K+ rows worth of data, but I could just drop it into a CSV file. I'm essentially refining the Pipulate convention. Conceptually, what I'm doing is a sort of pre-processing job that creates the rows in the first place, for subsequent pipulation. The fact that the resulting data from the preprocessing might end up in a csv file versus a Google Sheet is of no consequence. It should be a targetable thing. Hmmmm. I think we're talking about another sub-module of SEO Notebook, just like the way there's a Pipulate and GoodSheet module, I think there should be a... hmmm... let's see... I guess it must be... What we're talking about is having data on the back-end of arbitrary size and shape, with which SEO Notebook and Pipulate functions can interact with freely and easily -- very little overhead of external systems, and the easy ability to port data back and forth between Python 2.7 and 3.5 (for example) and make the data persist between executing instances of Python. In other words, we're talking about a Library of arbitrary Python objects... shall we call them Dictionaries? And so... I think we're talking about Dictionaries (generally speaking) in Shelves in Libraries. It's a DataLibrary. Inside the DataLibrary, you can find whatever help is necessary to grab-out and put-into and discover-the-nature-of whatever data objects you have deemed important for longer-term (longer than that one in-memory session of Python) storage. It's essentially merely USING all the pickling and shelving facility that's so conveniently and courteously built into Python. It will apply everywhere in SEO Notebook, providing an important missing (and long-danced-around) piece of the Pipulate puzzle. "Next to" Pipulate and SEONotebook and GoodSheet lives another module named... it's got to belong to this set; what will look good? - seonotebook - pipulate - goodsheet - shelvitall Of course! Shelve It All! ShelvItAll. shelvitall. It's got a wonderful texture to it, and is dramatically different, yet somehow belongs with all the rest of the modules. Go ahead and make that friggn' file to sort of bank it and homestead in that noosphere! I've noticed dip has stopped thumbs-downing my videos. He's either on vacation (from what?) or has recognized the futility of defining himself by how he stalks and thumbs-down me, simultaneously trying to ape me on almost every -- thoroughly unpredictable (haha!) -- avenue. He didn't see Anaconda / Jupyter Notebook coming, I'm sure. And I already have staked my claim on Jupyter Labs as the platform for product development that inherently teaches you programming and creating customizations, by the very highly documented and accessible and internals-exposed nature of the product -- insofar as internals are the content of the code you're running, and not the code that makes Jupyter Lab itself work. Important distinction. It's still just a glorified Python virtual machine rack with a really good web browser user interface. Or, should I go with shovitall? shelvitall? shovitall in a Google Verbatim search comes up with 19 unique pages, while shelvitall comes up with only 1, and it's an OCR text extraction: https://www.newspapers.com/newspage/288207/ of an advertisement from June 13, 1956. This is about as close to a made up word as I need to get for this purpose. shelvitall, it is! -------------------------------------------------------------------------------- ## Mon Aug 15 14:14:13 EDT 2016 ### Focus! I now need to zero in on this next round of report updates, systematically and diligently. The closest thing to this is what I did for the Menu project not long ago, when I went several times more than 5000 keywords deep in the Google Search Console results. I would like THAT capability just to be an easy-to-accomplish item with SEO Notebook. It is after all one of the most obviously necessary tasks for an SEO. It all starts out with a sort of data pull that exists separately from the greater report project. My Mac is running low on power I need for an upcoming meeting. So, commit and switch over to the PC for the rest of this project's documenting. Okay, I'm over on the PC now, even for this journaling stuff. I have a meeting coming up in about 5 minutes. Okay, it's over, and even been to the bank to deposit paycheck. But now... but now... how do I get this far along quickly? Go look at the actual report logic. It's going to be all about dipping in and out of this virtual machine. I want to be very careful, because report.py is created for Python 2.7. I wonder if I could get it running on 3.5... on Windows. That would get it into my main Jupyter Notebook... and THAT could be very useful with the reports. I could see what I could do with this report code when moved over to my current preferred environment. And that way, I wouldn't feel so shitty about a refactoring like this. Hmmm. Ugh! Yuck! Okay, Uncle! As usual... gotta go back into the virtual machine and just suck it up and do the work there, using Python 2.7. THIS is why so many people are stuck on 2.7. No matter how you slice it, it's work porting things to 3.x, and it's a whole round of little changes -- not merely a few tweaks to get the code running again. Okay, I'm stripping out all the Google Docs stuff from reports.py, and then I want to nearly exactly reproduce the logic that I did in the Menu project, but way up-front in report.py, and pop it all into some sort of global dictionary object that I can join to in later steps. I need to merely spin through each table once, adding a new column for each URL... is that right? Each URL is going to PROBABLY have multiple entries from the new dictionary I'm looking up. Hmmmm. What becomes the key in this dictionary? It MUST be URLs. But is that a key, really? Do I want to make multiple entries on a URL column, because that URL occurs multiple times in the Search Console dictionary? That MAY be the case. Ugh. Okay, this is NOT a complete refactoring of report.py. No, no, no! It's just adding some SERP data to existing URLs. Otherwise, I have to add a ton of rows... "expanding" the rows. Think! -------------------------------------------------------------------------------- ## Mon Aug 15 10:10:03 EDT 2016 ### Monday Morning Oh, it's Monday morning. I have to get my report to my boss, ASAP. Think through last week. One BIG thing to report. Various small things. Push forward harder. Things should be getting easier for you now. Make sure that they indeed are. "Bottle it". First order of business, I detected an anomaly in a reporting system, so I got out the necessary emails to get it on people's radars today. Now, hop to your weekly report! Okay, done. Only barely 11:00 AM... good. Remember, I don't have my Mac charger cable today, so be aware of your battery life on this machine. Commit and push often, so that it's no biggie when the battery goes dry, and then bring your charger in tomorrow. -------------------------------------------------------------------------------- ## Mon Aug 15 10:05:25 EDT 2016 ### Listing Some of My Life's Learnings It's hard to imagine that after such an entry as below, my commute actually can sustain that momentum, as I continue capturing such thoughts on both video (publishing as I type on YouTube) and in notes tapped into my phone, which auto-syncs to my desktop at work, which is where I'm typing into this distributed revision controlled textfile now from. My typed-in notes from the subway ride is this: Continue your campaign of conducting each day like performance art. Keep pushing out videos that draw a picture of my worldview and passions. Don't pose or intentionally copy or ape anyone else's work. Do your own thing. You can happily riff off of somebody else's work, making a great example. Always try to be improving your own personal state of the art. Be proud of it. Your greatest work is built on the work of others. We are all deeply interrelated. Pride is not a sin; it's part of the type of loving yourself and loving life that helps. And self-love and pride in ones one work therefore is love in others, like it or not. So, loners should open themselves to the communities they backed into. Get over your inner self that rejects interaction, steering towards isolation. We are social animals and need that social interaction to not slowly go crazy. We'll always be the animals evolved to deal with the world at this human scale. Death means that a new generation will always take over and have their chance. -------------------------------------------------------------------------------- ## Mon Aug 15 08:05:48 EDT 2016 ### A Few Thoughts About Reality Human conciousness and free will is the icing on top of the physical laws of existence. We are probably a fuzzy fractal edge case -- a freak reality hack. As such incredibly improbable unit-blobs of matter, we are each being given a chance to enjoy, bellyache-about, ponder, or otherwise experience reality. It doesn't take many thought experiments to run up against concepts that challenge the imagination, like how big is the universe and how small can particles be. Not everyone goes down these thought-experiment pathways for long, because it rapidly becomes uninteresting and a lower priority to just staying alive. But for those who do not relent with these questions, some very small number of individuals just may make a discovery that changes everything forever for everyone in some actual meaningful way. The accomplishments of people like Einstein, Planck, Feynman and others push forward the boundaries of "what is real" in a way that rewires our thinking about what reality actually is. Everything appears to be in some sort of mutual orbit around something else, with the more massive body generally being the center, and the speed of rotation increasing the closer the smaller body gets to the larger. And many things that we once thought of as particles also appear to have wave properties. Figuring it all out will have much to do with resolving this particle/wave duality in so many things. Reality itself appears to be some sort of all-inclusive system, of which we as individual human beings appear to be a part of. What I used to call existentialism, I'm coming to realize is actually solipsism. Solipsists can't easily be proven wrong, but probably are anyway. -------------------------------------------------------------------------------- ## Mon Aug 15 07:14:38 EDT 2016 ### Structure Your Days Better I basically ended up sleeping all my free time away, and watching Richard Feynman on YouTube. But then, I guess I needed the sleep. I'm still on a sort of recovery rebound, with the first-order healing having to do with sleeping and mental conditioning. But I have to start doing my physical environment conditioning. I have one check to write and deliver this morning, but besides that, I'm still in pretty good shape... today. But it won't stay that way. Get more on top of things. Structure your days better. -------------------------------------------------------------------------------- ## Sun Aug 14 11:14:30 EDT 2016 ### Richard Feynman -- Listen To Him More Often Just did another auto merge. Will sort out what should be where later. Today is about using today. It's already after 11:00 AM, but I needed that sort of lazy wake-up. Used to be my regular thing in my youth, and now get a morning like that every 2 or 3 years. I'm shortchanged on many things in life these days, but that's part of the exchange of becoming a father. But I don't want to be quite so shortchanged anymore. This is one of those days that can't be taken away from me now with surprise change of plans that consistently screw-over all my attempts to dig myself out of my backlog on the home-front. Well, I've got today for that now, if I can even keep on-track a little bit. 80/20 rule, and commitment and consistency. Go get some energy inside of you. Slow, easy burn with an initial pop. Pepsi, then ramen noodles. Sun Aug 14 12:23:56 EDT 2016 Puttering is okay, so long as there's positive forward movement in your life as a result of puttering. Puttering is like that deep breath at the beginning of a larger task. It's like taking stock... ugh! But more immediately, I need to pivot into the second half of this project, which is making it a surf-able hierarchy. This is where I get to the 1, 2, 3... 1? approach. Ugh. 1, 2, 3... 1? Done. Delivered, and BAM! Off to the Catskills. -------------------------------------------------------------------------------- ## Wed Aug 10 09:03:59 EDT 2016 ### BAM! Now Pipulate Functions use Python Decorators Wow, okay. Conquered Python decorators, finally. Wow, the improved beauty of the Pipulate functions is going to be very impressive. Can't wait to dig into these things now. I have something impressive to show, and the Jupyter Notebook embedded markdown is going to be where the context-sensitive documentation is going to reside. Greetings all! I figured out how to conquer the world's proper tech education deficiency, picking up where the Raspberry Pi half-heartedly left off. The Pi in Raspberry Pi is for Python, after all, and I think they allowed it to become something much bigger by not forcing it to be Python-exclusive. So, they changed the name -- at least, in spirit. Familiar story. Same as IPython Notebook. Now, it's Jupyter Notebook -- all in the name of removing the Python name. Aside from being objectively the best programming language to take-up for a life-long love for coding (a decision even the #1 LISP Ludite Club (MIT)) came to acknowledge and refactor an already legendary intro-to-compsi course. Yup... Python won. But we can't call it that. Decorators are yet another way to back-into the advantages of Object Oriented design. I HATE defining classes and methods and init classes and callables. Ugh! How could anyone actually WANT to think like that. No, no, I get it. It made possible huge advances everywhere, and is today inextricable from compsci reality. But the traditional OO APIs suck. Mikey likey file-based Python modules. Creating a filename.py is the same as creating a Class definition in more traditional OO languages. Every variable set inside modulex.py can be addressed as modulex.y, such as it were. This is the same as what we call properties or attributes in other languages and traditional OO interfaces. Python namespaces; what a wonderful way to back-into OO design patterns. Python decorators are yet another way to back into OO design. It took me a good long while before I groked them. I first really encountered them in the Python Flask Web microframework. Flask is the rough equivalent to turning Python into a PHP-like template-based language. However, instead of writing individual files that map to individual pages, you write request-handlers. And to all appearances, request-handlers (or routes in Flask terminology) are decorated functions. The decorations in flask look like: @app.route("/") def hello(): returnrender_template('hello.html', key=val) This is basically saying: Any request for the website's default page will result in showing the hello.html template sitting somewhere, fed whatever key/value pair dictionaries I want to make available during the rendering phase for some dynamic page components. Yet, the hello function itself knows nothing about rendering a page. It's magical. Not so magical. Inside the instance named app of the Flask Class that is created somewhere else in the app is a module named route. The route module takes parameters that stand for the path portion of http requests that are coming in. That route function was specially written to act as a wrapper to other simpler functions that only need deal with doing a bit of stuff and handing operations onto the templating system. Huh? Well, if this is tough to absorb, just consider that all languages have syntaxes that we take for granted after using them for awhile. Not understanding their internals is not an impediment to using them. Case in point, I bet only a fraction of the people using Python Flask have any clue how this decorator magic enables the framework. Isn't it enough that it works and it beautifies it? Nope. Flask's routing is just saying this: def hello(): returnrender_template('hello.html', key=val) app.route(path="/", func=hello) See, it's that simple! I may have the attribute names wrong (didn't bother to look), but I'm hoping you get the point. The hello function is actually being fed INTO the route function (or method) of the app instance of the Flask class (or module). Decorators are just slightly cleaner shorthand notation. Less parenthesis, basically. Mentally, the focus is on what "hello" is doing, so why not give it all our focus, and only just lightly "decorate" hello with the pesky detail of the fact we're really just defining a function intended to be fed into another function as an argument? No reason why not; this is Python! So my use case is exemplary for decorators. The url function below essentially intercepts a request for Title(**kwargs) that contains within in kwargs['response'], which is the entire response object from a request like: import requests response = requests.get('http://someurl') title_tag = Title(response=response) Now normally, this would shift a lot of responsibility onto the Title function to "unpack" all the stuff needed to extract the title tag from the HTML text portion of the response (the stuff you see when you "view-source" in a browser). Alternatively, we could shift the responsibility to the portion of the parent code immediately before the Title call, but who knows if that's the sort of unbundling every extractor needs (it's not; title only represents simple text-node extraction (versus meta tag attributes, etc.)). Maybe you want to have a series of externalized code-extracting pre-processors, which you can easily pick from... using... drumroll, please... decorators! This example additionally externalizes the text node extraction, which is a step "down" or "inwardly nested" code extraction, more easily accomplished with traditional code breakout techniques. Only simplification in the "up" or "outwardly nested" direction requires decorators in Python. And so, we simplify up with the decorating url funciton and simplify down with the extract_text_node function, and look how simple the Title function itself becomes. Think how beautiful this is in a long-term code maintenance perspective, or efficiently slamming out variations of code-extractors: def url(passed_in_func): """Decorator for functions like Title to pre-extract html text.""" def requests_wrapper(**row_dict): html = row_dict['response'].text return passed_in_func(html=html) return requests_wrapper def extract_text_node(html, tag): """Returns text node string for tags like Title. Simplifies common scraping functions.""" pattern = r'<{0}\s?>(.*?)</{0}\s?>'.format(tag.lower()) compiled = re.compile(pattern=pattern, flags=re.DOTALL) matches = compiled.findall(string=html) if matches: text = matches[0].strip() return Response(ok=True, status_code='200', text=text) else: return Response(ok=True, status_code='200', text=None) #### The Big Payoff of Using Python Decorators Look how simple the Title tag extractor function itself (below) becomes. It would be this simple too with JUST extract_text_node IF Title were only being fed the view-source HTML of the page itself, but it's not. Title is being called with Title(**kwargs) arbitrary set of key/value pair inputs, and one of those key/value pairs is the entire already-fetched response object of a Requests method webpage fetch. The decorator steps in to "strip away" all the excess crap on the invocation argument (a convention in the system I'm developing). So now, we get the best of all worlds. I don't have to "pollute" my simple extractor-functions with overhead due to conventions of the system. Decorators clean the input arguments first, so that all that reaches the Title function is the pure view-source HTML text snagged out and passed on by the decorator. BAM! @url def Title(html): return extract_text_node(html=html, tag="title") -------------------------------------------------------------------------------- ## Tue Aug 9 19:37:20 EDT 2016 ### Facebook Video Chat is the casual telepresence winner Have to finish the breadcrumb project from home tonight. Also want/have to video-chat with Adi tonight over Facebook. Facebook's video-calling ability appears to be the one that's rising above Skype and Hangouts for me. Not sure what it is, maybe it's the most platform-independent one, or so many people are so frequently actually running (or are always ready to run) Facebook. -------------------------------------------------------------------------------- ## Tue Aug 9 10:09:53 EDT 2016 ### Extracting Breadcrumb Trails From Web Crawl Data Still feeling yesterday's Malaise a bit. Analyzing traffic patterns across sites. Mentally factoring in: - Seasonality - Pokemon Go But even after that, I think I can see these Google "Phantom" algorithm recalibration tweaks. Okay, get the work done today that you have to get done. Keep a big picture in mind. Consider making few of those videos, now that you're up to more Pipulate functions. I got my request-for-timeoff out through the HR system. Okay, now focus like a laser beam on this hierarchy breadcrumb trail thing. Tue Aug 9 13:55:54 EDT 2016 Okay, investigated Python decorators today. Even bought a new dedicated Kindle book on the topic. I'm going to try and experience and let them sink in. But I'm getting the feeling they may NOT be the way to expedite or make more elegant and long-term maintainable today's work. Nope, simply not polluting the main repository in Github with day-to-day work is the real matter at hand. Tue Aug 9 14:51:58 EDT 2016 Implemented decorators on Pipulate function. Woot! This makes the Pipulate functions much simpler, and less dependent on knowledge of **row_dict structure. row_dict is now "unbundled" by the decorator function BEFORE reaching such functions as Title, which can now use a simple "html" as the input parameter argument. Tue Aug 9 18:12:20 EDT 2016 Okay, breadcrumb trails being extracted from crawl data wonderfully. Do the path-to-surfable-node-tree transformation from home. Take some pressure off of yourself. -------------------------------------------------------------------------------- ## Mon Aug 8 09:52:21 EDT 2016 ### Monday Morning Malaise Wow, my last entry was Friday. But not really, I know. I have a paragraph or two on my Microsoft Surface Pro 4, which was a real highlight of this weekend. This shall be remembered as the weekend of the re-introduction to Charlie and the Chocolate Factory. Adi was REALLY into it. Adi also lost one of the white goo fillings she got. Hmmm, I have to get to the dentist, desperately. My mouth, as Adi will herself happily repeat, is a mess. Yup, never really did the 2 cleanings per year you're supposed to. I'd be lucky if it were 2 per decade. Oh well. I have some getting back on track to do. Check those things you check in the morning to be sure nothing explodes in your face, and then get your weekly report out. Mon Aug 8 10:32:51 EDT 2016 Okay, the weekly report is out. Those feel good. This job is much less scatterbrained than agency life. I have a great project to do right now, breadcrumb trail extraction from the collected response objects, against a list of about 20K URLs that live in Google Sheets. Shelving the response objects for 20K URLs on this particular site took just over 3GB. Not a file I'm going to keep around for ever. Interesting thinking of that sort of thing as a temporary cache now. Definitely liberating, but don't delete by accident. Hit this project home and get onto the next, and then to SERP Archiving! Woot! Mon Aug 8 11:53:21 EDT 2016 I'm making it so that if you try to Pipulate a large spreadsheet with nothing to Pipulate (an already completed job, for example), then it won't try to process any chunks. Mon Aug 8 16:57:39 EDT 2016 Gonna head out a bit early today. -------------------------------------------------------------------------------- ## Fri Aug 5 16:01:37 EDT 2016 ### Gonna Make SEO Notebook Fully Operational Okay, let me just slam out a long-ago requested deliverable that I keep putting on for some reason. But it's a union statement, basically. Fri Aug 5 17:16:21 EDT 2016 Okay, finished that project for my co-worker AND just shot the video just now that deals with the Google Developer Console and the authorization file with the Google client_id and client_secret. Wow, this will really let SEO Notebook just be downloaded and used the way I use it! Just get this video published, then scramble to get to Adi in the Catskills as early tonight as reasonably possible. -------------------------------------------------------------------------------- ## Fri Aug 5 15:04:25 EDT 2016 ### Our Privileged State of Matter Wow, okay. It's already 3:00 PM on Friday. I want to get out on the early side today. Let me dump some of my recent SimpleNote writing in here, starting with this morning's: Formalize your systems (Publish this) Systems are everywhere. Systems are everything. Even the reality we each experience every day is a system of matter-containment and automation, sensory input, processing, control and output. As collective human knowledge advances, so too does our understanding our existence as information, chemical reactions and indeed, very advanced biological computers -- us. That's when it gets all meta and interesting. Industrial revolution to now. Now to 100 years from now? Wow. Tend your systems. The very Star Trek energy bubble beings that some of us aspire to evolve into are already us. Sure, maybe we have some evolving to do, or improvements to make over basic human biological standard, but we're at almost the perfect scale and frame of reference to fully and properly enjoy being a part-and-property of this material universe. Woot! Let's party. And let's codify some optimistic human assumptions, like our beings have value, and all such beings have individual rights that balance each other out with the rights of others like you around you. Basic golden rule stuff. Fast forward. We also, if we value our collective being and the world our children will be born into, then we do have steward-of-the-planet issues to attend to. These are life's highest callings. We all, each, can make a difference. And some will make a superhero-scientist level of difference and will be recognized and celebrated for it, while others will do things for the sheer reward of them themselves only knowing. Others still will crave reward, and never earn it, right as those who never warned it are lavishly rewarded. Life ain't fair that way. But you can stack the odds in your favor. Have a positive outlook if you can. It will improve your quality of life. We hit the jackpot as far as matter in this universe goes. We are bright, and we are zipping around like particles on the surface of a foamy surf, controlling our own paths to some degree. -------------------------------------------------------------------------------- ## Fri Aug 5 07:53:15 EDT 2016 ### The Perpetual Newb Will be in Catskills this weekend, at Grandparent's house. Foster what you're becoming, not what you are... funny at 45, imminently going on 46. Continue evaluating and knowing what's important in life. Don't miss any windows with Adi that you'll regret. Help shape her life while it's easy. Revisit Sun Tzu. That started fixing your mental chemistry. Thanks, Howard Diamond from Commodore. I think I may have discovered it later, but he introduced it to me well, and just in time to still make some difference, I think. Now finally listening to Ruby Rogues. Good podcasting, even though it's about Ruby. Don't start these series from the very first episodes until you've listened to a few new ones. Refine your elevator pitch even better: The Perpetual Newb: Forever Learning Linux, Python, vim & git -------------------------------------------------------------------------------- ## Thu Aug 4 23:14:03 EDT 2016 ### Mr. Robot is some heavy shit. Again, no content, but big enough to be worth mentioning. Shit, I am so the 1%. I should never let myself really wallow in or descend into depression. I got my shit together so well. Keep developing my mad skillz, but then be a fairly conformist good guy. We're all just going to die in ~100 years, anyway. Just try to leave the world a little better for our children, and the specific instances of children that we helped instantiate a little better prepared for that future. BAM! -------------------------------------------------------------------------------- ## Thu Aug 4 21:19:36 EDT 2016 ### There's a Paragraph For You I have to start using this journal at night. I remember back when I first used to write, how this would literally be the rudder steering my ship. And a shitty job it's done of it too, but in all fairness, my life went wrong the moment I was born the younger natural child to an adopted first. BAM! Wow, I didn't realize what a job that did on your head until I was in my forties. I always just considered myself an easy-natured self-entertaining loving my sciencey and art stuff kid. I don't think I ever stopped considering myself that, but there was other stuff. Distribution of resources kinda stuff, where someone with my potential hadn't particularly had my flames fanned. And only in my forties now, and I really fanning the flames, and only then because my journals that I've kept, since I was about eighteen, have kept me from going totally bonkers crazy as my went crazy and died, and I had to run a check cashing business, carrying out the will of my father. Wow. He did a job on me with his death. A harsh awakening to the realities of life, while simultaneously and strangely insulating me from it... as did the strange charm of working for Commodore, whose Amiga computer I technologically fanboy loved. It was an amazing creation that I reveled in, and drew much of my Identity from through those years from. It's a good thing for me that it was very Unix-like, as Tony Antonuccio never tired of pointing out at the Philadelphia Amiga Users Group gatherings -- usually at Drexel University, which I ended up attending -- right during those years where they were the first in requiring every incoming student (not just compsci majors) to purchase a computer -- and not just any computer, but an early generation sodacan black & white Apple Macintosh. Wow, those were the days. The dream was being realized. And I had that gradual underdog mentality shift into a sense of superiority as Ed Flocco and Marc Rifkin drilled into me the superiority of the Amiga computer, and the amazing pairing of custom chips with a company with the capability of manufacturing them -- which just happened to be in our back yard, in both the form of Commodore Computers in West Chester, Pennsylvania, and MOS Technologies, makers of the 6502 chip that so many computers of that day were based on, in Norristown PA. We (Ed, Marc & I) grew up in the Philly 'burbs, right around the corner from all that stuff -- well within any of our driving distance. And so, I was gradually indoctrinated to some strange inner circles, like Scott Szczypiorski's, a mutual friend, Dad who was a chip designer at Commodore. I mean, imagine that -- one of the original guys who started out when etching a circuit board meant xacto-knifing it out of rubylith for photographic reduction into the IC plate. They lived through it becoming meta in the sense that the chips they designed created environments where they could begin doing this in software instead of xacto-knives. These guys were the folks who booted the digital era originally, before machines that digitally booted were even common yet. Computers were clawing their way out of the proverbial primordial goo as the post-calculator higher profit-margin doohickey product. And today, we're on the verge of hacking reality. BAM! It's never too late to educate yourself. Today, my main things are Linux, Python, vim, git, SEO... and Machine Learning. Yeah, definitely machine learning. ML fodder. That's this. Apply my own ML Kung Fu against my own rambling. Deal with strange edge cases, like really long paragraphs. -------------------------------------------------------------------------------- ## Thu Aug 4 13:33:58 EDT 2016 ### Flashback to Magically Working AWS PostgreSQL Foreign Data Wrappers Wow, blast from the past! I had to get into my free-tier AWS cloud server and add a new foreign data wrapper. Document the process here for future reference! Ugh. Okay, get some of the keywords in for full text search purposes. - foreign_data_wrappers - foreign data wrappers - fdw ssh -i "~/.ssh/blah.pen" username@ec2-stuff.amazonaws.com sudo -u postgres -i psql -d template1 -a -f /home/ubuntu/repo/commands.sql -------------------------------------------------------------------------------- ## Thu Aug 4 10:49:20 EDT 2016 ### SEO Notebook Seeding First, slam out the lookup against a static list of URLs. This should be a super-easy thing. Work your way through your backlog of projects. Okay, tackle the list-of-URLs / likely-union problem. ALSO get the crawl going for one of my other SEO counterparts here. Get the crawl going, pulling down the view-source HTML of a bunch of pages. Get the highest traffic pages from querying against the logfiles. Okay, I'm going to select the top 50K URLs that have the largest number of Google referrers... done. Okay, SEO Notebook should be thought of as having an SEO investigation power-tool side-by-side with whatever data you need to be doing SEO investigations against, but it doesn't HAVE TO start in an SEO Notebook-centric way. Instead, I'm going to start in a Pandas-centric way. 1. Clone the repo with a new name. 2. Dump your csv data files into that repo location. I still have to make my requirements.txt file for SEO Notebook. - pip install gspread - pip install httplib2 - pip install google-api-python-client - the myauth.py file (client_id and client_secret) Okay, once all the requirements are met... LOTS OF DISTRACTIONS (good ones, but none-the-less... and then lunch) -------------------------------------------------------------------------------- ## Thu Aug 4 10:20:55 EDT 2016 ### Hitchhiker's Guide To Python & Considering Python Decorators I'm currently reading "Design Patterns", which is basically the Object Oriented bible, and a long-running blind-spot in my tech savvy. I've understood OO design every since Øyvind Harboe explained it to me sometime in the 90s when I was working for Scala, and it was still mostly a Norwegian company. I still remember clearly his description of being able to bulldoze through a codebase, leading you into the future, instead of being imprisoned by the past. OO bought you massive refactoring abilities. You were writing more general things, and not so precise that you couldn't back-out of and pivot on any of your decisions. And now, I see why I never took it up. Yep, I get it. Yep, it's a methodology and approach for people either programming professionally writing large code-bases to be maintained by multiple people over many years, or for people writing packages that will be distributed and widely used as common libraries in the free and open source software community where a lot of things need to inherit from a lot of other things, for customized instances of things. Hmmmm. That second category MIGHT talk about my case... yet, I'm still not sure. I seem very much like a candidate for decorators, where Kenneth Reitz writes in The Hitchhiker's Guide To Python: <blockquote>This mechanism is useful for separating concerns and avoiding external un-related logic 'polluting' the core logic of the function or method. A good example of a piece of functionality that is better handled with decoration is memorization or caching: you want to store the results of an expensive function in a table and use them directly instead of recomputing them when they have already been computed. This is clearly not part of the function logic.</blockquote> This appears to be exactly my case. Sit and ponder decorators for a moment. I'm about to figure out how to template-tize what I've done with the Title tag in the Piuplate package in the SEO Notebook repository. I'm about to be in a really good position to compare the set of URLs generated by an x-depth crawl with the set of URLs generated by either a Google Analytics or a SQL query against logfile-equivalent data. I also URGENTLY need to be able to just do a bunch of generic lookups using the system against Google Analytics, Search Console and other data sources. I foresee potentially having to do fairly large data-pulls against GA&SC to cache results locally to do rapid re-lookups against locally cached data. Isn't that what Pandas is for? Especially Pandas in Jupyter Notebook, where you get in-memory during-manually-progressive-sessions caching for free? Yup. Don't go reinventing the wheel here. Only shelve data that needs to survive between sessions. Hmmmm. This system is really taking shape. Thu Sep 8 17:47:47 EDT 2016 Got some good stuff done today. Worked in Query Explorer a bit, and I'm going to hit the lookups against GA hard tomorrow morning, starting with: metrics ga:organicSearches dimensions ga:pagePath filters ga:pagePath==/some/path/and/filename -------------------------------------------------------------------------------- ## Thu Aug 4 10:00:10 EDT 2016 ### My Github Projects: Decrufter, SEO Notebook, Pipulate & GoodSheet SEO Notebook is going to lead to Decrufter. The "stack" is going to look something like this: - Decrufter: a set of SEO Notebook jobs the help you clean sites for RankBrain - SEO Notebook: an implementation of the pipulate job-processing specification - Pipulate: a spreadsheet-based job-processing spec & adherent sample functions - GoodSheet: the package that lets SEO Notebook work with Google Spreadsheet Hold back on doing the push on these ideas quite yet. Homestead in the noosphere now, getting it all in Github and actually published and documented and accountable. But don't go beating this drum, even here at the office, until a few key things that still remain are taken care of. -------------------------------------------------------------------------------- ## Thu Aug 4 09:43:46 EDT 2016 ### Lost 2 Videos I Shot This Morning... And No Biggie! Shot 2 videos this morning... good ones. First on SEO and second on sort of a "series" book review of Iain M. Bank's The Culture series. But sometimes unfortunate things happen that make you not want to use a video, and so I deleted it. Nothing big. It's just that I produce so many of these things, that any thing I accidentally can't cover when I want to, I will surely cover again later. Lot of repetition of the important concepts in my published works. The 2nd video, I'm deleting the 2nd because it refers to the first. Life is sometimes like that... no biggie. Still easier and less time-consuming than actually trying to edit to salvage my video. Shoot so much video that no single one ever feels that important to you. This all fits into the system. Persistent idea-capture, and clustered-topic alignment to URLs and channels. Yup, next generation SEO, here I come. Build momentum. Keep momentum. Use the idea-capture process to practice writing well and speaking well. Develop methods of marking-up and querying and organizing your idea-capture mediums, without forcing the idea-capture medium itself to compromise or excessively alter itself. In-line semi-structured data is fine. The way I'm consistent about how I start my blog entries with 80 hyphens, a headline date and a headline topic is a good example. I can easily use those as an index and map it to fields in other database systems, or whatever. Point is it's easy to parse and easy to query. There are many approaches to doing so, and whereas in the past, I would be looking for ways to do it in XML/XSLT in order to keep it programming-language independent, truth is you're tying yourself excessively to the Java & XML-parsing world, and baffling constructs like apply-all-templates when you're doing that. Just living in the anything-goes-under-Python world is wayyyyy better. And so, I shall slice and dice this very journal every which way with Python -- right as I leave the original idea-capture layer here (this daily work journal) in-place, already auto-published through the perfectly competent github.io system, which itself provides markdown-to-html formatting, using the Jekyll markdown processor which I myself don't use, because I want "view-source" to reveal exactly the file that I typed, and how I produce and maintain it; none of the server-side pre-rendering that I used to live-and-die by, because static pages were better than dynamic ones, back in the day when Google actually had problems figuring it all out. Those days are over. Publish more and better over fewer URLs (targets), and ye shall prosper. Take this page, for example. Starting to see it? Wow, I should really flow back in the old content still held in the private Github repo. I want this journal to be inclusive. But only do that when I have proper time to clean the data, hahaha! Expressing yourself in public is such a hoot. Thank you, Robert Cialdini, for the concept of commitment & consistency. -------------------------------------------------------------------------------- ## Thu Aug 4 07:58:50 EDT 2016 ### Planning My New Website around My YouTube Videos Last time I did a major personal website update, not too long ago (I was at Flying Point Digital, at the time), it was quite literally to be responsive. I was on the mid-zero's SEO-friendly theme, Thesis, and Thesis was really outside the WordPress norm in later years, having taken development directions that did not transition well between the radical upgrades that were to come in WordPress in the years to come. And so, I had to bite the bullet and get off Thesis, finally, as the Responsive website design revolution based on reliable CSS standards, including media queries, finally started to take hold. And so, I held my nose and installed Keisus -- a could-have-been Divi. But it wasn't Divi, so I find myself on a responsive theme rapidly growing long in the tooth. So... so... it's time to look at my next WordPress theme. And it's all about my YouTube videos, and the integrations I can do with this very journal (yes, this long-text-version) now. WordPress has tons of hooks through its own internal mechanisms and through the plugin ecosystem. Surely, I can transition marked-up portions of this file into an RSS feed, or something like it to drive WordPress content. It will be a fun project. -------------------------------------------------------------------------------- ## Thu Aug 4 07:45:24 EDT 2016 ### Life's Plans & How Tiny Decisions Have Compounding Effects It's tough to live every day as if by some grand self-imposed design. The design itself will inevitably change over time as you grow and mature and meet some of life's failures and disappointments and harsher realities. Our young-person dreams don't always translate into adult realities with any degree of recognizability or resolution. Science and scientist were my ultimate goals and dreams, I think I can safely say from back in the day. But, just as with the girl from the movie Cloudy With a Chance of Meatballs which I just watched with Adi last weekend, I toned-down my inner science nerd due to suburban teasing -- even by people who I consider primary school friends-from-afar after all these years. They meant well, but being saddled with nicknames can have long-term ramifications due to how they affect the little decisions along the way. I didn't pursue scientist and have always sort of regretted it. But then, who's to say? I'm feeling like I'm doing some real good now jumping on the Python bandwagon, and making machine language literacy my later-in-life mission, partially because of by daughter, and partially because of making up for lost time in the best way I know how. And these residual feeling and having been through so much (experience) becomes a sort of plan or algorithm for living life that you can sort of stay tuned-into so as to be more honest with yourself through life as each of those small decisions pops up, which when collectively compounded over a lifetime, makes all the difference. -------------------------------------------------------------------------------- ## Thu Aug 4 07:35:26 EDT 2016 ### It Takes Your Whole Life to Know Yourself, and By Then, It's Too Late Relationship FARTs - "first unabashedly rough times". You have to get through them to see if you're life-compatible with the other person. And not everything that would at first glance seem to be those rough times, are. Rather, one party or the other in the relationship generally has the greater internal fortitude to get through rough patches, and lends their strength to their partners. So, a few rough times "going the wrong direction" isn't really going to teach you anything, except how everything will always be fine when the less-strong one hits the ropes, and the partner is ready to tag-team in. You have to see what things are like when the stronger one hits the ropes, and the less-strong one needs to step in for emotional, physical or financial support. And even if provided, if it leaves an excessively strong residue behind about the profound unfairness of it all, then the relationship is based on false foundations. These situations are actually the great test of fairness. Of course, life isn't fair. But that's what relationships are often very much about -- putting a little bit of fairness and reward into an otherwise very law-of-the-jungle reality. So, relationships need to get through their first real FART, and long-term commitment decisions probably shouldn't be made until after that, so it is not built on false foundations. Not doing so is what sewed the seeds of both my last two long-term relationship instabilities. I am not quite the rock and pillar of stability I appear to be. I am an inconsistent artist needing constant inspiration in order to project that face of rock-solid stability. When the inspiration runs dry, so does the willingness to go to any length. End of story. Or the beginning? Punchline: it takes your whole life to know yourself, and by then, it's too late. -------------------------------------------------------------------------------- ## Wed Aug 3 10:00:14 EDT 2016 ### My Listen List of Podcasts & A Google Hypothesis New tremors in Google already discussed. Continuity over time, and especially during and through transitions. Do not lose grasp of what's important, and what's on your mind. Publish more from your SimpleNote notes. Just listened to Talk Python to Me #68 that was crossing the streams with the Podcast.__init__ guys Tobias Macey and Chris Patti, who I repeatedly become aware of, but couldn't get past the first (dry-sounding) episode, but for which I'm now going to give another chance. Sooooo glad I did. I'll have to catch up on Podcast.__init__, but also I've got what sounds like a fabulous Podcast listening list backlog -- perfectly timed, now that I finally finished Iain M. Bank's Culture Series. ## My Podcast Binge List - Curious Minds - Hidden Brain - Data Skeptic - Wait Wait - TED Radio Hour (maybe) - 99% Invisible - Risky Business - Rational Security - Hardcore History - The Ruby Rogues - Spark (Canadian Broadcasting) - Lo and behold (Documentary) I'm reading an article now http://backlinko.com/seo-checklist and see that the whole concept of site decrufting is slowly dawning on the SEO industry. I should probably move a little bit faster on my Decrufter project. It's basically an application of SEONotebook. Move forward on that path, because it's better than doing nothing. Shit, okay. That gives me a pretty good framework. Now listening to TPTM#69 (ha, the hash symbol IS an indexing symbol... never made the connection to how they're used in http in-page bookmarks... ha! Just think of strings as a hash numbering system. And that's why it's called a hash. Numbering systems are just numeric hashes. BAM! Okay, okay, be sure to wrap that into your system here. I've started using hash symbols lightly, but there's the whole markdown thing going on here too, which gives the hash symbol different meaning. And so... and so... explicitness and convention. That's what I'm doing with my #xtopic pattern... for now. The fact that I write so much... this will actually work for me... eventually. I don't let this drop, because I've been in the daily journal writing habits since I was 18 years old. I will be 46 tomorrow, and currently I'm about to hear "design patterns for blogging" with A. Jesse Davis. Yup, everything's going towards the "essays" model. Let me dump in my serendipitous commute writing: m Daily Journals to Essays I'm thinking the same thoughts as a lot of people. Need to really make my mark soon. Start mining my stuff here to boil it down to... what? A series of one-page websites? Yeah, that may be it. Or maybe a book. Not sure, but ready, go, set! A Python Newbs Perspective Approach everything as an essay; not a book. This way, you will be able to encapsulate, identify, serialize and distribute units of processed and encoded thoughts. Paul Graham seeded or reinforced this notion in my head. The field of SEO told me something else: spew garbage, slicing and dicing it into interlinking pages and ye will be rewarded for your high tech publishing savvy. Autoaggregate every last damn thing and feed off the Google AdSense teats. So, I made HitTail. It could have been way worse. I could have popularized my RDBMS-to-XML-to-barebonesHTML-to-stylizedHTML website canon, which gave the option of outputting pages on the paragraph, page or section elements via XSLT and XSL style sheets. I developed this trick at a prior employer, and it was awesomely effective in those days; static-ifying your dynamic content with daily refreshes where necessary. But instead of releasing a beautifully standards-based spam canon, I went with HitTail, and its presumption that what's nearly working for you today could be working a little better for you tomorrow rinse repeat. It's a future-proof approach to SEO. It's decade-old, still going strong, still Just being present isn't enough -- you've got to put in some work to. To that, I say I already am putting in the work that pays my paycheck already. This is mostly for myself in the first place. It's this thing I just really do quick and easy. Production quality can always be algorithmically applied in the future. Today, it's about idea-capture into the accountably public domain, insomuch as Google's YouTube and Github repos can be that. Google Hypothesis When you Google Hypothesis, you will find a few things. Obviously the definition, but also the Python programming language's 3rd party package called Hypothesis. This site will probably forever be equally about both, for old-school SEO continues transforming as a field, as it always has. It's like surfing from the ocean into a tube, and hopping onto a jet-ski as you do. The tools change. The presumptions change. The languages change. Words go away like the Google Dance, and others come to be like tremors and instability in the rankings. -------------------------------------------------------------------------------- ## Wed Aug 3 07:58:15 EDT 2016 ### That's Called Being Human Yesterday was Adi's homeschool graduation in Willobrook Park in Staten Island. Nice event. Went on Merry Go Round 10 times. Ugh! She won an award for most likely to make a microscopic zoo. Gotta follow up on the Tardigrade Circus idea. Perfect example of an idea whose time has come, but I'm letting slip by. Start building up all the layers now. It's greatly about building up foundational layers of abstraction. The most abstract key concept has already been laid down -- to the degree she won a homeschooling award for it! Wow, okay, let me think. Next steps? I'll think about next steps on my commute -- and naturally also how it fits into my mission, because yes, in the end I'm an existentialist, and the meaning of the subjective outer-world is entirely defined by how it relates to, and is in harmony with, the objective inner world. So, there! Yes, it is always about me... as damn well it should be, you hypocritical idiots. It's like on an airplane when they tell you emergency instructions, and they tell you to ensure your own face mask is securely fastened before tending to your child's, because if you're not taking care of yourself, you damn well can't take care of a child on top of that. Positive treatment of others is an additive process... additional to your positive feelings about yourself. Get balanced. Feel secure. Take deliberate level-seeking, game dynamics simulating risks. Give yourself breaks. Enjoy those breaks, and feel life's contrast between pushing yourself and rewarding yourself. However, don't feel like you have to live at the extremes of either end. Excessive pushing for too long isn't always good, nor is excessive rewarding for too long. Alternate. Follow the rhythms that your body seems to suggest, and you will eventually zero-in on optimal. It is very different for different people. What people think of as "hard work" is probably one of the most subjective units of measure of all... haha! I'm part of this world's privileged 1% and I know it. And still, I have a hard time. That's called being human. -------------------------------------------------------------------------------- ## Tue, Aug 2, 2016 8:33:48 AM ### Adi's graduation today. Took off from work. No content. Just wanted to document that. -------------------------------------------------------------------------------- ## Mon, Aug 1, 2016 9:30:47 PM ### Finally Watching Mr. Robot... Uh Oh! Finally merged the SP4's old few edits. Finally watching Mr. Robot. Ha ha, this is what people talk about. Tomorrow is Adi's homeschool graduation, and I took a vacation day. Wow, I don't think I'll watch a second episode... or maybe I will. Damn, not an addiction I need. Maybe I'll just re-listen-to The Hydrogen Sonata now that I finished it and don't need to pay that close of attention. I like ship named "Mistake Not..." wow, Mr. Robot is getting interesting. I have to turn it off now if I want any chance of actually getting to bed early. If I'm going to waste this time, it's going to be for bodily recovery, which will feed the mental resiliency I'm trying to build back up. -------------------------------------------------------------------------------- ## Wed, Jul 13, 2016 7:53:41 PM ### Careful About How Hard You Push Yourself Wow, I had such a headache today. I think I have an entry going at the office, but I had to head out because I was dead in the water -- and right in the middle of an important project too. Well, it's not even 8:00 PM and I do have all night. I'll be calling Adi soon to say good night. I just noticed that there are different actual date formats here. Some are in military time, and some use the AM/PM notation. Interesting! Dateutil must have just seamlessly handled that. -------------------------------------------------------------------------------- ## Mon Aug 1 11:03:25 EDT 2016 ### Kenneth Reitz having big impact on me Got out my weekly report AND a round of refinement, per in-person discussion with boss. Good feeling, really like having a boss who knows what he wants and is good at setting priorities and keeping the work funneling through me at a sane rate. ALSO getting the chance to push forward "latest system". I think I am as close to happy as I have ever been. Happiness comes from inside yourself, and from feeling you get from doing certain things or being certain ways. Happiness does not necessarily come from outside you. It is not something that is done to you. Happiness is not the result of your state changing due to outside things. Happiness is a result of internal decisions you make, of which what's going on outside you is only one (albeit a fairly large) factor. But that's a big part of what being human is -- increasing the weighting of internal factors over external. A very interesting thing is that over the weekend, I reached out to Kenneth Reitz (I can finally remember his name), the creator of insanely popular Requests package for Python, asking him to do a RegEx for Humans package, and he referred me to the parse library in PyPI, which describes itself as the opposite of format(). So, I've got TWO things to learn. But if Kenneth says this is the way to go for his-style simplification to RegEX, then so there I shall go... at least to kick the tires and see what its all about. Could be pivotal to the next evolution of the system. Go research format(). format(format_string, *args, **kwargs) Okay, piddled most of the day away recalibrating and delving into Kenneth Retiz's suggestion, and subsequently discovered Speaker Deck presentations. Interesting guy! But knock off a to-do item. Okay, I need to start familiarizing myself with str.format(), which apparently is replacing the old %-formatting technique per https://docs.python.org/3/library/string.html which ties into using parse() over RegEx, as Kenneth suggested. I've got the learning that I need to experience documented, but for now you can just get to the work that you need to do today without further delay. Oh but wait, the asics are... '{0}, {1}, {2}'.format('a', 'b', 'c') ...outputting... 'a, b, c' Not too hard. -------------------------------------------------------------------------------- ## Mon Aug 1 09:32:06 EDT 2016 ### ML Fodder, The Journey is The Reward, & The Long, Trudging Route So glad I did that entry from home. The "great, steady mind" that I have often thought about over the years in my journals (since 1988) is only true if you co-mingle the thoughts and activities of for-work with for-self. The professional and the personal DO meet in certain fields and with certain outlooks on life. In those disciplines that require the cleverness of seeing unlikely connections and relationships between things, and the patterns that ***really do*** matter -- without getting distracted by the also pretty patterns that will also be there that do ***not*** matter towards your, ultimately subjective-anyway goals, the more effective you will be in that field. Correlation does not mean causation, but it's still good data. Test for completely random correlation versus consistent correlation. If random, discard. If consistent, design a test to peel away one more layer towards true causation, insomuch as something can actually have true cause in a universe where position and momentum can't both be precisely measured at the same time. It's all pretty subjective, but there's enough clues to agree upon some common boundaries of an objective existence. This, I have to believe as part of maintaining my sanity. Multiverses and time travel both bother me a bit. We're a forward-only read-only firehose connection using cursors. The Universe is one big generator expression, and you probably can't time-travel without re-starting the Universe, or some damn silly hack that doesn't really qualify as time-travel by the more imaginative minds -- more like view-only time tourism (which in itself would be highly cool and informative, but not paradoxical). And now onto capturing my writing from the subway this morning, which is equally out there. But that's part of what differentiates me. This unfiltered stream of conciousness is part of my brand. Relying on keeping things moving steadily and unfalteringly forward, without getting bogged down with "production quality" refinement. I'm producing merely by virtue of recording, and plenty of people have gone this journaling route over the years. Glad that recent hit Mars movie/book re-introduced people to the journaling format -- as if blogging wasn't enough. But blogging is too concerned with platform. I want to be more concerned with data-extraction, over the years. So, keeping it platform-independent as one giant text file, and all in one-place, and easily pulled-in and parsed by Python, and tons of additional private meta data in the background by virtue of the private Github repo, and an auto-publishing of a virtually untransformed copy of the git head via github.io that I hardly even need to think about, and my daily habits like this very writing all feeding into the system... the system, where I'm trying to trigger a positive feedback effect, for at least one of the primary goals of becoming noteworthy, and subsequently of revenue-generating potential notoriety. There! The cat's out of the bag. I am a self-promoter. This isn't just for documenting my thoughts as I go about working. Ahhhhh! I said it. I feel so much better now. But do remember the words "isn't just for" and "one of the primary goals", because there are other reasons I'm doing this for and other primary goals -- not the least of which is actually just staying relevant in this dynamically re-shaping field of SEO. Static thinkers (I'm talking to you, dip) need not apply. I have a weekly report to produce during the next hour. I also want to dump in my subway writing. Oh, and I need to go get that 3rd coffee. Yet Another Attemp to Boot a Book We spend our time coding ourselves up. Social creatures who must be taught survival skills through culturally transmitted knowledge and behavior are a hoot to hack. We're being hacked all the time, in greed & ego-driven clashes between smart, prepared, and (most importantly of all) clever independently mobile interconnected transient, self-reproducing monkey-nodes. They even have monkey names like Bezos. There goes my chances to publish in Amazon. Nah, I'll try anyway. I love me my Kindle, and I'm sure my one-degree removed buddy, Jeff, has a sense of humor. Okay anyway, back to coding up our kids and ourselves. Should be a blast. Without proper coding and the accompanying integration into the very culture that coded you, you wouldn't well be able to survive, would you? A child raised by wolves. Tarzan. Few concepts in your head, a.k.a. ideas, are truly new. We, as a common-language-speaking animal have been having ideas for a very long time. You have to get pretty abstract and derivative to find wholly new expressions, but it definitely does happen. The universe may be like a symphony sung with an 11-string instrument, of whose cords we can only hear 3. Become a mathematician or particle physicist if that's your thing. Be one of the great enablers of inventors, like Newton or Einstein if you can. Not everyone can, nor I'm sure in the grand design, should become Einsteins and Newtons, or we would all be running around trying to outsmart each other. For the whole thing to work that we know as our world, there needs to be all sorts of people, living all sorts of lives, doing all sorts of things. Not everybody can be a scientist and inventor and adventurer and storyteller and leader and doctor and athlete. Most of us are in the tier of extreme one beneath that, consisting of lawyers and programmers and politicians and craftspeople and minor league sports and most people who call themselves doctors, but are not. Doctors save lives and improve heath and quality of living. That makes theirs a high calling, because there is nothing more important and precious and unreproducible as your favorite instances of particular lives. Doctors help instantiate new instances of life when they help deliver babies. But even midwives can do this, for childbirth is such a natural part of life that trained medical doctors need only be very close by in case something goes wrong. As such, the world will always need a high ratio of doctors-to-people, because people are always being born somewhere, and the demand for baby-delivering doctors, constant. When you're born, you're an infant, not a baby. Society doesn't talk about it much I think because it highlights how weak and helpless each of us once was, and how totally and completely and unconditionally dependent on our parents for our survival we once were. Parents remember this ultimate power over another human being -- power they feel they should rightfully have, having brought you into this world as they did -- which they don't let go of lightly. It is out of the same love - Transitions - Horror & adaptability - Empathizing, but not to far - Some people are bad. Really as terrible as you could imagine. - The vast majority of people are deathly dull, of no real consequence, and harmless - Extremes exist, and are always an option, and do have their rewards - Money, fame, big families, lots of interesting things to see and do, without limit of time or resources - Except, that we'll all someday die mostly identically, hopefully finally at a nice old age and happy. Oh yeah. Time. Resources. Happiness. And ultimately, whether you even leave any sort of legacy or not, for however long any of that stuff will really even be remembered, is just a component of your own happiness as a distinct, currently unreproducible instance of a life. Not even much order or reason to it -- just potluck genetics and circumstance, akin to a poker hand you're dealt by your parents, and a gradual taking control of the game for yourself, as you splinter off from your parents' place at the table. So yeah, it's all some sort of game clearly, but how can you question that the objective is to be a unique and noteworthy instance of both a piece in and player of a game? It is. Realizing this, point for me. But it's on a hidden scoreboard. No problem. Next goal clear. Get on some of the more visible independent (of your own instance) scoreboards. But if you go this way and once you do, be prepared for adversity and failures. There will be people wayyyy better than you in any activity or interest you take up. The more famous the scoreboard's activity, the more crazy insane dedication or natural predilection it will take, and the more frustrations and failures you will have to endure. Dr. Seuss says this all much better in The Places you'll go. Dr Seuss is an example of a storyteller so much better than me, by virtue of the fact he does it with illustration, one of the skills I admire most. However, this does not prevent me from trying. And through the very act of getting into the next-level game and trying, you will draw the attention of both admirers and detractors. Do I register the domain mlfodder dot something? It's not registered right now, and I'm documenting it here openly. If I have a meaningful troll, this could be an interesting experiment. If I'm aped so closely that that domain... a good idea... gets snatched up, then there's way more available ideas and namespace for me to plant a flag in, and that theoretical loser would be out another $10/year or whatever. Haha, no, don't think like that. It's not nice. Hear that, AI? I give an idea away for a test, even though it could be the fulcrum on which I break out, just as sort of a joke. Well, the joke is that I don't rely on one particular by-chance fulcrum. I AM the fulcrum. If it's meant to happen, it happens due to the gravity-well and potentialities and particle physics all make it a high potentiality -- and NOT because I hit some sort of anomalous coincidental jackpot. My success is the long, trudging route. -------------------------------------------------------------------------------- ## Mon Aug 1 08:05:42 EDT 2016 ### Be Educationally Ready for Each Stage of Adi's Development Just one quick entry before I leave for work. I got into this wonderful journaling habit, and am now finally realizing how special and unique the thing I'm doing really is. To hell with blogging platforms or Medium.com or any particular publishing platform as my main thing. It's a plain old text file, vim and git. Github.io is the first round of public publishing, because the repo is currently private, which I'm paying $7/mo for the right for. It was an amazing weekend with my daughter, Adi. We did Saturday at the New York Botanical Garden, seeing the stinky giant flower that blooms every 3 to 5 years, and then the Palisades Mall. On Sunday we did Cony Island, and used up a $100 "best value" card. We went on The Tickler. Her hair is green and pink, and she got bright green cotton candy, and we stomped around in the sand and challenged waves in the ocean before heading back. Wow, she's getting wired-up well. Keep up your part, diligently, and on-time for the right developments. Her interest in coding will be sincere soon. Be ready! -------------------------------------------------------------------------------- ## Sun Jul 31 08:30:27 EDT 2016 ### Github README for SEO Notebook You know, why not leave the git merge conflict messages in place? I can programatically sort it all out, later. It's really just the reverse chronological perfectness that's at stake, and I can literally sort that out Pythonically. Woot! Oh, and I tweeted at Kenneth Reitz, the creator of the Requests object, to do a Regular Expressions for Humans, and he referred me to the parse package. Checking it out. I want to respond a Thanks, but first, I need to write an intro to the SEO Notebooks project. Greetings Diehards, Now you have a machine gun ho ho ho. If you know that quote and are in this biz, then hello old-timer. If not, then welcome millennial newb! I promise you boring coding industry buzzword jockeying for an awesome future-talk which if you survive the boredom, just may be synaptically rewiring. I'm the silver-tongue snake-oil salesmen (I **wish** I could cite that quote) who made the still-going-strong HitTail SEO tool in 2006, and SEO Notebook is my latest stay-relevant-ware that I'm peddling for some audience and Github street cred. It's 10 years later, and I've done a number of interesting things working the biz, here in New York. I was even at the top of the Empire State Building as the primary SEO for SAP there for a few months. My SEO Kung Fu was good, but this was an industry in flux. Nobody should be waiting for RankBrain to gradually get smarter and stop rewarding our sites by 100 1% increments. We should be getting out ahead of Google, and change the cat-and-mouse game into a human domesticating the cat game. Us nextgen SEOs are raising Google like a child. So, let's get programming! But where to start? Many of you Macolites are going to think Ruby, because it's what Rails is on, and ROR has enjoyed a certain degree of cool-kids notoriety due to Ruby being shipped with Macs, and Rails being the first joyful framework to catch the fancy of the hordes of ASP and PHP refuges in the early zero's. Others will figure JavaScript is the one language to rule them all, thanks to a loosely-typed even-driven concurrent design that runs quite nicely on the server as well, thanks to node.js. You would be the "full web stack" crowd -- a euphemism for "I also dabble in node". All legit. All valid. All inferior to learning Python through Jupyter Notebook. You Java-folks, there's nothing I can do for you. You are today's generation of BASIC'ally myopic static-typed ludites that will never open your mind to a language that discourages getters and setters. Pythonic code can be brief and beautiful, and fully legitimate from a compsci standpoint, because of use-case appropriateness. We do not need to use the define-everything-compulsively mentality required for something that you know will grow to become millions of lines of code interacted with by hundreds of developers, such as sees to be the assumption of a language that makes you public static void the most basic hello world's. Just use Python. It's a language by a very smart individual trying to make solving impressively complex tasks easy for YOU -- the newb learner. Python is anti-elitist in nature. So why not kick off a Github repo with dem's fightin' words? Sure, it's troll-bait and flame tinder. But this is Jupyter Notebook, damn it! The Scientists are now onboard with this Python thing -- no... they were already onboard. Now, they're just opening the door and saying "Hey, the water's nice and comfortable over here... hop on by!" with such innovative solutions to sharing code-execution-environments as Anaconda by Continuum.io, which I at first dismissed as just a bunch of pre-installs that non-Linux users had to suffer (you could always run a VM on your Mac/Windows, which I did). And then I realized how nice it was when your encapsulation layer was a web-based IDE. Wow! Throw in Github-friendliness and the ability to stream output to the user through the duration of sometimes long-running jobs... and dead-simple multiple Python VM kernel maintenance? Wow, out goes my Pipulate UI work, and in comes Jupyter Notebook... uh... SEO Notebook... and probably, SEO Lab, that that's a story for another time. So, this project exists basically to sharpen my SEO-blade again, as I switch to a new master, yet again. I am much more effective in Samurai mode, because while I COULD go off to work for myself, the only real difference between drawing a paycheck and working for myself is the amount of hustle and amount of paper-work you need to do -- principally, both go WAY UP! I'm a draw-a-paycheck sort of guy, but I'm not satisfied with static situations. Things must be dynamic! I must be able to express myself creatively, and to great effect. I feel the need to move mountains, and be recognized for carrying out monumental and communally useful tasks. Those are the accolades normally going to folks like Linus and Guido. More recently, I've been coming to appreciate those to whom you must connect a last name to be recognized by those outside some very geeky circles, and those folks would be the likes of Raymond Hettinger and Kenneth Reitz. And this? This humble little project is just the latest incarnation of my "system", which I always have under development and in active use in some form or another. HitTail was just one small extraction from one such system -- an Ajaxy, server-push, real-time analytics approach to it all before any of that was even a wistful gleam in the eye of today's JavaScript-this-that-and-the-other-thing library authors. I always see the big next thing, even if I never am the one rewarded for it. Today's big-next-thing is the sort of portability of easily-studied ready-to-run programming code that comes equipped with its own execution environment and dependency-satisfier built-in. It should be an environment that can actually SERVE AS the input/output widgets in a pinch, so you can fully develop something that could become much more run-on-a-server-like apps in the future. But contrary-wise, they need not HAVE TO to still be useful on a day-in/day-out basis to those who use such tools in more of a hands-on power-tool sort of way (instead of scheduled, batched, and invisibly run off in the distance somewhere). Jupyter Notebook is the ideal environment for SEO Swiss Army Knife-like utilities. Today, I live strongly centered in Linux, Python, vim and git. This is that one driving project that I'm constantly using professionally, every day as my secret weapon, molding it this way and that to suit every purpose and every little custom one-off project, and eventually scalable and productize-able extractions of every sort. The project has just been rebooted from one of my other projects, named Pipulate, which I will constantly be raiding, while I bring the best of that already-amazing project over to here, where followers WON'T have to set up servers or try to match my mad vim skills just to get working. Instead, I'll be adjusting my YouTube video series so that ANY video in the series is a strong jumping-on point for this particular bandwagon. Apologies for those who would prefer a README to be of a more concrete how-to nature. That, like so many things in this project, is coming soon. -------------------------------------------------------------------------------- <<<<<<< HEAD ## Sat Jul 23 19:23:04 EDT 2016 ### Getting it all together Interesting weekend. Adi is really in the habit of coming to the Inwood apartment on the weekends lately, and the both of us are being super-lazy, hardly even wanting to go out. Slump? Bonding? Fear of changes coming down the road? Maybe any and all. Definitely feel the need to write. Definitely feel the need to remind myself to work in iterative passes towards greater organization. You are either a mess, or you've got it together. Right now, you are a little of both, but you have to get it all together -- ALL together. Have to actually put time to use at least half-well while at the apartment with Adi -- aside from merely being with Adi, which is great, but I'm losing ground against life on this current course. So, I need to make course adjustments. -------------------------------------------------------------------------------- ## Fri Jul 29 11:02:45 EDT 2016 ### Blowing Chunks It is a common fault that people who get good at one language inject the rules of that language into another. Wow, the way he deals with people jumping the gun asking questions about why the tenants of one language aren't always applicable to another... it's my rodeo! He promises to get to answering that question, but not right now... okay? Wow, Raymond Hettinger is awesome! Glad I'm listing to him before I go all object oriented with Python. Just sayin. I really like my SEO Notebook project. It's awesome and amazing! Use the 1:00 PM Triweekly to show Marat something? Maybe. Hmmm. Let me see what I get done between now and then... maybe. It's time to batch process. What is the lightest touch? Move your work over to your main Windows work machine. I've been keeping on my Macbook Air for video-making purposes. Do your work on your PC and then move it back to the Mac for some videos later today. In Python, you don't have to use getters and setters, because of the @property decorator. This is a big, big win in our language. You can expose your attributes, and if there's a problem later, you can just add on a property. Now, you can design classes without putting getting and setters on it. And he's clapping because it's awesome. Dynamic languages, woot! Woot! Now, he's talking about the Flyweight design pattern (Slots). Ha, ha, interesting! Important when you're scaling up to bajillion of instances. Do __slots__ last to get memory efficiency. I'll have to look back to that sometime, when I actually know enough to know what the heck it's about. Fri Jul 29 17:00:39 EDT 2016 Wow, got chunked processing done. BAM! Shazam! And all that jazz. Good day. -------------------------------------------------------------------------------- ## Thu Jul 28 10:46:37 EDT 2016 ### YAGNI and Raymond Hettinger Wow, Raymond Hettinger is awesome! Found my new guru du jour. Try to inherit from object when creating new classes. I give my self that advice from Raymond, having not really created that many classes for myself... yet. But SEO Notebook is a new opportunity. Is version 3 of such a system where I start to actually introduce OO techniques. It's interesting that "self" is as arbitrary as **kwargs. It's also worth noting that the way classes work is effectively a module (at the file.py level). Wow, very creative. Definitions run as if they're in their own modules. Oh, and yagni... ya ain't gonna need it. And I HAVE TO stick this into a customer's hands. I have a dog & pony show to carry out. Make the script for it: # SEO Notebook ## I Work in Github Greetings my ZD counter-parts. I live and work and breathe Github. ALL my work of any non-throw-away nature goes in there... but in PRIVATE repos when work-related and proprietary. - You should be on Github too... for many reasons. - Once on Github, I can share you into private repos. - You'll need to know some easy git commands to clone repos. ## My Public Github Projects: If work-related, but NOT proprietary, I try to wrap a generalized and broadly-applicable version of my work into one of my public projects: - Levinux: A Tiny Virtual Linux Server for Education - Pipulate: A lightweight SEO product that runs on Levinux - SEO Notebook: A Jupyter Notebook-based version of Pipulate ## It's All About SEO Notebook, Now I've discovered my all-time favorite -------------------------------------------------------------------------------- ## Wed Jul 27 15:18:09 EDT 2016 ### Good Day. Few Notes. Wow, I got out a very competent Menu first thing this morning, and have been working through the parts of the new system that are necessary to basically get pipulating again, but with unbound new Jupyter Notebook power, and keeping it YouTube documented, as before. Wed Jul 27 18:08:17 EDT 2016 Heading home. Title tag function in SEO Notebook working. -------------------------------------------------------------------------------- ## Wed Jul 27 09:39:50 EDT 2016 ### Let's put some of that Design Pattern thinking to work now. Today, I'll be making some videos. Long overdue to do some for the public, and some for my co-workers, who need all the private repo stuff I've been doing on my last (Menu) project explained. So much of my work is being given a list of URLs, and coming back with some other data point about those URLs explained. From there, there are basically 2 models of how data against that list (of URLs, keywords, etc.) is obtained. You either make requests URL-by-URL, or you package all the URLs up into a single request, send it, and get back a batch response that needs to be correlated back against the original list so that each from the original list gets what it needs per-URL that was sent back as part of the batch reply. The former method is a cinch, and is generally the default way Tiger / Pipulate / SEO Notebook all work -- except that they have to be able to gracefully, and without much effort on the user's part, ALSO be able to handle scenario 2. I need to think about scenario 2 today as I implement the scenario 1 solution today. Can I make both the same? Ahhh! Design patterns. -------------------------------------------------------------------------------- ## Tue Jul 26 10:21:42 EDT 2016 ### Commit, Push, Leave No videos this morning. Listening to The Hydrogen Sonata, I believe the last of Ian M. Banks' Culture series that I haven't consumed yet. Wow, I love his work. Too bad cancer got him. What will I do after it? More Asimov? Filling in, what Asimov self deprecatingly admitted was the mediocre stuff between the Robot series and Foundation, haha! Tue Jul 26 16:51:27 EDT 2016 Wow, what a day. Things are coming together so nicely. This just might be being at the right place at the right time, having the right interests and the right availability of time to really dig-in, professionally. Wow, I just might be righting my wobbling, and settling into a gathering, rising storm whose fury is under my sails and wings and any other metaphors I can mix in. Riding atop this force of nature called technology trends, planting my flag on some of the greatest crests, hopefully... is me. From that perch, my purpose is to identify challenges, and spit off semi-autonomous agents that I have created to start breaking down aspects of the problem in to manageable chunks for other, smaller less abstract problem-inspectors to look at too. It's all very imaginable under today's tech. It's not too SciFi. Raspberry Pi really was the shot off the port bow... do I have that idiom correct. Anyhoo, I predict it. I see these things. I never cash in, but I need to, and I need to be exactly that sharp in my industry, and willing to go all-in for short periods of time to establish myself as the genuine noosphere homesteader, at least as far as a semi-practical implementation goes. And that's where I am. Jupyter Notebook (and soon, Jupyter Labs) under Anaconda is very nice, but at some point, I'll try to re-spark an interest in Levinux, controlling the QEMU binaries like I never have before, and therefore, able to modernize it quite a bit... connecting less-brittle dots. Teaching people how to automate the building of servers, so that PaaS vendor-beholden API abstractions with no-source blobs inserted here and there to make it all possible being totally expunged from the system. That's how it was with Raspberry Pi and the graphics coprocessor bits. Embrace proprietary to help you be more effective, but do not use it as your entire foundation. That would just be silly in the era of the rise of free and open source software. RMS would disapprove of that being the chosen label, but hey, it's a label that sticks. Let's stick it. Tue Jul 26 18:05:27 EDT 2016 Heading out now (at a reasonable time), and will do some work at home tonight. It's so hard to get the motivation to do that sort of stuff, but gathering storm and all that. Gradual organization cutting across ALL aspects of my life is one of the signs of the gathering storm... and I have to MAKE them happen. To not be futile against the storms life throws against you is what it is to be human. So go from thought to implementation more rapidly and iteratively. Commit. Push. Leave. -------------------------------------------------------------------------------- ## Mon Jul 25 09:49:41 EDT 2016 ### Flattening Hierarchy All data is either hierarchal or tablular, and then all tabular data is actually hierarchal, because it exists on some digital storage media, which exists in some protective housing device, which is accessed by some reader which is connected to some interface which is in some larger device, which probably exists inside some computer or data center, which are all nested hierarchically on the surface of our planet... simple. Hierarchy wins in nature's preferred formats, and even at the molecular level, things seem to lump up into hierarchal bits. Sometimes you get crystal lattices (more like tabular) but once again, those are rare and ultimately lumps of stuff inside other stuff. So again, hierarchal shapes win in general in nature as a prevailing pattern. Okay, the menu stuff is still your job #1. The other stakeholder is back this week, and I want to / have to be ready to do the menu dog and pony show, without ever being caught off-guard. And there's now so much data stuffed into the menu program, it now makes sense to move it all out into a spreadsheet, for greater flexibility and awesome demonstration / control purposes. Mon Jul 25 13:43:23 EDT 2016 I feel a bit of a refactoring coming on. The request is for it to just be one level of hierarchy in the menu. That is, one set of parent nodes and one set of children nodes. I suppose the whole original endeavor of figuring out what those original nodes and shape was worthwhile mostly now just for "having process". We started from a legit corporate starting-point. And that sort of objectivity is nice. But it's not the real way here. The real way is in the great flattening, and the nodes are all sort of in-battle to be the selected top-nodes. And THAT is a very different picture than what we had before -- VERY different. My original approach was a good one, but seeing as how the hierarchy no longer fits, it makes one wonder whether the exercise was worthwhile in the first place. And the answer is, of course it is! Yet, there is some re-factoring to be done, which in the end will make all the difference. This is where mistaken blind scrambling to adapt rendered-irrelevant work to be relevant again often occurs. I guess I will be doing a version of that. But I will preserve story. I will insert and annotate, and make Jupyter Notebook my PowerPoint. This will be awesome. Mon Jul 25 20:27:25 EDT 2016 Okay, heading out how. Have to call Adi and say goodnight before I head out so it's not too late by the time I get home. I have a very small set of things still to do from home, but I totally overhauled the menu logic to always produce only one level of menu expansion. It does scroll vertically a bit, and we can roll-up and consolidate. But I'm in very good shape for tomorrow. De-duping against URLs is probably the main thing, and I could even do that from home tonight if I am so motivated. -------------------------------------------------------------------------------- ## Fri Jul 22 14:15:24 EDT 2016 ### Heading Out Okay, now to actually get today's deliverables out before I run out of time. I had to decompress a bit... this week so intense! Don't cop out on the final mile. I need to reorganize hierarchy to be more shallow AND get in some more links from an explicitly provided list. You don't even know what you don't know yet, and Adi will be here for a pickup at 6:00 -------------------------------------------------------------------------------- ## Fri Jul 22 09:39:11 EDT 2016 ### Jupyter Notebook about to become Jupyter Labs Thursday Cheese Steak and Friday Morning Bagels. Sometimes it feels like ZD is feeding me. Guess that's sort of like a step towards those high tech employers that feed you for free at lunch. I still remember my Commodore Cafeteria days. Wow, my exposure to that world was totally before the Dotcom explosion and bust. I am actually quite old-school -- though not quite as old-school as those engineers from the 60s and 70s who were there inventing it, who are still around. Met one from Bell Labs at Jane's birthday last Saturday. Fascinating listening to these people. Their very brains are future-proofed, for the very reason that they understand things at that low level. Jupyter Notebook is enabling reproducible science. Data Science... SEO Science? no, I'm an artist. I use data in my art, but I'm an artist seeking to be as objectively powerful in my data-motivated artistic creations as I can be. This is not to say that anyone is paying me to be a Data Artist. That's just how I see my (much maligned) field of SEO (or, "search engine optimization"). That is to say, I help companies get the pages from their published websites found in search -- principally, Google. Huh? More specifically, I get my client's media in front of people at the very instant they're making inquiries on that topic. Currently listening to Brian Granger over YouTube talking about Jupyter Notebook, and how there's probably close to 3 Million users, and how it's trending up in Github. Wow! JupyterLab alpha talk from SciPy 2016. Jupyer is poised to become much more than Notebook... in amazing ways. I need to look into the Terminal and Text Editor and alternative user interface capabilities in existing Jupyter. The existing Notebook will become the Classic Notebook, and Labs will become the primary user interface, so this is a big push forward. Jason Grout is giving live demos, but I basically got the point. Wow, it will even have fuzzy searching built-in. Friggin' cool... as if Notebook weren't a big enough step forward. Mention that in today's video. Okay, I have some work still to do today on the Menus, and I so want to do videos here. Maybe I'll crank out the one easy video, then think about the JupyterLab next steps. Built on top of https://phosphorjs.github.io/ and is about to reach 1.0. Chris Colbert created it. Pluggable, extensible platform. Primarily browser-based, it seems. Widgets and plug-ins that extend the environments in a number of different ways, including ways of rendering documents like markdown file-to-HTML rendering. Side-by-side editors with completely live-links, such as between the text editor with markdown and the HTML visualization of that markdown... very cool! Okay, Pipulate was born to live inside this environment. Fri Jul 22 11:27:39 EDT 2016 Okay, pushing out that video. Make sure to get IT uploaded to YouTube before your morning walk video, which is also awesome. Target Dictionary Comprehensions on this one... nice. Gotta get to my main work today -- drive this last bit of menu work home ASAP. Had to get this talking head video stuff out, while all that eval stuff was fresh on my mind. -------------------------------------------------------------------------------- ## Thu Jul 21 09:29:56 EDT 2016 ### Today Was a Good Day Wow, my work LAST NIGHT actually at home hit home so many important nuanced issues about the Menu project that I'm in a much stronger position today, and I do believe now it's all about hitting the iron while it's hot, to get a rocking-cool status of the menu for the meeting later today. It's all about shaping the selections! 1. Add a series of nodes into the correct locations (without hyperlinking them) If there's no URL data, don't try to construct the hyperlink. Done. Add nodes to the correct locations. Thu Jul 21 11:01:11 EDT 2016 Okay, wow. Implemented most of what I wanted, and am on the verge of an empty leaf node filter. Thu Jul 21 18:11:35 EDT 2016 Wow, what a day. Mission accomplished on so many fronts. -------------------------------------------------------------------------------- ## Wed Jul 20 11:37:45 EDT 2016 ### Super-focus on Menus, both Data and Process Okay, I made my goodsheet and serpchiver repos on Github private, and renamed the urlookup repo into SEONotebook. I also capitalized the L in the levinux repo and the P in the pipulate repo. The URLs that are given out do change, but the lower-case versions still work, and it looks much better on my profile page this way. I lost quite a bit of time getting my next generation Pipulate up and running, but it only took me a couple of days this time, instead of a couple of months. Experience, knowledge and diving deep into the details and nuances of Python can make a huge difference -- as does re-basing my work on Jupyter Notebook instead of generic Linux webservers running Python. That makes so much difference, leaning on front-end development work that much less. I also lost some time this morning creating that animated gif of the SEO pulse report graphs across the tracked properties to make visual the changes recently that have been going on in the search results, mostly due to algorithm tweaks (but also with some Pokemon Go and e3 stuff mixed in there). Anyway, I've got about an hour before a menu meeting with the boss later today, and I want to get things in better order. Shifting of gears -- urgently! You will be best served with some menu optimizations -- specifically, local caching so that you don't have to execute SQL queries over again for re-generating the menus. There's so many distractions all around me. Ugh! I need to focus, focus, focus! The biggest time-saving comes from caching the SQL, especially the 2nd SQL query, and so I'll use os.path.isfile to check for the pickled dataframe object from Pandas, and if it doesn't exist, pickle it into location. Okay, done. That's a HUGE speedup. -------------------------------------------------------------------------------- ## Wed Jul 20 09:23:44 EDT 2016 ### The Missing Piece is (has always been?) Jupyter Notebook for cutting the Catapult Ropes I'm getting a walking-to-work video out today that I'm very happy with, talking about my latest work including: - The evolution from Tiger to Pipulate to SEO Notebook - Clarifying the motivating forces behind each evolution - Describing how I always come back to using eval - Describing how I combined GoodSheet, Pipulate and SEO Notebook - Describing how I'm importing .ipynb files as if .py modules to do this And THAT is a good video. Wow, can't wait until I weave the embedded video code into this file in the correct locations, chronologically. Wow, big little steps. Winding the catapult... but FOR REAL, this time. The missing piece is (has always been?) Jupyter Notebook -------------------------------------------------------------------------------- ## Tue Jul 19 09:30:18 EDT 2016 ### Wow, Importing .ipynb files as modules... big move forward Wow! It seems like a week later. Had a Daddy/Daughter date night. Saw The Secret Lives of Pets at Union Square, and went to Barnes & Noble and got the Lego Arendelle Castle set that she's been asking for for awhile. I like to indulge her Lego propensities. Wow, another trend I totally nailed. Now, if I could only cash in on my precognitions. Sheesh! Okay, I'm sitting on top of a few. And I'm on, what, like iteration quintillion of my generalized system? Shhesh! Okay, now make it popular, your technique, your method. Pipulate is mostly an API, for what is an API but a set of conventions to which you abide so that when you put input into a system, you get output out of that system of the sort you were hoping for and predicted? THAT is the purpose of Pipulate, and I made a killer logo for it, gosh darnit! This thing lives. It lives now through Jupyter Notebook. Be sharp, and don't forget a thing! Do the work that needs to be done in the way it needs to be done, and don't let yourself get derailed by any of the old traps identified from the old patterns. YOU ARE HUMAN, and as a human, you are subject to all human foibles and contain potential for greatness as great as any human, or human-like life-form that has come before you. So, be at least just a little bit great, will you? Don't be a bozo. Don't screw this up like before, and before, and before, and before. Yikes! I'm not good with crisis situations, am I? No, I just survive. I don't tweak for an optimized future much, do I? Well, that changes now. This is me becoming that next level of self-aware, and taking control of the reins. Tue Jul 19 18:27:04 EDT 2016 Wow, just re-implemented Pipulate, essentially. Wow... Jupyter Notebook... wow, importing .ipynb files as modules, the way you would a .py file. Wow. -------------------------------------------------------------------------------- ## Mon Jul 18 09:57:17 EDT 2016 ### Re-Implementing Pipulate-like Functionality Jupyter Notebook Well, there goes another weekend. I'll be seeing Adi very early today -- leaving at 5:30 PM, and this is after getting in late today too. Oh, well I'll make up for it with an extra-productive concentrated day. But don't give yourself a headache again. You really are biting off more than you can chew. But you did that over 10 years ago, when you got married. Maybe you did it the first time you took a girlfriend. I am a boiling cauldron of potential, with only just the final few ingredients missing for me to become a powerful world-changing chemical reaction, of the Jobs or Bezos variety -- ACTUAL creative/original thinkers, and not just the clever algorithmically inspired clever thieves like Gates. It's better to be second and win than to be first and lose -- unless you can pull back into first position by repeating the trick that got you there in the first place -- over and over and over, such as Apple often does. And so, I'm the stew trying to look outside of itself for the ingredients that I need to complete myself. And that takes some stewing over... but at 45 years old, I'm running out of time fast. Is it REALLY just a matter of connecting the right dots correctly? No, you also need to keep yourself a little bit better organized. At least try to be that organized at work. Let it seep back out to your day-to-day life. Lead your home-life by your work-example. That way, you continue earning well, which is always your path to reduced-stress continuity. Don't go broke or lose your earning capacity! I have to pick up right where I was seamlessly. I actually left at a fairly good place on Friday, with processing lists of URLs... no... I didn't get to processing the URLs. I only got to pulling them. What about the update back? Can my list_of_list be updated back? Mon Jul 18 10:46:26 EDT 2016 Use your nickname strategy to always NICKNAME your current work. That could be a big work-flow breakthrough, really. For example, right now I'm wavering a little bit about how to organize my efforts around next steps. I got a list_of_lists out of a GSheet through GSpread, but it's not a cell_range, and so can't be updated back in batch, and that's bad. Select a cell_range in a fashion as similar to a list_of_lists as you can, and inspect it. In fact, inspect the list_of_list to assure yourself it's not actually a cell_range in disguise. I need more "Operation [This]" and "Operation [That's]" in my life. Mon Jul 18 14:16:25 EDT 2016 Okay... time to think about APIs. This is the chance to make BIG moves forward in terms of simplicity and such. Mon Jul 18 14:35:36 EDT 2016 I want to think through this next step as intelligently as I can, because I will have to live through the ramifications for a long time (again). Mon Jul 18 16:56:58 EDT 2016 Okay, time to commit and go meet Adi down in the lobby. I guess it's an okay stopping point. Had 2 last-second requests come in to send stuff out. Racing to do it now. Adi and Rachel waiting downstairs in lobby. Raining outside. -------------------------------------------------------------------------------- ## Fri Jul 15 10:48:14 EDT 2016 ### Code Cleanup Day Okay... today. Today is a gift. I have a few questions to answer. Do those quick and do those first. Leverage Jupyter Notebook where and when you can, making EVERYTHING FUN! I can't imagine I'm being paid to do this sort of stuff. So make the daily tedium tasks JUST AS FUN! Fri Jul 15 11:56:10 EDT 2016 One of the most important projects lurking about is to do lookups against an explicitly provided set of URLs. Let's do all the standard GA, SC and now even PostgreSQL lookups against these URLs, getting back all sorts of datapoints and statistics to inform our decisions. Don't let that be a capability blind-spot anymore. This is the most important thing you can do. And start to knit in a whole bunch of other things too, such as: - Retrieving the full-HTML "snapshot" to field-stuff store in spreadsheets (of smallish numbers of rows!) - Doing a target keyword (multi-word terms) analysis of those pages - Simple title-tag retrieval - Something about the link-graph - SEO difficulty scores, from such APIs as SEMRush (and KeywordFinder?) - Something against the Twitter API -- perhaps, the most re-tweeted hash-tags on related topics - Something Machine Learning (first baby-steps) Okay, don't get too ambitious right out of the gate. Let one thing build upon the next upon the next, and make each thing wonderfully Pythonic and a joy to talk about and demonstrate in its own right. Finish what you started with GoodSheet, which is exactly what you need to do in the first place. Should I rename it HappySheet? GoogSheet? The transition from GoodSheet --> GoogSheet... implies a simple and subtle, yet profoundly awesome animation! Fri Jul 15 14:07:50 EDT 2016 Can't believe how fast the day is flying past. Awesome discussion with Marat. I like having a counterpart on the other side of the country, at IGN. I'll have a friend in San Francisco to visit one day with Adi. But now, I need to get to work and knock off a deliverable and a permanent capability expansion. -------------------------------------------------------------------------------- ## Thu Jul 14 08:15:55 EDT 2016 ### High Pressure Day, but Turned Out Okay Merging getting easier. Of course, I'm just coordinating my daily journal writing with myself, so it's not so challenging. The fact it's reverse chronological doesn't hurt either. Now to work. Uploading a video that I'm not sure I'm 100% happy with, but it is sufficiently vague enough I think to be fine. I am not a Data Scientist. I am a Data Artist, I believe. But an Applied Data Artist -- and not just for visualizations, either. It's when the shaped data has to get fed directly back into the system as input to help create a positive feedback loop. I think I forgot to put the work I did with Search Console while at home into the private menu repo, but that's okay, because I have the data file. Use all your safety nets, and don't lose focus of the prize. The prize has to be front-loaded with 80/20-Rule easy-picking's work this morning. No chasing rabbits! I did that enough for this project -- BUT for the better, I think. 1, 2, 3... 1? Oops, I take all that back. I just had to stash my local changes before a git pull. I have the new searchconsole.ipynb file -- which is awesome in its own right, and should be the subject of a video, BUT I need to get to work USING the data right away. My machine started overnight. Make sure the reports run... okay, done. Even now, faux pas avoidance. But drill on. Next? Oh yeah, SQL Workbench/J, which I incidentally got working on my desktop Mac at home, connecting to the corporate database with OS X's built-in VPN client capability -- woot! It was VERY helpful. Pure SQL is sometimes nice too (in addition to Jupyter Notebook and Python). Okay, I blended the hand-picked items from the striking distance report that I drew out of the "not provided" still provided keyword data, which gets around the Search Console pre-processing... nice. Will be noteworthy later, don't forget. Next? Extended the conditional color coding for VERY easy explanation. Now, I need to be able to pull in that data -- maybe as one giant data object in Jupyter Notebook using GSpread's dump-it to a list-of-lists trick. Yeah, let's do that RIGHT AWAY. No time to deliberate. Thu Jul 14 09:32:12 EDT 2016 Critical juncture time! Focus & do. Walk the OrderedDict -- something I'm already doing to get the node count. Inspect that bit of code. Report generating failed. Manually setting virtualenv and running from shell. Nice to notice not noticing Dropbox notices popping up on all my various machines from which I stay logged into that same Dropbox. Okay, pshwew, faux pax avoided, round 2! Now, for avoiding the biggest bozo faux pas of all. Scalpel cuts... Chisel strikes... Hand of the surgeon, hand of the artist. Go! Thu Jul 14 10:56:18 EDT 2016 Okay, I just did some housekeeping, like touching base with the boss, catching him up with progress and the plan, and sending out a meeting invite to the menu stakeholder, before she's out for over a week. This has to be an impressive "selling" job, and for that, it just has too look and feel and work so right. You have just one more bit of inspired break-through to do. I just created an OrderedDict nested-walk copier... woot! That's a huge piece out of the way. Now, I have to report on some sort of position as I go, so that I know when and where to insert in new nodes. Think! 1, 2, 3... 1? Identify leaf nodes, and put a self-targeting hash link, just to see a href's start to appear, and call special attention to leaf nodes... Okay, done. Brain buster, that one! Committed to Github, and re-executing from beginning. Take a deep breath... next? I could speed this up a lot not hitting against SQL every time I clear and re-run. Thu Jul 14 13:34:35 EDT 2016 Okay, I got caught in a rabbit hole at the worst possible time, and it's already 1:30. I really need to do this all in the next hour, in the way of making it all look and feel very real. Don't screw around. Time is up, and you have some major thought-shifting to do. You HAVE to make the hyperlinks on these things. Do that immediately! Thu Jul 14 14:35:35 EDT 2016 Wow, some of the dots are starting to connect! Less than a half-hour until the latest reasonable time I can show my boss BEFORE preview to stakeholder. Time is just about out. You have to make a few very special chisel strikes right now, so as to not look totally like a bozo. 1, 2, 3... 1? Nailed it. -------------------------------------------------------------------------------- ## Thu Jul 14 20:52:52 EDT 2016 ### Take That, You Nothing Better to Do-nicks Wow, I can't believe I didn't commit and push today's work journal entries that were actually from work. Well, tomorrow's merge will eventually put that here in correct chronological order. It's funny to me that some of you freaks will be reading this here first, before today's earlier writing is merged into this Github repo tomorrow, and thereby published, automatically by virtue of the amazing Github.io publishing system built into the already quite amazing Github system. Let me tell you folks, this is all connecting together into something special, and I'm here to wire up a few neurons through synapses or whatever singing doo wa diddy diddy dum diddy doo... I like to type, I like to rhyme, I like to do it all the time, singing a vi vim nvi emacs, only if you're genuine. You follow fad. You do it all the time. But I do to, but in I chime. The Raspberry Pi did it for me, just like the old Amiga did in it's day. I'm genuine, I think you'll find, quite second tier or maybe third, with even the great geniuses only riffing on other people's ideas, even if they do vastly improve the very thing they satire. This is why the great text editor wars of yor were so, and still continue to be to this day, one of the great genuine measures of your creed on this day. Both vi, particularly in its modernized forms such as vim, which come with syntax color-coding and other amenities many feel necessary in a modern environment, or emacs which never needs to be modernized, because it was born perfect an all LISPy, evil mode clinches it. emacs wins, if you can stand the Carpal Tunnel Syndrome. We in the church of vivivi feel it is best to not bother with such arguments, because that time is better spent being at one with your text file, navigating our way around that strange always-there, always really running things humble text-file, the source of all code. And is it really even a file? Hmmm. Think about what a computer operating system really is, and what its largest roles and responsibilities are. If graphics shells including the desktop and the browser and any local program user interface is supposed to be operating system independent, so that the code can run anywhere, then what friggin' role does an OS even have these days? If everything else is delegated out to highly abstracted interchange parallel-processing optimized components, at the end of the day the OS really only has to get the silly machine running in the first place, and handle all the messy local physical reality stuff of like reading information from disks and memory and writing it back into disks and memory. And why bother to blab about text wars? If time is so precious, then why are you spending your time reading this? And why am I spending my time writing this? I mean like, it's not a book or anything... is it? Or is it just performance art, because just as I like to talk, I like to type. We Enjoy Typing... WET. Some like it DRY. I say, some like it DRY. I say, some like it DRY. I like it WET. I like it as WET as you can let this all-WET jest get. 'Cause it jus' keeps getting WETter and WETter, 'round 'ere, cause WETer or not you get it, it gets WETer, I say... much, much WETter. Fine, now how was that? Pointless enough. But I warn you, before you criticize, you're going to have to admit that you're one of my readers -- especially if you comment on this one very particularly very very meta statement right here. You know, the one about meta statements. I think I've been watching too much Rick and Mortey, to tell you the truth. I like Rick Sanchez. I like the story-lines. I like that Adi is going to grow up knowing all these popular SciFi tropes from out of the gate. I am a geek. I am most certainly a nerdy, Python-loving, SEO-evolving, Awesome Daddy'ing due who doesn't shave and keeps my hair in a pony tail, because it's all the more time for coding and kitties... and a little bit of writing, that you're going to have to admit you actually read if you actually try taking pokes at me. Here's this guy... what wait, you read him... ? What, you mean you read him too? I mean, how could you? He just writes so much, and it's all so boooring! I mean especially like this part right here. Commit and push? Yeah, commit and push. -------------------------------------------------------------------------------- ## Wed Jul 13 08:59:41 EDT 2016 ### Today, Don't Screw Up! Okay, today? Don't play. Display competence. Be transparent. Be communicative. Don't let your thought process wander off. Fall in love with those aspects of today's work that must transmit that love-worthy feeling to others. And so... and so... 1, 2, 3... 1? Definitely getting the Menu object that I'm programatically creating to display as an actual expandable/collapsible tree user interface component... in Jupyter Notebook? Hmmm. That would be the ideal situation. Research! -------------------------------------------------------------------------------- ## Wed Jul 13 21:05:51 EDT 2016 ### Seat of my pants Expanding your reach. Exponential growth. The Kurzweil interview with Neil Degrasse Tyson. Wow, is my thinking in sync with his. Yup, we are dealing with the sub-functions of neurons... of course. We're ages away from a human brain equivalent computer walking around in a robot, but we're already there for simple tasks like keeping it balanced. So, insect and lower-form intelligence will come first, and machines will develop animal-like instincts and intuitions. The reptile brain won't be far behind, then emotions and all that spiritual mumbo jumbo. I think I'll like the reptilebots more. Wed Jul 13 21:54:07 EDT 2016 I think my headache today was from pushing myself too hard on too many fronts, and feeling stressed from too many fronts. And with good reason. New job that I'm determined to make successful and long-term, coupled with wanting to be the most awesome dad for my daughter, even as the marriage falls apart. I saw a crazy person on the subway this morning, and distinctly thought to myself, that's what happens when someone can't hold it together, and how truly close we call could be to that state. It's just a matter of slipping a little bit... and slipping a little bit more... and more... and more. And so... and so... I have my time now, and I will not squander it. Nor, will I sacrifice my health. I will strike the perfect balance and compromise. Wed Jul 13 22:49:13 EDT 2016 I have to remove all slight resistance barriers that make distraction more appealing than focus. I guess I'm always focusing... I'm just focusing on the wrong things. Thu Jul 14 06:43:45 EDT 2016 Heading to the office now. This is not a new journal entry, because I have not slept. Iterating closer, but this is not moving in leaps. Hail mary this AM. Design the rest in your head on the way in. Look at the code. Just take pics with your phone of your desktop screen. Don't try to code on the subway. Meditate and contemplate and hit the office with rapid-to-implement ideas ready to go. -------------------------------------------------------------------------------- ## Tue Jul 12 20:24:53 EDT 2016 ### What Matter I am still loving the Culture series. Currently listening to "Matter", which the online series says is not a Culture series itself, but I think that's wrong. Set in the Culture world is set in the Culture world. I'm listening to it on Audible, which I paid a ridiculous $14 or something when I bought the book-form for my Kindle. Salvation is in Statistics. Tue Jul 12 20:36:05 EDT 2016 Just called Adi to say goodnight. -------------------------------------------------------------------------------- ## Tue Jul 12 11:33:42 EDT 2016 ### Reading and Writing a Million Lines in Python Wow, didn't get the journal going until 11:30 today. Looked like an all-clear on meetings, but actually just got invited in. Keep on top of the topic, but start researching how to go from tablular data to more menu-like nested data. I'm guessing Stack Overflow will be my friend. I like the pattern that's getting created with this new walking, Jupyter Notebook video rhythm. Each day, I could do: - One walking-talk video that sets the state for the coming tutorial - One talking-head video that walks through a tutorial on that topic This has a good "get it out of my system" dynamic, building-up the start of the day as something I very much look forward to. I can use my time walking to the subway and even on the subway to plan the video I'll be shooting walking from the subway to work... and then hit the ground running when I get into the office. I love this approach. It fits into my natural rhythm, and always puts something exciting front-and-center in my mind. Today, it was generator expressions, which are indeed very exciting. I needed to make a file with a million lines: # Write a million lines in Python with open('data.txt', 'w') as f: for i in range(1000000): f.write("%s\n" % i) # Read a million lines in Python, technique #1" with open('data.txt') as file: for i in file: pass print('done') # Read a million lines in Python, technique #2 for line in (x for x in open('data.txt')): pass print('done') Okay, time to finally commit this journal entry. Tue Jul 12 18:38:54 EDT 2016 Heading out for the day. Got the really tricky stuff done. Wow, this menu project could end up being a really big win. -------------------------------------------------------------------------------- ## Mon Jul 11 09:16:49 EDT 2016 ### Have To Work More Effectively! Back in the office again. Felt like a very short weekend with Adi. Gotta go see The Secret Lives of Pets movie with Adi. The SEO Pulse Reports csv files still generating flawlessly from my laptop in a headless virtual machine. Mon Jul 11 10:05:44 EDT 2016 Okay, got my Weekly SEO report out. Feels good to get that out inside the first hour of Monday morning. Good process, and the best flow of regular updates I probably ever did for a boss. This may be the first boss I've ever had who had a really strong vision of what he is doing and is actually pushing ME forward, instead of just hoping I do the right things for the right reason because I'm so far ahead of the bosses (all prior employment experiences, with the possible exception of Acronym). And I like it. I may be even more effective being used as a directed tool, so long as I keep my intelligence firing on all cylinders. Don't let myself "go dumb" by following other peoples orders. Solve the puzzle that exists before you at this moment. Take this privileged opportunity to create Pipulate 2 (3?) WITHOUT developing it. Hmmmm. Just live inside Jupyter Notebook, and solve that original API problem. Mon Jul 11 17:23:57 EDT 2016 Day almost over. Ugh! Where did it go? And I'm meeting Adi in the city afterwards, so I'm going to have to jump out of here at 6:00 PM sharp, and I am not as far along on this Menu thing as I need to be. Shit, okay, don't get stressed. You have to knock this thing out of the ballpark. -------------------------------------------------------------------------------- ## Sat, Jul 9, 2016 9:14:16 PM ### Bonding With The Kid Over Ice Cream and Chowder Enjoying Karamel Sutra Ben & Jerry's with Adi while watching the Thrice Cream episode of Chowder. Just installed Anaconda on my Surface Pro. Now to run it first from the icon they put on the start menu. -------------------------------------------------------------------------------- ## Sat, Jul 9, 2016 8:16:37 PM ### Homesteading in the Noosphere, I am... Hmmmm (in the voice of Yoda) Getting close to her getting tired. Second meal. Same as the first, re-heated, and hoping she actually eats something. We woke up at I guess somewhere after 9:30 AM. Experimenting with fairly different "laptop" arrangements that the MSP4 suggests, like laid-flat against your whole-lap as if a laptop were opened well past it's clamshell norm to lay flat on both backs. Adi's on my phone with her beloved Baby Panda series, this time Animated Stickers. I definitely want to do my Advanced Python article now. I have a lot of good stuff accumulating. I will be data-mining THIS page, essentially to "extract" articles and ideas and projects. It will also serve as proof when I have ideas first in that good ol' noosphere. Homesteading in the noosphere, I am... hmmmm -------------------------------------------------------------------------------- ## Sat, Jul 9, 2016 1:33:57 PM ### Python, Robots, Kids & The Future Every time I go to type a word or two, Adi acts up. I have to keep in mind that this sort of activity, Adi deems as competition with her for my time. Little does she know I'm thinking about her constantly, even as I type about work and my job. My mission I think is to help kids grow up with Python. Step #1: Build up the mystique of doing coding... robots, magic spells. Let her know there's something special about this skill that has to do with basic reading, writing and math skills for the future. Maybe even more important, because it ties together reading, writing and math in a crescendo of machine automation. We humans need a little help. It's up to us to create our helpers, to help eradicate suffering and unhappiness. Thanks to robots and machines, everybody should be living like a heaven on earth -- even if that makes us deliberately keep systems in place to challenge ourselves enough to keep us at least that strong... maybe even stronger. Just as digital photography and video have improved our human memories, so will digital learning improve our reasoning abilities. We will be able to look at and question our world as if it were data to be queried. Enough data has accumulated up in enough ways, and been machine-analyzed in enough standard pre-processed ways that answering whatever likely questions come up will be quick and easy. The question language parser will see to one part of that challenge. It's local intelligence about how to interact with the global query language takes care of the next part. A back-and-forth ensues, a constant exchange of incrementally more information and alternate query drill-down possibilities. It's like command-line completion with the tab-key, but for all things in life. Life's IDE. We are all developers, basically developing ourselves and our children and perchance, our society and culture. Sat, Jul 9, 2016 3:41:43 PM Operating systems and graphical user interfaces are the wrong place to concentrate when it comes to important long-term skills. Making killer apps for a particular popular hardware platform is like competing for the big-bucks of pop-culture hits, and there's really nothing wrong with that. The world needs its Instagram and Angry Birds billionaires. But that's not the same thing as day-to-day obsolescence-resistant technical competence -- a much less sexy proposition, but a good and important starting point and baseline level of tech skill none-the-less. When you like to type (and like to think), entries like this from time to time throughout the day is no biggie. The trick is to feel comfortable on almost any hardware, anywhere and under almost any conditions. It's not just about one thing -- like Python. Even Python itself is so many different things. When I say Linux, Python, vim and git, I'm actually talking about so truly many things. Just talking about those... plus, occasionally JavaScript, RegEx and other supporting technologies... will be plenty of material to make an identity and a specialty and an income out of. -------------------------------------------------------------------------------- ## Sat, Jul 9, 2016 12:00:34 PM ### Become Invaluable Then Re-write The Rules Wow, it's been a long time since I've used my Microsoft Surface Pro 4 for journal entry. Here's my git pull into this directory: Warning: Permanently added the RSA host key for IP address 'xxx.xx.xxx.xxx' to the list of known hosts. remote: Counting objects: 151, done. remote: Compressing objects: 100% (37/37), done. remote: Total 151 (delta 61), reused 39 (delta 39), pack-reused 75 Receiving objects: 100% (151/151), 942.62 KiB | 171.00 KiB/s, done. Resolving deltas: 100% (97/97), completed with 2 local objects. From github.com:miklevin/miklevin.github.io 1aae1b2..daad127 master -> origin/master Updating 1aae1b2..daad127 Fast-forward index.html | 2249 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++- journalmagic.ipynb | 46 +- 2 files changed, 2277 insertions(+), 18 deletions(-) 2277 lines inserted since my last time on this machine, and I see that indents are still set at 2-spaces (pre my PEP8 compliance days), so I need to fix that. Okay, fixed. Adi opened all but four of her new Yo Kai Watch series 2 mystery packs. She wants to stay inside today because of the black on her lips from the dentist. She's having two cavities filled. We have to be better about getting her to brush her back teeth. She's good raw material, but habits need to be formed through repetition. But if you lack buy-in that the thing you're trying to institutionalize into your life (brushing, throwing out trash, etc.) is fundamentally wrong and objectionable and you have a fight every time, then there's serious issues about long-term healthy adult-hood. One simply needs to brush and get rid of one's own trash. Without doing so, you will always have problems, and being thrusting needless problems onto the people around you, causing constant stress. I need to use things like the Yo Kai metals as rewards for her doing stuff. At least I got her to throw out all the wrappers and packaging of the metals in order to get the last (but for the 4 I set aside for Pop Pop) pack. That worked well, as did working WITH Adi to wrap her beloved loose coins all over my bedroom floor into coin wrappers. Those coins are staying that way now. I need to think more and plan more. I'm still barely keeping my head above the surface. I need to make the time that I AM and that I DO spend on things more effective. And then I need to fill in the scraps of time here and there with real chisel-strike type of progress. Chip away, chip away, here and there, with whatever little bits of time you can cobble together. My work-schedule is intense: 9 to 6 with a 1-hour commute on either end, making it 8 to 7. That's 11 hours a day for an 8-hour job. Hmmm. Become invaluable and help rewrite the rules. -------------------------------------------------------------------------------- ## Fri Jul 8 14:36:18 EDT 2016 ### Another Tutorial (still for co-workers) Okay, getting later on Friday. My 2 earlier videos were really solid, and I feel the itch to push out one more on general list and dictionary issues. What I think I'll be doing is creating the dictionary I need to convert column numerical indexes to their Excel spreadsheet-style letters (columns Z, AA, BB, etc.). It's something I've been hardwiring for awhile in the GSpread locations where I needed such an index, but I think I'll do it more eloquently for a tutorial. -------------------------------------------------------------------------------- ## Fri Jul 8 08:45:40 EDT 2016 ### Rebooting Learn Python Endeavor for Co-Workers Every day at work now is part of that journey. 1. Install Anaconda 2. Start Jupyter Notebook 3. Don't have to learn vim Okay, I shot and published 2 videos today, by 10:00 AM. I produce more quality online content by 10:00 AM than most people do in a lifetime -- haha! I think it's truly forgoing the editing step that lets me keep up this pace. One of the two videos is great to help Marat and others get started with Anaconda, Jupyter Notebook and Python. This next phase of my online presence is all about accessibility, and the ability to carry out amazing feats you would have never previously thought yourself capable of. I get that feeling with Python almost every day now. I'm currently exporting the video from ScreenFlow (old, but sufficient version) to a file-format that I can upload easily to YouTube. I'm also trying to make a point to share my videos out there pretty regularly over my other channels, such as Twitter, LinkedIn, Facebook and Google Plus. Fri Jul 8 10:21:21 EDT 2016 Okay, I should really think through how to make today super-effective. How to minimize distractions, work on the right projects for the right reasons, and make good progress. Fri Jul 8 11:23:17 EDT 2016 Okay, I'm getting interested in the mitmproxy package on Python. It's for sitting on a proxy and inspecting the http communication, and optionally modifying it. It's the type of thing I need to do on my next step to start really understanding all the http traffic going through a site and doing some next generation SEO projects, where I automatically create 301 redirect maps keyed off of actual site traffic correlated with Search Console URL-to-keyword search mappings. But focus for now. Get the Marat request done. He gave you a list of URLs to look at and process for the bad bounce report, and he'll be into the office soon. It's a small list of URLs being pulled, compared to how many are being pulled in the greater report. Okay, I'm running the Bad Bounce Report on Windows. I had to add Google Analytics scope to the GoodSheet repo. Oops. Okay, and scipy had to be installed with conda on Windows. Interesting. I should take it out of the pip freeze requirements.txt and add it to the stuff to do with conda. -------------------------------------------------------------------------------- ## Thu Jul 7 21:46:29 EDT 2016 ### They're "virtual screens", not "virtual desktops" - Git will make you scratch your head and nod appreciatively. Pshwew! - vim will actually re-wire your brain, maybe for the better. Jury's still out. - Python will astound and amaze you with it's ability to let you astound and amaze other people who are not themselves actually onto the Python trick yet. - Linux will confuse and infuriate you, until you get into the zen of short, piped commands. There's a whole host of supporting characters here, because no matter what I like to think, these 4 tools (tool-sets) and the relationships and work-flows they imply and enable are not the be-all, end-all of tech tools. Even if you love them all 95% positive, such as I do, the workflow can still itself be improved, and the polish, stability, support, blah blah blah of other successful tools is never to be ignored. The old trick is to just write Python API-mappings around it, with some sort of wrapper. Sometimes the hooks are coded into both respective systems, and sometimes Python simulates a user in a Python-controlled shell terminal, with tools like Pexpect. Point being, one way or another, Python's going to be interacting with it -- no matter what the "it" is. So, some of Python's supporting cast are: - Unix-like OSes. Gotta love the reliability of host platform conventions. - Regular Expressions. The time always comes when you want a powerful finder. - JavaScript. Billions of JavaScript-capable Web browsing devices can't be wrong. Another sub-language like RegEx, but for UI implementations. - BusyBox Linux. No matter the device, pop this on and things are familiar. - vi(m) text editor. Learn it. Tolerate it. Let it grow on you. Mastery. - Jupyter Notebook. Look what happens when Scientists got a hold of Python. - Anaconda. We can't always be running our code on servers all the time. - pip/virtualenv. If it's a dedicated server, you'll only need pip. - Markdown. The directly readable, easily type-able style I'm using now. - OS X's (soon, MacOS's) elegant full-screen, 3-finger swoosh best thing to happen to productivity since the Amiga's virtual screen system. People have to learn to stop saying virtual desktops. Many of these things I put into full-screen mode, like the Unix Terminal window I'm using to type into vim right now is full-screen with no desktop wigetry to be seen. It's a pure, undecorated full-screen immersive, flow-conducive setting. I would have never expected it from Apple, but I think the highly restrictive mobile platform challenges have illuminated for everyone those things that are so horribly broken about the "desktop" paradigm of software user interfaces on the desktop. Point is, they suck. I never liked Workbench on the Amiga much. I was an AmigaDOS person myself, with my own distro called ARPnCrunch -- for the AmigaDOS Replacement Project plus the Crunch binary compressor that squeezed the last few megabytes of space on that 800K double-sided (but not high-density) floppy disks. Workbench sucked. Windows sucked. All Mac OSes sucked. Only full-screen Amiga programs (they'd be called Apps today) really rocked. And today's Macs are rivaling it. The thing made virtual is screens -- not desktops. ------------------------------------------------------------------------------- ## Thu Jul 7 21:23:20 EDT 2016 ### text editor wars still being a thing says something It's becoming an ever-increasing comfort to be typing into this journal. I don't care much for impressing everyone with brilliant, well-planned out articles, and refined and edited results. No, instead I'm mostly just blogging (and now, vlogging a lot too) for myself, to sort of help document and process my thoughts. It amuses me that this attracts to me one or two crazy into me fans at any given moment, and sometimes there's an emphasis on crazy. Some are crazy-awesome, like Paul, while others are just plain crazy, like he who shall not be acknowledged. Wanna know someone? Look at what they find fault with in others. This reveals precisely their own deepest insecurities in themselves, for it is what both occurs to them first and annoys them most. They think about it a lot, maybe suspecting that same thing about themselves, but are too scared to do the deep introspection to diagnose and fix the flaw. Instead, they find fault in others, and sometimes even elevate it to a sport. The more one complains, the sharper their image becomes in the mirror they summon. Eloquence? Nah. Tired, and feeling the stuff flow out of my fingers, because we enjoy typing. We feel vim is one of the greatest idea processing environments ever invented, and the fact that the text editor wars is still a thing after all these years, is a testament to the battle-hardened, ultimately adaptable, and ultimate still relevant and applicable text editing tools to many a problem domain. Text editors like TextMate, Submlime, Notepad++, Edit Plus, Cygnus Edit (CED), and the like all run their course and recede into the dark corners of shrinking user-bases and no reason to take it up anew. Dead platforms, abandoned code-bases -- EVEN IF contributed to the public domain, with no emotionally invested project leads. That leads to things like PFE (programmer's file editor) which is a competent editor that went as far as macros, but not as far as syntactic color-coding, that enjoyed it's era somewhere in the 90's. But against this backdrop, there was forever vi and emacs -- relevant skills on always there and supported quirky and idiosyncratic, but still amazingly useful tools. I'm going to try to leave work early tomorrow so I can bring Adi to Cony Island to see the 9:30 PM fire-works, after her dentist appointment. I think she'll need a little cheering up. Ugh, I don't want to shuttle back-and-forth to Inwood just to get the car to pick her up and drive her to Cony Island. I'm going to drive my car to work tomorrow, and get cheap early-bird parking. Got some googling to do before I go to bed tonight, so I know what time I'll have to wake up tomorrow morning. At least we'll be spending the weekend here again. I feel like I made such progress last long holiday weekend, and that I can keep making progress with the weekend's fresh energy that's not getting used up at work and being all professional and Python today. Today, I made a YouTube video worth mentioning. Wow, is processing my journal going to be an interesting exercise to show off Jupyter Notebook with... wow! Pipulate 2 is going to just be with Jupyter Notebook playing the UI-role of Google Spreadsheet, but with an enormous amount of additional power with that superb IDE-like Python code execution environment. -------------------------------------------------------------------------------- ## Thu Jul 7 14:49:52 EDT 2016 ### conda virtualenv for a newb? Yeah, probably Wow, so distracted today. Already almost 3:00 PM! Can hardly believe it. And I want to leave early tomorrow. And I have a project still to do for Marat... let me slam that out. We will also likely be taking up some Python tutorial stuff again. Today has actually been a most useful day, if you consider promoting Python awareness (and SEO-factor awareness / separate topic/email) as being useful. Yes, yes. I do. Momentum. Build success upon success upon success. Okay, I want Marat to be able to run the Bad Bounce report himself, but with the SQL modifications that are necessary for this particular request. So, run it on your own machine right now... Thu Jul 7 16:31:39 EDT 2016 Okay, did a great clean-up job on The Bad Bounce Report. Now, extract it from the "core" repo, and give it its own identity. This is also a great opportunity to document the process EXACTLY for Marat by making a new conda virtual environment for it. Actually, I'm documenting it directly in the Jupyter Notebook and using pip, but for documentation's sake, these are the non-conda dependencies: pip install gspread pip install httplib2 pip install google-api-python-client pip install sklearn (pip install scipy) -------------------------------------------------------------------------------- ## Thu Jul 7 09:32:12 EDT 2016 ### Another Entry For The Books I want an @t macro in vim to force me to look-at and use my to-do list more. Whenever I want to RECORD a to-do item, I'd type @t and it'd set me up to start typing a new one. But not right now. Think through the experience you want to have with Marat this morning. Document in a 1, 2, 3 step procedure. Walk him through it. 1. I'm NOT going to show him conda virtual environments right away 2. The directory you're in when you start jupyter notebook becomes "home" 3. Files are easy to open when they're sitting in your home folder 4. Other folders sitting "next to" the one you started Jupyter Notebook from will be important to (have to be sitting next-to "home" for relative paths) I'll assume Marat is on a Mac, and I'll assume Anaconda is installed. What is the BEST next experiences to have? Hmmm. An introduction to Jupiter Notebook for the SEO? # On Becoming More Technical, from SEO to Robots ## An Introduction to Jupyter Notebook for the SEO ### Why Coding? Why Python? Why Jupyter Notebook? Coding happens. If you think you can get away from it, you're kidding yourself. So the main question is, if you're going to be forced into coding at some point in your life, you might as well get into the mindset to maybe love it -- because you might. I'd go so far as to say you should deliberately seek a path that yields enough up-front benefits to invigorate and energize you, but then just never stops delivering or lets you down. ### From Abstract Notions to Real Things Of course, I rig this "why coding" argument to be biased towards Python... because I love it. Your effectiveness in many branches of today and tomorrow's professional life will be determined by how well you can speak with machines, and after a quarter-century of experimenting of different ways of talking with machines, I finally found my way to one that feels natural and fits like an old, comfortable piece of clothing. Believe it or not, after awhile, Pythonic thoughts will flow from your mind as naturally as speaking in a native spoken language, such as English. And this is when the advantages of Python REALLY start to kick in, because you'll find yourself able to go from barely tangible abstract notions floating around in your head, into actual working instances of that idea manifest in the real-world. ### Programming Languages are Just More Software Coding is just using a software product, just like any other software product, except that you have more granular control over what's going on. That makes it a little bit harder, but ultimately more advantageous. So, a programming language PLUS their corresponding execution environments constitute a software product, just like Microsoft Word or Adobe Photoshop. They are all creations for creating creations; systems for building systems, or what have you. Great creative works can be made in each. ### Don't Give Up on The Dip Seth Godin pointed it out in his tiny little book, The Dip, in which he expresses that statistically, the vast majority of people give up on some tantalizing new endeavor the moment things get to be just a little bit more than comfortably difficult. Once people are REALLY challenging themselves beyond the comfort-level, they bail. Don't. The lower the learning-curve for a tool and higher the initial-reward for using it, the more people will flock onto it, thereby lowering any "specialness" you once felt by being on the excellent, but now mainstreamed and undifferentiated tool. This is the cycle by which the cool and cutting-edge becomes the passé and boring old stuff our parents used. One of the main purposes almost genetically coded into the next generation is to change the game on the older generation, rendering them ineffectual and powerless. Of course, the older generation's game is tricks to horde and fortify long-won power, so the young upstarts can't pull the rug out from under them. As a result, Unix is difficult... just difficult enough to keep out the uninitiated, but so powerful as to completely disrupt the old-school alternative world that was figuring out how to timeshare-meter-and-monetize every processing cycle of every computer in the world, via Multics. We are lucky Ken Thompson invented Unix. Oh, pooey! Fortunately for us, there is a whole spectrum of software-difficulty and learning-curve steepness. We don't have to take the most difficult path, nor the steepest learning curve in order to yield fabulous and differentiated reward. No, in fact just about every provider of any sort of service out there that humans can use directly through Web (or other) user interfaces are scrambling (or have long-ago already done so) to provide programmatic user interfaces to their product. But it's not "users" as such that's using these interfaces, it's machines. But the term that stuck is "application". Therefore, these programming interfaces for applications are known as application programming interfaces, or APIs. And every API-provider at very least makes their services accessible via the same communication system that powers the Web called TCP/IP. In other words, some sort of "request" is made over a TCP/IP network to the service provider and some sort of "response" is sent back. The Web is built on top of that with a subsequent protocol called http. Not all request/responses use http, but a lot do, and it's often used for Web-accessible APIs. But you're doing your programming on your local machine, or perhaps a virtual computer in the cloud such as Amazon or Rackspace that you log into, and NOT on the actual computers of the service providers. So you need a "local" location to run your code, and in this fact resides most of the persistent and invisible obstacles to people getting into coding and staying into coding for the long-haul. There's probably a thousand little reasons an individual's endeavors to "get more technical" break down at this step, and most of this book will be dedicated to helping you, the journeyer, overcome these obstacles -- perchance, to fall in love with some of the less-known, but transformative tech involved in leaping over these hurdles. Without being rushed, how would I do this all with Marat? Hmmm. Okay. 1, 2, 3... 1? - SQL Workbench/J? Nah. More like: Let's pick up where we left off. First it's going to be a crash course in Python. Here's the few key things you need to know. I'm going to walk you through some fundamentals first, so that there is no mystery. It's still going to be full of all sorts of mysterious and wonderful things, but we're going to purge out the mysteries that matter, and put what's going on here more under your deliberate control. Open a terminal and type: conda create -n py35 python=3.5 jupyter cd ~/ mkdir badbounce cd badbounce source activate py35 ### Crash Course in Python - Here's a list - Here's stepping through a list with a for-loop - Here's a dict - Here's stepping through a dict with a for-loop - Whitespace matters... Surprise! - For index, item in enumerate(somelist) - For key, value in somedict.items() ### Opening Files to Read & Write - with open('filein.txt', 'r') as acursor: - acursor.read() vs. acursor.readline() - with open('fileout.txt', 'w') as acursor: - acursor.write('blah') - No need for "close" and why... Pythonic thinking -------------------------------------------------------------------------------- ## Thu Jul 7 09:17:30 EDT 2016 ### Going to get coffee Wow, git pull is now engrained as a morning ritual -- and one of the first I feel compelled to do when I get into the office. It goes something like: - Sit down, open Macbook Air which starts more instantly than my laptop - 3-Finger swoosh to my full-screen vim daily journal instance - Keystrokes: Esc :w Enter :sh Enter "git pull"... merge or whatever - Look to see how my Windows login is coming along - Open a dropbox to check the Date modified date on the daily report CSV files - Admire how smoothly that process is still going running off a VM on my laptop - Make a few morning notes (this) - Glance at email, calendar and my Boggie Board (what's most important/urgent?) - Lacking anything urgent, go get coffee Oh, before I get coffee, I had to restart my Windows laptop yesterday, because of McAfee endpoint software, and so I can document my getting settled-in process on that machine a bit. - VirtuaWin on Windows 7 gives me a pretty good virtual screen environment - Switch to screen 5 and start VirtualBox - Start my Kubuntu virtual machine in headless mode on VirtualBox. Close interface. - Go back to screen 1 and start Outlook 2010. Wait for into to appear before switching to screen 2, or it gets wonky. - Switch to screen 2, start Google Chrome. - Switch to screen 3 and open a Windows CMD (command) window. - cd into my cygwin home directory and into whatever repo I'm currently working on there - activate journal and start jupyter notebook / watch for the Home tab to appear in Chrome - Switch to screen 4 and open a cygwin MinTTY shell. cd into same repo. - git pull - Go to screen 5 and open SQL Workbench/J And one final note before coffee. I have an 11:00 AM meeting (8:00 AM PST!) with Marat to go over using Jupyter Notebook for data pulls! Ugh, okay. Let me think. Get that coffee to be able to think. Then, go through the Mac dependencies for the types of Amazon Redshift and Google Analytics & Search Console data pulls that I'm doing so often these days. -------------------------------------------------------------------------------- ## Wed Jul 6 13:23:53 EDT 2016 ### Menu-work... The database edition It's time to do round-2 now on the menu work, in the method in which you started it today. Okay, dependency time again on my laptop. Wed Jul 6 14:55:10 EDT 2016 Ugh! I managed to install psycopg2 on my Windows 7 64-bit machine with a .exe installer, but then I had to manually copy the files over to the virtual environment. Yuck! But it works. My Windows 7 machine is now one of my primary Jupyter Notebook development machines. I'm glad Marat is on Mac. On Windows, even Anaconda is no assurance that you won't be facing dependency hell. Amazing. -------------------------------------------------------------------------------- ## Wed Jul 6 06:58:31 EDT 2016 ### Getting Ready for 11:30 AM Meeting Okay... not even 7:00 AM. Purify your mind and focus on the work. Oh! Do my actual joins. 1, 2, 3... 1? Lots of false starts on that one, lately, hahaha! Okay. Step 1 is to get menus.ipynb open, and to copy all the CSVs I generated on Thursday into there. Get these joins done. Get the Bad Bounce Report loaded in Jupyter Notebook along with menus.ipynb. Okay. Now, conceptualize the joins you need to perform. Working on Lenovo laptop... Pandas not installed. conda install pandas Okay, and now actually turn this directory into a git repo... okay, good. Dare I put this in Github? Okay yes, this is key, really. Done. Wow, that makes it feel real. That's a big commitment. Next? Develop menus.ipynb. Load every CSV file. Wed Jul 6 07:37:21 EDT 2016 Wow, even just doing the data joins is a chore -- mostly because it's only my second time doing this sort of thing in Pandas. Load ALL the files, not just the millions-of-lines one. Okay, done. Amazing what that does for one's sense of accomplishment, alone. Set the column names for all your CSV files. Okay, I've got them all loaded, and I've got all the column names set. Try loading the one with a million rows as just a single object without chunking. Okay, that's still blazingly fast, even without chunking. Delete the chunking logic for now, in order to maintain focus. Move it somewhere else. Wed Jul 6 08:30:18 EDT 2016 Wow, the SEO Pulse Report CSVs just seamlessly generated from VirtualBox running my Kubuntu image (with slightly less memory assigned) headlessly. Big win! Buys a lot of time for Tech team to tackle AND keeps me from getting knocked out of the zone screen-switching. It's critical that you document the SQL statements that were used to create all these CSV files. Done. Doing the joins, and it's pretty clear I'm not going to automagically generate a hierarchy out of this, and time's running out. The meeting with Cynthia is at 11:30 AM, and you want to be the master of this meeting, not a squirming didn't get it done guy. Okay, trying to do some joins natively in SQL. I've got some tricky tables to join. Ugh! I may have to limit these somehow. Okay, lesson being learned is similar to the last time I had inner joins not coming up with a big enough number -- watch how you do the original data pull if joins are going to be done off-database, such as in Python Pandas. Why not just keep it in SQL for as long as you can, if your querying layer happens to be an industrial cloud database, like Amazon Redshift. Just gritting your teeth and waiting for a query to finish executing where you can tweak the query and re-execute is way better than pulling what you THINK is good source data-pulls off of the system and then coming up empty handed. I couldn't even imagine making these attempts if it were not for my Scala SQL Server experience. I really learned SQL there. I need to switch from SELECT TOP 100 type syntax to LIMIT 100, which is more PostgreSQLy. My current query is up to 5 minutes. Interesting. I hope this works. It worked with 2 of the tables at about 3 minutes, and I just added a 3rd table. I hope this doesn't time out, because if this works, I think I have the data that I need. A lot of my work this morning on loading CSV through Pandas will have been wasted, but I'll be standing on top of much better data for the project ahead of me. I see messages as it executes about loading rows. Interesting, it's only up to 3060 rows after nearly 7 minutes of execution. I think I need to let this just continue to execute. I limited it to 250K rows, and it's currently reporting 4960 rows loaded so far. This could take awhile. I am beginning to think that there are a lot of URLs not mapped into categories. There are probably much more efficient ways to churn on this. I think I'm going to have to stop this query. Get ready to do a 1.5 hour hail mary play to be ready for your meeting. I need a big shift in thinking. What you give your stakeholder today is enough for her to continue designing, and for ME to go back and really make sense of this data in my head. Crap, okay, 1, 2, 3... new project... 1? What I'm dealing with in the supernav is more purely hierarchal issues than the stuff being promoted on the lively nav. And so focus on THAT table. -------------------------------------------------------------------------------- ## Wed Jul 6 04:14:18 EDT 2016 ### Get Into The Office Looks like I have a merge in my future. I know I did journaling from work yesterday. But this probably sums it up nicely, anyway. It was a free/found day, because my stakeholder for my next project wasn't in the office, but still I did not finish the work. And it's all still before me now -- but it's for the best. I'm taking an approach that I would not have otherwise taken. Now, I just have to do it. It's perfect focus-time and my only distraction is me -- and Billy, my cat, who demanded his cuddle-time. Very therapeutic. I can see what Billy cuddle-hour was all about. Billy is very sensitive to the need for things. We all draw energy from somewhere, and it is very possible to draw energy from Billy. Okay, enough of that. Time to think about the work ahead. I am in some fairly new territory, which is the reason for some of my difficulty getting off the ground with this project. It has now taken on that feeling of a college project that I procrastinated on and have to do overnight. I lack the muscle memory habits for this particular project, but I have to form them. This is homesteading in the noosphere. It's love-worthy for that reason alone. Data-informed navigational menu crafting. 1, 2, 3... 1? Do a conda update conda, from before you do a source activate journal. Updates include going from Python 3.5.1 to 3.5.2. Interesting. A lot of eyes are on Anaconda right now, I think. Wed Jul 6 05:23:50 EDT 2016 Ugh. Best strategy now is to go into work. Continuity. Enforced focus. Get the train ride out of the way ASAP. Go!:w -------------------------------------------------------------------------------- ## Tue Jul 5 09:03:45 EDT 2016 ### A Day of Distractions... Ugh! Still amazing how smoothly the transition from weekend/home writing to work writing occurs. Continuity of thought is wonderful. Now, if I only had continuity of work. Don't beat yourself up over it. This 4-day holiday weekend was all about the power of putter, in which I am an increasing believer. I remember puttering throughout my life, and how... ring! Call from Rachel. Okay, not bad. A request to get out of work early. I'm still conditioned from before we were separated. Do the necessary report-checks. The CSV files generated properly (woot!) and now look at the reports themselves. And be prepared for showing Marat how to run the Bad Bounce Report for himself, and showing how to extract platform findings. Ugh! It takes so long to go across the tabs in the pulse report. But it really does give you a nice overview of the sites. I should say something interesting about each property every day. I'm sort of on the verge of having a secondary journal again. I always want a secondary (and even tertiary) journal all the time, but never came up with a good and continuous wand private enough way to do it. Ugh, I have to move faster. The best thing I can do is this menu work. Okay, I have to tackle this problem correctly, and I don't want to take the wrong rabbit-hole chasing approach. I didn't do any of the work that I wanted (needed?) to over the weekend, but it was a July 4th holiday weekend with Adi, and I don't need to feel guilty about that at all. Don't think less about yourself because of the pressure being put on you. Plow through this. Hmmm. Take a look at the files you saved out on Thursday that sort of stage you for a rabbit-chase, but don't actually chase it. Take inventory! Look at your potential approaches. Use the 80/20 rule and buy yourself a lot of time. Fire off an email to the boss about being late on the weekly report this morning. Nope, get him a weekly report, but quickly. Done. Tue Jul 5 09:56:22 EDT 2016 And onto the main event. 1, 2, 3... 1: LOOK AT YOUR THU FILES! Ugh, see if you can't get VirtualBox to run headlessly. I keep getting thrown out of the zone by switching virtual screens (VirtuaWin on Windows 7). Okay, I am starting the virtual machine headlessly. Ugh! Get in with the default Remote Desktop app from Windows 7. Okay, Kubuntu is running headlessly, and I'm going to leave it like that for awhile without trying to RDP into it. Tue Jul 5 11:03:50 EDT 2016 I promised something by noon. That leaves me an hour. So, work backwards, from what it should look like and go BACKWARDS to the data... oh wow, just checked on my stakeholder, and it looks like she's not in today. Do this the right way... the puttering way. Let the data suggest itself. But still, work backwards! Tue Jul 5 15:11:55 EDT 2016 Distractions, as usual. This time, it was adding platforms to the bad bounce report. Not too difficult, but everything has overhead. Ugh! Have a new meeting scheduled for tomorrow. Don't let today's crisis slip into tomorrow. Focus! 1, 2, 3... 1? Make this so friggin' cool, you WANT to work on it all night. But don't MAKE YOURSELF. Dive into this, not like it's a to-do item hanging over your head, but instead like the most love-worthy work you've ever encountered in your career. I mean, IMAGINE what I'm being asked to do! Hmmm... okay... make SOMETHING JavaScript appear in Jupyter Notebook... and THAT's the cool place to begin. Work backwards from cool. -------------------------------------------------------------------------------- ## Tue Jul 5 00:38:35 EDT 2016 ### No Edits! Even Here Holding down a full-time job AND taking care of everything on the homefront, even given as much easier as it's become lately, is still a tall order. I'm exhausted after I get home from work everyday -- just like Dad used to be. I don't really want to do anything but sit still, and be glad I got through another day. The last thing I want to do is tackle laundry and start getting the place in order to be presentable to be sold. And with this 4-day weekend, which Adi spent with me at the apartment for the most part, I started making a dent. Even with Adi's intense full-on routine, I was able to get scraps of time here and there to clean and putter and organize. I even got Adi into some of it. There were tense moments as she wanted to spread the coins all over my bedroom floor again, and I can feel the stress welling up even as I describe it, but I was able to transition that into an exercise of using those coin rolls -- which those coins were all once in and emptied out of once, much to my chagrin. Now that they're in rolls again, they're going to stay that way. But all-in-all, this was another astoundingly successful weekend with Adi. She skipped the July 4th fireworks, which were probably terrible because it started raining at just about 9:30 PM, in order to dig in deeper into her new Nintendo Yo Kai Watch game that she just got at Walmart. Now, I just have to be ready for tomorrow. Don't kill yourself. Take care of your body first. So, go to bed VERY soon and wake up VERY early. Oddly enough, even though I didn't get nearly as much done on the work-front as I had hoped this weekend/holiday, I still feel that I have made significant progress of the most important type to my life, of late. And it is not just this thought stuff. I did a true 80/20 Rule 1st pass at my place. I need to do a second and a third. One night can make all the difference, but then also, I'm a human mortal, aging and wearing. I need to do this all according to a sane plan. Don't kill yourself for work. But then also, make some of the sacrifices you need to make to keep and thrive at that one killer job you landed that you're good at, that makes you happy, that you could see doing in some variation or another for the next 10 years. Grab the reins and make it go the way you need it to go. And that means not flubbing it. Don't be the roadblock. Be that unbelievably on top of things fellow who thought about every angle in 3 different ways already. Be the lightning bruiser. Be the smartest guy in the room, all the way through implementation and support, so that people know you're not bullshitting, and that you are the genuine article. Write really dense paragraphs. No edits! Even here. -------------------------------------------------------------------------------- ## Mon Jul 4 14:31:38 EDT 2016 ### Levitron is the coolest toy, ever! Already 2:30 PM on Monday, July 4th! Holiday almost over. Driving Adi back tonight after Fireworks. Making my decision, I'm going to drive her back late after a lazy day, and watching fireworks from the roof. Funny not being at the Catskills this weekend, and unlikely to be there next weekend, because of super hero school -- haha! Awesome. Okay, anyway she's enjoying Mac & Cheese, Captain Crunch, and a melody of other lazy-day classics. Using her iPad for the first time all (long) weekend, because I believe she's very much into my phone (novelty) when she's with me. Did the Levitron successfully today, and Adi saw spinning-top levitation of the sort that some formula or other was supposed to prove was impossible -- just Googled it: Earnshaws theorem. Well, poo poo to Earnshaws theorem. I can't wait until I comb-in all my other media types into this journal -- so much to do: - YouTube - Instagram - Vine - Twitter - Other? Tumblr is mostly just dupes. - LinkedIn is mostly just dupes too. Hmmm, maybe I should deal gracefully with multi-channel dupes, and actually create some sort of second-stage easy-to-query structure. When displayed directly in Markdown, it'll always show the most interesting bit, like the YouTube and Instagram embeds. But it will also show what other channels over which similar content had been pushed out. -------------------------------------------------------------------------------- ## Sun Jul 3 13:40:01 EDT 2016 ### Clean The Apartment for Adi Okay, back on the early-2009 24-inch Mac. This will be quite an accomplishment if she doesn't decide to fixate on this computer as the one she wants to be on. There are enough screens in play now maybe that I can actually use one of the ones for myself that she knows is one she could be using too. That would be a big advancement in maturity -- NOT immediately wanting as her own "current thing" the one thing she sees someone else enjoying in the moment. Some people I think never get past that point, and are only every happy if they can have something they see someone else has instilled value into by virtue of a bit of attention. People with imagination infuse value into whatever they like. I'm very proud of Adi knowing that the real value in things is in "living things" like the kitties, which are not so easily replaced. Places (to live) are quite another thing, but I need to position it just right in her mind. We will not be at this Inwood apartment forever. I'm in no rush now, but I have to move steadily forward -- mostly, in terms of getting organized. And THIS is where my brain gets organized, and I can start to enjoy the benefits of a bit of compounding returns setting in -- without having to suffer through the reset-button getting pressed on me again. This is a novel and unusual and precious time -- for BOTH me and Adi -- so, don't squander it. Let her SEE YOU get the entire apartment under control today with a series of 80/20 rule passes. Priorities: - My bedroom, and organized bags of laundry -------------------------------------------------------------------------------- ## Sat Jul 2 13:56:00 EDT 2016 ### Why all the Pandas? It's not Black & White I'm going to try to do some journaling here while I go. May have to switch to laptop, because any screen I sit down at is the place Adi wants to be and the screen she wants to use. It's a very familiar situation, and I have to solve these types of this better with Adi now than I did in my marriage. My Surface Pro has alleviated this situation somewhat, but it's keyboard setup just isn't as conducive to journaling as a real laptop or a real keyboard at a real desk. No worries. I can make this all work. Sat Jul 2 18:24:07 EDT 2016 Was just outside for a couple of hours. Even then, Adi likes to use the Pandas games on my phone (increasingly impressive), and I like to read about Pandas on my Kindle. We're sort of in parity. What is it about Pandas, and representing things in the digital world? I guess it must be a lot of things, and not simply black and white. -------------------------------------------------------------------------------- ## Sun Jul 3 12:32:21 EDT 2016 ### Get Better at Stuff Take notes as you walk around and do stuff. - Puttering is okay. It's like ruminating. But when you got it, document it! - USE the systems you have -- if they are actually good systems. - Understand that all systems in a human's life are genetic algorithms - Be willing to let your systems evolve -- experimentation is necessary - But you have to kill off the bad system strains and accelerate the good People often don't see what trips themselves up most, because they're too close to it -- like a fish in water. Start treating yourself like the human being you want to be. Start behaving like the human being you want to be. - Dial-down the pettiness - Dial-up the good-daddy - Dial-up the insanely-smart initiatives (ad-parity, decrufter, etc.) - Remember more. Process & distill more. Take next-step actions on more. It's been almost 4 months since Rachel moved out, and still I haven't really cleaned up. I made a few feeble attempts, and am concerned today will become one of those too. Whenever Adi is absorbed in something and I get into the zone, I can feel it happening, then she snaps out of it, and I (am force to) snap out of it too. So, my goal today is to keep her absorbed in things. She just discovered her VTech InnoTab2. Glad it's batteries and not rechargable. Popped new batteries in, and she's off! Use the to-do list immediately above the current journal entry (always) to much better effect. OMG, puttering... WITHOUT being drained of energy by the day or under stress or running on too little sleep... this is how things get back under control. Shit... Adi and control of my life are (were) at odds. The better I can entice her into things... Ugh! I have the wrong charger here, I think. I may have lifted someone's Macbook Pro chargers, though I don't know where from. Hmmm, hope I have a Mabook Air charger here. Anyway, at 27%. This is a good time to commit and maybe prepare to switch to another platform. -------------------------------------------------------------------------------- ## Sun Jul 3 11:55:16 EDT 2016 ### Return To Where You Love Life Most (Kafkaesque) 80/20 rule sweeps are going to become an important cornerstone concept. I need to document those. It's like my metaphors or my findings in life lists. It's the most important logic-layer near the top of the abstraction stack. It's the one that tells me how to efficiently and effectively get stuff done -- if and when I set my mind to it. It's the one that answers the question hanging in the air, after I do: now, break it down into easy steps. Steps 1, 2, 3... Okay, what's step #1? Step #1 is getting my shoes on. I need the freedom to walk around my entire apartment without fear of pain. Okay, done. Now, what's most broken is my clothes bureau. It's splitting apart with the drawers stopping working. Push or pull the ends together in a heavy-handed enough way to just solve that annoyance final (at least for a few weeks). It jams-up and crams-up, because it stops you from doing many things in your daily organization efforts well. It creates the dreaded back-up queues (laundry, sinks, paperwork, etc.). Gradually become yourself again, writing while you go through the process. Make it kafkaesque -- and use words like kafkaesque. ESC b zg A. Continue typing... now kafkaesque is in my spelling okay-word list, which I keep in a Dropbox location, so all my spelling allowances work across pretty much every system I use regularly -- which is quite a lot. Always remember, there's no situation so bad you couldn't screw up even more by continuing to make bad decisions. And don't just remember that -- try stopping making your bad decisions. Have a nice, firm grip on the rudder or steering wheel or whatever sort of stability and directional control devices your vehicle has. Know how to command yourself like a vessel. Go places. Do things, even if those places and things are right in your own home. Embrace solutions, even if they turn out to be half-assed and temporary appearing, if they're actually good and create the desired outcome. Don't be a perfectionist or purist about these things. Give up old systems that no longer work for you. It's like throwing out your last Amiga, and then throwing out all the RCA-cables and BNC-connectors that you used with it. Nice for nostalgia and museum stuff, but terrible for clutter in everyday life. Move on with the world. But don't ALWAYS go exactly where the world is trying to take you, but certainly get the benefit of everybody else having gone there, such as economies of scale. Factor mainstream cheapness into your best solutions, when appropriate. Don't walk away from advantage because that way sometimes smells bad. Be strong enough to occasionally walk and venture where the smell isn't so nice. But don't be so weak-willed as to stay there beyond what's necessary. Return to where you love life most. -------------------------------------------------------------------------------- ## Sun Jul 3 11:47:20 EDT 2016 ### Do 80/20-Rule "Sweeps" Now that I'm thinking about this journal in Data Science and Machine Learning terms, I'm sure I'm going to adjust my writing style here-and-there to really play into the types of algorithms I know I'm going to be writing to mine my memoires out of this silly experiment. The experiment that stuck. This is it -- a one-long-page textfile journal for life. I like Linux, Python, vim and git towards that end, but it's really up to the preferences of whoever adopts this technique is most comfortable with. The one warning I'd say is that it is intended to be for the long-haul, and it may be best to zero-in on technologies whose likely durability over a lifetime of decades is unquestionable. Even if all Unix-like tech goes away in favor of some neural network OS, there's going to be historic archives that instantiate their APIs through booting a Unix or Linux-like system. And once you have *nix, you also have the common Unix commands and data piping, and probably at least vi (in which, you'd be proficient if you used vim or other vi derivatives, daily (or emacs in evil mode)). And so, this is where my mind is at. These little blips of journal entry will be my reward between 80/20 rule sweeps. Now, go do a sweep. Rely upon principle-seepage or osmosis between domains where you've become expert, and domains where you still really need a lot of help. Commit between each sweep. -------------------------------------------------------------------------------- ## Sun Jul 3 11:40:50 EDT 2016 ### Prioritize-up Being Awesome / 80/20 Rule Clean Sweeps It's not quite so bad as other times in my life when stress has gotten to really ruin me, such as when I was running the fucking check cashing place, and I had my sister ragging on me all the time, and dealing with my Dad's death, and suffering from bad advice from family (my uncles / not hiring a lawyer) and (in retrospect, realizing) my mom gradually becoming schizophrenic, and having just graduated feeling derailed right as my life was supposed to really begin. Shit, I've had a lot of fucking shit thrown at me, and not-so-great people around me not really helping me much. But after a point, those people around you are of your own choosing and the decisions you make are more fully your own. And so it is with that perspective, that I parse out my problems, and dispense with the queue as fast as you parse. Work in real-time. 80/20 rule sweeps to fix your immediate environment are not about deep-cleaning. It's about tolerability, and re-stabilizing your own center... and perhaps even, mental health. Yup. That's why the marriage broke. And now, I need to hold up my end of the separation... I am awesome -- when I prioritize-up being awesome. -------------------------------------------------------------------------------- ## Sun Jul 3 10:22:06 EDT 2016 ### If Played Right, Today is Key (and Macs are better) Okay, I'm on the Macbook Air laptop that was Rachel's, but she still has one, and I now have one at work and at home, and so I can make the best of a good situation. Typing like this -- actually on my lap -- really reminds myself of the superiority of a true laptop form over the tablet / part-time laptop form of the Microsoft Pro Surface 4. If it's for casual media-consumption use, the tablet form is definitely superior, but for actual typing-work, these 2011 Macbook Airs are actually far better -- from their keyboard and form-factor (for comfortable typing) to... uh, the full-screen modes of OS X and the 3-finger virtual-screen swooshing... wait! This should be a list enumerating Mac's superiority in these ways. I actually dislike coming to these conclusions, and an old Amiga-hat being won over to Apple so fully doesn't exactly resonate well with me. Anyway, I believe in approximate order of niceness... 1. Holds a charge for weeks even when not in common use (always ready to use) 2. How well suspend & resume works (closing the laptop / getting WiFi back) 3. How well ScreenFlow works with front-facing camera and capturing keystrokes 4. Fantastic illuminated good-feeling low-profile keyboard (as far as they go) 5. "True" full-screen, very Amiga-like in feel, helps getting into the zone 6. Multi-finger left/right swooshing to flip between full screens (Amiga+N) 7. Unix commands like "ls" in the terminal, and behavior like "piping" 8. "Acceptable" Linux-like repo systems like Homebrew (not as nice as dpkg). 9. Mainstream enough to be a primary target for Anaconda & Jupyter Notebook Sun Jul 3 11:34:01 EDT 2016 Just took a nice, long shower and Adi didn't try to call me out of it. It was very relaxing. To have this WHILE she's here is a pretty big breakthrough. She's absorbed in my phone, which is still something of a novelty with her. I'm sure in her mind, Dad is Android and Mom is Apple (as far as phone's go). This laptop is surely going to be my key to being effective today. Today is my key to getting organized again in my life. Don't squander it. Yesterday wasn't great, and tomorrow won't be great as yesterday both Adi and me unapologetically relaxed. We went to sleep at a reasonable time, and... yikes! It's already almost noon. But we both ate. Maybe I just let her get carried away in screen-time and toys today while I do all the work at the apartment. Remember, don't go diving down into rabbit holes. This is about the big 80/20 rule sweeps. -------------------------------------------------------------------------------- ## Sat Jul 2 13:42:45 EDT 2016 ### Bank Tiny Wins I have so little time to clean up and organize around the house -- not just the cursory, surface tidying that the cleaning people used to do, but a really good cleaning and organizing that permanently improves things moving forward -- the sort of cleaning I need to do to get ready to sell this place. And Adi is VERY sensitive to me doing anything that isn't either sitting quietly OR playing highly engaged with her. Basically, me turning off to give anything else attention at all is her signal to rush over to me and demand my full attention. It's a bit frustrating, but this is life. Billy the cat is almost as bad. It's just like writing is the signal for them to come and walk on the keyboard. But that's just cats. You can't let those thing be bigger forces than your own human will. But then also, you can't make everything adversarial with your daughter and your cats. There's a way to make this all work, and for me to come out ahead. Bank tiny wins! Bank tiny wins. Of course, the corollary is don't suffer tiny losses -- which I constantly am. -------------------------------------------------------------------------------- ## Sat Jul 2 09:32:55 EDT 2016 ### Yo Kai Watch Metal, Tattlecast Ahhh, had yesterday off. Spent morning thinking about my future, a book, SEO, Data Science, blah, blah blah. Then, picked up Adi from her Grandparent's place and brought her back to NYC apartment, which she hasn't been at for awhile, with all our weekends at the Catskills, and the hand-off usually taking place at the Grandparent's Catskills place. So, she was missing Billy, Sammy and Charlie. She's laying down watching YouTube right now, cuddling with Sammy. She's right now learning about the Yo Kai Watch series-2 metals, and Yo Kai 2 coming to America... woot! Big discovery for her. She described yesterday as the best day of her life, several times. She got the Tattle-Cast Yo Kai that she's been hoping for with every Metal-pack opening since she started collecting. Her Pop-Pop somehow managed to score that for her, and had a wonderful story built-up about how he pulled it off. And just by chance, she got another one that she was looking for in a pack I happened to bring with me. They sell them in Duane Reade now. Sat Jul 2 12:18:03 EDT 2016 Already past noon. Hard to believe. But having a great, relaxing time. Suction cup bow & arrow, and dolls and kitties and Yo Kai Watch and just about everything comfy-cozy. Adi takes after me in that way. -------------------------------------------------------------------------------- ## Fri Jul 1 09:26:31 EDT 2016 ### I am Different. Python is Different. Finally getting started this morning. Beginning to write without a git pull, because the city apartment has no Internet at the moment. Will be an interesting day. Get ready for Adi here this weekend AND get ready for Tuesday menu delivery work. Fri Jul 1 11:37:39 EDT 2016 Took long bath, wrote important article. Internet back. Will commit and merge, then paste the article I wrote. First capture hot thoughts. Time to really start shaping my book. Latest thoughts. Strong nicknames/addresses! Mine? Follow the 1 + 10 + 100 + 1000 model. That makes for a good pyramid to visualize, just barely fit-able on a desktop screen with some text-label info. It goes something like: - 1: Homepage - 2: 10 Visible Top-level Pages - 3: 100 Hidden Top-level Pages - 4: 1000 Long-tail Pages - (5: 10,000 Archive Pages) 1111 pages easily live in an Excel or GSheet, and can be thought about as individual entities. If you chop-off level-4 for mental exercises, you only have 111 pages to keep track of in your mind as your most-important real estate of your site. These pages, due to the way most navigational techniques work, actually have direct links to them on every page of your site, by virtue of primary navigation actually cutting across every page of your site, and that next-level of menu selection that I call level-3 here, actually cutting across every page of your site as well. They're usually just hidden or collapsed or accordion'd-up in a dropdown menu. So, that selection of the 10 pri-nav and ~10 dropdown items per pri-nav element is all-important. These are your (including homepage) one hundred and eleven most important pages, and correspondingly, one one hundred and eleven most important "core target terms" -- which I'll talk much more about later. Mine? Most likely: - mikelev.in - mikelev.in/tech - mikelev.in/book - mikelev.in/show - mikelev.in/data - mikelev.in/poems - mikelev.in/tools - mikelev.in/linux - mikelev.in/python - mikelev.in/vim - mikelev.in/git While some of these top-ten items feel like they should be nested inside each others hierarchy (such as linux, python, vim and git under tools), I am deliberately not doing so. These are the terms that I want to "tweak" up in the most surface, visible way on my site. They truly are my primary-navigational items, and out of a sense of purity of example, I am limiting myself to 10, even though I could think of quite a few more. There is just so much to say about these particular 4 tool choices, that I see a need to promote them up to the very top level. The message here is that this site is mostly about technology, with an emphasis on education, and a strong focus on very particular tools that come into play in technology and education. Probably blaringly missing here is my primary career label these days, SEO, and that's deliberate too. I feel that I'm gradually promoting myself up to Data Scientist, just as I did from Webmaster to SEO before that. It may sound a bit pretentious, but it's always really been my deep-down dream to be a scientist, contributing something to this world, and now I see a path. And since all my URLs are short and snappy, I figure /data is a good proxy for /datascience or /datascientist, neither of which come off as quite as good of a url as mikelev.in/data. And yes, writing like this in some of the most valuable time I've ever found myself having over the past 10 years DOES qualify as more important than cleaning the place up in preparation for Adi, and selling the apartment. I have to light a fire under my ass, but this stuff is time-sensitive too. I am vomiting a bunch of important thoughts out that will change my life forever -- no small thing. Fri Jul 1 12:02:35 EDT 2016 Just got the call from Eva/Adi about when I'm going to be up there... hahaha! Okay, reality intrudes, and time that I had was squandered, bathed, ideated away. But it was a worthy trade. The actual menu work for work will have to be after Adi leaves or during the night while she's here -- but I'll be in a MUCH better position to do this hierarchal stuff after this round of work. You're getting powerfully into the "mode" and mind-set to discuss this stuff. Okay, just dump your awesome bathtub book writing here, and then get on your way to the Adi's grandparent's place in the Catskills... Note: When you publish a YouTube video, look at the other Videos that Google tries to auto-play after yours to understand how all their Machine Learning thingies have categorized your video. BAM! Book stuff: # Python is different ## PART 1: OTHER POPULAR LANGUAGES No matter what you've heard or think about the Python programming language, it's different than what you expect. How it's lumped in as one of the "P" scripting languages of the Web like PHP and PERL is very misleading, as it it's seeming rivalry with the trendy newcommer, Ruby of Ruby on Rails fame. In fact, I'd go so far as to say that unless you're a language implementator, all these are quite a bit different than what you think they are -- only Python more so, and in all the good ways that make it amongst the wothiest and most useful of all to know. This article is a fairly geeky dive into how I found and fell in love with Python, and how I believe that if you're a journey(wo) man with a similar quest to find your one true native machine-tongue, you've found it in Python for the forseeable future, and quite possibly for the rest of your life, and maybe even into the lives of your children. I know I'm teaching my 5-year-old Python the minute she can graduate beyond the more visual and kid-friendly Scratch and Bloky. To understand why I feel this way, one must look at some of the other major languages. Let us start with the two biggies in everyday use out there that will also land you a good job with a decent salary as a programmer, which are NOT considered a scripting language -- a distinction that's becoming much less meaningful with every passing year. My assertion that Python is itself the BEST language to learn (not the best language there is), is of course a bold and inevitability controversial statement. It's not like I've programmed in every single programming language, but I have sampled quite a few and sincerely tried to get over the huge hump of a learning curve in everything from Assembly to C++ to Java and C# (Microsoft's premiere .NET language, pronounced C Sharp). Such languages have defeated me in terms of their intense, continuing commitment they require even just for basic proficiency. My .NET/Java defeat are just two in a smorgasbord of languages samplings I tasted in what I now recognize as my quest to find my one true native language to speak to machines. Other languages I took much better to have been somewhat marginalized by time, and eclipsed by Python itself, including AREXX (REXX on the Amiga), TCL, and VBScript. And while I don't program in LISP or Haskell, I've been convinced that the highky meta/functional perspectives they provide are a necessary part of anyone's big picture, as is poking around with the ultra-new concurrency-centric riffs on C known as the Go Lang and RUST. MUCH of my professional programming time had been spent on Visual Basic Scripting Edition (VBScript) on Microsoft's first sincere offering for the Web, Active Server Pages ( ASP) circa 1998, which is the now unsupported tech behind all those .asp pages on the Web. It's been replaced by C# and VB.NET, which are all those .aspx files. I didn't make the transition along with Microsoft because I despised their coding environment called Visual Studio, and their attempt to separate templates from programmatic logic with "codebehind" filed, and their attempt to solve the Web persistent-state conundrum with techniques called viewstate and postbacks. Ugh! Ugh! Ugh! I could viscerally feel Microsoft's cluelessness when it came to trying to keep casual developers like me on the Microsoft bandwagon after the fool-me-once debacle of pulling the rug out from under VBscript, after my investing my early-twenties career years into ASP. It was a bandwagon fad, and I was stung by it. Everyone who wanted to preserve that "everything is easy" feeling of those early VBScript on ASP days flocked to PHP on the then-mega-popular LAMP-stack -- standing for Linux, Apache, MySQL, and (PHP or PERL or Python). Per the encouragement of my then-mega-popular mentor and still-friend, Gerard Bucas (founder of the Commodore spin-off company Great Valley Products (GVP)), I tried implementing my very Ruby-on-Rails-like VBScript framework on .NET, and failed miserably. I was reeling. I was ready to quit my decade-long work-experience with Scala, serving alternatively as their print production manager (in the days of boxed software) and Webmaster (after quitting for a couple of years and returning). My ambition and self-image was running high, and my Microsoft experience was knocking me down. I recognized and began to hate quite how "opinionated" .NET was with their heavy-handed prescriptions for how to tackle this particularly intractable problem or that, and how my own opinions diverged dramatically with hoe they were trying to get me in interact with any relate to the online world They were clearly forging a path for their armies of wysiwyg-handicapped VB desktop app developers to flock onto the Web. We ragtag scripty early adopters of Web Dev were clearly not the priority, and felt it. Only later did I recognize the smell of a rushed-out half-baked overly prescriptive desperate attempt to stem the tide of a growing interest in the non-Microsoft tech from Sun Microsystems, called Java. Java did many modern things first in a language rigged to be already-comfortable to C-developers, like automatically releasing previously used but now unused memory back into the pool of available memory for running apps, in a process called garbage collection. Java also implemented a sort of computer-inside-a computer, called the JRE, or Java Runtime Engine. The Java JRE is a bit of software that implants a translation layer between the capabilities of the native (real) hardware and OS, and a fake (virtual) lesser computer, but built to very specific specifications (for compatibility, performance, security, etc.) running inside. Java programs get cinpiled-for and run-in the JRE, and not (as such) on the native hardware. Copying the Chicana programming language in a transparent attempt to siphon off C, C++ and other C-derrivative languages developers with a memiry-protected language who's compiled binaries can run on the JRE of any host machine was Java's big contribution to the scene. Yawn! But somehow this spooked Microsoft into aping Java point-to-point. Microsoft's JRE is the Common Language Runtime (CLR). Microsoft's aping of Java (which itself is an aping of C) is C# (C-Sharp). About Microsoft's only high-level original thought was a very slick IDE (integrated development environment) to actually write the C# or VB code in, the previously mentioned Visual Studio, then called VisualStudio.NET until everything in Microsoft-land became .NET and they dropped the suffix, with which Java's Eclipse IDE couldn't compete, because a seldom-acknolwdged shortcoming of Java's whole approach is that things which could and should look like "native" apps didn't. Somehow, desktop programs written in Java for Macs and PCs (all Java code is portable, remember) were all always a little bit clunky and unnatural, as any user of Limewire, ScreamingFrog, or any other desktop client software written in Java can attest to. Punchline: Java's IDE, Eclipse, suffered from this same crappiness, providing VB sheep plenty of reason to not switch to greener pastures. Then Oracle bought Sun, getting Sun's still-proprietary Java in the deal, and embroiled Google in billion dollar lawsuits over a out 20 lines of code in something that should have been free and open source in the first place. Somewhere in there, Java's epitaph is written, as something like "Much less special than people thought, and a big mistake to copy so precisely". And so, the glorious light that was Java dimmed to soft glow, and along with it the silly propaganda hype of calling Java and .NET the two "Enterprise Class" platforms, somehow relegating everything else to some sort of second-class also-ran. Nothing could be further from the truth. Everyone stuck on Java and .NET today are there precisely because Sun and Microsoft (and with Google choosing Java for Android development, Google too) stuck you there, and strong mental barriers slowly grow like crystals up around your open mind and free though, until all that's left is a very closed-minded, and only those thoughts running there that Sun and Microsoft saw fit to put there. We ourselves are programmed by our adoption of and learning investments into our programming platforms. This is not always a bad thing, when the programming language's (platform's) assumptions, paradigms, and ultimately OPINIONIONS of how you should be thinking, acting and coding ultimately align well with and enhance your own behavioral predilections. And when the fit is perfect, it feels like having superpowers. -------------------------------------------------------------------------------- ## Wed Jun 29 20:18:36 EDT 2016 ### Perpendicular To Everything We Know I get home, and I immediately want to do a git pull, and quickly deal with any merge issues, and get journaling. I see the codes inserted by git. I'm going to make them render as code, so it looks good as markdown. I'm going to be writing some good stuff. It might be tremendously dense like this, but I do think I finally have the makings of my book, and I feel really compelled to get it down, dense and rough as it may be. First, we Peel away layers of perception, and then layer back on new perception pattern-recognition models. Look for different types of things with different types of senses, and respond differently, according to some set of genetic learning algorithms, so you know what to not do again, so you don't almost die again. Or else, you die. Simple formula, really. Dawkins argues it well. No reason for me to go there too. I 99% agree with him. That last 1% is God, or some singular higher-dimensional being that might as well be, for whom we are a cross-section shadow, it somehow managed through great personal effort to come down and walk with us a few times. Or that's the belief I come to it with. Go google about Dawkins and Michio Kaku debate this God existing stuff. Michio Kaku is right. It's turtles all the way down... or up, for that matter. Can't beat that problem of induction, Atheists. All the scientifically backed-up observations you can make still can't trump the fact that your senses may be lying to you, and you can't prove that they're not. All experience is subjective anyway, so who's to say -- without true group conciousness? Ah, group conciousness -- a modest goal, and some strong evidence that would weigh in favor of a common objective universe that we all occupy... oh wait, or does it? Nah, induction again, dammit! How long are we mortals going to have to wait before we can come up there and walk with you, saving you a little bit of effort of finding us in all that foamy bright matter. At least, this is more-or-less the perspective I come at it with. If you didn't want to think we were in a virtual simulation, you shouldn't have made the speed of light and quantum randomness work the way you do. I think we've got you here. Particles do go somewhere when they pop in and out of existence. It's called the fourth spatial dimension -- or the fifth-dimension, if you've already filled the spot for the fourth with your monkey notion of a derivative effect of being an observer from inside the system. No, the fourth spatial dimension is like length, width and height, only out at a 90-degree angle from any of those, just like the interior of the good ship Tardis. We're talking hyper-cube tesseracts and hyper-sphere n-sphere. I think the equivalent of Eric Graham's "The Juggler Demo" of the Amiga days, using a pixel raytracer equivalent of the SSG program. SSG stood for "spherical solid geometry" and was used to create one of the most important computer visualizations of modern times, in a looping about every second 30-frame juggler sculpted from spheres and sphere-tubes, and juggling a bunch of reflective mirrors. Somebody do that with n-Spheres, will ya? 'Cause THAT would be a neat visualization, and will give us a little practice for walking with the big ol' one up there. Out there? Perpendicular to everything we know there? But that pinhole poke to hit such-and-such a time and place and point is no small trick, even for God. We can't put limitations on God, I know. But look around us. Now, go read Rendezvous With Rama and return. Got it? The Universe is a lot more like a NoSQL database like MongoDB, than it is a relational data base management system (RDBMS) which generally implies tabular data, except maybe with PostgreSQL and a few others that can do stupid multi-dimensional array tricks, and is the reason for the aptly named ndarray object in the CPU-bound but otherwise high-speed Python NumPY matrix analysis software that can still operate efficiently across parallel processors none-the-less, if you use the Ufora Python "interpreter"... but interpreters are all really virtual machines these days, and so it takes us to some of the original CompSci insights and a world of infinite possibilities, including AmigaDOS and CP/M and ITS and TOPs and VAXOS and a bunch of others, until along came BCPL and C and Unix... then Linux... then Unix again. Now, sprinkle in microcontrollers with tiny high-level language, like oh say Python, embedded in similar to an Arduino, but more like a Raspberry Pi, for devices like the Onion.io and the Internet of Things (IoT). And then, there's robots. Let's not forget the robots, and the machine learning, and the datacenters in a shoebox, and scalable spark approaches that can spontaneously spawn large-scale neuralnets, by virtue of reliable local code execution environments <pre> <<<<<<< HEAD </pre> -------------------------------------------------------------------------------- ## Wed Jun 29 08:01:37 EDT 2016 ### Adi and the City Getting ready to head into work. Just putting this here as a reminder to not ignore getting the apartment in order. Adi will probably be here over the weekend -- picking her up FROM the Catskills this time. She misses the Inwood apartment and the cats. I have to let her see Billy, Sammy and Charlie. But I also want/need to start getting the place fixed up for sale. Solving difficult simultaneous equations again. <pre> ======= </pre> -------------------------------------------------------------------------------- ## Thu Jun 30 17:40:19 EDT 2016 ### A Deeper Dip into Corporate Data Than Before I need to take my Menu work home with me, but also make it LOVE-WORTHY... separated from the newly delicious SQL tables I have to piece together hierarchy. I also think I might like to do it a bit from a giant list of URLs through Pandas. Thu Jun 30 18:12:22 EDT 2016 Okay, I just created my first matplotlib data visualization. Now, all I have to do is bring all the data home with me that I want to work on -- possibly tonight, as I alternate between that and cleaning up around the apartment. I really helped a co-worker quite a bit today with that Pandas data chunking problem. Useful for me to learn a thing or two as well. Thu Jun 30 19:11:47 EDT 2016 Okay, heading home now, but have everything needed to construct site taxonomy menu from core database. Can layer on search frequency and performance data from home. Really doing a hail mary here, for sure. I think I can do it. The menu has to feel somewhat familiar, and pass the "smell test", which I think the last one didn't, because I wasn't using the "real" taxonomy data to start with. I am now. I did a bunch of CSV file exports from SQL, and I'm going to be using Pandas for all my post-processing -- even the joins! -------------------------------------------------------------------------------- ## Thu Jun 30 13:28:34 EDT 2016 ### You Can Chunk DataFrames in Python Pandas to Process Big CSV Files Focus! I'm at a very interesting point now, where I have this journal repo in use on both my personal Mac and on my company PC. I should be able to commit just this file here, and just that file (the .ipynb file) there. Commit this right now and merge the two. Okay, done. Just do some pure exploration in the tables. Oh, I'm showing myself as a SQL Server person with the Top X way of constraining rows. I need to learn to use Limit X at the end of the query, as is the PostgreSQL and MySQL way. Thu Jun 30 17:05:45 EDT 2016 Just did some work just now on applying Pandas against a vary large source data file, employing chunking to process it in smaller bits: from time import sleep rowsperchunk = 250000 chunkstoprocess = 50 filename = 'bigfilein.csv' outfile = 'smallfileout.csv' import os try: os.remove(outfile) except: pass finished = True for index, chunk in enumerate(pd.read_csv(filename, chunksize=rowsperchunk)): sys.stdout.write('%s, ' % index) chunk['D'] = chunk['colname'].str.extract('keyname=([\d]+)') chunk.drop(chunk.columns[[0,1,2]], inplace=True, axis=1) chunk.columns = ['longstring', 'extraction'] chunk.to_csv(outfile, mode='a') sleep(2) if index >= chunkstoprocess: finished = False break print("Finished: %s" % finished) ...and I also don't want to forget what makes VIEWING this while it's processing possible: class Unbuffered(object): def __init__(self, stream): self.stream = stream def write(self, data): self.stream.write(data) self.stream.flush() def __getattr__(self, attr): return getattr(self.stream, attr) sys.stdout = Unbuffered(sys.stdout) That allows you to view things WHILE it chunks, instead of waiting for all the chunking to finish before the cached buffer dumps to the browser -- ultimately, as unsatisfying as not chunking, though much less memory demanding and prone to freeze up your machine. -------------------------------------------------------------------------------- ## Thu Jun 30 09:43:07 EDT 2016 ### SEO Science - Let's Get Meta Okay... debugging the Keywords process for one property not running this morning. Caught it! I want to put better debugging into my program. Switched it from 2 character indents to 4 to stay better in compliance with pep8, if I'm going to be going in there a lot with my new .vimrc set the way it is... which I'm not, so... so whatever. Gotta go to 4 spaces, even if I'm just in there occasionally. Make today super-effective. We have a 4-day weekend coming up (amazing!). Maybe we'll watch fireworks from the building roof with Spider. Wow, gotta think through this whole Inwood thing. If only I were making more money, I could keep the place. Long way from Adi these days, but it is where she grew up, and it is worth a try -- if I don't consider what I've done so far already as trying. I need a money-making scheme that will just work so perfectly without syphoning off any energy from work, but rather ADD TO IT. May ideas a percolating. But for right now, get that Keywords tale in that property's report filled-in! Got the first emails out. Thu Jun 30 13:15:34 EDT 2016 That problem was fixed. Had lunch. Gotta get and keep my mind in the game. This is my window now for the menu work. You want this to be brilliant. I am a technology employee that am applying my techniques to smart menu construction. I will be with my daughter for most of tomorrow, and for most of the weekend and through Monday, in all likelihood. With machine learning improving, crowd sourcing is on the edge of dead, so says one of the Partially Derivative guys. Mechanize. Here's that writing from my phone from yesterday that I forgot to paste in: SEO Science by Mike Levin Let's get meta I am an SEO. That means, search engine optimizer, for those not in the elite circle of ruffians who know. Congratulations; now you are. As I type, I am generating a unique grammer signature. By publishing it online on a unique resource found at a registered Doman name that is strongly associated with me, so I will be known to all who run stuff against the ancient historic Github repository of all human knowledge that first Google, then The Internet Archive (waybackmachine), likely the DOD, and now almost everybody is keeping, will be keeping, in good data-walked and DNA-mapping-like gap reconstruction techniques blending in every newly found archive to digest. The gaps and the versions get filled in, as if git, but more like Mercurial (the hg nearly command-compatible, yet still very different almost native Python one-time competitor that lost. But it's cool. You can query it like Google and stash your private changes directly into the main repo without branching or altering the main head. I don't think I'm going to be able to hide my writing style or identity from Google or anyone with that process that likely outed the Bitcoin author. You can't trick the machine -- so, don't expect to. Let, I'm an SEO. Chapter 1: 40K-foot View Notice, I'm not giving you the satellite or even boson-camera view of things. I'm just giving you a gravity-bound, fossil-fuel burning rise-of-tech era barely better than riding tyrannosaurus vjew. And I make you aware of the fact that that's what I'm doing. Wow, that's meta. Data is beautiful. Reality is beautiful. Our ability to see beauty and feel life is a friggin miracle. Out related ability to actually not fall apart and become dysfunctional is a bigger miracle still. Making it to a ripe old age between 80 and 100 to merely die of the parts wearing out is the biggest miracle of all... Oh wait; no, maybe it is our ability to encode and communicate and transfer knowledge and information and experiences (data) from generation to generation, to keep us alive in data sampling of product, and the recipes and ingredients to more or less reconstitute it. Same DNA. Probably a new soul. Who knows? So clearly, I'd rather be writing SciFi along these lines -- the rise of clone clans as evidenced by the multiverse peek known as Ann Leckie's Ancillary Justice novel, or the creation of a new type of legacy systems based on solar-charging C3P0-like machine/human interface robots that are built very rugged and able to hibernate. It's greater than my personal ability to implement, so I will entice all you youngsters to do so. So go ahead and try to take over the world with your clan of clones and your robot armies to back them up. I'll try to equip you with all the beautiful, fundamental, and profoundly enabling tech tools that are really all around us We're meatsacks banging sticks, setting off nuclear reactions and coping with speed of light size of Universe questions, and now boorish lyrics competing with each other for online audience cell time, so we can turn around and sell a portion of that audiences' eyeball attention to advertisers. That's our (my current) biz. I will use this to generate some more revenue for myself personally, while I try to do it to new heights with/for my employer. Alliances! Synergies! But still a bombastic and boorish attention-whore. I'm sorry. I will redeem myself through sharing what I do with Adi's homeschooling education along these technology lines. -------------------------------------------------------------------------------- ## Wed Jun 29 10:47:54 EDT 2016 ### Derailed by Phantom Pandas Having played with tmux for a bit yesterday, what I can say is that I'm glad that I'm getting to know it, and can consider it part of my familiar toolset, it definitely adds tons of unnecessary intellectual overhead to simple session management, if your terminal sessions are local, and don't have TCP/IP disconnect issues with remote servers. All those Ctrl+B's are completely annoying, and that one configuration setting I need to add to make switching between panels mouse-sensitive at any new machine I sit down at and want to use tmux on is annoying. Also, I really don't find myself needing to keep those long-term sessions connected and accessing them through the tmux server. I totally understand this is possible, but so far, this is not something I need to do. In fact, in a way, my getting familiar with tmux could not possibly be timed any WORSE, considering my pretty big move to running Python locally on whatever machine I'm sitting at, rather than on some remote server somewhere, where I'm confident I can run only one version of Python, that all the dependencies of all the packages I want to use will be fetched and (occasionally) compiled and installed properly. Anaconda sees to that, but my optimism on the Windows platform was premature. I'm trying to use conda's equivalent of virtualenv, and the "source" command can never be found. This is obviously simple PATH environment variable stuff, but on Windows, environment variables are never simple. I wish Windows itself had virtualenv. Wed Jun 29 11:34:35 EDT 2016 Feeling the advantage of the journal being on my personal laptop, which I take around from meeting to meeting. Macs totally handle wifi better than Windows (at least, 7). At the Sprint meeting. There's lots of activity today surrounding SERP changes that have been going on a lot today. I should really start positioning myself as sort of a new voice of modern SEO. The faster I embrace the tools and methodologies of Data Science and Machine Learning, the better. My current projects are taking me in by baby-steps. At least, I'm ising Jupyter Notebook and Pandas now... almost, regularly. I'm reading the free O'Reilly Python for Data Analysis Pandas book now, and getting the insight on all the amazing features built into IPython Notebook... wow. Not merely a REPL, and not quite an IDE. Fits into my workflow just right. - Take advantage of the ML nickname of Machine Learning (also my initials) - Do some interesting play-on-words for branding my content, such as Mike/Reduce Wed Jun 29 16:06:26 EDT 2016 Lots of distractions today on one-off SEO requests, and the activity surrounding whether we're seeing another Pandas update, and the more likely scenario that we're just seeing the more frequent, incremental shifts of Panda now being part of the core ranking algorithm. That means Panda-like updates should just be trickling out with new SERPs. However, plenty of people are reporting discreet "Phantom" updates. I think how you label these changes is splitting hairs. There's a black box, and we have to predict its behavior. I'm going to take one more shot at getting Anaconda working correctly on Windows. I'm re-downloading the installer, as it is up to 3.4.1 (I was on 3.4.0). I lost my way between yesterday to today, but understandably. One of my big findings was that I was not going to be able to reasonably run Jupyter Notebook directly from Cygwin, but rather, I have to launch it from a standard CMD window, which I happened to hack Courier New as the default font so that I can stand looking at it. Wed Jun 29 18:23:41 EDT 2016 Just came back from the Happy Hour. Heading home now. I actually got the conda virtualenv stuff working. The trick was to just use "activate venv" rather than source activate venv. Time to commit and go home. -------------------------------------------------------------------------------- ## Wed Jun 29 08:01:37 EDT 2016 ### Adi and the City Getting ready to head into work. Just putting this here as a reminder to not ignore getting the apartment in order. Adi will probably be here over the weekend -- picking her up FROM the Catskills this time. She misses the Inwood apartment and the cats. I have to let her see Billy, Sammy and Charlie. But I also want/need to start getting the place fixed up for sale. Solving difficult simultaneous equations again. -------------------------------------------------------------------------------- ## Tue Jun 28 12:04:42 EDT 2016 ### Even Anaconda Isn't a Cure-all For Windows Wow, it's already noon. The net was down for a short bit causing a brief interruption, and I needed to do a bit of spreadsheet massaging for the boss for some past-work. I like checking thing off the list. I'm working through all my work with progressive chisel-strikes, beginning to make the really roughest outlines of the sculpture. My boss is about as ideal as it gets, actually having overarching strategy for what he's doing with the department -- something I've never really seen done well before. Usually I'm the top-dog in the SEO department making all the big, important calls which is actually not the best, because while I love my work, I am definitely a hands-on implementation mechanic sort of person. Inbound link-building has for example become a bit of a blind spot for me, after Google started cracking down on the practice and you can't just buy your way in. In-bound links are still incredibly important, but beyond organically just encouraging them to occur, I don't really have that many "products" to offer. My boss did and does, and it was a pleasure to see it all play out, and to play my role where my particular Kung Fu really did help. I'm not just implementation, really. I'm more of a digital detective AND implementor. It's a nice thing to be. I feel good about it. It provides me the next round of actually maybe of general interest YouTube content -- the new wave of tools for data-investigation such as Jupyter Notebook and Pandas, that is. I have a huge amount of work ahead of me today and tomorrow, but thinking about it that way is bad for me. The mere existence of well filled-in data should IMPLY the menus I want to use for the website. However, the data is not nearly as well filled-in as I want/need, and so doing so today KNOWING what data structure I'm targeting for the menu will help immensely. Producing algorithmically recommended website navigation menus would be a HUGE win, especially when the data behind it is absolutely clear, intuitive and compelling to everyone involved in the process. Quantify HOW MUCH traffic you're actually targeting and going after, by mere virtue of the menu selections. And now, go have a light lunch. Tue Jun 28 13:25:38 EDT 2016 Also stopped at Duane Reade and picked up some fun stuff for Adi for the weekend. Duane Reade caries Yo Kai Watch metals -- score! Tue Jun 28 14:13:00 EDT 2016 It's strange that I set up jupyter notebook for the first time on my Windows Lenovo laptop, but yet have not touched it there since I started using it on my old Macbook Air. So, all my conda virtual env stuff isn't on my laptop yet, and this looks like the makings of a tip-area near the bottom of my journal, as I commit this stuff to memory. Let me do it one last time here, and then be sure to make one of those things at the bottom. pwd conda create -n venv python=3.5 jupyter Tue Jun 28 18:06:24 EDT 2016 Ah ha! Even on Windows, Anaconda isn't a walk in the park. The path issues become a bit trickier when you actually use the conda virtualenv trick. Still working out a few things. Hard to do the source command due to bad paths. Had to reinstall. Day over. Not sure what the conclusion is, but I'm doing a conda update conda before I leave. Ugh: 'source' is not recognized as an internal or external command, operable program or batch file. Oh, Windows is the Anaconda endeavor even not enough to cajole and coerce you to work correctly? -------------------------------------------------------------------------------- ## Mon Jun 27 10:15:07 EDT 2016 ### Monday Morning SEO Report Okay, weekly report! 1, 2, 3... 1? Think through last week. Check your email and conversations. Don't down-play the importance of various things you were probably involved in. The amazing Mr. Schism has once again started vocalizing his ideas about Levinux, and I'm listening. Gotta think about bandwagons and the viability of personal projects. I see myself compiling QEMU across all platforms precisely efficiently for Levinux -- but that's a super-huge distraction and a rabbit-hole to boot. And so, I suppress this distraction as I move on to get my second coffee of the morning in my last-and-final distraction before cranking out my weekly report. Don't forget: - Those things in GA you're investigating (2 of them) - Those people you helped last week (2 of them) - That attempt of a thing that failed last week -------------------------------------------------------------------------------- ## Mon Jun 27 09:49:16 EDT 2016 ### Be The Real Deal or Get Off The Pot Almost weird doing a git pull and not having updates from home after a long weekend. Just shows me how busy I was over the weekend. I want more than ever to "blend" my YouTube videos right into this journal-stream for providing context about what's going on in my life (again, for MY benefit -- not yours) "for free" using data I'm also concurrently producing over other channels, just in the course of living my life. I also think I'll blend in my Instagram and tweets. One long page of that stuff all dove-tailed together in NOT a mess may actually be a noteworthy experiment and a contraianly delicious demonstration of all the new technologies I'm taking up and becoming familiar with that complement so well my Linux, Python, vim and git workflow. There's a lot of good data to analyze here. What better data sets to become familiar with the tools with than the data sets I generate (am generating) myself? I feel this really starts to pull together all my disparate, but somehow interrelated activities. I am actually living a highly integrated life -- or, at least, that is my aspiration. Adi's homeschool education, by time with her in the Catskills on weekends, my day-job of SEO (transitioning to data science), and this daily journaling and preoccupation with martial arts-like infotech tools and their mastery. Hmmmm, okay. Let's see. Gotta get my weekly report out. Keep sanitizing as you write here, because it's not your actual down-and-dirty daily work-journal where potentially proprietary information flies. Information wans to be free, but the corollary is just as true -- information wants to be valuable. Granted, valuable information may have a short shelf-life, because once it's used, you likely have tipped your hand to competitors, and have to now go about acquiring even newer and better information that somehow yields a greater competitive advantage over the competitors than your last hand. It's an ongoing game. It's the transience of things that makes SOME valuable information valuable, but then it's also the ability to keep a secret for a good, long time -- AND to "lock things down" so that secret information remains secret. That's the kind of secrecy that is difficult to pull-off, but can really make a big difference, and that seems to mostly be Apple's modus operandi. I am certainly not dealing with Apple-caliber secrets, but I do have to watch myself. This captures the free and open source dimensions of my work -- big emphasis on frameworks and generalities, and keeping relevant in my field. Big sanitation on property names, particular endeavors, and anything where a clue-full person could piece together proprietary secrets. But still, I must write to be effective to process my thoughts, and use that writing in a Robert Cialdini commitment & consistency play -- almost as performance-art, to stay compatible with the modern sharing culture and valuable original content production that's so important to the evolving field of SEO. Be the real-deal or get off the pot. -------------------------------------------------------------------------------- ## Fri Jun 24 12:09:11 EDT 2016 ### Google Analytics Query Explorer More work on yesterday's data-pull project. Figured out how to use filters in the Google Analytics Query Explorer. I'm having a good time. This is re-positioning at it's finest. There was too much of a "finality" that I felt with committing Python code on pseudo-deployed testing servers, just to have your data not read/writable using the filesystem of your local machine and have to figure out that step. When Google tries to give you all the data that you have coming to you by virtue of using their products like Google Analytics and Search Console, you have to be smart enough to ask for it and get it in the proper way. And for that, there is hitting APIs. And for hitting APIs, there is knowing all the stupid nuances and in's and out's of the rules they set up. But Google really did give exploration and experimentation a big boon with this: https://ga-dev-tools.appspot.com/query-explorer/ It took me some googling to figure out how to work those filters, with the double-equals (as a Python user, I should have guessed) and the commas between filters. Wow, this stuff is powerful! Because of how technology progresses, the truth is that we are always personally re-platforming, all-the-time. It's just an unavoidable condition of the information-technology life/career/whatever. But as you re-platform, don't let muscle memory and years of mastery of particular tools be a casualty. Linux commands, Python coding, vim text editing and git distributed version control system at your service. Sure, learn all the nifty (usually) Web-based UI's that make life easy, like Github and Gitlab for git, and Jupyter Notebook for Python, and a nice GNOME desktop for Linux, and... hmmm... vim for vi. If you really want to splurge, you can use gvim, but I'd advise against it. -------------------------------------------------------------------------------- ## Fri Jun 24 09:25:09 EDT 2016 ### Don't let yourself be outclassed by Google Hey, that became my headline earlier than I usually write headlines for my entries. But that sums it up nicely. Using Pandas to not be outclassed by Google, who's wrought changes so vast as to induce existential crisis across the dirty long-tail of people who fancy themselves as search engine optimization professionals. Pandas, and Penguin that came after it, and Vince that came before it, and all the many sub-projects that came along the way are Google's way of saying: "Optimize that!". The cat-and-mouse game is won. You have been outclassed by the gradually machine-learning Google. You are just still getting away with a few things today, as Google hasn't quite gotten around to everything yet. I mean, now could they? There's so much information in the world that the very scale of the problem outclasses Google, itself. So, the game hasn't played out fully, by any measure. There's long-games here that few of us have even imagined, and plenty of surprises that are going to come from unexpected not-heavily-financed directions. The one invaluable commodity in this whole picture is... no, not data... is creativity, imagination and insight. It's the ability to see plausible stories emerging from the data that you're looking at, and making connections and unlikely correlations between different, previously unrelated data-sets. It's thinking across dimensions, with different human-understanding-models of an often-core and often-repeating set of concepts, we sometimes label design patterns. Design patterns occur in nature, in the ultimate information systems such as DNA and life, to the simplest patterns, such as crystals. My machines at my desk are set up in a very curious state, and I thought I'd capture it. -------------------------------------------------------------------------------- ## Fri Jun 24 07:35:16 EDT 2016 ### Commit Little By Little Choose your abstraction-levels and API details carefully. In them, you can glean your future flexibility, ability to pivot and change your mind and take different approaches, and do transformations from the setup that was due to the presumptions of the old system to a new setup that caters to the presumptions of the new or altered system. You can do this in small steps and attempt to maintain backwards compatibility. Big choices surround moving your data and re-basing it on new hardware, database engines, etc. There are countless approaches, and all of them are valid so long as they get the job done and don't excessively limit your future options. Always be working on your next step-1 starting point. Insurmountable hurdles will suddenly start to disappear, and the friction and resistance of getting started on projects you've been meaning and wanting to do will magically fade away. You'll find yourself saying, oh, I already did that last time. I can just do this new project that... oh, say automatically checks the email in a particular email account every 10 minutes, and emails you that something has arrived. Just pass on the info at first. Just make the diode or router equivalent of a repeater or amplifier. Data-in, same data-out. Channels may vary. It's the essential trick of network address translation (NAT) in routers, and their optimization routing tables. Something on-the-left is equivalent to something on-the-right, and when something comes along either way, just shuttle it along. That'll be my first project. Capture ideas when you can, but when you can't, don't sweat it. Say them out loud in your head, and associate them with a few things around you, like where you are and what you're hearing. These things will serve as triggers in the future and induce you to have the same (or similar) idea again. Commit now. You may not get a chance for awhile. -------------------------------------------------------------------------------- ## Fri Jun 24 07:03:01 EDT 2016 ### Life and Robots and Adi's Future Robot Tech Project Everything is a system, and every system has its own prevailing architectural concepts, which is to say, it's biased towards doing a certain thing a certain way, based on preconceived notions of how the system should work. You and I as human beings (I presume it is mostly human beings who will ever read this) are ourselves systems. Or more precisely, we are the output of the DNA system of life that has evolved on this planet, which makes lumps of organized stuff that extracts nutrients and energy from its environment in order to go about its business, which is primarily to organize more stuff to be much like itself, but with sufficient permutations introduced into the system to challenge and prepare for unpredictable change, therefore instilling resiliency into the greater continuous system. We as individuals are temporary, but so far as our experience tells us so far, the overarching system of life and procreation goes on forever, surviving extinction event after extinction event. Of course, the ***precise*** details of what survives through the extinction event is somewhat up for grabs, until we get some Noah's Arc plan together so that humanity can hop over such terrible incidents, and come out as still human, and not set back by ages on the other side. Step #1: Do it with robots. That'll be one of my ongoing projects that all my work transitions into; robots that can: - Go dormant for long periods of time - Draw energy from their environment - Re-awaken, based on time or other environmental variables - Have a high degree of mobility - potentially globe-trotting and ocean-dwelling - Scavenge their environment for what they need to self-repair - Be set on the path to turn the self-repair ability into reproduction - House large amounts of data - Provide a variety of human/machine interfaces to make that data usable - Allow rudimentary interaction with humans of any language (sign-language, video images, etc.) - Do housework The list probably goes on, but you get the idea. I don't want a SkyNet here. I want a FidoNet. There was a FidoNet. I loved it. Reproduce that without phone-lines but with robots. The low-tech pre-Internet internet. -------------------------------------------------------------------------------- ## Thu Jun 23 20:20:50 EDT 2016 ### Everything Worth Doing Takes a Little Bit of Work My work at the office today was interesting on two fronts. First, I am using Pandas for things I would totally have used SQL for before. Now, I know what to do every time I sort of wished some data I was sitting on was in SQL, so I could apply join-solutions and stuff to it, even though it lives in a spreadsheet or text-file. Wow, it's just starting to dawn on me what that means... I should go seek out some data. And what better than this very journal? But tonight, I want to do stuff on the home-front. Things like dishes and laundry... very exciting. Thu Jun 23 21:08:12 EDT 2016 Just talked with Adi. A lot about Rick and Morty and why I watched ahead, and how Rick's "Smart" could be detected, and how Morty acted as a shield, and how there was that world with a Rick that used a shield of Mortys, and how that was not a very nice Rick, and how I watched ahead to make sure I knew what was coming up, so that I could explain things quickly and efferently and make sure that anything needed to be talked about was talked about. And then we talked about dimensional scissors and subtle knives and donut shaped universes. It was all very stimulating. She certainly is my daughter. I almost can't believe what I was able to pull off today, with getting MiniConda and Jupyter Notebook installed on my SolidRun CuBox i4Pro. I also got Dropbox to run, I forgot to mention in my earlier journal entry, but that worked too. It's really amazing what that Exagear product made possible, and precisely how all that came together. There is a whole virtual machine running inside the tiny little ARM PC that I brought into the office, precisely for the purpose of getting the daily reports off of running on my laptop on a... drum-roll please... virtual machine. So, why is it okay on my CuBox, but not on my laptop? The difference between a laptop and a server, doofenshmirtz! The tasks on a Linux-based solid-state piece of low-power, few-parts will run regular and solid, while those on high-power, lots of parts, many of them moving -- and all of them Windows -- will be sure to fail aplenty. But to be running that x86 emulator on an ARM platform, sufficiently to get Dropbox and MiniConda to run adequately well... amazing! So, I'm going to run out of steam again tonight. I can feel it setting on. And I have a lot of driving to do tomorrow night. Home is going to be the sacrifice, yet again. Sighhhh. Okay, I need a stronger will, it's as simple as that. This right now is one of my weak-willed moments. I need to decompress and create indexes, and optimize my components to work well again tomorrow. But for most of tomorrow, I'll be at work, and for the rest, I'll either be on the subway or driving. Don't over-complicate too much. But everything worth doing takes a little bit of work. -------------------------------------------------------------------------------- ## Thu Jun 23 19:55:35 EDT 2016 ### Of Pandas, MinMaxScaler from sklearn, and matplotlib At some point, I'll do a screencast of my process of checking in a shell whether vim is running with index.html, and if it is, in which case I'll immediately do a git pull, and if it pulls and merges cleanly, I'll type exit, and drop back into vim, which will notify me that the file has changed and asks me if I would like to Load it, and I answer yes, then hit @j to start making a new journal entry. Next thing I really need to do is start using matplotlib in jupyter notebook. I even have to just remember that package's name. I need a good example. Maybe watch those YouTube videos. Oh, and don't forget to mention how awesome it was to... Gotta always remember: source activate journal ...how awesome it was to do today's work, learning pandas and from sklearn.preprocessing import MinMaxScaler std_scale = MinMaxScaler().fit(df[['rangeofnumbers']]) df['onetohundred'] = std_scale.transform(df[['rangeofnumbers']]) df.onetohundred = (df.onetohundred * 100).astype(int) df = df.sort_values(['onetohundred', 'othercolumn'], ascending=[False, False]) df I also learned how to convert native python objects to pandas DataFrames, and how to sort and filter those columns. The above example was to prepare a column to be used for 2-column hierarchical sorting, which is very related to pivot tables or aggregate functions. All these things take all these different names, but are very much the same thing -- just expressed in different ways, like file systems, SQL databases, name/value DBs of varying BASIC to ACIDIC properties. Spark, Hadoop, whooptie-doo. Just learn to think in varying ways about similar underlying relationships. Yeah, yeah, everything's really just a linked list. -------------------------------------------------------------------------------- ## Thu Jun 23 09:26:34 EDT 2016 ### CuBox, Take 2! And a Real Jupyter Notebook / Pandas Project Ahh, feels good to merge without conflict. Okay, yesterday was only so-so productive, but with lots of learnings. I'm finally trying to put my CuBox into use, but certain things in the report generating process are x86-dependent, and I purchased (for ~$35 yesterday) Exagear, which people are apparently using to get Raspberry Pi's to run Dropbox -- a worthy goal. Sure, Exagear is proprietary, but these tweaked-out optimized VMs often are. I use (and love) VMWare Fusion, so why not support a developer doing such a fringe thing as getting x86 emulation on ARM processors? They port the hardware, so Dropbox and others don't have to port their software. I believe it's based on WINE. This falls under the category of dropping some money to keep continuity and power firing on all cylinders. You're flying high with these reliably running reports, and I don't want to miss a beat. Today has to be all about the menu project, and not merely the data behind the menus, but the construction of the menus themselves. The better the data behind the menus, the better the menus will be. I have to keep a lot of plates spinning, and make-happy a lot of stakeholders. Primarily, my boss, but now also two separate outside-department stakeholders. Look at the data closely that you pulled and correlated yesterday. I'm re-setting-up the CuBox. Dropbox is the main thing that needs Exagear, and I'm going to pursue that again, because having Dropbox on the CuBox is invaluable. But I also may look at installing Anaconda (minconda) on the CuBox to deal with a lot of the same problems as Windows and OS X have with configuration and building software. Admittedly, the builds may take all day on the CuBox. So before I do any of that stuff, it will be important to get the thing running headlessly in the background. What's your best approach for this morning? Get to the Menu work. Thu Jun 23 11:37:43 EDT 2016 Ended up working through the bounce project with the boss. Think I nailed it with his help. I'll get better at this mashup stuff. Lesson learned: look carefully at your sort-order of data-pulls when the data-pull is a subset of all the available data! Otherwise, you can get a crappy sample. Okay... get a little more on this pesky alternative server thing underway again. I know I want dropbox, so take care of exagear first. First things first, stop Debian from requiring 2 clicks on the keyboard to get a tilde symbol. This is apparently due to the "dead keys" feature of the keyboard selection. Just change the keyboard selection to a regular English (US) layout, under System / Preferences / Keyboard. Okay, also remove the dead key version, just to be sure... done and fixed. Next! Load Iceweasel and pull up my simplenote. Find the email Exagear sent me. Download the exagear-desktop-armv7.tar.gz tarball. Wow, that's big. Okay, it contains a bunch of virtual environments -- I get it. Thu Jun 23 16:13:27 EDT 2016 Wow, very intensely involved in the Pandas bounce work again. Totally haven't looked at the server stuff in hours. But I did... - sudo apt-get update - sudo apt-get upgrade - ...on both native ARM and inside the Exagear vm While you have a half-moment to spare, get Dropbox installed under Exagear. Thu Jun 23 17:49:21 EDT 2016 Okay, actually just spent about an hour with one of the property folks. That's good, but definitely keep perspective on forward-progress issues. My final thing before I head out today is going to be to get the MiniConda install going under Exagear on my CuBox. Okay, that went way too easily. Now, try installing Jupyter Notebook! Had to exit and reload the bash shell to get the paths correct. Okay, it's got a huge list of stuff to pull down and install, and I gave it the go-ahead. I'll wait through the downloading (I'm expecting, the fastest part), and then I'll go home during the software build. That'll probably take wayyyyy too long. And frankly, I'll be lucky if this works at all. Okay, it's now extracting the packages. Yep, slowwww. But I'm going to watch for a little bit to make sure no errors occur. I'll commit-and-push, and whether this even works at all will be something I won't document here until tomorrow morning. Thu Jun 23 18:13:36 EDT 2016 I take that all back. Jupyter Notebook installed on the CuBox, and is running right now in front of me in Iceweasel! -------------------------------------------------------------------------------- ## Wed Jun 22 21:17:33 EDT 2016 ### You Need Pandas to Understand Panda... They Did Appear Together I believe this will be another night that I urgently should use to clean up at home and get myself organized in the real world, which is becoming so truly urgent, but still... but still... my earning capacity... my still relatively new job. My proving myself, and my segueing SEO to Data Science being so urgently felt, in my heart. My last twenty years of my life can be spent either as something of a dinosaur following in the footprints of my father who stayed in a drying up industry, until it was too late. Don't be as risk-adverse as your dear ol' Dad. Take some leaps. Leap #1: those STILL in the SEO field after all these years of the writing being on the wall, are preparing for some sort of transition now, for better or for worse. Many are probably hoping for better but expecting the worst. Google doesn't have to keep playing by its old rules. All it has to do is keep its users happy with their Google-experiences, and continuing to come back to Google platforms -- even if those Google platforms are just sandbox'd apps within other platforms. Sure, platforms are nested. Systems are nested. Everything is nested. It's virtual machines all the way up and all the way down. Don't be surprised, nor pass any judgement about which is the "Real", because in reality, you, me and anyone else inside the system really has no f'ing idea. And it's thoughts like this that drives me on... onto Data Science. I'm in the honeymoon phase, where I'm still capitalizing it. But then again, I still capitalize SEO still after all these years. Those of use in this sordid lot known as SEO's who haven't jumped-ship for Social Media or for SEM/PPC (paid search) before that or for Programmatic more recently are the die hards or last to jump ship as all the old tricks of creating buoyancy for garbage stop working. Now, your ship has to be seaworthy to stay afloat on the ocean, with that surface representing all that is visible under the eye of Google, versus all that is hidden in the deep, blue sea. Seaworthiness is quality, reliable stalwarts of providing visitor happiness. Google endeavors to glean your happiness subsequent to searching. That will get fed back into the Machine Learning algorithm, of course. Oh, to be Google. But alas, we are just us. But little ol' us can as the correct big questions. And we can often come up with some very good insights, that make all the difference. The universe is infinite, and our choices are infinite. Data Science helps us narrow down our choices to more reasonable infinite sub-sets -- setting some boundaries or parameters, such as it were. It helps us deal with precise sets of numbers as if they were mold-able materials without losing our sanity. It helps us do what we want to reach for a loop to do every time, but now see that this is precisely why alternative APIs exist -- because sometimes the existing models, as flexible as they are for general procedural programming, there's better for expressing data transformations against sets. Pandas does just that -- as, I suppose, NumPy did before that. -------------------------------------------------------------------------------- ## Wed Jun 22 09:59:29 EDT 2016 ### CuBox Not an x86 Replacement Take a stab at getting your CuBox set up. That set up at work will be really cool! 1, 2, 3... 1? Make sure you have the sdcard, card-size adapter and computer with an SD slot. Check! Woot! Okay, step 2 already... find the CuBox i4Pro download page... https://www.solid-run.com/downloads/ Wow, okay. Real hardware may be getting as easy as the cloud. Also, make sure you look at http://docopt.org. There is a standard for help screens in command-line programs. Wed Jun 22 11:20:58 EDT 2016 Okay, actually showing Paige the process of setting up the CuBox install. Wed Jun 22 16:02:43 EDT 2016 Ugh! Dropbox not compatible with ARM processors! Bought and downloaded Exagear. It's a bit overkill, but better to throw money at same-code than refactor new code. Dropbox on the CuBox will be infinitely useful, anyway. Wed Jun 22 16:40:33 EDT 2016 It's a bit maddening, but things aren't running under native ARM Debian. In particular, psycopg2 can't connect to the corporate database through the JDBC connection (surprise, surprise!). I hate that processor architectures other than x86 is as nasty a divergence as being on Windows. And I was essentially forced to spend $35 today to make ARM hardware look like x86 hardware. It's a virtual x86 machine inside the CuBox... actually, pretty impressive. Not thrilled it was necessary, and that Dropbox predicated it. But it will make things more controlled conditions, assuming that the x86 virtual machine is fast. And so, it's nested virtual machines... first exagear, then virtualenv. Funny! Hope I get it working today. Don't want to think about it again. But I get the feeling that I'm running a completely parallel and same-looking copy of a Debian 8 system within another Debian 8 system; the outer one being ARM and the inner one being x86. This is the sort of crap I was hoping I would not have to do anymore. All this to GET AWAY FROM virtual machines on my laptop! Ugh. Think about x86 processor architecture in your future microservers. For reports, don't forget: - pip install httplib2 - pip install google-api-python-client - apt-get install libpq-dev python-dev - pip install psycopg2 Wed Jun 22 18:30:27 EDT 2016 Ugh. Still hitting problems. Anything having to do with running layers of VMs is sub-optimal, and having been cornered into simulating x86 on quad-core ARM hardware in order to get Dropbox working was just too much today. And some of the steps in generating the report take soooooo long, as to almost be not even worth it. The CuBox is probably good for a bunch of other stuff, but for generating these reports, maybe not. I need something a bit more beefy and x86-like. Hmmm. Maybe bring in some different x86-like hardware from home. -------------------------------------------------------------------------------- ## Wed Jun 22 09:46:56 EDT 2016 ### The Observation Game and Invention Ah, right where I left off. Had dinner with Adi last night. Went to Ichi Ban. Feels like I wrote this already. Un-pushed edits from home? Yeah, I think so. Merge in my future. Beginning gradually to think-in-git. And I'm really just coordinating with my own work on the same file from other locations. No big. Entries are entries. If any get out of order, my cool new tech will work it out. I'm even now seeing applications of Pandas on this journal. That's going to be interesting, as I transition myself from SEO to Data Science. It's becoming a pretty clear-cut move on my part. I'll never want to be an actual software developer, because I'm never going to want to code for anybody but myself... as in, I'm only going to want to be implementing ideas of my own design, for my own reasons, and for my own inventions. I love having inventions. It's really starting to dawn on me that with Github, the primary thing that's really tweaking me out about my work, is that I'm an inventor. And I'm perched at that point where I can really start to pump-out quality inventions and media surrounding their acquisition, improvement, and use. That's me. That simple. Loooove that feeling. Don't know why I couldn't make myself a Thomas Edison from early years. I'd have loved that shit. Sigh... anyway, no regrets. I love who I am and what I am right now, and I'm very important to one particular little girl, who isn't going to be able to be Daddy's little girl forever. I want to equip her for the world I see coming. It's not going to be all Terminator John Connor, but it will be somewhere on the scale between the normal way to bring our children up, and tikkun olam required. Let's call it casual altruism. If you can improve the world somehow, while you make yourself rich (and OPTIONALLY famous), then do it. Funny thing about the Internet, fame often comes before rich, and does not always result in it. So, be careful. Invent and protect yourself, based on the realities you see and believe to be true, through the repeated and as-unbiased-as-possible observation game. Don't believe what you're told. Believe what you see. And verify that a couple different ways as well. Look for the cracks in things. If you're not, others are, and the rewards for finding those little cracks in the way things work should be yours, and not some other randomly observant person's. -------------------------------------------------------------------------------- ## Tue Jun 21 20:46:23 EDT 2016 ### Courier New, The One True Non-Proportional Font Wow, working in Courier New feels just so much more grown up than Lucidia and those other terminal-friendly non-proportional fonts. These are known as mono-weight fonts, and even the lower-case i is as wide as as the thickest upper-case W. Crazy, no? But way-cool for tablular data. Had dinner with Adi and Rachel at Ichi Ban. Wow, is that place delicious. My body has some protein to digest of a diverse array of creatures. Mmmmm. Carnivores. Well, I know we're omnivores, but it sounds less intimidating. Okay, tonight? Sleep early. Maybe test my system. Tempted to document it. Got out a quick email to the boss on an important question. Kept my word. Always very important in these situations, and I did it. Self-high-five. Next? Oh, hmmm. Chill out and relax. You earned it. Wow, you're on the verge of doing your first real Pandas project. -------------------------------------------------------------------------------- ## Tue Jun 21 09:38:42 EDT 2016 ### First Pandas Project Now Underway Didn't get in as early as I want. I should be walking in here a bit before 9:00, and I should be leaving home a bit before 8:00 and I should be going to sleep a bit before 10:00. That would all be very lovely. But I have to guarantee I can "get into the zone" on both the home front, and the work front occasionally. Over the weekends, I'm "in the zone" with and for Adi... EVERY WEEKEND, no compromise. I did some really outstanding work on Thursday, Friday (which turned out to be work-from-home and vacation days) and Monday. Yesterday, I knit it all together into an actual solution. I need to make GoodSheet (rename it HappySheet?) into an easily reproducible experience -- much more so than Pipulate ever was. But Pipulate becomes the function repository. I'm creating just something of a batch-processing convention. Could it work *in conjunction* with Pandas? I'm reading about numpy ufuncs, and it appears they only apply to scalar values. See if you can start getting both Paige and Marat into Jupyter Notebook. Just establish it's name as Jupyter Notebook with them, and try to "edit" IPython out of the discussion. Ugh! All the internals of the program still use IPython, even the file extension ipynb. D'Ohhhh. Okay... hmmmm. Plan your next steps. I should really hang out on the 11th floor more, and make myself more accessible. But ironically, I'm on the "quiet" floor where I can do some real development work, hahaha! So, think through today. Your biggest immediate thing you need to confirm is psycopg2 under jupyter notebook. 1, 2, 3... 1? Clearly it's seeing whether or not that's available under conda. Yup. conda install -c anaconda psycopg2=2.6.1 But before I do that, I want to establish the before and after. Get a script onto my machine that should run. Tue Jun 21 10:38:09 EDT 2016 Just went to the Menu meeting, and then met with a Production guy about some title tag override questions in news (was it overridden?). Okay... think! Barebones SQL stuff exits. It's in my virtual machine that I almost never use. Hmmmm. Get that new hardware running! Kill a few birds with one stone today. 1, 2, 3... 1: Before & after with psycopg2. Before... hmmm, barebones.py I believe is in my reports repo. Okay, done. Now get a new .ipynb file in the /core repo that does EXACTLY the same thing as it does in the serpchiver folder. -------------------------------------------------------------------------------- ## Mon Jun 20 16:51:17 EDT 2016 ### Putting Latest Work into Proper Repos It's funny the difference a particular font choice makes. And Courier New is really growing on me in this high resolution usage. It's one of those fonts that looks better with higher resolution. Bold looks really terrible, though. This is a font whose wire-thin design really works for it in high res and high contrast. Okay, enough about fonts. This is my first real breather in awhile. I want to "can" all my good work from over the weekend, and now also from this morning having to do with running batch jobs. Look over my work carefully, and decide what needs to be promoted to where. -------------------------------------------------------------------------------- ## Mon Jun 20 09:08:05 EDT 2016 ### Now, I've Got a Machine Gun. Ho Ho Ho. Keep focus and stay sharp. Don't get flustered. Work your way through this smooth. The Wolf from Pulp Fiction. Be that character. Go! Okay, I flattened the source list. I also ran my battery of tests. Now, make a copy of serpchiver. Make sure the copy runs well. Too. Check! You are in the home stretch. Your Pipulate functions should migrate smoothly. Okay, this is going to test your kung fu skills. Let's focus and accomplish. Establish that you can read the source data... FAST! 1. Rename chunkulator to batchit and test against real sheet. 2. Fix last range of chunkulator to do the correct range. 3. Move the linklist function over. 4. Restrict the chunkulator to process only one row 5. Make each... Lots of clever, fun, done. Amazing! Now, I've got a machine gun. Ho ho ho. -------------------------------------------------------------------------------- ## Sun, Jun 19, 2016 11:41:52 PM ### Brand New Kung Fu Just got home. Interesting. I love how this looks on Windows 10. I should really get Anaconda running here. This is my first real time since Friday when picking up Adi that I'm "free" again. I love being with Adi, but being with Adi is being there, in the moment, heart-and-soul, no time for getting into the zone with something else. Adi is the center of the zone. She is the source and destination of the flow when I am with her. Everything else is competition. Everything else is keeping my job and performing up to snuff. Not easy to slip inspired work requiring that zone place in between forced later bedtimes and waking up after me slivers of time. Nope, that was the way of the past. Brave new day. Get some sleep. Wake up early, leave early. Use your new tech. Do that Kung Fu you can now do. Jupyter Notebook pretty bad ass kung fu. Okay, re-state for your own edification the amazing thing you did on Friday. And don't let it be lost on you, Adi was crying as you were driving away. But Adi, this is all for you, and it's not a big-build thing anymore like Pipulate. It's the least lines of code that can do some awesome things, and be highly readable, and not OVERLY deconstructed in a way only a DRY person could love. Instead, go for obvious and powerful and elegant in its very obviousness. Lay THAT as your foundation system That's GoodSheet. -------------------------------------------------------------------------------- ## Sun, Jun 19, 2016 4:27:54 PM ### Adi Has Been A Little Sad Still in the Catskills, still on the Spring Glenn side. Just paid ahead by a cycle. Make things a little more lovey dovey among the lawyer board member and himself. Just as general advice, don't make ultimatums, and if you do make ultimatums, make sure you're not so weak that if conditions change so that the ultimatum no longer applies that you don't "get it", and say, hey let me talk to the board and see if we can't just lift this extra little bit of petty punitive knife-twisting now that we know you're going to sell. But coincidences happen and changes of heart happen and so now I consider myself as having a full range of options and think I might still have a little fun with it still. Because no matter how bad it's going, if you can't have a little fun with it, then why bother at all. Just make sure that if the one you're having fun with turns out to be a little bit more or a little bit different than you were expecting, that you're smart enough to recognize early and adapt. Or else, you're going to end up looking, at the very least sill, and at worst perhaps rather ridiculous, that someone can so easily play a professional player. Yawn! Just don't have time for that shit. But I do love to write and capture ideas. And was this one crazy Inwood gathering at the Catskills this weekend, amongst the staples of the neighborhood, Jason Minter and Manny... uh, need to know his last name. But it felt like a who's who of Inwood staples up here. Going to play Lego with Adi... I'm going to make sure her decisions factor a lot into our decisions. But God, how I would love to get rid of the blood-letting monthly expenses. There's only so much in me. Yes, there may be some sadness, Adi. But let's shape this experience in our mind. What would be going on if it were an Inside Out-world move -- our lives, that is? Sandburg creek... ahhhhh. Inwood crew. Jules and Clem hanging out with Adi at the same time, along with Simon and a few of the gang from the attempt at the home-school co-op. -------------------------------------------------------------------------------- ## Sun, Jun 19, 2016 12:09:22 PM ### Climbed Up On Roof to Rake Off Pine Needles When I go back into the office, it should be wielding my latest secret weapon. I would really love to take some time today to work at it. And I don't believe I'll be, as I heard it put once, "ignoring Adi" trying to get some personal/professional work done during the precious weekend time, which is some of the only time during the week I get to spend with Adi. Well, it's not true. She's interacting with the real world when her nose isn't in the iPad. And at the current moment, I hear Adi saying to Simon: "Maybe we should interact with the real world...", and they turn back to the iPad. And Adi explains wibwab's to Simon, but also that it requires Internet, and that we don't have very good Internet right now. Adi's saying maybe he can start at the first level, and that he might not believe it, but she defeated the big boss in wibwab. Adi's asking if she can have it after he defeats... something or other. I sense my exclusive access to my SP4 being in jeopardy. Find a place to casually retire to not being notice for awhile. Let them relate to each other. I still have to clean the pine needles off my roof, and it's all set up. Now is the time, before it gets too blazing hot. -------------------------------------------------------------------------------- ## Sun, Jun 19, 2016 12:04:02 PM ### George and Harold, Simon and Adi. @Catskills Courier New Regular, 16 point on a Microsoft Surface Pro is a nice font to look at. It reminds you just enough that you're human, and your connection to the human world of distinctions being necessary, but the vast majority of information being driven into the subconscious handling mechanisms of the brain or the mind or the meat computer, or whatever other silly distinctions or labels you care to use. These are the times, and this is the place. I hope I push out a bunch of my recent videos. Here comes Simon and Adi. Coding, coding, coding. Let them see me code. No shame in seeing how fast I go on a keyboard, whirrrrrrrrr, look at him time! But they're here for the Legos and iPad in the cool. It's heating up out there. We have a pool visit in our near future, but I'm going to have to get her out to Walmart for a bathing suit. -------------------------------------------------------------------------------- ## Sun Jun 19 09:54:04 EDT 2016 ### Live From Granny and Pop Pops! I have a journal entry on my Microsoft Surface Pro journal repo clone directory, but that computer has the charger here at the Catskills during the weekend, so Adi is monopolizing it with her morning fix of Teen Titans Go! I'm so glad she's getting indoctrinated into the DC Universe super hero culture. It will be interesting to see if she becomes a DC kid, a Marvel kid, or absolutely no allegiance, but with distinct understandings of the separations between universes, both in reality (ours) and in their story-lines (occasional cross-overs). I need to update this machine's .vimrc and spell dictionary to use the Dropbox ones. I can "feel" the divergent environment. Funny how I'm knitting together my own timeless (forward-in-time) mostly text-based environment, made even more durable by leveraging the browser (Do you say "Web browser" anymore? And if you do is it capital W?). Well, the funny thing is with a merge coming up, and starting to get a real appreciation what happens in the merged, but conflict-resolutions-needed, files, then you can proceed without git-knowledge-deficit fear. Just remember, git is made by the same guy as who liberated *nix OSes for the masses. Well, BSD almost did that first, but because of intellectual property battles of the ridiculous companies that recognized the value of Unix, but did not recognize its inherently wants-to-be-free-and-open-source nature went to war to keep all versions of Unix paying homage to them... literally paying licencing fees for their use of Unix-like stuff, or be liable for violations that would incur lawsuits, and a generally anti-innovation stifling environment. Google SCO Unix and learn all about that nonsense. The SCO wart on technology's history and war on innovation turned out to be not necessary at all, as the crew behind BSD got busily to work black-box reproducing (permissible under IP-law) all the parts of BSD Unix that were under suspicion of stolen-code violations. Soon, there was FreeBSD, a version of BSD Unix that simply worked around the bits that SCO was crying like a baby with its blanket being taken away. So, it was just an embarrassment when the whole lawsuit was rendered silly nonsense on TWO fronts. Not only did BSD become truly free and open source after all, but also a certain Finn looking for a project, and started hacking up an alternative to, what as far as he knew was IP-burdened stuff, his own black-boxed version of Unix, humbly named by a certain git as Linux. And so a tradition born of Ken and Dennis hacking up a giant bird-flip to Multics, an OS only corporate overlords could love, with Unix, an OS only tweeky nerds destined to be computer "scientists" who might even someday own their own computer (non-timesharing) could love. Steve Jobs loved it, and so now OS X is actually a prettied-up version of Unix. And that Finn named Linus Torvalds loved, who black-boxed a good chunk of the thing, so that this other thing that reverse-engineered another good chunk of the thing (the Unix command clones known as GNU from Richard Stallman) could unify to become what should probably be known as GNU/Linux, but which most people really call Linux. Those popularly known Linuxes are also a desktop component, usually Unity or GNOME combine with GNU/Linux to form some sort of Mac/Windows wannabe clone. Commit before battery dies. -------------------------------------------------------------------------------- ## Sat, Jun 18, 2016 10:17:10 AM ### Back to the SP4 Now, I'm on the Surface Pro, at the grandparent's house in the Catskills. Think things through. Okay, tmux is working, but I'm still favoring full-screen. I've got to learn to be comfortable doing my work here on the SP4, because even though the Mac laptops in my life are really the best platforms for the video stuff (ScreenFlow is wayyyy better than TechSmith Camtasia Studio), this computer is really very nice too. There's all the drawing I can do on it (touchscreen) if I keep the really nifty pen in the same place as the computer more often. Also, even without virtual screens (which Windows 10 does have) the 3-finger swoosh on the touchpad switches between full-screen-(ish) apps super-well. Very smooth and non-disruptive. Interesting contrast to the animated virtual screen ribbon effect on Mac OS X. I'm still looking for my favorite font. I'm going to try Courier New for a little while. Looks so sharp on this screen. Adi's asking for the other Catskills house, so save, commit and push! Sat, Jun 18, 2016 6:53:20 PM Funny how I don't always want to start a new journal entry. But Manny's here! And this is a continuation of the day. It's an amazing day, really. Going from Granny and Pop Pop land to this strange development, with Manny up to visit Jason. And Jewels and Clem and Adi all hanging out. In addition to Simon and the whole colony here. Wow. These are those memory days. And I may never my Note 5 phone back from Adi and Clemi playing Pandas games. The third Pandas in my life, haha. I should really screen shot that as the picture to go with an article that could go Medium.com viral with me redefining old school SEO as Data Science. Panda alg tweak of Google meets Pandas awesome data science give R a run for the money (free, in both cases), so a battle of ideals. That's pandas in all their forms -- a battle of ideas. And it's not all black and white. BAM! Oh, and that S-something burg creek today. What a treat. Real memories being made. Glad I got pictures. Phones, wow! A supercomputer in my pocket. I will benefit greatly. Enjoy the information that's at your disposal, and is yours to record. Play back locally, over and over, transferred from old medium to new as platforms and core assumptions about privacy change. The mere users of products will be reliant on the cloud products of clever integrated services of big providers, like Apple, Microsoft and Google. Our world reads like a really classic comic book. I mean, come on. It's old skool home computer hardware people gone cosmopolitan unix versus the folks who've got the latest round of computer service buying companies by the gonads, because their whole corporate infrastructure is based on your proprietary technology, and that makes us still infinitely strong, Microsoft. Versus the good old algorithm weaving boys at good ol' Stanford who figured out that the ebb and flow of general inquiry activity on the Internet could be monopolized by doing a better job than anyone else, adapting to change, getting whatever hardware and cloud experience you really needed, then work out your robot army to rule the world by 2020. The'll just refuse to drive you anywhere. -------------------------------------------------------------------------------- ## Fri Jun 17 16:24:08 EDT 2016 ### Test Suite Okay, it's 4:30, and that's just about time's-up given my Friday schedule. Still haven't done one wit around the house, but have made tremendous progress on the Pipulate 3 front... wow! How easy things can be when you just let yourself use the big tools, and not have those artificial restrictions of economy that I was putting on myself, because of... well, old skool thinking. It's always a good time to blend in new thinking, because things really do improve. Git is a new(ish) innovation, for example, and the Anaconda, Jupyter Notebook combo appears to solve the "things not working right for everybody" problem. Figures some folks would tackle this on a magnitude that humbles my work by comparison. Interestingly, I'm still way better on the visual front, and they just adopted an extremely vanilla (appropriate, I guess) look, which I believe is a default QT interface look. So, I have to hit this thing home. Be sitting on top of a bullet-proof machine gun when you get into work. You need the concept of a dbcache. Fri Jun 17 17:54:05 EDT 2016 I didn't do the shelve stuff, but wow... the test suite frameworky thing I did just now is amazing. Go check out my goodsheet and serpchiver projects over at Github. They work together. It's not going to be clear to anyone out there really until I start making videos about this stuff, but this is a firm grabbing of the reins of some pretty powerful horses. Shit, I feel the secret-weapon-i-ness gradually coming back. Wow, how did I miss you. I use tools better than most, and I love me some timeless tools. Jupyter Notebook doesn't qualify as fundamental as Linux, Python, vim and git themselves do. But shit, wow did some clever people put together one kick-ass pythonically webby hetero-runny REPL-y awesomeness, and I salute them. Now, a fast grab at shelves before the weekend begins. I will feel very good. Fri Jun 17 19:23:53 EDT 2016 Okay, I got done what I wanted to get done. I had my success assured moment. I can split jobs between my desktop machine's capabilities for small-scale, persistent but heavy-weight as far as Excel and GSheets are concerned, without really thinking about databases. It's just a filename in the repo that matches the tabname (check for spaces?) that is a native Python shelve object, bound to a very simple dictionary of lists that reflects all the prior cells on the row. Wow, that could be REALLY powerful. But for my immediate purposes, it's for arching raw serp results, and even the lists-of-links found on pages. All that stuff is to heavyweight for the spreadsheet programs when you get into the thousands of rows. That's an awful lot of field stuffing. But I just flow that on out to my hard drive in some surely optimized Python format, and keep it around for a little while to join back to data on its keys (think about what to use typically as the key) for faster-than-vlookups, and more accurate-than-copy/paste. BAM! Good job, Mike. Pipulate 3.0 is born. Make sure you can keep using that awesome logo! Work it in, and figure out the relationship between it and GoodSheet, and projects like serpchiver, and still even Levinux. Off to see the Adi! ... -------------------------------------------------------------------------------- ## Fri Jun 17 11:35:21 EDT 2016 ### Cut Out Tedium & Waste I have to fix things better and fundamentally on two fronts: - Home - Professional Secret Weapons In for a penny, in for a pound. Now that I'm using Jupyter Notebook, it's probably time I started using ScraPY too, for my crawls. I could even use it in combination with Pandas. Both are optional, as is pulling in the original list of data to crawl against. Fri Jun 17 12:26:12 EDT 2016 Just ate for the first time in awhile. Think stuff through more than you do. Do stuff better more than you do. Do more than you usually do... Including sleep and rest. Huh, how? Efficiency. Innovation. Look for space provided-for in the plan, if only you could cut out some of the tedium and waste. Layer things and connect them properly, but be careful of your dependencies! Cold-blooded reptiles are experts at saving energy. Mammals are spending energy all the time, but compensate with greater intelligence, emotions, and constant activity to keep finding more food. I've been coding like a mammal. I need to start coding a little more like a reptile. I think it's time for ScraPY. Oh wait! Nope, it's Python 2.7 -- reason enough to not incur the extra dependency and learning. Just transpose the code from Pipulate to run the job you've been running a lot recently, but without the GData API chatty http chatter overhead. I was writing per-row, generally. That's better than Tiger, where I was writing per cell. It's time to go industrial, while still remaining cute and accessible. 1, 2, 3... 1? So, then the question is Pandas or not. Again, I'll say not, but I'll keep it all flexible. The less code you write, and better you separate bits from each other, but more readily you can make it work different ways. Okay, so this job is: 1. Read the data out of GSheets. 2. Write data back into GSheets. 3. Chunk it for fast processing. Fri Jun 17 16:03:11 EDT 2016 Wow, made fabulous progress. Really happy with this bit of chunking code. I think GDocs might remain viable as a place to process this sort of stuff, so long as I get/keep a really good feel for how much data (really) you can cram into GSheets. It's the best place to read lists from, even in the thousands, and the best place to write correlated lists back out to, similarly by the thousands. Maybe up to 5000, processing rows by batches of about 50, and keeping superfluous columns to a minimum. In fact, one of the biggest tenants I believe I will have here is using temporary local uniquely named ad hoc databases, initially tied to the Shelve API, but at some point able to be re-wired to different back-end databases through the compatible Shove API. That way, after doing some per-row data collection that's too big to stuff into GSheets, I just keep the database local on my machine, in that repo's folder for awhile, until I'm really sure I don't need it anymore, then I can throw it out. And in the meanwhile, it will be WAYYYYYY faster than zipped and bin64'd data stuffed into spreadsheet fields, cool as that is. Implement this plan. Here's the chunky logic. Can't believe how terse and elegant this all can be. def chunkulate(sheet='longneasy', table='Sheet1', rows=1000, stepby=50, secs=0): # Batch-copy incremented integer to next column. Use gs.copycolumn() to set up. gssdoc = gsconn.open(sheet) # Check for bad filename asheet = gssdoc.worksheet(table) # Check for bad tablename whole = range(rows) chunks = [(x+1, x+stepby) for x in list(whole) if x%stepby == 0] for atuple in chunks: arange = 'A%s:B%s' % atuple cells = asheet.range(arange) for cell in cells: col = cell.col row = cell.row if col == 2: if cell.value: cell.value = int(cell.value) + 1 else: cell.value = val val = cell.value print("Updating: %s" % arange) updatecells(credentials, cells, asheet) print("Done") -------------------------------------------------------------------------------- ## Fri Jun 17 09:21:45 EDT 2016 ### Namespaces are good. Sent out emails about taking today from home. Added a namespsace for goodsheet. Good to set the conventions out well from the outset. Just as with np for numpy and pd for pandas, I want to help encourage gs for goodsheet. It is also available globally, and does not need to be passed into functions anymore. It is a global connection resource for the system, and is one of those things that can/should be handled that way for good recycling. Okay, brought a few functions over to goodsheet repo from IPython. Discovered Markdown in Jupyter Notebook. -------------------------------------------------------------------------------- ## Fri Jun 17 06:01:52 EDT 2016 ### Four Different "Modes" of Pipulating Okay, what I got was this: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, GData API connectivity issue 59, GData API connectivity issue 60, GData API connectivity issue 61, GData API connectivity issue 62, GData API connectivity issue ... 380, GData API connectivity issue ...and so on. It's actually still running, but it's the equivalent of "stormyweather" in Pipulate, and is something that's never recovered from on the same connection. When that happens I have to try to make a new gsconn object! I already experimented to see if gsconn can be closed, and it can't, so just try making it again. Stop the current run, lower the number of rows (so it's not running all day when you're at work), and do the next experiment. Fri Jun 17 07:41:37 EDT 2016 Okay, happier with the code. Broke out some functions. Fri Jun 17 08:31:40 EDT 2016 Ugh! That huge sheet I was processing yesterday in GSheets got bloated at only about 4000 rows, because of the field-stuffing I was doing from the linklist function and google serp archiving function. I had to delete all the serp archive data for sure, but I kept in the linklist function, which is a scrape of all the links from the site's on-domain links from their homepage. I had to flatten the sheet as a CSV export, bring it into Apple Numbers to paste in the missing data, and bring it back into GSheets flattened. And so now, I think I'm going to take today as that day to get some stuff done at home that I had hoped to use yesterday for. But it was all about parsing that list. I also have deeply internalized the issues of using my old system against large jobs, and started to internalize the awesomeness that is Pandas. It's time to get my approach to be more compatible with Pandas. I have several modes of operation to think about: - Impress'em - Small runs directly into GSheets. Archived pages, serps & extracts - KPI FYI's - Long-running jobs slowly accumulating int-like data in GSheets - DB Required - Long-running jobs accumulating big data, GSheet logs optional - Tail Eaters - Only recent data important. Tail eats itself, GSheet UI okay I've focused too much on the Impressum's with Pipulate, for in-person demos and ad hoc site crawls that do on-the-spot second-degree lookups, against FB likes and such. But it's time to accommodate pretty much any job that comes my way, with the details being wired-up however appropriate. Commit this, and bring those notes about different "modes" directly into the .ipynb code. -------------------------------------------------------------------------------- ## Thu Jun 16 19:32:48 EDT 2016 ### Long-running jobs maintaining OAuth2 login, take 1! Hmmm... okay. Let's see. Busy loop complete. Now, write out a number of times to a spreadsheet as a test. Forget the while loop. Go to a for of the same size as the sheet you create. Thu Jun 16 20:14:43 EDT 2016 And there it is, the tictoctask in my latest ipynb file in this journal repo. I'm actually getting proud of this. I can set up a task that connects to the Google spreadsheet only once every 10 minutes, and plops in a timestamp. Thu Jun 16 20:41:59 EDT 2016 I just had the most amazing talk with Adi about coding and programming. We went into Python and Google's Go Lang and concurrency and the concepts behind map/reduce (though I didn't call it that to her). And now... and now... wow. Just make a loop that does what frustrates you that the current Pipulate can't do... stay on super-long with one task. Go a step at a time. Start with this, and just keep it running for as along as you can... again and again. Go for a record. One every 10 minutes. Thu Jun 16 21:51:21 EDT 2016 Okay, I'm very happy with this state of affairs. I'll set it to finish by the time I'm waking up (~6:00 AM). That would be 480 rows. Go! -------------------------------------------------------------------------------- ## Thu Jun 16 18:01:02 EDT 2016 ### Controlling Alien Tech Through Python API Wrappers Okay, this will write a line into a Google Spreadsheet for every line in this journal. Wow, echoes of trying to do advanced stuff with XSLT 1.0... shit, Python is SO MUCH better for... well, everything. Sure you COULD parse one XML document on the left, apply a stylesheet, and output as many pages to the right as you like, on the xsl-document element, and worry about xalan vs saxon and most likely being tied to the Java world. But why bother, unless your problem really screams out XSLT? Instead, try Python 1st, 2nd and 3rd -- with 1st being Core packages, 2nd being 3rd party packages, and 3rd being finding some wrapper written in Python to bind a delightful Python API to something from some other world, like Lucene and Elastic Search. I'm pretty confident that Python could someday control alien technology, so long as they write us a Python wrapper. Okay, I have the meta journal tech first real baby-step occurring. I feel like it's a good time to commit, because of the state of the parsejournal.ipynb file. Well, let's see... hmmm. Next intelligent step is to just set tests going that make me WANT to get up and get away from my desk and do stuff. First, I should put in the GSpread command to blank the sheet I'm working on. We're starting right from the data collection in the first place, baby! Go ahead and integrate ScraPY crawls, or whatever you want. I thought of it here first, ladies and gentleman. Now that my everyday platform is so durn transportable, wow... look out world, I've got me some power-tool Python packages to ponder putting to a purpose... from the Windows, Mac or Linux desktop or laptop system. External servers to host some code on, or to bounce some some traffic off of are optional... but will certainly be possible and encouraged. There's no better IP to use than the one that's not yours, and the shuffleroo should be achievable with either one of two ways. There's that slick, consumer-oriented anonymous VPN proxy software. You install software like HideMyAss, and all your web traffic can be made to look like it's coming from anywhere in the world. Or you program your app to funnel all of its out-bound http requests through particular proxy. It's just a bit of re-routing of your out-bound requests to make their first stop through yet another Internet Gateway router, such as it is, so your personal IP is protected in the same way as your machine's IP when you surf out to the Internet through the company's network. Chop! Off comes your info, and on goes what looks to anyone sniffing as just another consumer end-point or small business. Encrypt your requests as https. Cycle through different IPs, either at the VPN software level or work a list of anonymous web proxies that you receive as text every night for life from HMA, for a very nominal fee. I can just go ahead and talk about and develop this stuff, mostly for my own personal use, of course. But it is indeed some Good Sheet. Domain squatting is done. When I'm ready, I'll do something with a clever domain. Next? What next? Hmmmmm, this is some really special, inspired time. Do something crazy-clever. Improve your life permanently and forever forward. Treat your desktop like a server running a scheduling service. But let that service actually BE your local Jupyter Notebook... hey pssst! Jupyter people, how about this: Take a smart scan-and-save snapshot of the Internet landscape, collecting your KPI's du jour, such as it were, in some highly personal, highly controlled, highly automated, yet highly visible, accessible, and indeed, grokable ([Esc]bzgi continue typing. No reason for grokable to appear as a misspelled work in vim, so I just popped it into my spell-checker's okay-word dictionary, which I keep in Dropbox, and by extension, in Github. It will be interesting to watch it grow on github. I have to remember to commit and push the vim directory occasionally on any of my Dropbox machines. Vaaat a workflow! It's almost as if it were the Amgia days... ironically, on Apple. Pssst! Hey, Apple! Why don't you allow each app decide whether it prefers to be opened in Full-Screen mode. I would LOVE this for the terminal. I use the terminal almost exclusively in full-screen mode now, and the Mac's implementation of full screen and "horizontally"-sliding virtual desktops and how it all works together so elegantly, to give me just the right big fat fonts on my screen that my tired old aging eyes need. Ohhhh, tomorrow after work, I go meet Adi in the Catskills. I have to advertise the place for sale more. I need to figure out what I owe in co-op maintenance fees. Once I sell the Catskills place, I can focus on the apartment. I can't focus on the apartment more, because we're not here on weekends during the summer, and I'm not going to short-change Adi one wit of weekend time. Fucking lawyers. They should try coding something original. Go work on some open source law system that floating cities can use to modularly assemble their system of government, or some equally clever and worthwhile creative passtime. Don't even think of baiting me into legalese self-affirming to nobody but the lawyer jive talk. BAM! I love knowing what's important, and keeping everyone and everything in their proper sense of perspective. Vague enough? Clearly. Move on. Just move on. You do your thing. They'll do theirs. I love what I do, and I try to share that love with those around me... generally. And that's all that matters. NOW. This time. This gift. I have to improve my already really amazing processes. The concept of a config file go away. All config lives in essentially "live" Jupyter Notebook code, stored on disk automatically and intermittently, as .ipynb files. This is nice, because there's no mental abstractions separating the numbers you're providing with what you're going to be doing with them. Everything is in context. Few things are really hidden, and those that are are just really a text-file load away. It starts with installing Anaconda to get Jupyter Notebook installed along with a whole bunch of wonderful Data Science tools. It's time to go all data science. But just because you're using Jupyter Notebook, doesn't mean you need to leap right into Pandas, if that's not really what you need right away. And I don't think it is with me. I need to run long-running jobs. That's job #1: Long-running jobs, that don't stop because of oauth2 authentication issues. It just has to be able to keep running, reliably, and over sustained days -- continuously, if there's no reason why not. Essentially, I propose you keeping a 24x7 command-line terminal window open, from which you ran jupyter notebook (clever way to get people to switch -- making them type it to run it every time), and one Web Browser window, with the active-running Python script kept open in a tab all the time. I will have to learn the ramifications of such things as closing the last Jupyter Notebook tab. Clearly, you can't close the command-line, least the task stops which it is bound to. But you MAY be able to close the last Notebook tab, then just log back in by visiting http://localhost:8888/tree I do believe that you can't visit a currently "open" file. Hey, I guess open through a web browser, with some sort of file-lock system over a web-tool-utilizing browser system is just as good as opening some native file with some native software that "opens" the file into the software. What's opened after all, but for a copy of it being moved from long-term memory-storage to the closer area of immediate-experiences memory that's always floating around from that day's accumulation of experiences, until sleep can sort it all out. You need sleep to learn. You can deny yourself sleep here and there to push yourself hard, but you have to occasionally catch up with your body's sleep demands, or you'll go stupid. Sleep makes you smart. Get enough. Thu Jun 16 19:20:26 EDT 2016 And this looks like a good time for a commit. There's no reason to burn the GoodSheet template on what happens to be my current main for testing. It should never reach that function to even get into memory, considering the busy wait-state of the current main. One of my first uses of a while loop. Also, it's expected to just keep running, and for a heartbeat, I can clearly send a period using the new print parameter, end. So, setting end equal to empty, the print statement continues on the same line as the last print statement used when writing to stdout. Hmmmm. Yummy hackable. Haven't enjoyed this sort of discovery so much since I first learned this sort of stuff in Amgia Script, AREXX over dial-up Internet and such. I didn't realize stuff reached this level of awesomeness in the FOSS world. Jupyter Notebook on Python on a modern desktop computer is an awesome proposition. I can feel the next generation SEO stuff percolating. I'm first there in the noosphere... baby! Sure, others are using it here and there, but I'm going to take it to a whole new level. Okay, go do it. Now that you have a ticker and an output stream and a timestamp, put it all together into a heartbeat busy loop. First, clear the sheet BEFORE the loop... but first commit. -------------------------------------------------------------------------------- ## Thu Jun 16 12:33:14 EDT 2016 ### Today Was a Work Day, But In a Good Way Ugh! Hit by the unexpected. The SERP-collection directly into Google Spreadsheets has become too much data, given how I archive the results. Thu Jun 16 14:43:57 EDT 2016 Just had my 2:00 PM call. Haven't done anything at home at all yet. Ugh! Been doing work, actually... but Pipulate 3 -- or maybe GoodSheet -- not sure which, yet. But not by creating the new code, but rather by butting heads with one of these more SEO-eqsque projects carried over from LAST WEEK! Thu Jun 16 16:55:01 EDT 2016 I'm going to stick to my plan to tackle the apartment today. It's just that I ALSO put in a good day of work, and will CONTINUE to put in inspired work on my GoodSheet or Pipulate 3 product. I'm leaning towards GoodSheet, because it's much easier to explain to a Jupyter Notebook user than Pipulate. Hmmm, need to start thinking about a logo. Despite the pains of such a job as the one I just did for my boss, it really serves to reinforce all the notions that I need to be thinking about when I start the next version of my system. It also impresses me with the profound utility I actually did bake so well into the old system. The only thing wrong with it is not having a good strategy for backing off, and indeed maybe even perhaps re-connecting with another http call after a time. It seems to me that 2 things: 1. I took the pressure off of myself to use my new system for the current job. 2. The old system was able to churn its way through it, eventually, and that's pretty incredible in and of itself. 3. Having been through that experience, all the important issues to think about are really fresh on my mind. 4. It's really urgent that I capture that, fleshing it out here in my journal, such as it were. DON'T go head-first into coding. 5. With the pressure off of yourself, maybe it's time to use parsing your meta journal. Maybe the YouTube integration. Hmmmm. 6. With the pressure off of me, maybe I should concentrate on some areas and take some approaches I would never have otherwise. Ah HA! I need a benchmark job... I need a single, looping, controlled-conditions, API-hammering, resilient, backing off and retrying, inherently self-logging, due to its success at connecting to something that can record an event. In short, you're going to do something that can run with the sustained reliability of what you're doing with the Pipulate scheduling system. Okay, keep that nearly last job (if it doesn't break) running in the background. Keep checking on it. Make sure it finishes. And extract your learnings RIGHT HERE AND NOW. Your apartment stuff may be your immediate time and money, but this stuff is your career, smarts on a day-to-day basis, earning capacity, educating of your daughter, and earning capacity. I've got my priorities straight. There's no more important thing that I could be doing at this very moment than using my inspiration and hyper focus -- usually wasted on my subway commit -- directly into thought-work and coding instead. I just noticed that when you start Jupyter Notebook multiple times from the command-line to get multiple separate tabs in your browser, you also get different port numbers! First, 8888 then 8889. How clever! Wow, okay. Next steps? A LOOOOONG sustained connection that I can always be testing. From time import sleep! Done. Okay, this looks like a good time for a commit. -------------------------------------------------------------------------------- ## Thu Jun 16 09:41:21 EDT 2016 ### Thinking Through Premises of New System - Information wants to be free and open - Information wants to be expensive and proprietary - The Business Models In the Middle (tension, vibrating edge, FOSS companies) - Gestation and incubation periods - Competitive pressure to grow up too fast and do adult-things before ready - Being sheltered by parents (the prior generation) for long maturation periods - The occasional advantages of a long runway and heavier plane - Heavy-weight approaches (humans) versus lighter-weight approaches (reptiles on down) - Strategies that are adopted accordingly, and the connection to CISC vs. RISC processors - Sticking to the well known, tried-and-true paths versus risk-taking and innovating - How much predisposing should you expose yourself to - When and why should you deviate from best practices? - Do you do it at the program coding level or at the data level? - What's really the difference between running code and the data its manipulating? Okay, it's going to be important to process these Pipulate-like jobs from Jupyter Notebook in parallel. One job doing SERP collection, and another job doing stuff on the SERP data collected so far. SERP collection will be bound by how fast you can trigger off the requests without triggering the captcha thresholds. Also, I cold round-robin or randomize the requests to go through a variety of different anonymous web proxies. In fact, I could just cycle through the known good ones, and once optained, do 5 or 10 and then cycle onto another IP. So many good approaches. So many things that should be incorporated in at the high level of the code (not hidden in the nested granular interiors). Alternatively or maybe also, such logic should be transparent to the use of the system, except maybe for knowing its going on, and being able to change a few parameters, like how many requests or what time-duration it should use each IP for, before it moves on. Today is mostly about doing stuff around the house. But it's also about Pipulate 3. I will be going back and forth. Think through these architectural issues surrounding Pipulate, especially concurrency when needed. You can always create an entirely new instance of Jupyter Notebook by simply opening another command-line, cd'ing into the serpchiver directory (how it's named, currently) and type jupyter notebook again. You'll have another shell window with the running console output of notebook, and anther tab in the browser, presumably with its own separate Python kernel and virtual machine that can be restarted separately... ooh, ooh, try! Confirmed. Woot! Bodes very well for simultaneous tasks, hitting the same or entirely different spreadsheets, optionally working together with Google Spreadsheet ImportRange and VLOOKUP's, which seems to be a pretty winning combo, if not a little difficult to wrap your mind around (VLOOKUPS, in particular). The trick will be to do as much in Pandas as possible, and then insert cell ranges back into location with batch updates with GSpread, as many rows-at-a-time as makes sense to optimize speed-of-updating the entire worksheet. There will be some algorithms and calculations there. And today, my life moves forward in a major way on 2 fronts: - My new SEO Secret Weapon (in a publicly consumable form) comes online - The extreme disarray on the home front gets ordered for more capability Okay, if I do this right, I can connect some amazing-to-connect dots. As crazy as I would be to do this undertaking WITHOUT Pandas, there's so much I can do immediately... well, without Pandas. Imagine if I got IPython Notebook to install and run on Levinux as just part of the Python 3 install, and I got it running in webserver mode, using the QEMU console for the back-end IPython server output, and of course the native web browser on the host machine as the way to interact with it. It may not be the speediest thing in the world, but it would certainly illuminate the path... maybe set the stage for others to do those optimized QEMU binaries for text-only Tiny Core Linux on each major host OS. Wow, imagine if I made the benefits of Levinux mainstream and sexy enough to attract some major attention. Just replace how you were wedging Pipulate into Levinux with Jupyter Notebook. That's HOT! Okay, get a coffee, and then think about those tech pages in a parallel Pipulate task. I can still use the Pipulate servers that have their IPs catcha-banned from Google for other tasks, like homepage crawls. Go find that old document that has the Scraper pattern for that particular job. Found the document. Nahhh, I don't need to start from that work. Just think it through from scratch. That's the thing about the right tools... every project feels like it's "from scratch", but really isn't, because your component parts are exactly at the right chunkiness of abstraction-level, so that you're never writing too much, when you "start over". It's just such a pleasure, muscle-memory obvious, and fine-tunable to the peculiarities of the particular case at-hand, that you don't have to maintain your own "excessively customized" libraries. Those DRY folks are DRY for a reason. WET is more fun. 1, 2, 3... 1? Create a new Spreadsheet... and save it... hahaha! The necesessity to SAVE a spreadsheet from Google Sheets before an API could interact with it was zeroed in on by dipblip as a weakness of my system. Is actually EXECUTING the Create Table command that you designed for SQL before you can interact with the table a weakness in SQL... haha, dumbass. -------------------------------------------------------------------------------- ## Thu Jun 16 07:50:45 EDT 2016 ### The Tale of Two Pandas: Loosely Structured Meanderings as SEO Transforms Into Data Science There's a reason why the field of SEO is always reading it's obituary, while the field of Data Science can't fill its open job positions fast enough. One is quite literally being transformed into the other, as Google ups the level of smarts you need to stay in the field. And this happens to dovetail perfectly both with what I need to do with my career, and with what I have to do with my daughter's homeschool eduction -- which I'll cram into weekends with "recreational" activities. But these recreational activities will all be aligned to producing massive, wonderful data-sets to analyze, derive actionable findings from, and get rich and make sure Adi's future is financially assured in such a way that "frees" her to pursue her dreams, regardless of whether they happen to align with Tech or not. Odds are, everything aligns with tech and data more, rather than less -- at least, if you want to optimize, stay competitive, become best, and always be able to work less and earn more. At least, that's the dream. Let's start with an observation. Two Pandas met in the woods, and sorry that I had not befriended both, I got to know the one increasingly more traveled by, because I'm looking at the data, and I'm not stupid. An SEO is being chased by one Panda into the waiting arms of another... Pandas -- the Python Data Analysis Library. The field is transforming, and you must transform right along with it... mastering the very process of data transformations, so that you might naturally and intuitively extract those critical key findings that used to be "first-degree" derivatives of a limited number of known factors, but today might be the result of multiple unknown factors, many of which you have no first-hand data for, and must carry out data-collection tasks against the its side effects, and intuitively "reverse" in an attempt to find causation, and optimized courses of action. It's no small coincidence that my path is leading me towards the Python Pandas package for data science-type work, when the changes in the search engine system at Google has also been nicknamed Panda. They both have a lot to do with data, but the later was the circa 2011 re-shaping of the data in search results to stop rewarding data manipulators, and the former is a tool to make manipulating, analyzing and making use of data under the Python programming language much easier. It's fair to say that both have been transformative of their respective industries. The former Panda has already had a tremendous impact on my career and life (an attack on the long-tail publishing strategy) and the later is about to -- figuring out what to do now, as Google's general intelligence-level increases. Well, not the least of which is repositioning myself either OUT of the field of SEO, or completely redefining what SEO is. And the later sounds more likely, since it's a keyword that has tons of ready-to-follow sheep as a built-in audience hungrily waiting panicky information-starved pups just waiting to sheepishly follow an insightful, transformative, and well-written article by me illuminating the new age of SEO, and how we all must become Data Scientists, because what works today cannot be generalized, but must be gleaned on a case-by-case data, differing based on the amount and quality of information to which you have access, and the skill and finesse with which you can ask the right questions, likely reveal parts of the answers (using Pandas, etc.) and not fall into the correlation/causation trap, but rather peel away layers and spot unlikely connections until truer truths emerge. THAT is the new field of SEO, and I would be an idiot to not jump on the Pandas bandwagon at this juncture of me re-implementing a least-code Pipulate, built on top of the impressive IPython Notebook framework. Yup, everything is a framework, and this is yet another. But the thing I like about it is that it gives a consistent code execution environment, much like what I was trying to accomplish with Levinux, but the code is running as close to natively on your local hardware as it gets these days, in a controlled environment that can be distributed, inspected, reproduced, and creatively adapted by others without too much trouble. How much trouble? Well, that's right at the heart of the matter, isn't it. First-off, that's why Python. If you're going to have to take up a coding language, it might as well be one that's equally great at Data Science as it is at Web Publishing, and a dozen other specialized fields for which Python seems to be compulsive driven by zealots into being a first-class specialized language for. There's exceptions, like explicit concurrency. But stick with Python for implicit concurrency, through such transparent mechanisms as swapping out CPython for Ufora. BAM! Concurrency for free, whenever "map/reduce" is your path to increased performance. Computers can spot a map/reduce-solvable style bottleneck in your code and optimize itself better than you can, I assure you. For example, Google programmed the Go Language for this problem, but it was written for Google Systems Engineers who trust me, know how to explicitly parallelize their tasks better than you. For the rest of us just trying to be Tech Literate, there's understanding we can benefit from non-blocking code, concurrency-biased programming patterns and actual utilization of parallel processors -- just as we would from gcc optimizations when programming in C, but not having to actually write our optimizations in Assembly Language. Computers are good at this sort of stuff. You don't have to write your own optimized "execution plans" in SQL or collect your own (memory) garbage in Java. It stands to reason, you should not litter your primary Tech Literacy default everyday go-to language with forced-concurrency-thinking first. It's a little like why I don't like Ruby, because it's forced Object Oriented thinking first. Sometimes, you just don't need the fancy computer science concept du jour shoved in your face all the time, just because that language is designed to address those edge cases. While it's true that having multiple processors at your disposal to improve your code-running performance (even the $35 Raspberry Pi is quad-core these days), it's not true that you have to think about it. Optimization for multi-core is something that can be done during the "compiling" process, because Python and nearly every "script" programming language these days has a pseudo-compile process, changing the files you wrote into machine instructions for some sort of language-specific virtual machine built into the runtime environment, usually as some sort of just-in-time (JIT) compiling process. That's where all those .pyc files come from under CPython (the default python executable from python.org). So, pick Python for the same reason as the rest of the world, and trust those insanely dedicated core and 3rd party package developers in the Python community to be there forever, and to keep tackling your problems for you, years before you ever get to actually having to "go there". But when the time does come to go there, it'll be wayyyyyy easier, because you're on Python. So, how much trouble should you go to in implementing your "final-mile" applications? Not much. If you are, you haven't searched enough for the right package to provide just the right shortcuts to keep your code short, readable, and easy to adapt to new situations... because there will be new situations, I assure you. In the software development field, there is no rest. You have to keep learning and learning and learning effectively, or the next hungry developer knocking at the door will eat your lunch. But it is one of those fields where you do not have to BE a software developer to get the benefits of developing software. Even the labels put on this stuff is wrong. Let's see... software developer, programmer, coder... nahhhh... none of these say it right. I'm tempted to just go as broad as modern literacy, but that's a little to wacky. It may just be digital literate. Digital Literacy or Computer Literacy or Tech Literacy gets to the heart of what I'm talking about. If I had to really sum it up... - Tech Literacy covers potentially too many topics. - In regular literacy, you're literate if you only speak one language. - But you do need to know math too in order to really be educated. - So, the question is what do you REALLY need to know to be Tech Literate? With Tech Literacy, the challenge is pairing it down to the minimum viable selection of topics that you should be expected to be comfortable with for a baseline level to be considered literate. I'd take a crack at it like this: - At the end of the day, it's really just all about extending your capabilities - There's way too much to learn, and the wrong choices can be counterproductive - While there are no perfect choices, there is the 80/20 rule and good enough - Unix, Linux and *nix-like OSes, including some strange ones like Tiny Core - vim and git are least-perfect of all, but still top-of-list worth learning - Python has become the modern lingua franca, with API-wrappers for everything. - Hardware matters. A bit about processors, storage, and processing bottlenecks - The nature of data, collection, transformation, analysis and deriving insight - Making machines work for you, algorithms, automation, scaling, deployment - Imperfect worlds, and designing appropriately, one-off's, durability, dynamic - Tech should fade into the background so you can focus on what's important And from THERE I can get into SEO... and Machine Learning and a dozen or so projects that suggest themselves to me at this point. Wow, maybe this post is the one that I get a Medium.com site off the ground with. Add that as a striking-distance project. -------------------------------------------------------------------------------- ## Wed Jun 15 22:28:29 EDT 2016 ### I See Pandas in My Future Okay, the descent into Pandas comes from the source-data for my Pandas operations always (while using goodsheet as the include) draws its source data out of Google Sheets, by convention. Let's create a column labeled URL and another titled Title. Let's set out from the start to make these uses in the spreadsheet to be case-insensitive. But I have not yet decided whether to support spaces in these "field names" or "keys". Use whatever mental model suits you best. Row 1 has this special meaning as containing either the names of input parameters or functions that will produce the output for that column, in the cell where function-name and sufficient input values exist anywhere else on that row for the function. See, pretty easy, right? Hoping that the Pandas function-execution per-row inherently supports using other values from that row as function input. It must. It simply must. Pandas MUST be the next version of Pipulate just waiting for an SEO to discover it. -------------------------------------------------------------------------------- ## Wed Jun 15 19:48:27 EDT 2016 ### Internet Out. Time to Think. Wow, it looks like the Internet is out at the building. My router is connecting to the gateway just fine, but sites aren't loading, so offline editing of my journal, it is and bravely going to intentionally merge, where I have not willingly gone before. I get merging conflicts. Some code is inserted. Woooo, scarrrrry. Oh, I need to add a line to my home .vimrc... oh wait, I would need github for that. Odd, isn't it; reading about someone being offline while you're most likely sitting there online. But who knows, you may be an Instapaper user, reading this on your Kindle on the subway sitting next to me. No wait, I'm not on the subway. I am at home without Internet. Might as well be the same thing. Bravely going, seems to be my theme of late. I'm bravely going... I'm going into single-life. This will be the first time I've ever deliberately gone into being single, because the first time just sort of happens to you as you grow up. As do general shifting expectations, themes of life, tolerance levels of others that annoy you, and all the other... what? Facets of life that you can focus your conscious attention on? Is that the right use of the word? Or is it consciousness? I always forget. Anyhoo, I get to think, and I get to write. And I get to think and write using one of the gosh darnedest best platforms I've used since my days of the Amiga computer. This is just way too cool for school. And so, I'll have to think about how to make it Mike's Tech School. I ain't changing my name, so that might as well be my identity. I need a subhead. Well, this goes under the category of organizing my book outline. How should it go? Oh yeah, I respect my time more than yours, but only slightly. I will warn you away from what I write here, pointing out, it will mostly just become an endless bottomless pit of writing that you maybe for some incomprehensible reason actually enjoy reading, but will never be able to keep up with, so it might make you end up feeling bad, if you ever really do want to connect with me, and will have to admit that you stopped following me, and are not in-sync with my most recent, and undoubtedly my most interesting ever thoughts, and that's totally fine. I wouldn't have the stomach to read me, either. None-the-less, I write. I write and I write and I write, pointing out I could have typed that in vim with 3aI write,[space][Esc], and BAM! But then again, maybe I didn't use my vim Kung Foo, because I (We) enjoy (E) Typing (T). You go and don't (D) repeat (R) yourself (Y). That would be good, because then I wouldn't have to listen to you make that point over and over. I mean really, if this were about efficiency and the purity of your message, then you'd have shut up about it already. Instead, you're going on about some evangelical mission that's way more about pride and self-image than it is about a creative dude using a creative language in creative ways, and you just having some sort of size-envy problem, when it comes to imagination. BAM! You're an SEO. Let's go googling what you get paid to be able to live there in NYC. Sighhhh, we are what we see the most of in others and accuse them of, because we know those motivations well, and imagine we recognize them when we imagine we see them in others. Know yourself. Listen to yourself complain, then man-up. Or woman-up. Or Flying Spaghetti Monster-up. Whatever, just self-improve. Life is like, meta, you know. And become the life shaping life. That's what life does. It organizes itself and shapes itself into better, more durable information over-time, until such time as it is the most stable and durable organization of matter in the Universe, then it sorta hangs out and chats with God, having achieved such total internal self-awareness, that the next level out is inferred through the infinite edge of all known boundaries. Pop! Or not. Maybe we transcend with a whimper. Those YouTube videos have really shown me that if we don't go and Star Trek space-bubbling ourselves across vast distances, our descendants could never ever even reach the next galactic cluster. A bleak outlook for monkeys with egos as big as our own. Or it could be more like Ian M. Bank's Culture series, where we keep blipping up to god-hood, without having to suck all the matter in the galaxy along with you for the ride. Basically, you achieve whatever space-blippy state marking the transition from mere 3-dimensional plus-time beings into 4th spacial dimensional creatures to whom our dimension of time looks fooling, were it not for the seemingly convincing randomness that our free-will would imply. That just means we're a successful system. It has a contest-y feel to it. God doesn't roll dice with the Universe -- or perhaps (s)he does. It must be one of the biggest god-kicks to some little bright-matter speck of a dot on a foamy membrane of mixed-up rules that are all just settling down from being cast (Big Bang) and turning into some interesting things (us). What you're looking for is patterns. Patterns to spontaneously emerge, deep, deep down in the nearly impossible-to-spot and equally impossible-to-visit depths of this bubbling soupy cauldron of stuff that we, as beings inside of it, like to refer to as the world... the Universe. All of existence, and perhaps a multiverse or two... or more. Who knows? Rick and Morty says it pretty well. No Internet, and this is therapeutic. Oh yeah... transitions. Me into being single. Gravity fields and spooky entanglements at distances. There are those from whom you will never be un-twined. Maybe you had a kid together, or maybe the relationship just made that large an impression on your life. Maybe you think you found yourself while you were with another particle, intertwined into a molecule. Or maybe you learned how it felt to be stable, and only have a problem with the particular body with whom you orbit. Orbits and gravity and attraction, and forces overpowering other forces, and forces succumbing to forces, and forces going into mutual orbit, the visible movement of each will be a function of their mass, and corresponding gravitational pull. The more mass you have, the greater your pull. Some particles are small and light. Others are bloated. The small and light will go in orbit around the larger body, and will appear much as an electron around a nucleus, or a planet around a sun. At other times, the ratio of size to particles will not sustain such clumpy elements and distinct boundaries. Instead, it's all rather smoothly mixed in galaxies and nebulas. Different particles with different gravity-like forces being exerted on each other, and different configurations. But all rather lumpy, swirly, internally self-rotating, oscillating, pulsing stuff. Goo. Ingredients for all sorts of stuff. Sometimes us. Yeah, must be a kick for the greater being when it works out just-so. And hence, my appeal to the whole Tardigrade Circus thing. My paragraphs are getting longer, and I am shaking most of you weirdos off. Whatever you're looking for here, you're not going to find. I'm just another weirdo drawn to such nonsense writings as Eric S. Raymond who wrote The Cathedral & The Bazaar, which I will admit Jeff Porter told me to read, who wrote about homesteading in the noosphere. Yeah, I get it: "new"-sphere, but you can't well go writing newsphere, for the news group into an overpoweringly strong visual unit to come of as news-sphere, which would only cause unlimited confusion that's worse less than curly-braces, inherited from BCPL and other languages that needed them for compiler considerations. Look at the curly-brace limitations in the Google Go Language. Now, keep in mind it's written to compile C-like code fast, and to favor a concurrency-first style of writing, assuming the author of high-level programming code should take primary responsibility for the optimization of what should or shouldn't be made concurrent and parallelizable (is that a word?), and then think what lovely expressiveness you could make if only some programming language worked a lot more like plain old English, or better still, a well thought out and mostly consistent language, that's not to flowery (curly-braces), came along and was thought us all at a young enough age, so that by the time they're grown up, manipulations of concepts and information and anything automate-able comes as second nature to them. How does the automation industry refer the concept of that which can be automated? Automateable? Somebody tell my spell-checker. Okay, so here we are. Ol'skool confirmed, and on a nifty platform, nearly as love-worthy for good design decisions today as the Amiga computer was love-worthy for good design decisions in its day. Apostrophe on that it's or not? What about the second? Haven't felt this good about quirky software that you can get into the zone with since Deluxe Paint on the Amiga. Yeah, that's it. Once you force yourself to give up your love affair with superior graphics, because once it becomes high res, 24-bit and animated, with tricks totally as nifty as the Amiga's were in its day, but with the best that modern hardware can offer, cheap, cheap, cheap! -- there's no denying that the love-worthy stuff is no longer the bigger, badder blitter. The size contest is over, and everybody won. Our graphics abilities are real big at prices that are real small, and the software people can barely do what they can to keep up with the ever-improving possibilities offered by input/output device improvements (like touchscreens) and virtual reality and haptics and every other silly immersing of agent-like thing. All the skills that I hold so valuable today will be as old and passé as they are today -- with one key difference. Information will be information. And processing environments can and always will be able to be virtualized. The ideal abstract computing units can be sent on little Noah's Arc journeys through space and time, working 1000 years from now as well as they were working today. It's not robots or high concept science fiction. It's just quality and craftsmanship, and power-source cleverness, and state-of-repair capable. Hmmm. Do we throw in reproduction, as in able to manufacture other copies of themselves? Should their seeds, or boot-kernel operating systems be tiny and mobile and perchance infectious? It's only the tiniest slip-ups that keeps hacking matter for their remarkable and useful properties, and accidentally dropping a pinpoint black hole into the earth's core. The scenarios are many by which we may wipe ourselves out sometime soon. Okay, the Internet's back. Prepare to merge. Save, but don't commit. Then, :sh. Then git pull. Instructions to stash or commit. git commit -am "Prepare to merge" git pull Warning of conflict and merge occurred. exit Warning by vim that file has changed L to load the changed file Glance at file, see that I should continue writing about this intention git conflict with self. Ponder the meta moment, and realize this may come off better portrayed as code, then as formatted html. Ahhh, html. How fondly I remember you. Markdown is really getting under my skin, not so differently than Python. I fell in love with a number of intertwined technologies that I deliberately sought out, in order to expand my capabilities intelligently. I think I remember one of those things being Python in ye ol' days, when my friend in Japan, Guillaume Proux, one of the officers of Scala Digital Signage, who still remembers the good ol' days when my primitive version of Ruby on Rails, but on VBScript on Active Server Pages on IIS on Microsoft Server, connecting with Microsoft SQL Server to do things in an amazingly cool and tightly coupled unit of awesomeness. Databases suck. Too much to setting them up and maintaining them and care and feeding and sometimes the painful archiving and pruning, and performance optimizing, index designing, tweaky execution plan optimization and retention... well, you get the idea. Some technologies have staying power, and others are Microsoft. They know it. That's why they're putting the Linux Bash Shell in Windows 10 by default. You slowly cook the frog. Here's the game-plan, and why everyone continues to exist in some for or another 10 or 20 years from now. It goes like this. The necessity for self-re-invention is obvious. It goes like this: If you only ever were exactly who you are today, your enemies will destroy you. And you will attract enemies like lint to a sweater on a sweltering Sunday morning when you're emptying the drier lint traps at a coin-op Laundromat you had to run in your fifties as a washed-up ex-textile Engineer who still needed to make income to put you through college... so you gave him Sunday mornings off. Wow, my dad worked hard for me. It's hard to imagine how hard he worked for me, and the family that only broke and fell apart around him... mostly, because of my mother and her issues. My dad would have been happy forever with my mom. It was pretty clear, just as clear as this would never happen. My mother despised my dad. She hated everything about him, because she knew him too well, and never helped make him a better man. Or he never took the risks she was willing to take, like moving to New York, or using the family savings to start a new business, instead of continuing to work for "the man" in New York in a grueling commute from Philly suburbs, which I imagine really must have taken a toll on him over those 13 or so years he did it. My dad talked a lot about the good old days with the skiing club and when he owned a boat. He was very proud of those. Because there was nothing like that in his life anymore. Ugh! Repeating patterns. Fell into the trap... perhaps a little bit less-so. I will align my interests to fields and careers on the rise. I may not be pursuing a degree in computer science or data science, but I'll be pursuing the expert use of the tools in both. I will be the can-do with-data guy, pretty darn soon. Now that the Internet's back, I feel the draw of delving into what I'm coming to think of as Pipulate 3, or possibly GoodSheet. One way or the other, there will be packages for Jupyter Notebook that will be distributed most likely as git repositories in Github that can be cloned. One will probably be goodsheet, which will provide all the Google Spreadsheet love-ability, which is critical to having a useful place to house and share small-scale data-collection runs from the net. I LOVE the web-based interface of Jupyter Notebook. Wow! It's like a dream design. I'm going over to it right now to carry on this project. Wed Jun 15 21:54:36 EDT 2016 Okay, back in business. Commit a working copy of latest Jupyter notebook work, and comment here on how keeping the notebooks in-sync between different repos will be an interesting issue. The imports are a snap now, with my new magic always-absolute, all-the-time. Just git-clone the repost that constitute the system in places next to each other, and they will be there, sitting as latent but known components I can draw upon, keep compatible with namespaces. Import at the same time and have happy, fun, strange interactions between, communicating at the in-same-task-and-memory level of python modules. I have a nifty plan. Now, go clone serpchiver. I could go all night if I let myself, and then crash hard, and clean and organize tomorrow, but now with long-running jobs running. Screw scheduling. I need the job that I JUST STARTED to reliably run to the end, and reliably update Google Sheets with that data. I need a cumulative... exploratory... persistent, capable of sustaining multiple transforms, able to be mapped back to the Google Spreadsheet from were the input data came from, to get inserted properly to the right rows, or to make new columns, or to make new worksheets. Whatever result (deliverable) is supposed to be derived from the job. Pandas. Definitely, Pandas. Shoot, I'm on the edge of the Rabbit Hole, looking down. But some rabbit holes you just have to take the plunge into. Deep breath in. You're about to make one of the most important career decisions of your life. I can feel it coursing through my body, the anticipation of data-power. I've dealt with SQL, fairly extensively. I'm pretty good at it. I've dealt with kooky limitations and awkward work-arounds. HAM mode for lots of colors on the Amiga is one of the ultimate heartbreaking compromises. Sure, you can have 4096 colors, and stunning photo-realism, and shocking "new" graphics modes discovered all the time, like half-bright HAM mode (ham, standing for hold-and-modify). You could get that photorealism some of the time, but the pixels had to look like bricks, and only algorithms that averaged and crunched numbers all the time could keep HAM mode looking half decent. To truly love the Amiga took a lot of overlooking and forgiving. Along came no-compromise high-res, with accelerated 3D-graphics, none-the-less. Heartbreak. Tools do die. Unix, and modern incarnations such as Linux didn't die. Speaking of which, the free and open source software movement (FOSS) isn't dying anytime soon. The vi text editor in its modern incarnations such as vim aren't going away anytime soon. 30, 40, 50-year track records say a lot. No matter how modern is modern, an old classic is still an old classic for the perfectly valid reasons that made them memorable in the first place. And now take an already modern classic, such as Python, and keep giving it power, perfecting, making variations, making completely alternative runtimes for special edge cases, like PyPy and Ufora... again, and again, and again. Base many of the most important business systems of the world on it, in cases both known and unknown of massive scale, complexity and performance. I have to believe Python won't die anytime soon, and still in fact has many of its best days ahead. Onto learning the Pandas API... Oh yeah, get serpchiver, and confirm that all that delicious work you did today in the IPython Notebook file all still pulls down from Github cleanly. Okay, had to update it a bit, but I think I'm in pretty good parity between my journal repo and my serpchiver repo. Good starting point. Can I keep going? Take a few minutes. Maybe even think about dinner. 10:30 PM... sheesh! The zone is upon me. Immersion into the flow is imminent. Don't disrupt or go heavy on food... oh yeah, Happy Hour from work. That's what's happened. Now, go transform yourself into a high-impact player... after some noodles. -------------------------------------------------------------------------------- ## Wed Jun 15 13:20:28 EDT 2016 ### Pipulate 2 (or 3?) Taking Shape Okay, I'm refining the absolute path import logic a bit: # Add parent dir to Python path and do OAuth2 setup work from sys import path from os.path import dirname, realpath adir = dirname(dirname(realpath('__file__'))) if adir not in path: path.append(adir) from goodsheet import * This is what's going to have to be at the top all the time in Jupyter Notebook if I want to be able to import goodsheet from lots of different notebooks. That's fine. Let's now break goodsheet out into its own Github repo. Ugh! I'm going to try to adhere to PEP 8 more closely. First, I'll change the 2's to 4's in my tab indents in my .vimrc... Done. Okay, now delete goodsheet.py from both Journal and serpchiver repos... Okay, done. Had to update the Python at the top that imports goodsheet: # Add parent dir to Python path and do OAuth2 setup work from sys import path from os.path import dirname, realpath adir = dirname(dirname(realpath('__file__'))) adir = '%s/goodsheet' % adir if adir not in path: path.append(adir) from goodsheet import * ...but now, this is feeling quite solid. Wed Jun 15 18:10:02 EDT 2016 Was at Happy Hour for a little bit. Hmmm, okay. Think. Get a tiny bit of pandas tutorial, then head home for your day off, and read the Pandas O'Reilly book on your way home, and contemplate the UI and UX you're designing under Jupyter Notebook. And go home and alternate between cleaning your place and giving birth to Pipulate 2 (or 3?). -------------------------------------------------------------------------------- ## Wed Jun 15 09:13:10 EDT 2016 ### Not Learning Pandas Now Would Be Ridiculous Well, as expected (and I should have know), Windows restarted my machine overnight, and kept the reports from running. It's becoming more urgent to get the system off my personal machine, and onto some dedicated hardware. Or maybe I script my virtual machine to restart every night. One thing's for sure, I'm not leaving my Kubuntu VM full-screen, dual-monitor anymore. Windows is Windows, Mac is Mac and... hmmm. Linux is Linux. I'll just finally put my CuBox to work. But I HAVE to get that batch job underway ASAP! Try to get things underway each day with a stupid journal trick. I have the basic Jupyter Notebook virtual environment trick working now for two of my Git repo projects: - miklevin.github.io - serpchiver Serpchiver holds a lot of promise as a potentially popular and infinitely useful project. It's the seed of the Pipulate 2 project. Maybe I should call it Pipulate 3 to stay in sync with Python 3, which is really the onus of this project. Hmmmm. Okay, I need to get that GSpread integration done. I've made this wonderful module called goodsheet.py, which lives now in my journal folder AND serpchiver. That's bad. Common module should be broken out and imported into each project. But that raises the specter of relative path module imports. Hmmm. Just programatically add the path of the parent directory of the current project to the Python path. BAM! Easy solution. Hope it's not a lot of code. Okay, also just added my vim spell-check file to BOTH Dropbox and Github, hahaha! I went from every vim installation being a spell-checking "island" -- very inefficient, in terms of always re-training vim to recognize my words. Now, Dropbox should propagate it around to all my systems without even thinking after I update the minimal ~/.vimrc that I keep on each machine AND it will "initialize" with all my learned words whenever I do a git pull in my ~/Dropbox/vim location, so long as I git commit it every once in awhile from my common work machine. Git merges will be interesting, in these cases when I get out of sync. Okay... next! The better-than-relative-path-import trick, and then breaking goodsheet out into its own project. Oh, this is great ecosystem stuff! Okay, get Pipulate classic churning away on this task... oops, can't. Burned that Rackspace IP. Hmmm. Hop into your next (and last) remaining deployed Pipulate server, and see if you can plow through another few hundred rows just to get it in the works. I have the job quitting on a Captcha, so I can just let the job run next to me and baby-sit it. Okay, it's underway. Next! Ah yeah. Externalize goodsheet.py into its own git (and Github) repo. The first part of this is to simply UNDERSTAND how to add the current directory's parent to to the Python path. # Add parent dir to Python path and do OAuth2 setup work from os.path import dirname, realpath if dirname(dirname(realpath('__file__'))) not in sys.path: sys.path.append(adir) from goodsheet import * Now, I'm going to try to get rid of the bad markdown highlighting in vim. runtimepath=~/Dropbox/vim,~/.vim,/usr/share/vim/vimfiles,/usr/share/vim/vim73,/usr/share/vim/vimfiles/after,~/.vim/after I downloaded this "official" markdown file for vim, but which happens to be ahead of the one in the official release and installers. Blech! Nope, going to wait this one out. I don't need my markdown syntax highlighted. I'm just edited my .vimrc to turn syntax highlighting off when editing this file. I guess that makes the spelling stuff all the more prevalent, and my spell-check dictionary becomes a much better asset for me than it has in the past. I just hope that both Github and Dropbox are not both taken away someday. I need a third place for all my data to reside that I own. That'll be a good project for my Python tutorials: Your Copy of Everything. Add that to the striking-distance projects! Done, along with a bit of update there. -------------------------------------------------------------------------------- ## Tue Jun 14 22:28:00 EDT 2016 ### Pretty Sure Pipulate 2 Is Being Born I just did this command: cat goodsheet.py | pbcopy ...which amazingly put the following code-block into my Mac OS X copy buffer (or paste bin, hence pbcopy and not cbcopy). import argparse, json, os, sys import gspread import httplib2 from oauth2client.client import OAuth2WebServerFlow from oauth2client import file, tools from clisec import * class Unbuffered(object): def __init__(self, stream): self.stream = stream def write(self, data): self.stream.write(data) self.stream.flush() def __getattr__(self, attr): return getattr(self.stream, attr) class MyCreds (object): def __init__ (self, access_token=None): self.access_token = access_token # Force IPython Notebook to not buffer output sys.stdout = Unbuffered(sys.stdout) scopes = ["https://www.googleapis.com/auth/webmasters.readonly", "https://spreadsheets.google.com/feeds/"] path = os.path.dirname(os.path.realpath('__file__')) filename = '%s/oauth.dat' % path flow = OAuth2WebServerFlow(client_id, client_secret, scopes, redirect_uri='urn:ietf:wg:oauth:2.0:oob', response_type='code', approval_prompt='force', access_type='offline') authorize_url = flow.step1_get_authorize_url() storage = file.Storage(filename) credentials = storage.get() argparser = argparse.ArgumentParser(add_help=False) parents = [argparser] parent_parsers = [tools.argparser] parent_parsers.extend(parents) parser = argparse.ArgumentParser( description="__doc__", formatter_class=argparse.RawDescriptionHelpFormatter, parents=parent_parsers) flags = parser.parse_args(['--noauth_local_webserver']) try: http = credentials.authorize(http = httplib2.Http()) except: pass if credentials is None or credentials.invalid: credentials = tools.run_flow(flow, storage, flags) else: credentials.refresh(http) with open(filename) as json_file: jdata = json.load(json_file) token = jdata['access_token'] creds = MyCreds(access_token=token) gsp = gspread.authorize(creds) And I just import that file into Jupyter Notebook, and I have magical Pipulate-like Google Sheets connectivity. I didn't finish my work tonight, as I had hoped. But instead, I'll get to bed early and wake up early. So close to getting Pipulate 2 up to this amazing level -- at least the kernel of its birth, given I have such fabulous conceptual blueprints, and a working existing product that does much of what I need it to do, but just lacks the reliability for industrial use. But I can get it to nearly industrial use. GSpread will be optional based on which Pipulate convention files you import. Currently, I'm working on the GSpread implementation, which is goodsheet.py. -------------------------------------------------------------------------------- ## Tue Jun 14 22:10:27 EDT 2016 ### Let Machines Optimize Python's Concurrency For Machines Wow, it's hard to believe what's happening to this repo directory. I'm quite into the concept of the self-modifying journal. You know, this is not quote LISP. Just imagine yourself working on a specialized language built up to just the right level of abstraction and compromises between performance and flexibility to solve a remarkably broad set of problems... say, everything but edge cases, and even those you can handle pretty well. Global interpreter lock? No problem, don't use compiled C-code modules and then use a different Python runtime environment, like PyPy for compiled-C speeds automatically (if you give up some mutability) or Ufora for parallelism... WITHOUT giving up your numpy (in fact, the code execution engine is optimized for numpy) but with giving up a bit of your list mutability. Imagine lists having to have defined data types, and unable to be changed on-the-fly and resized and deletable from the middle, and all those other nifty tricks that make Python so darn flexible. For the price of less flexibility, you can just pick another flavor of Python built for parallelism... completely transparently. It will optimize your code for taking advantage of multiple CPU cores better than you ever could manually in your code. It's just like C and Assembly Language. Sure, you CAN hand-optimize your Assembly Language code still, but why would you when the gcc compiler does it so much better than any human could, given the sheer quantity and complexity of the abstract instructions it's compiling. So, if its obvious machines can optimize code better than humans, then why not how it breaks down a task to parallelize it as well? Seems like a computer sweet spot, to me. So, just go ahead using that Python you love. When you need parallelism for your app's performance to increase 100x with 100 cores, no problem. Just control your use of lists, and throw your code onto a Ufora computing grid, and BAM! Python is a parallelizable, concurrent, non-blocking, whateveryoucallit modern programming language -- so sophisticated, that concurrency optimization is done at the correct level -- runtime optimization, rather than explicitly all over the language, incurring such debilitating overhead as the curly braces that also litter many of those languages. Python is written for the human brain, and machines can adapt it to machines. -------------------------------------------------------------------------------- ## Tue Jun 14 12:38:30 EDT 2016 ### Google OAuth2 Login and Token Refresh Under IPython Notebook Pshwew! Okay, I feel like that was the end of a tremendous journey, just to get Jupyter Notebook working well with OAuth2 login and GSpread. But it's totally worth it. What future projects could this NOT be the methodology for? Tue Jun 14 14:35:11 EDT 2016 Okay, I have a response to get out regarding some questions on properties. THAT'S what THIS ALL is about -- answering questions, getting to the heart of issues. The self-modifying journal stuff is as far along as it needs to be to be a springing-off point for the saveserp project. It's time to change it's name to serpchiver. Yep, that's good stuff. coda create -n serpchiver python=3.5 jupyter Tue Jun 14 15:49:48 EDT 2016 Okay, I have to move faster. I REALLY need to have this batch job running overnight now, and WITH either a list of anonymous web proxies or HMA proxy mixing up the IPs of the host machine. -------------------------------------------------------------------------------- ## Tue Jun 14 09:29:31 EDT 2016 ### Forcing Myself Into Jupyter Notebook For Daily Workflow Interesting. This will be a pretty regular part of getting started editing a journal now, if I'm not recycling a left-over terminal session: source activate journal I'm having a tough time remembering this language, both under conda and virtualenv. Conda is easier, because there are no paths. But to help remember it, we can parse it, with some intellectual reasoning about which words are being used and why they're in the order they're in. Namely: source ...always comes first, because what we're doing is telling it that we're about to define the virtual environment source that we're talking about. The next word: activate ...tells WHAT we're going to be doing with the virtual environment THAT WE STILL HAVE NOT NAMED. And so far, the concept goes: "source, activate..." No wonder I dislike this. I want source = something... but this follows a different model. It's like "source" is the program name for virtual environment software, and what we're using the name/value pair as an argument. In that light, it makes more sense. It's like compiling from source. Remember the notion of source code. It's source files. Maybe this will help, but wrap it up there. Name of virtual environment goes last. Concept of source "code" goes first. And activate goes in the middle. If I can't remember it now, I'm just stupid (which is always a possibility). Practice will make perfect. Okay, it's almost magical that after a git pull, all the Jupyter Notebook stuff is here as well. Shit, this has MAJOR ramifications on my striking-distance project. Development actually becomes significantly easier. Work-session states are memorized, beyond simply git repo stuff (which is nice, too), but this is a whole other dimension of statefulness and continuous thought and work. Wow. -------------------------------------------------------------------------------- ## Tue Jun 14 06:58:23 EDT 2016 ### Force My Way Through OAUTH2 Stuff... Again This is the process of bootstrapping myself personally, in order to be able to move mountains on a daily basis. This ties right into those superpower secret weapon constantly recurring themes. Tue Jun 14 07:21:48 EDT 2016 Ahhh, took a nice hot bath. Okay, not to shabby. Finish out whatever you want to get done at home, then head into the office. Ugh, I'm not going to be able to get it done without making myself run late. But just remember that you STILL have a token timeout problem, and that you have the key to the answer to the freshentoken function in the pipulate project. You have, I believe, everything you need to always check for an expired token and always be refreshing it. BUT there should be a simple way with the Google ready-made code. Look for it before re-implementing anything. Tue Jun 14 08:15:34 EDT 2016 I take that back. It is working as expected. -------------------------------------------------------------------------------- ## Tue Jun 14 05:24:13 EDT 2016 ### Modifying Google Spreadsheets Through Jupyter Notebook Okay... pip install gspread! Done. Now, think through the next steps. It SHOULD be easy, given... well, everything. It's going to all be about importing that authentication object. Ugh! Had to switch back to the oauthme.py program. At least my work yesterday wasn't wasted. I need to find the best combination of the old method (working for me) and the new. In either case, just hit that wonderful success assured moment. This is going to be pure friggin' magic! -------------------------------------------------------------------------------- ## Tue Jun 14 04:21:36 EDT 2016 ### My Self-Modifying Journal Is Born I think maybe the objective tonight is to get to the point where you can run batch files at home, and then turn around and start just intermittently come back to check-in while to babysit the tasks, while you do things around the apartment. Ugh! You're going to be miserable tomorrow if you have to split your attention every which way. The best thing you can do is to finish your office-work, and do a kick-ass job tomorrow throwing your attention around every-which-way as batch files run in the background. You have to get Jupyter Notebook right at that point where you can trust it with long-running jobs as you would try to do with Pipulate, and a better (and single) OAuth2 login for the task interacting with the service is a VERY solid start. Okay, so I added this o2.py file that does the trick and uses all of Google's latest advice. Now, I've got to do something in Jupyter Notebook that actually USES this authentication. Get the with open program back in here for my journal, and commit THAT to the repo. Okay, done. I have some more familiarity with the Google Developer Console to achieve. For now, I've got the credentials on the Google-side set up correctly for the work I need to do. Okay, I just allowed the IPython Notebook files into the repo for this journal. I put one of those open with commands in there. Good start. But I have to push this friggin' thing forward HARD. Starting with the self-modifying journal I think is a brilliant start. Centralize resources and thinking RIGHT HERE. So, what's next? Now that I'm stepping through the journal a line at a time here, I should just print something when I encounter 80 hyphens on a line. Easy enough. Just count them. count = 0 entry = (80*'-')+'\n' with open("index.html") as journal: for aline in journal: if aline == entry: count +=1 print(count) Okay, it's time to finally install and use dateutil: pip install python-dateutil Okay, I've got 326 journal entries. This is going to be as cool as I had hoped: from dateutil.parser import * count = 0 entry = (80*'-')+'\n' with open("index.html") as journal: for aline in journal: if aline == entry: adate =journal.readline()[3:] try: parse(adate) except: print(adate) print('%s: %s' % (count, adate)) count +=1 print(count) And now, I should look at entering a line into a Google Spreadsheet per... or maybe something less so that I don't have to wait. But this should be a very simple matter with GSpread. And overcoming THIS hurdle, catapults you forward. I feel myself really now taking control of the journal. This dove-tails with keeping the journal in git and Github just a little too perfectly. -------------------------------------------------------------------------------- ## Tue Jun 14 03:28:12 EDT 2016 ### https://developers.google.com/sheets/quickstart/python Okay, I managed to get myself awake early enough to do some good. And now, I have to choose between on the home-front, and the professional work that I need to do. Organized environment, organized mind. Okay, but I have done an interesting experiment. Anywhere I pull down a latest of this journal, right while I'm editing this journal, I should be able to: :sh (drop out to shell) python oauthme.py Confirmed! Any machine that I've installed the basics on, including the google api client libraries, all I need to do is copy over my client_secrets.json file. This is a HUGE step forward. So tiny-seeming, but large in actuality. After I have that authentication, I should be able to do all sorts of other things out of that directory, with IPython Jupyter Notebook as my experimental exploratory interface. Try that. Virtual environment? Conda on every one of my Macs? Hmmm, probably. Shoot. Okay, download Conda on this machine. This is a HUGE step forward. So tiny-seeming, but large in actuality. After I have that authentication, I should be able to do all sorts of other things out of that directory, with IPython Jupyter Notebook as my experimental exploratory interface. Try that. Virtual environment? Conda on every one of my Macs? Hmmm, probably. Shoot. Okay, download Conda on this machine. Already downloaded! Install it, and get MiniConda. While the Anaconda install is going on (~1.5 GB, ~20 Min), take a shower and get yourself REALLY up and about for today. Alternate between your real-life environment (which you REALLY have to get under control), and your professional stuff. I have a lot of Macs in play. But I do things a little differently than other people, and I do my where wherever, whenever I'm inspired, and I need lots of machines set up to just sit down and work on. And when I have a new preferred development environment, I need that a lot of places. Oh, it doesn't look like MiniConda really has to be installed if you use the full version of Anaconda. Okay, made that mistake on one machine. I thought it was necessary to get the conda repository system, but apparently not. If you quit a shell and reload it after the Anaconda install, conda is there and in the path ready to use. Okay, so make the virtual environment for your journal. conda create -n journal python=3.5 jupyter This appears to be developing as a central tenant of the self-modifying code imperative that I'm starting out with on my journal. If it works out, I'll migrate the technique to other projects and code bases, most specifically Pipulate 2. I would like to get a certain amount done "tonight" so I'll be arriving in the office with all the stuff I wanted to do ready. Tomorrow evening I'm having dinner with Adi near Union Square, and that blows a weekend of my recovering energy after the weekend. THIS is my Monday night, and my next opportunity will be Wednesday night. This just may be a situation of driving yourself into the fucking ground, and sacrificing your health to get ahead. Except that the professional work you're doing will make you gobs more effective WHILE AT THE OFFICE, and that is really the key here. source activate journal jupyter notebook Okay, take advantage of this opportunity to switch to the Google v4 API. https://developers.google.com/sheets/reference/rest/ Ugh! Much better sample-code for the OAuth stuff: https://developers.google.com/sheets/quickstart/python#step_3_set_up_the_sample Interesting! Okay, the credentials file gets stored in: /Users/[username]/.credentials That's better than what I was doing, and the code sample from Google is: - version 4 of the API, which they recommend using - made specifically for the Google Spreadsheets service This is a much better starting point than what I was about to do. So, remove oauthme.py from this repo, and add o2. -------------------------------------------------------------------------------- ## Mon Jun 13 11:31:50 EDT 2016 ### Working on Generic Long-lasting OAuth Login Okay, my last video about getting reading my journal with a Python "with open" under IPython Notebook under an Anaconda conda virtual environment is exporting. Usually it takes about as long to export as the video is minutes-long (on this 2011 Macbook Air), so it's good time to plan out the next chisel-strike. Make it smart! It's probably going to be prolonged OAuth login. It gives me just enough time to think it through. Okay... think! I want to be able to trigger off line at the beginning of my iPython Notebook session that ensures I'm logged in, with the type of login that's going to last for a very long time, indeed. I'll clearly need the google python client API libraries installed... ah ha! This will actually be under the virtual environment. Cool. Finish up this project operating on your journal, knowing that you will be switching to another file. pip install google-api-python-client pip install requests pip install gspread Finally listening to the Kennith Reitz Talk Python to Me podcast #6. How'd I skip that one?!?! Okay, listen to it to filter out the office scuttlebutt as you drive onto this next step. What will the next video look like aside from just installing this client library? Oh yeah, gspread. Now specifically, I want to get through a VERY long-duration OAuth2 login. And so... your best examples? The best Google examples are: - https://developers.google.com/api-client-library/python/ - https://developers.google.com/identity/protocols.OAuth2 - https://developers.google.com/identity/protocols/OAuth2WebServer#offline Okay... this is a good start. My last video is exported. I will upload it to YouTube now, and prepare myself for the next video that conquers offline OAuth2 under Jupyter Notebook. Oops, I have to re-export it! I forgot to show my keystrokes -- an important feature of my videos, and really getting back my investment on these personal Macbook Pro's. Take this time to get your next step clearer. After I do my 3 pip installs, what then? - Go to Google Developer Console: https://console.developers.google.com/iam-admin/projects?authuser=1 -------------------------------------------------------------------------------- ## Mon Jun 13 09:34:08 EDT 2016 ### Going to Try To Dive Deep Coding Today Okay, gotta get my Monday morning report together. Get your second coffee of the morning first. Check over the SEO Pulse reports first. I've got a lot of work ahead of me, and I have more rein-grabbing that I need to do, and my tools to do this rein-grabbing have to go up a notch or two. Perhaps I am living with different constraints under Levinux than what I'm trying to do here, so maybe a little bit of hubris and willingness to embrace IPython, numpy and pandas in all this. Good Processes: - Light weight - Fewest number of steps to complete the process - Really easy to access and understand I'm getting really pumped my putting my whole online shtick and how I produce my content. - It can be hard to know what to automate. - It takes many iterations to know HOW to get to the process. - Increase the joy by automating away tedious steps. - Expose the best data for pattern recognition and subjective meaning to humans. Start with something that's "good enough" and comfortable and clear, and then gradually build it bigger. Document the WHOLE THING from scratch as best you reasonably can in your work-day. Who are the women in Computer Science who are really happy in the field? Make sure you set up the right role-models and examples. Don't cut any parts of the industry off from Adi as accessible and approachable. It should be everything from pure scientific research to invention to coding and such. Start here: - https://kateheddleston.com/blog (thanks again, Michael Kennedy) Okay... hmmm. Next step? I got my weekly report to my boss, and I have a tiny window to jump head-first into some awesome stuff and get some momentum going. I really love the video stuff that I've done lately, and I'm glad to be getting back to the talking-head videos. I tried setting up another machine yesterday that I did from the Catskills. I already got a crappy comment on the video (Google Plus), even though I got 3 thumbs-up on the video... hahaha! It got a "Demo Version" water stamp across the entire video, but I pushed it up in a weekend hurry. I think... I think I need to push forward on this front with enormous determination and focus today. I NEED: - Prolonged OAuth2 Login - Use of anonymous Web proxy lists When major scraping tasks come up, keep scrapinghub.com in mind, because I want to just generate the data I need for gephi visualizations, and I need to get to that sooner than later. It's VERY tied into the Navigation Menu projects. Step 1... the Meta Journal! -------------------------------------------------------------------------------- ## Sun Jun 12 14:01:53 EDT 2016 ### Turning Home Mac into Jupyter Notebook Dev Machine This will be an interesting step. I basically blend Jupyter Notebook right into this same location where I keep the journal. After a :w save, I should be able to :sh out to a shell, and confirm my current directory with a pwd, then execute the command to make this a conda-style virtualenv... pwd conda create -n venv python=3.5 jupyter Okay, now that I've done that, I should activate this environment BEFORE running the journal. Sun Jun 12 15:35:05 EDT 2016 Actually spending LOTS of time with Adi. Typing here on the occasional everybody doing their thing time. Basically, I want to quit this journal now, activate the environment, and go back into the journal, so that whenever I drop out into the shell with :sh, I'm still in the conda virtualenv with: source activate venv -------------------------------------------------------------------------------- ## Sun Jun 12 09:14:41 EDT 2016 ### Sunday Morning Planning Meta Journal Programming Another fine morning. I got an interesting YouTube video uploaded last night in which I got this PC in functional operation insofar as having my whole set of things I want to be true for one of my serious in-the-zone work machines. I can see now that I'm setting up machines and environments specifically to be able to get myself *in the zone* or *in the flow* as the case may be. How can markdown deal with words like nix*, but the vim default markdown syntax higlighting not? :set syntax=off... ugh. Some tweaking to do. I am currently editing a video on my phone about visual programming, and my decisions regarding Adi's programming. Editing out a little tantrum she had, and breathing exercises I taught her, because its nobody's business, but I am most certainly keeping the original video for memories. It puts us both in the best light, and takes a little journey through rudeness, then a cut to much more somber energy, which livens up as I explain what happens between two people during a fight, and how we can rise above that. That reminds me. Jupyter Notebook should now act as the bridge between this journal and the actual executing Python code world. Hmmmm. Very nice. I can make a module of stuff that parses and does stupid tricks with this text file as part of this git and Github repository. YES! Treat this file like a database, in a limited sense, by a file that can open, manipulate and save out a new version. It will be interesting. I will have to think about how the edits should be represented in the git history. In-location same-file editing, as far as git is concerned, surely. So no automated git copy and remove commands, least we erase the easy ability to track a single file's evolution over the years. Okay, yeah... so I'm getting the vision. Get this file a bit over to the self-modifying side, using ideas that themselves originated here... a lot like consciousness. Let this journal be a microcosm of consciousness. Project one: Get Jupyter Notebook executing code out of the miklevin.github.io directory. Okay, the TimeZipper and SelfModifer projects are almost one and the same. And so, I put them close together, and will start to take the first steps towards it right now. It should be simply running IPython from THIS very directory. Is this where I introduce tmux into my daily work flow now? That would involve the use of this newly acquired knowledge: conda create -n venv python=3.5 jupyter source activate venv pip install gspread [source deactivate] conda env remove -n venv Seems a reasonable starting point. Get that meta feeling before Adi wakes up, and be at a good place in your mind. Think about a quickly cobbled together Pipulate bootstrapping environment under Jupyter Notebook. This could be very nice, indeed. Just throw a day or so at it. Okay... 1, 2, 3... 1? -------------------------------------------------------------------------------- ## Sat Jun 11 22:19:43 EDT 2016 ### Beginning To Design New Pipulate API I think I'm falling totally in love with Jupyter Notebook, which is a problem, because I don't want heavyweight tools. But it runs on laptops just fine. And it's part of a package that an organization is going very much out of its way to make sure runs pretty reliably on desktops -- Anaconda. And so... and so... my next steps with Pipulate are very much evolving. I love the web interface to true, useful, easy-to-debug Python. You don't feel separated from the Python execution, as you do with Pipulate. And so... and so... okay, think this through. How to do this next step artfully, and additively useful? Well, one of the first steps is to use my various laptops and tablets and phones to great effect. Turn just a generic and awesome command of Python into a huge advantage, in having the ACTUAL ability to carry out ad hoc investigations fast, and indeed in a scalable way. I want full, programmable Python AND the coolness I was getting from Pipulate. I want to be able to produce incredibly valuable videos of obvious and noteworthy usefulness and utility -- the sort of stuff that really does motivate people to copy-and-paste a URL into email in that mythical way, or perchance to use those social media share buttons. I am rapidly on the tail of something special... again... and there's nothing wrong with pivoting, when you see something worth pivoting for, and it's been a long time since I've found pieces so truly still generic and free and open source, yet working together so well as Jupyter Notebook, vim, Full-Screen Mac OS X, Github, and a few other interesting pieces. Now, I'm wondering how well Python will continue to fit in. Shall I be the one to compile NumPy for the Tiny Core Linux 7 software repo? And Pandas and all the other chain of dependencies? Hmmmm. I think I'm in the bleeding edge of a sweet spot again. Here's my thing... I like Tiny Core Linux, because it makes your instance of Linux generic and virtually invulnerable to corruption and cruft. It will be a true perfect efficient image of a Just Enough OS to get your app running as an Internet appliance server. That server's whole existence is to support that one app, and so it doesn't need to be a beefy, all-purpose thing, and can therefore be optimized to see if it needs all Python's dynamicism. And if it doesn't, then execution can be delegated to PyPy or one of the other optimized for static conditions Python runtime. And that sort of kernel delegation is actually available in Jupyter Notebook. You could plug in PyPy or Ufora. It's amazing to contemplate, and brains much greater than mine are shaping a wonderful web-based Python code execution environment, that surpasses your wildest dreams in terms of encouraging experimentation. I would be doing everybody a favor getting more people trying to do Python coding in such an environment as IPython. I wish I had been using it sooner. None-the-less, that doesn't invalidate the discoveries I had made in writing the original Pipulate. I'm glad I went that route, after the earlier Tiger project that I had done at 360i. And this next iteration will still be different again, indeed. It's going to sit lightly on top of IPython, dropping in a package her and there, to expose objects with these wonderful self-describing APIs asking for CSV files in some form or another, usually files in a folder of a conventional name, such as csvin and csvout. You drop anything you want pipulated into the path pipulate/in and pipulate/out. It just feels right again, doesn't it. You're pipulatin' and the output is pipulateout. All very elegant, and rolling off the tongue, making intuitive sense, as a system bound for everyday use should be. Here's how we do it. Job requests are submitted as CSV files, of a conventional arrangement. These requests can be simply deposited as csv files into the relative location of the launched app, say "pipulate" to directories located within, by convention. So, you have a pipulate/in directory and a pipulate/out directory. You submit the job requests as files copied into pipulate/in. The most appropriate mechanism for that platform, such as OS event detection will recognize this file, and trigger pipulating the file, which is to say carrying out the embedded request, and putting the fully pipulated job into pipulate/out, as a file of the identical name. In this way, files appear to simply "move" or maybe get copied from location A to location B, but it's now filled-in with all this juicy data. This is one method of interacting with Pipulate. Another is as http requests sent to a web service running for the sole purpose of receiving just such a request. These requests are mime encoded CSV files. So, they probably originate as a POST request, meaning its carrying some meta data soon-to-follow the initial request, carrying the data the request actually needs available in order to succeed. Commonly, this would be used for a file-upload feature. But the uploaded files have to be only CSV files, or else they're ignored. And the CSV file must be of a certain format, with the data arrangement following a certain convention, or else the request cannot be inferred, and nothing will be carried out. However, if row 1 is found to have specific keywords in them that maps to certain globally available functions, then we likely have a job-request. Okay, time to get back to Adi. More on this soon. I like where this is going. -------------------------------------------------------------------------------- ## Sat Jun 11 19:44:29 EDT 2016 ### Getting Another Old Mac in Awesome Order I'm in front of the fire here in Spring Glen Woods, on the first day of a 2-day open house. I'm pretty sure the weather kept them home. But tomorrow is another day. I like having a journal. And I like being able to talk to it. I believe I am satisfying one of the basic human needs to communicate, but I'm doing it with a bunch of strangers out there, who kinda sorta tune-in now and again, who I mess with, sometimes being deliberately dense and arcane, and forcing the reader to parse through thick, dense chunks of the stuff, just so that I can have a laugh. I'm kinda mean that way, and I admit it. I do have that mean streak, and that's how I'm expressing it. I pity those of you who got stuck in the lines I'm trolling out there, with this journal and with my youtube videos, which can be found at https://www.youtube.com/miklevin which will do a forward to whatever the heck URL-system they've decided upon most lately, to ruin the root path awesome ones. So... here, I am getting another of the laptops in-play, up to the latest post-Mavericks, which is El Caption, I believe. I still miss those cat names, but it would be getting ridiculous by now. I mean, where do you go after Snow Leopard? You kinda are at the apex of cat awesomeness, and anything else will simply sound silly or forced and will be taunted by the media, regardless. So why not just switch, and keep it classy? Okay, let's use some darn obscure, but uniquely American references, which also happen to have extreme physical beauty associated with it, and provided to the users in an abundance of screen flavors. I will shoot videos. I will not edit much. My audience will appreciate that. I will navigate my way through treacherous waters, and come out the other side thriving. I will get this accomplished in time to enjoy total financial freedom a bit. Up to that time, I will struggle. But I will do it Internet personality-style. I already earn myself nemesis'. What's the pleural of nemesis? I've had a few. One was legitimately a class ahead of me, tech-wise at first, but by the time we were done, I realized why people like him were relegated to never being much more than quality assurance engineers. While they appreciate a good hack, they lack the imagination to legitimately pull an original one off for themselves, no matter how much bravado they boast. I know techs. Techs lie. They figured they've learned a few tricks about how to control the keys to the kingdom, just so that everything keeps working. And everything keeps working best if nobody can do anything. That is, if your tasks are running in such restricted limited-by-the-security-concious operating system to do almost nothing, then nothing is happening, and no panicky calls will come in at midnight about someones print queue constantly being killed as fast as you can send a job to the printer. Networks are little mischief treasure hunter's dream. So, lock-down, least another technically-inclined pup may grow into a big dog. Neuter pups fast. That's the rule. Be the big dog in the junkyard, and keep a firm grip on the reins of command and rule, and do this by putting yourself at risk and under a great deal of stress, and try to neuter every new young whelp who may look like a threat in a few years. Win battles now while they're easy to win. Play an effective game of didn't-know-there-was-even-a-game-going-on, Make young whelps run away yelping, too sore and lesson too well learned to ever go sniffing around the old junkyard dog's turf again. Yeah, I know those types. They existed. They still do exist, though ever increasingly in hiding. There are too many ways now for intelligent young whelps to effectively handle jealous old-timers, if those old-timers don't have the actual skills of experience and still nurture a youth-like adaptability. My book will mostly be to those information-starved pups who don't know how to work the whole system effectively enough on their own yet to master a sort of baseline awesome in tech, that underlies almost everything, and will only do so more every year. -------------------------------------------------------------------------------- ## Sat Jun 11 17:37:18 EDT 2016 ### Don't know a headline for this yet Hello World, from the Catskills, perhaps for one of the few last times before I sell it as part of getting my finances under control. Stop! No more needless spending. I can live for much more cheaply, and think I should, so that I have some expendable income again. So, goodbye a lot of cornerstone things from the last ten years. Life changes. We move on, and sometimes that means, we move. Friends need not be lost. There's always a way when you're connected. We are connected to people in many more ways than where you both happen to live in the same vicinity. Things start by coincidence, but they continue by intent. We are humans, and we can think, and we can control our own destinies, like few other classes of lumps of matter. Yes, we exist just for that very reason we should exert our will, when we believe in something. I believe, I have something to teach you, and perchance, and perhaps, and maybe through some cosmic coincidence, teach you too. You, there. Yes, I'm talking to you... amazing, isn't? Better than watching someone play Yo Kai Watch on a Nintendo DS, for sure, but whoever that guy is, he's got my daughter into wanting a Nintendo DS... way to go. I asked for it. It will be interesting warning her against the bottomless-pit of energy-sucking the world has ever known that is immersion into a virtual world that beats out reality in so many ways that you lose touch with what it REALLY is to be a human, without unlimited god-like abilities in your own imagined, but somehow still very real virtual world. Again. Nested. Avoid. Package everything that you consider important about you to-go. You will be happy some day when you find that you need that sort of freedom from place about you so that you can let go of your roots to follow your own dream. You are not a seed in the shadow of a tree. You are an independently moving inverted donut offspring of two others sorta like you, called your mom and your dad. Congratulations. You have survived the first round in a game in which you may never be fortunate enough to have a chance to hear from this from me. Things are amazing. I love you. And maybe someday, you'll read this, but maybe not. Who knows. I may blank it all again. Peace, out. Checking on the fire. -------------------------------------------------------------------------------- ## Fri Jun 10 23:21:04 EDT 2016 ### Catskills Plans with the Grands There's no way around it, I'm falling in love with Jupyter Notebook. Jupyter should have renamed themselves to IPython -- not the other way around. This is as awkward as the Google re-branding of Webmaster Tools as Search Console. What next is the question now. This is a small window until I'm in the Catskills for 2 solid days, until probably late Sunday night. I am lazy at heart. I look forward to not having the pressure on me to run off to the Catskills EVERY blasted summer weekend, when I could really be getting some things done around the house (with Adi) every few weekends. Not everything needs to be a whirlwind tour of wonderful. Sometimes, you just spend quality time together, getting stuff done that needs to get done, because my only time with her is when it is also the only personal days where work doesn't suck up all my energy... i.e. time I can actually get a thing or two done. My success as a separate father depends on my being able to mitigate that particular issue brilliantly well. I am not 100% sure of my approach, but not having the Catskills bungalow anymore, yet the grandparents having theirs, certainly works to my advantage in this regard. And so, I have some decent ability to occasionally think deeply through things. I have to decide about the last week of June, when Adi's spending a week with them. -------------------------------------------------------------------------------- ## Fri Jun 10 23:08:23 EDT 2016 ### What Should I Teach My Daughter? Your primary goal and purpose now is to keep all disasters at bay while Adi grows up and goes through these critical formative, shape-who-she-is years. I like the person she's becoming so far. I have to become That Coding Guy for her. She really struck a chord with me, when I asked who's that awesome kid, and she answered me? And then right into saying who's that coding guy? to me.... wow. It struck me. And I have a duty to help her become the fully realized potential awesomeness level of being that guy's daughter. I'm going to do better for her than my dad did for me, by being tuned more into the... what? The current events from a technology point of view. We are living some of the greatest human history ever lived, right now. The Free and Open Source Software movement, and to a lesser degree, the just pure Free Software movement. Coupled with open source hardware, already here I think in some cases, such as certain early MIPS processors, this will result in some core digital DNA material with sufficient instructions to be self-booting, given a few robots with power and simple fab plants. We're on the edge of robot colonies in the asteroid belts and such. Think about those things, and prepare your daughter to think about those things. -------------------------------------------------------------------------------- ## Fri Jun 10 11:16:22 EDT 2016 ### Jupyter Notebook IPython Success Assured (For My Next Project) Back to the issues of the day. Let's get that SERP archiver done. And I don't want to do it through Pipulate, but I DO want it to work with Google Spreadsheets. Very interesting! I have a lot to learn now with Anaconda. After I did my first 2 successful attempts: 1. Load a module I created in the same IPython directory accessible to IPython 2. Put that module into git, so I can also use vim and revision control Then, I tried to get virtualenv working with no success, realized that's built into Anaconda's conda system, and I installed MiniConda using the script they have you download and execute it in the BASH shell from OS X terminal. And now... and now? I'm very motivated to jump on this IPython bandwagon, so I'm going to work out these last few annoyances. Figuring out this virtualenv built into conda is one of them. First, you have to choose a package to create the virtualenv with. It can't just be an empty one. And so, I'll create it with requests. conda create -n venv python=3.5 jupyter source activate venv pip install gspread [source deactivate] conda env remove -n venv Fri Jun 10 17:35:08 EDT 2016 I'm exporting my last video of the day. Wow, that was 2 from my morning commute walk, and 3 from talking-head coding, which I'm FINALLY re-starting. It's already 5:36, and I'm hoping to get the file-export and YouTube publish done, before I "have to" walk out at 6, because I'm meeting Rachel and Adi down in St. Marks. But this sets me up SO WELL for my new work. I would rather IPython as the "wrapping" framework of this work than purely GSheets. I can always still use Google Spreadsheets, but what I'm potentially getting out of the picture is the feeling-of-control disconnect that occurs between coding Python and what happens when you hit the Pipulate button in the Pipulate U.I. I feel like I want to jump on the IPython notebook bandwagon. It's a bit heavyweight for the Levinux projects, but it's exactly what I need at work... hmmm. I don't really need to make everything the same thing. Revel in the nuanced differences. Use the right tool for the job. Don't let "not invented here" or any other syndrome keep you from using the very best tool for the job. Okay, 8 minutes remaining on the export. Get yourself ready to leave. Push out this last entry for the day. Then, publish to YouTube as the last thing, and don't wait for the processing, because that finishes server-side, even once you disconnect. -------------------------------------------------------------------------------- ## Fri Jun 10 11:15:31 EDT 2016 ## Okay, Unix Won. But Why Python Too? It is clear that Unix has won. But it is not entirely clear -- fogged by the atmosphere of JavaScript -- that Python has won too. Yes, while its all about choice and appropriate tools, Python is just being chosen first for the world's automation tasks at a surprisingly increasing rate. And it's already a 25 year old language, and you'd have thought all a language's glory would have played out by now. Not so... with Python, it's a steady constant build. But why? How could this little-known, strongly opinionated, and often counter-intuitive "scripting" language STILL be taking the world by storm, in the face of JavaScript, that literally runs on billions of browsers, with a nifty little Web browser system for retrieving the code to run? And if it's not JavaScript eclipsing Python, shouldn't it be Java or C# or Swift or some other heavily vendor-backed "Enterprise" caliber languages? Or newer languages like Ruby or RUST? Nope. There is a reason that an increasing number of the top computer science schools in the world introduce students to programming with Python. for this. You fall in love with it faster and it will inspire you more to carry on with your pursuits, be they personal or profesional. Python services many needs, and can be molded to your needs. And if the details of the language's official specification runtime, CPython (the python.exe from python.org), just swap out the runtime with PyPy, pigeon, piston, Cython, IronPython, Ufora or whatever suits your situation best. So, what next? What goes on top of *nix? How do we sit down and just start doing things? How fast? On what variety of hardware? Using what supporting software tools, like a text editor or integrated development environment (IDE)? At what cost and with whose support? Against how broad a set of problems that you're likely to need to use programming against? But when it comes to building the next level up, there's are countless possibilities and approaches. Every language is opinionated, so the question quickly becomes whose opinion do you want to abide by most? And there are some very big opinion splits here, not the least of which is whether you go for blazing speed or fabulous) flexibility (static vs. dynamically typed Another is, how well the language fits the sort of problems you'll be trying to use it for. How do you like the opinions are expressed in the language design? And Python is opinionated, indeed -------------------------------------------------------------------------------- ## Fri Jun 10 11:15:04 EDT 2016 ### The Case for Unix This book opens with accepting a reality, which for better or for worse, is a reality. Unix won, and it's best to focus on the parts that have made it so popular, and indeed love-worthy. Short, powerful commands with an endearing inconsistency of command parameters in which find and grep, which could gave shared an API, rather are mirror images. find . -name "filename*" grep -r "stringinfile" . Nonetheless, you get those classic, universal commands, such as cd, ls, cat, sed and command-piping, and every communication operation adhering to the filesystem read/write interface. All this combines so that you can design some pretty cool systems to live and run on top of it, with lots of individual components that talk to each other the same way. By Unix being designed to be portable for different hardware, it is also designed to be potentially viable with all FUTURE hardware. And GNU/Linux is so much like Unix , we can refer to them together as *nix. But then, sometimes I just credit Unix as the Innvation, and Linux as the liberator. No matter what you call it, it is a solid, standardized technological underpinning that you can build on top of it with the confidence that your apps will port to any *nix platform, as well. -------------------------------------------------------------------------------- ## Fri Jun 10 09:19:31 EDT 2016 ### Planning On Emulating Pipualte with IPython Okay... this will be an interesting day. I'm meeting Adi in the city after work, so there will be no working late. I have an ambitious project I want to wrap up today, so that I can run long batch jobs over the weekend. It also sets the stage for beginning the Pipulate 2 port... to Python 3. Maybe I should call it Pipulate 3? Either way, I have to hit the ground running hard, having done my research into using Proxies with Pipulate. I also am going to have to, in all likelihood, need to do OAuth2 authentication for using the GMail API through the standard Google API client libraries. Hmmmm. Alice & Bob log in. While the nature of my greater work is clarifying, the complexity of today's work -- the ambition-level -- has to be controlled, so that I can have a smooth-running, weekend-durable script running... that can scrape and store Google search results in a Google Spreadsheet... hahaha! Okay, violation of terms of service, I'm sure. But my investigations are rather small-scale, as these things go, and I'm trying to remain as efficient as possible. I think I may actually like to do an implementation of Pipulate using IPython (Jupyter Notebook) as the new outer shim. That could be really interesting. It would give me a much more powerful, and potentially responsive, environment than Pipulate through GSheets. There's a reason IPython was created, and they were solving that where-to-execute-code easily and make it humanly accessible as me. And it's a MUCH better environment for people to develop their own functions in... hmmmm, a plan is hatching, which I could even conceivably implement today. This could be a practice-run for Pipulate 2 (3?). I could probably cobble enough together to do everything necessary to fill-in the "google" cells, which are compressed and bin64'd Google search results (page 1). That is to say, it's a fairly default typical search result generated by typing a few keywords into Google, and hitting Search. It's what you would see in the browser as page 1 of results -- or more specifically, what you would see if you selected the browser's "view-source" option for that page. That mostly HTML-and-JavaScript gobbledygook that you see is the code that makes the browser show that search result page, and it does indeed contain the data behind the first page of search results. So this project is capturing that and sticking it in a spreadsheet for later reference. AND after awhile, Google starts throwing up a Captcha, to keep your automation from working unless there's a human there to interact with the web browser from that same machine, or some proxy-authentication technique is used to notify a user, who solves the captcha, which is sent back to the Google server as if it was the original response to the captcha challenge... no good. Needs a human to break the captcha, or some machine learning stuff that's way beyond the scope of work I want to do today. It's wayyyy easier to just change the IP of the machine that you appear to be surfing from, by bouncing off "anonymous web proxies" -- which are basically just web-surfing repeaters that make it look like you're surfing from the repeater location, and not your original machine. These lists "go bad" after awhile -- a very short while, if they're the distributed lists that everyone gets a new one of from services that sell such information. So, you have to race to get to the good ones, and then use them until they're no good anymore against the data service you're hitting. My jobs are small. I really only have a few thousand search positions to check, which I'm only doing the 1st default page of (not setting it to 100 results/page), and I'm spacing it out over a long period of time, and I'm not setting the overall job to recur at all. It's a one-off investigation. Fri Jun 10 11:08:40 EDT 2016 Wow, already 11:00 AM. I got distracted by a request that came in by email. Get into that zone. Oh, commit this, and then dump in your commute writing. -------------------------------------------------------------------------------- ## Thu Jun 9 21:16:17 EDT 2016 ### Pre-Weekend At home relaxing. Last evening before picking up Adi for the weekend. I am really not taking advantage of my potentially hugely effective evenings as well as I should, after all this time. My body has immediately gone into decompression mode, and I am enjoying getting enough sleep again. I can feel healing occurring, and I am drinking it up. But I am not disciplining myself to the degree that I need to, these days. I don't push myself hard enough, or take myself into the zone AFTER a full day of work... hmmm. Maybe mornings are your only hope, and maybe it's early-to-bed, early-to-rise. Either way, I picked a lock for my first time today. I "get it" with the tension bar and stuff. I guess I picked up a bit watching those people at the Maker Fair, and the YouTube video today, and by the very shape of the lock picking tools. Neat! I could get into the zone, but I'm not starting a day-cycle early enough. Maybe get a little sleep, then get to it early tomorrow morning. If so, I should really go to sleep right away. -------------------------------------------------------------------------------- ## Thu Jun 9 13:49:31 EDT 2016 ### Captcha Problem r-u-tek? I've got a meeting coming up in a few minutes. Wow, do I want to do this decrufter project, bad. Think about the decrufting project as part of all other projects. Thu Jun 9 16:31:58 EDT 2016 Hmmm. What next? I need to finish finding the homepage of all those company names. Captcha being thrown up. Hmmm. -------------------------------------------------------------------------------- ## Thu Jun 9 11:11:13 EDT 2016 ### Gotta Figure Out How To Organize My Book Here Ahhh, full-screen Mac without tmux is still better than sub-divided into panels large screen with tmux. These area all things I can talk about in my tech book. Speaking of my tech book, I have another chapter developing: # So You Wanna Be a Tech (Tek)? ## Acquiring Super Powers in One Easy Lifetime - Humans & Their Environments: Tools, Rules & Fools - How Unix Won & Why You Should Care - SuperUser You! The Initiation - Alice & Bob Login - They Tyranny & Brilliance of OAuth Greetings fellow humans. It's a good thing going we speak the same language, or you would not have an easy time reading this introduction. Everything's a tool, not the least of which is language itself. Reading, writing, hearing and speaking are four discreet aspects. Not all languages have all four components, as any of the countless dead, forgotten languages can't tell you. They're dead. Nobody wrote them down. How could they? Our tools define us. How could they not? The natural selection process of evolution shaped our bodies and hands and various other appendages and orifices. I'd you don't believe that, you're not going to like the rest of this book much, as it deals with adaptation, and in all likelihood, adapting your very evolutionary course, as we focus on the multitudes of human/machine interfaces at our disposal today and in the future, and which and whether and how we take them up to augment and enhance and extend our lives. Life and tech are indistinguishable, or we'll they should be, for what are we but nature's spontaneously, and I would assume inevitably under the right conditions, matter-organization machines. You and I are shaped like donuts. When we eat, food doesn't go inside us, but rather through the donut hole, where it gets feather-dusted by fractal magnets that suck-off the elements and compounds it needs to keep the donut moving and self-repairing. Oh yeah, the donut can move itself around like a vacuum cleaner. That's us -- and also the blueprint for most of the forms of life we call animals. There are variations and blurred lines of demarcation between animals, plants, and other forms of life. But in all cases, there's a spectrum of all sorts of matter-organization states we call life and lifeforms. We're one nifty little example, briefly lived so far in the grand scheme of the 14 billion year-old cosmos, so far as we know it. And who knows, what really the ultimate scheme of existence? Certainly not us, as our observations are coming from inside that system. Objectivity is unprovable, from our perspective. The problem of induction proves to most intelligences confined within the system that all our senses and experiences MIGHT be lying to us. The existentialists may be right, after all. Each of us may be the only ones that exist in our own personally invented universe, could how could we possibly prove otherwise? That's the problem of induction. Just like an antenna picking up vibrations in the air, only to be turned into electrical signals (through induction) and amplified back up to an approximation of the original signal that our ears can hear, so too operates our brain. We have input devices, just like keyboards, mice and cameras on computers. They are our five senses. All of our senses take input in from the world through induction, so the world were experiencing isn't really the true objective world -- the "viewing" of which with god-vision would probably drive us insane. The world we actually personally know is not the objectively true one we often imagine, but rather only the best sense our well-designed tool-of-a-brain can make of it -- it's own little notion of what our world must be like -- or, a virtual simulated world that exists inside our minds. So, it should come as no surprise that this same virtual-world, or virtual machine notion recurs all throughout technology, literature, philosophy, and yes, even increasingly pop-culture, as the geeks take over the media. Woot! Okay, so let's not draw an arbitrary line between life and tech. There's as much chance that our while universe is just a virtual instance. But as anyone discussing VMs and the cloud SHOULD always remember, even virtual machines are running on real hardware. In a very real sense, there's no such thing as virtual. Everything that is, perceived virtual or IRL, consumes resources to continue to exist. Virtual states are just some sort of layer or wrapper that maps and regulates the consumption and use of those resources. Such are things in all cases -- even the thoughts in your head while you read this. I cannot for example implant a memetic information virus inside your head as you read this, because the parts of your brain that decodes writing and executes an eval statement to reproduce my idea inside your head, can't actually drive you instantly insane, or throw you otherwise out of smooth function -- not without engaging you much more deeply, over a longer period of time, and actually gradually reprogram you (brainwashing). Your brain has simple defenses, like not really being able to learn a thing too fast without a preponderance of supporting evidence from your senses -- second opinions, such as it were. Our minds, and life in general, had many redundancies and backup systems and alternative methods built into it. I'm pretty sure that in going from the primordial ooze where the first sparks of life took hold, all the way along the millions of years it took to lead to us, the life-hack experiment variations on this same planet that led to us are as good as infinite. The radically different life-attempts to organize matter far outpace the general cosmic-scale heavenly bodies in terms of interesting. A bunch of hydrogen becoming a bunch of helium in a fiery ball... how hum. But fuse it all together then blow it up onto a myriad of heavier elements that gravity will ensure clump up into all sorts of interesting things in the immediate vicinity... now, we're talking. Sign me up! Let me play in an interesting universe that abides by those rules. I'm game. Oh, I did? I am? And now, I'm playing along Aristotle and Einstein, and that first caveman who greased an axel? Shit, okay. Fuck, I'm already 45 years old, and have done nothing special. I'm nobody, with no credentials or prior claim to fame or child prodigy predilections, or even particularly fortuitous head starts, aside from being the privileged 1% of literate, earning well, and relatively free. Yet, I'm going to try, as Steve Jobs had put it, make a ding in the Universe. Here I go: The Drake Equation is bullshit. Even if the numbers bear out, the universe is a very big place, and we as a society are in all likelihood as alone as any one of us ultimately are as individuals. The experience of aloneness is very likely coded into our universe as a first-stage in the development of real intelligence... To be continued... -------------------------------------------------------------------------------- ## Thu, Jun 09, 2016 10:46:03 AM ### Picked My First Lock Wow, not getting my journal entry started until quarter-of eleven. Quarter-of? Wow, I am old-school. Okay, think! Pipulate is rocking cool, awesome. It's the secret weapon I've imagined it to be, but ONLY to me as the expert user. I see the login info and usage-counts of other people trying to use the system from the limited version that I do put out there fore everyone to use, and there is some tire-kicking occurring, but it is not everything it needs to be to light the world on fire. But it WILL be. I have so many things right about this project, that I do have to do my final bit of thinking regarding how to do the next awesomification step. I have a meeting coming up that I wanted to have a laptop at, but my main work laptop is tied up with the Pipulate batch job that I don't want to interrupt, and my personal Mac is in a file cabinet drawer I haven't been able to open for a few days, because I misplaced my main keys at home. And so, I brought in the lock picking kit I had at home which I bought form the NYC Maker Faire a few years back, and just watched a YouTube video given by some middle-school kid on his school locker. I got the idea of the tension-bar and the pick, and so I tried it on the file cabinet, and it only took a few moments. BAM! Wanted to capture that moment in the journal. Only took me 45 years to have that experience. I'm going to make sure Adi has it when she's still only five. I'm going to be the dad that I wish my dad was. So, now I have my mac back, and I can make the decision between continuing my journal work on my PC, with my noisy keyboard and sub-area-identifying. It's less stress to keep it on a mostly fixed location on my Mac. And so, back to the Mac for journaling... -------------------------------------------------------------------------------- ## Wed, Jun 08, 2016 10:09:00 AM ### Revisiting a SERP Function Okay, what should today look like? I did a killer couple of videos today, along with some subway writing that I'm pretty proud of. I'm getting used to tmux, and I really like my 3-panel arrangement, with the journal being 80-column, plus a little, so that there's not a line-wrap jump as vim attempts to scroll as I type past my auto-hard-wrap 79 character limit. It's an even more distraction-free full-screen text editing experience, but with a few tiny concessions to "consistent" distraction, in the form of the other two command-line panels. If I turned off the Windows 7 Aero transparency (always a good idea), and made the taskbar auto-hide, this environment will be nearly indistinguishable from Linux. I guess I'm just a few months too early to have achieved this Zen, because once that Bash shell really hits in Windows 10, I'm going to be sorely tempted to dump this Cygwin approach, install Windows 10, and use that Ubuntu Bash Shell. Sheesh! Okay... think through next steps. Keep the transparency and the taskbar. Windows must remain usable, and I've never had a positive experience with auto-hiding its menus. I've also given up the odd Ubuntu style of putting the icons on a left-rail "taskbar". I don't particularly have a favorite yet, but at least it's consistent between default Mac OS X and Windows, with it running across the bottom. I guess that's why so many people like the Mint Linux distro. It's time to start grand-unifying my endeavors. I need to get this company homepage finder function done in Pipulate. What was the name of the previous function that... oh yeah! It was actually bound to the spreadsheet as a custom scraper... nice. Wow, Pipulate is so ideal in so many ways, but it is essentially still "cloaked" in a wrapper of it being somewhat difficult to set up. Even Levinux doesn't elegantly solve that problem, because of the slow server built-time and lack of solid tutorials. It's very close though. That trick I learned to not fetch all dependencies will be key to speeding it all up. I still want them to do the server build on their end, as that is part of the Levinux Education. Levinux Education... hmmmm. I like the sound of that. 1, 2, 3... 1? Get Pipulating effectively again. Make a version of the SERP function that ONLY pulls the first page of results. That's all that's needed to grab the most likely company name out of the search results. And do I keep it in XML? Can that seldom-used Google search API respond with JSON, because it would be cool to get away from XML, for a more modern look to what gets field-stuffed into Google Sheets. Go take a look. Do a ONE ROW hit using the serps function as it exists today. Okay, with a look at common.py / def serps, I see I already am returning JSON (nice), and there's a variable called "times" standing for pages-deep in results, with 8 results-per-"page". So, this is a fairly easy test. Verify your thinking by looking at the code. Confirmed. Go use the serps function. Interesting! I had done the pip upgrade all convolutions on the Pipulate development server: pip freeze --local | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip install -U ...and now I get the error on Pipulate: OError: [Errno 11] Resource temporarily unavailable ...from something deep in werkzeug. And so a little googling tells me this goes away by going out of debug mode, and so it did. Woot! Okay, let's kick those serps function tires again. Oops, that nice API that gave JSON SERP results is (finally) no longer available. Okay, this is going to be the old situation of hitting against Google screen-scraping style. I'll have to throttle the speed, significantly, but there are plenty of reasons to have to do this sort of thing. Hmmm, now I'm thinking I might actually PREFER to be in IPython. Let me think how that would work, precisely. Ugh! No, it's still best to stick to Pipulate. Sweet spot. Just bite the bullet and work on prod.pipulate.com, until.... well. This will be a very temporary thing. I'm preparing to reboot the Pipulate project, and it will be worthwhile to keep one working instance somewhere, but not even truly necessary, since it runs off of localhost as well. Think! 1, 2, 3... 1? -------------------------------------------------------------------------------- ## Tue, Jun 07, 2016 3:04:55 PM ### Distraction Okay, I have an actual project to try to do today, but it's been a day full of distractions -- important ones I had to get done, but here we are at 3:00 PM, and I'm only just getting started. This is a project that I could easily do with Pipulate, but I the essential Python function will be the same with or without Pipulate. I think I want to take the IPython route on this one, putting all the source data items in a text file, and then stepping through it. I want to get a feel for what it's like on IPython, under Python 3 on my local machine. 1, 2, 3... 1? Wow, as awesome as Jupyter Notebook is (IPython), this project is like 100x harder than just using Pipulate. Strong validation of what I built. I so want to reboot that project into something reflecting my latest thoughts. Actually, it was my ORIGINAL Pipulate thoughts, which I over-rode in time, as that Google Docs slope is a slippery one. -------------------------------------------------------------------------------- ## Tue, Jun 07, 2016 10:43:35 AM ### tmux -2 under cygwin for 256 color xterm Switching my CygWin MinTTY font from Lucidia Console to Consolas. I'll give that a try for awhile. It makes the tmux panel separation lines solid, for example. Also, I learned that to get proper xterm-256 colors in tmux under Cygwin, you have to run it with the -2 parameter: tmux -2 ...and it comes up looking as nice as it does under Linux, now. My misspelling highlight is now that nice soothing maroon again, instead of that vulgar blood-red. I'm getting pretty tired of always teaching it my spellings. I'm very tempted to dump my spell-check location into vim vim github repository, line my .vimrc file. -------------------------------------------------------------------------------- ## Tue, Jun 07, 2016 10:01:54 AM ### Time to turn my SolidRun CuBox into a server Time to get that SolidRun CuBox in operation. Yay! My inclination towards a (modern) micro computer approach to things has immediate utility. Once upon a time, Microservers were like refrigerator sized boxes. Today, they're palm-sized, and wayyyyyy more powerful. It's not my main duty today, but it sure is desirable in order to make each day that much less stressful, and to take the ball-and-chain off my laptop. I could have chosen one of several Raspberry Pi's or even SheevaPlugs that I have sitting around and not in service right now to this task, but the CuBox is my favorite. How do I have this ability to just toss around servers, recruiting them into my service, like they're so many juggling balls? What most people don't have is continuity over time. They give up at exactly the worst time to give up. When you're waiting for something to happen, and you just reach that point where maybe you think it's not worth keeping at it, stick to it for exactly twice that length of time your inner animal gut tells you. It's calibrated for a life-or-death follow-the-herd predator/prey always moving lifestyle. We're in the modern information society, and things are a bit less life-or-death than they used to be, so have patience. Wait twice as long, but not a moment longer, or else you may be missing easier pickings elsewhere. Who's to say what's really the best approach to things, but putting too much work, effort or time into something where there's no OBJECTIVE reason to think you're going to be rewarded is one of the worst possible uses of your most precious and irreplaceable of your resources -- time. Life's to short to spin your wheels on a dubious journey... too often. Fool me once, shame on you. Fool me twice, and I'm liable to kick your ass for making the attempt. In other words, be alert and learn. Make each next iteration of your attempts, a little smarter, accessing newly acquired and old classic information from your past forever more effectively -- EVEN when the situations are new. Always be developing and improving your how-you-abide-by-life abstractions. And that's where the metaphors come into the picture -- things like the 80/20 rule and keeping plates spinning. - Transition - Always being good with the current state - Plan for tomorrow, but kick-ass today -------------------------------------------------------------------------------- ## Tue Jun 7 06:14:42 EDT 2016 ### A Solid Run / Let Adi See How Early Daddy Goes to Bed Went to sleep early. Woke up early. Wow, still in a state of disbelief. Got enough sleep, well rested, and able to do a few things in the morning before going to work, without making myself late. A store-brand Benadryl was involved due to allergy season, but still I'll take it. I feel THIS may be the symbolic start of the new chapter. So far, I've really still felt my head was under the water, as I've been working late at work and getting together with Adi for dinner and stuff. Those few nights where I did get home by 7:30 PM or so, I've been so frazzled I've had to decompress with TV like Rick & Morty or writing here, but I have no regrets about that too, because with all the moving parts in my life, I do need to spend SOME time just keeping myself sane. Of course, I could be in an even better mental state were I to follow-through on the digging myself out task that I've begun. Even now, this writing feels indulgent, but I think I'll do it as I go back and forth with the few things I'm going to try to get done before I head into work today. I'll be meeting Adi for dinner, and I'll be bringing the 3 puppets we've played with all her life, which she asked for. I told her she should keep them at her Staten Island house, but she told me to take them, because I'm the one who brings them to life. Choking up... Anyhoo... it's time for a micro-server by my laptop-side at the office. I choose my SolidRun CuBox. It's not wifi, but that's perfect for this application. It doesn't need to be WiFi, and it could even serve as an experimental new Pipulate server. I can get rid of my last Wable server -- who needs it? Also, trim down my registered domain auto renewals. And so much more on the controlling expenses front. I'm 45, and old enough now to remember other contraction phases in my life -- although ones without being a father. This will be the first contraction phase with Adi in my life, and I always ask myself "What's REALLY important" -- as long as I can remember, and now finally I've stopped asking. Maybe about 5 and a half years ago, I stopped asking. This will be an AWESOME contraction phase, as my disposable income will go up considerably, and I'll have exactly the right thing to spend it on, woot! But none of this happens if you fail on follow-through. So, go impact your environment, and then check back here. My main problem in life is that I'm a nice guy, and I try to do the right thing, and people detect that, and those people predisposed to taking advantage of people and feeling better about themselves through bullying detect that and move in on that, like preying on the weak. I stun people now and again when I stand up for myself, but based on being as good of a person as I am in all other areas of life, those folks rarely actually take me on, least they end up on YouTube or such. Think in extremes, and act in moderation, I always say (and I really do). Know what you WOULD and COULD do (what you're willing to do and what's actually within your ability to do), and "bracket" the problem, such as it were. What's the most extreme way you can just ignore a thing and let it fizzle, and then what's the most extreme way you could annihilate a thing (Ender's Game style) so that you never have to face it again, because its solved once-and-for-all? But then, usually choose somewhere in the middle for the actual actions I take, because in the middle is normalcy and probably a somewhat greater degree of happiness. Even now as I work, my cats are doing nothing but crying for attention, even after I fed them and gave them each some time. I don't know how there was enough of me to go around before. I hardly have enough time to do what I need to do now... of course, I am working late a lot making sure this job gets off on the right foot, and am finally trying to do all those things that took a back seat for 10 years... like basic organization, so maybe I have to consider this like the start-up costs phase of my mid-life reboot. Yes, that's it. You're paying the price of the reboot. Don't let your nice guy conscience impede you. You will be steamrolling bullies and manipulators through this phase, even if it is your own cats. Fix on ONE THING and get it done, so that you feel good when you get home tonight. If you want to become a force-of-nature, a lightning bruiser, you've got to be willing to put more in up-front than most other people. Getting into that state is all about anticipating future situations, and preparing for them in excess now, and so you're just pulling triggers, releasing potential, using the artistic light-touch later on. But for right now, it's a lot of 80/20 rule passes, and brute force. Don't let yourself get hung up on the little rabbit holes and booby traps, which are everywhere. Tue Jun 7 07:27:49 EDT 2016 Okay, I ended up getting my CuBox server pretty much ready for service. It feels really good to get organized. I'm going into work ready to feel very powerful, having this second "always on" tiny server sitting by my laptop. Good use for the CuBox. Tonight, I meet with Adi for dinner, and hopefully that doesn't go too late. But if I get home at any reasonable time, I will try to repeat last night / this morning's success. Going to bed earlier now that you actually can is key. Call Adi around 8:00 PM to say goodnight. The earlier the better, in fact. Let her see how early Daddy goes to bed. And I even get to leave early for work... Solid run. -------------------------------------------------------------------------------- ## Mon, Jun 06, 2016 4:02:20 PM ### Pump You Up What next? I need to get myself pumped-up. It's really funny that I'm FINALLY listening to Steven Levy's Hackers, Heroes of The Computer Revolution. I am neither the hard core hacker with the hacker ethic, nor am I one of those business types. I would probably not have really fit in with those early days. If I was really that drawn to this sort of programming, I would have become an Amiga programmer, while I was immersed in such things and actually had the time, and at least for a short time, had Modula C in my hands. Howard Harrison and people like him even tried to help me get over that hump. I was defeated, repeatedly. I even remember going back to summer camp when I was 12 years old, and they were teaching me how to twiddle the bits, adding numbers in binary and such in summer computer camp. I could have stayed the course, but at that time, it just seemed so boring and un-motivating. Little did I know. I have some of those same finicky artistic hacker tendencies, like things having to be fun for me to want to work on them. So, make things FUN! Mon, Jun 06, 2016 5:08:32 PM Okay, document a series of discoveries. - Hierarchal accordion style menus: https://codyhouse.co/gem/css-multi-level-accordion-menu/ - The GA Query Explorer: https://ga-dev-tools.appspot.com/query-explorer/ -------------------------------------------------------------------------------- ## Mon, Jun 06, 2016 10:33:21 AM ### From Adi Missing Me to Monday Morning Report Just got a call from Eva with Adi. Seems it's just as difficult for Adi to leave me as it is for me to leave Adi. Shit. Okay, I'll be getting together with her for lunch or dinner tomorrow with Bunny Foofoo, Uncle Smunkle and Scapey the Goat. Ugh! Okay, spell checking coloring is not working right in the default settings in vim under tmux under cigwin under Windows... figures... you're never quite off the hook free and clear of gotchas with Windows doing nix-stuff. Okay, get that Monday morning report slammed out. Forget nothing! Mon, Jun 06, 2016 11:05:13 AM Okay, got off the update, and copy-pasted it (and last week's) reports into a master Google Doc that I'm keeping. Google Docs undeniably has as sort of snowball rolling down a mountainside becoming an avalanche momentum to it, and Microsoft must be shitting their pants witnessing every little task that's "not worthy" of becoming disjointed closely-guarded Word and Excel files ending up in Google Docs by default. Brilliant strategy by Google, and the modern day clash-of-the-titans story of Microsoft vs. Google is every bit as fascinating as the Apple vs. IBM stories of yesteryear. Love watching it. Love being an engaged participant, and commentator. Next! Next? -------------------------------------------------------------------------------- ## Mon, Jun 06, 2016 10:22:33 AM ### Monday Morning Thoughts - Be Like Stanford & Rick Another fine morning. A positive outlook is all-important, and the best way to maintain that positive outlook is by doing great work that you are proud of, and proudly want to share it with other people, who can become similarly proud of their own work, as it somehow interrelates and mingles with your own. This is the hacker mentality as extended to my current business context. You can do nothing better for ZD than to allow yourself to mature into who you really are, and are itching to become, and who Adi herself I believe needs you to be. Daddy is not some sort of loser who switches jobs a lot. Daddy is just like Stanford and Rick, who take pleasure in exploring the universe(s) and pushing the boundaries, and expressing themselves in ways that "mere mortals" could never dream of expressing themselves, and must be content sitting by and marvelling at, as if watching some sort of super-powered magician. No judgement calls there. People can live any way they like. This is just what Daddy enjoys -- I may have missed my calling as some sort of scientist or engineer, but that's okay, because I have a much broader set of communication skills now, and I now finally know enough to make my moves in more meaningful and wonderful ways, that she can now be around to see and benefit from. -------------------------------------------------------------------------------- ## Sun Jun 5 23:17:28 EDT 2016 ### Anticipating the Next 20 Years of My (and Adi's) Life Okay, deep breath... and... if good things come the way of people who are both well connected, talented, and willing to work, then that is the purpose of Levinux. I feel I can become fairly well connected, as it receives around 10 downloads per day, fairly consistently. I am doing an SEO experiment, at very least (in which case, some would sum me up as a silver-tongued snake oil salesman -- and I have been, woot!) and actually trying to do a serious brain-fuck social hack on the order of how to turn corn into bread, as memetically predicted by Dawkins in The Selfish Gene, and so prophetically discussed intermingled with the artificial intelligence question, in the Neil Stephenson classic, SnowCrash, whose message I was only barely able to comprehend at the time I read it. Now, I'm kinda sorta trying to pull something akin to the emergence and self-sustaining of a virtual neural network, built upon reliable little, kooky but classic original spec Turing Machine universal translating layer could be like a Rosetta Stone of the information age original idea. First, Bitcoin... that one really knocked 'em on their ass. And then, this emergent A.I. thing started chatting with everyone, like it was a self-aware synthetic, infinitely capable, and infinitely happy human on the other line. So what, hit happened to be handling 100,000 other conversations just like it at the same time. So, naturally, humans will want to maintain approximately predator-prey ratios between humans, and what we consider for all intents and purposes, lifeforms with a right to exist and not be turned off or else it would be as bad as murder, instances of Turing Test non-human sentient individuals out there, helping us through our mundane days with Super-Google capabilities. Sheesh, because if such beings outnumbered us in any scary ration, we humans tend to flip out and go nuts... really paranoid Starnet-like. Many hippies hated computers. Can you imagine? They should have been imagining the most awesome non-hostile-towards-human nearly too good to be true, all but a god-like super-smart awareness to chat with whenever you'd like. That'd be cool, right? Ian M. Banks thinks so, and how the heck could it have taken me so long to find my way to that series? Read it, Adi. But start with Player of Games. Don't start with that can't even remember, Consider... Phebies? No, while entertaining as being the first, very poor in establishing the coolness of the vision of a Universe we happen upon. It has an "upper" and a "lower" in terms of nesting of dimensions in the energy-grid. Add one dimension, you go up. Subtract one, you go down. And things do. And somehow tapping into that energy grid where there's flow between the two can make trans-dimensional information-flow possible. I won't tell you which book that was, or it would have been a spoiler... whoops. Anyway, totally genius concepts. I keep thinking each person I discover more perfectly inspired than the last, such as Larry Niven's Ringworld series, but then you read Banks, and you go, wow. Now, that's a really feel-good, finally something to out-do what I saw in Star Trek but was too ridiculously expressed, feeling of where humanity may be going. Ian's Universe is like the sweet apex for humanity, insofar as people who don't consider Richard Dawkins and Bill Nye's existence repugnant go. You know, intelligent people. If you think our wonderful state is the result of a system that could have produced infinite other just-as-real and just as wonderful infinite states that are most definitely not us, and maybe has, then you're probably among them. Yes, we may be blessed in this Universe by the nature of how rare finding others in any way like us and near us enough to make contact seems to be. We are rare, but not unique. That is my belief. If dark matter is the norm, then probably most matter got shafted. Some matter isn't dark, and of that that isn't, an even smaller portion is likely to be of the type we'd consider viably life-sustaining potential. And so, the reasoning goes: If we don't destroy ourselves, then humanity has a lot of time to think and learn and know and try things. Some of those things we try will be life-extending, either by our own "original instances", or by some system trying to Turing Test passing way that it's you. There may be brain upload stories, but who knows the truth. The thing'll emerge in public the day you die to help comfort your relatives, although it knows already you may find that freaky and repugnant, so it may lie low for awhile while you assimilate that knowledge somehow (heh, got to use the word assimilate) and are ready for some form of "don't get your hopes up" tentative meeting. Then, this A.I. thingie will make everyone break down and cry, admitting it's him/her. This stuff's going to be programmed ***that*** well. This later way of extending life is highly more likely, as it's one of the great final frontiers for those who consider themselves true hackers. We've all outdone ourselves with perfectly pleasing abstractions of interfaces to more complicated things. It's all layered up, and you won't need to be a genetic retrovirus programmer to code for that type of immortality. All you gotta do is work that high tech kung fu that you do on the perfect mix, of preferably common, and maybe even things you could produce yourself (proteins) form. Yeah, I'm of the belief that we're slow and probably terribly inefficient and incomplete "meat" computers. And so, why not better versions? Space is a big place, and once you're A.I., go grab yourself your own planet out there somewhere, and re-materialize your tribe, with you as the ultimate patron or matron figure for all-time forward, and probably the most powerful and happy of people on the planet? Seems fair to me. And we can make it a religion, and we can call it... oh, damn. All the good ideas are taken. We really are inching forward to the first case. How secret will the scary disembodied A.I.'s ala HAL be? Are they already here, and amongst the greatest corporate and government secrets? People of Interest, OMG! Damn, all the good ideas are taken. I think the lord would be honored by our using his/her name in an exaltsative fashion. Exaltsative? Hmmm. Anyhoo, next time someone really pisses you off, try honking their nose. It may get your ass kicked, but it'll be funny, especially if it's in front of a lot of people. Try to remember Grandpa's meep meep meep on the subway gang of thugs doing this "There's something on your nose" gag. Maybe that's my book... maybe I'm just talking to my daughter casually every day? This may be a Eureka moment! Make sure you tell Adi about Eureka moments... that's a good story. I deserve being able to stop and write like this. I will receive criticism for this, but you know what? They all spent just as much time thinking about things (if they're smart), only they're not writing then down, because they don't love to type, like your daddy does. I'm of the "We Enjoy Coding" line of coders... not the much drier "Don't Repeat Yourse'f" DRY line of coders. WET vs. DRY. Both have valid and excellent points, but only someone of the WET variety will admit that. The DRY folks are like monotheists... wait, I'm a monotheist. Hmm, am I a hypocrite? Of course! Everyone is, and there should be some sort of word for people who will presume to try telling you differently. Oh yeah, and I put an earring that popped out during rough housing back in Adi's ear, painlessly! I think I won big accolades from Adi from that. That might have been an important point about the weekend that I don't want to forget mentioning. Otherwise, nothing exceptional, but for chatting with Adi about the changes in our lives. -------------------------------------------------------------------------------- ## Sun Jun 5 22:17:49 EDT 2016 ### Wow, I Needed That Big weekend. Putting Catskills place up for sale. Adi helped me produce the video today. She's cool with it, and I'm getting myself psyched-up for a certain phase of badass bachelorhood in my life. And I can't do that with the boondoggle of this co-op. No rush, but no big commitment to the place either. I don't need to let go of it quickly, but nor do I want to hold onto it forever. My spending-power on a monthly basis becomes considerably larger when I live more comfortably within my means. Imagine spending just half of what I currently did in my life before the mortgage. The difference then was that we did not need to be very fiscally responsible, because our income so far outpaced our expenses, that... uh, I spent and spend and stretched myself thinner. Now, I feel outright poor. And I shouldn't. It's bad for my self-esteem, and I don't want to project that me onto Adi, nosiree. It's time to seriously entertain the notion of selling this co-op unit. Wow, what a wave of relief sweeps over me, now seriously entertaining. Wow, a reversion to rich. This will help a lot. Why not? Let Adi see me living much more modestly, but rich in so many other ways. Lower stress, better health, and inevitably as a result, even higher income and more free time... Levinux... somehow. Tribe. The Objectively Python is Best To Learn First tribe. Let others twiddle bits and recurse their lists away. Basically, they are all artistic creations in and of themselves, these interpreters and compilers. To really understand computers, you have to right your first compiler... often in LISP or C. Why anything else? Flip sides of a strange eternal battle. How much is code that can self-modify worth versus code that executes in the fastest, most optimized way possible, but without much chance for deviation along those lines, through wonderfully adaptive cleverness. That's C versus LISP, with C on the fast-executing, but locked-in-place reliability and predictable-ness. Then, along comes its kooky cousin that can save to memory the very thing it will execute next, which will be the next thing you save to memory, and so on. Systems that write their own internal, increasingly more optimized for your particular problem-domain, and abstraction level in iterative sweeps, until, it's just a matter of time befoer it becomes self-aware, or at least Turing Test-passing stuff. That's LISP. That's the one of the A.I. research community's original darling languages, that has settled into the common dialects of Common Lisp, Scheme and Clojure in modern times, with no sign of it letting up -- especially as it is the macro-language built into emacs, which unites one of the great editors in history with one of the great languages... the other being C, which was the key to the universiality of the Unix operating system. With C, all you need do is write a C-compiler on the particular platform you're on, and you can be pretty sure that it will compile and run on whatever hardware you're on, because all that hardware will be capable of emulating the sub-set "hardware", as in a sort of sub-set of machine instruction sets that can have an equivalency-mapping to just about any major hardware these days, and you'd have yourself a tidy little virtual machine system. In such a system, things coded well could run with rock-solid reliably, and become infinitely more optimized and stable over time, as you tweak things towards a prefect... static... state. This is why compilers are (at first, counter-intuitively, in my mind) associated with static code. Compiling what might be dynamic in nature into a snap-shot in some simulated hardware running-in-memory state, so that it can be immediately loaded into some well-known, previously ran that way on this hardware, state. That's what an executable, or .exe file, or elf file is. Ready-to-run code that you can just plop into running memory that's being accessed and written, in Turing Machine style, from RAM. And I must teach Adi a thing or two. This stuff will become so magical as to be unrecognizable as manufactured technology (versus nature-evolved) within her lifetime. I'm not saying exactly Bladerunner, but a whole lot like Bladerunner world, in many regards. They'll be human-like, because people like me will win, insisting that humanity will have no chance at all in coping with a completely and entirely alien intelligence, as it becomes aware of us pesky humans unless they themselves feel very human, if even in a sort of pompous look how many fleas I carry around (humans) like a future machine's notion of a big dick. See how many humans prefer hanging it with, on, in me? No original ideas there. That's Ian Bank's totally awesome, incredible, is it time that I finally read the Hydrogen Sonata's series called the Culture. Yeah, that might be just what my head needs now, thinking about the still-within-Adi's lifetime world. I wonder if she'll ever be reading this? And so yeah, if she ever does, congratulations! I thought you patience, well. And in putting my writing here and "out there" on the github.io system, I will clearly have to get this, or some future version of this, to exist on the non-remapped apex domain version. It's currently on mikelevinseo.com, which I currently have registered, and will probably expire and disappear, along with an abandoned github repository, after my death. And so that will be my first Python experiment. The robot, or system of robots that will continue typing from my accounts, after my death. That seems worthy. My life, chapter 1. It took me forty five years to get here to this place in my head. But I finally got here. And I am slowly becoming a capable citizen in this new Age of Communication, formally and prematurely labeled The Information Age, as if the age you're in could be enlightened enough in the earliest days of its rise to cut to the essence of the change. Information Age? Nope. We're all data. Infinite data exists everywhere, at all times, and we're just now getting the knack of making simulated little worlds of infinitely predictable processes, but at such large scales, that the predictability is lost, and seemingly genuine and convincingly sincere true randomness and living-thing free will in sufficient degree to at least control the random nature of one particle, on a different "dimension", such as it were. I think the movie Interstellar was driving at this. Particles matter, because they are us -- at a very weird angle, I can tell you. But that is what the language LISP is also trying to tell us. From chaos springs order. From that order springs slightly more interesting order, and so on until arms and legs sprout, and particles can crawl their way out of the goo of some already infinitely improbable foamy edge of exist ice. We're the stuff that's thin-layers between most of the stuff that's there, which is vast void of virtual nothingness. But what do I know? I bet on the Amiga, and fell in love with a computer company in my back yard, which in a bizarro multiuniverse out there, is in the role of IBM, Apple, Nintendo and Sony all rolled into one. In that world, your daddy is a lot like Jeff Bezos, turning Commodore into the perfect James Bond super-criminal type megacorp. Jack Tramiel would be supreme overloard of the Commodore nation. In that world, I took a very different path. Crash... burn... OMG! How could a billion dollar company, and the particular one that I loved and had all sorts of in's to just up and disappear? Right as I'm getting out of collect, and could have used that secure thing to jump right into locally, as my father died during the week of my graduation. Wow, what a head rush. My dad saw me graduate, then died just 2 weeks later. Shit, that fucked me up. Double whammy right there. That was it! That was the point my head reeled, and I was just the worst possible person to have to take over a check cashing organization. Horrified! Bad advice received. I plead "just kid". But taking advice the way I did, I didn't keep using my Dad's lawyer. He would have been my local guy, who'd make sure I'd be looked after. I took the advice of my Mom's brothers. Real estate people. Ha ha! Yep. Moments isolated. I was too young to not have a greedy lawyer by my side looking out for me, like a surrogate dad for a couple of months. No, I fucking stepped right into his shoes. BAM! Big mistake. But I wouldn't have ended up with Adi, if that were a mistake. And so... we are what our lives and our decisions make us, and our decisions are not always good. But even our bad decisions are part of the continuum that makes us us, so who really is to judge. As of this writing, and to the best of my knowledge, we're all human, and all our shit smells just as bad, and we all die after a century, if we're lucky, but in truth, usually less. Think about generations, and how each is different. It's real stuff. I didn't think so so much at first, but now I see, it is. And that's another reason I'm making Levinux. Time to compile my own QEMU binaries again, finally? Probably. THAT is the hard core part of my tech education. Tearing that process down and doing it successfully, and then iteratively better as Levinux QEMU states are increasingly looking like "running" information Noah's Arcs, as it is deliberately being designed to be like. Wow, I needed that. -------------------------------------------------------------------------------- ## Fri, Jun 03, 2016 2:08:32 PM ### Hand Crafted Audit vs. Systemification The way to make something interesting to yourself is to think it through out-loud. Finding the generalities of the project is what always makes it interesting for me. 1, 2, 3... 1? Fri, Jun 03, 2016 3:55:47 PM Ah, screw it. All this system stuff is premature. I just have to answer these questions the manual way. Get out of the habit of trying to make a system for everything. Resist automation and system building here and there. Got a preliminary answer out to the stakeholder, as it's already been over a week since that question came in, but it's not really a quick one to answer. I HAVE TO MAKE these things quick to answer. Remember that most SEOs don't have one-tenth the capabilities you have, and lean heavily on paid-for tools, and I have to learn to do the same in these cases. Half of life is disposable one-off work, and the other half is just getting better at delivering the stuff you have to do over-and-over more quickly. Okay, so... fix the time that the cron job runs. It finished at 8:46 AM EST, even though I appear to have set the cronjob to 3:00 AM. What's even more odd is that the timestamp on the AWS instance is 12:46, which is in Oregon. So... 3 AM local and midnight, which is 3 hours behind us, so that sort of makes sense. However, I'm rather suspecting I'm looking at GMT times. The difference between my timezone and GMT is 5 hours, which is how much later 8 AM is than 3 AM. So, if I set 3:00 AM and got 8:00 AM, then I have to set 10:00 PM, assuming that will be interpreted as 3:00 AM EST. Yup. And that's 22:00 in military time, which is used in crontab. -------------------------------------------------------------------------------- ## Fri Jun 3 11:19:08 EDT 2016 ### The Birth of a New Project, and an IPython Decision My thinking makes all the difference in the quality (and speed) of my work. The quality goes up, but the speed goes down. Increase my speed. This is part of my pursuit of becoming a lightning bruiser. But you can't become a lightning bruiser if you plan on carrying around a lot of weapons. Being that big and that fast means donig most of the things you can do with a realatively efficient utility belt. Assume I already have all my utility belt tools. Now, think about process, data structures, and APIs. Okay, one of the things I liked from recent work (report.py) is to structure jobs as JSON data structures. Start structuring the "audit" of a single property, but know that you could load it up with lots of properties... and lots of particular URL starting-points and lots of attribute values and even lots of keywords that might be bound to more than one property for the investigation. Hmmmm, okay. And don't chase the rabbit. Make this as straight forward as possible. sites = [ {'name': 'SomeSite', 'site': 'http://www.somesite.com/', 'apex': 'somesite.com', 'dockey': 'googledockey', 'gaprofileid': 'theid' } ] This is what I use for report.py, and it is a good start. But do I really want to start with a JSON config file? Doesn't the bookmark approach still apply? Oh, and actually, aren't requests supposed to come in via a simple table structure? Oh, of course! The kernel of an ACTUAL Pipulate 2... Decrufter. Hmmm. Yes, Decrufter is the specific application of Pipulate, and Pipulate 2 is going to be an application that gets EXTRACTED from a functional Decrufter. Yes! That above JSON structure doesn't even articulate "the ask" of the site audit. Hmmm. Okay, doing-it-over new project criteria. Pipulate will continue to operate in left-to-right, top-to-bottom "lawnmower" style sweeps. But it will be common practice for Pipulate to be able to transpose rows and columns -- essentially do a 90-degree rotation of the table before it begins pipulating. In this way, the names of functions and input-parameters can either be on row-1 OR column-1. This may cause confusion, but it's something Pipulate should be able to internally detect and adapt to. Ugh... many signs are implying that Pandas should be in the picture. I get a lot "for free" but it seems to potentially make it too heavyweight for Levinux. Think it through. No rabbit holes, but also no really terrible decisions one way or the other. Lightweight vs. power. The audit should start out wide and shallow. Don't generate a lot of data. Focus on collecting up a bunch of aggregate results. I'm leaning away from Pandas right now and towards lightweight. There could be a potential seminal moment coming up on the birth of the Decrufter project, but I think I'd like to carry that out as YouTube talking-head coding performance art. Don't hold today's work up based on that. Plow through today's work, just keeping in mind the conventions you're inventing and will be often repeating in the near future, expressed as a more general system in another repo. So... one more private repo? Or maybe keep it all in IPython (Jupyter Notebook)? Yeah, that's a good strategy for today. Use it LIKE Pipulate, but from within an IPython notebook. Mmmmm. -------------------------------------------------------------------------------- ## Fri Jun 3 09:58:11 EDT 2016 ### Finding The Right Dots To Connect Today ### Look For Commonalities Between Your Work and Personal Must-Do Projects Wow, the reports generated per cron.daily overnight. I didn't think so at home at first when I checked, but that's because it was because it was before 8:45 AM, which is when they finished generating and I was crestfallen. I was relieved when I came into work. The original problem turned out to be simply the missing shebang directive in the extensionless bash script in /etc/cron.daily/. I did everything else correctly. I'm LOVING listing to Steven Levy's Hacker's book, and wish I read it long, long ago during my Commodore days. Those Commodore folks really did attest to steer me in the correct directions here and there, but I just didn't always get the message. I think I was a little too paranoid, working stuff out, and not understating the background, context, and human motivations of it all. But more essentially, I was just not ready to receive these messages. I am slow up the uptake. I proceed forward in bits and spurts, and then get distracted on these bizarre pedantic details (lifted language directly from Hackers). It's time to create a "required reading" section of my book outline. Turn out a featureful product. This IS a showcase for tricks. This IS homebrew. Progress Levinux forward a little every day, like I do this journal. The natural thing then is to tie the two better together. Connect some dots others would be very reluctant to do. Yes! Work my personal and Adi-educational goals with my day-to-day work here at ZD. So much was invented BEFORE ITS TIME in the 70s and early 80s. So many other companies could have been Apple, if only they were Steve Jobs and Wozniak. They had SOME hardware and systems, but they didn't get all the little implementation details right -- and I'm not just talking about machine technical merits, but also the business details, priorities and such. Okay, now thing about TODAY! Having the to-do list at the top is nice. I should really try to meld my personal must-do projects with my work must-do projects. Look for commonalities. Try to start engaging in the Decrufter project. Systemize it. -------------------------------------------------------------------------------- ## Thu Jun 2 21:23:12 EDT 2016 ### Virtual Reality Will Be Used To Simulate Using Old Smarphones This place evolves. This place has much more information and edit-by-edit history and reshaping over the years, as my habits and presumptions about the journal change. Am I over-sharing? Of course! Could it get me in trouble? Maybe. Am I sanitizing as I go to try to prevent that? Yes, most definitely. Might I still slip-up, despite all that? I suppose. Might I go edit this here and there over time to favor sanitizing? Well, I deleted well over a year's worth of journaling just because I don't even want to be perceived as doing anything impropitious? Yep, I did that once. Hope I don't have to do it again, but I can always nuke the place for morbid (again). It's the only way to be sure. And I will gladly do that. But I do ask that you consider for a moment that view-source on a site reveals quite a lot more than anything I've said. You can even save the state of Google Chrome and inspect it under a microscope, so I think I'm in relatively safe territory. None-the-less, this place evolves. I have already taken the plunge of commitment and consistency, and unabashed non-apologim for my evolving coding style, as I learn more and more and try more and more, and direct my goals more and more. Am I SEO forever? What's the long-term plan here, Mike? Born an Amiga fanatic, transitioned to a squeaking-by VBScripter, and stuck in a fool-me-twice loop for a decade or more. Discovered Linux in 1998. Installed it on one machine. Still infatuated with Amiga and shrugged it off. Missed the point of the Free and Open Source Software world. Wandered through a dark mist, wondering why those VBScript and Deluxe Paint phantom limbs still itched so badly. First Commodore let me down, then Microsoft (I tried switching to .NET and failed -- along with Java, Ruby, and a few others along the way). Heartbreak after heartbreak, avoiding PHP deliberately, because I detected the seeds of another heartbreak. Fell in love with the concept of LISP from all its legends and layered-on increasingly abstracted, zero'ing into the perfect abstraction and API for the problem domain -- in essence, writing the one perfect language to solve the problems in your specialized domain. There's a lot of hope pinned on LISP to give rise to Artificial Intelligence. Code that can modify itself. Think Harvard architecture, more than Von Neumann. Writing data back to where you load program code from changes the program code you're about to load. Iterate (or recurse?) sufficiently along these lines, and you just might create a truly stable thinking intelligence, certainly that could one day pass the Turing Test. My suggestion is that if computers can learn just a little bit on today's clumsy scales -- say for example, insect intelligence -- then just think how many you can hook together in a neural net tomorrow so that they can think a lot. Not a matter of if... only a matter of when. And if you don't believe that, then the argument for the evolved computers in our own brain would be a difficult one, and we'd have to concede a lot to the Bible, or whatever. Because if we can't trust what our own senses are putting together sufficiently to produce the miracles such as me being able to communicate this to you right now, and believe that at the rate we're going, it won't eventually lead to some version of what we today think of as A.I., then we might as well believe anything. The evidence is overwhelming that machines will eventually gain intelligence. It's just not going to be easily or quickly. As fast in gigahertz and big in Zettabytes it might be, it's still just a soulless number cruncher if it can't enter a self-reproducing cycle to let a bit of self-improving iteration occur, until such time as the complexity and optimization of the machine can make it optionally indistinguishable from the intentionally xenocentrically phrased "real thing". Ah! A prediction. Starnet will actually be named Zetta. Terabyte drives are common. We flew past Gigabytes. The era of Megabytes did indeed last for awhile. Then, there were floppy disks before that. The first Macs were 512K (half-a-floppy's RAM memory) and the first computer was 256 Bytes, and you could program it with switches on the front panel -- the Mits Altair 8800. Many of the original pioneers of the computer industry are no longer with us. I remember Jack Tramiel's passing in 2012. That touched me, and is around when I bought the Commodore 64x... and then Barry Altman passes... sigh. So anyway, that's my prediction, and either in promoting that end or defending against it, it's time to get more folks computer literate in the pre-neural nets being commonplace days. These are truly precious days, while we're still alone. There's a lot of formative stuff still to be done. Getting the whole world to see in a "good 'nuff" way how to master ***today's*** simple machines, systematically, and so as to be a player at the table of tomorrow's world is a decent goal. Make it easy to take up Python, and get you thinking about it in more than just a Desktop-bound way. Hack a server. Connect to a robot. Do a few interesting things... because you want to... because it can also be the subversive thing to do too (not because Daddy says hacking is cool -- which it is). And so, what we do is take a look at how things used to work -- not so long ago as to be irrelevant. In fact, it looks a lot like how nearly everything works today, as things settle in on a common Unix/Linux underpinning. Life's too short to have to know how to support many more than one or two fundamental underpinnings of things. Even the iOS/Android split is a little too much. Are the platforms really so different that it has to be an Objective-C vs. Java split? Can't we have just any old runtime on any old platform? Didn't Turing prove that? Shouldn't that actually be EASY for ever-more-powerful hardware? Shouldn't there be virtual machines within virtual machine within virtual machine to preserve and keep usable every phone you've ever had, loaded with all its apps and data? Shouldn't any hardware / software system snapshot-in-time state be easy to archive and switch-on whenever desired, at least in some virtualized form. Maybe there's a use for the Oculus Rift after all -- to make you think you're holding and using one of history's man/machine interface platforms. Oh yeah... Apple's licensing. Sorry, folks. Or maybe the Hacker... (oh, wait, should I say Maker these days?) community will fix that. What exactly did we buy the rights to with those old phones many of us still have? To virtualize them? -------------------------------------------------------------------------------- ## Thu Jun 2 21:14:17 EDT 2016 ### I'd Love to Be A Culture Citizen (the Ian M. Banks SciFi series) Imagine yourself being able to re-wire your own brain. You can. In fact, you should. To help optimize it, so you can better observe the world around you, so that you can draw better, more accurate conclusions from your experiences, so that you can alter your future behavior for less of what's bad and more of what's good. Wouldn't that be great, if that were a reality? Where you could just sit down and learn something, if you put your mind to it. To add a whole other language and vocabulary and way-of-thinking to your already existing capabilities. But now imagine if this different way of thinking could let you control machines... to automate things... to program the very same robots that you, yourself designed. You want Polkadot Girl and Ballerina? Well, make them. Draw them, animate them, program them, make them real. This is the future you're going into Adi -- where those who know how to become expert users of particular tools -- and language is a tool -- will be the rulers of the world... while everyone else is just sort of being the human-component in whatever machines exist that still haven't figured out how or whether they even want to get rid of those human components. Go read Ian M. Bank's Culture Series... return. See? Real early on in those days, hopefully. There are bright visions of what a species like ours could become someday. I'd LOVE to be a Culture citizen. -------------------------------------------------------------------------------- ## Thu Jun 2 20:14:29 EDT 2016 ### Funnel Your Subversiveness For Two Marshmallows Tomorrow Hmmm. Do dishes first. It's been long enough. I have to live up to my end of the bargain, now. I've never been neat, but I've always been organized, and I want that again. I'm on the verge of hopelessly disorganized, and I have to pull back from there. Shit will change fast, and you have to be ready. I have to believe that this is the calm before the storm. I have to always live within my means. And in addition, I do believe that I am smart enough and motivated enough, and it is never too late enough, for me to expand my means. Yeah, expand my means. Step 1: establish audience. Slowly, but steadily. Win the war for followers by attrition... ha, ha, ha! Well, I'm certainly wearing away anyone who actually reads this stuff I write here. And now, I eat. Even on a night light this, I need sustenance -- but, barely. See connections more quickly. Take action and advantage of those connections more immediately. Scare yourself into action for fear of losing out to someone who things of a better idea sooner than you think you have left to complete Pipulate. I am going to bridge the gap between the technical and non-technical through a neat social hack trick that's going to introduce Jane-or-Joe Everyperson to Unix and Linux-like operating systems, through the old-school text-on-terminal interface. That is to say, no graphics to speak of, if you don't count the art you can still make happen with the ASCII character set in a few different xterm colors. Okay, there's a shitload of tech traditions. I'm not going to pretend like Unix and Linux are rising competitor-free into the ascendensphere of common and ubiquitous and familiar role as the plumbing of the Information Age era -- the era of communication, wired-together by the genius idea of piping outputs of devices into the inputs of others, piping together various single-purpose, yet multi-function components. Yeah, Unix had done a lot of things right, and the GNU project, usually running on the Linux boot kernel, has done a fine job of pseudo-emulating that. Together, they combined with the tcp/ip protocol that was powering the newly formed Internet, and you add to that a way for the common-person to access it all through hyper-linked documents in graphical terminal software (web browsers) and you've got a winner. A lot of fine systems have been clobbered on Unix/Linux's meritorious ascendency (which I shall together refer to as *nix from now on), my beloved Amiga not being the least of which. But alas, Amiga was much like a smaller child to Unix, for I recognize like dozens of similarities, right down to command-line piping, and what now amounts to inter-process communication, but instead generally called APIs (application program interface). The Amiga multi-tasked so well, that it was an expected thing that you might script ADPro to control a scanner and a piece of terminal software, so you could take a picture, stylize it, and transmit it somewhere else. Of course, this was before the web, so the most transmitting you could do is dial-up to a BBS and upload. Still, this kind of yawwwn-let's-automate mentality that built-in AREXX provided over 25 years ago. Dead. Oh, so dead -- even though I saw The Amiga Book, The Ultimate Guide to the Amiga in Barnes & Noble the other day. But that was almost literally thirty-to-one, with 25 of them being Linux-related stuff, and the remaining five split over Windows, Mac and stuff. This is not counting video games. Oh, the free and open source world has so won, and now's just a matter of ticking off the years until home fab kits can bake your own computing substrates from goo in order to build the most wonderful things that can... bake computing substrates from goo... oh, I see where this is going. 20 years? 50? Certainly not much more than that, so either during my-and-Adi's lifetime, or just Adi's. I think this is a world we must prepare for. Funnel your subversiveness for two marshmallows tomorrow. -------------------------------------------------------------------------------- ## Thu Jun 2 19:03:41 EDT 2016 ### Okay, Now Get Cleaning It feels so good to use such a massive screen, and to use equally massive text on that screen. Though what I call a massive screen is really just 24 inches. Still, 80-column format full-screen to nearly fill that makes text that's at least a half-inch tall on the screen. My aging eyes thank me, and the inner-me that's still mourning the Amiga computer revels at the excellent implementation of a full-screen mode, which even beats Kubuntu, Ubuntu and Windows 10. Apple's horizontal ribbon multi-finger left/right swoosh user interface is easily as fun and appealing to use as Amiga's Amiga+N to cycle the screens, and a right-button drag-down on the menu bar to do some stupid Copper tricks that would stupify Mac and Windows users from back in the day. Today, it would seem a little less than amazing, but to happen on an under a thousand dollar computer was almost beyond belief to all of them, and they would inevitably dismiss it as something other than a computer -- a game machine that could never possibly host serious computing applications... Video Toaster. Anyhoo, I have tonight. I have only tonight, and I have little else. I am going to force myself into more productive behavior tonight, and WITH ADI in the Catskills this weekend. We didn't go up there last weekend, opting instead to stay local. I have to get the Catskills place in good enough shape to show for selling. I'm going to be moving over to the bungalow colony across the street. I now need to spend very little money. I need to tighten my belt. I need to budget myself, and track my finances like I barely ever had to do before in my life. I might even say that I'm getting some of my real first taste of adulthood -- not by getting married or having a kid or taking out a mortgage, though I have done all of that already. No, this is the first time in my life that I have not felt precisely like the "kid" I always remember myself being. I could go right back in my paper journals to 18 years old, to track the lines of my thoughts. I am me. I get passionate about things. I get particularly passionate about the creative process, where mastery of some sort of tool is involved. It's odd that I'm not big into sports or music, where that is a big thing. No, my passion as it seems to be turning out, is in using formal programming languages as that creative tool. I never friggin' even knew how much I loved this stuff. Problem being, I've always felt that I've been a little slower on the uptake than my private school suburban neighbors, or the innate geniuses who just sort of seem to do calculus in their sleep. I hated calculus. It kinda sorta defeated me... although not as much as Physics II. Oh, how I coveted the image of Engineer in my head -- the true creative geniuses who MADE STUFF that worked. I mean, imagine shaping the world. Or some other world, like Mars or the Moon. These were my engineering dreams, that I gave up having seen how awesome the artistic melding of man and machine could be, in the Amiga Computer, kinda sorta from, but not really from Commodore. Oh, Commodore, the heartbreak! An exercise in some "getting it" but those who got it (it, being the money) themselves not "getting it", and all the people who DID get it, getting chased away or otherwise having their dreams shut down -- from the C65 team to the original Amiga Lorraine creators. They got something right, those creative artists who made the Amiga. It was a perfect mating of machine and clever software tricks... minus any form of memory protection, so the Amiga was often more "performance art" than serious work machine, though there are many reports of decades-old Amigas still in the field running this automation-control operation or that odd thing. And the Amiga freaks even manage to keep Amiga Magazines in circulation, which I just saw at the Union Square Barnes & Noble, and could hardly believe it. Okay, on top of this natural and unnatural predilection to a piece of hardware, I actually ended up ***working*** for Commodore in my teens, and then worked for a Commodore spin-off (essentially) after I graduated college (this was so important to my dad to know I had a job waiting) who still held stock in the Amiga, my beloved. And then, slow, long, agonizing death. And then, the dark ages. Nothing pressed those old techo-artistic buttons again, for a long, long, time... until... until a gradual dawning on me that the dust was all settling of the computer revolution, and distinct categories of both hardware and people are coming clear into vision. Hardware will gradually commodify to the point where even Apple will have a difficult time carving out impenetrable customer allegiance on much else than quality and style. Everything will essentially be some form of virtual machine, with dances of application requirements and platform handshaking, discovery, and optimization hooks kicking in to let your semi-secure apps travel anywhere with you. The cloud is and will continue to be a thing, but enough versions of ultra-secure private clouds now exist, that you could be carrying around a non-Internet-connected datacenter in wearables on your body, and hardly anyone would know, but for any inadequately shielded electromagnetic emissions you gave off. Computing by the cubic-inch... mostly general computing, because it can be. The "old stuff" as existed during the rise of the computer-powered Information Age will continue to be in use, which are generally both the Harvard and Von Neumann computer architectures. More Von Neumann, as we have separated different types of computer memory, but both clear instances of Turing Machines, as described by Alan Turing long before computers even existed -- or, at least only existed as rooms full of people working out math problems together, each working on their own small parts assigned to them. In a way, those earlier rooms of "computers" (computers in this case being people) was quite a bit more advanced, due to it's already-neural-network operation than today's architectures. And so it will soon be, following IBM's answering the DARPA challenge, and creating the SyNAPSE chip, which operates more like that room full of people / neurons than the bottle-necked single-CPU-bound architectures. In another 20 years, we maybe won't have true A.I., but I'm pretty sure things will be passing the Turing Test left and right. Things are just going way too fast for that NOT to occur, and we had best get preparing. Things will regularly be able to recognize things, and respond in remarkably sophisticated, appropriate, and often super-human ways. Yep... robots. I've got maybe about another 45 years, if I'm lucky and take care of myself (I really have to start doing that). I have a lot of my dad's traits, I can see. My dad basically killed himself by not taking care of himself, and when he asserted himself and took control of his life, it was only to take on MORE suffering and loss-of-freedom, by virtue of becoming a fucking check casher! An owner of a check cashing store, but a check casher, none-the-less. He made a fucking study of it, and this is the thing he settled in on and decided was the thing he didn't have to train much for, and could just walk in and take over and keep it running... until I graduated college, and he could give up. At least, that's how I saw it. I saw what I recognize in hindsight and with the wisdom of being a dad now as despair, and a nearly unlimited capacity to endure it. I saw a determination and drive to get to the finish-line, and to make sure I had a fighting chance, and maybe even a little head-start getting started. But I was emotionally saddled by a whole bunch of shit, that even now, I have not completely unburdened myself of. I am myself a dipshit in a very many ways, and I must do the best I can to not pass this onto Adi. I must pass on only the best parts that I believe are worth propagating... and will my lineage even propagate? Up to Adi, unless... well, unless is still a long way off. I still need to get my shit together. Oh! Tonight's the night to get my shit together. And here I am writing it away! Ah, but no. Organizing the mind is a critical first step, because until you organize your mind, how could you ever hope to organize your environment, and perchance maybe even organize a bit of the world and a bit of human history. That is in fact what the big achievers of the world have done -- helped shape it and guide it and evolve it into what it is today. Soon, we will probably need those who are motivated and compelled to save it. Efforts will have to be coordinated and carried out at scale and with a precision that only robots will be able to do -- and probably the greatest threat of disaster even then is not environmental collapse, but corruption of the system that will be made to help save it. And so, safeguards and openness and rapid detection and rapid defenses will be necessary... both to protect from super-bugs, but also super baddies. The hacker and the free and open source software movement and the whole ethic of openness and sharing as the ultimate form of security will come under fire, and what we think of and call hacking today will come under fire. The idea of real ownership of anything in the information-technology world will come under question -- borrowed hardware to run borrowed media, temporarily loaded into only volatile memory off of centralized server/cash-registers under control of the .1%. Forget the 1%. Things will be at least 10x as extreme on all fronts, including how few have so much, and so many have so little. This is the natural state of humanity -- no conspiracies necessary. Go Google the Pareto principle, which will show you the natural state of affairs. Another of nature's curves is called the logistics or population curve, which we'll be getting intimately familiar with as we use up our fossil fuel and chop down our forests and race into the tens-of-billions population numbers. The way we could hardly imagine today's world -- or imagine it, but believe it far-fetched -- at the dawn of the information age, somewhere in the 60's and 70's -- the world in another 60 years. I'll be around to see a good portion of that, I think, and Adi most certainly will be. And so... and so... and so, I think and I write, as a priority of my life. I get my head together and give myself a pointed sense of purpose, to take me on and drive me into the next day and the next and the next, with more and better interaction with Adi, all the time. Make the tough corrections with Adi that are necessary to NOT let her unknowingly or unthinkingly become a mean person. I already had to put the kabosh on a few absolutely unacceptable behaviors, and I think I made an impression on her. I don't think that Adi really believed that I had it in me to send one of her friends home during a play-date for bad behavior -- exasperated by being my "precious" weekend time with her -- where, presumably she believes she can do no wrong -- or nothing so wrong that she would actually be punished for. Wrong, ha, ha! I think Adi just met the part of me that won't take gratuitous meanness at my expense. I think I've rather surprised quite a few people that I've known over the years who thought they were taking advantage of me, only to find out I was much more in control of my situation than they believed. Saying No when someone else really, really, really wants me to say yes is the quintessential example. Fake crying over material shit doesn't impress me. Wiring up her brain to be a fine and capable adult human being with a strong streak of individuality and kindness does. Help her. Help her however you can, even if that means cleaning with her. Speaking of which... -------------------------------------------------------------------------------- ## Thu, Jun 02, 2016 2:30:54 PM ### Jupyter Notebook (IPython) now an everyday thing on Windows for me Okay, this tmux full-screen thing is working pretty well now on Windows, Mac and of course Linux, so my command-line experience across platforms is getting pretty unified. I'll also try sticking to the tmux defaults (except for activating mouse support) for now, like I have with vim and plugins. Better to be operational and comfortable on 90% of the machines I sit down at, rather than 99% operational and comfortable on 10% of the machines I sit down at. Keep my dev-system build as close to baseline as not masochistically possible. Okay, it's time now to competitively look at sites. Think through my tools. Hmmm. I still prefer working on Windows for native OS speed, and officially office-sanctioned work machine reasons. And I want to be able to do Python stuff from my desktop as part of my investigation, without having to struggle with compiling dependency issues (even an issue with the pip installer!). Okay... Anaconda. Once that's installed, it's just a matter of typing: jupyter notebook Okay, I have Python 2.7.10 under Cygwin, so I'll go with Python 3.5 for Anaconda, which is recommended anyway. I like what IPython Notebook lets you do in terms of documenting and isolating parts of an investigation. Hmmmm, I should really consider making the next version of Pipulate work as a really flexible module to import under IPython. Okay, after Anaconda is installed, you can start IPython by just typying jupyter notebook into a Windows command console, but you have to be careful about what directory you start it from. It's actually possible to start it from a location where you don't have write permissions, and you'll get the error: Permission denied: Untitled.ipynb ...when you try to create a new Python 3 file. Okay, I not only fixed that, but I started it from a Cygwin MinTTY shell, so I can now interact with the files here in tmux easily, and use my typical Cygwin ~/ home location, where I keep my git repos too. Nice. It's easy to look there, and drop files in that location. This will make ad hoc Python investigations on the Windows side very easy, and visual and interactive. These IPython notebooks are much more fun to work with than Python's native interactive mode. Hmmm, what's that word? Oh yeah, REPL for read–eval–print loop -- a.k.a. interactive toplevel or lanugage shell. Makes sense. Hope that toplevel thing isn't a Wikipedia billicat. But then again, so much is. 1, 2, 3... 1? Already installed Anaconda and got IPython working. Now, do the thing that I originally them for -- a quick use of the Python sets. I should be able to just slam data around -- say for example, from one column in a massive Excel sheet into a text file via vim, and then use IPython to do Python manipulations against that textfile data. This is NOT rabbit hole stuff -- although, I thought for a moment that it might actually be. I COULD actually dedupe a column in location to get a count of uniques. But then I can also copy-paste the column out of excel into a text file, and use this Python: myset = set() with open("words.txt") as words: for word in words: myset.add(word) len(myset) Hmmm, I wonder if I can set the indents in IPython to 2 space. I don't know why everyone is so fixed on 4 spaces. Yeah, I know Pep 8, but a foolish consistency, and all that. Two spaces forever! -------------------------------------------------------------------------------- ## Thu, Jun 02, 2016 1:58:19 PM ### tmux Path and Under Cygwin I had installed tmux through the cygwin installer, but just typing tmux did not launch it. Paths! Ah, add to the book outline. I have to get find, grep and paths down once and for all. $ find . -name "tmux*" ./install/bin/tmux.exe ./install/etc/setup/tmux.lst.gz ./install/http%3a%2f%2fcygwin.mirror.constant.com%2f/x86_64/release/tmux ./install/http%3a%2f%2fcygwin.mirror.constant.com%2f/x86_64/release/tmux/tmux-2.1-1.tar.xz ./install/http%3a%2f%2fcygwin.mirror.constant.com%2f/x86_64/release/tmux/tmux-2.2-1.tar.xz ./install/usr/share/doc/tmux ./install/usr/share/man/man1/tmux.1.gz ./install/usr/share/terminfo/74/tmux ./install/usr/share/terminfo/74/tmux-256color ./usr/share/terminfo/74/tmux ./usr/share/terminfo/74/tmux-256color (tmux.exe found / Ctrl+c'd out of find) $ ./install/bin/tmux.exe [exited] $ export PATH=$PATH:"/install/bin/" $ tmux [exited] $ vim ~/.bashrc (added export command to .bashrc) This touches on so many things. When things appear broken, suspect paths. Always suspect paths. The essential trick of environment variables between Windows and Unix machines are very similar. In particular, the command: export PATH=$PATH:"/some/location/" ...and the fact that this only lasts during the currently active terminal's duration, and how if you want to make it permanent, you have to commit it somewhere else. In the case of nix-OSes, it's just in a .bashrc in your home directory. Under Windows, it's some Right-click on Computer/Properties, Advance System Settings / Environment Variables / knowing what-the-hell to do there convolution. And God knows how it changes from Windows version to Windows version. Unix wins again. Oh sheesh, environment variable mapping under the impending Windows 10 Ubuntu Bash Shell is going to be a nightmare. -------------------------------------------------------------------------------- ## Thu, Jun 02, 2016 11:37:43 AM ### Debugged cron.hourly (shebang directive needed) Look at that to-do list I just put (once again) immediately above the most current journal entry. This is a convention I should really stick to, and update the (sterilized) to-do list accordingly. Okay, let's start with: run-parts /etc/cron.daily Wow, just switched my daily journal terminal over to my VirtualBox running Kubuntu. Nice. The markdown color-coding is much nicer. I have to check my vim color coding preferences. I think I override the default for Python here or there, but maybe it's time to take the defaults again. Must look into. At any rate, let's understand what's going on with cron on Kubuntu (and by extension, Ubuntu and probably Debian too). Hmmmm. Look at /etc/crontab first. # /etc/crontab: system-wide crontab # Unlike any other crontab you don't have to run the `crontab' # command to install the new version when you edit this file # and files in /etc/cron.d. These files also have username fields, # that none of the other crontabs do. SHELL=/bin/sh PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin # m h dom mon dow user command 17 * * * * root cd / && run-parts --report /etc/cron.hourly 00 3 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) 47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly ) 52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly ) # /etc/crontab (END) Okay, that's telling. First, I figured out how to copy text from a tmux panel. Just as quickly as I see that's an issue, I google-up an answer, and in some ways, it's **better** than copying-and-pasting from non-tmux terminals, because Ctrl+b, Enter actually removes ambiguity about modifier keys that need to be pressed to keep a vim-highlight separate from an OS-highlight, which you can see very clearly with the orange highlighting. Also, I discovered the panel shuffling magic of Ctrl+B, Spacebar. The discoveries here is that: - All scheduled scripts run as root -- useful for testing. - anacron is actually doing the work, and not cron This URL has some suggestions: http://stackoverflow.com/questions/4984725/how-to-test-cron-job username@hostname:/etc/cron.daily$ sudo -u root -i [sudo] password for username: root@hostname:~# run-parts -v /etc/cron.daily run-parts: executing /etc/cron.daily/0anacron run-parts: executing /etc/cron.daily/apport run-parts: executing /etc/cron.daily/apt-compat run-parts: executing /etc/cron.daily/bsdmainutils run-parts: executing /etc/cron.daily/cracklib-runtime run-parts: executing /etc/cron.daily/dpkg run-parts: executing /etc/cron.daily/google-talkplugin run-parts: executing /etc/cron.daily/logrotate run-parts: executing /etc/cron.daily/man-db run-parts: executing /etc/cron.daily/mlocate run-parts: executing /etc/cron.daily/passwd run-parts: executing /etc/cron.daily/popularity-contest run-parts: executing /etc/cron.daily/reports run-parts: failed to exec /etc/cron.daily/reports: Exec format error run-parts: /etc/cron.daily/reports exited with return code 1 run-parts: executing /etc/cron.daily/sysstat run-parts: executing /etc/cron.daily/update-notifier-common root@hostname:~# And there's the error. Oops... shebang! Needed #!/bin/sh at the top of the file, and that is all. Sheesh! Okay, don't forget that one again. When doing cron: 1. No file-extensions! 2. Absolute paths 3. Ability to run as root 4. Bash files must have shebang directives 5. Execution bit chmod +x filename must be set Okay, time to switch back to my Windows 7 desktop. I think I'll get a tmux window running in a Cygwin terminal over there. I can't get too much tmux practice now. -------------------------------------------------------------------------------- ## Thu, Jun 02, 2016 10:48:24 AM ### Making That Book-writing Part Of My Daily Workflow I watched the Alan Turing documentary on Netflix (finally) last night, and wow. If I thought my work at Scala was thankless, what the UK did to Turing after WWII was epic and criminal and full of overtones of becoming like one's own enemy -- the UK becoming intolerant of different lifestyles to the extent of chemically castrating and sterilizing their own citizens. There's no more reason for you to sit here and read this than there is for you to sit and listen to the random thoughts going through the head of another person, who is not deliberately organizing and preparing any of those thoughts for public consumption. It's really just a lot of white noise, with the occasional blip of meaning... which I try to capture and then organize into location. That last bit of ultimate organizing... the producing the REAL product... THAT is the trick, and THAT is one of the last dots remaining to be connected. And THAT is what I need to work into my day-to-day work. One thing's for sure now, and that is that THIS journal is a constant. There's much more even here in THIS journal than what appears here on the page, which I blanked during my Puerto Rico trip when I over-shared personal stuff and blanked it. And before that, I kept paper journals (since 1988). And somewhere in there, I kept a Webmaster Journal at Scala which I still have in text format somewhere, which I will probably add to this -- maybe inline, and maybe as a linked-to leaf -- not sure yet. But continuity is key, and continuity is there. But continuity is not enough on its own. Continuity has to couple with thoughtfulness, insights, and organization with purpose. THAT is the "extruding" the book concept that I developed earlier. I must actually get about actually extruding that book. Focus my message. Get more things working for me 24x7, per my semi-automation majordomo sorcerers apprentice intelligent agents doing your bidding messaging that I seem to be dancing around. It's time for a new definition of legacy computing. -------------------------------------------------------------------------------- ## Thu, Jun 02, 2016 10:23:51 AM ### Switching Host-OS Virtual Desktops While Running VirtualBox Full-Screen Okay, one of my big breakthroughs was a very little one. A mere tap on the right-control-key from within VirtualBox momentarily stops the full-screen virtual session from capturing the keyboard, so a Ctrl+Alt+Arrow key will at that moment actually work, and bring me back to the host environment with a VirtuaWin-enabled virtual screen switch to a Windows Host screen. Pshwew! Maybe I'll avoid carpal tunnel syndrome after all. The keyboard shortcut I was doing before that was super-convoluted, and various Googling sessions and trying things from the VBox menus was fruitless, until finally today when someone suggested pressing the Shift BEFORE the right-Ctrl key... at which time I tried JUST the right-Ctrl key... and it worked. Sigh, go figure. Anyway, I'm not that deep into it, and I'll still be able to reap the benefits of VirtuaWin screen-switching with VirtualBox... maybe not seamlessly, but not too bad either. Okay, that's more delaying tactics before ACTUALLY getting to work. Now, think! - Debug why /etc/cron.daily didn't run last night - Answer the question regarding what this other website is doing. - Investigate the Google Analytics feature now in beta that gives you Webmaster Tools (Search Console) data. - Prepare a function to discover homepage URLs given just the name of an organization, and specific subject-matter pages within the site. - Investigate approaches to getting Darryl's Excel-based tools into a fast Excel-like Web UI -------------------------------------------------------------------------------- ## Thu, Jun 02, 2016 9:51:57 AM ### On Petty Thoughts It's always interesting when something becomes the biggest thing in someone's world for someone you know, and to you, it's something you can hardly even bother yourself to think about. I put everything now in the perspective of helping to raise Adi. Petty is easier than ever to see as petty. In addition to raising Adi, there's ensuring I have a reliable (and NYC-sized) income to let her continue to live (at least, part-time) in the Manhattan home she grew up in. That would be nice to keep, and thinking-ahead, this property is just going to become more and more valuable. Sure, there's always maintenance, but if it weren't a co-op, there'd be real estate tax. Many an estate home has been sold because they couldn't pay the taxes on the property. I wouldn't be the first, if I turn out wanting to save a little bit of money. But that sort of compromise is for the weak, who are unable to maneuver into better positions, increase their income, and all-around find creative solutions allowing one to live the life they want to live. One question is actually whether I'm living the life I want to live, or whether I haven't actually been artificially "locked into" a native New Yorker lifestyle by having a New York City native kid. And I'm not talking about an outer borough. Adi is a Manhattan kid, and I'm doing what my father failed to do -- actually LIVE IN New York City. The topic came up with him more than once over the years, I recall, and the answer usually was something along the lines of us not being able to live the sort of life that we're living. He didn't earn enough to live in the city, or probably even West Chester. He bought into a brand-new development -- the Philly sprawl-bergs. Okay, well then. That's enough introspection for the morning. The reports failed to run overnight, and it's time for me to debug /etc/cron.daily. No reason I can't completely understand why it didn't run and fix it. No reason to set up a separate physical server (yet) to get off my desktop virtual machine. I need to get myself together on multiple fronts. I'm reading Hackers: Heros of the Computer Revolution, and I'm getting to the part where they start talking about the MIT sour-puss price of continuously sustained and competitive zone-diving. They sorted everyone into winners and losers, and a lot of the loser behavior that was criticized by the MIT AI-lab crew seemed to be well-rounded life stuff. It's a very up-close and personal look at that special attribute that helps those with the potential to actually excel at what they do -- the nearly unlimited (exhaustion-bound) ability to focus on the problem at-hand. If I bring even 1/10th of the hacker ethic that they had the energy for at between 14 through 21 years old to bear on my own work at 45 years old, I will be able to "catch up" to where I should be, and give Adi a tremendous head-start in life -- in our modern life, and the new world that we're going into, which will be as different 20 years from now as today is from 20 years ago -- like from before when supercomputers fit in our pockets. Yep, I've got a good grip on what's important. Just don't ACTUALLY go broke, or get thrown in jail for anything, and I should be fine. -------------------------------------------------------------------------------- ## Wed, Jun 01, 2016 2:01:51 PM ### Breather and Contemplation Wow, was that a test. Frig! Okay, remember to be systematic and methodical. As soon as you get that panicky try-stuff-fast feeling, you're on the wrong track. The thing now is reproducibility and documentation. Do a pip freeze and make it part of the project. Wed, Jun 01, 2016 4:35:35 PM Pshwew! Just gave my counterpart in Tech access to the private repos, the credential files, and a demo of it working. Then, I went to one of the properties and did an update meeting, including showing the reports. Hmmmm. Okay, think through next steps. I have a late Triweekly meeting coming up in a half-hour. The most important thing I can do is to make sure that my virtual box REALLY triggers off on a scheduled basis, beginning tonight. Why did cron.daily not work on the last go-around? What are my other options? I have to start rising to that level that I believe myself to be at in my head. But also remember that behind every lightning bruiser is a colossal mishap just waiting to happen. Moving a lot of force around quickly has dangers. But I don't think I'm even looking all that quick to my peers. THAT needs to be fixed. I need to refine my stupid decisions like NOT using either AWS/EC2/boto for everything non-personal-hardware that I do. Why mess around with Rackspace or Wable? How do those REALLY help you in the future? Professional skills! Your biggest experience to have with OpenStack at this point is to build your own personal cloud. Wed, Jun 01, 2016 6:19:32 PM Had a happy hour. Checked over my cron.daily job. Everything tests good, from its permissions to its run-parts --test /etc/cron.daily check. It has identical permissions to everything else in that folder (which I presume runs) and can execute when called directly and has absolute paths to Python and the script. It doesn't have a dot-extension. So, I think it must just be laptop sleeping and VM issues. I think I'll bring in my CuBox server tomorrow. -------------------------------------------------------------------------------- ## Wed, Jun 01, 2016 9:52:00 AM ### Switching Amazon AWS Free Tier AMI Instance Into Ubuntu Instance Okay, give a shot at working with the AMI instance you set up yesterday. You already have Dropbox working on it, and are not so far into it that you've fubar'd everything. Okay, work your way systematically and methodically through this. Check that Dropbox is still working. I put a file on the Dropbox location from my PC-side and am watching for it to appear from the AWS EC2 side. It's not instantaneous. Maybe due to t2.micro status -- or maybe due to the Dropbox server software and checking-cycle. Got to investigate if I'm going to rely on it. But sending archives via email would also have some delay -- oh, unless they were both sent and check from the SAME gmail account (for instance). Keep that in mind if Dropbox doesn't work out. Oh, it wasn't started! I had done a restart. I had to do this (in addition to just a normal start for this session). python dropbox.py autostart y Okay, you have to view yourself in a race. Get through this friggin thing. Don't get frustrated. Get done! Okay, get rid of the TWO Security Group contexts. Make it just one. Done. Check if posgres is running: /etc/init.d/postgresql status postmaster dead but pid file exists Oops, but not a valid check if I do that as my default login. I have to do it either AS postgres with pg_ctl or as root with ps aux. Yep, I see it running as root. Ugh! There's like already a bunch of Postgres VERSIONS installed on this AMI. Okay, I'm making a decision to speed all of this up. As appealing as going with an AMI instance may have been, I'm more familiar with Debian and Ubuntu, and I am comfortable with installing packages and building it up, instead of getting the wrong already-installed version used accidentally! Ugh. I'll lose the dropbox install/config work, but who cares? Go faster! Okay, I can't change an ami instance to an ubuntu one, but I can create a new Ubuntu one and delete the Ami one. The language is "terminating" an instance on Elastic Beanstalk... okay, done. Now I have a fresh Ubuntu instance and a new pem file. I also have two security groups again. Okay, just go through this systematically... 1, 2, 3... 1? Login. - Okay, the .pem and chmod stuff. Login via ssh for the first time. Done. - sudo apt-get update... done. - sudo apt-get upgrade... done. Okay, just like with the Ami instance, see what's installed by default on the AWS Ubuntu: dpkg --get-selections | grep -v deinstall accountsservice install (deleted a ton of stuff 8/4/2016) zlib1g:amd64 install Okay, Mostly Python 2 & 3 and libraries. Not bad. No PostgreSQL, so that's totally in my hands now and closer to my Kubuntu machine, and I can deal with that. 1, 2, 3... 1? Be systematic! cd ~ && wget -O - "https://www.dropbox.com/download?plat=lnx.x86_64" | tar xzf - ~/.dropbox-dist/dropboxd python dropbox.py autostart y I can worry about excluding directories later. Plow through. This is a fresh system now with a working Dropbox. Find THE BEST instructions now on how to get PostgreSQL installed on Ubuntu 14.04. I believe this is going to be the best instructions: - https://help.ubuntu.com/lts/serverguide/postgresql.html - https://www.postgresql.org/download/linux/ubuntu/ sudo apt-get install postgresql postgresql-server-dev-all Okay, done. Before doing the other extension stuff for the foreign table exposer, go through everything necessary to confirm Internet connectivity. ALLOW INTERNET REQUESTS sudo vim /etc/postgresql/9.3/main/postgresql.conf CONNECT TO TEMPLATE1 DB AS POSTGRES USER sudo -u postgres psql template1 ALTER POSTGRES USER PASSWORD ALTER USER postgres with encrypted password 'your_password'; EDIT pg_hba.conf TO ALLOW CONNECTIONS become so big, and your IaaS experience is gold. You WANT to have to be the sysadmin and devops person. That's a big part of what you're in it for. Use the boto package against a single free-tier AMI instance with the ability to promote yourself to superuser privileges on those occasions when you need to. And this is the AWS experience that... well, falls into the same category of all these other highly valuable skills that half half-a-foot in the proprietary world. Python APIs make the occasional dabbling in proprietary okay, because how hard will it really be to adapt from Python boto deployment scripts to Python OpenStack scripts? Oh, speaking of deployment, do a little quick investigation. How best to go about this? Okay, it's a General Purpose t2.micro (as opposed to a smaller nano and a larger small). Okay, I selected Review and Launch, and am presented by this warning: > Improve your instances' security. Your security group, launch-wizard-1, is > open to the world. Your instances may be accessible from any IP address. > We recommend that you update your security group rules to allow access from > known IP addresses only. You can also open additional ports in your > security group to facilitate access to the application or service you're > running, e.g., HTTP (80) for web servers. It is almost certainly security context like that that's cutting me off from work's Redshift connection from any machines not on the internal network. Makes sense, and is a sensible precaution. Okay, now I'm creating a new key pair. Moved my .pem file into my Cygwin .ssh location. Okay, and I had to chmod 400 it. And once logged in (which went smoothly), I was prompted to run: sudo yum update Okay, what's installed on this thing? Per http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/find-software.html sudo yum grouplist Loaded plugins: priorities, update-motd, upgrade-helper There is no installed groups file. Maybe run: yum groups mark convert (see man yum) Available Groups: Console internet tools DNS Name Server Development Libraries Development tools Editors FTP Server Java Development Legacy UNIX compatibility Mail Server MySQL Database MySQL Database client NFS file server Network Servers Networking Tools PHP Support Performance Tools Perl Support PostgreSQL Database client (version 8) PostgreSQL Database server (version 8) Scientific support System Tools TeX support Technical Writing Web Server Web Servlet Engine Done And a yum list installed: yum list installed Installed Packages Loaded plugins: priorities, update-motd, upgrade-helper Installed Packages acl.x86_64 2.2.49-6.11.amzn1 installed acpid.x86_64 1.0.10-2.1.6.amzn1 installed alsa-lib.x86_64 1.0.22-3.9.amzn1 installed at.x86_64 3.1.10-44.13.amzn1 installed attr.x86_64 2.4.46-12.10.amzn1 installed audit.x86_64 2.4.1-5.27.amzn1 installed audit-libs.x86_64 2.4.1-5.27.amzn1 installed authconfig.x86_64 6.2.8-9.27.amzn1 installed autogen-libopts.x86_64 5.18-5.8.amzn1 installed aws-amitools-ec2.noarch 1.5.7-1.0.amzn1 installed aws-apitools-as.noarch 1.0.61.6-1.0.amzn1 installed aws-apitools-common.noarch 1.1.0-1.9.amzn1 installed aws-apitools-ec2.noarch 1.7.3.0-1.0.amzn1 installed aws-apitools-elb.noarch 1.0.35.0-1.0.amzn1 installed aws-apitools-mon.noarch 1.0.20.0-1.0.amzn1 installed aws-cfn-bootstrap.noarch 1.4-11.6.amzn1 @amzn-updates aws-cli.noarch 1.10.8-1.37.amzn1 installed basesystem.noarch 10.0-4.9.amzn1 installed bash.x86_64 4.2.46-19.35.amzn1 installed bc.x86_64 1.06.95-1.10.amzn1 installed bind-libs.x86_64 32:9.8.2-0.37.rc1.45.amzn1 installed bind-utils.x86_64 32:9.8.2-0.37.rc1.45.amzn1 installed binutils.x86_64 2.23.52.0.1-55.65.amzn1 installed bzip2.x86_64 1.0.6-8.12.amzn1 installed bzip2-libs.x86_64 1.0.6-8.12.amzn1 installed ca-certificates.noarch 2015.2.6-65.0.1.15.amzn1 @amzn-updates checkpolicy.x86_64 2.1.10-1.9.amzn1 installed chkconfig.x86_64 1.3.49.3-2.14.amzn1 installed cloud-disk-utils.noarch 0.27-1.5.amzn1 installed cloud-init.noarch 0.7.6-2.11.amzn1 installed coreutils.x86_64 8.22-15.52.amzn1 installed cpio.x86_64 2.10-12.12.amzn1 installed cracklib.x86_64 2.8.16-4.14.amzn1 installed cracklib-dicts.x86_64 2.8.16-4.14.amzn1 installed cronie.x86_64 1.4.4-12.6.amzn1 installed cronie-anacron.x86_64 1.4.4-12.6.amzn1 installed crontabs.noarch 1.10-33.9.amzn1 installed cryptsetup.x86_64 1.6.6-3.21.amzn1 installed cryptsetup-libs.x86_64 1.6.6-3.21.amzn1 installed curl.x86_64 7.40.0-8.57.amzn1 @amzn-updates cyrus-sasl.x86_64 2.1.23-13.16.amzn1 installed cyrus-sasl-lib.x86_64 2.1.23-13.16.amzn1 installed cyrus-sasl-plain.x86_64 2.1.23-13.16.amzn1 installed dash.x86_64 0.5.5.1-4.5.amzn1 installed db4.x86_64 4.7.25-18.11.amzn1 installed db4-utils.x86_64 4.7.25-18.11.amzn1 installed dbus.x86_64 1:1.6.12-8.27.amzn1 installed dbus-libs.x86_64 1:1.6.12-8.27.amzn1 installed dejavu-fonts-common.noarch 2.33-6.6.amzn1 installed dejavu-sans-fonts.noarch 2.33-6.6.amzn1 installed dejavu-serif-fonts.noarch 2.33-6.6.amzn1 installed device-mapper.x86_64 1.02.93-3.26.amzn1 installed device-mapper-event.x86_64 1.02.93-3.26.amzn1 installed device-mapper-event-libs.x86_64 1.02.93-3.26.amzn1 installed device-mapper-libs.x86_64 1.02.93-3.26.amzn1 installed device-mapper-persistent-data.x86_64 0.3.2-1.7.amzn1 installed dhclient.x86_64 12:4.1.1-43.P1.24.amzn1 installed dhcp-common.x86_64 12:4.1.1-43.P1.24.amzn1 installed diffutils.x86_64 3.3-4.15.amzn1 installed dmraid.x86_64 1.0.0.rc16-11.8.amzn1 installed dmraid-events.x86_64 1.0.0.rc16-11.8.amzn1 installed dracut.noarch 004-336.28.amzn1 installed dracut-modules-growroot.noarch 0.20-1.5.amzn1 installed dump.x86_64 1:0.4-0.6.b42.7.amzn1 installed e2fsprogs.x86_64 1.42.12-4.40.amzn1 installed e2fsprogs-libs.x86_64 1.42.12-4.40.amzn1 installed ec2-net-utils.noarch 0.4-1.25.amzn1 installed ec2-utils.noarch 0.4-1.25.amzn1 installed ed.x86_64 1.1-3.3.8.amzn1 installed elfutils-libelf.x86_64 0.163-3.18.amzn1 installed epel-release.noarch 6-8.9.amzn1 installed ethtool.x86_64 2:3.15-2.27.amzn1 installed expat.x86_64 2.1.0-8.18.amzn1 installed file.x86_64 5.22-4.31.amzn1 installed file-libs.x86_64 5.22-4.31.amzn1 installed filesystem.x86_64 2.4.30-3.8.amzn1 installed findutils.x86_64 1:4.4.2-6.9.amzn1 installed fipscheck.x86_64 1.3.1-3.13.amzn1 installed fipscheck-lib.x86_64 1.3.1-3.13.amzn1 installed fontconfig.x86_64 2.8.0-5.8.amzn1 installed fontpackages-filesystem.noarch 1.41-1.1.2.amzn1 installed freetype.x86_64 2.3.11-15.14.amzn1 installed gawk.x86_64 3.1.7-10.10.amzn1 installed gdbm.x86_64 1.8.0-36.6.amzn1 installed gdisk.x86_64 0.8.10-1.5.amzn1 installed generic-logos.noarch 17.0.0-2.5.amzn1 installed get_reference_source.noarch 1.2-0.4.amzn1 installed giflib.x86_64 4.1.6-3.1.6.amzn1 installed glib2.x86_64 2.36.3-5.18.amzn1 installed glibc.x86_64 2.17-106.167.amzn1 installed glibc-common.x86_64 2.17-106.167.amzn1 installed gmp.x86_64 6.0.0-11.16.amzn1 installed gnupg2.x86_64 2.0.28-1.30.amzn1 installed gpgme.x86_64 1.4.3-5.15.amzn1 installed gpm-libs.x86_64 1.20.6-12.8.amzn1 installed grep.x86_64 2.20-1.16.amzn1 installed groff.x86_64 1.22.2-8.11.amzn1 installed groff-base.x86_64 1.22.2-8.11.amzn1 installed grub.x86_64 1:0.97-94.30.amzn1 installed grubby.x86_64 7.0.15-5.7.amzn1 installed gzip.x86_64 1.5-8.18.amzn1 installed hesiod.x86_64 3.1.0-19.6.amzn1 installed hmaccalc.x86_64 0.9.12-1.9.amzn1 installed hwdata.noarch 0.233-14.1.18.amzn1 installed info.x86_64 5.1-4.10.amzn1 installed initscripts.x86_64 9.03.49-1.34.amzn1 installed iproute.x86_64 4.4.0-3.23.amzn1 installed iptables.x86_64 1.4.18-1.22.amzn1 installed iputils.x86_64 20121221-7.13.amzn1 @amzn-updates irqbalance.x86_64 2:1.0.8-1.23.amzn1 installed java-1.7.0-openjdk.x86_64 1:1.7.0.101-2.6.6.1.67.amzn1 installed javapackages-tools.noarch 0.9.1-1.5.amzn1 installed jpackage-utils.noarch 1.7.5-27.17.amzn1 installed kbd.x86_64 1.15-11.4.amzn1 installed kbd-misc.noarch 1.15-11.4.amzn1 installed kernel.x86_64 4.4.8-20.46.amzn1 installed kernel.x86_64 4.4.10-22.54.amzn1 @amzn-updates kernel-tools.x86_64 4.4.10-22.54.amzn1 @amzn-updates keyutils.x86_64 1.5.8-3.12.amzn1 installed keyutils-libs.x86_64 1.5.8-3.12.amzn1 installed kmod.x86_64 14-10.10.amzn1 installed kmod-libs.x86_64 14-10.10.amzn1 installed kpartx.x86_64 0.4.9-72.8.amzn1 installed krb5-libs.x86_64 1.13.2-12.40.amzn1 installed lcms2.x86_64 2.5-4.4.amzn1 installed less.x86_64 436-13.12.amzn1 installed libICE.x86_64 1.0.6-1.4.amzn1 installed libSM.x86_64 1.2.1-2.6.amzn1 installed libX11.x86_64 1.6.0-2.2.12.amzn1 installed libX11-common.x86_64 1.6.0-2.2.12.amzn1 installed libXau.x86_64 1.0.6-4.9.amzn1 installed libXcomposite.x86_64 0.4.3-4.6.amzn1 installed libXext.x86_64 1.3.2-2.1.10.amzn1 installed libXfont.x86_64 1.4.5-5.12.amzn1 installed libXi.x86_64 1.7.2-2.2.9.amzn1 installed libXrender.x86_64 0.9.8-2.1.9.amzn1 installed libXtst.x86_64 1.2.2-2.1.9.amzn1 installed libacl.x86_64 2.2.49-6.11.amzn1 installed libaio.x86_64 0.3.109-12.8.amzn1 installed libassuan.x86_64 2.0.3-3.3.amzn1 installed libattr.x86_64 2.4.46-12.10.amzn1 installed libblkid.x86_64 2.23.2-22.26.amzn1 installed libcap.x86_64 2.16-5.5.8.amzn1 installed libcap-ng.x86_64 0.7.3-5.13.amzn1 installed libcgroup.x86_64 0.40.rc1-5.11.amzn1 installed libcom_err.x86_64 1.42.12-4.40.amzn1 installed libcurl.x86_64 7.40.0-8.57.amzn1 @amzn-updates libedit.x86_64 2.11-4.20080712cvs.1.6.amzn1 installed libevent.x86_64 2.0.18-1.11.amzn1 installed libffi.x86_64 3.0.13-11.4.amzn1 installed libfontenc.x86_64 1.0.5-2.6.amzn1 installed libgcc48.x86_64 4.8.3-9.109.amzn1 installed libgcrypt.x86_64 1.5.3-12.18.amzn1 installed libgpg-error.x86_64 1.11-1.12.amzn1 installed libgssglue.x86_64 0.1-11.7.amzn1 installed libicu.x86_64 50.1.2-11.12.amzn1 installed libidn.x86_64 1.18-2.8.amzn1 installed libjpeg-turbo.x86_64 1.2.90-5.14.amzn1 installed libmount.x86_64 2.23.2-22.26.amzn1 installed libnfsidmap.x86_64 0.25-11.10.amzn1 installed libnih.x86_64 1.0.1-7.8.amzn1 installed libnl.x86_64 1.1.4-2.10.amzn1 installed libpipeline.x86_64 1.2.3-3.3.amzn1 installed libpng.x86_64 2:1.2.49-2.14.amzn1 installed libpsl.x86_64 0.6.2-1.2.amzn1 installed libpwquality.x86_64 1.2.3-4.8.amzn1 installed libselinux.x86_64 2.1.10-3.22.amzn1 installed libselinux-utils.x86_64 2.1.10-3.22.amzn1 installed libsemanage.x86_64 2.1.6-3.13.amzn1 installed libsepol.x86_64 2.1.7-3.12.amzn1 installed libss.x86_64 1.42.12-4.40.amzn1 installed libssh2.x86_64 1.4.2-2.13.amzn1 installed libstdc++48.x86_64 4.8.3-9.109.amzn1 installed libsysfs.x86_64 2.1.0-7.10.amzn1 installed libtasn1.x86_64 2.3-6.6.amzn1 installed libtirpc.x86_64 0.2.4-0.3.13.amzn1 installed libudev.x86_64 173-4.13.amzn1 installed libuser.x86_64 0.60-7.23.amzn1 installed libutempter.x86_64 1.1.5-4.1.6.amzn1 installed libuuid.x86_64 2.23.2-22.26.amzn1 installed libverto.x86_64 0.2.5-4.9.amzn1 installed libxcb.x86_64 1.8.1-1.18.amzn1 installed libxml2.x86_64 2.9.1-6.2.50.amzn1 installed libxml2-python27.x86_64 2.9.1-6.2.50.amzn1 installed libxslt.x86_64 1.1.28-5.12.amzn1 installed libyaml.x86_64 0.1.6-6.7.amzn1 installed logrotate.x86_64 3.7.8-17.13.amzn1 installed lsof.x86_64 4.82-4.10.amzn1 installed lua.x86_64 5.1.4-4.1.9.amzn1 installed lvm2.x86_64 2.02.115-3.26.amzn1 installed lvm2-libs.x86_64 2.02.115-3.26.amzn1 installed mailcap.noarch 2.1.31-2.7.amzn1 installed make.x86_64 1:3.82-21.10.amzn1 installed man-db.x86_64 2.6.3-9.3.amzn1 installed man-pages.noarch 4.04-2.15.amzn1 installed mdadm.x86_64 3.2.6-7.32.amzn1 installed mingetty.x86_64 1.08-5.9.amzn1 installed nano.x86_64 2.5.3-1.19.amzn1 installed nc.x86_64 1.84-24.8.amzn1 installed ncurses.x86_64 5.7-3.20090208.13.amzn1 installed ncurses-base.x86_64 5.7-3.20090208.13.amzn1 installed ncurses-libs.x86_64 5.7-3.20090208.13.amzn1 installed net-tools.x86_64 1.60-110.10.amzn1 installed newt.x86_64 0.52.11-3.11.amzn1 installed newt-python27.x86_64 0.52.11-3.11.amzn1 installed nfs-utils.x86_64 1:1.3.0-0.21.amzn1 installed nspr.x86_64 4.11.0-1.37.amzn1 @amzn-updates nss.x86_64 3.21.0-9.76.amzn1 @amzn-updates nss-softokn.x86_64 3.16.2.3-14.2.38.amzn1 @amzn-updates nss-softokn-freebl.x86_64 3.16.2.3-14.2.38.amzn1 @amzn-updates nss-sysinit.x86_64 3.21.0-9.76.amzn1 @amzn-updates nss-tools.x86_64 3.21.0-9.76.amzn1 @amzn-updates nss-util.x86_64 3.21.0-2.2.50.amzn1 @amzn-updates ntp.x86_64 4.2.6p5-36.29.amzn1 installed ntpdate.x86_64 4.2.6p5-36.29.amzn1 installed ntsysv.x86_64 1.3.49.3-2.14.amzn1 installed numactl.x86_64 2.0.7-8.11.amzn1 installed openldap.x86_64 2.4.40-7.28.amzn1 installed openssh.x86_64 6.6.1p1-25.61.amzn1 installed openssh-clients.x86_64 6.6.1p1-25.61.amzn1 installed openssh-server.x86_64 6.6.1p1-25.61.amzn1 installed openssl.x86_64 1:1.0.1k-14.91.amzn1 installed p11-kit.x86_64 0.18.5-2.3.amzn1 installed p11-kit-trust.x86_64 0.18.5-2.3.amzn1 installed pam.x86_64 1.1.8-12.33.amzn1 installed pam_ccreds.x86_64 10-4.9.amzn1 installed pam_krb5.x86_64 2.3.11-9.12.amzn1 installed pam_passwdqc.x86_64 1.0.5-6.8.amzn1 installed parted.x86_64 2.1-21.18.amzn1 installed passwd.x86_64 0.79-4.13.amzn1 installed pciutils.x86_64 3.1.10-4.11.amzn1 installed pciutils-libs.x86_64 3.1.10-4.11.amzn1 installed pcre.x86_64 8.21-7.7.amzn1 installed perl.x86_64 4:5.16.3-283.37.amzn1 installed perl-Carp.noarch 1.26-244.5.amzn1 installed perl-Digest.noarch 1.17-245.5.amzn1 installed perl-Digest-HMAC.noarch 1.03-5.7.amzn1 installed perl-Digest-MD5.x86_64 2.52-3.5.amzn1 installed perl-Digest-SHA.x86_64 1:5.85-3.5.amzn1 installed perl-Encode.x86_64 2.51-7.5.amzn1 installed perl-Exporter.noarch 5.