Categories: sumana | Programming
Software development, system administration, and similar
# 11 Mar 2018, 10:42AM: Recent Debugging And Confidence:
I am proud of myself for some recent debugging I've done on and with codebases and tools that I hadn't worked on before.
A few weeks ago, I was sitting next to a friend who co-maintains a web app and hadn't looked at it for a while. The styling was screwy. I asked whether some CSS or JS he depended on had upgraded, like jQuery or something. He said no, his site hosted all its dependencies. I opened up the site and checked the Network tab in Firefox Developer Tools and saw that it pulled in Bootstrap from a CDN. Ah, one of the other maintainers had added that! And updates to Bootstrap had screwed up the page's styling.
That same day, as a freshly minted co-maintainer of twine (a utility to upload packages to PyPI), I investigated a problem with our CHANGELOG. Twine's changelog, as represented on Read The Docs (example) and when I built the docs locally, only displayed version number 1.4.0 (2014-12-12) and two associated GitHub issues. This was inaccurate since the source file changelog.rst had 70+ items and ran up to version 1.9.1 (2017-05-27). I figured out that this was happening because changelog.rst is meant to be formatted so the Sphinx extension releases (which I hadn't used before) can parse it, and the current file wasn't syntactically (or semantically) adhering to releases's conventions. (Since then, with advice and help from some folks, I've released Twine 1.10.0 and started a new maintainer checklist.)
And then, a couple days later, I fixed my friends' blog. Their front page had reverted to a ten-year-old index page. I had never touched Movable Type before and hadn't used their particular managed hosting web GUI before, but I poked around (and checked for backups before changing anything) and managed to figure it out: during a May 2008 outage, someone had hand-made an index.shtml page, which was now overriding the index.html page in the server config. I figured it out and found and fixed it.
My mom says that when I was a kid, I took apart alarm clocks and spare hose attachments and so on, and put them back together just fine. She once came upon me taking something apart, and when she drew breath to admonish me, I said, "Amma, if I don't take it apart, how do I know what is inside? Don't worry, amma, I'm just looking at it, I'll put it back together when I'm done," and I did. She told me that I took apart a mechanical alarm clock, carefully spreading all the parts out on some newspaper, and put it back together, and it didn't quite work properly, so I took it apart again and then put it back together, and it worked, and I jumped for joy and said "I fixed it!" (I still feel that way when I fix something.)
At some point along the way I feel like I lost that calm confidence in my abilities, that "things are made of stuff" and what one person made another can fix. But I have it again, now, at least for some bits of software, and some purely mechanical stuff (yesterday, helping friends move, deciding to break down a big empty cardboard box, responding to "but it's so big, it won't fit on the stack" with "we have knives"). It doesn't feel courageous at the time, just sensible, but then I look back and feel like a badass.
If I had to point to the single biggest cause of this regained confidence, I'd point to the Recurse Center, where I got way more comfortable with bravery and failure in programming.
# 06 Feb 2018, 09:48AM: The Ambition Taboo As Dark Matter:
PyCon just rejected my talk submission,* so I'll try to finish and post this draft that I've been tapping at for ages.
My current half-baked theory is that programmers who want any public recognition from our peers, recognition that meaningfully validates our personal mastery, basically have to do that through one of a few fora that therefore accrue less-spoken emotional freight. And two of those places are code review in open source projects** and proposal review in tech conference talk submissions, and the fact that we don't talk enough about the role of ambition when talking about these processes leads to unnecessary hurt feelings.
For context: We give talks for varied reasons. To teach, to make reusable documentation, to show off things we've made or things we know, to burnish our credentials and thus advance our careers, to serve our corporate brands' goals, to provide role models for underindexed folks from our demographics, to give a human face to a project and make it more approachable, it goes on.
A conference talk is a tool in a toolbox that has a lot of other tools in it. (The Recompiler, Linux Journal and LWN pay for articles, for instance.)
And conferences are more than lecture halls, of course -- they're networking opportunities, communities of practice, parties, vacations, sprints, and so on.
But when we talk about the particular pain or joy of having a talk accepted or rejected from a conference, there's an emotional valence here that isn't just about the usefulness of a talk or the community of a conference. We're talking about acceptance as a species of public professional recognition.
I've found it pretty useful to think about public professional recognition in the context of Dr. Anna Fels's book Necessary Dreams. She points out that the childhood or adolescent desire for fame is often a precursor to a more nuanced ambition, combining the urge to master some domain or skill with the desire for the recognition of one's peers or community. This influences how I think about awards, about job titles, and about encouraging technologists in the public interest, and about the job market's role in skill assessment.
So how can a programmer pursue public mastery validation? Here's what I see:***
- contributing to open source software (mastery validation: maintainers merging commits and thanking/crediting contributor for work)
- presenting at conferences (mastery validation: program committee accepting talk)
- posting comments to gamified platforms like Reddit, Hacker News, and Stack Overflow (mastery validation: upvotes and replies)
- publishing academic research (mastery validation: journal accepting paper, peers reviewing paper positively)
- writing books (mastery validation: publisher accepting & publishing book)
- starting and architecting technically challenging projects (mastery validation: skilled technologists cofounding with or working for you, or relying on or praising your work)
So, this stuff is fraught; let's not pretend it's not. And we get rejected sometimes by conferences and talk about it, try to take the perspective that we're collecting "no's", we remind others that even successful and frequent speakers get rejected a lot and you can choose not to give up. And we give each other tips on how to get better at proposing talks. And that's all useful. But there's also another level of advice I want to give, to repeat something I said last year:
I try not to say "don't get discouraged," because to me that sounds like telling someone not to cry or telling someone to calm down. It's a way of saying "stop feeling what you're feeling." Instead, I try to acknowledge that something is discouraging but also -- if the other person's ready to hear it -- that we can come back from that: your feelings are legitimate, and here are some ways to work with them.
Some advice I hear about bouncing back from a conference talk rejection involves formalizing, creating systems to use to get better at writing proposals (my own tips mostly fall into this category) -- after all, in programming, you can learn to make better and better things without directly interacting with or getting feedback from individuals. The code compiles, the unit tests pass. And that can be soothing, because you can get the feedback quickly and it's likely to be a flavor of fair. (But that computer rarely initiates the celebration, never empathizes with you about the specific hard thing you're doing or have just done, and rarely autocredentials you to do something else that has a real impact on others.)
To formalize and abstract something makes it in some ways safer; it's safer to say "I'm working to pass the [test]" or "I'm building a [hard thing] implementation" or "I'm submitting a talk to [conference]" than to say "I am working to gain the professional respect of my profession". But that is one motivation for people to submit talks to tech conferences and to feel good or bad about the talks they give.
So part of my advice to you is: go ahead and be honest with yourself about how you feel. Rejection can be hard, working to get an unaccountable gatekeeper's acceptance**** and failing to get public professional recognition in your chosen field is a cause of anxiety, and so on. Be honest about how discouraging that can feel, and why, and what you wanted that you didn't get.
And another part of my advice is that I will ask, like the annoying programmer I am: what problem are you trying to solve? Because there are probably a lot of ways there that don't involve this particular gatekeeper.
And the most annoyingly empowering part of my advice is: Humans created and run PyCon and TED and Foo Camp and all the other shiny prestigious things; you're a human and you could do so too. Especially if you acknowledge not just your own but others' ambition, and leverage it.
* Maybe we'll do it in an open space anyhow.
** Another blog post for another time!
*** I've left some things out here.
We have some awards, e.g., ACM Distinguished Member, that you might get if you work really hard for decades in certain fields. That feels too far away for the kind of thing I'm thinking about.
I've left out the possibility of being promoted at your job, because many technologists perceive engineering job promotions as not particularly correlating with the quality of one's work as a programmer, which means a promotion doesn't send a strong signal, understood by peers outside one's organization, of validation of programming mastery. Then again, if your organization is old enough or big enough, maybe the career ladder there does constitute a useful proxy for the mental models of the peers whose judgment you care about.
I've left out various certifications, diplomas and badges because I don't know of any that meaningfully signal validation of one's mastery as a programmer industry-wide. And there's a lot of stuff to parse out that I feel undecided about, e.g., I find it hard to distinguish the status symbol aspect of admission from the signal that the final credential sends. And: A lot of people in this industry find it impressive when someone has been admitted to certain postsecondary engineering programs, regardless of whether the person graduates. And: In my opinion, the Recurse Center is an experience that has an unfortunate and unintended reputation for gatekeeping on the basis of programming skill, such that a big subset of people who apply and are rejected experience this as an authoritative organization telling them that they are not good enough as programmers (and Google Summer of Code and Outreachy have a related problem).
Of course, go ahead, write your own blog post where you talk about how wrong I am about what I list or exclude, especially because I come from a particular corner of the tech industry and I'm sure there's stuff I don't perceive.
**** Some conferences' gatekeepers are more unaccountable than others'; regardless, the feeling from the rejectee's point of view is, I bet, mostly the same. And you can start your own conference or join the program committee of an existing conference to see what it's like from the other side of the desk and wield a bit of the power yourself.
# 02 Jan 2018, 10:46AM: 2017 Sumana In Review:
Four years ago, during my first batch at the Recurse Center, every day I'd write in a little notebook on the subway on my way home, jotting down a few bullet points about what I had learned that day. I found it helped in a variety of ways, and the keenest was that on bad days, reviewing my notes reminded me that I was in fact progressing and learning things.
On any given day in 2017 I often did not feel very happy with my progress and achievements and how I was using my time. I fell ill a lot and I was heartsick at the national political scene and current events. It is genuinely surprising to me to look back and take stock of how it all added up.
I went hiking in Staten Island and in the Hudson Valley. I got back on my bike and had some long rides, including on a canal towpath in New Jersey and over the Queensboro bridge. (And had my first accident -- a car in my neighborhood rear-ending me at a traffic light -- and thankfully escaped without damage or injury.) I learned how to bake bread. I got to meet Ellen Ullman OMG. And I tried to travel less than I had in previous years, but I still had some fine times in other places -- notably, I had a great time in Cleveland, I witnessed the total solar eclipse in Nashville, and I visited Charlotte, North Carolina (where, among other things, I visited the NASCAR Hall of Fame).
I did some of the same kinds of volunteering and activism that I'd done in previous years. For instance, I continued to co-organize MergeSort, participated in a fundraising telethon for The Recompiler telethon, signal-boosted a friend's research project to get more participants, and helped revitalize a book review community focusing on writers of color. Also, I served again as the auctioneer for the James Tiptree, Jr. Literary Award fundraising auction at WisCon, which is a particularly fun form of community service. The Tiptree Award encourages the exploration & expansion of gender. I wrote this year about what an award does, and the reflections I've seen from winners of the Tiptree Awards and Fellowships tell me those honors are doing the job -- encouraging creators and fans to expand how we imagine gender. This year I also deepened my commitment to the Tiptree Award by accepting the organization's invitation to join the Tiptree Motherboard; I am pleased to have helped the award through a donation matching campaign.
But the big change in my community service this year was that I tried to prioritize in-person political work. I called, emailed, and wrote postcards to various government officials. I participated in my local Democratic Club, including going door-to-door petitioning to get my local city councilmember onto the ballot for reelection.
And I found that I could usefully bring my technologist perspective to bear on the city and state levels, especially regarding transparency in government software. I spoke to my local councilmember about my concern regarding public access defibrillator data (the topic that led me to file my first-ever Freedom of Information Law requests, for government health department records) and this inspired him to sponsor a bill on that topic. (Which is now filed as end-of-session partly because of the limbo in potentially getting PAD data from NYC's open data portal -- I need to send an email or two.) I was invited to speak to a joint committee of the New York State Assembly on the software side of our forensics labs, and got particularly interested in this aspect of due process in our criminal justice system, publicizing the issue in my MetaFilter posts "'maybe we should throw an exception here??'" and "California v. Johnson". I testified before the Committee on Technology of the New York City Council on amendments to our open data law (I didn't prep my public comment, so this text is reconstructed from memory; video), and then spoke before the same committee on an algorithmic accountability measure (and publicized the bill, especially keeping the Recurse Center community apprised as best I could). And I did research and outreach to help ensure that a state legislature hearing on protecting the integrity of our elections included a few researchers and activists it wouldn't have otherwise.
In 2018 I want to continue on this path. I think I'm, if not making a difference, making headway towards a future where I can make a difference.
This was by far Changeset Consulting's busiest year.
I had a mix of big projects and smaller engagements. First, some of the latter: I advised PokitDok on developer engagement, with help from Heidi Waterhouse. For Open Tech Strategies, I wrote an installation audit for StreetCRM. And, working with CourageIT, I came in as a part-time project manager on a government health IT open source project so the lead developer could focus more on architecture, code, and product management.
Some larger and longer projects:
Following a sprint with OpenNews in December 2016 to help write a guide to newsrooms who want to open source their code, I worked with Frances Hocutt to create a language-agnostic, general-purpose linter tool to accompany that guide. "The Open Project Linter is an automated checklist that new (or experienced but forgetful) open source maintainers can use to make sure that they're using good practices in their documentation, code, and project resources."
I spent much of the first half of 2017 contracting with Kandra Labs to grow the Zulip community, helping plan and run the PyCon sprint and co-staffing our PyCon and OSCON booths, running English tutoring sessions alongside Google Summer of Code application prep, and mentoring an Outreachy intern, along with the usual bug triage, documentation updates, and so on. We wrapped up my work as Zulip's now such a thriving community that my help isn't as needed!
From late 2016 into 2017, I've continued to improve infrastructure and documentation for a Provider Screening Module that US states will be able to use to administer Medicaid better (the project which spurred this post about learning to get around in Java).
And just in the last few months I started working on two exciting projects with organizations close to my heart. I'm thrilled to be improving HTTPS Everywhere's project workflow for developers & maintainers over the next few months, working with Kate Chapman via Cascadia Technical Mentorship (mailing list announcement). And, thanks to funding by Mozilla's open source grants program and via the Python Software Foundation, the Python Package Index -- basic Python community infrastructure -- is getting a long-awaited overhaul. I'm the lead project manager on that effort, and Laura Hampton is assisting me. (Python milestone: my first time commenting on a PEP!)
Along the way, I've gotten a little or a lot better at a lot of things: git, bash, LaTeX, Python (including packaging), Sphinx, Read the Docs, Pandoc, regular expressions, CSS, the Java ecosystem (especially Gradle, Javadocs, Drools), the Go ecosystem, Travis CI, GitHub Pages, Postgres, sed, npm Linux system administration accessibility standards, IRC bots, and invoicing.
Talks And Other Conferences:
This year, in retrospect, instead of doing technical talks and expository lectures of the type I'm already good at, I played with form.
At LibrePlanet 2017 I gave the closing keynote address, "Lessons, Myths, and Lenses: What I Wish I'd Known in 1998" (schedule, video, in-progress transcript). I tried something aleatoric and it worked pretty well.
At Penguicon 2017 I was one of several Guests of Honor, and spoke in several sessions including "Things I Wish I'd Known About Open Source in 1998" (which was different from the LibrePlanet version, as intended) and "What If Free and Open Source Software Were More Like Fandom?" (further links).
Then, at PyGotham, Jason Owen and I co-wrote and co-starred in a play about management and code review: "Code Review, Forwards and Back" (video on YouTube, video on PyVideo, commentary).
I also attended Maintainerati and led a session, attended !!Con, worked a booth for Zulip at OSCON, attended PyCon and helped run Zulip's sprint there, and co-sponsored a post-PyGotham dinner.
Other Interesting Things I Wrote:
I did not write this year for magazines; my writing went into this blog, MetaFilter, Dreamwidth, microblogging, and client projects, mostly. I also wrote an entry for a local business competition (I didn't make it very far but I'm glad I did it, especially the finance bits) and started two book proposals I would like to return to in 2018.
I've mentioned already some of the posts I'm happy about. Some others:
"On Noticing That Your Project Is Draining Your Soul" (every once in a while someone emails me and mentions that this has helped them, which means a lot)
"How to Teach & Include Volunteers who Write Poor Patches" (12 things you can do)
"Inclusive-Or: Hospitality in Bug Tracking", a response to Jillian C. York and Lindsey Kuper.
I turned part of "Some posts from the last year on inclusion" into "Distinguishing character assassination from accountability", a post about pile-on culture and callout culture where I pulled out quotes from 11 writers on how we take/charge each other with responsibility/power within communities.
I loved Jon Bois's 17776 and discussed it with other fans on MetaFilter, and then, to try to understand its amazingness better, wrote "Boisebration", collecting links to fiction and nonfiction by Bois about class, feminism, aging, sports, politics, wonder, education, & art (and 17776 precursors/callbacks).
I found out about Robert E. Kelly, like so many did, when his kids crashed his BBC interview, then collected some links in a MetaFilter post about his writing on Korea, US foreign policy, international relations, and academia.
I wrote up a bit about "1967's most annoying question for women in Catholic ministry" on MetaFilter to signal-boost another Recurser's cool project.
I enjoyed the learning and the plot twist in "The programmer experience: redundancy edition", in which I discovered a useful resource for Form 990 filings and learned to use the Arrow library for Python date-time manipulation. And was grateful to Pro Publica.
And I made a few jokes on social media I particularly liked:
yesterday, was trying to explain virtual environments/containers/VMs to a friend and said "they range from Inception-style fake computers to putting a blanket on the floor and pretending it's lava"
today a friend and I explained leftpad & Left Shark to someone and I began sketching out a hypothetical HuffPo piece connecting them
We habitually crowdsource infrastructure from, expect unsupportedly high levels of performance from unsuspecting participants -> popcorn.gif
Public notice I received:
I got some public attention in 2017 -- even beyond the Guest of Honor and keynote speaker honors and my amazing clients -- that I would like to list, as long as I'm taking an inventory of 2017.
I rode the first revenue ride of the new Q train extension in Manhattan and really loved the art at the new 72nd Street MTA stop. A journalist interviewed me about that on video and my experience got into the New York Times story about the opening.
Presenters at the code4lib conference said their project was specifically motivated by my code4lib 2014 keynote "User Experience is a
Social Justice Issue" (written version, video). I was honored and humbled.
And -- this is out of place but I need to record it -- as someone who knew Aaron Swartz, I consented to be interviewed by artists working on a play about him, and so someone briefly portrayed me (as in, pretended to be me and repeated my words aloud) in that play, Building a Real Boy.
Finally, Hari Kondabolu looked at the English Wikipedia page about him, much of which I contributed, and was amazed at how thorough it was. So that was awesome and I was proud.
I got on Mastodon as part of my effort to improve how I use social media. I started using a new task tracker. I got back on my bike, and got somewhat into a habit of using it for some exercise and intra-city travel. A new friend got me into taking more frequent photos and noticing the world I'm in. Two new friends caused me to look for more opportunities to see musicians I love perform live.
I consumed a fair bit of media this year; didn't get into new music but enjoyed music podcasts "I Only Listen To The Mountain Goats" and "Our Debut Album". I did some book and reading reviews and will catch up to other 2017 reading sometime vaguely soon.
Leonard's film roundups & TV spotlights are a good way to see or remember most of what I saw in the last few years. TV highlights for me for 2017 are The Good Place, Jane the Virgin, The Great British Baking Show (which led me to write a tiny Asimov fanfic), Steven Universe, and Better Call Saul; I also saw Comrade Detective and Yuri!!! On Ice. Films I'm really glad I saw: The Big Sick, Schindler's List, Get Out (I fanned in MetaFilter Fanfare), In Transit, A Man For All Seasons, Hidden Figures, and Lemonade -- and a rewatch of Antitrust.
I made a few new friends this year, most notably Jason Owen and Mike Pirnat. My friends Emily and Kris got married and I got to hold up part of the chuppah for them. I took care of some friends at hard times, like accompanying them to doctor's visits. I got to see some friends I rarely see, like Mel Chua and Zed Lopez and Zack Weinberg, and kept up some old friendships by phone. My marriage is better than ever.
This year I shall iterate forward, as we all do.
# 03 Dec 2017, 01:23PM: The Programmer Experience: Redundancy Edition:
Sunday morning, one thought led to another, and I realized I'd like to read a US nonprofit's Form 990 (annual report on assets, revenue, etc.). I started looking for 990s, and stumbled upon a big set of machine-readable 990s from 2011 through sometime in 2016. Is the one I seek in there? How recently was the dataset updated?
So I broke out bpython, the json module, and wget and started figuring it out.
Some things I learned/realized/remembered along the way:
- json.load() does pretty much exactly what I hoped it would even though I had forgotten the exact wording -- yay.
- Gar -- why can't datetime Just Do The Right Thing regarding turning very datestamp-looking strings into datetime objects? Aha, Arrow is my answer (Arrow : datetime :: requests : urllib2).
- Hey, if I've started up a virtualenv (being careful to create one whose Python is my installation of Python 3), how come I can't invoke bpython? Ohhhh, I have to pip install bpython inside the venv (or -- just to be safe -- I used pip3 install bpython which worked).
- I am fine with iteratively searching through a huge dataset in an incredibly unoptimized and cut-and-paste way at first, and if I get irritated by waiting then I can fix things up with refactoring and optimization and so on.
- According to the index of available filings, the most recent update reflected in those filings is from January 11, 2017.
- I'm one step closer to reflexively doing this kind of stuff in a Jupyter notebook and publishing it but I'm not there yet, despite encouragement from Julia Evans and how much easier the infrastructure's gotten. Next time, I think.
- I remembered that Bradley Kuhn would like for far more people to pay attention to 990 filings, and has co-maintained a repository of FLOSS foundations' filings for some time. I started searching around in case there was a more recent version of the repo* -- and found, via an actually relevant textual ad served by a search engine, ProPublica's Nonprofit Explorer.
Use this database to view summaries of 3 million tax returns from tax-exempt organizations and see financial details such as their executive compensation and revenue and expenses. You can browse IRS data released since 2013 and access over 9.6 million tax filing documents going back as far as 2001.
For example, attractively presented data, and full 990s, for the Organization for Transformative Works.
So, uh, turns out I didn't need to actually get all "make a new GitLab repo! how does Arrow work? gotta refactor this!" and so on, but ah well. Now I will have a fresh new project to use for testing as I delve into Python packaging and test Warehouse, the new Python Package Index.
* Update: Mike Linksvayer, on Identi.ca, edifies: "The archived gitorious floss-foundations filings repo is now actively maintained by Martin Michlmayr at https://gitlab.com/floss-foundations/npo-public-filings".
# 27 Jun 2017, 02:52PM: Learning To Get Around In Java:
One of Changeset Consulting's clients is working on modernizing a legacy web application; we're improving both its structural underpinnings and its user interface and outgoing APIs. It is like we are Chief O'Brien in the first season of Star Trek: Deep Space Nine, surveying and retrofitting Terok Nor. But that's not a fair comparison; O'Brien has to not only grapple with alien engineering approaches, but with the resentful and deliberate trashing the Cardassians inflicted on the station before handing it over. I haven't seen Stargate Atlantis but perhaps that's a better analogy; with every component of this long-asleep lost city that we resuscitate, a new console or room shimmers to life. Which is pretty rewarding!
The original authors wrote this application in Java. I've never worked on a Java application before, so the last few weeks have been quite an education in the Java ecosystem, in its tools and frameworks and libraries. We're improving the installation and deployment process, so now I'm more familiar with Ant, Gradle, Maven, OpenJDK, JDBC, Hibernate, and WildFly. I've gotten some API documentation in place, so now I know more about Spring and Javadoc.
As I was explaining to a friend this weekend, the overwhelming thing isn't Java as a language. It is a programming language and you can program in it, fine. The overwhelmption is the seemingly endless chain of plugins, platforms, and frameworks, and the mental work to understand what competes with, supersedes, integrates with, or depends on what.
Imagine you come to visit New York City for the first time, and wish to visit a specific address. First you need to work out where it is. But you do not have a map; there is no unified map of the whole place. Surely you can figure this out. Watch out: if someone doesn't tell you what borough an address is in, it's probably in Manhattan, but then again maybe not. There are multiple streets with the same name, and "31st Street and Broadway" in Queens is quite far from "31st Street and Broadway" in Manhattan. The avenue numbers go up westwards in Manhattan, eastwards in Brooklyn, and northwards in Queens. And so on.
You ask around, you see sketches of maps other people have made on their journeys, and eventually you feel pretty confident that you know the rough distance and direction to your destination. Now, how do you get there from your hotel room?
You probably don't want to walk all the way; for one thing, it's illegal and dangerous to walk on the freeways. This is why we have the subway (express and local), and buses (express and local, both privately and publicly run), and government-regulated taxis (street-hailable cabs and private car services), and bike rental, and commuter rail, a funicular/tram, car rental, ferries, and so on. Also there are illegal rideshare/taxi services that lots of people use. You try to learn some nouns and figure out what sort of thing each is, and what's a subset of what.
A MetroCard works on some of these modes and not others, and some transfers from one ride to another cost you nothing, and you can't use an unlimited-ride card twice at the same station or on the same bus within 18 minutes.* You can bring a bike on some MTA-run services but not all, not all the time. There are whole neighborhoods with no subway service, and whole neighborhoods with approximately no street parking. At rush hour the trains get super full. Service changes at night, on the weekend, and on holidays. Cars and buses get stuck behind accidents and parades. People and signs in Manhattan refer to "uptown" and "downtown" as though they are cardinal directions; they often correlate to "north" and "south" but not always. Metro North trains terminate at Grand Central, but Long Island Railroad trains terminate at New York Penn Station, which is named after Pennsylvania because it's where you can catch a train to Pennsylvania,** and there's a Newark Penn Station too but over a crackling loudspeaker those two station names sound very similar so watch out. And so on.
You're lucky; you find a set of cryptic directions, from your hotel to the destination address, based on a five-year-old transit schedule. It suggests you take a bus that does not exist anymore. Sometimes you see descriptions of travel that you think could be feasible as a leg of your journey, and you read what other people have done. They talk about "Penn Station" and "the train" without disambiguating, refer to the subway as "the MTA" even though the MTA also runs other transit, talk about "the 7" without distinguishing local from express, and use "blocks" as a measure of distance even though some blocks are ten times as long as others.
Aaaaagh. And yet: you will make it. You will figure it out. New Yorkers will help you along the way.
The decades-old Java ecosystem feels overwhelming but this application overhaul is like any other task. Things are made of stuff. Human programmers made this thing and human programmers can understand and manipulate it. I'm a human programmer. I made Javadoc do what I wanted it to do, and now the product is better and our users will have more information. And every triumph earns me a skill I can deploy for other customers and groups I care about.
* Just long enough for you to enjoy a little break from the podcast you're making with President Nixon!
** Also see St. Petersburg's Finland Station.
# 15 Oct 2016, 01:55PM: New Zine "Playing With Python: Two of My Favorite Lenses":
MergeSort, the feminist maker meetup I co-organize, had a table at Maker Faire earlier this month. Last year we'd given away (and taught people how to cut and fold) a few of my zines, and people enjoyed that. A week before Maker Faire this year, I was attempting to nap when I was struck with the conviction that I ought to make a Python zine to give out this year.
So I did! Below is Playing with Python: 2 of my favorite lenses. (As you can see from the photos of the drafting process, I thought about mentioning pdb, various cool libraries, and other great parts of the Python ecology, but narrowed my focus to bpython and python -i.)
Playing with Python
2 of my favorite lenses
[magnifying glass and eyeglass icons]
by Sumana Harihareswara
When I'm getting a Python program running for the 1st time, playing around & lightly sketching or prototyping to figure out what I want to do, I [heart]:
bpython & python -i
[illustrations: sketch of a house, outline of a house in dots]
bpython is an exploratory Python interpreter. It shows what you can do with an object:
>>> dogs = ["Fido", "Toto"]
append count extend index insert pop remove reverse sort
And, you can use Control-R to undo!
[illustrations: bpython logo, pointer to cursor after dogs.]
Use the -i flag when running a script, and when it finishes or crashes, you'll get an interactive Python session so you can inspect the state of your program at that moment!
$ python -i example.py
Traceback (most recent call last):
File "example.py", line 5, in
toprint = varname + "entries"
TypeError: unsupported operand type(s) for + : 'int' and 'str'
[illustration: pointer to type(varname) asking, "wanna make a guess?"]
More: "A Few Python Tips"
This zine made in honor of
NYC's feminist makerspace!
CC BY-SA 2016 Sumana Harihareswara
Everyone has something to teach;
everyone has something to learn.
Here's the directory that contains those thumbnails, plus a PDF to print out and turn into an eight-page booklet with one center cut and a bit of folding. That directory also contains a screenshot of the bpython logo with a grid overlaid, in case you ever want to hand-draw it. Hand-drawing the bpython logo was the hardest thing about making this zine (beating "fitting a sample error message into the width allotted" by a narrow margin).
Libby Horacek and Anne DeCusatis not only volunteered at the MergeSort table -- they also created zines right there and then! (Libby, Anne.) The software zine heritage of The Whole Earth Software Review, 2600, BubbleSort, Julia Evans, The Recompiler, et alia continues!
(I know about bpython and python -i because I learned about them at the Recurse Center. Want to become a better programmer? Join the Recurse Center!)
# 26 Sep 2016, 09:33AM: iCalendar Munging with Python 3, Requests, ics.py, and Beautiful Soup:
Leonard and I love seeing movies at the Museum of the Moving Image. Every few months we look at the calendar of upcoming films and decide what we'd possibly like to see together, and put it on our shared calendar so we remember. And for every showing (example) the MoMI provides an iCalendar (.ics) file, to help you add it to your electronic calendar. But it's a pain to individually download or refer to each event's .ics file and import it into my electronic calendar -- and the museum's .ics files' DTEND times are often misleading and imply that the event has a duration of 0 seconds. (I've asked them to fix it, and some of their calendar files have correct durations, but some still have DTEND at the same time as DTSTART.)
Saturday morning I had started individually messing with 30+ events, because the MoMI is doing a complete retrospective of Krzysztof Kieslowski's films and I am inwardly bouncing up and down with joyous anticipation about seeing Dekalog again. And then I thought: I bet I can automate some of this tedious labor!
So I did. The create-fixed-ics.py script (Python 3) takes a plain text file of URLs separated by newlines (see movie-urls-sample-file.txt for an example), downloads iCalendar files from the MoMI site, fixes their event end times, and creates a new unified .ics file ready for import into a calendar. Perhaps the messiest bit is how I use a set of regular expressions, and my observations of the customs of MoMI curators, to figure out the probable duration of the event.
Much thanks to the programming ecology that helped me build this, especially the people who made RegExr, Beautiful Soup (hi Leonard), Requests, ics.py, and the bpython interpreter, and the many who have written excellent documentation on Python's standard library. Thanks also to Christine Spang, whose "Email as Distributed Protocol Transport: How Meeting Invites Work and Ideas for the Future" talk at Open Source Bridge 2015 (video) introduced me to hacking with the iCalendar format.
- It can be a bit slow as the number of URLs adds up -- it took maybe 5 minutes to process about 31 events. I oughta profile it and speed it up. But I usually only need to do this about six times a year.
- This script is not careful, and will overwrite a previously created .ics file at the same address (in case you're running it twice in one day). It has no tests and approximately no error-checking. This was a scratch-my-own-itch, few-hours-on-a-Saturday project. No Maintenance Intended.
- Absolutely not an official project of the Museum of the Moving Image.
# (1) 27 Jul 2015, 02:00PM PST: Slides & Code from HTTP Can Do That?!:
My slides are up, as is demonstration code, from "HTTP Can Do That?!", my talk at Open Source Bridge last month. I am pleased to report that something like a hundred people crowded into the room to view that talk and that I've received lots of positive feedback about it. Thanks for help in preparing that talk, or inspiring it, to Leonard Richardson, Greg Hendershott, Zack Weinberg, the Recurse Center, Clay Hallock, Paul Tagliamonte, Julia Evans, Allison Kaptur, Amy Hanlon, and
Video is not yet up. Once the video recording is available, I'll probably get it transcribed and posted on the OSBridge session notes wiki page.
I've also taken this opportunity to update my talks and presentations page -- for instance, I've belatedly posted some rough facilitator's notes that I made when leading an Ada Initiative-created impostor syndrome training at AdaCamp Bangalore last year.
# 18 Jun 2015, 06:53AM: HTTP Can Do That?! and Comedy:
On Wednesday of next week (June 24th) I'm presenting "HTTP Can Do That?!" at Open Source Bridge in Portland, Oregon.
I have explored weird corners of HTTP -- malformed requests that try to trick a site admin into clicking spam links in 404 logs, an API that responds to POST but not GET, and more. In this talk I'll walk you through those (using Python, netcat, and other tools you might have lying around the house).
I practiced this talk Tuesday night at the Recurse Center and it went well; people learned a lot about headers, verbs, status codes, and odd HTTP loopholes, and gave me constructive criticism so next week's version will be clearer.
I have also suggested a Birds of a Feather evening session called "Nothing Is Totally Incomprehensible If We Try Together" but don't yet know whether or when it will happen.
Then, at AlterConf Portland on Saturday, June 27th, I'll be performing some stand-up comedy for hippie nerds. I thought about trying to cram 100 punchlines into my 45-minute HTTP talk, but I don't think I'll be able to achieve that -- people need to understand something before they can understand a joke about it -- so it'll be nice to get 4 or 5 laughs per minute during the stand-up on Saturday.
# (1) 24 Apr 2015, 03:26PM: Technothriller Book Review Partially In The Form Of A Python Exercise:
I am glad I read Hackster: The Revolution Begins..., a technothriller by Sankalp Kohli and Paritosh Yadav taking place in modern-day India. It's plotty and passionate and tense, and it's about Indians to whom India is the center of the universe. But it's also got major problems. Here are some quotes:
It was now time to attain answers. And he had found his answers in SNAGROM -- a device conceptualized by his father, but built and made operational by him with a few modifications to avenge the death of his patriotic father who had sacrificed his whole life for the progress of beloved country, India, only to be publicly humiliated and pronounced a terrorist with links to Pakistan's ISI by the ruling party of India, The Democratic Alliance Party. [p. 23]
Mr. Bedi, Vikram's father, was a scientist. He had the unique ability to solve problems by using concepts of one domain, into an altogether different one - something which most academicians couldn't do. His papers and theories on early meta-systems had brought a fresh perspective and direction into the scientific community. In his papers, he reduced the bigger problems into simple ones. He put it very simply, a meta-system is a system based on other systems. [p. 35]
Arjun could feel this guy getting to him.... he was not a person who took even the smaller defeats sportingly. For him defeat was accompanied by a splurge of vengeance. [p. 68]
"It seems like he had conceptualized a system that replicated the modern day concept of Big Data trackers and used it to come out with trends which were closer to reality." Vikram whispered to himself. [p. 78]
But, was it all because of one man? How could a single man cause so much havoc? It must have been 'the system'. [p. 111]
For ten years, he had used his peculiar ability to suppress all sorts of mutiny within the alliance with an ease that always surprised everyone around him. Nobody had ever seen him running across the country to meet the influential people in times of crisis. He would simply make a private phone call and follow up the next day. The matter would be resolved. [p. 152]
So I didn't love the prose or the characterization. And one plot thread in Hackster disproportionately bothered me.
In the scene below, two guys are investigating a break-in by Vikram, a super-elite hacker. Vikram broke into the Srinagar police department's "criminal database" to remove his friend Ashfaq's name from "the list of arms dealer with a pending investigation" (sic). Initially, police investigators had overlooked the incursion: "They termed it a routine hack failure." [p. 17-18] But this new anti-cybercrime unit digs deeper. For context, both authors of Hackster have MBAs, one "in the field of telecom technology," and in the Acknowledgement they thank someone for cybersecurity advice.
"He deleted one entry and then used a jumbler on all the others."
"After deleting the entry, he covered his track by jumbling up the names of all the people in the list. I tried running a point to point match between the shuffled copy of this list with an older correct copy, but none of the names matched. In short the whole list is corrupted, and we will not be able to make anything out of it easily. It is a long list. It has too many names. This guy is a genius." [p. 51-52]
But then Aarti, a top-shelf cybersecurity expert, succeeds at extracting the name "Ashfaq Ahmed Karim":
"He didn't know that entire data of servers of police department gets automatically stored in tape drives at the end of each month. These tape drives are detached from the servers and are stored in a secret location. I took out an older version of Illegal Arms Dealer List from the backup tape drives and then wrote a program to match each word of the older list with the newer one and rearranged the new list accordingly."
Sumit and Rao watched her with awe as she continued further, "Even the most advanced computer of ours took two days to complete this activity and give us this one name. This one lead should help us to take a step closer to our target." [p. 82]
My suspension of disbelief at this point broke so hard that it sent shards into nearby brick walls, where they remain, softly vibrating. I'm willing to set aside, for the sake of fiction, how badly guarded this data is, and why does Aarti have to go to the tape drive if there's an older version of the list more readily available, and why are they acting like this is a giant string rather than a set of rows in a table in a relational database and thus amenable to additional forensic techniques. Even so: this kind of puzzle is practically a junior programmer's intro-to-Python exercise. You could do this in bash; you could do it in Excel. And unless the Srinagar police department is tracking pending investigation against literally millions of arms dealers, a bog-standard developer's laptop could run that script in, mmm, 20 minutes.
Hmmmmmmmm, how long would it actually take? I decided to try to replicate this, without even trying very hard and while listening to a Taylor Swift album on repeat. I took the 417 names from the Nielsen Haydens' old blogroll, put them into a file separated by newlines (bloggers-archive.txt), and then removed one name, and saved the new file as bloggers.txt. Ah but now I want to obfuscate it! So I pulled all the names apart into their component words and shuffled them randomly and then wrote that back to a file (code: obfuscate.py). The new, jumbled list looks suitably forbidding:
My findmissingname.py script does not bother to "rearrange the new list accordingly" because what Aarti really wants is the missing name. findmissingname.py spits out the two words in the missing name, and it takes 0.04 seconds to do so on a ThinkPad. And I'm bone certain I could optimize performance further.
This points to an asymmetry I had not previously noticed regarding what will and will not break my suspension of disbelief. When I'm reading scifi or technothrillers, I am reasonably fine with magic zoom-enhance, encryption, robotics, and other implausible advances. I can deal with it if you have way cooler toys than exist in my world, if you tell me something hard for me is easy for you. But if you try to tell me that something easy for intermediate-skilled me is hard for hella competent world-class experts with best-of-breed gadgets, I laugh, because you're ridiculous.
I am married to a programmer whose code has literally been used to catch an illegal arms dealer. I highly doubt this repository is going to have a similar impact. But hey, I learned something new about my genre reading conventions and I practiced my Python 3.
# 19 Jan 2014, 04:51PM: Interesting-Looking Talks at PyCon 2014:
This year I'm going to visit PyCon! In fact, I'm presenting a poster: "What Hacker School Taught Me About Community Mentoring". You should register soon if you're coming, especially to take advantage of heavily subsidized childcare or to register for one of the tutorials.
Someone on one of my mailing lists asked what sessions people are particularly looking forward to. I tend to follow Skud's conference tips, which mean skipping sessions when I need to do self-care. But with such great-sounding talks, I may not be able to pull myself away!
- Allison Kaptur's "Import-ant decisions". Kaptur is a facilitator at Hacker School and I enjoy her thinking
process, areas of interest, and speaking style. I know I'll learn more about package management in general, and about Python specifically, from this talk.
- Jessica McKellar's "Building & breaking a Python Sandbox". McKellar did a residency in my Hacker School batch during which I got to see a preview of this talk, so I may not go again, but I found it thought-provoking; it helped me understand how Python works in a new way.
- Erik Rose's "Designing Poetic APIs". I met Erik Rose at a Google Summer of Code Mentor Summit and thought he was a really smart and fun guy. I've never designed an API before (thus I also want to go to the novice-focused "So You Want to Build an API?" by Megan Speir), and I like that his discussion will include anthroplogy, psychology, and history.
- Nathan Yergler's "In Depth PDB". I am one of the programmers the speaker sadly mentions; I rarely use it, if at all, even though it would be great to help me see when and why things are breaking!
- A localisation talk, maybe this one, because I think the Python application I'm working with right now just has hardcoded strings (aiee, bad).
- One of the SQLAlchemy talks, maybe this one, because I don't grok how to use SQLAlchemy yet. However I have registered for the SQLAlchemy in-depth tutorial so one may duplicate the other.
- Julia Evans's "Diving into Open Data with IPython Notebook & Pandas".
I enjoy Julia Evans's investigation process and her speaking and writing style. (See my post "Why Julia Evans's Blog Is So Great".) I have never used IPython Notebook nor matplotlib, numpy, pandas, or any of the other awesome science/data-related Python tools, and keep meaning to; this talk should help me with that.
- Greg Wilson's "Software Carpetry: Lessons Learned".
I am a tremendous fan of Wilson's work - Software Carpentry, the books he's edited, etc. The SC crowd has collected a lot of data (e.g. surveys of learners at their bootcamps) and I will probably want to soak in their lessons learned and shout about them to every other teach-y group I'm in.
- Naomi Ceder's "Farewell and Welcome Home: Python in Two Genders". I want to learn more about the experiences of trans women in my open source communities.
- Kate Heddleston, Nicole Zuckerman, presenting "Technical on-boarding, training, and mentoring". I do this task in my open source communities so I want to learn more best practices.
I'm thoroughly looking forward to my first PyCon. (I stopped by one for like an hour in 2003 and helped at the registration desk; I guess it took me eleven years to get to the other side of the desk!)
# (2) 22 Dec 2013, 10:42AM: Why Julia Evans's Blog Is So Great:
Some writing is persuasive; it aims to cause you to believe or do something. Some is expository; it aims to cause you to understand something. A lot of tech writing is persuasive or expository.
Some writing is narrative. It aims to cause you to feel or experience something. In personal narrative, the writer shares a personal experience and invites you to walk with her on that journey, experiencing it as she did, emerging with a new perspective. I really like narrative-style tech writing.
What I call the "Amazing Grace" story (previously) is, in a sense, all three of these. "Amazing grace! (how sweet the sound) / That sav'd a wretch like me! / I once was lost, but now am found, / Was blind, but now I see." Or, in more modern terms, "An English Sailor Found Salvation Through This One Weird Trick."
- Exposition: My experience started in sordid terror and ended in divine ecstasy
- Narration: Bask and wonder with me in the intricacy of my journey and the unexpected yet inevitable emergent properties of my condition
- Persuasion: Thus, if you are enthralled to sin, if you are a fallen resident of our fallen world, you should follow my example
I started thinking about this because my Hacker School colleague Julia Evans has a super-engaging blog. During our batch, she dove into operating system internals, and blogged about what she learned and how she learned it. She's consistently inspired me and made me laugh. Two of her fans (fellow HSers) even made a loving Markov-chain tribute, Ulia Ea.
One reason we love it is that most entries narrate her daily learning and illustrate a journey through confusion into wonder. See "Day 37: After 5 days, my OS doesn't crash when I press a key", which is possibly the most "Amazing Grace"-esque of her posts. Excerpt:
5. Press keys. Nothing happens. Hours pass. Realize interrupts are turned off and I need to turn them on....
It's not just the large-scale rhetorical structure; her diction and even her punctuation delight me. I particularly marvelled at her sentences in "Day 43: SOMETHING IS ERASING MY PROGRAM WHILE IT’S RUNNING (oh wait oops)". Excerpt:
12. THE OS IS STILL CRASHING WHEN I PRESS A KEY. This continues for 2 days....
As far as I can tell this is all totally normal and just how OS programming is. Or something. Hopefully by the end of the week I will get past "I can only receive one IRQ" and into "My interrupt handler is the bomb and I can totally write a keyboard driver now"....
I'm seriously amazed that operating systems exist and are available for free.
SURPRISE MY CODE IS NOT WORKING BECAUSE SOMETHING IS ERASING IT.
Can we talk about this?
- I have code
- I can compile my code
- Half of my binary gets overwritten with 0s at runtime. Why. What did I do to deserve this?
- No wonder the order I put the binary in matters.
It is a wonder that this code even runs, man. Man.
The disarmingly informal ALLCAPS adds to the intimacy more explicitly created with the question "Can we talk about this?" which invites the reader into one-on-one conversation. Moreover, I specifically call your attention to the statement "Why." and the repetition "man. Man." They demonstrate how Julia acknowledges mystery, with a tinge of disbelief.
As Patrick Nielsen Hayden observed,
A great deal of science fiction is about what the field's insiders often call "sense of wonder," a quality not entirely unrelated to the good old Romantic Sublime. Many of the genre's classics are in essence carefully-tuned machines designed to attract readers whose primary conscious loyalty is to rationalism, and lead them by a series of plausible contrivances to a sudden crescendo of mystical awe. This is an important part of SF from Olaf Stapledon to William Gibson and beyond.
And Julia Evans.
You can hire me through Changeset Consulting.
This work by Sumana Harihareswara is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.
Permissions beyond the scope of this license may be available by emailing the author at email@example.com.