Python announcements, programming and tools
# 16 Apr 2019, 02:36PM: PyCon NA, !!Con, and WisCon:
I'm back in New York City, and preparing to travel a bit; May is my big conference month this year.
I'll start the month at PyCon North America in Cleveland, Ohio, practically the whole conference, 1-9 May. I'm co-organizing The Art of Python, an arts festival -- several short plays, plus a fanvid and live music -- the night of Friday, May 3rd. And during the sprints, 6-9 May, I'll be concentrating on the world of Python packaging and distribution, e.g., PyPI.
I'll go home to New York City, then go to !!Con 11-12 May, where I am not organizing or speaking or suchlike.
And then I'll be at WisCon in Madison, Wisconsin, for the whole convention, 24-27 May plus a little extra on either side. I will once again be the comedy auctioneer for the auction on Saturday night that benefits the Tiptree Award -- if you have something to donate to the auction, please let us know by 15 May. I may not make it to the Floomp or the vid party. I will probably be on a few panels; several panels are still seeking volunteer panellists (sign up by 19 April). I do plan to be at the Gathering and the Dessert Salon (heads-up, changes to Dessert Salon entry flow).
If you're going to be at any of these events, perhaps we can share a beverage! If you want to make sure that we do that, let's actually set up at least a tentative appointment soon, so I can put it in my calendar.
I will be more responsive to emails and text messages than to social media while at these conferences, and in particular, I may see mentions and direct messages on Mastodon but I probably won't see mentions or direct messages on Twitter. Also, I am pretty forgiving about being called mispronunciations of my name, but here's a recording in case you want help -- I also respond to "Vikki" which is the fake name I use with strangers at restaurants. And I will probably not hug you unless we know each other well.
# 03 Oct 2018, 05:16PM: Tidelift Is Paying Maintainers And, Potentially, Fixing the Economics of an Industry:
As the founder of Changeset Consulting, I keep my eye on consultancies and services in and near my niche, open source leadership, maintainership, and sustainability.* And I've known Luis Villa for years and got to work with him at Wikimedia. So yeah, I noticed when Tidelift announced its big new launch. And -- now, as a very-part-time consultant who helps Tidelift understand the Python world -- I am excited about their commitment to pay more than USD$1 million to maintainers (including "a guaranteed minimum $10,000 over the next 24 months to select projects").
Here's my take on the new Tidelift subscription model, the "lifter" role, and whom this works for.
For software businesses, this provides that missing vendor relationship, SLA, release cadence expectations, and general peace of mind for all of that unseen infrastructure you depend on. It's often easier for businesses -- of many sizes -- to pay a regular fee than to put open source project management work, dependency-updating, compliance checking, dependency security audits, or FLOSS volunteer relations on the engineering schedule.
For individual programmers and community-maintained open source projects, Tidelift is a potential source of substantial income. As a Pythonist, I hope to reach people who are currently core code contributors to open source projects in Python, especially on the Libraries.io digital infrastructure/unseen infrastructure/improve the bus factor lists. And I would like to reach projects like the ones Nathaniel Smith calls out in a recent post:
that (1) require a modest but non-trivial amount of sustained, focused attention, and (2) have an impact that is large, but broad and diffuse
and projects in the "wide open", "specialty library", and "upstream dependency" categories identified by the Open Tech Strategies report "Open Source Archetypes: A Framework For Purposeful Open Source".
For such people and projects, becoming a lifter is a promising model -- especially since the required tasks are fairly few, and are things maintainers should do anyway. I'm encouraged to see Jeff Forcier (maintainer of Fabric, Alabaster, and more) and Ned Batchelder's coverage.py getting onto the Tidelift platform.
And you can see estimated monthly income for your package right now. For some people, especially those whose healthcare doesn't depend on an employer, Tidelift payments plus some side consulting could be a sustainable, comfortable income.
Then there are folks like me whose contributions are only partially visible in commit logs (management, user support, testing, and so on), and groups that work together best as a team. Tidelift is also a potential source of income for us, but it's a little more complicated. Tidelift can send lifter payments to individuals, for-profits, and nonprofits, but: "If a package has multiple co-maintainers, you'll need to agree as a group on an approach." If you thought code of conduct conversations with your community were uncomfortable, wait till you bring up money! But, more seriously: I've been able to talk frankly with open source colleagues about thorny "who gets paid what?" questions, and if you're candid with your co-maintainers, the benefits may be pretty substantial. You can get advice on this conversation during the next live Tidelift web-based Q&A, Thursday, Oct. 11 at 2 p.m. Eastern Time (sign up at the bottom of the lifter info page).
Nonprofits, companies, and working groups that maintain projects can sign up now as lifters. Even if it's just a trickle of money right now, it might build over time and turn into enough to fund travel for an in-person sprint, contract work to improve continuous integration, an Outreachy internship, etc.
(One gap here: right now, Tidelift isn't great at supporting system-level packages and projects, like tools that get installed via apt or yum/DNF. I'm pretty sure that's something they're working on.)
What about noncommercial users or users who can't afford Tidelift subscriptions? The more lifters and subscribers sign up, the more those users benefit, too. Subscribers' funding means maintainers have time to make improvements that help everyone. And lifters agree to follow security, maintenance, and licensing best practices that also help everyone. Plus, Tidelift stewards libraries.io, a great resource for anyone who uses or develops open source (more on that). More money for Tidelift could mean libraries.io gets better too.
So I'm tooting a horn here and hoping more people sign up, because this is one of the more plausible ways open source sustainability could possibly work. Tidelift could be a real game-changer for the industry. Check it out.
* Examples: new competitors like Maintainer Mountaineer and OpenTeam, new funders like OSS Capital, and colleagues/referrals like Open Tech Strategies, VM Brasseur, Otter Tech, and Authentic Engine.
# 29 Jul 2018, 03:02PM: "Python Grab Bag: A Set of Short Plays" Accepted for PyGotham 2018:
Fresh from the waitlist onto the schedule: Jason Owen and I will be performing "Python Grab Bag: A Set of Short Plays" at PyGotham in early October. If you want to come see us perform, you should probably register soon. I don't yet know whether we'll perform on Friday, Oct. 5 or Saturday, Oct. 6.
(The format will be similar to the format I used in "Lessons, Myths, and Lenses: What I Wish I'd Known in 1998" (video, partial notes), but some plays will be more elaborate and theatrical -- much more like our inspiration "The Infinite Wrench".)
To quote the session description:
A frenetic combination of educational and entertaining segments, as chosen by the audience! In between segments, audience members will shout out numbers from a menu, and we'll perform the selected segment. It may be a short monologue, it may be a play, it may be a physical demo, or it may be a tiny traditional conference talk.
Audience members should walk away with some additional understanding of the history of Python, knowledge of some tools and libraries available in the Python ecosystem, and some Python-related amusement.
So now Jason and I just have to find a director, write and memorize and rehearse and block probably 15-20 Python-related plays/songs?/dances?/presentations, acquire and set up some number of props, figure out lights and sound and visuals, possibly recruit volunteers to join us for a few bits, run some preview performances to see whether the lessons and jokes land, and perform our opening (also closing) performance. In 68 days.
(Simultaneously: I have three clients, and want to do my bit before the midterm elections, and work on a fairly major apartment-related project with Leonard, and and and and.)
Jason, thank you for the way your eyes lit up on the way back from PyCon when I mentioned this PyGotham session idea -- I think your enthusiasm will energize me when I'm feeling overwhelmed by the ambition of this project, and I predict I'll reciprocate the favor! PyGotham program committee & voters, thank you for your vote of confidence. Leonard, thanks in advance for your patience with me bouncing out of bed to write down a new idea, and probably running many painfully bad concepts past you. Future Sumana, it's gonna be ok. It will, possibly, be great. You're going to give that audience an experience they've never had before.
# 11 Apr 2018, 10:17PM: My LWN Story Summarizing PyPI's Overhaul:
This coming Monday, April 16th, we plan to flip the switch on the new PyPI and redirect https://pypi.python.org web browser requests and pip install requests so the codebase serving them is Warehouse (which is in beta right now at https://pypi.org). I'm proud of our team's work and hope you find it useful.
I haven't blogged here in a while, but I've been writing a lot, mostly announcements and explanations listed on, or a few hyperlinks away from, the onwiki index to my PyPI work. When I can't give people choices (and, unless your organization sets up a private package index/repository, PyPI can feel like the only game in town), I want to give them a lot of lead time to test, file bug reports, and migrate, and I want to provide backstory.
So: today LWN publishes a new article by me, "A new package index for Python". In it, I discuss security, policy, UX and developer experience changes in the 15+ years since PyPI's founding, new features (and deprecated old features) in Warehouse, and future plans. Plus: screenshots!
This summary should help occasional Python programmers understand why a new PyPI codebase is necessary, what's new, what features are going away, and what to expect in the near future.
If you aren't already a subscriber, you can use this subscriber link for the next week to read the article despite the LWN paywall. Thanks to LWN for the venue and the subscriber links, and thanks to Jake Edge in particular for thorough editing. Thanks to my Warehouse team for fact-checking me.
# 11 Mar 2018, 10:42AM: Recent Debugging And Confidence:
I am proud of myself for some recent debugging I've done on and with codebases and tools that I hadn't worked on before.
A few weeks ago, I was sitting next to a friend who co-maintains a web app and hadn't looked at it for a while. The styling was screwy. I asked whether some CSS or JS he depended on had upgraded, like jQuery or something. He said no, his site hosted all its dependencies. I opened up the site and checked the Network tab in Firefox Developer Tools and saw that it pulled in Bootstrap from a CDN. Ah, one of the other maintainers had added that! And updates to Bootstrap had screwed up the page's styling.
That same day, as a freshly minted co-maintainer of twine (a utility to upload packages to PyPI), I investigated a problem with our CHANGELOG. Twine's changelog, as represented on Read The Docs (example) and when I built the docs locally, only displayed version number 1.4.0 (2014-12-12) and two associated GitHub issues. This was inaccurate since the source file changelog.rst had 70+ items and ran up to version 1.9.1 (2017-05-27). I figured out that this was happening because changelog.rst is meant to be formatted so the Sphinx extension releases (which I hadn't used before) can parse it, and the current file wasn't syntactically (or semantically) adhering to releases's conventions. (Since then, with advice and help from some folks, I've released Twine 1.10.0 and started a new maintainer checklist.)
And then, a couple days later, I fixed my friends' blog. Their front page had reverted to a ten-year-old index page. I had never touched Movable Type before and hadn't used their particular managed hosting web GUI before, but I poked around (and checked for backups before changing anything) and managed to figure it out: during a May 2008 outage, someone had hand-made an index.shtml page, which was now overriding the index.html page in the server config. I figured it out and found and fixed it.
My mom says that when I was a kid, I took apart alarm clocks and spare hose attachments and so on, and put them back together just fine. She once came upon me taking something apart, and when she drew breath to admonish me, I said, "Amma, if I don't take it apart, how do I know what is inside? Don't worry, amma, I'm just looking at it, I'll put it back together when I'm done," and I did. She told me that I took apart a mechanical alarm clock, carefully spreading all the parts out on some newspaper, and put it back together, and it didn't quite work properly, so I took it apart again and then put it back together, and it worked, and I jumped for joy and said "I fixed it!" (I still feel that way when I fix something.)
At some point along the way I feel like I lost that calm confidence in my abilities, that "things are made of stuff" and what one person made another can fix. But I have it again, now, at least for some bits of software, and some purely mechanical stuff (yesterday, helping friends move, deciding to break down a big empty cardboard box, responding to "but it's so big, it won't fit on the stack" with "we have knives"). It doesn't feel courageous at the time, just sensible, but then I look back and feel like a badass.
If I had to point to the single biggest cause of this regained confidence, I'd point to the Recurse Center, where I got way more comfortable with bravery and failure in programming.
# 26 Jan 2018, 06:28PM: Preserving Threading In Google Group or Mailman Mailing List Replies with Thunderbird:
Have you ever wanted to reply to a mailing list post that wasn't in your inbox? I had that problem yesterday; here's how I fixed it.
Context: I'm the project manager for Warehouse, the software behind the new Python Package Index (PyPI) which -- thanks to funding from Mozilla and support from the Python Software Foundation -- is on its way to launching and replacing the old PyPI. I've been in the Python community for years, but -- just as when I went from "casual Wikipedian" to "Wikimedia Foundation staffer" -- I'm learning about lots of pockets of the Python community that I didn't yet know about. Specifically, Python packaging has a lot of different repositories and mailing lists. One of them is the Google Group pypa-dev, a mailing list for developers within the Python Packaging Authority.
I joined pypa-dev recently -- and, in skimming the archives, I found a months-old message I wanted to reply to while preserving threading (so that future folks and longtime subscribers would see the update in context). So I clicked on the dropdown menu in the upper right corner for that post and clicked "Show original", which got me the Message-ID header. But how could I get Thunderbird to let me write a reply with the appropriate In-Reply-To header? Preferably without having to install some extension to munge my headers?
This reply to a StackExchange answer got me most of the way there; the basic approach is the same whether you're working with a Google Group or a Mailman list. (If it's a Google Group or a Mailman 3 list, you can of course reply via the web interface, but maybe you want to cc someone or have the history in your Sent folder, or you just prefer composing in Thunderbird.)
- First, you need to get the raw text, so you can get the Message-ID.
If you're looking at a Google Group message (example), click on the dropdown menu in the upper right corner for that post and choose "Show original" (example), then click the "Show only message" button to get a raw text page like this.
If you're looking at a Mailman 2 message (example), then navigate to the monthly archive. You can get there by clicking on the "More information about the [name] list" link at the bottom of the page, which takes you to a list info page (example), and from there, the "Visit the [name] Archives." link (example). Here on the archives-by-month page, download the archive for the month that has the message (using the "[ Gzip'd Text [filesize] ]" link in the "Downloadable version" column). And now you can, for instance, gunzip 2018-January.txt.gz in your terminal to get 2018-January.txt which you can search to find the post you want to reply to.
If you're looking at a Mailman 3 message (example), look at the bottom of the left navbar for a "Download" button (hover text: "this thread in gzipped mbox format"). If you gunzip that you'll get a plain-text .mbox file which you can search to find the post you want to reply to.
- Now, no matter what mailing list software you had to wrangle, save the raw message as a temporary file with a .eml extension, e.g., /tmp/post.eml, to smooth the way for Thunderbird and your OS to think of this as a saved email message. If you're looking at a Mailman archive, this is where you select just that one message (headers and body) from the .txt or .mbox file and cut-and-paste it into a standalone .eml file.
- Open that file in Thunderbird: File menu, select Open, select Saved message, and navigate to /tmp/post.eml and open it.
- If all's gone well, the message pops open in its own window, complete with Reply and Reply All buttons! Go ahead and use those. Note that the From: and To: lines have been obfuscated or partially truncated to protect against spammers, so you'll probably need to fix those by hand, e.g., replacing at with @ and fixing any ellipses (...).
- Hit Send with the glow of thread-preservation satisfaction. Watch for your post to show up, properly threaded, in the list archives (example).
You can hire me through Changeset Consulting.
This work by Sumana Harihareswara is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.
Permissions beyond the scope of this license may be available by emailing the author at email@example.com.