Cogito, Ergo Sumana

Categories: sumana | Programming:Python

Python announcements, programming and tools

: My LWN Story Summarizing PyPI's Overhaul: Python Package Index logoThis coming Monday, April 16th, we plan to flip the switch on the new PyPI and redirect web browser requests and pip install requests so the codebase serving them is Warehouse (which is in beta right now at I'm proud of our team's work and hope you find it useful.

I haven't blogged here in a while, but I've been writing a lot, mostly announcements and explanations listed on, or a few hyperlinks away from, the onwiki index to my PyPI work. When I can't give people choices (and, unless your organization sets up a private package index/repository, PyPI can feel like the only game in town), I want to give them a lot of lead time to test, file bug reports, and migrate, and I want to provide backstory.

So: today LWN publishes a new article by me, "A new package index for Python". In it, I discuss security, policy, UX and developer experience changes in the 15+ years since PyPI's founding, new features (and deprecated old features) in Warehouse, and future plans. Plus: screenshots!

This summary should help occasional Python programmers understand why a new PyPI codebase is necessary, what's new, what features are going away, and what to expect in the near future.

If you aren't already a subscriber, you can use this subscriber link for the next week to read the article despite the LWN paywall. Thanks to LWN for the venue and the subscriber links, and thanks to Jake Edge in particular for thorough editing. Thanks to my Warehouse team for fact-checking me.

Filed under:

: Recent Debugging And Confidence: I am proud of myself for some recent debugging I've done on and with codebases and tools that I hadn't worked on before.

A few weeks ago, I was sitting next to a friend who co-maintains a web app and hadn't looked at it for a while. The styling was screwy. I asked whether some CSS or JS he depended on had upgraded, like jQuery or something. He said no, his site hosted all its dependencies. I opened up the site and checked the Network tab in Firefox Developer Tools and saw that it pulled in Bootstrap from a CDN. Ah, one of the other maintainers had added that! And updates to Bootstrap had screwed up the page's styling.

That same day, as a freshly minted co-maintainer of twine (a utility to upload packages to PyPI), I investigated a problem with our CHANGELOG. Twine's changelog, as represented on Read The Docs (example) and when I built the docs locally, only displayed version number 1.4.0 (2014-12-12) and two associated GitHub issues. This was inaccurate since the source file changelog.rst had 70+ items and ran up to version 1.9.1 (2017-05-27). I figured out that this was happening because changelog.rst is meant to be formatted so the Sphinx extension releases (which I hadn't used before) can parse it, and the current file wasn't syntactically (or semantically) adhering to releases's conventions. (Since then, with advice and help from some folks, I've released Twine 1.10.0 and started a new maintainer checklist.)

And then, a couple days later, I fixed my friends' blog. Their front page had reverted to a ten-year-old index page. I had never touched Movable Type before and hadn't used their particular managed hosting web GUI before, but I poked around (and checked for backups before changing anything) and managed to figure it out: during a May 2008 outage, someone had hand-made an index.shtml page, which was now overriding the index.html page in the server config. I figured it out and found and fixed it.

My mom says that when I was a kid, I took apart alarm clocks and spare hose attachments and so on, and put them back together just fine. She once came upon me taking something apart, and when she drew breath to admonish me, I said, "Amma, if I don't take it apart, how do I know what is inside? Don't worry, amma, I'm just looking at it, I'll put it back together when I'm done," and I did. She told me that I took apart a mechanical alarm clock, carefully spreading all the parts out on some newspaper, and put it back together, and it didn't quite work properly, so I took it apart again and then put it back together, and it worked, and I jumped for joy and said "I fixed it!" (I still feel that way when I fix something.)

At some point along the way I feel like I lost that calm confidence in my abilities, that "things are made of stuff" and what one person made another can fix. But I have it again, now, at least for some bits of software, and some purely mechanical stuff (yesterday, helping friends move, deciding to break down a big empty cardboard box, responding to "but it's so big, it won't fit on the stack" with "we have knives"). It doesn't feel courageous at the time, just sensible, but then I look back and feel like a badass.

If I had to point to the single biggest cause of this regained confidence, I'd point to the Recurse Center, where I got way more comfortable with bravery and failure in programming.

Filed under:

: Preserving Threading In Google Group or Mailman Mailing List Replies with Thunderbird: Have you ever wanted to reply to a mailing list post that wasn't in your inbox? I had that problem yesterday; here's how I fixed it.

Context: I'm the project manager for Warehouse, the software behind the new Python Package Index (PyPI) which -- thanks to funding from Mozilla and support from the Python Software Foundation -- is on its way to launching and replacing the old PyPI. I've been in the Python community for years, but -- just as when I went from "casual Wikipedian" to "Wikimedia Foundation staffer" -- I'm learning about lots of pockets of the Python community that I didn't yet know about. Specifically, Python packaging has a lot of different repositories and mailing lists. One of them is the Google Group pypa-dev, a mailing list for developers within the Python Packaging Authority.

I joined pypa-dev recently -- and, in skimming the archives, I found a months-old message I wanted to reply to while preserving threading (so that future folks and longtime subscribers would see the update in context). So I clicked on the dropdown menu in the upper right corner for that post and clicked "Show original", which got me the Message-ID header. But how could I get Thunderbird to let me write a reply with the appropriate In-Reply-To header? Preferably without having to install some extension to munge my headers?

This reply to a StackExchange answer got me most of the way there; the basic approach is the same whether you're working with a Google Group or a Mailman list. (If it's a Google Group or a Mailman 3 list, you can of course reply via the web interface, but maybe you want to cc someone or have the history in your Sent folder, or you just prefer composing in Thunderbird.)

  1. First, you need to get the raw text, so you can get the Message-ID.

    If you're looking at a Google Group message (example), click on the dropdown menu in the upper right corner for that post and choose "Show original" (example), then click the "Show only message" button to get a raw text page like this.

    If you're looking at a Mailman 2 message (example), then navigate to the monthly archive. You can get there by clicking on the "More information about the [name] list" link at the bottom of the page, which takes you to a list info page (example), and from there, the "Visit the [name] Archives." link (example). Here on the archives-by-month page, download the archive for the month that has the message (using the "[ Gzip'd Text [filesize] ]" link in the "Downloadable version" column). And now you can, for instance, gunzip 2018-January.txt.gz in your terminal to get 2018-January.txt which you can search to find the post you want to reply to.

    If you're looking at a Mailman 3 message (example), look at the bottom of the left navbar for a "Download" button (hover text: "this thread in gzipped mbox format"). If you gunzip that you'll get a plain-text .mbox file which you can search to find the post you want to reply to.

  2. Now, no matter what mailing list software you had to wrangle, save the raw message as a temporary file with a .eml extension, e.g., /tmp/post.eml, to smooth the way for Thunderbird and your OS to think of this as a saved email message. If you're looking at a Mailman archive, this is where you select just that one message (headers and body) from the .txt or .mbox file and cut-and-paste it into a standalone .eml file.
  3. Open that file in Thunderbird: File menu, select Open, select Saved message, and navigate to /tmp/post.eml and open it.
  4. If all's gone well, the message pops open in its own window, complete with Reply and Reply All buttons! Go ahead and use those. Note that the From: and To: lines have been obfuscated or partially truncated to protect against spammers, so you'll probably need to fix those by hand, e.g., replacing at with @ and fixing any ellipses (...).
  5. Hit Send with the glow of thread-preservation satisfaction. Watch for your post to show up, properly threaded, in the list archives (example).
Filed under:


You can hire me through Changeset Consulting.

Creative Commons License
This work by Sumana Harihareswara is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.
Permissions beyond the scope of this license may be available by emailing the author at