Cogito, Ergo Sumana

Categories: sumana | Programming:Python

[Add new subcategory] [Delete]

Python announcements, programming and tools


(1) : MOSS Video, BSSw Honorable Mention, and The Maintainership Book I Am Writing:

Video

Mozilla interviewed me about the Python Package Index (PyPI), a USD$170,000 Mozilla Open Source Support award I helped the Python Software Foundation get in 2017, and how we used that money to revamp PyPI and drive it forward in 2017 and 2018.

From that interview, they condensed a video (2 minutes, 14 seconds) featuring, for instance, slo-mo footage of me making air quotes. Their tweet calls me "a driving force behind" PyPI, and given how many people were working on it way before I was, that's quite a compliment!

I will put a transcript in the comments of this blog post.

(Please note that they massively condensed this video from 30+ minutes of interview. In the video, I say, "the site got popular before the code got good". In the interview, I did not just say that without acknowledging the tremendous effort of past volunteers who worked on the previous iteration of PyPI and kept the site going through massive infrastructure challenges, but that's been edited (for brevity, I assume).)

This video is the first in a series meant to encourage people to apply for MOSS funding. I mentioned MOSS in my grants roundup last month. If you want to figure out whether to apply for MOSS funding for your open source software project, and you need help, ping me for a free 20-minute chat or phone call and I can give you some quick advice. (Offer limited in case literally a hundred people contact me, which is unlikely.)

BSSw

The Better Scientific Software (BSSw) Fellowship Program "gives recognition and funding to leaders and advocates of high-quality scientific software." I'm one of three Honorable Mentions for 2020.

The main goal of the BSSw Fellowship program is to foster and promote practices, processes, and tools to improve developer productivity and software sustainability of scientific code. We also anticipate accumulating a growing community of BSSw Fellowship alums who can serve as leaders, mentors, and consultants to increase the visibility of those involved in scientific software production and sustainability in the pursuit of scientific discovery.

Exascale Computing Project logoThat's why I'll be at the Exascale Computing Project Annual Meeting next week in Houston, so if you're there, I hope to meet you. In particular I'd like to meet the leaders of open source projects who want help streamlining contribution processes, growing more maintainers, managing communications with stakeholders, participating in internship projects like Google Summer of Code and Outreachy, expediting releases, and getting more out of hackathons. My consulting firm provides these services, and at ECPAM I can give you some free advice.

Book

And here's the project I'm working on -- why I received this honor.

In 2020, I am writing the first draft of a book teaching the skills open source software maintainers need, aimed at those working scientists and other contributors who have never managed public-facing projects before.

More than developer time, maintainership -- coordination, leadership, and management -- is a bottleneck in software sustainability. The lack of skilled managers is a huge blocker to the sustainability of Free/Libre Open Source Software (FLOSS) infrastructure.

Many FLOSS project maintainers lack management experience and skill. This textbook/self-help guide for new and current maintainers of existing projects ("brownfield projects") will focus on teaching specific project management skills in the context of FLOSS. This will provide scalable guidance, enabling existing FLOSS contributors to become more effective maintainers.

Existing "how to run a FLOSS project" documentation (such as Karl Fogel's Producing Open Source Software) addresses fresh-start "greenfield" projects rather than more common "brownfield", and doesn't teach specific project management skills (e.g., getting to know a team, creating roadmaps, running asynchronous meetings, managing budgets, and writing email memos). Existing educational pathways for scientists and developers (The Carpentries, internships and code schools) don't cover FLOSS-specific management skills.

So I'm writing a sequel to Karl's book -- with his blessing -- and I'm excited to see how I can more scalably share the lessons I've learned in more than a decade of leading open source projects.

I don't yet have a full outline, a publisher, or a length in mind. I'll be posting more here as I grow my plans. Thanks to BSSw and all my colleagues and friends who have encouraged me.


[Edit me!]

: My New Title, Improving pip, Availability For Work, And SSL (No, The Other One): A few professional announcements.

Seeking developers for paid contract on pip; apply by Nov. 22

One is that I helped the Packaging Working Group of the Python Software Foundation get funding for a long-needed improvement to pip. I led the writing of a few proposals -- grantwriting, to oversimplify -- and, starting possibly as soon as next month, contractors will start work. As Dustin Ingram explains:

Big news: the Python Packaging Working Group has secured >$400K in grants from multiple funders (TBA) to improve one of the most fundamental parts of pip: its dependency resolver. https://pyfound.blogspot.com/2019/11/seeking-developers-for-paid-contract.html

The dependency resolver is the algorithm which takes multiple constrained requirements (e.g. "some_package>=1.0,<=2.0") and finds a version of all dependencies (and sub-dependencies) which satisfy all the constraints.
https://pip.pypa.io/en/stable/user_guide/#requirements-files

Right now, pip's resolver mostly works for most use cases... However the algorithm it uses is naïve, and isn't always guaranteed to produce an optimal (or correct) result.

.....

These funds will pay multiple developers to work on completing the design, implementation and rollout of this new dependency resolver for pip, finally closing issue #988.

Not only will this give pip a better resolver, but it will "enable us to untangle pip’s internals from the resolver, enabling pip to share code for dependency resolution with other packaging tooling". https://pradyunsg.me/blog/2019/06/23/oss-update-1/

This is great news for pip and Python packaging in general. Huge shout out to @pradyunsg for his existing work on the resolver issue and guidance here, and to @brainwane for all her tireless work acquiring and directing funding for Python projects.

If you or your organization is interested in participating in this project, we've just posted the RFP, which includes instructions for submitting proposals, evaluation criteria and scope of work.
https://github.com/python/request-for/blob/master/2020-pip/RFP.md

If you're interested, please apply by 22 November.

NYU, Secure Systems Lab, and my new title

Working at the new space on NYU Tandon's campus, left to right: Sumana Harihareswara, a volunteer with the PSF's Packaging Working Group, a contracted project manager for the Python Packaging Index, and a visiting scholar in NYU Tandon Professor Justin Cappos's Secure Systems Lab; Stephanie Whited, communications director for the Tor Project and visiting researcher in the Secure System Lab; and Santiago Torres, a computer science doctoral candidate working in the Secure Systems Lab. Photo by NYU publicity.In further news: I am now a visiting scholar in Professor Justin Cappos's Secure Systems Lab at New York University's Tandon School of Engineering. And I get to use an office with a door, shelves, whiteboards, and so on (per the picture at right). If you contribute to Python packaging/distribution tools and live in/near or sometimes visit New York City, let me know and perhaps we could cowork a bit?

The Secure Systems Lab stewards The Update Framework (TUF) and related projects, and works to improve the security of the software supply chain. The Python Package Index is likely going to implement TUF to add cryptographic signatures to packages on PyPI, and so I've gotten to give TUF's developers some advice to help that work move along. (I won't be the manager on that project but I'll be watching with great interest.) PyPA may also choose to use more of SSL's work in implementing further security improvements to the package distribution toolchain, and I'm learning more to work out whether and how that could happen. Also, Cappos's research on backtracking dependency resolvers has been helpful to the pip resolver work.

Edited 19 Nov 2019 to clarify role.

PSF projects

I'm grateful to get to help connect the Python Software Foundation with more resources and volunteers. Changeset's current and recent projects have mostly been for the PSF. Last month we finished accessibility, security, and internationalization work on PyPI that was funded by the Open Technology Fund, and Changeset's work on communicating about the sunsetting of Python 2.x continues and will go through April 2020.

Availability for one-day engagements in San Francisco in February

But I am interested in taking on new clients for short engagements starting in February 2020. In particular, I will be in the San Francisco Bay Area in mid- to late February. If you're in SF or nearby, I could offer you a one-day engagement doing one of the following:

  • developing a contributor outreach/intake strategy
  • researching potential funders and writing a rough draft of a grant proposal
  • auditing and improving your developer onboarding documents

I'd spend a little time talking with you, then sit in your office and finish the document before leaving that afternoon. (Photo at right provides a sample of how I look while sitting.) Drop me a line for a free initial 30-minute chat and we can talk pricing.


[Edit me!]

: Futureproofing Your Python Tools: The people who maintain Python and key Python platforms want to help you protect the code you write and depend on.

If you write software in Python, or depend on something that's in Python, this is for you.

Some of you are writing Python 2, or you still have software you depend on that is written in Python 2. January 1st, 2020, is the day that official support for Python 2 stops. So this is a fresh heads-up that you should really have a migration plan and start working on it, to move to Python 3. A lot of stuff you depend on already works on Python 3, and is even pledging to end Python 2 support in or before 2020. And it's easier than it's ever been to port your own code from 2 to 3.

You should probably upgrade to Python 3.7. If you want to test out 3.8, it has some changes in how it does warnings, and the first release candidate is out.

But, speaking of futureproofing:

Code authors move in and out of projects and companies. Six months or 18 months later, maybe you want to update and re-use, re-release or re-deploy code someone else wrote. Or you want your team to be able to reuse what you did after you leave, which means you need the code to run, and you need them to have the password so they can update stuff.

Have you written Python code that they have published as a package on the Python Package Index, pypi.org, so other people can use pip install to install it?

(And if you want to do that but don't know how? Check out this recently improved tutorial to help you do that.)

Publishing that package is a great way of making it so other people can run and deploy it, even within other parts of your organization.

But -- who actually has the keys to the castle? Who can upload a new version, or delete a version that has a problem?

You should probably make sure multiple people have either "owner" or "maintainer" privileges on the project on PyPI.

And you should review your project security history display, which lists sensitive events (such as "file removed from release version 1.0.1") in your PyPI user account and your PyPI project. We just added this display, so you can look at things that have happened in your user account or project, and check for signs someone's stolen your credentials.

And then how do you make it a little harder for vandals, spammers, and thieves to take over your account and upload malware or delete all the packages? Add two-factor auth for your login, like you would with your bank. Use an app on your phone to give you a six-digit code, or use a physical security device like a Yubikey.

And how do you make it easier to automate publishing new versions of your package, and make it safer to save your credential in the cloud? We've made it possible for you to create an "API token" where all it can do is upload, so you can use that instead of your PyPI password.

With well-tested Python 3 migration tools and new PyPI security features, now is a great moment to invest in robustness for Python software that you make or depend on.

[This blog post is kind of unwieldy, because it's about too many different things. I won't be publicizing it that much and instead will probably reuse text from it in more focused announcements elsewhere. But I'm publishing it here as a summary of my recent work, because management and communications for the projects above are what I've been working on recently for the Python Software Foundation. (A different kind of summary is on the Clients page for Changeset Consulting.)]


[Edit me!]

: Kickoff for Communications Work on the Python 2 Sunsetting: Python's 2.x line will reach End of Life on January 1, 2020, meaning that the maintainers of Python 2 will stop supporting it, even for security patches. Many institutions and codebases have not yet ported their code from Python 2 to Python 3. And many of them haven't even heard yet about the upcoming EOL. Volunteers have made many resources to help publicize and educate, but there's still more work to be done.

So the Python Software Foundation has contracted with my firm, Changeset Consulting, to help communicate about the sunsetting of Python 2. The high-level goal for Changeset's involvement is to help users through the end of the transition, help with communication so volunteers are not overwhelmed, and help update public-facing assets so core developers are not overwhelmed.

During this project (starting now and going till the end of April 2020), Changeset's goals are:

  • Create a concise page on python.org giving community guidance on the why and what of EOL
  • Create and publicize a formal press release through the PSF
  • Audit and update every mention of Python 2 on python.org, including in documentation and the wiki
  • Develop a plan to handle redirecting https://docs.python.org/2/* (especially deep links)
  • Publicize the new "what's up with EOL" page to a variety of audiences, respond to questions, and keep the page updated until PyCon US 2020 based on questions that come up

So, towards those goals, you'll see me with my colleagues:

  • Researching, writing, and editing technical documents and the press release
  • Corresponding with stakeholders, writing and publishing announcements and other communications in multiple media
  • Initiating and facilitating meetings (I will try to make them very minimal and short; if I invite you, please let me know your response)
  • Filing, triaging, labelling, prioritizing, and linking issues, such as in the website repo

For accountability, Changeset will provide reporting via:

I'm going to be asking for a lot of help along the way from the Python community: meeting with us, answering our questions, double-checking our drafts for accuracy, publicizing that EOL page to your circles, setting up some parties for January 1st. Thanks in advance and let's get the Python user base further along towards enjoying the shininess of Python 3.


[Edit me!]

: Beautiful Soup is on Tidelift:

I've been doing a tiny bit of consulting for Tidelift for a little over a year now, mainly talking about them to open source maintainers in the Python world and vice versa. (See my October 2018 piece "Tidelift Is Paying Maintainers And, Potentially, Fixing the Economics of an Industry".) And lo, in my household, my spouse Leonard Richardson has signed up as a lifter for Beautiful Soup, his library that helps you with screen-scraping projects.

Leonard writes:

There was a period of about a year in 2017-2018 when I wasn't interested in doing Beautiful Soup work, but Tidelift changed that. Tidelift gathers subscription money from companies that rely on free software, and distributes the money to the developers in exchange for a level of support that I find sustainable.

Nobody builds an entire product around Beautiful Soup (or at least nobody will admit do doing this), but thousands of people have used Beautiful Soup to save time at their day jobs. Bundling Beautiful Soup together with bigger projects like Flask and numpy is a solution that works really well for me.

The other day I looked at the list of featured supported packages and was happy to see that a bunch of Python projects have signed up as lifters, including SciPy and numpy, Flask, setuptools, Werkzeug, websockets, urllib3, celery, coverage, a bunch of Django packages, Jinja2, keyring, Pylint, coverage, and pytest. And I think over the next 6-12 months we're going to see some effects of Tidelift support -- not just in the security and release cadences of the supported projects, but on other issues stemming from unfairness and a lack of reciprocity in open source, like maintainer burnout and expectation-setting. The list of licensing, security, maintenance, and marketing tasks lifters agree to do may end up being a benchmark, like the open source Independent Verification & Validation checklist by Open Tech Strategies, that even non-lifter maintainers use to set realistic expectations for what "supported"/"maintained" means.


[Edit me!]

: PyCon NA, !!Con, and WisCon: I'm back in New York City, and preparing to travel a bit; May is my big conference month this year.

I'll start the month at PyCon North America in Cleveland, Ohio, practically the whole conference, 1-9 May. I'm co-organizing The Art of Python, an arts festival -- several short plays, plus a fanvid and live music -- the night of Friday, May 3rd. And during the sprints, 6-9 May, I'll be concentrating on the world of Python packaging and distribution, e.g., PyPI.

I'll go home to New York City, then go to !!Con 11-12 May, where I am not organizing or speaking or suchlike.

And then I'll be at WisCon in Madison, Wisconsin, for the whole convention, 24-27 May plus a little extra on either side. I will once again be the comedy auctioneer for the auction on Saturday night that benefits the Tiptree Award -- if you have something to donate to the auction, please let us know by 15 May. I may not make it to the Floomp or the vid party. I will probably be on a few panels; several panels are still seeking volunteer panellists (sign up by 19 April). I do plan to be at the Gathering and the Dessert Salon (heads-up, changes to Dessert Salon entry flow).

If you're going to be at any of these events, perhaps we can share a beverage! If you want to make sure that we do that, let's actually set up at least a tentative appointment soon, so I can put it in my calendar.

I will be more responsive to emails and text messages than to social media while at these conferences, and in particular, I may see mentions and direct messages on Mastodon but I probably won't see mentions or direct messages on Twitter. Also, I am pretty forgiving about being called mispronunciations of my name, but here's a recording in case you want help -- I also respond to "Vikki" which is the fake name I use with strangers at restaurants. And I will probably not hug you unless we know each other well.


[Edit me!]

: Tidelift Is Paying Maintainers And, Potentially, Fixing the Economics of an Industry: As the founder of Changeset Consulting, I keep my eye on consultancies and services in and near my niche, open source leadership, maintainership, and sustainability.* And I've known Luis Villa for years and got to work with him at Wikimedia. So yeah, I noticed when Tidelift announced its big new launch. And -- now, as a very-part-time consultant who helps Tidelift understand the Python world -- I am excited about their commitment to pay more than USD$1 million to maintainers (including "a guaranteed minimum $10,000 over the next 24 months to select projects").

Here's my take on the new Tidelift subscription model, the "lifter" role, and whom this works for.

For software businesses, this provides that missing vendor relationship, SLA, release cadence expectations, and general peace of mind for all of that unseen infrastructure you depend on. It's often easier for businesses -- of many sizes -- to pay a regular fee than to put open source project management work, dependency-updating, compliance checking, dependency security audits, or FLOSS volunteer relations on the engineering schedule.

For individual programmers and community-maintained open source projects, Tidelift is a potential source of substantial income. As a Pythonist, I hope to reach people who are currently core code contributors to open source projects in Python, especially on the Libraries.io digital infrastructure/unseen infrastructure/improve the bus factor lists. And I would like to reach projects like the ones Nathaniel Smith calls out in a recent post:

that (1) require a modest but non-trivial amount of sustained, focused attention, and (2) have an impact that is large, but broad and diffuse
and projects in the "wide open", "specialty library", and "upstream dependency" categories identified by the Open Tech Strategies report "Open Source Archetypes: A Framework For Purposeful Open Source".

For such people and projects, becoming a lifter is a promising model -- especially since the required tasks are fairly few, and are things maintainers should do anyway. I'm encouraged to see Jeff Forcier (maintainer of Fabric, Alabaster, and more) and Ned Batchelder's coverage.py getting onto the Tidelift platform.

And you can see estimated monthly income for your package right now. For some people, especially those whose healthcare doesn't depend on an employer, Tidelift payments plus some side consulting could be a sustainable, comfortable income.

Then there are folks like me whose contributions are only partially visible in commit logs (management, user support, testing, and so on), and groups that work together best as a team. Tidelift is also a potential source of income for us, but it's a little more complicated. Tidelift can send lifter payments to individuals, for-profits, and nonprofits, but: "If a package has multiple co-maintainers, you'll need to agree as a group on an approach." If you thought code of conduct conversations with your community were uncomfortable, wait till you bring up money! But, more seriously: I've been able to talk frankly with open source colleagues about thorny "who gets paid what?" questions, and if you're candid with your co-maintainers, the benefits may be pretty substantial. You can get advice on this conversation during the next live Tidelift web-based Q&A, Thursday, Oct. 11 at 2 p.m. Eastern Time (sign up at the bottom of the lifter info page).

Nonprofits, companies, and working groups that maintain projects can sign up now as lifters. Even if it's just a trickle of money right now, it might build over time and turn into enough to fund travel for an in-person sprint, contract work to improve continuous integration, an Outreachy internship, etc.

(One gap here: right now, Tidelift isn't great at supporting system-level packages and projects, like tools that get installed via apt or yum/DNF. I'm pretty sure that's something they're working on.)

What about noncommercial users or users who can't afford Tidelift subscriptions? The more lifters and subscribers sign up, the more those users benefit, too. Subscribers' funding means maintainers have time to make improvements that help everyone. And lifters agree to follow security, maintenance, and licensing best practices that also help everyone. Plus, Tidelift stewards libraries.io, a great resource for anyone who uses or develops open source (more on that). More money for Tidelift could mean libraries.io gets better too.

So I'm tooting a horn here and hoping more people sign up, because this is one of the more plausible ways open source sustainability could possibly work. Tidelift could be a real game-changer for the industry. Check it out.


* Examples: new competitors like Maintainer Mountaineer and OpenTeam, new funders like OSS Capital, and colleagues/referrals like Open Tech Strategies, VM Brasseur, Otter Tech, and Authentic Engine.


[Edit me!]

: "Python Grab Bag: A Set of Short Plays" Accepted for PyGotham 2018: Fresh from the waitlist onto the schedule: Jason Owen and I will be performing "Python Grab Bag: A Set of Short Plays" at PyGotham in early October. If you want to come see us perform, you should probably register soon. I don't yet know whether we'll perform on Friday, Oct. 5 or Saturday, Oct. 6.

(The format will be similar to the format I used in "Lessons, Myths, and Lenses: What I Wish I'd Known in 1998" (video, partial notes), but some plays will be more elaborate and theatrical -- much more like our inspiration "The Infinite Wrench".)

To quote the session description:

A frenetic combination of educational and entertaining segments, as chosen by the audience! In between segments, audience members will shout out numbers from a menu, and we'll perform the selected segment. It may be a short monologue, it may be a play, it may be a physical demo, or it may be a tiny traditional conference talk.

Audience members should walk away with some additional understanding of the history of Python, knowledge of some tools and libraries available in the Python ecosystem, and some Python-related amusement.

So now Jason and I just have to find a director, write and memorize and rehearse and block probably 15-20 Python-related plays/songs?/dances?/presentations, acquire and set up some number of props, figure out lights and sound and visuals, possibly recruit volunteers to join us for a few bits, run some preview performances to see whether the lessons and jokes land, and perform our opening (also closing) performance. In 68 days.

(Simultaneously: I have three clients, and want to do my bit before the midterm elections, and work on a fairly major apartment-related project with Leonard, and and and and.)

Jason, thank you for the way your eyes lit up on the way back from PyCon when I mentioned this PyGotham session idea -- I think your enthusiasm will energize me when I'm feeling overwhelmed by the ambition of this project, and I predict I'll reciprocate the favor! PyGotham program committee & voters, thank you for your vote of confidence. Leonard, thanks in advance for your patience with me bouncing out of bed to write down a new idea, and probably running many painfully bad concepts past you. Future Sumana, it's gonna be ok. It will, possibly, be great. You're going to give that audience an experience they've never had before.


[Edit me!]

: My LWN Story Summarizing PyPI's Overhaul: Python Package Index logoThis coming Monday, April 16th, we plan to flip the switch on the new PyPI and redirect https://pypi.python.org web browser requests and pip install requests so the codebase serving them is Warehouse (which is in beta right now at https://pypi.org). I'm proud of our team's work and hope you find it useful.

I haven't blogged here in a while, but I've been writing a lot, mostly announcements and explanations listed on, or a few hyperlinks away from, the onwiki index to my PyPI work. When I can't give people choices (and, unless your organization sets up a private package index/repository, PyPI can feel like the only game in town), I want to give them a lot of lead time to test, file bug reports, and migrate, and I want to provide backstory.

So: today LWN publishes a new article by me, "A new package index for Python". In it, I discuss security, policy, UX and developer experience changes in the 15+ years since PyPI's founding, new features (and deprecated old features) in Warehouse, and future plans. Plus: screenshots!

This summary should help occasional Python programmers understand why a new PyPI codebase is necessary, what's new, what features are going away, and what to expect in the near future.

If you aren't already a subscriber, you can use this subscriber link for the next week to read the article despite the LWN paywall. Thanks to LWN for the venue and the subscriber links, and thanks to Jake Edge in particular for thorough editing. Thanks to my Warehouse team for fact-checking me.


[Edit me!]

: Recent Debugging And Confidence: I am proud of myself for some recent debugging I've done on and with codebases and tools that I hadn't worked on before.

A few weeks ago, I was sitting next to a friend who co-maintains a web app and hadn't looked at it for a while. The styling was screwy. I asked whether some CSS or JS he depended on had upgraded, like jQuery or something. He said no, his site hosted all its dependencies. I opened up the site and checked the Network tab in Firefox Developer Tools and saw that it pulled in Bootstrap from a CDN. Ah, one of the other maintainers had added that! And updates to Bootstrap had screwed up the page's styling.

That same day, as a freshly minted co-maintainer of twine (a utility to upload packages to PyPI), I investigated a problem with our CHANGELOG. Twine's changelog, as represented on Read The Docs (example) and when I built the docs locally, only displayed version number 1.4.0 (2014-12-12) and two associated GitHub issues. This was inaccurate since the source file changelog.rst had 70+ items and ran up to version 1.9.1 (2017-05-27). I figured out that this was happening because changelog.rst is meant to be formatted so the Sphinx extension releases (which I hadn't used before) can parse it, and the current file wasn't syntactically (or semantically) adhering to releases's conventions. (Since then, with advice and help from some folks, I've released Twine 1.10.0 and started a new maintainer checklist.)

And then, a couple days later, I fixed my friends' blog. Their front page had reverted to a ten-year-old index page. I had never touched Movable Type before and hadn't used their particular managed hosting web GUI before, but I poked around (and checked for backups before changing anything) and managed to figure it out: during a May 2008 outage, someone had hand-made an index.shtml page, which was now overriding the index.html page in the server config. I figured it out and found and fixed it.

My mom says that when I was a kid, I took apart alarm clocks and spare hose attachments and so on, and put them back together just fine. She once came upon me taking something apart, and when she drew breath to admonish me, I said, "Amma, if I don't take it apart, how do I know what is inside? Don't worry, amma, I'm just looking at it, I'll put it back together when I'm done," and I did. She told me that I took apart a mechanical alarm clock, carefully spreading all the parts out on some newspaper, and put it back together, and it didn't quite work properly, so I took it apart again and then put it back together, and it worked, and I jumped for joy and said "I fixed it!" (I still feel that way when I fix something.)

At some point along the way I feel like I lost that calm confidence in my abilities, that "things are made of stuff" and what one person made another can fix. But I have it again, now, at least for some bits of software, and some purely mechanical stuff (yesterday, helping friends move, deciding to break down a big empty cardboard box, responding to "but it's so big, it won't fit on the stack" with "we have knives"). It doesn't feel courageous at the time, just sensible, but then I look back and feel like a badass.

If I had to point to the single biggest cause of this regained confidence, I'd point to the Recurse Center, where I got way more comfortable with bravery and failure in programming.


[Edit me!]

: Preserving Threading In Google Group or Mailman Mailing List Replies with Thunderbird: Have you ever wanted to reply to a mailing list post that wasn't in your inbox? I had that problem yesterday; here's how I fixed it.

Context: I'm the project manager for Warehouse, the software behind the new Python Package Index (PyPI) which -- thanks to funding from Mozilla and support from the Python Software Foundation -- is on its way to launching and replacing the old PyPI. I've been in the Python community for years, but -- just as when I went from "casual Wikipedian" to "Wikimedia Foundation staffer" -- I'm learning about lots of pockets of the Python community that I didn't yet know about. Specifically, Python packaging has a lot of different repositories and mailing lists. One of them is the Google Group pypa-dev, a mailing list for developers within the Python Packaging Authority.

I joined pypa-dev recently -- and, in skimming the archives, I found a months-old message I wanted to reply to while preserving threading (so that future folks and longtime subscribers would see the update in context). So I clicked on the dropdown menu in the upper right corner for that post and clicked "Show original", which got me the Message-ID header. But how could I get Thunderbird to let me write a reply with the appropriate In-Reply-To header? Preferably without having to install some extension to munge my headers?

This reply to a StackExchange answer got me most of the way there; the basic approach is the same whether you're working with a Google Group or a Mailman list. (If it's a Google Group or a Mailman 3 list, you can of course reply via the web interface, but maybe you want to cc someone or have the history in your Sent folder, or you just prefer composing in Thunderbird.)

  1. First, you need to get the raw text, so you can get the Message-ID.

    If you're looking at a Google Group message (example), click on the dropdown menu in the upper right corner for that post and choose "Show original" (example), then click the "Show only message" button to get a raw text page like this.

    If you're looking at a Mailman 2 message (example), then navigate to the monthly archive. You can get there by clicking on the "More information about the [name] list" link at the bottom of the page, which takes you to a list info page (example), and from there, the "Visit the [name] Archives." link (example). Here on the archives-by-month page, download the archive for the month that has the message (using the "[ Gzip'd Text [filesize] ]" link in the "Downloadable version" column). And now you can, for instance, gunzip 2018-January.txt.gz in your terminal to get 2018-January.txt which you can search to find the post you want to reply to.

    If you're looking at a Mailman 3 message (example), look at the bottom of the left navbar for a "Download" button (hover text: "this thread in gzipped mbox format"). If you gunzip that you'll get a plain-text .mbox file which you can search to find the post you want to reply to.

  2. Now, no matter what mailing list software you had to wrangle, save the raw message as a temporary file with a .eml extension, e.g., /tmp/post.eml, to smooth the way for Thunderbird and your OS to think of this as a saved email message. If you're looking at a Mailman archive, this is where you select just that one message (headers and body) from the .txt or .mbox file and cut-and-paste it into a standalone .eml file.
  3. Open that file in Thunderbird: File menu, select Open, select Saved message, and navigate to /tmp/post.eml and open it.
  4. If all's gone well, the message pops open in its own window, complete with Reply and Reply All buttons! Go ahead and use those. Note that the From: and To: lines have been obfuscated or partially truncated to protect against spammers, so you'll probably need to fix those by hand, e.g., replacing at with @ and fixing any ellipses (...).
  5. Hit Send with the glow of thread-preservation satisfaction. Watch for your post to show up, properly threaded, in the list archives (example).


[Edit me!]

[Main]

You can hire me through Changeset Consulting.

Creative Commons License
This work by Sumana Harihareswara is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.
Permissions beyond the scope of this license may be available by emailing the author at sh@changeset.nyc.