Python announcements, programming and tools
# 03 Jul 2021, 05:32PM: Some Novel Python Packaging/Distribution/Inspection/Installation Projects:
People who program in Python have an easier time hearing about package-related tools that have been around for a while and that are under the banner of the Python Packaging Authority, or that are commercially supported (this simplified diagram showcases a lot of them). And if you're looking for canonical guidance on what tools to use, check out packaging.python.org and tell your colleagues.
But -- since open source and open standards make things interoperable -- people also develop new tools for their specific needs in packaging, distribution, inspection, and installation, and sometimes I come across them when people announce them. I haven't tried any of these yet but here's a list of some stuff I noticed from the last few years.
Pypitoken, "A library for generating and manipulating PyPI tokens"
Thoth, "an enhanced server-side resolution offered to the Python community" (related: thoth-solver: "A tool for aggregating Python package metadata" and Dependency Monkey which "can compute all the possible combinations of packages that can occur in a resolved software stack and verify the given stack works well")
installer, "a low-level library for installing wheel distributions"
Dotlock "is a package management tool similar to pipenv, but with a different philosophy: instead of acting as a wrapper around pip, dotlock handles package resolution natively."
simpleindex provides "a lightweight PEP-503 private index/proxy" that declares routing rules to serve files from local directories. Also see pywharf.
Mach-nix "allows one to package Python projects and environments with Nix, requiring minimal knowledge of Nix.... Why would you want to use this tool? Reproducible builds with all build and run-time dependencies provided by the same package manager, regardless of whether they're Python dependencies or not."
ipwhl: a downstream repository in which "Each repo release will ensure a single version for a project for each platform, and one can use it to replace PyPI for both build and runtime dependencies for reproducibility." Per the repo for "interplanetary wheels (or floating cheeses)": "platform-unique, singly-versioned Python binary distributions backed by IPFS for security and reproducibility."
Python devirtualizer: "a preliminary implementation which manages
shared packages so that only one copy of each package version is
pip-deepfreeze: "a small tool that aims at managing the dependencies of a Python application in a virtual environment."
And one more thing that is a PyPA project: the Python Advisory DB. After public discussion, there's a new community-owned repository of security advisories for packages published on https://pypi.org.
# 01 Jul 2021, 12:58PM: Software Bill of Materials & the US Federal Government:
In February, the United States's President Biden signed an executive order on the US's supply chains; he followed this up with an EO in May specifically concentrating on improving cybersecurity. To quote Tidelift's summary, "in essence, this order is a striking attempt to create a new global standard for cybersecurity that all organizations around the world will need to ensure their software supply chain meets or exceeds in the near future."
Because of the EO, the US's National Telecommunications and Information Administration requested comment "on the minimum elements for a Software Bill of Materials (SBOM), and what other factors should be considered in the request, production, distribution, and consumption of SBOMs". I heard about this thanks to Jacob Kaplan-Moss, who suggested the Python Software Foundation could share its open source perspective. So I led an effort to submit a comment on behalf of the Packaging Working Group of the PSF - thanks to Morgan Mayo (PSF Director of Resource Development) for some of the prose!
The Packaging WG comment, along with comments from 80+ other individuals and groups, are now up at the NTIA website. Check out Section II ("Background context about the Python packaging toolchain and ecosystem") for a simplified yet still confusing diagram. Section IV of our comment ("Infrastructure funding") is pretty short; for a longer treatment on a related topic, see Tidelift's comment "Software bills of materials are important—but they won't work at scale if we don't pay the maintainers".
# 13 Apr 2021, 07:49PM: Python Packaging Tools: Security Work And An Open Position:
Two exciting bits of news regarding massively improving how we package, distribute, and install Python software!
First: a new grant. New York University (specifically Professor Justin Cappos) and I have successfully asked the US National Science Foundation for a grant to improve Python packaging security. The NSF is awarding NYU $800,000 over two years, from mid-2021 to mid-2023, to further improve the pip dependency resolver and to integrate The Update Framework further into the packaging toolchain. I shared more details in this announcement on an official Python packaging forum.
I'll be part of this work, paid to work on this part-time, doing some outreach, coordination, project management, and similar. Thanks to the NSF, Justin, the Secure Systems Lab at NYU, and all the people who work on Python packaging tools!
Second: the Python Software Foundation is hiring a full-time project manager and community manager for Python's packaging toolchain. Thanks to Bloomberg for the funding! Please check out the job description and spread the news. Please apply by May 18th, 2021.
The job is remote and you can apply from anywhere in the world. As the description says: "Total compensation will range from $100k-$125k USD based on qualifications and experience." And you'd report to Ee W. Durbin III, a colleague I strongly recommend and love working with.
I'm thoroughly grateful that we've now gotten to the point where the PSF can hire for a full-time person for this role. As a volunteer and as a contractor, I've performed -- in many cases initiated -- the activities that this person will do, and I've seen the critical need. We deeply need a full-time coordinator for holistically assessing and improving the user and developer experience of Python packaging, because -- as Russell Keith-Magee said in his PyCon US 2019 keynote -- the status quo poses "an existential threat" to the future of the language. And so one of the desired qualifications for the role is: "Belief that Python packaging problems are of critical importance for the Python language... but that those problems are solvable."
We've gotten better and better at attracting corporate and grant funding -- and yes, I'll take some credit for that, with my past work researching and writing grant proposals, leading funded projects, and volunteering with the Packaging Working Group and cofounding the Project Funding Working Group. So, now, what should we focus on? We need to prioritize improvements for strategic value (e.g., should we first concentrate on overhauling the Warehouse API, or making a generic wheel-builder service, or tightening metadata compliance, or ....?). What can we learn from other package management toolchains, especially those that emerged after PyPI and pip (e.g., yarn, npm, cargo), and what should we copy? In my opinion, you do not need to already have an opinion on these questions to apply for this role -- you just have to be interested in talking with a bunch of stakeholders, poking through past discussions, and collaboratively developing some answers.
I won't be applying for this PSF role -- I'm going to be, instead, excited to collaborate with that person and help them learn all the stuff I know, so that in the long run, we'll have more people, with that set of skills and domain knowledge, working on Python packaging. I'll concentrate on the Python supply chain security piece specifically (via the NSF-funded work at NYU), plus finishing my book and maybe creating and leading associated trainings, and taking what I've learned to other languages and ecosystems through client work.
So: please spread the word and apply!
# (2) 13 Apr 2021, 04:16PM: Trying to Notice What's Missing:
I'm ploughing through some open source project email threads and thinking:
In 2010, people got together in Berlin for a Wikimedia developers' meeting .... and then a bunch of them hung around a lot longer than they'd expected, because a volcano erupted and so their flights got cancelled. As I understand it, you can trace certain architectural decisions and improvements to the discussions and pair programming from that chunk of unexpected extra in-person time.
It's conference season, at least in the northern hemisphere, and we're going into our second year of virtualized or missing technology conferences. The maintainers, users, and stakeholders of the open source software you depend on have gone more than a year without getting to quietly gossip with each other over a snack or while walking to a sponsored party. It's been more than a year since one guy has been able to make that other guy laugh and remember "ah, he's not so bad really". It's been more than a year since people could easily scribble boxes and arrows together on the back of a conference schedule or poke at the demo on someone's laptop.
We come together every once in a while to refill on trust and camaraderie and a shared understanding of what we're trying to do and who we're trying to do it for; I assume that, for some folks, those wells have now run dry.
In a tree's rings you can see the years of drought. Where, in our code and our conversations, will we see the record of this separation? Do you already see it?
# 30 Sep 2020, 12:24PM: Changes Coming To Pip In October 2020:
People who deal with Python: Changes are coming to pip, Python's package installation tool, in October 2020. Please share this migration guide and our video with your circles.
I'm working on improving the Python packaging toolchain, foundational work that will (in the long run) make the whole Python experience way less confusing. In the short term this may mess with some people's workflows, so we want lots of people to hear about it now.
The pip team made a 2-minute video to explain what's up:
We are also doing user experience studies, and want you to sign up if you ever do anything with Python (whatever your level of skill/experience).
Please boost this toot or retweet this tweet if you want to help us get the word out.
Computers need to know the right order to install pieces of software ("to install x, you need to install y first"). So, when Python programmers share software, like when they publish packages on the Python Package Index or internally in large companies, they have to precisely describe those installation prerequisites. And then pip needs to navigate tricky situations when it gets conflicting instructions.
Up until now, pip's been very inconsistent in handling this stuff, which makes it easy for your Python environment to get messed up. That's why we successfully applied for $407K in funding from Mozilla and the Chan Zuckerberg Initiative to finish and roll out a proper dependency resolver for pip. The goal is that pip will get better at handling that tricky logic, and easier for you to use and troubleshoot.
You can test the new behavior (in beta) right now by using an optional flag in pip 20.2. And in pip 20.3, coming in October, the new behavior will be the default.
Once you're using the new resolver, pip is going to be stricter and more consistent. So things won't mysteriously break as much, and we can add more features that lots of people want.
But! Right now, a ton of people unknowingly have Jenga towers of wobbly dependencies in their environments and will run into pain when we make the resolver stricter and more consistent. And this may lead to you getting stuck in troubleshooting, assuming that pip caused the problem, when actually the deeper cause is conflicts among how your upstreams specify requirements (TensorFlow just fixed a related thing, for example).
So: We're trying to get Python users to try out the beta of the new resolver that's available in the current stable release of pip (20.2), fix your own environments, report bugs in your upstreams in advance, and report bugs to us so we can fix them in the next couple weeks. We started spreading the word about this a few months ago. And now: video! People watch videos, I hear? I hope this helps.
# 08 Sep 2020, 10:13PM: Breaking Release Bottlenecks -- What Changeset Can Do:
I did some volunteer work earlier this year, helping rejuvenate pipenv (a command-line tool that some people use to help handle Python packages they make and use). Here's what I did, how long it took, and how you can do the same.
Pipenv's maintainers had not released a new version since November 2018, and users were concerned (in many cases switching to competitors). In early March of this year, someone suggested that perhaps the official Python Packaging User Guide should stop recommending it. I saw that suggestion and went into the relevant Internet Relay Chat (IRC) channel to nudge one of pipenv's maintainers and to ask: what do you need? What's blocking you?
Dan Ryan ("techalchemy") knew what was blocking him:
techalchemy: if you're purely evaluating 'how do we release the code', yeah I might just be the main roadblock?
techalchemy: someone to yell at me to stop doing things that are not related to the goal?
sumanah: let us assume that a successful release -- even as a pre-alpha -- is something that does not instantly break every user's life
techalchemy: yeah longer term planning though would require tech writing for sure and onboarding help, god do I struggle with that
techalchemy: have you heard me explain something...
sumanah: if you JUST want someone to yell at you to stop doing those unrelated things, just for about a month, then that can be cheaper .... would you actually _listen_ to that person?
techalchemy: historically speaking, I'd insist I was doing something important briefly but probably reassess, I do know what needs to happen
sumanah: :-) ok. So, how frequently do you need those checkins? like, 4 times a day, 5 days a week?
techalchemy: hopefully not that much, but I could see a few checkins being helpful especially if we were also onboarding some new people
sumanah: techalchemy: ~10 minutes of conversation, via IRC, 4 times a day, 5 days a week, for 3 weeks....
That would have worked out to about ten hours. We underestimated how long it would take Dan to address some nasty continuous integration bugs, so instead of three weeks it took three months. Over those months, I donated about 15 hours of my time, helping him release two betas, then a stable version in May. Given my current hourly rates, this constitutes a donation worth a few thousand dollars, which is infinitesimal compared to the value unlocked by expediting a pipenv release.
Dan needed someone to help him with prioritization, release management, and communications. So I:
Phil Gyford noticed that initial IRC conversation and said:
It's kind of fascinating as an example of bottlenecks in open source development and the importance of project managers.
And Yoz Grahame replied:
I regularly have conversations like this, and it's a toss-up as to which of the two roles I play.
Yeah. An external perspective can help a lot. And, ideally, a project manager is a supportive accountability partner who helps with the bits you're not great at.
If you/your company depends on an open source project and you're getting annoyed or worried because the release cadence has slowed to a standstill, there's a strong chance you can turn that around. If someone on your team can spend a few hours complementing the existing maintainers and helping unblock them, that could save you a bundle compared to forking or switching dependencies. Try talking with the maintainers about what they need. And I do mean talking, as in, synchronous conversation via voice or chat, so you can build some trust and get the kind of conversation you see in the IRC log above.
Or: contact Changeset Consulting for a free initial 30-minute chat. Maybe it'll only cost you a few thousand dollars to get that bottleneck unblocked. Let's find out!
# 28 Aug 2020, 04:20PM: Apply For Grants To Fund Open Source Work, and Career Thoughts:
Apply For Grants To Fund Open Source Work
When I tell people about grants they could get to help fund work on open source software projects, sometimes they are surprised because they didn't know such grants existed. Therefore, a month ago, I delivered a ten-minute PyOhio talk in which I shared:
I've now posted a transcript (with slides), and the ten-minute video is up.
The Python context, and followup Q&A
Have some additional thoughts and info I could not fit into the talk!
Background and ideas: People who work on Python's packaging tools usually aren't paid to do so. It's usually but not always a side project. We have gotten grants (or similar funds) of $80,000 USD or $150K or $207K and they are a huge reason why we
- have the new PyPI (Warehouse) in production, instead of the legacy site
- have 2-factor auth on Warehouse
- are much closer to releasing the new pip dependency resolver
On March 22, 2019, I started a wiki page to list some fundable projects in Python packaging. I figured that if we had a structured, public list of well-scoped, shovel-ready tasks that would move faster with funding, it would make it easier to find grants, sponsors, and other directed funds. And, heads-up, corporate readers: if you can help the PSF get some money to do these things, they are much more likely to happen, and Python will get much easier to deal with.
We've now moved that list to GitHub. If you know that filling a particular gap would improve Python packaging and distribution, go ahead and make a pull request!
And that doesn't just go for packaging. The Project Funding Working Group is seeking project ideas on a wider scale. If you make a well-structured list of fundable TODOs for your Python project, it can go in that list -- this will, in the future, make it easier for you to get funding.
Eligibility and how the money flows: often, only public charities or similar institutions are eligible to apply for certain kinds of funds, which is why getting the Python Software Foundation involved is helpful -- they are often eligible to apply in cases where you as an individual are not eligible. The way we've been doing it for the past few years is: some volunteers research a grant opportunity and write a proposal (often with the assistance of a PSF staffer) and the Python Software Foundation submits it. If it gets funded, the PSF hires contractors to do the proposed work, and then those contractors perform the proposed work and (via the PSF) report back to the funder. I cofounded the new Project Funding Working Group to centralize those efforts, bring enthusiastic volunteers together, teach more people to do this stuff, and capitalize on the momentum of the last few years.
We know some things about what some funders are seeking, and want to help match you up with funders who might be a good match. Depending on where you live, there may be country-specific grants that the existing members of the Project Funding Working Group know less about! Like, there is the Prototype Fund for people in Germany, and Innovation Fund Denmark, and there are a bunch of European Union grant opportunities that I know very little about like Horizon 2020.
Erika Owens asked some followup questions:
...how to assess if a grant is worth applying for - how do you know the foundation is legit? how do you balance the amount of time proposing/reporting with the likely grant amount?....
also, from a documentation mindset—where can you go for help with this type of work? what public writing is useful to others? (and possible given so much secrecy with funding)
When is it worth applying? & more: Different projects need different amounts of money. A one-time gift of $10K (about 60-100 hours of work at USD$100-150/hour) is not enough to make a significant dent in some of the listed fundable packaging tasks, but $10K could support a "get the next release out and clean up docs/bugtracker/patch queue" effort for your favorite library. This is why the Project Funding Working Group is trying to amass informational resources (a list of funders, "how to write a proposal", etc.) and point a lot of people at those resources so many people can self-serve -- the volunteers in the Working Group do not have time to write a dozen grant proposals from scratch in a year, each of which is for $10-30K. It might take 10-20 hours to research and draft a multi-page grant proposal from scratch (it gets easier when you can copy and paste from previous proposals or planning documents for the same software project). Sometimes it takes longer if a bunch of stakeholders (such as project maintainers) have to agree on priorities and scope. I hope that the working group writes a few proposals each year and that they're for at least $80,000 each, and that we advise or otherwise help a bunch of other volunteers to write grants for varied amounts. Let's see how the first year goes. Maybe I'll be wildly off.
I feel pretty lucky in that most of the grant-and-similar-funded projects I have worked on had fairly light reporting requirements, things that ended up taking maybe an hour per month plus maybe three hours at the end of the project. (Then again, I prep along the way by collecting meeting notes and updates on a public wiki page.) At the end of this year I need to make a fairly substantive report to the Chan Zuckerberg Initiative, so I may eat my words then!
What public writing is useful to others? I see people sharing actual grant proposals and budgets, and the Project Funding WG will too (soon), and I think that helps a lot. I bet a before-and-after of a proposal before and after editing would also be helpful. Timelines like this slide here (from my PyOhio talk) help others set expectations for the process. Project retrospectives like this Read the Docs post -- ditto. Lists of funders including real talk about what they're looking for. Reminders of deadlines (like: apply for the Better Scientific Software Fellowship by September 30th). If you have more suggestions, please open an issue!
How do you know whether the foundation is legit? Hmmm. I have confidence that I could tell whether a funder was fishy before getting into anything I'd regret, but I am having a hard time articulating the evaluation criteria I would use. Something to work on.
And where can you go for help with this kind of work? I know I ought to learn more from the world of professional grantwriters and other nonprofit experts, like Candid, and perhaps I will have a spurt of energy sometime soon to go dive in.
This week, on the way to posting this, I rearranged my conference talks page to be more navigable. I have given 35-50 talks on tech-related topics in the past 10 years, depending how you count. I used to talk a lot about Wikimedia, then about HTTP, then Python packaging and the plays.
The response to my one talk about grants has already been strong and, if I just reacted to that, I'd give more talks about grants. Already my blog posts and talk about grants have led to Changeset Consulting client leads and some client work. And there's a logic here -- I have succeeded in doing something lucrative that other people would like to replicate.
And it can be easy to get sucked into grantwriting (the work of researching and creating grant proposals), because I am clearly expert enough to be helpful, there are deadlines to motivate me and colleagues, and every available grant is an explicit invitation with a concrete amount of money attached. It's a trail others have already blazed.
But grantwriting is a treadmill that ties projects to rich funders with short timescales, a topic ably covered by Nonprofit AF, The Revolution Will Not Be Funded: Beyond the Non-Profit Industrial Complex, and many others. As with recruiting, or diversity/equity/inclusion work, there's a lot of toil here.
So, I am excited about the potential of the Project Funding WG, which will help many open source groups pursue not just grants but "external grants and similar funding." Including talking with companies who want to sponsor particular improvements, or generally sustain their dependencies (via Tidelift or a direct gift) and avoid the headache of forking or switching.
And I'm open to Changeset Consulting doing some paid consulting work on grantwriting, and I'm doing some unpaid grantwriting to garner funding for projects where Changeset would do paid work. And I'm particularly interested in joining together with folks who are making big proposals to big foundation-style or government funders, upwards of USD$500K or in the millions.
But, in the near future, I mostly want Changeset to work on paid projects, funded by for-profit companies, to rejuvenate and level up open source projects that they depend on but do not control. And that's looking promising. I'm using many of the same skills you use in grantwriting, because they're sales skills. And I am looking forward to blogging about one of these projects soon.
# 27 Apr 2020, 06:00PM: Remote Sprint Tips:
Every year, many developers of Python (the language itself, not just stuff written in Python) get together for a sprint. This year it will probably be virtual. How should that work? I offered to share my experiences and tips, the folks in the core development group asked me to do so, and I listed some tips. My approach is less "top-down schedule" and more "here's how to adapt to and support the emergent ways people will act".
# 21 Apr 2020, 12:18PM: Persisting:
Today is our wedding anniversary. Instead of going out for a nice dinner, we'll .... do something at home. Maybe we'll remember a bunch of nice memories from the last fourteen years. Maybe we'll go through the Anniversary Gifts bot output and see if there's something we can make at home.
I've now sewn three fabric facemasks. For fabric I used old tech company tee shirts. For ties: elastic from free airline eyemasks, shoelace-like handles from fancy shopping bags, and the hemmed bits of the tee shirts. All of them are serviceable. I'll be trying to improve and, if I can get better, give some away.
We used this approach to gather and grow yeast using raisins, sugar, and water in a jar on a windowsill. Today Leonard's using it to make bread. We have some powdered dried yeast but are trying to save it. And we've been growing green onions in some jars of water on another windowsill. Their stalks keep pushing out new green growth. The most successful watercolor painting I've done so far is a portrayal of one of those bunches.
The pip 20.1b1 beta release is out. And Python 2.7.18 is out marking the very last, final, release of Python 2.7 and the end of the 2.x era. My household contributed to both of these things. Here's Leonard's pull request that adds an informational banner to the 2.7 docs. When I can concentrate on work or exercise or media it's better. The news is awful. I try to only listen to or look at it a few minutes per day.
There is light through the windows, along with the rain and lightning, and I see the tree branches in the wind, falling and rising, falling and rising. Every night at 7pm I know it's 7 because people start clapping and ringing bells from our windows and balconies, a gesture of support for the health care workers and all the other essential workers who are trying to keep us all going. I do it too. The other night I got out a little temple bell and started using that. Someone has a tambourine. A few nights ago someone started chanting "USA! USA!" and I recoiled; as I joked to a few friends, better to chant "South! Korea!" or "Germany!" since they're actually doing it right. And someone else has, a few times, played a recording of "The Star-Spangled Banner". As I mentioned to friends: well, the thing that works about that song now is that it's a question. Does that banner still wave? We don't know!
I also have joked: And is this the land of the brave and the home of the free, or the land of the scared and the home of the at home? But it's all those things, of course. And the rhetoric of that joke, as though you cannot be both at home and free, plays into the hands of foolish, even malicious shouters who prefer to swan around shedding and catching viruses, and to mob streets while braying about government restrictions, and refuse to love their neighbors.
I'm glad of the rain. It feels natural to be inside when it's raining.
# (1) 29 Jan 2020, 12:12PM: MOSS Video, BSSw Honorable Mention, and The Maintainership Book I Am Writing:
Mozilla interviewed me about the Python Package Index (PyPI), a USD$170,000 Mozilla Open Source Support award I helped the Python Software Foundation get in 2017, and how we used that money to revamp PyPI and drive it forward in 2017 and 2018.
From that interview, they condensed a video (2 minutes, 14 seconds) featuring, for instance, slo-mo footage of me making air quotes. Their tweet calls me "a driving force behind" PyPI, and given how many people were working on it way before I was, that's quite a compliment!
I will put a transcript in the comments of this blog post.
(Please note that they massively condensed this video from 30+ minutes of interview. In the video, I say, "the site got popular before the code got good". In the interview, I did not just say that without acknowledging the tremendous effort of past volunteers who worked on the previous iteration of PyPI and kept the site going through massive infrastructure challenges, but that's been edited (for brevity, I assume).)
This video is the first in a series meant to encourage people to apply for MOSS funding. I mentioned MOSS in my grants roundup last month. If you want to figure out whether to apply for MOSS funding for your open source software project, and you need help, ping me for a free 20-minute chat or phone call and I can give you some quick advice. (Offer limited in case literally a hundred people contact me, which is unlikely.)
The Better Scientific Software (BSSw) Fellowship Program "gives recognition and funding to leaders and advocates of high-quality scientific software." I'm one of three Honorable Mentions for 2020.
The main goal of the BSSw Fellowship program is to foster and promote practices, processes, and tools to improve developer productivity and software sustainability of scientific code. We also anticipate accumulating a growing community of BSSw Fellowship alums who can serve as leaders, mentors, and consultants to increase the visibility of those involved in scientific software production and sustainability in the pursuit of scientific discovery.
That's why I'll be at
the Exascale Computing Project Annual Meeting next week in Houston, so if you're there, I hope to meet you. In particular I'd like to meet the leaders of open source projects who want help streamlining contribution processes, growing more maintainers, managing communications with stakeholders, participating in internship projects like Google Summer of Code and Outreachy, expediting releases, and getting more out of hackathons. My consulting firm provides these services, and at ECPAM I can give you some free advice.
And here's the project I'm working on -- why I received this honor.
In 2020, I am writing the first draft of a book teaching the skills open source software maintainers need, aimed at those working scientists and other contributors who have never managed public-facing projects before.
More than developer time, maintainership -- coordination, leadership, and management -- is a bottleneck in software sustainability. The lack of skilled managers is a huge blocker to the sustainability of Free/Libre Open Source Software (FLOSS) infrastructure.
Many FLOSS project maintainers lack management experience and skill. This textbook/self-help guide for new and current maintainers of existing projects ("brownfield projects") will focus on teaching specific project management skills in the context of FLOSS. This will provide scalable guidance, enabling existing FLOSS contributors to become more effective maintainers.
Existing "how to run a FLOSS project" documentation (such as Karl Fogel's Producing Open Source Software) addresses fresh-start "greenfield" projects rather than more common "brownfield", and doesn't teach specific project management skills (e.g., getting to know a team, creating roadmaps, running asynchronous meetings, managing budgets, and writing email memos). Existing educational pathways for scientists and developers (The Carpentries, internships and code schools) don't cover FLOSS-specific management skills.
So I'm writing a sequel to Karl's book -- with his blessing -- and I'm excited to see how I can more scalably share the lessons I've learned in more than a decade of leading open source projects.
I don't yet have a full outline, a publisher, or a length in mind. I'll be posting more here as I grow my plans. Thanks to BSSw and all my colleagues and friends who have encouraged me.
# 12 Nov 2019, 06:25PM: My New Title, Improving pip, Availability For Work, And SSL (No, The Other One):
A few professional announcements.
Seeking developers for paid contract on pip; apply by Nov. 22
One is that I helped the Packaging Working Group of the Python Software Foundation get funding for a long-needed improvement to pip. I led the writing of a few proposals -- grantwriting, to oversimplify -- and, starting possibly as soon as next month, contractors will start work. As Dustin Ingram explains:
Big news: the Python Packaging Working Group has secured >$400K in grants from multiple funders (TBA) to improve one of the most fundamental parts of pip: its dependency resolver. https://pyfound.blogspot.com/2019/11/seeking-developers-for-paid-contract.html
The dependency resolver is the algorithm which takes multiple constrained requirements (e.g. "some_package>=1.0,<=2.0") and finds a version of all dependencies (and sub-dependencies) which satisfy all the constraints.
Right now, pip's resolver mostly works for most use cases... However the algorithm it uses is naïve, and isn't always guaranteed to produce an optimal (or correct) result.
These funds will pay multiple developers to work on completing the design, implementation and rollout of this new dependency resolver for pip, finally closing issue #988.
Not only will this give pip a better resolver, but it will "enable us to untangle pip’s internals from the resolver, enabling pip to share code for dependency resolution with other packaging tooling". https://pradyunsg.me/blog/2019/06/23/oss-update-1/
This is great news for pip and Python packaging in general. Huge shout out to @pradyunsg for his existing work on the resolver issue and guidance here, and to @brainwane for all her tireless work acquiring and directing funding for Python projects.
If you or your organization is interested in participating in this project, we've just posted the RFP, which includes instructions for submitting proposals, evaluation criteria and scope of work.
If you're interested, please apply by 22 November.
NYU, Secure Systems Lab, and my new title
In further news: I am now a visiting scholar in Professor Justin Cappos's Secure Systems Lab at New York University's Tandon School of Engineering. And I get to use an office with a door, shelves, whiteboards, and so on (per the picture at right). If you contribute to Python packaging/distribution tools and live in/near or sometimes visit New York City, let me know and perhaps we could cowork a bit?
The Secure Systems Lab stewards The Update Framework (TUF) and related projects, and works to improve the security of the software supply chain. The Python Package Index is likely going to implement TUF to add cryptographic signatures to packages on PyPI, and so I've gotten to give TUF's developers some advice to help that work move along. (I won't be the manager on that project but I'll be watching with great interest.) PyPA may also choose to use more of SSL's work in implementing further security improvements to the package distribution toolchain, and I'm learning more to work out whether and how that could happen. Also, Cappos's research on backtracking dependency resolvers has been helpful to the pip resolver work.
Edited 19 Nov 2019 to clarify role.
I'm grateful to get to help connect the Python Software Foundation with more resources and volunteers. Changeset's current and recent projects have mostly been for the PSF. Last month we finished accessibility, security, and internationalization work on PyPI that was funded by the Open Technology Fund, and Changeset's work on communicating about the sunsetting of Python 2.x continues and will go through April 2020.
Availability for one-day engagements in San Francisco in February
But I am interested in taking on new clients for short engagements starting in February 2020. In particular, I will be in the San Francisco Bay Area in mid- to late February. If you're in SF or nearby, I could offer you a one-day engagement doing one of the following:
- developing a contributor outreach/intake strategy
- researching potential funders and writing a rough draft of a grant proposal
- auditing and improving your developer onboarding documents
I'd spend a little time talking with you, then sit in your office and finish the document before leaving that afternoon. (Photo at right provides a sample of how I look while sitting.) Drop me a line for a free initial 30-minute chat and we can talk pricing.
# 22 Sep 2019, 02:03PM: Futureproofing Your Python Tools:
The people who maintain Python and key Python platforms want to help you protect the code you write and depend on.
If you write software in Python, or depend on something that's in Python, this is for you.
Some of you are writing Python 2, or you still have software you depend on that is written in Python 2. January 1st, 2020, is the day that official support for Python 2 stops. So this is a fresh heads-up that you should really have a migration plan and start working on it, to move to Python 3. A lot of stuff you depend on already works on Python 3, and is even pledging to end Python 2 support in or before 2020. And it's easier than it's ever been to port your own code from 2 to 3.
You should probably upgrade to Python 3.7. If you want to test out 3.8, it has some changes in how it does warnings, and the first release candidate is out.
But, speaking of futureproofing:
Code authors move in and out of projects and companies. Six months or 18 months later, maybe you want to update and re-use, re-release or re-deploy code someone else wrote. Or you want your team to be able to reuse what you did after you leave, which means you need the code to run, and you need them to have the password so they can update stuff.
Have you written Python code that they have published as a package on the Python Package Index, pypi.org, so other people can use pip install to install it?
(And if you want to do that but don't know how? Check out this recently improved tutorial to help you do that.)
Publishing that package is a great way of making it so other people can run and deploy it, even within other parts of your organization.
But -- who actually has the keys to the castle? Who can upload a new version, or delete a version that has a problem?
You should probably make sure multiple people have either "owner" or "maintainer" privileges on the project on PyPI.
And you should review your project security history display, which lists sensitive events (such as "file removed from release version 1.0.1") in your PyPI user account and your PyPI project. We just added this display, so you can look at things that have happened in your user account or project, and check for signs someone's stolen your credentials.
And then how do you make it a little harder for vandals, spammers, and thieves to take over your account and upload malware or delete all the packages? Add two-factor auth for your login, like you would with your bank. Use an app on your phone to give you a six-digit code, or use a physical security device like a Yubikey.
And how do you make it easier to automate publishing new versions of your package, and make it safer to save your credential in the cloud? We've made it possible for you to create an "API token" where all it can do is upload, so you can use that instead of your PyPI password.
With well-tested Python 3 migration tools and new PyPI security features, now is a great moment to invest in robustness for Python software that you make or depend on.
[This blog post is kind of unwieldy, because it's about too many different things. I won't be publicizing it that much and instead will probably reuse text from it in more focused announcements elsewhere. But I'm publishing it here as a summary of my recent work, because management and communications for the projects above are what I've been working on recently for the Python Software Foundation. (A different kind of summary is on the Clients page for Changeset Consulting.)]
# 05 Aug 2019, 09:01AM: Kickoff for Communications Work on the Python 2 Sunsetting:
Python's 2.x line will reach End of Life on January 1, 2020, meaning that the maintainers of Python 2 will stop supporting it, even for security patches. Many institutions and codebases have not yet ported their code from Python 2 to Python 3. And many of them haven't even heard yet about the upcoming EOL. Volunteers have made many resources to help publicize and educate, but there's still more work to be done.
So the Python Software Foundation has contracted with my firm, Changeset Consulting, to help communicate about the sunsetting of Python 2. The high-level goal for Changeset's involvement is to help users through the end of the transition, help with communication so volunteers are not overwhelmed, and help update public-facing assets so core developers are not overwhelmed.
During this project (starting now and going till the end of April 2020), Changeset's goals are:
- Create a concise page on python.org giving community guidance on the why and what of EOL
- Create and publicize a formal press release through the PSF
- Audit and update every mention of Python 2 on python.org, including in documentation and the wiki
- Develop a plan to handle redirecting https://docs.python.org/2/* (especially deep links)
- Publicize the new "what's up with EOL" page to a variety of audiences, respond to questions, and keep the page updated until PyCon US 2020 based on questions that come up
So, towards those goals, you'll see me with my colleagues:
- Researching, writing, and editing technical documents and the press release
- Corresponding with stakeholders, writing and publishing announcements and other communications in multiple media
- Initiating and facilitating meetings (I will try to make them very minimal and short; if I invite you, please let me know your response)
- Filing, triaging, labelling, prioritizing, and linking issues, such as in the website repo
For accountability, Changeset will provide reporting via:
I'm going to be asking for a lot of help along the way from the Python community: meeting with us, answering our questions, double-checking our drafts for accuracy, publicizing that EOL page to your circles, setting up some parties for January 1st. Thanks in advance and let's get the Python user base further along towards enjoying the shininess of Python 3.
# 22 Jul 2019, 11:35AM: Beautiful Soup is on Tidelift:
I've been doing a tiny bit of consulting for Tidelift for a little over a year now, mainly talking about them to open source maintainers in the Python world and vice versa. (See my October 2018 piece "Tidelift Is Paying Maintainers And, Potentially, Fixing the Economics of an Industry".) And lo, in my household, my spouse Leonard Richardson has signed up as a lifter for Beautiful Soup, his library that helps you with screen-scraping projects.
There was a period of about a year in 2017-2018 when I wasn't interested in doing Beautiful Soup work, but Tidelift changed that. Tidelift gathers subscription money from companies that rely on free software, and distributes the money to the developers in exchange for a level of support that I find sustainable.
Nobody builds an entire product around Beautiful Soup (or at least nobody will admit do doing this), but thousands of people have used Beautiful Soup to save time at their day jobs. Bundling Beautiful Soup together with bigger projects like Flask and numpy is a solution that works really well for me.
The other day I looked at the list of featured supported packages and was happy to see that a bunch of Python projects have signed up as lifters, including SciPy and numpy, Flask, setuptools, Werkzeug, websockets, urllib3, celery, coverage, a bunch of Django packages, Jinja2, keyring, Pylint, coverage, and pytest. And I think over the next 6-12 months we're going to see some effects of Tidelift support -- not just in the security and release cadences of the supported projects, but on other issues stemming from unfairness and a lack of reciprocity in open source, like maintainer burnout and expectation-setting. The list of licensing, security, maintenance, and marketing tasks lifters agree to do may end up being a benchmark, like the open source Independent Verification & Validation checklist by Open Tech Strategies, that even non-lifter maintainers use to set realistic expectations for what "supported"/"maintained" means.
# 16 Apr 2019, 02:36PM: PyCon NA, !!Con, and WisCon:
I'm back in New York City, and preparing to travel a bit; May is my big conference month this year.
I'll start the month at PyCon North America in Cleveland, Ohio, practically the whole conference, 1-9 May. I'm co-organizing The Art of Python, an arts festival -- several short plays, plus a fanvid and live music -- the night of Friday, May 3rd. And during the sprints, 6-9 May, I'll be concentrating on the world of Python packaging and distribution, e.g., PyPI.
I'll go home to New York City, then go to !!Con 11-12 May, where I am not organizing or speaking or suchlike.
And then I'll be at WisCon in Madison, Wisconsin, for the whole convention, 24-27 May plus a little extra on either side. I will once again be the comedy auctioneer for the auction on Saturday night that benefits the Tiptree Award -- if you have something to donate to the auction, please let us know by 15 May. I may not make it to the Floomp or the vid party. I will probably be on a few panels; several panels are still seeking volunteer panellists (sign up by 19 April). I do plan to be at the Gathering and the Dessert Salon (heads-up, changes to Dessert Salon entry flow).
If you're going to be at any of these events, perhaps we can share a beverage! If you want to make sure that we do that, let's actually set up at least a tentative appointment soon, so I can put it in my calendar.
I will be more responsive to emails and text messages than to social media while at these conferences, and in particular, I may see mentions and direct messages on Mastodon but I probably won't see mentions or direct messages on Twitter. Also, I am pretty forgiving about being called mispronunciations of my name, but here's a recording in case you want help -- I also respond to "Vikki" which is the fake name I use with strangers at restaurants. And I will probably not hug you unless we know each other well.
# 03 Oct 2018, 05:16PM: Tidelift Is Paying Maintainers And, Potentially, Fixing the Economics of an Industry:
As the founder of Changeset Consulting, I keep my eye on consultancies and services in and near my niche, open source leadership, maintainership, and sustainability.* And I've known Luis Villa for years and got to work with him at Wikimedia. So yeah, I noticed when Tidelift announced its big new launch. And -- now, as a very-part-time consultant who helps Tidelift understand the Python world -- I am excited about their commitment to pay more than USD$1 million to maintainers (including "a guaranteed minimum $10,000 over the next 24 months to select projects").
Here's my take on the new Tidelift subscription model, the "lifter" role, and whom this works for.
For software businesses, this provides that missing vendor relationship, SLA, release cadence expectations, and general peace of mind for all of that unseen infrastructure you depend on. It's often easier for businesses -- of many sizes -- to pay a regular fee than to put open source project management work, dependency-updating, compliance checking, dependency security audits, or FLOSS volunteer relations on the engineering schedule.
For individual programmers and community-maintained open source projects, Tidelift is a potential source of substantial income. As a Pythonist, I hope to reach people who are currently core code contributors to open source projects in Python, especially on the Libraries.io digital infrastructure/unseen infrastructure/improve the bus factor lists. And I would like to reach projects like the ones Nathaniel Smith calls out in a recent post:
that (1) require a modest but non-trivial amount of sustained, focused attention, and (2) have an impact that is large, but broad and diffuse
and projects in the "wide open", "specialty library", and "upstream dependency" categories identified by the Open Tech Strategies report "Open Source Archetypes: A Framework For Purposeful Open Source".
For such people and projects, becoming a lifter is a promising model -- especially since the required tasks are fairly few, and are things maintainers should do anyway. I'm encouraged to see Jeff Forcier (maintainer of Fabric, Alabaster, and more) and Ned Batchelder's coverage.py getting onto the Tidelift platform.
And you can see estimated monthly income for your package right now. For some people, especially those whose healthcare doesn't depend on an employer, Tidelift payments plus some side consulting could be a sustainable, comfortable income.
Then there are folks like me whose contributions are only partially visible in commit logs (management, user support, testing, and so on), and groups that work together best as a team. Tidelift is also a potential source of income for us, but it's a little more complicated. Tidelift can send lifter payments to individuals, for-profits, and nonprofits, but: "If a package has multiple co-maintainers, you'll need to agree as a group on an approach." If you thought code of conduct conversations with your community were uncomfortable, wait till you bring up money! But, more seriously: I've been able to talk frankly with open source colleagues about thorny "who gets paid what?" questions, and if you're candid with your co-maintainers, the benefits may be pretty substantial. You can get advice on this conversation during the next live Tidelift web-based Q&A, Thursday, Oct. 11 at 2 p.m. Eastern Time (sign up at the bottom of the lifter info page).
Nonprofits, companies, and working groups that maintain projects can sign up now as lifters. Even if it's just a trickle of money right now, it might build over time and turn into enough to fund travel for an in-person sprint, contract work to improve continuous integration, an Outreachy internship, etc.
(One gap here: right now, Tidelift isn't great at supporting system-level packages and projects, like tools that get installed via apt or yum/DNF. I'm pretty sure that's something they're working on.)
What about noncommercial users or users who can't afford Tidelift subscriptions? The more lifters and subscribers sign up, the more those users benefit, too. Subscribers' funding means maintainers have time to make improvements that help everyone. And lifters agree to follow security, maintenance, and licensing best practices that also help everyone. Plus, Tidelift stewards libraries.io, a great resource for anyone who uses or develops open source (more on that). More money for Tidelift could mean libraries.io gets better too.
So I'm tooting a horn here and hoping more people sign up, because this is one of the more plausible ways open source sustainability could possibly work. Tidelift could be a real game-changer for the industry. Check it out.
* Examples: new competitors like Maintainer Mountaineer and OpenTeam, new funders like OSS Capital, and colleagues/referrals like Open Tech Strategies, VM Brasseur, Otter Tech, and Authentic Engine.
# 29 Jul 2018, 03:02PM: "Python Grab Bag: A Set of Short Plays" Accepted for PyGotham 2018:
Fresh from the waitlist onto the schedule: Jason Owen and I will be performing "Python Grab Bag: A Set of Short Plays" at PyGotham in early October. If you want to come see us perform, you should probably register soon. I don't yet know whether we'll perform on Friday, Oct. 5 or Saturday, Oct. 6.
(The format will be similar to the format I used in "Lessons, Myths, and Lenses: What I Wish I'd Known in 1998" (video, partial notes), but some plays will be more elaborate and theatrical -- much more like our inspiration "The Infinite Wrench".)
To quote the session description:
A frenetic combination of educational and entertaining segments, as chosen by the audience! In between segments, audience members will shout out numbers from a menu, and we'll perform the selected segment. It may be a short monologue, it may be a play, it may be a physical demo, or it may be a tiny traditional conference talk.
Audience members should walk away with some additional understanding of the history of Python, knowledge of some tools and libraries available in the Python ecosystem, and some Python-related amusement.
So now Jason and I just have to find a director, write and memorize and rehearse and block probably 15-20 Python-related plays/songs?/dances?/presentations, acquire and set up some number of props, figure out lights and sound and visuals, possibly recruit volunteers to join us for a few bits, run some preview performances to see whether the lessons and jokes land, and perform our opening (also closing) performance. In 68 days.
(Simultaneously: I have three clients, and want to do my bit before the midterm elections, and work on a fairly major apartment-related project with Leonard, and and and and.)
Jason, thank you for the way your eyes lit up on the way back from PyCon when I mentioned this PyGotham session idea -- I think your enthusiasm will energize me when I'm feeling overwhelmed by the ambition of this project, and I predict I'll reciprocate the favor! PyGotham program committee & voters, thank you for your vote of confidence. Leonard, thanks in advance for your patience with me bouncing out of bed to write down a new idea, and probably running many painfully bad concepts past you. Future Sumana, it's gonna be ok. It will, possibly, be great. You're going to give that audience an experience they've never had before.
# 11 Apr 2018, 10:17PM: My LWN Story Summarizing PyPI's Overhaul:
This coming Monday, April 16th, we plan to flip the switch on the new PyPI and redirect https://pypi.python.org web browser requests and pip install requests so the codebase serving them is Warehouse (which is in beta right now at https://pypi.org). I'm proud of our team's work and hope you find it useful.
I haven't blogged here in a while, but I've been writing a lot, mostly announcements and explanations listed on, or a few hyperlinks away from, the onwiki index to my PyPI work. When I can't give people choices (and, unless your organization sets up a private package index/repository, PyPI can feel like the only game in town), I want to give them a lot of lead time to test, file bug reports, and migrate, and I want to provide backstory.
So: today LWN publishes a new article by me, "A new package index for Python". In it, I discuss security, policy, UX and developer experience changes in the 15+ years since PyPI's founding, new features (and deprecated old features) in Warehouse, and future plans. Plus: screenshots!
This summary should help occasional Python programmers understand why a new PyPI codebase is necessary, what's new, what features are going away, and what to expect in the near future.
If you aren't already a subscriber, you can use this subscriber link for the next week to read the article despite the LWN paywall. Thanks to LWN for the venue and the subscriber links, and thanks to Jake Edge in particular for thorough editing. Thanks to my Warehouse team for fact-checking me.
# 11 Mar 2018, 10:42AM: Recent Debugging And Confidence:
I am proud of myself for some recent debugging I've done on and with codebases and tools that I hadn't worked on before.
A few weeks ago, I was sitting next to a friend who co-maintains a web app and hadn't looked at it for a while. The styling was screwy. I asked whether some CSS or JS he depended on had upgraded, like jQuery or something. He said no, his site hosted all its dependencies. I opened up the site and checked the Network tab in Firefox Developer Tools and saw that it pulled in Bootstrap from a CDN. Ah, one of the other maintainers had added that! And updates to Bootstrap had screwed up the page's styling.
That same day, as a freshly minted co-maintainer of twine (a utility to upload packages to PyPI), I investigated a problem with our CHANGELOG. Twine's changelog, as represented on Read The Docs (example) and when I built the docs locally, only displayed version number 1.4.0 (2014-12-12) and two associated GitHub issues. This was inaccurate since the source file changelog.rst had 70+ items and ran up to version 1.9.1 (2017-05-27). I figured out that this was happening because changelog.rst is meant to be formatted so the Sphinx extension releases (which I hadn't used before) can parse it, and the current file wasn't syntactically (or semantically) adhering to releases's conventions. (Since then, with advice and help from some folks, I've released Twine 1.10.0 and started a new maintainer checklist.)
And then, a couple days later, I fixed my friends' blog. Their front page had reverted to a ten-year-old index page. I had never touched Movable Type before and hadn't used their particular managed hosting web GUI before, but I poked around (and checked for backups before changing anything) and managed to figure it out: during a May 2008 outage, someone had hand-made an index.shtml page, which was now overriding the index.html page in the server config. I figured it out and found and fixed it.
My mom says that when I was a kid, I took apart alarm clocks and spare hose attachments and so on, and put them back together just fine. She once came upon me taking something apart, and when she drew breath to admonish me, I said, "Amma, if I don't take it apart, how do I know what is inside? Don't worry, amma, I'm just looking at it, I'll put it back together when I'm done," and I did. She told me that I took apart a mechanical alarm clock, carefully spreading all the parts out on some newspaper, and put it back together, and it didn't quite work properly, so I took it apart again and then put it back together, and it worked, and I jumped for joy and said "I fixed it!" (I still feel that way when I fix something.)
At some point along the way I feel like I lost that calm confidence in my abilities, that "things are made of stuff" and what one person made another can fix. But I have it again, now, at least for some bits of software, and some purely mechanical stuff (yesterday, helping friends move, deciding to break down a big empty cardboard box, responding to "but it's so big, it won't fit on the stack" with "we have knives"). It doesn't feel courageous at the time, just sensible, but then I look back and feel like a badass.
If I had to point to the single biggest cause of this regained confidence, I'd point to the Recurse Center, where I got way more comfortable with bravery and failure in programming.
# 26 Jan 2018, 06:28PM: Preserving Threading In Google Group or Mailman Mailing List Replies with Thunderbird:
Have you ever wanted to reply to a mailing list post that wasn't in your inbox? I had that problem yesterday; here's how I fixed it.
Context: I'm the project manager for Warehouse, the software behind the new Python Package Index (PyPI) which -- thanks to funding from Mozilla and support from the Python Software Foundation -- is on its way to launching and replacing the old PyPI. I've been in the Python community for years, but -- just as when I went from "casual Wikipedian" to "Wikimedia Foundation staffer" -- I'm learning about lots of pockets of the Python community that I didn't yet know about. Specifically, Python packaging has a lot of different repositories and mailing lists. One of them is the Google Group pypa-dev, a mailing list for developers within the Python Packaging Authority.
I joined pypa-dev recently -- and, in skimming the archives, I found a months-old message I wanted to reply to while preserving threading (so that future folks and longtime subscribers would see the update in context). So I clicked on the dropdown menu in the upper right corner for that post and clicked "Show original", which got me the Message-ID header. But how could I get Thunderbird to let me write a reply with the appropriate In-Reply-To header? Preferably without having to install some extension to munge my headers?
This reply to a StackExchange answer got me most of the way there; the basic approach is the same whether you're working with a Google Group or a Mailman list. (If it's a Google Group or a Mailman 3 list, you can of course reply via the web interface, but maybe you want to cc someone or have the history in your Sent folder, or you just prefer composing in Thunderbird.)
- First, you need to get the raw text, so you can get the Message-ID.
If you're looking at a Google Group message (example), click on the dropdown menu in the upper right corner for that post and choose "Show original" (example), then click the "Show only message" button to get a raw text page like this.
If you're looking at a Mailman 2 message (example), then navigate to the monthly archive. You can get there by clicking on the "More information about the [name] list" link at the bottom of the page, which takes you to a list info page (example), and from there, the "Visit the [name] Archives." link (example). Here on the archives-by-month page, download the archive for the month that has the message (using the "[ Gzip'd Text [filesize] ]" link in the "Downloadable version" column). And now you can, for instance, gunzip 2018-January.txt.gz in your terminal to get 2018-January.txt which you can search to find the post you want to reply to.
If you're looking at a Mailman 3 message (example), look at the bottom of the left navbar for a "Download" button (hover text: "this thread in gzipped mbox format"). If you gunzip that you'll get a plain-text .mbox file which you can search to find the post you want to reply to.
- Now, no matter what mailing list software you had to wrangle, save the raw message as a temporary file with a .eml extension, e.g., /tmp/post.eml, to smooth the way for Thunderbird and your OS to think of this as a saved email message. If you're looking at a Mailman archive, this is where you select just that one message (headers and body) from the .txt or .mbox file and cut-and-paste it into a standalone .eml file.
- Open that file in Thunderbird: File menu, select Open, select Saved message, and navigate to /tmp/post.eml and open it.
- If all's gone well, the message pops open in its own window, complete with Reply and Reply All buttons! Go ahead and use those. Note that the From: and To: lines have been obfuscated or partially truncated to protect against spammers, so you'll probably need to fix those by hand, e.g., replacing at with @ and fixing any ellipses (...).
- Hit Send with the glow of thread-preservation satisfaction. Watch for your post to show up, properly threaded, in the list archives (example).
You can hire me through Changeset Consulting.
This work by Sumana Harihareswara is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.
Permissions beyond the scope of this license may be available by emailing the author at firstname.lastname@example.org.