Blog by Sumana Harihareswara, Changeset founder

21 Aug 2023, 11:35 a.m.

User Support, Equanimity, And Potential Cross-Project Tools and Practices in Open Source

I met David Lord at this year's PyCon, talking about maintainer burnout and how to make projects more resilient. Since then, we've been chatting regularly to help us develop material for our respective writing projects. David co-maintains Pallets, Flask, and other widely-used open source projects, and is documenting his practices to teach them to future maintainers. His 8+ years of maintainership experience give him a set of experiences and problems I'm grateful to learn from. And I'm writing my book on maintainership.

Springing from those conversations: a couple of related topics I've been thinking about:

User support frustration and the infrastructure of equanimity

One bit of conversation started as a refrain of the old song: why won't the users help us help them?!

Users who ask for support for free have a responsibility to do some pre-work themselves to solve problems first. When using Python packages as dependencies for their own high-priority work (or, as industry lingo puts it, "in production"), users ought to pin dependencies, and use some consistent process to carefully upgrade their direct and transitive dependencies in a way that balances stability and staying up-to-date. Also, by default when you're running Python (most of the time), deprecation warnings only show up if you explicitly ask for them. So, if a project maintainer has done the right thing by trying to warn you "hey! you're using a feature that will be removed in the next release! you should use this other method instead!" you have to be extra-careful to check for that. For instance, you could regularly use a testing suite like pytest and invoke the flag that turns warnings into errors.

So this got me thinking about the work necessary to promulgate these practices to our users, as things they ought to generally do, yeah, but especially as prerequisites for asking for free support. Some projects' maintainers are willing to spend time catering to the desires of users who want to avoid pinning their dependencies. But that makes it harder for the maintainers who do not have time to do that, because users develop unrealistic expectations that open source projects (as a whole) will support their use case at no cost.

Thus: would it be a good idea to build a solidarity movement among Python open source maintainers, developing a consistent minimal set of hygiene practices we ask users to take care of before we'll help them? This sounds -- to understate things tremendously -- like a difficult task. But it might be worthwhile anyway.

More generally: individual projects can clearly state and publicize policies that are meant to reduce the scope and difficulty of their user support workload, such as deprecations, supported environments, issue templates, codes of conduct, and so on. And many do. But enforcing those policies still requires labor and can be demoralizing.

And: every thriving project inherently will have to deal with some number of user misunderstandings, well-meant off-topic discussions, and ghostings in the middle of an otherwise promising back-and-forth. In my estimation, even with great automation and terrific "saved reply" boilerplate responses and attention-saving notification dashboards, an irreducible amount of labor is necessary, of a type that many of us find dispiriting over the long haul. It involves correcting people, and saying "no," and demonstrating maturity and respect even when others aren't, and so on.

It's just hard.

And it's not just us. These dynamics happen in every public-facing organization that actually tries to be accountable to the people it affects (governments, for instance). Imagine an elected official holding a local town hall on health care reform, or a healthcare-related resource and help fair: of course some number of the residents who show up will

  • ask for help with a problem with their driver's license, which these constituent services staffers can't help with
  • ask for help with a problem that's handled at a different level of government
  • ask for help with their healthcare -- but they haven't brought the relevant paperwork with them or they've forgotten important details, so the constituent services folks can't help them right away
  • demand support that is not feasible
  • criticize the official for a stance she does not even hold
  • try to turn the conversation to immigration reform

And those are problems of success. Successful outreach that gets people to show up will inevitably bring along people who have been reminded of adjacent questions, ideas, and needs. This is partly because most people don't mentally subdivide those things according to which government department works on them, but also because many people discover what they need or what they want to talk about iteratively, in the process of interactive conversation. And you've provided a stimulus for thought, and they bring it into the contexts of their wildly varying lives and respond with as many different (often skewed) assumptions as there are humans in the universe.

The very vitality that motivates a project and keeps it moving also arouses strangers' interest, and they come from different contexts, and most of them won't dance fluidly with you, and there'll be a lot of stubbed toes. Or, to switch metaphors: the sun and rain and soil that nourish your project inevitably also feed the spores and seeds that the wind brings; weeds are a sign of success. But that doesn't mean weeding is fun! It can be really tedious and draining.

And the dread of that burden can cause, in us, an aversion to the more public side of maintainership. If we know that every time we send out a release announcement we'll get some nonzero number of off-topic complaints, hard-to-comprehend bug reports, and bad-idea feature requests, then we will likely develop an aversion to making frequent releases. If the contradictory demands of different user constituencies lead to us feeling stuck in no-win arguments, and we think that's likely to happen every time we ask our users what they want, we'll likely get less interested in asking for their opinions.

So: given that: Can we, collectively, agree on practices and make tools to reduce the emotional drain of this irreducible minimum user response labor? How do we build the infrastructure of equanimity?

I'm talking about helping maintainers feel better in the face of the equivalent of the unruly town hall. Like, reading a user's complaint can feel bad, and telling a user "sorry, no, see the deprecation policy" can feel bad. Beyond fostering better atmospheres, beyond automation, what are the specific mindset shifts that can help a maintainer stay compassionate but also maintain equanimity? There's something more sophisticated here than the simple "care less/grow a thick skin" advice, and my intuition is that some parts of it will include:

  • Each maintainer has to seriously reflect on power and economics enough to develop their own analysis of their freedoms and obligations within open source, and the freedoms and obligations of others. This ought to be nuanced enough to differentiate between users who contribute to sustaining the project and users who do not, and to acknowledge that maintainers have power over users unless they actively work to have power with those users.
  • Each maintainer needs to genuinely reflect on their own personality. What do you want? Why are you doing this? What activities do you dread and endlessly put off? Have you been pretending to yourself that you will someday get around to them? Realistically, as you imagine the months and years to come, can you keep up the current pace of work? Are you taking enough breaks, including multi-day or multi-week vacations that let you reset your perspective and come back refreshed? What activities satisfy you or even energize you? Are you using open source activity to avoid thinking about difficult stuff in the rest of your life? What drew you to steward this project, and what does it mean to you to honor that responsibility?
  • Based on these two analyses, each maintainer needs to define (and iterate on) their limits, and strongly consider publicizing them to set expectations.
    These will probably include limits on their time spent on open source projects -- likely with different quotas for tasks that the person finds more energizing or more draining. Bounding the scope of work is a common practice in contracting work and it's a tool maintainers should use more often. (Perhaps: a maximum of 4 hours per week on direct user support, 6 hours of teaching/discussion time, and ten hours of programming. And you can set an exclusion on types of work you simply never want to do -- maybe you find in-person events unbearable, or never want to write a release announcement again. If you set that expectation publicly, you can make room for someone else to step up.)
    Another kind of limit: principles to guide the kinds of support one offers (e.g., mailing list, GitHub issues, plus twice-monthly live office hours on Twitch, but not social media, Stack Overflow, or Discord), and potentially offering different tiers of support for different kinds of users (e.g., "students and nonprofits get more help than profit-seeking companies and hobbyists" or "I will make extra time to help people from my region").
    And, collectively with co-maintainers, defining the scope of work for the project, and publicizing it -- "things we won't do" -- can head off some grumbles.
  • Collectively, we have to continue making and improving tools that make the work easier and that set expectations appropriately for all people interacting with maintainers. Some tool ideas follow!

Potential cross-project tools and practices

A shared blocklist, or at least a warning flag. Every maintainer seems to have to learn for themselves about particular users who consistently file annoying, useless bugs and do not otherwise productively contribute. Could there be a shared blocklist to help with this? Perhaps GitHub could tell you: "this person has been banned! A lot!"

(One reason this would help: when a maintainer needs to delete abusive comments from a thread or ban a user from their project, they often end up, as a side effect, removing the ability for an outside observer to understand the context for their moderation actions. This effect is magnified if the moderator is acting because of a pattern of behavior happening over a long period of time, perhaps in several different threads, fora, or projects. Since the end result of the moderation is often sharply visible, but the context is opaque or diffuse, audiences who expect transparency get confused and upset. I don't want administrators making utterly indefensible moderation decisions, and I do want them to be appropriately accountable for their power over others. But I would like to give administrators tools to help them prevent individuals whom we already know to be abusive and extremely uncooperative from leeching contributors' energy in the first place. And one beneficial effect of that is that we might have fewer "wait, why'd you ban him?" threads, which take significant maintainer facilitation to prevent from eating weeks of time and spiraling into emotionally draining flamewars.)

Boilerplate policies. Beyond licenses and codes of conduct, we could use a shared set of customizable policies on deprecation, security reporting, release announcement, vendoring, user support, contributing guidelines, testing, architectural change, and data privacy. We could also share new issue templates, and replies to reuse for common misunderstandings (like "you'll need to rebase" or "we're not able to prioritize this right now but a patch or a donation could change that").

Flagging old resources as old. Users searching for help are usually not starting by searching your own written documentation and only then moving on to others' resources. We all have to recognize that most of the documentation people find and use to learn about our projects is made, and controlled and maintained, by other people! This includes StackOverflow and Reddit and Discord threads, and TikTok and YouTube videos, and blog posts and mailing list archives and GitHub issues within other adjacent projects. This is particularly true if your project is an infrastructure layer that lots of other tools interoperate with -- Python packaging maintainers would love for users to first seek help at our canonical tutorial site, packaging.python.org, but it's an uphill battle against a decade of search engines ranking other pages higher. Users find now-obsolete blog posts and reference guides and don't check their publication date, and end up super confused.

As such: Heidi Waterhouse suggests that we programmatically make our wiki pages expire after a year to trim cruft and improve accuracy. I think this needs to go beyond wikis, and that -- in recognition of our interdependent information ecology -- we all ought to take some responsibility for at least signalling THIS IS OLD AND MAYBE OBSOLETE! archived how-tos that we host, as individuals and as projects.

What if we all programmatically added a "heads-up, this is old!" disclaimer on old blog posts? Chris Rose recently added one, inspired by me. Now that the start of Chris's 2015 "Using Docker to run tox without adding Pythons to your system" says:

Hi, reader. I wrote this in 2015. Technology referenced may have changed and I nearly certainly have. I may disagree now with what I wrote then, or it may refer to obsolete terms and tech. I will usually only edit old posts to refresh links that have changed. If this post is particularly offensive please contact me and let me know.

it's less likely that readers will get misled.

If you want to do this too, take a look at my slightly different wording, and at Jacob Kaplan Moss's wording, which I also like:

... It may be very out of date, partially or totally incorrect.... If something in this post is actively harmful or dangerous please get in touch and I'll fix it.

And: Google indexes old, closed issues and pull requests on GitHub. Which means that people who Google search for the text of an error message often first come across closed issues, 10+ years old, whose content is now misleadingly obsolete, if they're related at all to the actual current problem the user's having. And they comment, "I'm still having this problem!" or "what about this?" on it, and: headaches for everyone.

David addressed this with a bot that locks all closed issues, 2 weeks after the most recent comment, so that only trusted project collaborators can reopen/comment on it. It's important to communicate in the "locking this now" message that the bot is only locking old, closed, no-updates issues. This is not a "stalebot" which might lock open issues or pull requests after some period of inactivity; that provokes ire. David reports that this has significantly helped with notification load and increasing issue quality, and that he wishes this was the GitHub default.

Shared user experience research efforts. David wants more information about his users' experience so he can make better decisions. Such as:

  • He puts effort into consistent deprecation warnings. Are they effective? Are people seeing those warnings early enough to avoid unpleasant surprises when functionality changes? If not, what could he do to communicate this information to them more effectively?
  • Who's using this software? That is: do their needs cluster around a few major use cases/workflows? If so, what are they?
  • Which, if any, of the new features from the past few years do they use?
  • How do they seek help? Do they make an effort to start with the official docs, or just search the web and often find random blog posts written by enthusiasts?
  • What level of detail do they need in an error message spouted to the command line, and what would be so long that they'd be more likely to skip reading it?

In 2019, when I started writing grant proposals for work on pip to finish revamping the dependency resolver, I made sure to include a big chunk of budget for user experience research. Because these are the kinds of questions we can answer with UX research: outreach to recruit a big variety of users, surveys, interviews, studies where we watch over-the-shoulder as a user tries to accomplish a task and thinks aloud about it, prototypes (paper or software), "buy a feature" games, analysis, reports, persona development, and so on. The UX team wrote up what they did -- it looks like some key material is still awaiting merge into the main pip docs, but here's the UX guidance they wrote for Python packaging maintainers, and here's the research synthesis with several useful reports. [Edited 15 April 2024 to note: now that the work's been merged, I can properly link to the main pip UX research and design page, UX guidance written for Python packaging maintainers, and research results including reports from user surveys and interviews.]

We didn't limit ourselves to only researching the UX of pip, because that wouldn't make sense; pip is an interdependent part of the Python package creation and distribution ecosystem, and we needed to understand how users reasoned about, researched, and learned about packaging broadly. The resulting analyses and training have helped massively improve pip's user experience design, including error messages and docs as well as the direct ergonomics of invoking the right commands and having them do what users expect. And -- I believe -- learning these tools and facts has been helpful to other packaging tools maintainers beyond pip.

I see this as a model that we can replicate to other ecologies. Pool funds and invest in UX research for a suite of tools that people often use together, and learn surprising and helpful things that help you all improve your projects' UX.

A start

As I said while exploring what it might look like if open source were healthy: there are a bunch of giant infrastructural barriers to significantly improving the working conditions of open source maintainers. At least in the United States, we don't have guaranteed reliable health care. We don't have reliable direct funding for infrastructure, research, and development from companies, governments, the nonprofit sector and/or rich universities. We don't have high quality, universal, early childhood childcare.

But, given what we do have -- each other -- here are some ideas for ways we could join together to help each other out.

Comments

Camille
https://camilleacey.com
22 Aug 2023, 22:50 p.m.

As you might imagine, I have a lot of thoughts here. This is a really thorough post and I want to give you a worthy response. So I'm publicly declaring my intention to draft said response soon. Thanks for writing this all out and giving me a lot to think about!

Justin du Coeur
https://social.coop/@jducoeur
30 Aug 2023, 13:54 p.m.

Very useful stuff! I'm going to pass this around a bit in the Scala community -- the problems you identify are all familiar-sounding, and the ideas are well worth thinking about. Thanks!