I propose the formation of one or more “Open Source Quality Institutes”. An OSQI is a public-sector organization that employs software engineers. Its mission would be to improve the quality, and especially safety, of popular Open-Source software.

Why? · The XZ-Utils backdoor (let’s just say #XZ) launched the train of thought that led me to this idea. If you read the story, it becomes obvious that the key vulnerability wasn’t technical, it was the fact that a whole lot of Open-Source software is on the undermaintained-to-neglected axis, because there’s no business case for paying people to take care of it. Which is a problem, because there is a strong business case for paying people to attack it.

There are other essential human activities that lack a business case, for example tertiary education, potable water quality, and financial regulation. For these, we create non-capitalist constructs such as Universities and Institutes and Agencies, because society needs these things done even if nobody can make money doing them.

I think we need to be paying more attention to the quality generally, and safety especially, of the Open-Source software that has become the underlying platform for, more or less, our civilization. Thus OSQI.

They’re out to get us · For me, the two big lessons from #XZ were first, the lack of resources supporting crucial Open-Source infrastructure, but then and especially, the demonstration that the attackers are numerous, skilled and patient. We already knew about numerous and skilled but this episode, where the attacker was already well-embedded in the project by May 2022, opened a few eyes, including mine.

The advantage, to various flavors of malefactor, of subverting core pieces of Open-Source infrastructure, is incalculable. #XZ was the one we caught; how many have we missed?

What’s OSQI? · It’s an organization created by a national government. Obviously, more nations than one could have an OSQI.

The vast majority of the staff would be relatively-senior software engineers, with a small percentage of paranoid nontechnical security people (see below). You could do a lot with as few as 250 people, and the burdened cost would be trivial for a substantial government.

Since it is a matter of obvious fact that every company in the world with revenue of a billion or more is existentially dependent on Open Source, it would be reasonable to impose a levy of, say, 0.1% of revenue on all such companies, to help support this work. The money needn’t be a problem.

Structure · The selection of software packages that would get OSQI attention would be left to the organization, although there would be avenues for anyone to request coverage. The engineering organization could be relatively flat, most people giving individual attention to individual projects, then also ad-hoc teams forming for tool-building or crisis-handling when something like #XZ blows up.

Why would anyone work there? · The pay would be OK; less than you’d make at Google or Facebook, but a decent civil-service salary. There would be no suspicion that your employer is trying to enshittify anything; in fact, you’d start work in the morning confident that you’re trying to improve the world. The default work mode would be remote, so you could live somewhere a not-quite-Google salary would support a very comfortable way of life. There would be decent vacations and benefits and (*gasp*) a pension.

And there is a certain class of person who would find everyday joy in peeking and poking and polishing Open-Source packages that are depended on by millions of programmers and (indirectly) billions of humans. A couple of decades ago I would have been one.

I don’t think recruiting would be a problem.

So, what are OSQI’s goals and non-goals?

Goal: Safety · This has to come first. If all OSQI accomplishes is the foiling of a few #XZ-flavor attacks, and life becoming harder for people making them, that’s just fine.

Goal: Tool-building · I think it’s now conventional wisdom that Open Source’s biggest attack surfaces are dependency networks and build tools. These are big and complex problems, but let’s be bold and set a high bar:

Open-Source software should be built deterministically, verifiably, and reproducibly, from signed source-code snapshots. These snapshots should be free of generated artifacts; every item in the snapshot should be human-written and human-readable.

For example: As Kornel said, Seriously, in retrospect, #autotools itself is a massive supply-chain security risk. No kidding! But then everyone says “What are you gonna do, it’s wired into everything.”

There are alternatives; I know of CMake and Meson. Are they good enough? I don’t know. Obviously, GNU AutoHell can’t be swept out of all of the fœtid crannies where it lurks and festers, but every project from which it is scrubbed will present less danger to the world. I believe OSQI would have the scope to make real progress on this front.

Non-goal: Features · OSQI should never invest engineering resources in adding cool features to Open-Source packages (with the possible exception of build-and-test tools). The Open-Source community is bursting with new-features energy, most coming from people who either want to scratch their own itch or are facing a real blockage at work. They are way better positioned to make those improvements than anyone at OSQI.

Goal: Maintenance · Way too many deep-infra packages grow increasingly unmaintained as people age and become busy and tired and sick and dead. As I was writing this, a plea for help came across my radar from Sebastian Pipping, the excellent but unsupported and unfunded maintainer of Expat, the world’s most popular XML parser.

And yeah, he’s part of a trend, one that notably included the now-infamous XZ-Utils package.

And so I think one useful task for OSQI would be taking over (ideally partial) maintenance duties for a lot of Open-Source projects that have a high ratio of adoption to support. In some cases it would have to take a lower-intensity form, let’s call it “life support”, where OSQI deals with vulnerability reports but flatly refuses to address any requests for features no matter how trivial, and rejects all PRs unless they come from someone who’s willing to take on part of the maintenance load.

One benefit of having paid professionals doing this is that they will blow off the kind of social-engineering harassment that the #XZ attacker inflicted on the XZ-Utils maintainer (see Russ Cox’s excellent timeline) and which is unfortunately too common in the Open-Source world generally.

Goal: Benchmarking · Efficiency is an aspect of quality, and I think it would be perfectly reasonable for OSQI to engage in benchmarking and optimization. There’s a non-obvious reason for this: #XZ was unmasked when a Postgres specialist noticed performance problems.

I think that in general, if you’re a bad person trying to backdoor an Open-Source package, it’s going to be hard to do without introducing performance glitches. I’ve long advocated that unit and/or integration tests should include a benchmark or two, just to avert well-intentioned performance regressions; if they handicap bad guys too, that’s a bonus.

Goal: Education and evangelism · OSQI staff will develop a deep shared pool of expertise in making Open-Source software safer and better, and specifically in detecting and repelling multiple attack flavors. They should share it! Blogs, conferences, whatever. It even occurred to me that it might make sense to structure OSQI as an educational institution; standalone or as a grad college of something existing.

But what I’m talking about isn’t refereed JACM papers, but what my Dad, a Professor of Agriculture, called “Extension”: Bringing the results of research directly to practitioners.

Non-goal: Making standards · The world has enough standards organizations. I could see individual OSQI employees pitching in, though, at the IETF or IEEE or W3C or wherever, with work on Infosec standards.

Which brings me to…

Non-goal: Litigation · Or really any other enforcement-related activity. OSQI exists to fix problems, build tools, and share lessons. This is going to be easier if nobody (except attackers) sees them as a threat, and if staff don’t have to think about how their work and findings will play out in court.

And a related non-goal…

Non-goal: Licensing · The intersection between the class of people who’d make good OSQI engineers and those who care about Open-Source licenses is, thankfully, very small. I think OSQI should accept the license landscape that exists and work hard to avoid thinking about its theology.

Non-goal: Certification · Once OSQI exists, the notion of “OSQI-approved” might arise. But it’d be a mistake; OSQI should be an engineering organization; the cost (measured by required bureaucracy) to perform certification would be brutal.

Goal: Transparency · OSQI can’t afford to have any secrets, with the sole exception of freshly-discovered but still-undisclosed vulnerabilities. And when those vulnerabilities are disclosed, the story of their discovery and characterization needs to be shared entirely and completely. This feels like a bare-minimum basis for building the level of trust that will be required.

Necessary paranoia · I discussed above why OSQI might be a nice place to work. There will be a downside, though; you’ll lose a certain amount of privacy. Because if OSQI succeeds, it will become a super-high-value target for our adversaries. In the natural course of affairs, many employees would become committers on popular packages, increasing their attractiveness as targets for bribes or blackmail.

I recall once, a very senior security leader at an Internet giant saying to me “We have thousands of engineers, and my job requires me to believe that at least one of them also has another employer.”

So I think OSQI needs to employ a small number of paranoid traditional-security (not Infosec) experts to keep an eye on their colleagues, audit their finances, and just be generally suspicious. These people would also worry about OSQI’s physical and network security. Because attackers gonna attack.

Pronunciation · Rhymes with “bosky”, of course. Also, people who work there are OSQIans. I’ve grabbed “osqi.org” and will cheerfully donate it in the long-shot case that this idea gets traction.

Are you serious? · Yeah. Except for, I no longer speak with the voice of a powerful employer.

Look: For better or for worse, Open Source won. [Narrator: Obviously, for better.] That means it has become crucial civilizational infrastucture, which governments should actively support and maintain, just like roads and dams and power grids.

It’s not so much that OSQI, or something like it, is a good idea; it’s that not trying to achieve these goals, in 2024, is dangerous and insane.


Comment feed for ongoing:Comments feed

From: Des Dougan (Apr 02 2024, at 15:42)


This makes so much sense on many levels. The $64K question is how it can be made attractive such that governments will see the benefits (benefits which are pretty obvious, I'd posit, to anyone who's used (me) or coded Open source software).

Thank you for developing and posting your idea.


From: reagan gibbs (Apr 02 2024, at 20:59)

This... just this. I am too new to network administration to do anything but lock up at night and unlock the building in the morning, but I will follow the story and put my shoulder to whatever wheel needs a shoulder. This needs to happen


From: Juha Autero (Apr 02 2024, at 21:09)

I've been thinking recently the biggest problem in social media is that we focus on how the world has to change for it to be better for us instead of what we can do to change world for better. Our solutions end up being ineffective because we are trying to change things we cannot affect.

I blame Aristotle and classical Greek philosophers' way of looking the world from outside point of view like some Great Architect. You end up with just another "if we all just..." silver bullet.

In a complex world this is not going to work. You only end up hiding complexity in sweeping oversimplifications.

It is possible that someone reading your article is in right position in a government to start the ball rolling, but I feel there even isn't enough expertise in the political system to generate necessary political will.

Free software has never relied existing power hierarchies, mainly because it didn't have access to them. But like they say, you become more conservative as you age. As you become more familiar with structures of power, you start to rely them more.

Sorry about incoherent rambling, I'm still trying to think things through and a web form on a mobile device is not the tool to re-edit the text into something better. What I'm trying to say is, you used to have better understanding of open source to suggest government as a solution.


From: Dave Pawson (Apr 03 2024, at 00:31)

An excellent idea Tim. How to turn it into a sales pitch for x.gov? You've made the tech pitch quite well. A good sw QA engineer might jump at the chance of such a job. Even the funding sounds more than practical... but how might it compete with 1001 other cash grabs?


From: Michael sokolov (Apr 03 2024, at 02:09)

Have you done any thinking about what existing structures might become incubators for this kind of work? I think you rightly point out governments are not really in the game, but maybe we are missing something that is already starting to happen. I see there is something called the EU open source policy summit, although I didn't really know much about it. Also I see the US has something called the cyber safety review board https://www.dhs.gov/news/2022/07/14/cyber-safety-review-board-releases-report-its-review-log4j-vulnerabilities-and . Maybe the fact that is called cyber makes it feel a bit dated already, but it could be a place to start


From: Juha Autero (Apr 03 2024, at 05:50)

We may not have people deciding government budgets reading this, but we have people who can:

- Register domain names.

- Setup Web sites.

- Setup working groups.

- Create making lists.

- Do fundraising.


From: Adam Rice (Apr 03 2024, at 07:11)

I know a lot more about volunteer management than I do about software, and one thought I have about "maintenance" is that an OSQI could do the following:

- Shine a light on poorly supported but important projects. The simple existence of this list could invite more XZ-type situations, but it could also invite better scrutiny.

- Maintain a list of volunteer developers who are willing to jump in and help with these. Obviously vetting would be a huge issue here.


From: Hugh Fisher (Apr 04 2024, at 16:58)

Perhaps a suitable model is the US Natonal Transportation Safety Board?

Not an expert, but my understanding is that they would tick most of your boxes if 'aircraft' was substituted for 'software'. In particular any data / information / etc that the NTSB collects during investigation can't be used by third parties as evidence in a lawsuit, encouraging companies to be less defensive when cooperating.


From: Nathan (Apr 06 2024, at 14:36)

It feels like the surface area is too vast here. Every time I need to install some Python software, it seems like I have to download somewhere on the order of a dozen dependencies. This is bizarre to me because at this point I must have accrued well over a hundred python packages already in my system. The fact that it seems I always need more indicates the scale of just this one ecosystem. Add in node.js packages, Ruby gems, and of course the innumerable C and C++ packages out there and you already have a target-rich environment.

And that's just the libraries! Once you start also considering the software that consumes those libraries, you have a truly Herculean task ahead of you.

And as you point out, the adversaries are patient and smart. If your OSQI prioritizes the top 100 packages, they can target the 101st.

And the broader the OSQI spread themselves, the less deep they can go. Consider your average OSQI employee; a harried civil servant with the goal of auditing 3 open source packages a week. Is this hypothetical employee going to do the due diligence to dig into each of these commits from the XZ exploit and find the subtle bug?

I can tell you that commit hygiene in many OSS projects is very poor. It's not entirely uncommon for something that's supposed to be fixing the spelling in an error message or rewording a wonky comment to also have a seemingly-unrelated change piggybacking on top of it.

It's not a terrible idea. The work seems boring enough that I don't think I'd take a pay cut to do it; the explicitly-not-feature-development clause pretty much guarantees that the work would be mostly drudgery rather than creative in nature, but I imagine some would jump at the chance. I just don't feel that it would be worth it.


From: Rachel Crowther (Apr 06 2024, at 14:46)

We *must not* have one of these. There must be more; they must be independent of each other; and they must each be funded by mutually distrusting governments.

It is a certainty that an OSQI run by the US government will be quietly told to ignore backdoors inserted by US state actors; it is likely that it will also be told to ignore similar quality issues from "friendly" actors. Therefore, if one wants a measure of certainty, one needs something like this run by each of US, Russia, China, Iran, perhaps Germany or France within the EU. Given that combination, it seems very unlikely that all of them would be instructed to ignore the same backdoor.

This lack of trust between multiple OSQIs is essential for the rest of us to trust the combined output.


From: Ryan Barrett (Apr 11 2024, at 22:38)

Compelling! Sounds like OpenSSF and Alpha-Omega, among others.


author · Dad
colophon · rights
picture of the day
April 01, 2024
· Technology (90 fragments)
· · Software (83 more)

By .

The opinions expressed here
are my own, and no other party
necessarily agrees with them.

A full disclosure of my
professional interests is
on the author page.

I’m on Mastodon!