Atom as a Case Study

This is adapted from my talk of the same name at ETech 2006. The talk’s sections were entitled Why?, How?, What?, and Lessons?; I’ve left out What?, the description of what Atom is, since we’ve had plenty of that around here. That leaves Why we built it, How we built it, and what Lessons you might want take away from the experience.

Why? · Q: RSS is the world’s most successful application of XML; why monkey with a good thing? A: Tech problems and people problems.

The most obvious technical problem is that there isn’t just one “RSS”, there are several incompatible versions. But RSS 2.0 has the biggest mindshare and market share; deservedly, in my opinion. Most people, starting from scratch, would (correctly) pick 2.0 as the RSS version to go with.

And there are some issues with RSS 2.0. Nothing that keeps it from working just fine across the universe of the Blogosphere, but real irritants. Item: It’s not clear whether you’re allowed to have more than one enclosure. Item: An item title is either in HTML, or not, and there’s no way to find out which, which can cause silent data loss, and make RSS 2.0 unsuitable for some applications, like Bugzilla. Item: Relative links don’t work: If you have pictures right there in the same directory as an item’s HTML, you still can’t say <img src="cute-cat.jpg"/>, you have to fill in the server and directory and everything, which makes your feed bulky, more brittle and less portable. For example, see my RSS feed. Item: Increasingly, people want to use non-ASCII characters in URIs, but in RSS 2.0 you can’t do that.

Then there’s the issue of APIs. RSS is a wonderful tool to help make the web writeable by everyone, but most web-writing tools suck. If we want them to suck less, it would be great if there were a standard API that anyone who wanted to develop a writer’s tool could use to talk to lots of different publishing systems. Unfortunately, the Blogger and MetaWeblog APIs are all we have now, and they’re pretty lame; see here, here, and here.

The conclusion is obvious; while RSS 2.0 is a wonderful success story, there’s still work to do in the syndication format space. So you might think the thing to do would be to get together informally and improve what we have now, and ship it as RSS 3.0. That would be a bad idea. Getting together informally would lead to an implosion, and calling it RSS 3.0 would be against the rules.

Bad Craziness · The fact of the matter is that for reasons that nobody seems to understand, working on syndication standards flips the asshole bit, even in the most reasonable people, leading to seriously nasty behavior.

Everyone who’s been within shouting distance of any of these processes knows all about it, but for the benefit of the ETech audience I displayed two instances of clueless childish abuse which had occurred within the week prior to the conference. But you know, I could have found that sort of stuff for nearly any week stretching back years now.

Even if you were determined to ignore the bad behavior and tough it out, there are some rules in effect. The RSS 2.0 specification includes a Roadmap section that says “Therefore, the RSS spec is, for all practical purposes, frozen at version 2.0.1. We anticipate possible 2.0.2 or 2.0.3 versions, etc. only for the purpose of clarifying the specification, not for adding new features to the format. Subsequent work should happen in modules, using namespaces, and in completely new syndication formats, with new names.” Which is clear enough, and there’s real value in clarity.

How? · Producing Atom has been a Whole Lot of Work, and it’s not (quite) done yet. First, let’s quote Sam Ruby:

Remember that Echo effort Jon told you about? We started based on experiences with RSS, but otherwise with a clean slate. We had interminable discussions over the name of the effort. Then we had interminable discussions over which standards body to pick. Then we had interminable discussions over dates. Eventually we solved all the issues above as well as other issues like relative URIs. And we had to wait while the slow standards process ground away. And finally, we got an official RFC issued by an internationally recognized standards body: the IETF. The same organization that issued the HTTP and SMTP standards — these are the protocols that power the web and email respectively.

Let me make that quantitative: After we’d worked for months on Sam’s Wiki, we trotted over to the IETF. The production of RFC 4287 took 17,944 email messages; the archive is 62.4MB in size.

The work so far on the Atom Publishing Protocol has been, relatively, child’s play, a mere 4,361 messages occupying a mere 17.0MB.

After we’d finished hashing it out among ourselves, the whole IETF had a chance to take a look, and caught some dorky things we’d missed.

There has been plenty of bad behavior, snarling, and name-calling in the process. It is not the case that RSS in particular provokes bad behavior: it’s something about syndication in general that flips that you-know-what bit. I and my Atom co-chair Paul Hoffman have had to warn and lecture and ban and ban again and tut-tut and generally spend way too much time treating adults like children. One WG member called me “criminally insane” over an issue that’s so trivial I don’t remember what it was. And in fact, some participants were driven away by the bad behavior of the others.

It would be fair to describe the process as slow, grinding, bureaucratic, difficult, and tedious.

Lessons? · In no particular order, my conclusions. Many will, I’m sure, disagree with one or more.

RSS is Good Enough · Really; it hasn’t significantly got in the way while the blogosphere went from nowhere to tens of millions strong. Initially, Atom will be used by people who are actually getting stung by one of those technical gotchas I mentioned above (e.g. Bugzilla, or high-value financial feeds, or whatever), and/or by purists who dislike the RSS 2.0 spec, and/or by people/institutions who believe in capital-S Standards, and/or by people who don’t like Dave Winer. But I’m pretty sure that there’ll be tons of RSS feeds around a decade or two out, and why not?

Reading and Writing · Right now, a general-purpose feed reader has to be able to handle three flavors of RSS and Atom 1.0. On the other hand, if you’re generating feeds, all you really need to support are RSS 2.0 and Atom 1.0, and within a year, more or less all the software will read Atom 1.0 and that’ll start to look really attractive for people who only want to generate one flavor.

The Protocol Is More Important · The Atom Publishing Protocol, which everyone will probably call “APP”, is a bigger deal than the data format. The reason is simple: RSS 2.0 works well enough to get work done and interoperate pretty well. MetaWeblog basically doesn’t, and there’s a huge place in the ecosystem for a portable API. If “Web 2.0” means anything, it means the Writeable Web, and right now writing requires special tools and most of them aren’t very good. You should be able to write the web from your emailer and outliner and browser and camera and spreadsheet and phone, and we need the APP to make that happen.

Since APP wants you to publish by posting Atom data there’ll be a lot of Atom data; the protocol will drag the data format behind it.

The IETF Process Was a Good Idea · Yes, even though it was a wearing bureaucratic grind. If people get along, you don’t need process and rules and so on; but people don’t get along; especially around syndication. The IETF’s virtue, shared by most sane standards organizations, is that it recognizes that fact and has a process where you can deal with it and get work done and still be reasonably open and inclusive.

Here’s an analogy; most people think that it’s pretty important that we have a government of Laws not Men. Which is to say, courts and lawyers and litigation generally suck, but are better than the alternative.

Here’s a strong claim: Atom is arguably the canonical example of the place where a formal standards org is appropriate: there’s a substantial body of prior art, we kind of know what works and what doesn’t, and the culture is broken enough that informal processes aren’t gonna get the job done.

Standards · It’s a dirty job, but someone has to do it. But not me, for a while. When we get through Atom, I’m awarding myself a few years’ break from the standards trenches.

---