This is a lengthy note to myself. I initially wanted to capture the thinking that went into the construction of mod_atom while it was still fresh in my mind, and dumped out the first dozen or so sections. Then as I expanded and refactored the code, I find that I’m keeping this up to date. This mostly by way of putting it in a place where I won’t lose it. I can write stuff for ongoing faster than for any other medium, and “On the Net” is a good place not to lose stuff. If mod_atom eventually gets picked up and used, this may be useful to me or anyone else who’s maintaining it; and if it doesn’t, there’ll still eventually be an AtomPub server module for Apache, and this might be useful to whoever builds it. But this is not designed to be entertaining or pedagogical; among other things, it’s in essentially random order.
I’m publishing it in November 2008, 14 months after I started writing it, because I’m giving the first-ever public speech about mod-atom at ApacheCon 2008 and I want something to point at should anyone be interested.
What’s a “Publication”? · mod_atom is built around the notion of a “Publication”, which has a one-to-one correspondence with a “Service Document” as described in RFC5023. There is a strong notion of a publication which is expected to have at least one Entry collection (to which Atom Entries can be POSTed) and one Media collection (to which any old bag of bits can be POSTed).
Abbreviations & Nomenclature · By “Apache” I mean the code produced by the Apache Software Foundation’s HTTP Server Project, often called “httpd” but this is misleading as the executable sometimes runs under the name “apache” or “apache2”. Specifically, I’m referring to the version against which mod_atom was developed, 2.2.*.
To keep function names reasonably short, MLE stands for Media Link Entry, and FP stands for Front Page, i.e. the “index.html” thingie that is the face a publication shows to the world. Pub stands for “publication”, which in the mod_atom context means the set of collections described by one AtomPub Service Document.
A “Feed” is an Atom document whose root is
“Collection” is a Feed as used in RFC5023, it can accept POST requests to
create new resources.
Files and URIs · The default operations of Apache assume file-backed resources. Thus mod_atom’s mapping from URI into filesystem space gets quite a bit of support from the infrastructure.
Mostly Static · mod_atom tries to impose the minimum possible tax on the URI processing path. It has to check every URI going through the server to see if it applies to a pub, and re-write those that do from URI into filesystem space. Then almost all GET requests can be tossed back to Apache’s static-file processor without further intervention.
There’s an exception when you have to fetch
anything but the first twenty entries out of a feed. The feed URI takes
start= query parameters and generates the
feed dynamically when it sees them.
Otherwise, in general each URI in a publication is mapped to an actual static
Use the Source, Luke! ·
The main body of code is reasonably well commented, and there are several
page-fulls of explanation at the top of
mod_atom.c, which are
really essential to understanding the details.
Where Stuff Goes ·
Suppose there’s a publication with its URI space rooted at
the corresponding directory space rooted at
The URI space for a pub has two
In fact, they are backed by a single directory structure (beginning
mod_atom only allows POST, PUT, and DELETE requests (as used in AtomPub) to be
routed to the URI space beginning with
The goal is to keep your security setup simple. With
<Location> directive, you can
apply rigorous TLS+authent security to the
atom/ subtree, while
pub/ subtree wide-open.
The actual subtree layout is described at exhaustive length in
the comments at the top of
mod_atom.c. The software has
knowledge of this directory layout wired in at a deep level. Changing it much
would probably be quite expensive in terms of refactoring/recoding cost.
To support this, the top-level
x/ (for “extras”) subdirectory, which starts out populated with
mod_atom allows unrestricted
DELETE to anywhere under
atom/x/; the idea is that
you can use it as a sandbox (probably with the same access-control that
applies to the whole
atom/ subtree) for your ancillary files.
It allows unrestricted creation and deletion of directories, with automatic recursion as required.
ID Elements ·
Every Atom feed and entry has to have a unique ID. mod_atom uses the
urn:uuid style. The directory structure is such that there’s
only one feed per directory, which also contains a file named
(note the dot) that contains only the feed’s unique identifier.
This exists so that when you re-create a collection you don’t have to read and parse the existing collection to find the ID string.
Element Updates and the Iterator ·
When you create a new resource with POST, it goes into a directory whose
pathname encodes the current date along YYYY/MM/DD lines. Feeds are generated
by a directory iterator (see
iterator.c) that walks backward
through these YYYY/MM/DD directories, returning entries one at a time.
When you update a resource with PUT, it creates a new “Link” file with an uninteresting name in the current (that is, date of the PUT) YYYY/MM/DD directory, which contains only the actual full pathname of the file that was updated. The iterator uses these to generate a feed which is still reverse chronologically ordered in the face of updates.
When an entry is updated multiple times, or updated then deleted, there’s no effort made to remove obsolete link files. These are detected in later iterator scans and removed on a side-effect basis, making the Link/PUT system self-maintaining.
Object-Orientation, or Not ·
The Apache code manages to have a fairly object-oriented feel even though
it’s in C, because the key routines all expect to take a
request_rec as an argument. mod_atom
has three important structures:
packages of information about the things implied by
their names. These are read and written in a fairly ad-hoc way and mostly
exist to keep function argument lists under control; so despite appearances,
the design of mod_atom really isn’t very O-O at all. There’s a very linear
per-request control flow starting at the beginning and proceeding through to
I wonder, if I were re-doing this now, knowing how it’s done, I’d refactor to have seriously Object-like constructs for Entry and Pub and Media. I don’t think it would help that much, but at the same time I’m sure that if I were re-writing it in Java or Ruby, wouldn’t think twice. I wonder if there’s a lesson here?
Generating HTML ·
mod_atom can operate either as a pure “Atom Store”, performing CRUD only on
Atom Entries and Collections; or, it can operate as a blog publishing system,
generating HTML and public-facing Atom Feeds and so on. HTML publishing is
turned on by the existence of a resource named
HTML and Tidy · When mod_atom receives a POSTed or PUT entry, and HTML is to be generated, the content, in particular the “text constructs” (see RFC4287) need to be parsed, so that sanitization rules can be applied. For those that are marked as XHTML, this is trivially accomplished using Apache’s built-in parser.
Text that comes in marked with
type="html" is a tougher nut to
crack. I found two plausible candidates for HTML parsing: TidyLib and
libxml2. My research did not find anything online that suggested either was
qualitatively better than the other, so I chose TidyLib on the basis that the
library you link to is an order of magnitude smaller.
Since Apache’s XML object model is different from Tidy’s, this means there is replicated sanitization and persistence code; oh well.
XML In · Apache has expat compiled in, and there’s a call that makes it easy to point it at either the client request or file and get a tree structure, both of which mod_atom uses. Over the years, I’ve done relatively little with trees, preferring stream parsers, for robustness. It turns out that Apache already has settings to set a hard limit on the size of XML object it’ll try to parse, so I figured it would be OK to allow it to parse individual Atom entries into a tree.
Apache’s XML support has a whole bunch of special-purpose connections to DAV, but you can mostly ignore them.
The object model itself is kind of nice; it forces you to walk through
lined lists of adjacent text chunks, but aside from that I found that it
pretty well got out of the way.
I eventually created a separate function bundle in
to capture a few repeated patterns of child-walking to pick up this
Tidy’s isn’t as slick, but it didn’t cause any pain. I’ve already complained about the interface you have to use for a memory allocator. Whatever.
There’s one instance of fairly horrible code that makes me think I’m doing
something wrong: when I get some media bits POSTed, I have to cook up a Media
Link Entry. So I hand-construct an XML tree for the purpose; see
XML Out ·
mod_atom needs to write quite a bit of XML. Apache does have a an
apr_xml_to_text call, but there was a pretty severe impedence
mismatch with my needs. To start with, I wanted to pump stuff into the XML
output straight from the program, without having to wire it into a tree.
Second, pumping out the XML as you generate was more natural to code (and also
probably a memory-saver).
So I used my own Genx library for the purpose. It has the virtue of being small-ish and pretty well-tested, in production here & there around the Net. It makes it really difficult to generate output that’s not well-formed.
It has a few problems, too. The API was a little kludgy; enough that
there’s a chunk of code in
genx_glue.c that automates
common cases and makes code more readable.
Also, since it generates not only well-formed but canonical XML, you’ve got
a real problem when you find something like
incoming HTML-encoded data. As a result, there’s actually a table of all the
XHTML 1.1 entity names in the code, and mod_atom turns all those into the
actual Unicode character values in the output. And, as a side-effect, all
Text Constructs that come in marked
type="html" go out marked
type="xhtml". Which I think isn’t actually harmful.
Error Reporting · There are two kinds of errors; client protocol errors (e.g. an Entry without a title) which should be reported back to the client in enough detail that there’s a chance of fixing them, and internal errors (e.g. “Can’t open file”). In each case, the error should probably be recorded in the apache logfile.
Apache doesn’t make it particularly easy to send a body back to the client with an error return code. mod_dav uses a fairly strained-looking techniqe, and at the moment mod_atom does really nothing useful, just returning the best-available HTTP status code with no explanation.
Wiring in mod_dav style error bodies is on the to-do list. This should
probably be done anywhere you see an instance of
X is anything but
Also, the error-reporting structure inside mod_atom is kind of ad-hoc and
maybe needs refactoring. Some returns return an OK or an HTTP error code;
others return an
apr_status_t, and still others return a
char *; NULL on success, otherwise an error message.
There may actually be a case for having both these options, but I’m not sure
the optimal choices have been made.
Finally, when you get an error in the HTML-generation phase, which doesn’t start until the Atom Store work is done, this does not cause the client to see an HTTP error (although it is sent to the server logfile). Is this correct?
Startup · When httd starts up, it invokes mod_atom to process each AtomPub directive in the config file. There are two directives; here’s a sample of each:
AtomPub /joe /Users/joe/Public/blog "Joe’s Blog" "Joe Smith" AtomMetaPub /blogs /var/blogs "Default title" "Default user"
In each case, the first argument is the root in URI-space, the second the
root in filespace.
AtomPub declares a single pub,
AtomMetaPub a facility for managing pubs (more on that later).
mod_atom checks to see if a blog specified by the
directive exists, and if it doesn’t, creates an empty directory framework, and
empty feed files, from scratch.
Apache startup is in itself a little weird. It runs through the config-processing process twice, once to check whether the file is correct and the directive processing doesn’t blow up, the second time “for real”. At the moment, mod_atom just does all its initialization twice in a row, not trying to avoid doing anything twice. This doesn’t seem to cause any problems.
Pub CRUD ·
A publication declared with
AtomMetaPub (let’s call it a
meta-pub) has an Atompub Service
Doc and talks the Atompub protocol, but new publications (let’s call them
sub-pubs) are created,
updated and deleted as a side-effect.
(Let’s just say “meta” and “sub” like in the comments.)
Thus, the collection feed for a meta constitutes a directory of subs.
Suppose the meta is rooted at
/foo. Then there will be 100
s99. When a new sub is
created, it gets some sort of short name via regular Slug processing (say
submarine), and one
of the subdirectories (say
s37) is chosen at random. Then the
sub’s root is
To help discovery, in the meta’s feed, the entry will contain a link like
<link rel="publication" href="/foo/s37/submarine/atom/pub.atomsvc" />
When you DELETE the entry in the meta collection that represents (in some sense) the sub, the whole sub is just deleted lock, stock, and barrel.
Templating · The HTML generation system is described in Autumnal mod_atom. I invented a templating system because I couldn’t find one that seemed a very good match for mod_atom’s needs. It’s straightforward if fairly tedious code; check out run_fp_template and run_fp_template_element. The only trick is that for the front-page template you need to interrupt generating the page in the middle of the template where it says “put the entries here”, so there’s a little bit of recursive shuck-and-jive with a state variable; the first time, as an adult programmer, that I really wished I’d had continuations.
Globals · Apache modules are by default constructed with the use of only one global variable, the “module” structure, which has pointers to structures describing how to handle that module’s directives and which which event hooks the module would like to be invoked for.
Since mod_atom’s code extends across several C source files, this doesn’t work. However, we’d like to be careful not to pollute the global function namespace. So, everything that’s visible outside the scope of one source file has one of the prefixes “atom_”, “genx_”, or “Tidy”. This makes for some long and ugly function names.
In general an Apache module can’t have mutable global variables because Apache can run in threaded mode, requiring you to implement access control. mod_atom does have some global variables, and even writes to them, but only at initialization time, which is single-threaded; accesses are read-only at run-time.
In the process of mapping between URI space and filesystem space, mod_atom
does a tremendous amount of cutting and splicing of UNIX-style pathnames. The
book-keeping that goes witih this in order to keep the slashes in the right
space is awkward and tedious, so the
atom_join() function does
this; it takes a variable number of arguments, the last of which must be
NULL, and splices them together, making sure the segments are
separated by exactly one slash.
Concurrency · It’s hard; let’s go shopping.
For individual resource creation, mod_atom pushes the concurrency issues down into the filesystem, which seems pretty bullet-proof.
There is exactly one place where it is necessary mutex:
When you’re doing a PUT to a resource, there’s a critical session while
you compute its ETag and compare that to the
header. I think there’s no way around mutexing this; you have to
out while you compute the ETag, decide whether or not to take the update, and
then maybe apply it.
This is accomplished with APR’s file-locking code, not on the resource itself but on a stub file of the same name with “.mutex” appended.
Sluggishness · Slug processing is pretty brutal. The text is de-%-escaped, which is supposed to give you UTF-8. Then any characters which aren’t legal UTF-8 or XML are silently crushed out. Then we remove leading and trailing dashes, and all characters which are not XML NameChars, or are ‘.’, are replaced with hyphens.
If there’s no slug, or there’s something horribly wrong with it, and it’s
an Atom entry that’s being posted, we try to use the text from its
Failing that, filenames are just random numbers.
If you send multiple posts in successions with the same Slug, mod_atom tries appending random numbers, and if that doesn’t work after a couple of tries, gives up and declares an error.
Security · In a typical publication, there are parts that you want to expose to the world and parts that require access control. There are a variety of Apache Directives that you can use to support access control. mod_atom enforces two run-time policies to aid in managing your security setup:
POST, PUT, and DELETE operations are rejected unless their target URI
/atom/ immediately after the publication’s root.
GET operations are also rejected unless the
path-step occurs similarly in the URI.
Let’s assume the same customization discussed above:
AtomPub /joe /Users/joe/Public/blog "Joe’s Blog" "Joe Smith" AtomMetaPub /blogs /var/blogs "Default title" "Default user"
Requiring basic authentication for access to the
parts of the URI space could be done with Apache configuration directives as
<Location /joe/atom> AuthType Basic AuthName JoeBlog-admin AuthUserFile /var/blog-admin/passwd Require user joe </Location> <LocationMatch /blogs/s[0-9][0-9]> AuthType Basic AuthName Blog-Publishers AuthUserFile /var/blog-admin/passwd Require valid-user </LocationMatch>
This could be done in a similar fashion using
directives if you wanted to work from file paths as opposed to URIs.
If an entry is POSTED with app:draft set, an Atom entry is created and the
collection, but neither the public-facing feed nor the HTML version, is
updated to show it. Should it eventually be PUT with app:draft removed, the
public-facing HTML and feeds are created.
When the HTML version of an entry is generated, its extension depends on
the value of the
type attribute of its
element. If it’s absent or
.text file is
created. If it’s
.html file is created.
Lazy Feed Generation ·
Whenever an entry is created, updated, or deleted, a file named
timestamp is updated in the root directory of its collection.
When mod_atom is handling a
GET request for the collection file,
or the public-facing feed file or front page files which are generated from
it, the timestamp of the collection file is compared against that of the
timestamp file, and all three (collection, feed, and index.html) are
regenerated prior to serving the request.
This removes some tricky race-condition scenarios and also makes heavy update streams more survivable.
To Do ·
Allow PUT with
app:draft to un-publish an entry. (This
might work, but I haven’t tested it.