In A post-mortem on the previous IT management revolution William Vambenepe writes, on the subject of standardization: “The first lesson is that protocols are easy and models are hard.” I agree about the relative difficulty, but think when it comes to interoperation, protocols are very difficult and shared models usually impossible. A couple of examples occur to me.
[This is the third time recently that I’ve referred to Mr. Vambenepe; I only ran across him when I wandered into this dodgy Cloud neighborhood. Given current circumstances I suspect it’s illegal for us to converse, but if things end well I’ll have to buy him a beer.]
We are re-visiting a theme here: I have long believed and repeatedly written that bits on the wire are the only serious reliable basis for interoperability, and worried in public about the feasibility of shared models. I believe the existence and success of the Internet and the Web are strong evidence in my favor. They have no object models or APIs; nor could they: they are just a collection of agreements what bits we send each other, with accompanying narrative about what those bits are supposed to mean.
Case Study: Atom and/or JSON · Recently I’ve seen a few instances of protocols specified as allowing messages in either XML or JSON; this included drafts of the Sun Cloud APIs before I got involved.
I think the reason this happens is that popular programming frameworks including JAX-B and Rails make it easy to generate either on demand; “Why,” the protocol designer asks, “shouldn’t we provide both? It’s free!”.
Well, it’s free for you. But you’ve potentially doubled the work for someone implementing the other side of the protocol, who must be prepared to deal with either data format. So don’t do that.
The second problem is subtler and more serious: since there are two data formats and you don’t want to describe both of them at the bit-pattern level, you have to invent an abstract model for the protocol messages and define two mappings, one to each payload format.
This extra work adds zero value to the protocol, and introduces several opportunities for error. And it really only works well if your data structure is a relatively flat, frozen set of fields without much narrative. Which means that you don’t need XML anyhow. And if you do need XML, attempts to model it in JSON quickly spiral into sickening ugliness and complexity.
So don’t do that.
Case Study: Modeling a Number ·
What’s the simplest possible protocol element? I’d suggest it’s a single
Suppose we are designing a simple XML protocol, and that protocol includes a
registered which is used to communicate the
number of students registered for a course. Here is its specification:
The content of the
registeredelement gives the number of students registered in this class, updated at 5PM each Friday. The offering faculty’s current rules governing when in the semester courses may be added or dropped must be consulted to predict the possible changes in the value from week to week.
This value could in principle be computed by retrieving the course roster and is included in this message to simplify certain operations which are frequently performed with respect to courses.
I like this description; it’s very specific as to the application semantics and what you have to watch out for, and even provides some justification as to why the field exists. So, is that enough specification? Is my protocol-design work done?
At this point, some readers will be shifting nervously in their seats, wondering whether I shouldn’t offer some syntax guidance. Well, that’s not rocket science; let’s add a paragraph to the specification:
The content of
registeredMUST match the regular expression
I’m not actually sure we’ve just added value to the protocol. On the one hand, we’ve disallowed empty elements and fractional students. On the other, suppose the library that generated the XML sent you this:
There’s been a pretty-printer at work here and a superfluous
+”. Should this be thrown on the floor? I’m not a
Postel’s-Law absolutist, but tossing this does feel a bit ungenerous. If you
went with just the first version, it might encourage implementors to be
intelligently defensive about their number parsing. Is that bad?
I’ll tell you this: As a TDD disciple, if I read the initial spec language, I’d have cooked up a bunch of test cases that would have explored the value space pretty thoroughly before I shipped any code. Tighten the spec and I might have gotten lazy.
Still, a serious data-modeling fan might argue that I’m not going far enough. For maximum interoperability, data should be typed, right? Fortunately for our fan, XML Schema provides a rich repertory of primitive types. Here’s a picture, from XML Schema Part 2: Datatypes Second Edition:
Now let us consider what kind of a thing we have here; it’s probably safe
to rule out
decimal, or rather its subtype
integer. But what kind? Well, it’s just a bunch of people in a
class, so let’s use
unsignedByte and not waste memory. Only
Java doesn’t do that and, oops, consider those big undergrad
“Whatever 101” courses. OK, let’s say
unsignedShort and hope
this never gets used for one of those online self-improvement courses by
Oprah and Eckhart Tolle. So maybe just
positiveInteger, because the class can’t be empty. Can
Then go have some fun reading the “lexical space” constraints that come with each of these types. I really hope that your XSD-checking library is compatible with the one at the other end of the wire.
Enough already; it seems obvious to me that in the vast majority of scenarios, we would be investing substantial effort but not really adding any value in the process.
At this point I should acknowledge that to some extent I’m whacking at a straw-man here. But I do believe the following things about designing protocols:
You can’t provide too much English prose about what the data means and how it relates to the real world.
You may add value by constraining syntax beyond what’s implied by the specification prose. But not always; you may introduce complexity and rigidity without corresponding benefit.
Trying to get implementors at both ends of a protocol to agree on a data model is at best difficult and requires many decisions, each of which is easy to get wrong and each of which may introduce unnecessary brittleness.