At AWS, I’m now in the Server­less or­ga­ni­za­tion, which in 2018 is big fun. Some­one asked me to check out the work be­ing done at the Cloud Na­tive Com­put­ing Foun­da­tion (CNCF), par­tic­u­lar­ly around CloudEvents. There’s been a par­tic­u­lar­ly in­ter­est­ing ar­gu­ment go­ing on around there that I think has use­ful lessons for any­one who cares about de­sign­ing net­work pro­to­col­s.

I’m nat­u­ral­ly in­ter­est­ed in Event­ing be­cause it’s cen­tral, not just to server­less com­put­ing, but to mod­ern ap­pli­ca­tion con­struc­tion in gen­er­al. Events are a good way to think about a lot of dif­fer­ent things: Ac­tu­al events from the re­al world (“Garage door opened”), in­fras­truc­ture hap­pen­ings (“database failed over”), us­er ac­tiv­i­ties (“Leila signed in”), or da­ta move­ment (“Object 894t7 up­load­ed to buck­et JXYT8-33”). Events are nice, par­tic­u­lar­ly in well-designed mod­ern app­s, be­cause among oth­er things you can feed them to func­tions and drop them on­to mes­sag­ing queues.

My first project at AWS was CloudWatch Events, and one of the es­sen­tial things about a CloudWatch Event is that it’s got a fixed JSON wrap­per with a bunch of top-level fields that are guar­an­teed to be there. We nev­er wrote down a for­mal spec but there’s a rea­son­ably straight­for­ward de­scrip­tion here. CloudWatch events JSON ob­ject­s, and that’s all they are; noth­ing fan­cy about them.

Ev­i­dence sug­gests those choic­es were good; the ser­vice has been pret­ty suc­cess­ful; loads and loads of cus­tomers do­ing all sorts of ba­sic meat-and-potatoes au­toma­tion, and then some pret­ty imag­i­na­tive apps com­bin­ing built-in and cus­tom events. So, I have a lot of sym­pa­thy with the CNCF work.

CNCF CloudEvents have an ab­stract def­i­ni­tion not tied to any da­ta for­mat, with the idea that there could be mul­ti­ple dif­fer­ent rep­re­sen­ta­tion­s, al­though most ex­am­ples and con­ver­sa­tions still re­volve around JSON.

The Pull Re­quest · The prob­lem is summed up Pull Re­quest #277 and is­sue #294. It’s ba­si­cal­ly about whether it’s OK to put un­known fields not de­fined in the spec (“extensions”) in­to the top lev­el of a CloudEven­t, or in­stead, ban­ish them to an ex­ten­sions con­tain­er field. It’s not that cru­cial an is­sue and I can see both sides of it.

The ar­gu­ment be­ing ad­vanced in is­sue 294 and by Thomas Bouldin in Code­lab: Ver­sion­ing is Hard (a­ka the “SEF theorem”) is that if you al­low adding “extensions” at the top lev­el, that might break some soft­ware. In par­tic­u­lar, it’s go­ing to break any­thing that re­lies on Pro­to­col Buf­fers (ev­ery­one says “protobufs”). Be­cause they’re not tex­tu­al and self-representing but bi­na­ry and re­ly on an ex­ter­nal schema to help soft­ware un­pick the bi­na­ry bit­s; and that doesn’t leave room for any old ran­dom new bits to be dropped in­to the top-level record.

It turns out that some or­ga­ni­za­tions have bought in­to pro­to­bufs heav­i­ly; for the pur­pos­es of this dis­cus­sion it doesn’t mat­ter what their rea­sons were, or whether those rea­sons were good. So deal­ing with CloudEvents is go­ing to be eas­i­er for them if they can re­ly on map­ping back and forth be­tween CloudEvents and JSON. Which they can’t if ex­tra­ne­ous “extensions” might show up at the top lev­el.

Les­son 1: The In­ter­net isn’t ab­stract · I think the CloudEvents com­mit­tee prob­a­bly made a mis­take when they went with the abstract-event for­mu­la­tion and the no­tion that it could have mul­ti­ple rep­re­sen­ta­tion­s. Be­cause that’s not how the In­ter­net work­s. The key RFCs that de­fine the im­por­tant pro­to­cols don’t talk about send­ing ab­stract mes­sages back and forth, they de­scribe ac­tu­al re­al byte pat­terns that make up IP pack­ets and HTTP head­ers and email ad­dress­es and DNS mes­sages and mes­sage pay­loads in XML and JSON. And that’s as it should be.

Time af­ter time, peo­ple have got the idea of shar­ing ab­stract ob­jects across the In­ter­net, and time af­ter time it’s led to prob­lems of one sort or an­oth­er. There was a time when a lot of peo­ple thought that some­thing like CORBA or DCOM or WCF would make objects-on-the-wire not on­ly pos­si­ble but straight­for­ward, and free us from the tyran­ny of think­ing about the bits and bytes in mes­sage for­mat­s. But as you may have no­ticed, those things are pret­ty well gone and the Web has out­lived them; its klunky old ad-hoc tags and head­ers are how ev­ery­thing work­s, most­ly.

To make this con­crete: If CNCF had start­ed out say­ing “A CloudEvents is a bag of bits which is a JSON Text” or “…which is a pro­to­buf message”, well, is­sue #294 just wouldn’t ev­er have arisen. And nei­ther choice would have been crazy.

Les­son 2: S, E, and F · Bouldin’s Ver­sion­ing is Hard in­tro­duces the “SEF Theorem” where “S” is for Struc­tured, by which he means “you need an ex­ter­nal schema and you can’t just throw in ex­tra fields”, “E” is for Ex­ten­si­ble, i.e. you can go ahead and put in unan­nounced for­eign fields with­out chang­ing ver­sion­s, and “F” is for For­ward Com­pat­i­ble, which means you can add ver­sions with­out break­ing ex­ist­ing de­pen­den­cies.

Giv­en the choice, I’ll take “E” and “F” any day. When you’re pump­ing mes­sages around the In­ter­net be­tween het­ero­ge­neous code­bas­es built by peo­ple who don’t know each oth­er, shit is gonna hap­pen. That’s the whole ba­sis of the We­b: You can safe­ly ig­nore an HTTP head­er or HTML tag you don’t un­der­stand, and noth­ing break­s. It’s great be­cause it al­lows peo­ple to just try stuff out, and the use­ful stuff catch­es on while the bad ideas don’t break any­thing.

So what hap­pened? · The com­mit­tee took the trade-off I like. Which means you can ex­tend CloudEvents pret­ty freely (good), but you can’t use pro­to­bufs and JSON in­ter­changably and ex­pect things to work (un­for­tu­nate). This way is less brit­tle but a lit­tle hard­er to deal with. Not gonna say that the right choice is a slam-dunk, but it is the right choice.


Comment feed for ongoing:Comments feed

From: Paul Hoffman (Aug 31 2018, at 06:02)

An article that keeps saying both "protobufs" and "standards" has a few problems. CBOR is a standard, it feels like JSON in that an object is trivially and clearly extensible, there are plenty of tested language implementations, and it requires no external schema.


From: Geoff Arnold (Aug 31 2018, at 07:31)

Didn't we learn anything from years of futzing with SNMP MIBs?


From: John Cowan (Sep 01 2018, at 17:38)

I don't believe in theorems without proofs. In particular Protobufs v2 supports both E and F as well as the obvious S, provided that you partition the field numbers between insiders and outsiders, as by assigning odd numbers to the standard and even numbers for extensions. If two different extensions use the same numbers for semantically different fields, you lose, but that's true of JSON too.

Unfortunately, Protobufs v3 removes the feature that maintains unknown fields when copying a buffer, and CapnProto (a reinvention of v2 by Kent Varda) removes the feature of assigning numbers arbitrarily (but there are a lot of benefits in exchange, notably zero-copy behavior).


author · Dad · software · colophon · rights
picture of the day
August 30, 2018
· Technology (84 fragments)
· · Internet (111 more)

By .

I am an employee
of, but
the opinions expressed here
are my own, and no other party
necessarily agrees with them.

A full disclosure of my
professional interests is
on the author page.