So, I helped build Ama­zon CloudWatch Events (blog, AWS con­sole), which just launched. Been a while since my last ex­tend­ed spell of be­ing an ac­tu­al soft­ware en­gi­neer. Ship­ping feels good.

What it does · The cloud’s asyn­chronous; changes hap­pen when they hap­pen. Maybe you called an API a minute ago, maybe a database failed over, maybe your app saw a traf­fic surge. As­sum­ing you want to know when they hap­pen, the tra­di­tion­al ap­proach is POLL LIKE HELL. Oop­s, I be­lieve the po­lite us­age is “repeatedly call De­scribe APIs”.

The idea here is for our ser­vices to broad­cast “Events” (OK, they’re re­al­ly lit­tle JSON blobs) and for you to write “Rules” that match events us­ing “Patterns” (OK, they’re re­al­ly lit­tle JSON blobs) and route ’em to “Targets”, which are of­ten Lamb­da func­tions but can al­so be var­i­ous kinds of queues and streams and so on.

So for ex­am­ple you catch an EC2 instance’s tran­si­tion to “running” state and run a Lamb­da to tag it, or fix up its DNS. Or you catch a cer­tain class of API calls and route them to a DevOps Slack chan­nel. Or… well, a bunch of oth­er things, both ob­vi­ous and strange.

Which is to say: Less polling, less la­ten­cy, more au­toma­tion. We’ve been run­ning a be­ta for a while and the peo­ple who’ve seen Events have most­ly said “Yeah, we can use that” and then they get on the air more or less right away. So I’m op­ti­mistic.

Is it in­ter­est­ing? · Be­ing event-driven isn’t a nov­el­ty, nor are mes­sage bus­es. But from the biz point of view, cloud au­toma­tion is pret­ty hot these days. And build­ing any­thing at AWS scale gets fun fast.

This ser­vice has a part that soaks up the events from all the oth­er AWS ser­vices. Then there’s the part that fig­ures out which events match the rules. And of course, there’s a rules database. Fi­nal­ly, match­ing events have to be sent off to the ap­pro­pri­ate tar­gets  —  Lambdas or Topics or what­ev­er.

Of those, on­ly the Rules database is sort of ob­vi­ous. The rest would be too, ex­cept they have to op­er­ate at AWS scale, which means there are re­mark­able num­bers of Events and Rules and Tar­gets in flight. And of course the Cloud in­fras­truc­ture we build this on is fal­li­ble, so all the de­signs have to as­sume that Shit Hap­pen­s.

When there’s a change in your AWS de­ploy­ment and you’ve post­ed a rule to route it to a Lamb­da or what­ev­er, it gets there pret­ty damn fast. That makes me hap­py.

What I did · The thing wasn’t my idea; that came from Jesse Dougher­ty, the guy who hired me in­to AWS Van­cou­ver. His pitch­ing me on the project was one of the rea­sons join­ing up seemed like a good idea. I didn’t touch the database, nor the ma­chin­ery that de­liv­ers matched events. But I did leave fin­ger­prints on those JSON blob­s, and on the event in­ges­tion and match­ing soft­ware. I think this bit ex­hibits my simplest-thing-that-could-possibly-work en­gi­neer­ing aes­thet­ic.

But to be hon­est, I prob­a­bly added more val­ue work­ing on six-pagers and con­vinc­ing peo­ple the whole thing was worth do­ing.

Where’s the mag­ic? · The databas­ing and stream­ing and sync­ing in­fras­truc­ture we build on is pret­ty slick, but that’s not the se­cret. The man­age­ment tools are nifty, too; but that’s not it ei­ther. It’s the trib­al knowl­edge: How to build Cloud in­fras­truc­ture that works in a fal­li­ble, messy, un­sta­ble world.


Comment feed for ongoing:Comments feed

From: Marcel Lanz (Jan 14 2016, at 14:29)

this reminds me of a "business event monitoring" where we did collecting, matching and visually display the system state using time series. it gave an operator a view of the events within a collection of business systems and enabled them to correlate possible dependet events. I could stare the whole day on these timeseries monitor.

what we did:

friday, exceptions and user behavior:



author · Dad · software · colophon · rights
picture of the day
January 11, 2016
· Technology (78 fragments)
· · Cloud (4 more)

By .

I am an employee
of, but
the opinions expressed here
are my own, and no other party
necessarily agrees with them.

A full disclosure of my
professional interests is
on the author page.