On Thurs­day we launched some add-ons for AWS Step Func­tions, on which I helped a bit. As usu­al, there’s a nice Jeff Barr blog. This is to add de­sign notes and ex­tra col­or.

Our an­nounce­ment de­scribes these as “Integrations”  —  internally, while we were build­ing them, we called them Con­nec­tors, and I’m go­ing to stick with that be­cause it has one less syl­la­ble and feels id­iomat­ic.

Tl;­dr · Up till now, Step Func­tions knew how to hand work to Lamb­da func­tions and to polling “Activity Workers”. As of now, it can al­so make use of Dy­namoDB (read­/write), Batch (s­tart a job in ei­ther fire-and-forget or wait-for com­ple­tion mod­e), ECS (reg­u­lar and Far­gate fla­vors), SNS (write-only), SQS (write-only), Glue (like Batch, async or sync), and SageMak­er (same).

Of course, you could do all this be­fore, by run­ning a lit­tle Lamb­da func­tion to call what­ev­er API, but now Step Func­tions knows how to make those calls di­rect­ly. Which means few­er Lamb­das to own and main­tain. Al­so this should run a lit­tle faster with­out an in­ter­posed func­tion, and fi­nal­ly, Step Func­tions can be smarter about deal­ing with re­tries and throt­tling and so on.

How it works · Noth­ing es­sen­tial in the Ama­zon States Lan­guage has changed. Just as be­fore, you use a Task State to get work done, and you iden­ti­fy the work in the val­ue of the Re­source field, which is a URI. Used to be, the on­ly URIs rec­og­nized were Lamb­da ARNs and Activity-worker Task ARNs. Yeah, as far as I know, nobody’s ev­er reg­is­tered AWS’s “arn” URI scheme, but for all prac­ti­cal pur­pos­es they’re per­fect­ly good URIs.

So all we re­al­ly had to do to make this work was teach Step Func­tions to rec­og­nize new fla­vors of ARNs: One for each of the op­er­a­tions I men­tioned above. For ex­am­ple, the Re­source val­ue that re­quests fetch­ing a Dy­namoDB item is arn:aws:s­tates:::­dy­namod­b:getItem. All the oth­er stuff in Task states about Retri­ers and Catch­ers and so on goes on work­ing just as it did be­fore.

This no­tion of short strings that iden­ti­fy a “unit of in­for­ma­tion or service” is a straight­for­ward use of URIs, and shouldn’t be sur­pris­ing to any­one who un­der­stands how the Web work­s.

In most cas­es, the im­ple­men­ta­tion is sim­ple enough; the ser­vice just feeds the ap­pro­pri­ate in­put da­ta to the in­di­cat­ed API. But a cou­ple of the new Con­nec­tors go fur­ther, for ex­am­ple run­ning a Batch job in syn­chronous mod­e. It turns out that the Batch ser­vice on­ly has a fire-and-forget API, so what the ser­vice does in this case is write a rule in­to the caller’s CLoudWatch Events ac­count which catch­es Batch’s job-finished event and routes it to an SQS queue, which Step Func­tions has a long-poll post­ed on to find out when the work is done.

Once again, cus­tomers could pre­vi­ous­ly have set this up for them­selves (and in fact some have) but it just makes more sense to of­fer it as a built-in.

Pa­ram­e­ter­i­za­tion · Step Func­tions has al­ways had a tool, the “InputPath” field, to fil­ter in­com­ing in­put and se­lect bits and pieces of it to feed to work­er­s. When we start­ed work­ing on the first few Con­nec­tors, we re­al­ized it wasn’t quite up to the task of as­sem­bling the cor­rect in­put for an ar­bi­trary col­lec­tion of API call­s. We were at risk of re­plac­ing the dumb lit­tle Lamb­das that ex­ist­ed just to call APIs with even dumb­er lit­tle Lamb­das that just wran­gled JSON in­to the right shape to call the API.

Thus the States Language’s brand-new “Parameters” field. To ex­plain this, I’m go­ing to re-use the ex­am­ple from Jeff’s blog linked above:

 1 "Read Next Message from DynamoDB": {
 2      "Type": "Task",
 3      "Resource": "arn:aws:states:::dynamodb:getItem",
 4      "Parameters": {
 5        "TableName": "StepDemoStack-DDBTable-1DKVAVTZ1QTSH",
 6        "Key": {
 7          "MessageId": {"S.$": "$.List[0]"}
 8        }
 9      },
10      "ResultPath": "$.DynamoDB",
11      "Next": "Send Message to SQS"
12    }

You can see the magic read-from-Dynamo ARN there in the Resource field on line 3. But it’s the Parameters field value that’s interesting. It has the right JSON shape to hold the GetItem API arguments, but buried down in the Key field on line 7 there’s a little weirdness going on. It turns out that DynamoDB wants you to pass a string argument by sending JSON that looks like:

"S": "My own personal string key"

In line 7, you see a field whose name is “S.$” and value is “$.List[0]”. That “.$” suffix is the new thing; whenever you see that in the Parameters block, it means that the value is to be interpreted as a JSONPath, applied to the state’s input, and then the whole field is replaced by one whose name is the same with the “.$” suffix subtracted, and whose value is whatever you got from the JSONPath.
[Tim, what if their API already uses a field whose name ends in “.$”? -Ed.]
[That would be unfortunate. -T]

We think this should provide people with most of what they need to compose the arguments for almost anything you might want to invoke. By the way, the Parameters-block idea wasn’t mine, it was cooked up by folks on the Step Functions team, notably Ali Baghani. And because it’s not mine, I can say: Way cool!

Permissions · If you go to the console and set up a state machine like the one in the example, we can do an extra trick, namely look at all the Connectors in your machine, figure out what permissions you need to make those calls, and synthesize a Role for you, designed to be used in running that state machine.

API mapping? · Now, if you look at the way we’ve provided mappings for a few of the AWS APIs, you might reasonably wonder “Why not all of them?”. After all, I notice that Diego ZoracKy recently published a general-purpose Lambda function that does just that — give it the API name and the right arguments and it’ll do the call.

It’s not a crazy idea, but we’re not going to do it. Blindly calling APIs without having thought it through a little could be a recipe for unhappiness. We want to make sure that when we make those calls, we’re being sensible about buffering, polling mode, retrying, checking for impossible arguments, and so on. For example, we support sending a message to SQS, but receiving one will require some head-scratching about whether and when to delete after a successful read.

Also, in some cases we might want to do prep work, as we already do to make those asynchronous fire-and-forget APIs look synchronous when you put them in a state machine.

So, we’re going to reserve the right to add Connectors at our own speed and in a thoughtful way.

Future directions · The fact that we identify workers with URIs leaves the door open for any kind of future Connector you can think of. There are lots of obvious candidates in the AWS SDK, to start with. One especially obvious one is supporting the ARNs of other Step Functions state machines, giving you nested child-workflow invocations.

Another is allowing the use of any old HTTP endpoint URL, so your state machine could talk to any Web API in the universe. I suppose we’ll need to add an "HTTP-method" field or equivalent to specify whether you want GET or POST.

Anyhow, there are lots of Step Functions features on the road map that aren’t just Connectors; but I suspect that we’re going to come under pressure to keep adding them, starting today and going on more or less forever.

States-Language housekeeping · The States Language spec has been updated, and so has its source on GitHub. So has the statelint state-machine validator (and the Ruby Gem updated).

All these years into my career, I still get a little thrill being part of the Launch Day dance, a few little pushes with my own pinkies.

I tested my parts pretty hard but there might be bugs; I do look at pull requests and have taken some in the past. In particular I have to say thank-you for jezhiggins’ updates to the supporting j2119 parser generator that made it possible to add Parameters validation to statelint.


Comment feed for ongoing:Comments feed

From: Dan Kibler (Dec 03 2018, at 15:06)

These are some welcome additions. I'm glad you took the trouble to make them reliable and easy to use (e.g., security).

An obvious extension useful to many would be to write to and from S3.

As I eluded to in our brief Twitter exchange, increasing the max size of the data that can be passed into or between step would be very useful too.


author · Dad · software · colophon · rights
picture of the day
November 30, 2018
· Technology (85 fragments)
· · Cloud (10 more)

By .

I am an employee
of Amazon.com, but
the opinions expressed here
are my own, and no other party
necessarily agrees with them.

A full disclosure of my
professional interests is
on the author page.