I did some recre­ation­al pro­gram­ming over Christ­mas and the blog I wrote about it is now guest­ing in Jeff Barr’s space for your amuse­men­t; try the soft­ware at IsItOnAWS.­com. What I didn’t do there was re­lay the lessons I picked up along the way; one or two are around AWS, but most fol­low from this be­ing my first non­triv­ial ex­pe­di­tion in­to the land of NodeJS. So (ac­knowl­edg­ing that on­ly 0.8% of my pro­fes­sion aren’t al­ready Nodester­s), here they are. Spoil­er: I don’t like Node very much.

Les­son: Lamb­da has his­tor­i­cal­ly been used for behind-the-scenes work. But with the re­cent ar­rival of new API Gate­way and Cer­tifi­cate Man­ag­er good­ies, it’s be­come pret­ty easy to con­vince a func­tion to serve HTTP re­quests point­ed at your own web-space. Will this be a pop­u­lar id­iom? Beats me.

Les­son: I can now work with Node’s everything-is-a-callback world­view, but stil­l, at the end of the day I think it’s wrong. What I want to do is fetch data, then pro­cess data, then write data, and if a damn com­put­er lan­guage can’t give me a se­quen­tial ab­strac­tion when I want to do se­quen­tial things, well screw it.

Yeah, I ac­knowl­edge the kozmick per­for­mance gains Node achieves, even when liv­ing in a single-threaded en­vi­ron­men­t, by push­ing de­vel­op­ers in­to callback-or-die ter­ri­to­ry, but you know, there are things like pre-emptive mul­ti­task­ing and thread pools that should let the sys­tem in­ter­leave IO and com­pute for per­for­mance with­out mak­ing me wor­ry my pret­ty lit­tle head over it.

Hav­ing said that, async/wa­ter­fall is a straight­for­ward way to re­me­di­ate the dam­age.

Les­son: Node pro­vides a very ser­vice­able lit­tle JavaScript REPL on my Mac. There is no pro­gram­mer on whose life JavaScript doesn’t im­pinge some­times, and a com­mand line is aw­ful­ly help­ful in ex­plor­ing ar­ray com­bi­na­torics and re­lat­ed weird­ness.js.

Les­son: Con­struct­ing a zip was pret­ty easy with jszip. Ex­cept for, de­spite the fact that a zip is a bunch of bytes, jszip in­sist­ed on emit­ting a Node Stream. But it seems that NPM gen­er­al­ly con­tains cor­rec­tives for its mis­fea­tures, in this case raw-body.

Les­son: Node’s HTTP-fetch func­tion is kind of dumb and clum­sy. Every lan­guage should have a one-liner that says “Here’s a URL, gimme back an ob­ject with the content-type and the re­sponse body’s bytes, or let me know if you can’t.” Of the lan­guages I’ve used in re­cent years, on­ly Go and Ru­by do.

Les­son: Upon pub­lish­ing this, I will re­ceive much pity­ing feed­back along the lines of “Well of course you could have done it in a one-liner us­ing TheNewHotness.js.” And al­so point­ing out many oth­er bet­ter ways to have done this us­ing things my In­ter­net search skills were in­suf­fi­cient­ly ad­vanced to dis­cov­er. Draw your own con­clu­sion.

Les­son: The IPv6 address-literal syn­tax is stupid­ly human-hostile.

Les­son: NPM has at least one of ev­ery­thing you can pos­si­bly imag­ine.

Les­son: NPM de­pen­den­cies are a ful­mi­nat­ing can­cer­ous mess. This lit­tle Lamb­da that runs when the JSON up­dates needs fif­teen freak­ing megabytes in its node_­mod­ule di­rec­to­ry, and the zip is like 2.5M. For the lit­tle func­tion that ac­tu­al­ly han­dles the IsItOnAWS re­quest­s, I con­scious­ly tried to keep the de­pen­den­cies down, but I still end­ed up need­ing async, ipad­dr.js lo­dash, and sprintf-js for an­oth­er 2½ meg. Feaugh. What’s a “lodash”, any­how?

Les­son: The Lamb­da and S3 APIs are min­i­mal, sen­si­ble, and well-integrated in­to Node’s resistence-is-futile you-will-learn-to-love-callbacks paradig­m.

Les­son: The best Node code is Non Fan­cy Node.

Les­son: The tape unit-test har­ness Just Worked for me out of the box, had a nearly-zero learn­ing curve, and was min­i­mal­ly in­tru­sive. I’m a fan.


Comment feed for ongoing:Comments feed

From: Brent Rockwood (Mar 30 2017, at 03:20)

Hi Tim,

If I were to make one change, it would be to link to some of the blog posts/Github repo from https://isitonaws.com so people can discover the neat technology behind it.


From: John Cowan (Mar 30 2017, at 05:20)

I left a comment at Jeff Barr's blog, but it doesn't seem to have been approved, so I'll leave it here. There's a nice method of using binary search on ranges that I learned about when messing with Unicode predicates like isBurmese and isUpperCase before every language had them built in. I can't tell from the description if you are already doing this.

Use an ordinary vector of numbers of size 2n, and fill it in thus: start1, end1 + 1, start2, end2 + 1, .... When you need to know if a number is in some range, do your binary search to see what vector element is just greater than or equal to the input. If it's at an odd index, the number is in a range; if it's at an even index, the number is not in any range.

You can do set operations on these vectors as well, though you don't need them for this problem. To negate, prefix the vector with 0 and drop off the last value, unless the vector already begins with 0, in which case you drop it and append the largest value + 1 (0x110000 in the Unicode case). Union is just merge, and intersection is another kind of merge that drops duplicates.


From: Patrick Quinn-Graham (Mar 31 2017, at 05:29)

Node encourages the use of streams because they allow you to handle data more efficiently (passing it from the zip compressor directly to a writable stream such as a file on disk, or in your case a remote server as bits are available) than buffering it all in ram first.

s3.putObject should allow you to pass the stream object instead of buffering it yourself first with raw-body. If not, s3.upload definitely does.


From: Gavin B. (Mar 31 2017, at 09:53)

The nice thin about Elixir on top of Erlang is that it simulates the sequential programming that you crave in node.js while under the hood its all tail callback. They also have a half decent web server now - Phoenix. Next vac maybe take a try. elixir-lang.org


From: Peter J. (Mar 31 2017, at 11:45)

"What I want to do is fetch data, then process data, then write data". Something like


looks an awful lot like promises. Any reason (other than "Non Fancy Node") you didn't go that way, since v6 is supported as a Lambda execution environment?


author · Dad · software · colophon · rights
picture of the day
March 29, 2017
· Technology (81 fragments)
· · Cloud (8 more)
· · Web (387 more)

By .

I am an employee
of Amazon.com, but
the opinions expressed here
are my own, and no other party
necessarily agrees with them.

A full disclosure of my
professional interests is
on the author page.