I’ve been using RSpec in a way that’s probably inappropriate, but has got me thinking about Test-Driven Development and REST.

RSpec? · Rubyland is all about TDD, but if you listen to the Kool Kids, they don’t talk about “tests” or even “unit tests” any more, but about “specs”. Which comes from RSpec, a Ruby framework that is positioned as not for “unit testing” but for “Behavior-Driven Development” or BDD. I was first exposed to it in an April-2006 presentation by Dave Astels at Canada On Rails.

I’d explain BDD but that’s covered very nicely on the RSpec page and also in Bruce Tate’s Behavior-driven testing with RSpec. I’ve never actually managed to write any software with a pure BDD approach, but the closer I’ve come the happier I’ve been.

Now you could say that as frameworks go, RSpec is just a retake on xUnit with heavy lashings of syntax sugar, and I think there’s justice in that; but on the other hand, readability is important and I generally like syntax sugar.

The Trouble With mod_atom · I’ve been grinding away part-time on mod-atom for over a year, but I really have to get it somewhere near 1.0 so I can do performance testing so I’ll have something interesting to say at Apachecon even if I’m speaking late Friday thus to an audience of 8½ people.

The development of mod-atom has been test-driven, because The Ape existed before I started. But not, I’m afraid, unit-test-driven. In the early days I had unit tests for some of the low-level path-munging and directory-walking routines, but they were brutally difficult to maintain (I had to mock out essentially all of APR) so eventually I junked ’em. But I always had the Ape and Joe Gregorio’s AtomPub client keeping me honest.

However, recently I started going beyond the limits of what the Ape currently tests: publication CRUD, HTML generation, and app:draft support. The good news is that the Ape code has generic support for GETting and PUTting and POSTing and checking media-types and extracting links; it wasn’t actually designed to serve as a REST testbed, but with a bit of refactoring it’s given me what I need.

Some REST Tests · An example is worth a thousand words:

  it "Should create a sub-pub if posting to a meta-pub" do
    m = meta(@meta)
    c = Ape::Collection.new(m.href)
    entry = new_entry(:title=>'Posting to meta',
      :summary=>'To see if it works', :content=>'Does this matter?')
    p = c.post( :data => entry, :slug => 'sub-pub-title')
    p.should be_kind_of(Ape::Entry)

    # find the <link rel="publication"
    @sub = p.link('publication')
    exists?(@sub, 'application/atomsvc+xml').should be(true)

@meta is the URI of the meta-pub Service Doc. There are a few utility routines, obviously. Some are built into the Ape like Entry#link, others were cooked up for this purpose but probably belong in a generic framework.

Here’s another:

  it "Should respect app:draft" do
    ce, c = entries(@pub)

    entry = new_entry(:title=>'Just a post',  :draft => 'yes',
      :summary=>'Just text', :content=>'Wanna see text')
    p = c.post( :data => entry, :slug => 'Want Text')
    p.should be_kind_of(Ape::Entry)
    
    h_uri = p.uri.sub(/\.atom$/, '.text')
    @to_clean_up << p.uri
    exists?(h_uri).should_not be(true)

    # now make sure it also isn't in the feed.atom or index.html
    feed_uri = ce.href.sub(/collection.atom$/, 'feed.atom')
    h = Ape::Getter.retrieve(feed_uri, 'application/atom+xml')
    h.should be_kind_of(Net::HTTPSuccess)
    rel_uri = h_uri.sub(%r{^http://[^/]*/}, '/').sub('/atom/', '/pub/')
    h.body.index(rel_uri).should be(nil)

In this one, there’s some cheating happening; the test code knows what what the resource names created by mod-atom look like and peeks into URI space to see if they got created. And it looks to see if there are links to the shouldn’t-exist public version of the resource by just scanning representations for the URIs.

Anyhow, URI-driven: check. Resources and representations: Check. Hypertext as engine of application state: check.

What’s Wrong With This Picture? · Well, it might be RSpec, but it sure ain’t BDD. As Headius pointed out when I was telling him about this, it’s actually functional not unit testing. And in fact I wrote a ton of C code first and the tests later. And furthermore, either I’m not using RSpec right (quite likely, it’s my first time) or it has an impedance mismatch with the notion of working with a stateful system, where you create resources in one test and interact with them in the next and destroy them in a subsequent one.

But hey, it sure helped me with mod-atom and it’s pretty readable and it wasn’t too much work to sketch in the tests. So I’m managing to restrain my guilt.

What Would RESTUnit Look Like? · I’m early in this thought process, but some basics occur to me:

  • Access to GET/POST/PUT/DELETE with fine control over the headers, and utilities to wrangle ETags and so on.

  • Easy XPath for pulling stuff out of messages which happen to be in XML.

  • Convenience functions for running around the DOM digging URIs out of message bodies.

  • Possibly some Atom-specific stuff, on the assumption that a lot of the payloads in RESTful conversations will be Atom. For example, you really need Entry#link(rel).

  • A place to stash information about the resources you’ve created and are interacting with, and a cleanup function so you leave the URI space the way it was when you started.

As an RSpec newbie, I get the feeling that a lot of what I need would be easy to add on or may already be there.

What Other People Are Doing · I thought “Surely others have thought about Test-Driven RESTful Development” and went poking around, but didn’t find much. There’s RESTClient, but it seems more like a debugging aid.

Then of course there’s Joe Gregorio’s APP Test Client; Joe tells me that, just as I’ve been doing with the Ape, he’s been pulling apart the generic-AtomPub-client and AtomPub-test-driver pieces, so I suspect that it’d be at least as good a basis for TDRD as the Ape.

Aside from some Rails-specific stuff here and there, that’s all I’ve turned up. Let me know I’m wrong in the comments below.

Which is odd; the combination of TDD or BDD and REST feels natural to me; write code to exercise the service, then start filling in the service.

What Am I Going to Do? · Not over-think it, for now. I still have a lot more tests to write and then I need to build a bunch of scripts to exercise performance... hmm, seems that a RESTful test suite and performance exerciser might share a lot of code. When I get that done, I’ll think about it and develop an opinion as to whether there’s maybe a BDD framework in there trying to get out, or at least signposts for what one ought to look like.

Or not; It’s perfectly possible that there’s no sweet spot here, that the essential details of REST apps differ so much that there’s not really much shared testing framework to pull out.

But I think the idea is stimulating enough to share while still half-baked.



Contributions

Comment feed for ongoing:Comments feed

From: Peter Keane (Sep 11 2008, at 22:42)

Excellent. This strikes me as exactly the sort of "tool support" we need in the REST/Atom space, as per your previous post on REST.

[link]

From: John Cowan (Sep 11 2008, at 22:48)

Unit tests are to testing as syntax is to programming languages: important, but also wildly overrated.

Today I was reviewing a small code change by another team, and I saw that they had corrected a situation whereby A elements were assumed to have the content model rightly belonging to B elements and vice versa. The unit tests ran fine, since they don't give a hoot for semantics, and because the same code was used on both client and server, the end-to-end tests were also running fine.

I pinged the lead developer to see what had happened. He sheepishly admitted that the error had been caught only by old-fashioned code eyeballing! I suggested that what that showed was the lack of effective functional tests, which would have exposed the problem in a heartbeat. The code can be totally symmetrical and perfect-looking, but in the end it has to Do The Right Thing or it's worthless.

[link]

From: Martin Probst (Sep 12 2008, at 01:09)

Testing frameworks usually don't have a good way to "create something in the first test, modify it in the second, delete it in the third test", because iirc the public opinion on tests is that they should be entirely self contained.

In particular unit tests are supposed to need no configuration, no setup, and no external dependencies.

I know that this can be quite painful at times, and testing without e.g. a database can be impossible, so these laws should probably not be taken too strict.

However I'm quite convinced that tests that depend on the order in which they are executed are a pain. You will get weird interdependencies and in the end test failures where you (or the poor maintainer) cannot be sure which test exactly did something wrong. So for stuff like collections it might be better to factor out the resource creation in helper methods, and do it for each test case.

[link]

From: Charles Oliver Nutter (Sep 12 2008, at 03:38)

Obviously, if you come up with a nice REST testing framework, you need to call it RESpec.

[link]

From: Paul Moser (Sep 12 2008, at 05:27)

Have you taken a look at the story framework that is part of RSpec? - that's aimed at functional/acceptance testing.

We've been using that along with Net::HTTP for our acceptance tests of a REST based system.

A trivial example might be:

Scenario: a client requests an existing item

Given a item "<root><child>some text</child></root>" exists with id 1

When a client requests a GET of /items/1

Then the response should be successful

And the response should have a /root/child node with a value 'some text'

with some Ruby code like:

When "a client requests a GET of $path do |path|

req = Net::HTTP::Get.new(path)

@@resp = Net::HTTP.new('localhost', '3010').start {|http| http.request(req)}

end

Then "the response should be successful" do

@@resp.should_be_a_kind_of(Net::HTTPSuccess)

end

Then "the response should have a $path node with a value '$value' do |xpath, value|

# some XPath magic on @@resp.body

end

This seems to be working quite well for us at the moment.

[link]

From: Bob Aman (Sep 12 2008, at 05:42)

I have a project I've been needing something like "RESTUnit" for. I've just been using a custom HTTP client in my tests, but I really should be using something purpose-built for testing. It's becoming too much of a hack at this point. So there's demand. But I might write my own.

[link]

From: Dave (Sep 12 2008, at 06:47)

I've been doing TDD for a REST service for Gliffy. The code I wrote to do that is being refactored into RestUNIT, which is written in Java, but could certainly test any REST service.

The tests I created have a URL, parameters, and headers. The expected results contain a status code and content. By default content is compared verbatim, but the comparison mechanism can be customized.

Further, the test engine derives additional tests from the tests its given. For example, if a test is for a GET, it will then check to see if "Last-Modified" and/or "ETag" was provided and, if so, re-run the test as a conditional GET to ensure a 304 was returned. Or, if your API allows tunneling, tests can be derived to test that without having to duplicate a test.

At any rate, the engine I built at Gliffy is tailored to our API and was not created with a lot of planning and foresight. That's why I'm starting from scratch, building on the lessons learned.

As for <b>how</b> I did things TDD, I decided to test in-container (since the setup was relatively simple) against a known dataset. For tests taht do not modify the dataset (GETs, failed PUT/POST/DELETE), I simply created the URL and described the expected result. This runs against my server, breaking and I implement the feature.

For tests that <b>do</b> modify the data set, I created a concept of "functional" unit tests that basically perform several REST requests in a predefined order. If data was created, that data was removed at the end of the "functional" test. This allows all tests to run in any order without me having to rebuild the database each time.

Not the 100% pure ideal, but very effective for me.

[link]

From: Matt Liggett (Sep 12 2008, at 13:47)

While at Socialtext, I created a Perl library for testing HTTP (REST) services, called Test::HTTP. It consists of 2 parts: the first is a library that makes it easy to test some assertions one might have about a REST service.

The 2nd part is a syntax filter which lets you write nearly literal HTTP packets and have them translated in to calls said library.

A simple test looks like the below.

test_http 'echo test' {

>> GET /echo/foo

>> Accept: text/plain

<< 200

~< Content-type: ^text/plain\b

<<

<< foo

}

This sends a GET request with the given Accept header and then asserts that we get back a 200, that the Content-type header matches the regex /^text\/plain\b/, and that the body is exactly the string 'foo'.

Perl code may be interspersed with the HTTP packets, e.g. to parse an HTML, JSON, or XML body.

Download: http://search.cpan.org/dist/Test-HTTP/

There is a great opportunity to take this concept and fully develop it.

[link]

From: drewp (Sep 12 2008, at 22:45)

You might enjoy http://code.google.com/p/nose-trestle/ or at least some of the ideas in it. It's python's doctest concept, for a REST service.

With trestle, you write detailed docs about the service, and then trestle makes sure that the service behaves exactly as your docs said it would. Then you publish the docs for the users :)

[link]

From: Ali (Sep 13 2008, at 02:38)

twill is pretty good for this kind of thing:

http://twill.idyll.org/

I think you can add most of the other stuff you're looking for (xpath, atom) through the python interface.

[link]

From: JulesLt (Sep 16 2008, at 17:13)

Surely RESpecT, rather than RESpec. Aretha Franklin would be turning in her grave, if she were actually dead and all that.

[link]

From: Ken MacLeod (Sep 23 2008, at 19:55)

I've used WebUnit on a couple of projects. I don't remember which of these it was:

http://webunit.sourceforge.net/

http://www.mechanicalcat.net/tech/webunit/

Python's XPath really helps out.

[link]

From: Ole Matzura (Sep 27 2008, at 15:29)

Hi guys,

Ole from the soapUI (http://www.soapui.org) team here. We've just released the first beta of soapUI 2.5, which has rather extensive support for testing REST services, both with or without WADL definitions. There isn't much documentation yet, but I've created a blog entry to get one going (http://www.eviware.com/blogs/oleblog/?p=11) and I'll cover more topis (parameters, representations, etc) in the coming days..

I'm sure this can be improved, and input from REST-users like you would be awesome.. if you have the time..

Check it out at http://www.soapui.org/new_and_noteworthy_2_5.html, download from sourceforge: http://sourceforge.net/project/showfiles.php?group_id=136013&package_id=163662&release_id=628758

cheers!

/Ole

eviware.com

[link]

author · Dad · software · colophon · rights
picture of the day
September 10, 2008
· Technology (77 fragments)
· · Atom (91 more)
· · Ruby (93 more)
· · Web (385 more)

By .

I am an employee
of Amazon.com, but
the opinions expressed here
are my own, and no other party
necessarily agrees with them.

A full disclosure of my
professional interests is
on the author page.