Testing with FIRE
November 18, 2016
I forgot to put something on the Internet.
For almost 8 years now, I've held the belief that effective automated test suites have four essential attributes. These attributes have been referenced by other authors, and were the subject of a talk I gave at the Agile 2009 conference. But I was shocked to discover (that is, remember) that the only place they are formally documented is in my Continuous Testing book [Pragmatic Bookshelf, 2011], which is now out of date, out of print, and totally inaccessible to most of the Internet.
And so now, we blog.
The Four Attributes
Neatly summarized in the acronym, FIRE, the four attributes I look for in a test suite are:
- Fast
- Informative
- Reliable
- Exhaustive
I often consider these attributes when making decisions about automated testing. My understanding of how essential they are to my work has progressed to the point where almost all the software I have written professionally in the last few years--in different domains, languages, and platforms--has had such an accompanying unit/micro test suite. Let's examine them in detail...
Fast
When I say Fast, I mean that an entire suite of hundreds of tests runs in a second or so. This means that my individual tests should run in a millisecond or two. When writing code with tests, the duration of the test suite determines the length of my shortest feedback loop. Since this loop creates the very foundation of my development process, it's really, really important. As I described in One Second Services, keeping this feedback loop at a second or less also naturally creates an upper bound on the size of individual services in a larger system, and can serve as a guide when decomposing them.
Informative
An Informative test suite tells me what's wrong, and nothing more. If I introduce one bug, it should cause one failing test. It should tell me exactly what's wrong, and not spew lots of errors into the console. It should give me the error in context so that I know not only what the expected and actual behavior were, but also why the program was expected to behave that way. I find nested example structures in tests to be a good way to achieve this, but there are lots of ways.
Reliable
A Reliable test suite passes or fails consistently across test runs and environments. The results are completely deterministic, and solely a function of whether or not the system under test has the expected behavior. It is not dependent on the order the tests are run, user input, the state of a database, network access, the presence of ephemeral files, or any other state or process that is not completely controlled by the test suite itself (which, again, runs in about a second). This means that any sort of external database setup/teardown is probably not feasible on a per test basis.
Exhaustive
Exhaustive, in some ways, is the inverse of Informative. Just as I don't want a bug to produce multiple failing tests, I also don't want a bug to produce zero failing tests. An exhaustive test suite tests everything I care about. Does this mean every line of code I write is tested? Not necessarily. If I can change a bit of code and still feel confident enough to push that change to production with no further testing (manual or otherwise), then I don't need to write a test before making that change. Logging is a common example...although some log statements are important or complex enough that I do test them. I also miss tests sometimes, even when doing Test Driven Development, and you have to be prepared for that. We're all human, and I find that planning for the fact that you will make mistakes is essential when building software that people can trust.
So there we go. Officially on the Internet now. So if I start talking about the FIRE attributes to someone, and they give me that crazy look that reminds me I need to define my terms, I'll have a place to refer them. In fact, to help me remember where it is (and to give you a easy way to refer to it) I'm going to redirect to this post from fire.benrady.com. Feel free to use it in place of "that thing Ben said."
Comments