One Second Services

Microservices have problems. Monoliths have problems. How do you wind up in a happy middle? Here's what I do.

As I talked about in my new book, I'm skeptical of starting systems with a microservice architecture. Splitting a new system across a bunch of different services presumes you'll know how to organize things up front, and lots of little microservices can make refactoring difficult. 
So I start with a small monolith. As I build, I add tests. My tests run very fast...hundreds of tests per second. I run the tests automatically on every change to the code, so speed is essential. 
When the entire test suite for the monolith starts creeping up into the 800-900ms range, I start to notice the time it takes to run the tests, and then I know it's time for this monolith to split into two smaller services. By then, I usually know enough about the system to know how to split it cleanly, and because I have a good test suite, refactoring it into this shape is easy. 
It usually doesn't split in half...80/20 is more common, but it's enough to speed my tests up again. After that, I just keep growing the services and splitting as necessary. The last system I built like this wound up with dozens of services, each with dozens or hundreds of tests which never take more than a second to run. Right in the Goldilocks Zone, if you ask me.

Candy or Death: The Automatic Halloween Candy Dispenser

Let's start with a word problem. Assume you live in a busy trick-or-treating neighborhood and that, on average, a group of four rings your doorbell every minute and takes 1/2 oz of candy per person. If you leave a bowl full of 2 lbs candy on your front step, how much time will elapse before it will all be gone?

Answer: It's a trick question. Given that that the NIST's strontium lattice clock, the most precise clock in the world, is only capable of measuring time in femtoseconds, nobody knows how long it takes. Humanity has no device capable of measuring the infinitesimal amount of time it takes for unattended candy to disappear from a doorstep on Halloween.

To work around this problem, I decided to build a device to hand out candy on Halloween in a more...civilized...manner.

Continue reading "Candy or Death: The Automatic Halloween Candy Dispenser" »

Refurbishing a Mail Slot and Doobell

When I moved into my house, the mailbox was in pretty sorry shape. It was corroded, and the mail flap was stuck open. On top of that, it had an integrated doorbell that didn't work. Lastly, the entire border of the mailbox was covered with an ugly and aging caulk job, complete with rotting natural fiber insulation that had been there since god-knows-when.

2013-11-19 12.24.01

The first thing I did was try to get the doorbell working. Here you can see the original cloth insulated wires that were probably installed when the house was built in 1929. 
2013-11-19 12.25.14
Surprisingly, the wires still seemed to be connected, and I found where they came out in the basement. So after replacing the button with a metal momentary pushbutton, I tried building a simple doorbell using an XBee radio and connecting it to the wires. I had a Raspberry Pi XBee basestation that I use for other projects, and I wrote a simple script to send a Pushover notification to my phone whenever someone pressed the button.
2013-07-15 21.48.43
Unfortunately, there was an intermittent short in the wires. After getting 10 or so spurious doorbell notifications on the first day, I knew I had to take more drastic measures.
I found the Honeywell RCWL300A wireless doorbell on Amazon. I ordered it hoping that I could modify it to be activated by the doorbell that I had already bought, since I didn't want to just stick a big plastic button on the outside of my house. I opened it up and started tracing the circuit with the macro lens I have for my iPhone.
2014-08-29 19.10.38
There were 5 solder points on the board, and I found two of them seemed to be connected to the doorbell button. Using a couple of leads, I confirmed that connecting the 2nd and 4th pins would trigger the doorbell. Huzzah! Then I soldered on a couple of wires, drilled two holes in the case, and then tucked the device up inside my mail slot. I then ran the wires up to the doorbell button, and I had a working doorbell (with a wireless chime to boot!).
2014-09-01 13.39.09
Then I turned my attention to the mail slot itself. The first step was just to take it apart, and considering that it had completely rusted open, this was rather challenging. The mail slot cover moved on an axle that was either welded or glued into the surrounding cover.
2014-08-18 18.29.02
I wound up just cutting it out with a dremel. I replaced it with a threaded rod and held it in place with a couple of nuts and lock washers. Then I took off all the corrosion with a combination of sandpaper and a wire brush wheel. Then I painted it with about 5 coats of Rust-o-leum Oil Rubbed Bronze metallic spray paint. I had my doubts about the paint adhereing to the surface, but it turned out pretty good.
2014-09-01 14.51.20
Next, I replaced the natural fiber insulation with some closed cell spray foam, which worked pretty well, although it was really a mess to apply. I then repositioned the mounting screws to shift the mailbox up a bit to cover the gap. 
2014-09-14 15.10.13
Of course, like every project, there a still a few rough spots I'd like to clean up. But as you can see, the end result is a big improvement. 


Why The Post Scarcity Society Will Not Be Star Trek

As a technologist, I often think about Marc Andreesen's assertion that software is eating the world. It's a very provocative statement, but I can't really disagree with it. Whether we like it or not, we are building a new society in which labor is devalued. Thought workers are quickly becoming the only essential employees for many organizations. The middle class, who up until now has been dependent on their ability to trade labor for capital, is being destroyed.

Hope has been offered by the idea that we may be building a "post-scarcity society." One in which trading your labor for subsistence is no longer necessary. If we are able to optimize to cost of everything down to free or nearly free, the proponents argue, we might wind up with a new society that looks something like Star Trek. And who wouldn't want to live in the Star Trek universe?

Creating a society that even remotely resembled the Star Trek universe would surely be mankind's greatest achievement. Neil DeGrasse-Tyson once looked into why civilizations do great things like that. He was trying to figure out how to rekindle interest in space exploration. He found that all civilizations throughout history have only ever done something great for one of three reasons:

  • Defense (aka War)
  • Economics
  • Religion

In the canon of Star Trek, humanity's modern renaissance happened when we were first visited by the Vulcan race. Alerted to our existence by the first successful test of a warp drive, Vulcans landed on earth with a message of peace and friendship. The course of all of humanity was changed in an instant, because that event has _all three_ of the elements that DeGrasse-Tyson describes. The Vulcans represented a potential ally in a galaxy of previously unknown aggressors. They were a new conduit for trade and commerce, opening new markets and providing new technology. Finally, proof of the existence of an intelligent race other than humans, was for the bulk of humanity, something that completely reshaped their sense of self and spirit. If you doubt the religious significance of that event, consider this: Spock was only _half_ vulcan. If having a new species to breed with doesn't change your ideas about God, nothing will.

The thing that created Star Trek was not post-scarcity. Post-scarcity was the effect, not the cause, of human-extraterrestrial first contact. The Star Trek universe was created through the unification of all of humanity into a singular guiding goal: The exploration of space. That single event was so powerful as to bring about all the changes necessary for humanity to move past the industrial revolution, and view an individual's contribution of labor not as a prerequisite for societal approval, but as an inefficiency to be happily optimized away.

We don't have Vulcans. We have the Internet. And they are not the same thing.

While the Internet is born out of military roots, its effects are primarily economic. It does not have the transformative effect that contact with a sentient alien species would have. In absence of this, we have no reason to believe that the world we are building will, in any way, resemble the sci-fi fantasy that we all hope it would.

The world we are building does not have a powerful, unifying force behind it. It has only self-interest and the legacy of societal structures that are unable to deal with new realities. America, in particular, is culturally ill-equipped to handle these new realities. The new world we are building is much more likely to be a technological feudalism than it is to be a utopian commune. If we don't take steps to shape its direction now, we will not be given a second chance.

Vim's undo list isn't a list. It's a tree.

Vim's undo list isn't a list. It's a tree...meaning that it keeps track of all the edits you make after having "undone/redone" something. Putting this power to use can be a bit daunting, unless you keep a couple of simple vim commands handy.

First, let's create an example to work with. Make a new buffer and type three things (switching back to normal mode after each line to produce three separate changes). You should wind up with something like this:


Now let's say I undo the change that created "third", and then change "second" to "2nd". Now I have this:


You can undo and redo to remove and re-add "second" and "first", but there's no way to bring "third" back, right? In most editors, it would be lost.

Actually, in Vim, there's at least three ways to get it back.

The :earlier and :later commands will move you back and forward in time across the undo tree. It's basically a time machine built into vim. At this point, to bring back "third" we just need to use the 'earlier' command, like so:

:earlier 1

That will bring us back one edit in time (rather than in the undo/redo path) leaving a buffer that looks like this:


If you prefer using 'g' rather than command mode, g+ and g- from normal mode will move you across the edit tree one step forward or backward, respectively.

And of course, no self-respecting time machine would be complete without time, so :earlier and :later both take time as an argument as well. To jump back to the state of your code 30 seconds ago, just type this

:earlier 30s

It works with [m]inutes, [h]ours, and [d]ays too. Being able to jump back and forth between changes I was making days ago, in just a few keystrokes, is just one of the many reasons I love Vim.

Ben Rady is the author of "Continuous Testing with Ruby, Rails, and JavaScript". If you've enjoyed this post and would like to read more, you should follow him on Twitter.

Lines of Code is the Best Software Productivity Metric

Lines of code is a great metric for productivity. Not only is it not broken, I would argue that's it's clearly the best. The important question to ask about this metric is "How does programmer productivity relate to value delivered to a customer?"

If you want to measure what programmers produce, lines of code added is the only metric that makes sense. Firstly, it's objective. It's easily obtained from source code repository logs. Many existing tools already can measure and track it. It can be applied to almost any programming language. Finally, because their workflows are (usually) so highly automated, they're able to measure their work product far more accurately than most. And while there are other things that programmers create (emails, documents, meeting invites, coffee stains...) by definition they write code. They are uniquely able to solve problems by writing software, so measuring the production of that software makes sense.

So if we can measure productivity this way, why doesn't it seem to be useful when we do? In manufacturing, higher rates of production almost always leads to more value. I know that all you Lean Production advocates are yelling and screaming about inventory management and muda right now, but if you were producing and selling 1000 things a day and now all the sudden you can produce 2000 of them at the same cost, there's net value there. If you can find a counterexample, please go ahead and produce those extra 1000 things and give them to me. I'll sell them on Amazon.

Software does not work like this at all. If your team was producing 1000 lines of code a day, and now all the sudden they're producing 2000, you really have no idea what impact this has on value delivered. Are you creating twice as many features? Are you refactoring to pay off technical debt? Or are you just thrashing around trying to find a design that fits your domain? Maybe you're adding features, but not ones that users need. Maybe you're just making your system harder to use.

The reason for this productivity paradox is simple: Software is not an asset. It is a liability. From a financial standpoint, creating it is much more akin to leasing office space than it is to producing finished product. It represents an ongoing cost of doing business, that may or not actually result in any value being delivered to a customer. Lines of code in a codebase are liabilities that you have to write, test, document, compile, deploy, etc....and the less of it you have for a given solution, the better.

Programmers solve problems, which can be assets. They often do so by creating software, thereby producing liabilities. In software, production is an expense. Competent programmers are able to create more value than cost when they do this, but not all programmers do. The best programmers can create value by removing code and the best solutions in software can often be achieved through simpler functionality rather than just more of it.

So there's nothing wrong with "lines of code" as a productivity metric. It's fine. You've been using it wrong. To understand the net value created by programmers, you have to look past productivity. You have to look how and why and when people are using the solutions you create. You have to create feedback loops that extend all the way to the point where value is actually delivered, and you have to find ways to make those feedback loops fast and responsive. Otherwise, you're going to spend your time optimizing for metrics that don't matter.

Ben Rady is the author of "Continuous Testing with Ruby, Rails, and JavaScript". If you've enjoyed this post and would like to read more, you should follow him on Twitter.

Poor Man's Location-based services with arp-scan and cron

Have you ever wanted to have your computer do something when you came home, or when you left? Here's a cheap and easy way to do it.

The arp-scan utility (available in most decent package managers) can scan your local network for active network clients. Just use the --localnet option. If get a response from your mobile phone (identified by mac address) on your local network, then you're (probably) home! Here's an example:


if arp-scan --retry=3 --localnet | grep 'a4:a4:a6:a7:aa:ad'; then 
  echo "Daddy's home!"
  echo "Daddy's gone."

Something like this could be easily run from cron (as root). A more sophisticated version would probably need to track the current state, possibly by touching and deleting a file, as to only act when it changes.

Something else to note: My phone, it seems to ignore too many arp requests in a row. Of course YMMV. Once a minute seems to work though. So the --retry option is in there just in case one of the arp packets gets lost.

Arduino Controlled RC Car

The problem with RC cars and little kids is that RC cars are actually rather hard to control. And really, rather than standing there controlling the car, most kids would rather be chasing it around (at least, most of my kids would).

So I decided to see if I could build an RC car that drove itself, using an Arduino nano rather than a remote control.


Here's the parts list (aside from wires, solder, etc...)

The first step was figuring out where the RC chip leads were connected.

The answer turned out to be:

  • R17 -- Backward
  • R18 -- Forward
  • R23 -- Right
  • R24 -- Left

After that, I just needed to wire those leads into the arduino. I started out testing with an Arduino Uno, and once everything was working, I wired it up to the Nano I bought to actually use in the car.

After I confirmed that was working, I put the car back together, drilled a hole in the top for the LED, and hot-glued the PING))) sensor to the front at about a 10 degree angle upwards.


Arduino's libraries are so easy to work with, you almost forget you're writing C++. So I wrote a simple program to control the car. Since there's only one ultrasonic sensor, I needed a simple approach to avoiding obstacles (Summary: turn left).

Final Result

Here's the car driving around my toy-littered basement.

There are still some improvements to be made here. Finding a car that ran on 9 volt power would have greatly simplified the design...although most of those cars are much more expensive than the one that I bought. And I need to add a switch to control the arduino.

Of course, this begs the question, is it still an "RC" car if it's not controlled remotely?

The Simplest Continuous Testing

Here's a simple bash script I use to do things to files whenever they change. It takes a command as an argument and runs that command with the name of a changed file. Many times, it's all I need to do continuous testing.

# Requires that the inotify-tools package be installed.
wait_cmd="inotifywait -m -r --format %w%f -e modify ."
$wait_cmd | grep --line-buffered $filter | while read file; do 
  $cmd $file