In Praise of the ./go Script - Part II

In the first article in this series I introduced the concept of the ./go script, a unified interface to all the dev tooling on your project. In this article I'd like to talk about one of the most important properties of a good ./go script - isolation.

A well-maintained ./go script brings many benefits to a delivery team. Of those benefits two of the biggest are consistency across dev machines and easy onboarding of a new team member (or a new dev box). However, both of these benefits can only be fully realized if your ./go script is able to act in isolation of the dev machine's setup. Put another way, your ./go script shouldn't depend upon things which it can't manage itself.

"It works on my machine"

Let's dig into what I mean by isolation by way of a counter-example. We are on a team building a Java web app, but we use ruby to automate our browser-based testing. Our ./go script should be the one unified access point to tooling, so we've added a browser-tests command to it. This command simply shells out to "rake test:browser_based" in order to actually run those browser-based tests. The ruby code that makes up those tests depends on a variety of third-party libraries. Unfortunately that means our command will fail unless we've previously installed the correct gems [ruby library packages] onto our dev machine. Our ./go script is not isolated from the system setup of the machine it's running on. The presence or absence of certain ruby gems - actually specific versions of those gems - as well as the presence of ruby itself has an effect on the outcome of our ./go script command. We're stepping into the world of "works on my machine".

Those of you who are ruby developers are probably already aware of one solution to this particular example - Bundler. Bundler's job (well, one of its many overloaded jobs...) is to manage which versions of which gems are installed and to isolate a ruby runtime instance to only use those specific gem versions. By using Bundler the right way (hint: --path is your friend) we can create a ./go script which is able to manage its own set of gems, fully isolated from the machine's globally-shared gems. Similar technologies exist in most language ecosystems. There's virtualenv for python, npm for node, gradle/ivy/maven for java, and many more.

So something like Bundler can help to isolate our dev tooling from library dependencies. Good ./go scripts will take this isolation principle further by isolating - or at least managing - the language runtime entirely. Tools like rvm and rbenv allow this in ruby, and there are similar options in other ecosystems. If you're not inclined to build a ./go script that's sophisticated enough to actively manage the language runtimes it depends upon I would strongly encourage at least checking that the version available is appropriate, and failing fast with a verbose error message if that's the case. A big error message saying "Ruby < 1.9 is unsupported" is infinitely preferable to an obtuse error about ruby syntax half-way through a seemingly unrelated build process.

Isolate all the things

Besides language runtimes and libraries, another common thing for ./go scripts to depend upon is pre-compiled binaries such as zip, or perhaps phantomjs, chromedriver or terraform. These binaries are usually command-line tools which are used by some part of your build/test/deploy process. As such these binaries are all dependencies which should be actively managed. You have a few options in how to do so. The easiest solution is to simply check the binary into your project's source control repo.

An alternative is to introduce an ensure_foo script (where foo is your dependency). This script will check to see if the binary is available in some isolated project-specific directory. If not, the script will download a package for the binary from somewhere on the internet and unpack it into that directory. Your ./go script is then responsible for calling the appropriate ensure_ scripts before attempting to perform some command which depends upon the binary. These ensure_ scripts are checked into source control just like your ./go script.

A third option is to have your ./go script just check that a binary of the appropriate version is installed globally. If it's not, your ./go script should fail fast with a nice verbose error message which includes instructions on how to install said binary. That last part is important - remember that one of the aims of the ./go script approach is to reduce your README file to a singular instruction: "run ./go to get started". You can only get away with this smug one liner if the ./go script is self-documenting on how to resolve any dependency issues it runs into.

Abandoning your host

There will always be dependencies which are hard to manage in complete isolation of the host operating system. Yet a hermetically sealed, fully isolated dev toolchain is the best way to ensure maximum benefits from the ./go script approach. A solution which I like for achieving full isolation is to host your entire toolchain within a virtual machine using something like Vagrant, or better yet within a lightweight container using something like Docker. Your ./go script would then manage that VM or container to install all dependencies, and delegate all build operations into it. A neat aspect of this is that team members may not even know that you have virtualized the tools. After all, one of the benefits of the ./go script is that it presents an abstract interface into your tooling, allowing you to change how the tooling is implemented, and in this case even where the tooling is run. Teams which use this strategy may add a ./go nuke command which destroys the entire VM or container and then rebuilds it from scratch. This is a useful option to have available if you're seeing some strange behavior on your machine which others aren't seeing.

If you reach this level of isolation you can expect to be free of any disparity between machines running fresh instances of the toolchain. Each instance should be byte-for-byte identical from the OS up. This is true for both the tooling on developer workstations and importantly also for the tooling used by the agents building software destined for production deployment.

Shared state, the enemy of the good

As with many things in the world of software, shared state is the cause of much woe in the world of build tooling. Hopefully this article has shown how a disciplined isolation of your tooling from that shared state can help to avoid wasting energy on debugging confusing error messages. Having a ./go nuke in your back pocket is a great way to spend less time frowning at a screen and more time building awesome software. In addition, consistency between the way your code is built in dev and the way it's built for prod might help you sleep sounder at night.