Continuous integration doesn’t get rid of bugs, but it does make them dramatically easier to find and remove.
Continuous integration (CI) has become a staple in the world of agile software development, largely because it allows teams to detect problems early. In CI, developers integrate code into a shared repository several times a day, and each check-in is verified by an automated build.
However, despite its many benefits, CI can sometimes go terribly wrong — often because of harmful 'antipatterns' or bad practices. This article explores the antipatterns and bad practices that make CI ineffective and how you can avoid them.
What is an antipattern?
An antipattern is just like a pattern, except that instead of a solution it gives something that looks superficially like a solution but isn’t one.
Antipatterns are common 'solutions' to development problems that actually do more harm than good. While antipatterns appear to solve problems, they also create new problems and for CI to work properly antipatterns must be replaced with more effective and reliable alternatives.
Five common CI mistakes
#1 Infrequent check-ins
This antipattern makes development teams go against the fundamental principle of CI — they let code stay on local laptops for too long without it being committed and checked-in.
This is usually because the feature they’re working on needs an exhaustive set of changes to be complete. But, as the feature is being developed, other project members continue to check-in their work. As more code is gradually checked-in, merge conflicts arise that are difficult to solve and can severely delay integration.
How to avoid it
Split the feature into smaller, independent tasks that are followed by a commit and check-in. Small commits and frequent check-ins will make integration faster and hassle-free. Small commits also allow for specific tasks to be reverted, if necessary, rather than the whole feature.
For example, tasks for implementing a write to a database could include:
Add a repository layer to a write to database
Introduce a new endpoint and integrate it with the repository layer
Introduce different error responses when a write fails
Add validation checks for the request body
#2 Feature branches
This antipattern dictates that all feature development should take place in a dedicated branch. This makes it easy for multiple developers to work on features without disturbing the main codebase.
However, if feature branches don’t get changes from the mainline (the main branch for the repository) until it’s finished, the project isn’t even close to following continuous integration. In fact, this approach is more like continuous isolation, or CI theatre.
An isolated feature branch leads to multiple conflicts because by the time the feature is completed, the mainline will have tons of changes. This forces all developers to sit together and merge or important changes might be missed.
Let’s look at an example.
Source: Feature Branch by Martin Fowler
The image above shows a repository containing many long-lived feature branches. The branch by Professor Plum has local commits P1, P2..P5 and she has taken changes from the mainline at points P1–2 and P3–4. Finally at P1–5 she merges her feature to the mainline.
The branch by Reverend Green, which started at the same time, has local commits G1, G2..G6 and has changes from the mainline at G1–2, G3, which are before P1–5. When he decides to merge it to the mainline, he needs to incorporate all the changes done by Professor Plum. And because there are a lot of commits, it will be difficult to integrate them all.
How to avoid it
Avoid long-lived feature branches and maintain only one branch as a source of truth. Begin all development work with checking out from this branch — commonly known as the 'trunk' (or 'main' in Git terminology) — and pushing to it. This is called trunk-based development, because everyone develops against one branch: the trunk.
In projects which need to have pull requests approved before the merge, use short-lived branches for each task. You can merge these branches when the task is completed. These branches can be deleted and the next task could be taken on a fresh branch taken from mainline.
This approach increases the number of PR approvals, but also it makes the PRs shorter and crisp, which can be approved quickly. The larger a PR is the more review effort it requires.
Source: Feature Branch by Martin Fowler
In the image above, Reverend Green makes a local commit G1 and merges it to the mainline. Professor Plum makes a local commit P1 and before she can merge it into mainline, she needs to take commit G1, merge it locally with her changes and then push. Reverend Green does the same when he needs to push the next commit. It’s a series of small merges, without needing a huge merge event.
#3 Broken builds
In reality, a broken build is not a problem — the build pipeline is designed to give early feedback when something goes wrong. The problem, however, arises when a build stays broken for a long time.
If a broken build isn’t fixed quickly, it blocks other check-ins. Check-ins on a failing build then create more files, more changes and more dependencies that make it difficult to detect and isolate the defect.
How to avoid it
Be disciplined about fixing broken builds as soon as possible. When fixing the build requires a lot of time, revert the commit instead, so that one person can look into the build issue, while others get unblocked to check-in.
#4 Build time
One of the main reasons for taking a CI approach is to get faster feedback. A long-running build defeats this purpose by increasing the wait time for developers after check-in. Often, this leads to our first mistake, infrequent check-ins, because developers tend to check-in multiple commits at once to avoid long wait times.
How to avoid it
One reason for a longer build time is running all sorts of checks and tests as part of the build. To overcome this, configure the build to just compile the code and run quick unit tests. Push all the other checks and tests that need time, such as testing integration among different components, to the next stages in the pipeline.
The goal is to find the right balance between tests/checks and build time so a build is stable enough to move to the next task. Use the test pyramid (see image below) to achieve this balance.
In this method, only the unit tests will be run as part of the build, which take minimal time. As you can see from the pipeline image below, everything else can run later without blocking developers from moving to the next task.
A demo pipeline that runs different kinds of tests in different stages
#5 Build feedback
If teams don’t realize their build is broken, they’re likely to keep making check-ins on a failing build, leading to the issues we highlighted in mistake #3.
How to avoid it
Set up a CI dashboard showing the status of the build, and configure CI servers to notify teams by email or on Slack about broken builds. As soon as the team receives a broken build notification that should become their priority.
Source: GoCD - Slack Build Notifier by Ashwanth Kumar
While doing this, though, it’s important to avoid alert fatigue. When people become inundated with build notifications, they’re likely to start ignoring them, rendering the whole process ineffective.
So, instead of configuring the CI server to send notifications after every check-in, you should notify people when:
A new check-in fails, so that your team can fix it
A new check-in fixes a broken build, so that those who were waiting for the build to be fixed can check-in their code
Avoiding CI mistakes: a checklist
Make small, independent and frequent commits to the remote repository
Use trunk-based development or short-lived branches for small tasks
Make fixing a broken build your top priority, and don’t allow check-ins until the build is fixed
Reduce build time by getting faster feedback if something goes wrong
Set up a dashboard and a notification mechanism to inform team members about build status. Only send important CI notifications to avoid alert fatigue
Above are the antipatterns or bad practices of continuous integration that occur in a day-to-day project. Hopefully, the solutions shared will help avoid these problems and put your project one step closer to continuous integration and eventually continuous delivery.
Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Thoughtworks.
