Part 2. Part 1 covered trunk-based development. This is what comes next: the practice that most teams claim to do but almost none actually are.
The difference between "we have a CI pipeline" and "we practice continuous integration" is the difference between owning running shoes and running.
What it is
CI is the practice of integrating code to trunk (main, master) continuously, at least once a day. Every integration is verified by an automated build and test suite. If the build breaks, fixing it is the team's immediate priority.
Nobody pushes on a broken build. If the build is not fixed within a reasonable timeframe, the breaking change is reverted to unblock the rest of the team.
How to actually do it
If you have trunk-based development in place, CI requires three things:
- Small commits. Each commit is either a complete, working increment or an unfinished increment that lives under one of the decoupling strategies mentioned in Part 1. It is a change that leaves the system working.
- A fast automated build. The regression tests must run on every single commit. If this takes more than ten minutes, it is too slow. Developers will stop running it.
- Fix broken builds immediately. A red build means trunk is broken and nobody can integrate until it is fixed. The team stops and fixes it, pulling a cord and "stopping the line".
On separate QA teams. Teams that practice CI bake quality into the product through executable specifications written in tandem with the production code instead of trying to inspect it in later. A separate QA process hinders integration because it introduces a batch-and-queue stage between writing code and knowing if it works. The developer commits, moves on to the next thing, and days later gets a defect report from a different team with different context.
Quality is built in to the product by the people writing the code through TDD, pairing or ensemble programming (QA included), and automated tests that run on every commit. A separate QA team testing after the fact is an attempt to inspect quality into a system after it has already been built. At that point, it's too late.
The pipeline
A CI pipeline verifies every commit to trunk. Keep it fast.
Stages
- Lint and static analysis. Catches formatting, style violations, some security concerns, and obvious errors.
- Unit tests. Fast and isolated. The bulk of the test suite.
- (Component) Integration tests. Verify components work together across local boundaries.
- Build artefact. Produce the deployable thing.
- Deploy. Automated deployment to non-production and production environments.
On distributed systems. The above example is the shape of a bare minimum pipeline for a single deployable unit/monolith. Distributed systems introduce complications, as each service now needs its own pipeline, and a mechanism to detect if a change to one service breaks another without having to wait for every other service to be deployed to a shared environment.
Contract-driven development solves this problem by allowing each service to be tested in isolation without standing up the whole system.
One path to production
The CI pipeline should be the only way to get code from the local machine into production. Shortcuts and "hotfix" branches are a symptom of a missing CI practice.
Test-driven development
TDD matters for CI because of what it produces:
When you write the test first, you get two things: a test suite that actually covers the behaviour (because every behaviour was driven by a test), and small, incremental steps (because you only write enough code to pass the next test). Both of these are exactly what CI needs, good coverage and small commits.
Critically, you are building up a regression suite beside the code. This regression suite serves to replace the separate "testing phases" that denote waterfall development.
Teams that write tests after the fact tend to have gaps in coverage and larger, riskier commits. Teams that practice TDD tend to commit more frequently and with more confidence.
TDD doesn't just include unit tests. The process often begins with something higher level, such as a contract test or integration test.
Common misconceptions
"We have a CI pipeline, so we do CI"
Most teams that say they do CI are running a pipeline on pull requests against feature branches. CI requires the pipeline to run against trunk on every commit. Everything else is just a build system.
"We can't do CI because we need code review for compliance"
Pair programming satisfies review requirements. Two people wrote and reviewed the code together, in real time. The commit log can be used to show who was on the story. If your compliance framework requires evidence that code was reviewed before production, a pairing record is stronger evidence than a rubber-stamp approval on a PR that sat in a queue for days, then was read by someone else, who was working on something else.
If your organisation genuinely requires asynchronous sign-off, find out exactly which policy mandates it and what it actually says.
Always ready to release
If you are doing trunk-based development and CI, the head of master is always in a releasable state. Every commit passed the pipeline. Every commit is a complete increment or under a deployment-release decoupling strategy.
On how long it takes. Ask your team: how long does it take to get a one-line change to production? If the answer is days or weeks, the bottleneck is delivery. The code was ready in minutes. Everything after that is process, ceremony, and waiting.
In this case, the Theory of Constraints and Value Stream Mapping can be used to help to identify and release bottlenecks in the system.
Goldratt, E.M. (1984) The Goal. Great Barrington: North River Press.
This is part of the Mainline documentation. Mainline is a project management tool for teams that ship continuously.
