One path to production

Most “platform projects” die in month two: a catalogue of tools, three environments that do not resemble each other, and nobody willing to press the production button on a Friday. The antidote is boring and it works: one path, commit to production, for one real service.

This note is the long version of that path: what each stage contains, what guarantee it buys, what we measure, and where it usually breaks. The examples use GitHub Actions, Kubernetes, Terraform and Argo CD because that is what we meet most often, but the contract translates without drama to GitLab CI, Nomad or ECS.

What counts as a path

A path to production is not a green pipeline. It is a chain of guarantees where every link can be audited after the incident, at three in the morning, by somebody who did not build it.

• Reproducible build: the same commit produces the same artifact, without depending on the cache on anybody’s laptop.
• Immutable, versioned artifact: one image by digest, never a moving tag.
• Provenance and signature: you know which pipeline produced it and from which source.
• Infrastructure declared in Git, with a reviewable plan before apply.
• Progressive delivery with automatic abort criteria.
• Rollback in one command, rehearsed, with a known time budget.
• Traceability: from an alert to the line of code in under five minutes.

If any of those is missing, you do not have a path yet — you have a rehearsal. And rehearsals collapse exactly when the traffic is real.

Stage 1 — The reproducible build

Almost every pipeline we inherit fails here and nobody knows it. The classic symptom: rebuilding the same commit two weeks later produces a different image, because the Dockerfile drags an unpinned package install, a floating base tag, and a lockfile that gets regenerated in CI.

The rules we apply are short and not negotiable: base pinned by digest, dependencies installed from a lockfile in strict mode, multi-stage build so the final image carries no toolchain, non-root user, and normalised timestamps if you need bit-for-bit identical builds.

# syntax=docker/dockerfile:1.7
FROM golang:1.22-alpine@sha256:2c8e...f01d AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY . .
RUN CGO_ENABLED=0 go build -trimpath \
      -ldflags="-s -w -X main.version=${GIT_SHA}" \
      -o /out/api ./cmd/api

FROM gcr.io/distroless/static-debian12@sha256:9be3...aa71
COPY --from=build /out/api /api
USER 65532:65532
ENTRYPOINT ["/api"]

The proof it works is not that it compiles: it is a weekly job that rebuilds the commit currently in production and compares digests. If they differ, reproducibility broke, and you want to know that before the next incident, not during it.

Stage 2 — The checks that actually pay

The usual mistake is bolting twenty gates onto the critical path and ending up with a 40-minute pipeline the team learns to skip. We split by cost and by consequence: whatever blocks a merge must be fast and deterministic; slow things run in parallel or post-merge, without holding anybody up.

Blocking (target: under 10 minutes)

• Compile, unit tests, and contract tests for the APIs another team consumes.
• Lint and static analysis with security rules, not style rules.
• Secret detection on the diff, not on the whole history.
• Manifest and infrastructure plan validation: policy as code against what will actually be applied.
• Migration compatibility check: no migration may break the previous version of the service.

Non-blocking

• Container vulnerability scanning, with a debt budget and a deadline instead of a “failure” everybody ignores.
• Long end-to-end suites and load tests against the pre-production environment.
• Infrastructure drift detection against the declared state.

The practical rule: if a check fails and the team’s answer is “re-run it”, that check is not a gate, it is noise. Either you fix it until it is deterministic, or you take it off the critical path.

Stage 3 — Promote by digest, never rebuild

The most expensive antipattern we see: each environment rebuilds the image from its own branch. Then staging and production do not run the same binary, and “but it worked in staging” becomes company policy.

You build once, on merge to the main branch. What travels between environments is the digest, and the only difference between environments is configuration injected at runtime.

# promotion: a commit in the state repo, not a new build
$ ARTIFACT=registry.internal/api@sha256:41b0...cc9e
$ yq -i ".spec.template.spec.containers[0].image = \"$ARTIFACT\"" \
     envs/prod/api/deployment.yaml
$ git commit -am "promote api $ARTIFACT to prod" && git push

That commit is the audit trail. Who promoted, which digest, when, and against which configuration revision. You do not need a parallel ticketing system to reconstruct the history of a deploy.

Stage 4 — Infrastructure with a reviewable plan

Terraform applied from laptops is not infrastructure as code: it is a shared script with hidden state. The plan has to run in CI, get posted on the pull request, and be applied only from the main branch, with state locking and federated credentials instead of static keys.

Two details that save incidents: split state by environment and by domain so a networking apply cannot touch the database, and explicitly mark the resources whose destruction requires human approval.

permissions:
  id-token: write        # OIDC to the cloud, no static credentials
  contents: read
  pull-requests: write

jobs:
  plan:
    steps:
      - run: terraform plan -lock-timeout=120s -out=tf.plan
      - run: conftest test tf.plan --policy policy/   # policy gates
      - run: terraform show -no-color tf.plan > plan.txt
      - uses: ./.github/actions/comment-plan

Stage 5 — Progressive delivery with an abort criterion

A canary without an automatic abort criterion is a progress bar. What makes a canary useful is the opposite promise: if the signals leave the agreed band during the analysis window, the system rolls back on its own, without paging anybody.

The signals we use are almost always three: the service’s own error rate, p95 or p99 latency compared to the stable version, and saturation — memory, queues, database connections. We add one business metric when it exists, because a service can return two hundreds and still be doing damage.

strategy:
  canary:
    steps:
      - setWeight: 5
      - pause: { duration: 10m }
      - analysis:
          templates: [{ templateName: error-rate-and-latency }]
      - setWeight: 25
      - pause: { duration: 15m }
      - setWeight: 50
      - pause: { duration: 30m }
# analysis fails -> automatic abort -> stable version comes back

The window matters. Five minutes at 5% of traffic will not catch a memory leak or a problem that only shows up with a cold cache. We prefer long windows at low traffic over short windows at high traffic.

Stage 6 — The rollback contract

“We can roll back” is a belief until somebody times it. Our contract is explicit: one command, a time target, and a short list of what a rollback does not fix.

1. Revert the deployment to the previous known-good digest, in under two minutes.
2. Database migrations always go first and are backward compatible: add the column, ship code that writes it, ship code that reads it, and only then retire the old shape.
3. New behaviour lands behind a flag, so turning it off does not require a deploy.
4. Queue messages are versioned, because rolling back a consumer while new events are in flight is an incident inside another incident.

And it gets rehearsed. One rollback per quarter, during working hours, on real production. If that sounds frightening, the fear is the data point: it means you do not have the contract, you have a paragraph in Confluence.

What we measure along the way

Four numbers, taken before we touch anything and reviewed weekly. Not for a pretty dashboard, but because they are the argument when somebody asks why this deserves budget.

• Commit-to-production time, at p50 and p90. The p90 is the one telling the truth.
• Deploy frequency per service, not aggregated across the org.
• Share of deploys requiring manual intervention or rollback.
• Time to restore service after a failed deploy.

A pattern that repeats: deploy frequency does not rise because the pipeline got fast, it rises when rollback stops being frightening. The bottleneck is almost always trust, not CI minutes.

Where this breaks in practice

• Secrets: as long as long-lived credentials sit in CI variables, the pipeline is a privilege escalation path. Federated identity per job, least privilege, and no secret that outlives the workflow.
• Environments that do not match: if pre-production has a third of the memory and synthetic data, the canary is not measuring what you think. We prefer a smaller pre-production that is identical in shape over a large one that is different.
• Flaky tests: 2% random failures across thirty jobs means nearly half your merges start with a false failure. Quarantine it, fix it, or delete it, but do not live with it.
• Migrations coupled to the deploy: if schema and code must move together, there is no rollback, there is backup restoration.
• A single human approver as the gate: it does not add safety, it adds a queue. The useful review lives in the pull request, not in front of the red button.

Then you copy it, you do not redesign it

We do it first on the service that deploys the most, not the cleanest one. The point of weeks 2–4 is not to cover the org chart. It is that someone on the client team can ship without us on the call.

Once that path exists, the rest of the work is modules: the same reusable workflow, the same checks, the same rollback contract, the same metric naming scheme. The second service should take days, not weeks; if it takes weeks, the first one was not properly encapsulated and we go back to it before moving on.

Kubernetes, Terraform or GitHub Actions are not the product. The product is that Friday stops being an event.

If you are stuck in a six-month platform redesign, send us two lines about the service that hurts most. You get an initial read and a ballpark quote within 24h.