Almost every organisation we work with already has Terraform. What they do not always have is infrastructure as code. The difference shows up in one simple question: if somebody deleted the production account tomorrow, how long would rebuilding it from the repository take, and what would be missing?
The honest answer usually includes “well, the network was built by hand in 2021” and “that database is not in state”. This note is about closing that gap without a big bang, and about the design decisions that stop it from reopening.
State is a production database
The state file is not an implementation detail: it is the index of everything that exists. It deserves the same treatment as a database with customer data, and it almost never gets it.
- Remote backend with locking. Without locking, two concurrent applies produce a state matching no reality at all.
- Versioning and backups, with a rehearsed restore. A corrupted state with no backup is repaired by importing resources by hand, for days.
- Encryption at rest and audited access, because state contains sensitive values even when you marked them sensitive in the code.
- One state per environment and per domain. Never a single giant state for the whole organisation.
Splitting is the decision with the most medium-term consequences. A monolithic state makes every plan take ten minutes, makes any small change require permissions over everything, and lets a typo propose destroying the shared network. Our criterion is blast radius and change frequency: what changes daily does not share state with what changes once a year.
infra/
foundation/ # accounts, network, DNS, base IAM — rarely changes
prod/ staging/
platform/ # cluster, ingress, observability — changes monthly
prod/ staging/
services/ # databases, queues, buckets per service — changes daily
prod/checkout/ prod/search/ ...Layers communicate through explicitly published data — a read-only remote state or a parameter store — never through implicit coupling. If the services layer can modify the network, the split is decorative.
Modules: fewer than you think
Enthusiasm for modules produces two symmetric pathologies. The first is the wrapper module that just forwards twelve variables to the underlying resource and adds no decision at all: indirection with no value. The second is the universal module with forty variables and nested conditionals, which nobody understands and which breaks every time somebody adds a new case.
A module earns its place when it encapsulates a decision you want to make once for the whole organisation. Our favourite example is a database: the module does not exist to save lines, it exists to make it impossible to create one without encryption, without backups, without a maintenance window, and without alarms wired up.
module "orders_db" {
source = "git::ssh://git@internal/modules//postgres?ref=v4.2.1"
name = "orders"
environment = "prod"
size = "db.r6g.large"
# encryption, 30d backups, multi-AZ, alarms and credential rotation
# are not parameters: they are part of the module contract
}Modules are versioned with immutable tags and consumed at a pinned version, never from the main branch. A shared module pointing at main means a merge in the modules repository can change another team’s production plan without anybody asking for it.
The plan is the reviewable artifact
Reviewing an HCL diff reviews the intent. Reviewing a plan reviews the consequence, and they are different things: a three-line change can propose replacing an instance, and the diff does not say so.
So the plan runs in CI, gets posted on the pull request, and is saved as a file. The apply then runs against that exact file rather than re-planning: things can happen between plan and apply, and you want the apply to fail if the world no longer matches what was reviewed.
Policies run on top of the plan. That is where the rules we do not want to repeat in every human review get enforced.
# conftest / OPA over the plan JSON
deny[msg] {
r := input.resource_changes[_]
r.change.actions[_] == "delete"
r.type in {"aws_db_instance", "aws_s3_bucket"}
not has_approval_label
msg := sprintf("destroying a stateful resource: %s", [r.address])
}
deny[msg] {
r := input.resource_changes[_]
r.type == "aws_security_group_rule"
r.change.after.cidr_blocks[_] == "0.0.0.0/0"
msg := sprintf("rule open to the internet: %s", [r.address])
}With two dozen rules of this kind, human reviews stop being spent on checking encryption and tags and go to what only a person can judge: whether the change makes sense.
Credentials: the pipeline should not hold master keys
A CI runner holding long-lived admin credentials is the highest-value target in your organisation: anybody who can merge a malicious workflow owns the entire cloud. The practical alternative is federated identity per run, with different roles for planning and for applying.
- The plan role is read-only. It can run from anybody’s pull request, including external contributors, without risk.
- The apply role can only be assumed from the main branch, with an explicit condition on repository and ref.
- Credentials live as long as the job. There is nothing to rotate because nothing persists.
- Secrets the infrastructure needs are not written into variables: they are generated and placed in the secret manager, and the application reads them at boot.
Drift, and how to stop pretending it does not exist
There is always drift. Somebody touched a rule in the console during an incident, a managed service upgraded itself, a Kubernetes operator changed an annotation. The problem is not that it exists: the problem is discovering it in the middle of an urgent change.
We run scheduled drift detection that fixes nothing on its own and opens a ticket with the diff. Correction is always a commit, never a manual apply from somebody’s laptop. And when drift repeats on the same resource, the conclusion is not that people are disobedient: it is that the resource legitimately belongs to another system and should stop being managed here.
For emergency access we keep a break-glass role: broad permissions, short session, two-person approval and automatic auditing. It exists because in a serious incident the pipeline is not the fastest path, and pretending otherwise just pushes people to shared credentials in secret.
Bringing what already exists under Terraform
We never start by importing the whole estate. We start with what hurts most, which is usually what changes most or what already caused an incident through a manual setting.
- Write the resource in code with the configuration it already has in the cloud, not the one you wish it had.
- Import it with declarative import blocks, so the operation is reviewed in a pull request too.
- Iterate until the plan is empty. An empty plan is the proof that the code describes reality.
- Only then change the configuration, in a separate commit, so the diff of the change is readable.
- Close the door: remove manual write permissions on that resource type.
That last step is what makes the work permanent. Importing without removing console access is cleaning a room with the window open.
Expensive mistakes we keep seeing
- Workspaces used as environments with the same code and different variables: it works until production needs something staging does not, and then the code fills with conditionals.
- Count instead of keys for repeated resources: inserting an element in the middle shifts indices and proposes destroying and recreating everything after it.
- Providers without a pinned version: on a random Monday the plan changes because the provider changed a default.
- Secrets in Terraform variables: they end up in state in plain text, even when the output is marked sensitive.
- Automatic apply with no gate in production: infrastructure is not an application, and some changes have no rollback because the resource is gone.
The final goal is measurable and not at all abstract: anybody on the team, in their first week, should be able to open a pull request that changes production infrastructure, read the plan, understand it and merge it without phoning anybody.
If only one person runs your Terraform, send us two lines about how your state is split. You get an initial read and a ballpark quote within 24h.