The cluster you can explain

There is a recognisable moment in the life of a cluster: the day somebody asks why a pod restarted and the team’s honest answer is “we don’t know, but it fixed itself”. From then on, every decision is made by superstition. Replicas get added just in case, memory limits get doubled just in case, and the bill grows without anybody being able to point at what produces it.

An explainable cluster is not one with fewer features. It is one where four behaviours — scheduling, scaling, eviction and rollout — are configured on purpose and documented on a single page.

Requests and limits: the setting everybody copies wrong

Almost all the lost performance and almost all the overspend in a cluster come from here, and the reason is that the two fields look symmetric and are not.

The request is what the scheduler reserves: it determines which node the pod fits on and, in practice, it is what you are paying for. The limit is a ceiling enforced very differently depending on the resource, and that asymmetry is the whole point.

• Exceeding the CPU limit does not kill the process: it throttles it. The container waits for quota and tail latency explodes while dashboards show comfortable CPU usage. It is the hardest failure to diagnose without knowing it exists.
• Exceeding the memory limit does kill the process, ungracefully and without a useful dump. Memory is not compressible.

Hence the default recipe we apply: set the CPU request to what the service actually needs and set no CPU limit unless there is a concrete isolation reason; and set memory request and limit to the same value, calculated with headroom over the observed peak.

resources:
  requests: { cpu: "250m", memory: "512Mi" }
  limits:   {              memory: "512Mi" }
# no cpu limit -> the pod can use spare node CPU during a spike
# memory request == limit -> Guaranteed class, last candidate for eviction

That equality has a side effect that matters: the pod lands in the guaranteed quality-of-service class and is the last to be evicted when the node runs out of memory. Pods with a low request and a high limit are the first to go, and they are usually exactly the ones somebody configured “with headroom” to feel safe.

The numbers are not guessed. They come from the 95th percentile of real usage over two weeks, and get revisited quarterly because the service changes. A resource recommender in advisory mode does that calculation for you; in automatic mode, on services with irregular traffic, it produces surprise restarts.

Scaling: three loops that do not talk to each other

In a typical cluster three scaling mechanisms act at once, at different latencies and with no explicit coordination. Understanding their interaction avoids most surprises.

1. Horizontal pod autoscaling reacts in tens of seconds and can only use capacity that already exists on the nodes.
2. Node autoscaling reacts in minutes, because a machine has to boot, join the cluster and pull images.
3. Vertical autoscaling changes requests and, in doing so, recreates the pod, which is exactly what you do not want during a spike.

The practical consequence is that if you scale on CPU, by the time the new node is ready the spike has already run you over. For HTTP-facing services we prefer scaling on a signal that leads the pain: concurrent requests per pod, or queue depth. And we keep spare capacity as low-priority pods that do nothing but hold space, so a real spike evicts them immediately instead of waiting for a new machine.

behavior:
  scaleUp:
    stabilizationWindowSeconds: 0     # scale up fast
    policies: [{ type: Percent, value: 100, periodSeconds: 30 }]
  scaleDown:
    stabilizationWindowSeconds: 600   # scale down slowly, avoids flapping
    policies: [{ type: Percent, value: 10, periodSeconds: 60 }]

The asymmetry between up and down is not cosmetic: scaling down as aggressively as you scale up produces flapping, and flapping costs more money than the leftover capacity would.

Probes: the most common cause of a self-inflicted outage

The three probes have distinct purposes, and confusing them turns a degradation into a full outage.

• The startup probe protects slow-booting services: while it runs, the others do not count.
• The readiness probe decides whether the pod receives traffic. It is the one that should reflect temporary dependencies.
• The liveness probe restarts the container. It should only fail when restarting is genuinely the fix: an internal deadlock the process cannot escape.

The classic failure is a liveness probe that checks the database. When the database gets slow, Kubernetes restarts every replica at once, each one reopens its connection pool, and the database — which was merely slow — goes down entirely. Checking external dependencies belongs in readiness, never in liveness.

And clean shutdown matters as much as startup: when the termination signal arrives, the pod should stop reporting ready, wait a few seconds for the load balancer to notice, and only then drain in-flight connections. Without that grace period you will see errors on every deploy and blame something else.

Multi-tenancy: boundaries before trust

A cluster shared between teams needs four controls from the start, because adding them later means breaking workloads that already worked.

• Namespace quotas, so one team cannot consume all capacity with a misconfigured deployment.
• Default requests and limits, so an unconfigured pod does not inherit unlimited permission.
• Network policies with default deny. Without them, any pod in the cluster can talk to your internal database.
• Pod security standards enforced at admission: no privileges, no root, no writable filesystem except where genuinely needed.

Priority classes are the other control almost nobody configures and which decides what survives when the cluster fills up. If batch jobs and user-facing APIs share a priority, the scheduler will choose at random at the worst possible moment.

Cost, explained per service

A cluster bill has an uncomfortable property: you pay for nodes and consume with pods, so spend does not attribute itself. The useful conversation starts when each team sees three numbers for its own namespace.

• What they requested, which is what they pay for.
• What they actually used at the 95th percentile.
• The gap between the two, in euros per month.

In most clusters we audit that waste sits between 40% and 70% of requested capacity. Recovering it needs no expensive tooling: right-size requests with data and delete the pre-production environments nobody turns off at night.

Then come the structural levers, in order of return: interruptible instances for anything that tolerates eviction, with an on-demand pool for what does not; capacity commitments for the stable baseline; and consolidation of underused nodes, which is where a modern autoscaler clearly beats the classic one.

One warning about disruption: pod disruption budgets are what make all of this safe. Without them, a nightly consolidation can drain all three replicas of a service at once, entirely correctly from Kubernetes’ point of view.

GitOps and the real state

All of the above only holds if the cluster is declared in Git and reconciled continuously. The point of GitOps is not automatic deployment: it is that at any moment you can compare what should exist with what does, and that direct write access to the cluster can be removed because it is no longer needed to get work done.

The addition we always recommend is a minimal service catalogue: which namespace, which team, which SLO, which runbook, which dependencies. Half a page per service, in the same repository. It is what turns a cluster into a map instead of an inventory.

If your cluster works but nobody can explain why, send us two lines about the last restart you did not understand. You get an initial read and a ballpark quote within 24h.