Alerts that earn the page

When we audit a platform, one of the first numbers we ask for is how many alerts reached a phone last month and how many ended in a real action. The usual ratio sits between five and fifteen percent. The rest are notifications somebody muted, acknowledged without looking, or resolved by waiting.

That noise is not a comfort problem. It is a reliability problem: a team that has learned to ignore the pager reacts more slowly when the alert that mattered finally arrives. The cure is not a prettier dashboard, it is shrinking the set of things allowed to wake you up until every one of them justifies itself.

Three destinations, not one

The most common structural mistake is having a single channel for everything the system wants to say. We split by urgency and by owner, and the rule is that the destination is decided when the alert is written, not once it is already ringing.

• Page: something degrading users right now that a human can fix right now. If either half is missing, it is not a page.
• Ticket: something to fix this week. A disk at 70%, a certificate expiring in twenty days, a dependency with a known vulnerability.
• Dashboard: context for diagnosis. It notifies nobody and does not need a threshold at all.

Applying just this split usually removes half the noise in an afternoon, because it exposes how many alerts existed simply because somebody wanted to see the number somewhere.

Alert on symptoms, diagnose with causes

A CPU-at-90% alert says nothing about the user. It could be a pod doing its job perfectly well. A 4%-error-rate alert on the checkout endpoint says something, and it survives any internal refactor.

The practical rule: alerts based on observable user experience — availability, latency, correctness, data freshness — and cause metrics available on the dashboard for the diagnosis moment. If you have twenty cause-based alerts per service, what you have is a mental model frozen into thresholds that age badly.

There are legitimate exceptions and they are few: resources that exhaust irreversibly with predictable lead time, like disk space or database sequence identifiers. There, alerting on the cause buys you the margin the symptom would not.

SLOs: the boring part that does the work

An SLO is not a marketing number full of nines. It is an agreement about how much failure is acceptable within a window, and that figure is what turns “this feels bad” into a decision.

We always define it per user journey, not per service. “The checkout API responds in under 400 ms 99% of the time” is measurable; “the orders microservice is healthy” is not. A journey may cross six services, and nobody cares which one was at fault while the cart would not load.

# objective: 99.9% successful requests over 30 days
# error budget: 0.1% -> ~43 minutes of total failure per month

slo:
  service: checkout
  objective: 0.999
  window: 30d
  indicator: |
    sum(rate(http_requests_total{route="/checkout",code!~"5.."}[5m]))
    / sum(rate(http_requests_total{route="/checkout"}[5m]))

The error budget is the piece that changes conversations. If budget remains, the team ships features. If it is spent, the next iteration goes to reliability. It stops being an opinion contest between product and infrastructure and becomes a consequence of a number both sides accepted before the incident.

Fast burn and slow burn

Alerting directly on the SLO does not work: it either fires too late or fires on every micro-blip. What works is alerting on the rate at which the budget is being consumed, across two windows.

• Fast burn: at this rate the month’s budget is gone within hours. That is a page, at any hour.
• Slow burn: a small, sustained degradation that eats the month in six days. That is a ticket during working hours.

The short window inside each rule stops you from receiving notifications after the incident is already over, which is exactly why people end up muting entire channels.

- alert: CheckoutFastBurn          # 14.4x -> budget gone in ~2 days
  expr: |
    slo:error_ratio:rate1h{service="checkout"}  > 14.4 * 0.001
    and
    slo:error_ratio:rate5m{service="checkout"}  > 14.4 * 0.001
  for: 2m
  labels: { severity: page }

- alert: CheckoutSlowBurn          # 3x -> ticket, not a page
  expr: |
    slo:error_ratio:rate6h{service="checkout"} > 3 * 0.001
    and
    slo:error_ratio:rate30m{service="checkout"} > 3 * 0.001
  for: 15m
  labels: { severity: ticket }

Cardinality, or why your metrics bill looks like that

Observability cost is not driven by the number of services, it is driven by the number of distinct time series. And a distinct series is born every time a label takes a new value. Putting user ID, full URL with parameters, or trace ID into a label multiplies storage by orders of magnitude.

What we do: normalise routes before labelling, keep a short allowlist of labels per metric, and validate in CI that nobody introduces a high-cardinality label. Fine detail goes to traces and logs, which are designed for it and get sampled.

# bad: one series per product and per user
http_requests_total{path="/product/8a91f/reviews?user=41288"}

# good: one series per route template
http_requests_total{route="/product/:id/reviews", method="GET", code="200"}

Trace sampling deserves an explicit decision. Head sampling is cheap and throws away exactly the interesting requests; tail sampling costs more infrastructure but guarantees you keep the errors and the slow tail, which is the only thing you will actually look at.

The alert should carry context, not a riddle

A notification that reads “CheckoutFastBurn firing” forces the person on call to rebuild the state from scratch while half asleep. Every alert that reaches a phone carries at minimum four things: what the user is seeing, since when, what changed recently, and direct links to the runbook and to the dashboard already filtered to the affected service.

The link to the last deploy is the biggest time-saver. In most incidents the cause is a recent change, and the first useful question is almost always “what shipped in the last hour?”. If that answer sits inside the alert itself, you save the first ten minutes of every incident.

A useful runbook is short and operational: how to confirm it is real, how to mitigate, how to escalate and to whom. It is not architecture documentation. If it is longer than one screen, nobody reads it at three in the morning.

The weekly on-call review

No alerting configuration survives contact with reality intact, so the mechanism matters more than the initial state. Half an hour per week, with the list of everything that fired, and for each item one of three decisions.

1. It was real and needed a human: it stays, and if diagnosis was slow, the runbook improves.
2. It was real but could wait: it drops to ticket.
3. It needed no action, or the action was automatable: delete it or automate it. No sentimental exceptions.

The two numbers we track in that meeting are the share of actionable alerts and out-of-hours interruptions per person per week. When the first rises above two-thirds and the second drops below one, on-call stops being a tax on the team.

What observability does not fix

• If there is no owner, the alert bounces. Every SLO has a named team, not a generic mailing list.
• If there is no fast mitigation, the alert only tells you earlier that you are about to suffer. A flag, a rollback or a rate limit is worth more than one more dashboard.
• If the alerting system depends on the cluster it watches, it will go quiet exactly when you need it. The notification path is tested and hosted separately.
• If nobody rehearses, the runbook is fiction. One drill a quarter is enough to find the broken links and the missing permissions.

If your team is on call and nobody trusts what fires, send us two lines about what fired last week. You get an initial read and a ballpark quote within 24h.