Recover Faster from Incidents with Observability

Every system fails eventually. What separates resilient companies from the rest is not the absence of incidents but how quickly they notice, understand and fix them. And the cost of being slow is real: in ITIC’s 2024 survey, over 90% of mid-size and large enterprises said a single hour of downtime costs them more than $300,000.

Observability is what turns an incident from “something is wrong, nobody knows what” into “the payment service started timing out after the 14:02 deploy”. In New Relic’s 2024 Observability Forecast, organizations with full-stack observability reported 79% less downtime than those without it.

Monitoring vs. observability

Monitoring tells you when something you anticipated goes wrong: CPU above 90%, a health check failing. Observability lets you ask new questions about failures you did not anticipate — why are requests from one region slow, but only for customers on the new plan?

You need both, built on three kinds of telemetry that work together:

• Metrics — cheap, aggregated numbers over time. Great for dashboards and alerts.
• Logs — detailed, structured records of individual events.
• Traces — the path of a single request across services, showing where time is spent.

The magic happens when they are correlated: jump from a latency spike on a dashboard to the slow traces behind it, and from a trace to the exact log lines for that request.

1. Define what “healthy” means with SLOs

Without a clear definition of healthy, every graph looks alarming or none do. Service Level Objectives fix that by describing reliability from the user’s point of view:

• SLI (indicator): e.g. the share of checkout requests that succeed in under 500 ms.
• SLO (objective): e.g. 99.9% of them should, measured over 30 days.
• Error budget: the 0.1% you are allowed to miss — and a shared signal for when to slow down and invest in reliability.

2. Alert on symptoms, not causes

Alert fatigue is the enemy of fast recovery. When on-call engineers receive dozens of noisy alerts, they stop trusting them — and the important one gets missed. Page a human only when users are affected or about to be.

Burn-rate alerts on your SLOs do exactly that: they fire when you are consuming error budget fast enough to matter, and stay quiet for brief blips. Use two windows per alert, as the Google SRE workbook recommends: a long window proves the problem is significant, and a short window proves it is still happening, so the alert resolves quickly once you have fixed it. A single window is either noisy or slow.

# Recording rules: checkout error ratio over several windows
- record: checkout:error_ratio:rate5m
  expr: |
    sum(rate(http_requests_total{service="checkout",code=~"5.."}[5m]))
    / sum(rate(http_requests_total{service="checkout"}[5m]))
- record: checkout:error_ratio:rate1h
  expr: |
    sum(rate(http_requests_total{service="checkout",code=~"5.."}[1h]))
    / sum(rate(http_requests_total{service="checkout"}[1h]))

# Page when the 99.9% SLO's budget burns 14.4x too fast
# (2% of a 30-day budget in 1 hour) in both windows
- alert: CheckoutErrorBudgetFastBurn
  expr: |
    checkout:error_ratio:rate1h > (14.4 * 0.001)
    and
    checkout:error_ratio:rate5m > (14.4 * 0.001)
  labels:
    severity: page
  annotations:
    summary: "Checkout is burning its error budget fast"
    runbook: "https://runbooks.example.com/checkout/high-error-rate"

Add a second, slower pair (for example 6 hours and 30 minutes at 6× burn) that opens a ticket instead of paging, to catch gradual degradation.

Everything else — disk filling slowly, a single pod restarting — becomes a ticket or a dashboard, not a page.

3. Instrument once with OpenTelemetry

OpenTelemetry has become the standard way to instrument applications for metrics, logs and traces. Instrument once and send data to whichever backend you choose — Prometheus and Grafana, Datadog, New Relic, or your cloud’s native tools — without rewriting code when you switch.

• Start with auto-instrumentation for HTTP, gRPC and database clients.
• Propagate trace context across every service boundary, including queues and async jobs.
• Add business attributes (customer tier, region, feature flag) so you can slice problems the way users experience them.
• Use structured JSON logs that include the trace ID.

4. Make deploys and changes visible

A large share of incidents follow a change: a deploy, a configuration update, a feature flag flip. Annotate your dashboards with every deployment and flag change. The first question during any incident should be answerable in seconds: what changed?

5. Dashboards that tell a story

Fifty graphs on one screen help nobody at 3am. For each service, build a top-level dashboard around the golden signals — latency, traffic, errors and saturation — with drill-downs into dependencies. Someone unfamiliar with the service should be able to tell within a minute whether it is healthy.

6. Practice the response, not just the tooling

Tools shorten diagnosis; routines shorten everything else.

• Runbooks linked from every alert, with first steps and rollback instructions.
• Clear roles during incidents: an incident lead, a communicator, and people fixing.
• Mitigate first, investigate later. Roll back or fail over before hunting for the root cause.
• Blameless post-incident reviews that produce concrete follow-ups — better alerts, missing telemetry, automated rollback.
• Game days that rehearse failures in a controlled way so the first real one isn’t a surprise.

7. Keep observability costs in check

Telemetry volume grows faster than traffic. Control it deliberately: sample traces intelligently (keep all errors and slow requests, sample the rest), drop high-cardinality labels you never query, and tier log retention. Good observability is about the right data, not all the data.

The payoff

If you’d like help setting this up, see our observability and monitoring service.

Fast recovery is a capability you build before you need it. Clear SLOs, a handful of trustworthy alerts, correlated telemetry and a practiced incident routine turn outages from hours of guesswork into minutes of focused work. Engineers also get their time back — New Relic’s same survey found teams spend around 30% of their time dealing with interruptions — which means more time building the product.