Observability Explained: Logs, Metrics and Traces

Monitoring tells you when something is wrong. Observability helps you understand why, including for problems you didn't anticipate. As systems become distributed (services, queues, third-party APIs, AI features), the question "why is checkout slow for some users?" can't be answered from CPU graphs alone. You need well-designed logs, metrics and traces that work together.

#Key takeaways

• Metrics: cheap, aggregated numbers over time. Best for dashboards and alerting.
• Logs: detailed, discrete events. Best for debugging specifics. Make them structured.
• Traces: the path of a request across services, with timing. Best for finding where latency and errors originate.
• Correlate all three with shared IDs (trace IDs in logs) and consistent attributes.
• Define SLOs around user experience and alert on symptoms, not every internal cause.

#The three pillars

#Metrics

Numeric measurements aggregated over time: request rate, error rate, latency percentiles, queue depth, CPU.

• ✅ Cheap to store and query, and ideal for alerts and trends.
• ⚠️ Low detail; high-cardinality labels (user IDs, request IDs) can explode storage costs.

Two proven frameworks:

• RED (for services): Rate, Errors, Duration.
• USE (for resources): Utilization, Saturation, Errors.

Record latency as histograms so you can compute percentiles (p50/p95/p99). Averages hide the slow requests users complain about (latency numbers every engineer should know).

#Logs

Timestamped records of discrete events. Make them structured (JSON) so they're searchable and aggregatable:

json{"ts":"2026-09-28T10:14:03.221Z","level":"error","service":"checkout",
 "trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","user_id":"u_123","order_id":"ord_987",
 "event":"payment_failed","provider":"stripe","error_code":"card_declined","duration_ms":842}

Logging best practices:

• Consistent fields across services (service, environment, version, trace_id).
• Log events, not essays: machine-readable fields plus a short message.
• Right-size log levels: errors that need attention, warnings for anomalies, info for key business events, debug sampled or off in production.
• Never log secrets or sensitive personal data; redact at the source.
• Control volume and cost: sample noisy logs, and set retention policies.

#Traces

A trace follows a single request through your system as a tree of spans, each representing an operation with timing and attributes:

textTrace 4bf92f… (GET /checkout, 1,240 ms)
├─ api-gateway                         12 ms
└─ checkout-service                 1,210 ms
   ├─ SELECT cart_items (postgres)      8 ms
   ├─ pricing-service.calculate        35 ms
   ├─ inventory-service.reserve        60 ms
   └─ payment-provider.charge       1,080 ms   ← here's your latency

Traces answer questions like "which dependency makes this endpoint slow?" in seconds. They're indispensable for microservices, async workflows (event-driven architecture) and AI agents with many tool calls (making AI agents reliable).

#OpenTelemetry: the standard

OpenTelemetry (OTel) is the open, vendor-neutral standard for generating and exporting telemetry: APIs and SDKs for many languages, auto-instrumentation for common frameworks, and a Collector for processing and routing data to any backend.

pythonfrom opentelemetry import trace
tracer = trace.get_tracer("checkout")

def charge(order):
    with tracer.start_as_current_span("payment.charge") as span:
        span.set_attribute("order.id", order.id)
        span.set_attribute("payment.provider", "stripe")
        result = provider.charge(order.total)
        span.set_attribute("payment.status", result.status)
        return result

Instrumenting with OTel avoids lock-in: you can switch or combine backends without re-instrumenting code.

#Correlation: making the pillars work together

The real power comes from moving between signals:

1. An alert fires on a metric (checkout p99 latency is up).
2. You open exemplar traces for slow requests and see the payment span is slow.
3. You jump to logs filtered by those trace IDs and see timeouts from one provider region.

Enable this by:

• Injecting trace IDs into logs automatically.
• Using consistent resource attributes (service name, version, environment, region).
• Propagating context across service calls and message queues (trace context headers).

#SLOs and error budgets

Define reliability from the user's perspective:

• SLI (indicator): what you measure, such as "the percentage of checkout requests that succeed in under 1 s".
• SLO (objective): the target, such as "99.5% over 28 days".
• Error budget: the allowed unreliability (0.5% of requests). When it's healthy, ship faster; when it's burning, prioritize reliability.

SLOs turn reliability debates into data-driven decisions and align product and engineering (technical debt management).

#Alerting that doesn't burn people out

• Alert on symptoms users feel (error rates, latency SLO burn, failed jobs), not every internal cause (CPU at 80%).
• Use multi-window burn-rate alerts on SLOs to catch both fast and slow burns.
• Every alert must be actionable, with an owner and a runbook link.
• Route by severity: page for urgent user impact, create tickets for slower issues.
• Review alerts regularly and delete noisy ones. Alert fatigue causes missed incidents (incident management and postmortems).

#Dashboards

Build a small set that answers key questions quickly:

• Service overview: RED metrics, SLO status, and recent deploys (deploy markers help correlate regressions with releases: zero-downtime deployments).
• Dependencies: database, cache, queues and third-party APIs.
• Business health: orders per minute, signups, payments. These often catch problems technical metrics miss.

#Observability for data and AI systems

• Data pipelines: freshness, volume and quality metrics per dataset (data quality guide).
• ML models: prediction drift and performance (MLOps explained).
• LLM applications: token usage, cost, latency per step, tool-call traces, and quality signals like user feedback and eval scores (LLM cost optimization).

#Cost control

Telemetry can get expensive:

• Sample traces (for example, keep all errors and slow requests, and a fraction of normal ones via tail sampling).
• Limit metric cardinality: avoid unbounded label values.
• Tier log retention: hot for days, cheap archive for longer.
• Drop noisy, low-value data at the collector.

#FAQ

#What's the difference between monitoring and observability?

Monitoring watches known failure modes with predefined dashboards and alerts. Observability is the ability to explore telemetry and ask new questions about unexpected behavior. Good monitoring is part of an observable system.

#Do small applications need tracing?

Even a monolith benefits from tracing database queries and external calls. It quickly shows where time goes. For microservices and async systems, tracing becomes essential.

#Which should I implement first: logs, metrics or traces?

Start with structured logs and core RED metrics with alerting on user-facing symptoms. Add distributed tracing (OpenTelemetry auto-instrumentation makes this easier) as soon as you have multiple services or performance questions.

#What is an error budget?

The acceptable amount of unreliability implied by an SLO. If your SLO is 99.9% success, your error budget is 0.1% failures. Teams use it to balance feature velocity against reliability work.