Overview
Observability is the ability to understand a system's internal state by examining its outputs. The three pillars — logs (events), metrics (aggregates over time), traces (request journeys) — answer different questions. Logs tell you what happened. Metrics tell you how much and how fast. Traces tell you where time was spent across a distributed request. You need all three to debug production issues effectively.
Architecture
THREE PILLARS OF OBSERVABILITY
════════════════════════════════════════════════════════
LOGS (what happened)
├── Discrete events with context
├── Structured JSON (not free-text)
├── Correlation ID → link logs across services
└── Tools: ELK (Elasticsearch+Logstash+Kibana), Loki+Grafana
METRICS (how much, how fast)
├── Numeric time-series: counters, gauges, histograms
├── High cardinality problem: don't use user_id as a label (millions)
├── SLIs: p50/p95/p99 latency, error rate, throughput
└── Tools: Prometheus, Grafana, Datadog, CloudWatch
TRACES (where time was spent)
├── Single request through N services
├── Trace = tree of spans, each span = one operation
├── Parent-child spans, timing, error state
└── Tools: Jaeger, Zipkin, OpenTelemetry, Datadog APM
REQUEST TRACE ANATOMY
════════════════════════════════════════════════════════
Trace ID: abc-123
│
├── Span: API Gateway [10ms] (traceId=abc-123, spanId=s1)
│
├── Span: Order Service [45ms] (spanId=s2, parentSpanId=s1)
│ ├── Span: DB query [12ms] (spanId=s3, parentSpanId=s2)
│ └── Span: Kafka publish [3ms] (spanId=s4, parentSpanId=s2)
│
└── Span: Inventory Service [30ms] (spanId=s5, parentSpanId=s1)
└── Span: Redis GET [2ms] (spanId=s6, parentSpanId=s5)
Total: 55ms (gateway + max(order, inventory) in parallel)
ALERTING STRATEGY: USE 4 GOLDEN SIGNALS
════════════════════════════════════════════════════════
Latency → p99 > threshold
Traffic → requests/sec (sudden drop = problem)
Errors → error rate > 1%
Saturation → CPU > 80%, queue depth > threshold
Alert on SLO violations, not metric thresholds:
Wrong: alert if p99 > 500ms
Right: alert if error budget for p99 < 200ms SLO is being burned fast
Technical Implementation
Structured Logging with Correlation IDs (Spring)
// MDC (Mapped Diagnostic Context): thread-local key-value pairs added to all log lines
@Component
public class CorrelationFilter extends OncePerRequestFilter {
@Override
protected void doFilterInternal(HttpServletRequest req, HttpServletResponse res,
FilterChain chain) throws IOException, ServletException {
String traceId = Optional.ofNullable(req.getHeader("X-Trace-Id"))
.orElse(UUID.randomUUID().toString().replace("-", "").substring(0, 16));
MDC.put("traceId", traceId);
MDC.put("userId", extractUserId(req));
MDC.put("path", req.getRequestURI());
res.addHeader("X-Trace-Id", traceId); // propagate back to caller
try {
chain.doFilter(req, res);
} finally {
MDC.clear(); // prevent leakage in thread pools
}
}
}
// logback.xml — structured JSON logging
// {
// "timestamp": "2026-05-01T10:00:00.000Z",
// "level": "INFO",
// "traceId": "a1b2c3d4e5f6",
// "userId": "user-123",
// "path": "/api/orders",
// "message": "Order placed successfully",
// "orderId": "ord-456",
// "duration": 42
// }
Prometheus Metrics with Micrometer
@RestController
@RequiredArgsConstructor
public class OrderController {
private final MeterRegistry meterRegistry;
private final Counter orderCounter;
private final Timer orderTimer;
@PostConstruct
void initMetrics() {
// Counter: monotonically increasing (total orders)
this.orderCounter = Counter.builder("orders.placed.total")
.tag("service", "order-service")
.description("Total orders placed")
.register(meterRegistry);
// Timer: latency distribution (records count + sum + histogram buckets)
this.orderTimer = Timer.builder("orders.processing.duration")
.tag("service", "order-service")
.publishPercentiles(0.5, 0.95, 0.99) // p50, p95, p99
.sla(Duration.ofMillis(100), Duration.ofMillis(500))
.register(meterRegistry);
}
@PostMapping("/orders")
public ResponseEntity<Order> placeOrder(@RequestBody OrderRequest req) {
return orderTimer.record(() -> {
Order order = orderService.place(req);
orderCounter.increment(1, "status", "success");
return ResponseEntity.ok(order);
});
}
}
// Prometheus scrape endpoint: GET /actuator/prometheus
// Grafana dashboard: p99 latency, error rate, throughput
OpenTelemetry Distributed Tracing
// Add OpenTelemetry auto-instrumentation agent — zero code change
// java -javaagent:opentelemetry-javaagent.jar -jar app.jar
// Agent instruments: Spring MVC, JDBC, Kafka, Redis, etc. automatically
// Manual spans for custom operations
@Service
public class OrderService {
private final Tracer tracer = GlobalOpenTelemetry.getTracer("order-service");
public Order processOrder(OrderRequest req) {
Span span = tracer.spanBuilder("order.process")
.setAttribute("order.userId", req.getUserId())
.setAttribute("order.itemCount", req.getItems().size())
.startSpan();
try (Scope scope = span.makeCurrent()) {
// All downstream calls within this scope inherit the trace context
Order order = placeOrder(req);
span.setAttribute("order.id", order.getId());
span.setStatus(StatusCode.OK);
return order;
} catch (Exception ex) {
span.recordException(ex);
span.setStatus(StatusCode.ERROR, ex.getMessage());
throw ex;
} finally {
span.end();
}
}
}
Interview Preparation
Q: What is the difference between monitoring and observability?
Monitoring is checking known failure modes — alerting when CPU > 80%, error rate > 1%, disk full. You know what to look for. Observability is the ability to ask arbitrary questions about system behavior using telemetry data, even for failure modes you didn't anticipate. A system is observable when you can debug novel production issues without deploying new instrumentation. Monitoring is reactive (known unknowns). Observability is the foundation for debugging unknown unknowns. In practice: you need both — monitoring for SLOs and basic health, observability for root cause analysis of incidents.
Q: What is distributed tracing and why does it matter for microservices?
In a microservices system, a single user request touches 5-20 services. When the request is slow or fails, traditional logs don't tell you which service was the bottleneck. Distributed tracing assigns a unique Trace ID to each request and propagates it through all service calls via HTTP headers (traceparent in W3C Trace Context). Each service creates spans (start time, duration, metadata) and reports them to a trace collector (Jaeger, Zipkin). The collector reconstructs the full call tree, showing: which service was slowest, which downstream call failed, and the exact sequence of operations. Without tracing, diagnosing "why is checkout slow?" in 10 microservices requires manually correlating log timestamps across 10 services — with tracing it's a single query.
Q: How do you avoid cardinality explosions in Prometheus metrics?
High cardinality occurs when a label has many unique values — user IDs, order IDs, UUIDs. A metric like http_requests_total{user_id="user-abc-123"} with millions of users creates millions of time series in Prometheus, exhausting memory. The rule: labels should have low cardinality (< 1000 unique values). Good labels: method=POST, status_code=200, endpoint=/api/orders. Bad labels: user_id=xxx, order_id=xxx, trace_id=xxx. For per-user analytics, use your log aggregation system (Elasticsearch/Loki) with structured logs — these are indexed differently and can handle high cardinality. Prometheus is for aggregate behavior, not individual-level tracking.
Learning Resources
- OpenTelemetry Documentation
- Prometheus Best Practices
- Google SRE Book — Monitoring Distributed Systems
- Grafana + Loki — log aggregation
- Observability Engineering — Majors, Fong-Jones, Miranda — O'Reilly