Log Correlation: Tracing Errors Across Services

Learning Objectives

By the end of this lesson you will be able to:

• Explain what a correlation ID is and why it is essential in distributed systems
• Use grep to find all log entries belonging to a single request
• Follow a date parse failure across API, service, and database log layers
• Reconstruct an event timeline from log timestamps

What Is a Correlation ID?

A correlation ID (also called a request ID, trace ID, or X-Request-ID) is a unique identifier assigned to each incoming HTTP request. It is:

• Generated at the entry point of the system (API gateway or first service)
• Attached to every log statement made while processing that request
• Forwarded to downstream services via HTTP headers so they can log it too
• Returned to the client in the response header so it can be reported in bug reports

Without correlation IDs, if ten users hit an error at the same time, their log entries are interleaved and indistinguishable. With correlation IDs, you can isolate exactly one user's journey through the system.

How to Find a Specific Request in Logs

The most basic and reliable tool is grep:

grep "req-abc123" app.log

This outputs every line in app.log that contains the string req-abc123 — the full request lifecycle in order.

In structured log systems (JSON logs, Datadog, Splunk, CloudWatch), you use a query:

request_id = "req-abc123"

or

fields @timestamp, @message | filter request_id = "req-abc123" | sort @timestamp asc

The result is a complete, time-ordered view of what happened to that specific request.

Following a Date Parse Failure Across Layers

Here is an example of how a single bug produces entries across three log files:

api.log (the HTTP layer):

2024-11-14T14:32:10.001Z INFO  [req-abc123] POST /api/bookings received — body: {"start_date":"31-13-2024"}
2024-11-14T14:32:10.047Z ERROR [req-abc123] Request processing failed — returning 500

service.log (the business logic layer):

2024-11-14T14:32:10.012Z DEBUG [req-abc123] BookingService.create() called
2024-11-14T14:32:10.039Z ERROR [req-abc123] DateTimeParseException: Text '31-13-2024' could not be parsed — expected format yyyy-MM-dd

db.log (the database layer):

(no entries for req-abc123 — the exception was thrown before any DB query was made)

Reading these three snippets in chronological order tells the complete story: the API received a malformed date string, passed it to the service layer, which tried to parse it and failed with a DateTimeParseException, and the API layer caught that exception (or didn't) and returned a 500.

Using Timestamps to Reconstruct a Timeline

When you don't have a correlation ID, timestamps are your next best tool. The approach:

1. Identify the failure moment — from a user report or alerting system ("error at 14:32 UTC")
2. Collect all ERROR/WARN entries in a ±2 minute window across all log files
3. Sort by timestamp — sort combined with grep or your log UI's time filter
4. Look for a cluster of errors — a bug that affects one request often produces 2–5 log entries within milliseconds of each other

Example grep that collects errors across two log files, sorted by time:

grep -h "ERROR\|WARN" api.log service.log | sort

The -h flag suppresses the filename prefix so entries from both files sort together by their ISO 8601 timestamps.

Summary

Correlation IDs and timestamps are the two primary tools for navigating logs when investigating a bug. A correlation ID gives you surgical precision: every log entry from one request, nothing else. Timestamps give you contextual reconstruction: the sequence of events leading to a failure, even across services you did not know were involved.