Logging & Observability | Domain 4

🔭 What is Observability?

Observability is the ability to answer questions about your system's internal state using only the data it outputs (without needing to change or probe the system directly). A system is "observable" if, when something goes wrong, you can figure out why — just from the data it emits.

📌 Real World Analogy

A doctor cannot see inside your body directly. But they can observe your temperature, blood pressure, heart rate, and test results — and from these signals, determine what's wrong. Your app is similar: you can't watch every line of code run, but if it emits good metrics, logs, and traces, you can diagnose problems quickly.

🏛️ The Three Pillars of Observability

📊 Pillar 1 — Metrics

What: Numbers aggregated over time.

Examples: Requests/second, error rate %, CPU %, memory MB, latency ms.

Good for: Trending, alerting, capacity planning. Tells you THAT something is wrong.

AWS tool: CloudWatch Metrics

📝 Pillar 2 — Logs

What: Text records of discrete events with timestamps.

Examples: "User login", "Payment failed", "Request processed in 340ms".

Good for: Debugging specific incidents. Tells you WHAT happened and the context.

AWS tool: CloudWatch Logs

🔗 Pillar 3 — Traces

What: The path of one request through all services.

Examples: Request went to API Gateway → Lambda → DynamoDB (3.7s bottleneck).

Good for: Distributed systems, finding WHERE a bottleneck is.

AWS tool: AWS X-Ray

🎯 Exam Tip — Logs + Metrics + Traces Together

The exam repeatedly emphasizes that you need ALL THREE for proper observability. Metrics alone can't tell you why. Logs alone can't tell you where. Traces alone don't tell you about your whole system's health. Together they give complete visibility.

📐 Structured Logging — Make Logs Machine-Readable

Unstructured logging is when your logs are just random text strings. Structured logging means writing logs in a consistent, machine-parseable format — usually JSON. This makes logs searchable, filterable, and processable by tools like CloudWatch Logs Insights.

❌ Unstructured Log (BAD)

User login successful for [email protected]
ERROR payment failed order 12345 user 789
Request processed in 340ms status 200

Problems: Hard to parse, inconsistent format, can't easily extract values, hard to search across logs.

✅ Structured Log (GOOD)

{"level":"INFO","event":"login","userId":"[email protected]","timestamp":"2024-01-15T14:32:07Z"}
{"level":"ERROR","event":"payment_failed","orderId":"12345","userId":"789","timestamp":"..."}
{"level":"INFO","event":"request","duration":340,"statusCode":200,"path":"/api/orders"}

Benefits of Structured Logging

Queryable: With CloudWatch Logs Insights, you can query filter userId = "789" and find all log events for that user instantly
Consistent: All team members and all services log the same fields in the same format
Machine-readable: Automated systems can parse and act on log data
Metric extraction: You can use Metric Filters or EMF to automatically extract numbers from structured JSON logs

📋 Standard Log Fields to Always Include

Field	Purpose	Example
`timestamp`	When it happened (ISO 8601)	`"2024-01-15T14:32:07Z"`
`level`	Severity	`"INFO"`, `"WARN"`, `"ERROR"`
`service`	Which service logged it	`"OrderService"`
`correlationId`	Unique ID to trace across services	`"req-abc123"`
`requestId`	Lambda/API GW request ID	`"a1b2c3d4-..."`
`message`	Human-readable description	`"Order processed successfully"`
`userId`	User context	`"user-789"`
`duration`	How long the operation took (ms)	`340`

🏢 Centralized Logging — One Place for All Logs

When you have 10+ microservices, each writing logs separately, you have a problem: to debug a user's issue, you'd need to search logs in 10 different places. Centralized logging means aggregating all logs from all services into one searchable place.

Centralized Logging Architecture

λLambdaService A logs

🖥️EC2Service B logs

🐳ECSService C logs

🏢On-PremLegacy system

↓ All logs flow to one place ↓

☁️ CloudWatch Logs Centralized log store

↓ Use for analysis ↓

🔍Logs InsightsQuery logs

📊DashboardsVisualize

📦S3 ExportLong-term archive

🔎OpenSearchSearch & visualize

🔗 Correlation IDs — The Key to Cross-Service Debugging

When a request flows through 5 services and fails at step 4, how do you find all the related log entries across all 5 services? By passing a Correlation ID (also called Request ID or Trace ID) through every service call, and including it in every log entry.

📌 Example

User places an order. API Gateway generates: correlationId = "REQ-abc123"
This ID is passed to Lambda → Lambda passes it to DynamoDB calls → Lambda passes it to SES → etc.
Every log entry from every service includes "correlationId": "REQ-abc123"
Now search CloudWatch Logs Insights: filter correlationId = "REQ-abc123"
You instantly see the complete story of that one order across all 5 services.

⚡ AWS Lambda Powertools — Simplified Observability for Lambda

Lambda Powertools is an open-source library (available for Python, Node.js, Java, .NET) that provides structured logging, distributed tracing (X-Ray), and metrics (EMF) with minimal code. It implements best practices out of the box.

🔧 Three Core Utilities

📝 Logger

Structured JSON logging with automatic injection of: Lambda context (function name, request ID, cold start indicator), log level filtering, correlation IDs.

📊 Metrics

EMF-based metrics with a simple API. Creates CloudWatch metrics from log writes. No separate API calls. Auto-flushes at end of each Lambda invocation.

🔍 Tracer

X-Ray tracing with automatic annotation, subsegment creation for AWS SDK calls, and response/error capture. Works with Active Tracing enabled.

        Python — Lambda Powertools Complete Example
from aws_lambda_powertools import Logger, Metrics, Tracer
from aws_lambda_powertools.metrics import MetricUnit

# Initialize once at module level
logger  = Logger(service="OrderService")
metrics = Metrics(namespace="MyApp", service="OrderService")
tracer  = Tracer(service="OrderService")

@logger.inject_lambda_context(correlation_id_path="headers.X-Correlation-Id")
@tracer.capture_lambda_handler()   # auto X-Ray tracing
@metrics.log_metrics()              # auto EMF flush
def lambda_handler(event, context):
    # Structured logging — automatically includes requestId, service, etc.
    logger.info("Processing order", order_id=event["orderId"])

    # Add a custom metric (EMF — no API call)
    metrics.add_metric(name="OrdersProcessed", unit=MetricUnit.Count, value=1)

    # X-Ray annotation
    tracer.put_annotation(key="orderId", value=event["orderId"])

    try:
        result = process_order(event)
        logger.info("Order completed", duration_ms=result["duration"])
        return {"statusCode": 200}
    except Exception as e:
        logger.error("Order failed", exc_info=True)  # auto stack trace
        metrics.add_metric(name="OrdersFailed", unit=MetricUnit.Count, value=1)
        raise
      

🎯 Exam Tip

Lambda Powertools is the answer for: "simplify observability in Lambda", "structured logging with minimal code", "combine logging/metrics/tracing in Lambda". The exam may test this as an alternative to manually writing logs + calling PutMetricData + X-Ray SDK.

⚠️ User Behavior Logging & Anomaly Tracking

Beyond technical metrics, good observability includes logging user behavior and detecting anomalies (unusual patterns that might indicate a bug, attack, or business problem).

👤 User Behavior Logging

Log what users do in your application to understand usage patterns and identify issues:

Which features are used most? (product analytics)
Where do users drop off in the checkout flow? (funnel analysis)
Which users are generating the most errors? (identify problematic clients)
What did a specific user do right before they reported a bug? (reproduce issues)

🔍 Anomaly Detection

CloudWatch has Anomaly Detection — a machine learning feature that learns the normal patterns of a metric and automatically alerts when values deviate from the expected range.

Learns seasonal patterns (traffic spikes on Mondays, low traffic at 3am)
Automatically adjusts expected range based on historical data
You don't need to set hard thresholds — it adapts automatically
Use case: "Alert me when traffic drops unexpectedly" (even if you don't know what the threshold should be)

📊 System State Logging

Log system state at important transitions — not just errors:

Application startup/shutdown
Configuration changes
Database connection pool state
Cache hit/miss rates
Queue depths

This data helps you understand why a spike occurred — was it a deployment? A config change? An external event?