AWS X-Ray | Domain 4 — AWS Dev Associate

❓ The Problem — Why Do We Need X-Ray?

In the old days, you had one server running one application. When it was slow, you checked that server. Simple.

Modern applications (microservices, serverless) are distributed — a single user request might touch 5-10 different services:

One User API Request — Touches 5 Services

📱User BrowserMakes API call

→

🚦API GatewayRoute & auth

→

λLambdaBusiness logic

→

🗄️DynamoDBRead data

→

📧SESSend email

→

📱User gets response

The user says: "The API is slow, it takes 4 seconds!"

Without X-Ray, you'd check each service manually. Is it API Gateway? Lambda? DynamoDB? The email service? With 5+ services, this is extremely time-consuming.

With X-Ray: You see that the DynamoDB query is taking 3.7 seconds. Found it. Fix it.

ℹ️ Distributed Tracing

Distributed tracing means tracking a single request as it flows through multiple services. Each service adds its timing information. At the end, you have a complete timeline showing exactly how long each step took. X-Ray is AWS's distributed tracing service.

🔤 Core X-Ray Concepts — The Building Blocks

Trace

A trace is the complete story of one request — from start to finish, across ALL services. Think of it as a timeline showing every step. A trace contains one or more segments. Each trace has a unique Trace ID (like a tracking number for the request). Example: When a user places an order, the entire journey from API Gateway through Lambda to DynamoDB is ONE trace.

Segment

A segment is the contribution of ONE service to the trace. Each service that X-Ray instruments creates a segment. A segment records: what the service did, how long it took, whether it succeeded or failed. Example: The Lambda function creates a segment. API Gateway creates a segment. DynamoDB (called by Lambda) creates a subsegment inside Lambda's segment.

Subsegment

A breakdown of work WITHIN a segment. When your Lambda calls DynamoDB, that DynamoDB call becomes a subsegment within the Lambda segment. Other subsegments: HTTP calls to external APIs, SQL queries, custom code blocks you want to time. Subsegments give you granular timing within one service — like which specific database query is slow.

Inferred Segment

When Service A (which has X-Ray) calls Service B (which does NOT have X-Ray installed), X-Ray creates an "inferred segment" for Service B based on the call data from Service A. You can still see Service B in the service map — it just won't have the same level of detail as a real segment. Example: Your Lambda (with X-Ray) calls an external REST API that doesn't have X-Ray → X-Ray infers timing from Lambda's side.

Anatomy of a Single X-Ray Trace

🔍 Trace: ORD-1234 (Total: 4.2 seconds)

📦 Segment: API Gateway 0ms → 120ms (120ms)

📦 Segment: Lambda Function 120ms → 4100ms (3980ms)

↳ Subsegment: DynamoDB.GetItem 3700ms ← BOTTLENECK!

↳ Subsegment: SES.SendEmail 180ms

From this trace, you can immediately see: DynamoDB.GetItem took 3700ms out of the 4200ms total. That's the bottleneck. You'd then look at the DynamoDB query — maybe it's doing a full table scan instead of using an index.

🏷️ Annotations vs Metadata — Critical Difference

You can attach custom data to X-Ray segments. There are two types: Annotations (filterable) and Metadata (not filterable). This distinction is heavily tested.

🏷️ Annotations — Indexed, Filterable

Simple key-value pairs (string, number, boolean)
CloudWatch X-Ray indexes them — you can use them in filter expressions to search traces
Limited: max 50 annotations per trace
Use for: data you need to search/filter on — userID, orderId, environment, tenantId, feature flag
Example: "userId": "user-123"

📋 Metadata — Non-Indexed, Not Filterable

Can store any JSON-serializable data (objects, arrays, complex structures)
NOT indexed — you cannot filter traces based on metadata values
No limit on size (reasonable amounts)
Use for: additional context you want to see when you look at a trace — request body, response data, debug info
Example: Full DynamoDB response object

🎯 Key Exam Rule — Annotations for Filtering

The exam repeatedly tests this: If you want to FILTER traces (find traces where userId = "user-123"), you MUST use Annotations.
Metadata cannot be filtered. If someone says "I want to search for all traces from tenant X" → add an Annotation, not Metadata.

        Python — X-Ray SDK
from aws_xray_sdk.core import xray_recorder

def process_order(order_id, user_id):
    # ANNOTATIONS — can filter traces by these values!
    xray_recorder.put_annotation("orderId",  order_id)  # indexed!
    xray_recorder.put_annotation("userId",   user_id)
    xray_recorder.put_annotation("environment", "production")

    # METADATA — stored in trace but NOT filterable
    xray_recorder.put_metadata("order_details", {
        "items": ["widget-1", "widget-2"],
        "shipping_address": "123 Main St...",
        "payment_method": "credit_card"
    })
      

🗺️ X-Ray Service Maps — Visual Dependency Analysis

A Service Map is an automatically generated diagram showing all the services in your application and how they connect. Each node (box) shows the service, and the edges (arrows) show calls between them, with health and latency information.

📊 What a Service Map Shows

Example Service Map — E-commerce Order Processing

🌐 API Gateway OK ✓ | 120ms avg

↓

λ Order Lambda OK ✓ | 3.9s avg ⚠️

↓ ↓ ↓

🗄️ DynamoDB 3.7s avg 🔴 SLOW

📧 SES OK ✓ | 180ms avg

💳 Payment API Inferred | 800ms

From this Service Map, you can instantly see:

DynamoDB is the slow node (3.7 seconds average) — highlighted in red/orange
Payment API is an "inferred" service (no X-Ray SDK) — shown in different color
API Gateway and SES are healthy — shown in green

🎯 Service Map Benefits

Automatic discovery: You don't draw the diagram manually — X-Ray discovers service dependencies from actual traffic
Health indicators: Color-coded to show which services are having issues
Latency visualization: Shows average and percentile response times per service
Drill down: Click on a service node to see its traces, then click on a trace to see the full timeline

🎯 Exam Tip

Service Maps answer: "how do I visualize the dependencies between my microservices?" and "how do I find which downstream service is causing the bottleneck?" — answer is X-Ray Service Map.

🎲 X-Ray Sampling — Cost vs Detail Trade-off

Tracing every single request in a high-traffic application would be expensive (costs money to store traces) and could add latency. Sampling means only recording some requests as traces, not all of them.

📊 Default Sampling Rule

X-Ray's default rule: record the first request per second, then 5% of additional requests.

📌 Example

Your API handles 1000 requests per second.
Default sampling: Record 1 (first) + 5% of remaining 999 = ~51 traces per second.
Instead of storing 1000 traces/sec, you store ~51/sec. Much cheaper, still statistically significant.

⚙️ Custom Sampling Rules

You can create custom sampling rules to sample different rates for different types of requests:

Rule Name	Condition	Rate	Why
HealthCheck	URL = /health	0%	Health check noise is useless
PaymentAPI	URL contains /payment	100%	Always trace critical transactions
ErrorTracing	HTTP status = 5XX	100%	Always trace errors for debugging
Default	Everything else	5%	Statistical sample for normal traffic

🎯 Exam Tip

Sampling rules are the answer when: "you want to trace ALL requests for a specific high-value endpoint but reduce tracing for normal traffic" or "reduce X-Ray costs while keeping important traces".

🔎 Filter Expressions — Finding Specific Traces

With thousands of traces, you need to search for specific ones. Filter Expressions are queries you use to find traces that match specific criteria.

        X-Ray Filter Expressions
# Find traces that took more than 2 seconds
responsetime > 2

# Find all failed traces (HTTP 500)
http.status = 500

# Find traces for a specific URL path
http.url CONTAINS "/api/orders"

# Find traces with an annotation value (MUST be annotation, not metadata!)
annotation.userId = "user-123"
annotation.environment = "production"

# Combine conditions
http.url CONTAINS "/api/payment" AND responsetime > 1

# Find traces that have an error or fault
error = true

# Find traces for a specific service
service("OrderLambda") { fault = true }

# Find traces by annotation AND response time
annotation.tenantId = "tenant-456" AND responsetime > 5
      

🎯 Exam Tip

Remember: You can only filter on Annotations (not Metadata). If a question asks "filter X-Ray traces by user ID or customer ID" → first you need to add those as Annotations in your code, then use a Filter Expression.

⚙️ X-Ray Daemon — The Data Collector

The X-Ray SDK in your application doesn't send trace data directly to the X-Ray service. Instead, it sends trace data (UDP on port 2000) to a local X-Ray Daemon process, which batches the data and sends it to the X-Ray service.

How X-Ray Data Flows

💻Your App CodeX-Ray SDK calls

→ UDP:2000

⚙️X-Ray DaemonBuffers & batches

→ HTTPS

🔍X-Ray ServiceStores traces

📍 Where Does the Daemon Run?

Platform	Daemon Setup
AWS Lambda	X-Ray daemon runs automatically when you enable Active Tracing. You don't install or manage it.
Elastic Beanstalk	Enable X-Ray in the EB console → daemon is installed automatically. No manual setup needed.
EC2 Instance	You install the daemon manually as a background process. Available as a package for Linux/Windows.
ECS Container	Run as a sidecar container alongside your app container.
On-Premises	Install daemon manually on the server. Needs IAM user credentials (not IAM role, since not on AWS).

🔑 IAM Permissions for X-Ray Daemon

The daemon needs permission to write to X-Ray. The IAM policy needs:

        IAM Policy
{
  "Effect": "Allow",
  "Action": [
    "xray:PutTraceSegments",     // Send trace segments
    "xray:PutTelemetryRecords",   // Send telemetry
    "xray:GetSamplingRules",      // Get sampling rules
    "xray:GetSamplingTargets"
  ],
  "Resource": "*"
}
      

λ X-Ray with AWS Lambda — Active Tracing

Lambda has built-in integration with X-Ray through a feature called Active Tracing. When enabled, Lambda automatically creates segments and runs the X-Ray daemon — you just need to add the X-Ray SDK to your code.

⚡ Enabling Active Tracing

Three ways to enable:

AWS Console: Lambda function → Configuration → Monitoring → Enable Active Tracing
AWS CLI: aws lambda update-function-configuration --function-name my-fn --tracing-config Mode=Active
CloudFormation/SAM: Tracing: Active in the function resource

        SAM Template
MyFunction:
  Type: AWS::Serverless::Function
  Properties:
    Tracing: Active   # Enable X-Ray Active Tracing
    Policies:
      - AWSXRayDaemonWriteAccess   # Required IAM policy
      

📊 What Lambda Automatically Creates in X-Ray

A segment for each Lambda invocation with: cold start duration (if applicable), initialization time, invocation time
Subsegments for each AWS SDK call inside Lambda (DynamoDB, S3, SQS, etc.) — automatically created if you use X-Ray SDK
Metadata about the function: function name, version, memory, request ID

ℹ️ Passive vs Active Tracing

Active Tracing: Lambda always creates traces for all invocations (subject to sampling rules).
Passive Tracing: Lambda only traces if the upstream service already started a trace and passed the trace ID in the request (called "trace propagation"). If there's no incoming trace, Lambda doesn't create one.

🔄 Trace Propagation — How Traces Flow Between Services

When one service calls another, it passes the trace ID in a special HTTP header: X-Amzn-Trace-Id. The receiving service picks up this ID and continues the same trace. This is how one trace spans multiple services.

        HTTP Header
# This header is automatically added by X-Ray SDK
X-Amzn-Trace-Id: Root=1-5e272390-8a7cf1af82c8d8f012345678;Parent=4abc123def456;Sampled=1
                    ↑ Trace ID               ↑ Parent Segment ID  ↑ 1=sampled, 0=not
      

🌱 X-Ray with Elastic Beanstalk

Elastic Beanstalk has native X-Ray support. When you enable X-Ray in the Elastic Beanstalk environment, EB automatically installs and configures the X-Ray daemon on all instances.

How to Enable

AWS Console: Elastic Beanstalk → Environment → Configuration → Software → Enable X-Ray
Configuration file: Add .ebextensions/xray-daemon.config to your app package

        YAML — .ebextensions/xray-daemon.config
option_settings:
  aws:elasticbeanstalk:xray:
    XRayEnabled: true
      

🎯 Exam Tip

With Elastic Beanstalk + X-Ray: You enable it in the console (or config file) and EB handles everything. You DON'T need to manually install the daemon. This is a key difference from EC2 (where you must install it manually).

👥 X-Ray Groups — Organize Your Traces

X-Ray Groups let you create named collections of traces that match a filter expression. Instead of searching every time, you define a group once and all matching traces are automatically organized in it.

📌 Example

Create a group called "SlowRequests" with filter: responsetime > 2.
Now you have a dedicated view of all slow requests, with separate service maps and metrics for just those traces.
Create another group "PaymentErrors" with filter: annotation.service = "PaymentService" AND error = true.

📊 X-Ray Insights — Anomaly Detection

X-Ray Insights (not to be confused with Logs Insights) automatically detects when fault rates in your traces spike. It groups related anomalous events into "insights" so you don't have to manually notice the pattern.

Detects sudden increases in error rates, latency spikes
Groups related traces that show the same anomaly
Can send notifications via SNS

📋 X-Ray Exam Quick Reference

Scenario / Question	X-Ray Feature to Use
Find which service in the chain is slow	Service Map — visual latency per node
See the detailed timing of one specific request	Trace Timeline — drill into one trace
Find all traces for a specific user ID	Annotations + Filter Expression
Attach complex JSON data to a trace for debugging	Metadata — not filterable but stores rich data
Only trace 10% of requests to save cost	Sampling Rules
Always trace payment requests but sample others	Custom Sampling Rules
Integrate X-Ray with Lambda (easiest way)	Active Tracing (enable in console)
Integrate X-Ray with Elastic Beanstalk	Enable in EB console — daemon auto-installed
Service B doesn't have X-Ray SDK but is in the map	Inferred Segment — auto-created by X-Ray
How does X-Ray SDK send data to X-Ray service?	Via X-Ray Daemon (UDP port 2000)
Find all errors/faults in traces	Filter Expression: `error = true` or `fault = true`

🔍 AWS X-Ray — Distributed Tracing

❓ The Problem — Why Do We Need X-Ray?

🔤 Core X-Ray Concepts — The Building Blocks

🏷️ Annotations vs Metadata — Critical Difference

🏷️ Annotations — Indexed, Filterable

📋 Metadata — Non-Indexed, Not Filterable

🗺️ X-Ray Service Maps — Visual Dependency Analysis

📊 What a Service Map Shows

🎯 Service Map Benefits

🎲 X-Ray Sampling — Cost vs Detail Trade-off

📊 Default Sampling Rule

⚙️ Custom Sampling Rules

🔎 Filter Expressions — Finding Specific Traces

⚙️ X-Ray Daemon — The Data Collector

📍 Where Does the Daemon Run?

🔑 IAM Permissions for X-Ray Daemon

λ X-Ray with AWS Lambda — Active Tracing

⚡ Enabling Active Tracing

📊 What Lambda Automatically Creates in X-Ray

🔄 Trace Propagation — How Traces Flow Between Services

🌱 X-Ray with Elastic Beanstalk

How to Enable

👥 X-Ray Groups — Organize Your Traces

📊 X-Ray Insights — Anomaly Detection

📋 X-Ray Exam Quick Reference