🎯 Domain 4 Practice Quiz

EMF (Embedded Metric Format) lets you write metrics as structured JSON to stdout, which CloudWatch Logs automatically extracts into metrics. There's zero extra API call latency since it piggybacks on the existing log write. PutMetricData (A) adds network latency per call. CloudWatch Agent (B) cannot be installed on serverless Lambda. Kinesis (D) is overly complex.

Latency = total end-to-end time at API Gateway. IntegrationLatency = time waiting for the backend response. If Latency is high but IntegrationLatency is low, the overhead = Latency − IntegrationLatency is large, meaning API Gateway itself is adding delay. This points to issues like request validation, Lambda authorizer calls, response mapping templates, or caching operations — all within API Gateway, not the backend.

CloudWatch Logs Insights is purpose-built for ad-hoc log analysis — fast, uses a simple query language, and can aggregate by time bins. Option C (Metric Filter) is better for ongoing continuous monitoring (it creates a metric going forward), not for querying existing historical logs. Logs Insights can query existing historical data immediately. Both B and C are valid approaches; for a one-time or ad-hoc count, Logs Insights (B) is more efficient.

INSUFFICIENT_DATA is the initial state of a new alarm — it transitions to OK or ALARM once enough data points exist to evaluate. It can also occur when a metric stops reporting data (e.g., EC2 stopped, but then it would be explicitly INSUFFICIENT_DATA due to missing data). A newly created alarm always starts in INSUFFICIENT_DATA until its evaluation period completes with actual metric values.

CloudWatch Synthetics lets you create canaries — scripts that run on a schedule to simulate user transactions (like checking a URL). They call your endpoints, measure response time, check availability, and publish metrics to CloudWatch. You can then set alarms on those metrics. This is the purpose-built solution for proactive endpoint monitoring without real user traffic.

The PutMetricData API is how you publish custom application metrics to CloudWatch. You choose a custom namespace (e.g., "MyApp/Orders"), define the metric name ("OrdersPlaced"), and provide a value. Once published, the metric appears in CloudWatch like any built-in metric — you can graph it, create alarms on it, and add it to dashboards.

CloudWatch Composite Alarms allow you to combine multiple alarms using AND/OR logic. Create: ALARM("CPU-Alarm") AND ALARM("Memory-Alarm") → the composite alarm fires only when both individual alarms are in ALARM state simultaneously. This reduces alert fatigue (no pages when only one condition is true) and is the exact use case Composite Alarms were built for.

EMF writes metric values as structured JSON to CloudWatch Logs (stdout). CloudWatch Logs asynchronously extracts these as real metrics — zero added latency to function execution since the log write happens in the background Lambda infrastructure. The function doesn't wait for PutMetricData network round-trips.

CloudWatch Shared Dashboards generate a public URL (or a URL for specific AWS accounts) that allows people to view dashboard data without needing AWS credentials. The link shows a live, read-only view of the dashboard. This is the proper, low-maintenance solution for sharing metrics with non-technical stakeholders.

CacheHitCount is the actual CloudWatch metric that counts requests served directly from the API Gateway cache (no backend call made). CacheMissCount counts requests that went through to the backend. Cache Hit Rate % = CacheHitCount / (CacheHitCount + CacheMissCount) × 100 — you calculate this using Metric Math, as CacheHitRate itself is not a native metric.

CloudWatch Metric Math allows creating expressions that combine multiple metrics mathematically. Define m1 = Errors, m2 = Invocations, expression = m1/m2*100 (this is your error rate %). Create the alarm on this expression value > 5. This is cleaner and more accurate than alarming on raw error count, which doesn't account for varying traffic volumes.

CloudWatch metric retention by resolution: 1-second (high-res) → 3 hours, 1-minute → 15 days, 5-minute → 63 days, 1-hour → 15 months (455 days). After the retention period, metrics are automatically aggregated into the next coarser resolution. This is why you can see detailed recent data but only hourly data for older periods.

An X-Ray Trace collects all the segments and subsegments generated by a single request from entry point (API Gateway) through every downstream service (Lambda, DynamoDB, SNS, etc.) until a final response is returned. It's identified by a unique Trace ID that's propagated via the X-Amzn-Trace-Id HTTP header.

Annotations are key-value pairs added to X-Ray segments that are indexed and therefore searchable via Filter Expressions. To filter by userId, add it as an annotation: putAnnotation("userId", userId). Then use a Filter Expression: annotation.userId = "user123" to find all traces for that user. Metadata (C) is NOT indexed and cannot be searched.

The key distinction: Annotations = indexed key-value pairs (searchable). Metadata = non-indexed key-value pairs (not searchable, but visible in trace detail). Use Annotations when you need to search/filter traces. Use Metadata for rich contextual data you want to see when inspecting a specific trace but don't need to search across traces (e.g., full request body, user preferences object).

The X-Ray Service Map is a visual diagram showing all services as nodes connected by edges. Each node shows average latency, request rate, and error/fault/throttle rates. It gives an instant visual overview of the entire system health and where bottlenecks are. The Trace Timeline (A) shows one specific trace, not the overall system.

When Active Tracing is enabled on a Lambda function, Lambda automatically: (1) runs the X-Ray daemon as a sidecar, (2) sends a segment for the Lambda service itself, and (3) injects the trace context so the function code can instrument subsegments. This eliminates the need to manually deploy and configure the X-Ray daemon. Without it, you would need to run the daemon yourself and set up trace propagation manually.

An Inferred Segment is automatically created by X-Ray when your code (instrumented with the X-Ray SDK) makes calls to AWS services like S3, DynamoDB, or SNS via the AWS SDK. X-Ray captures the timing of that outbound call and creates a segment representing the downstream service — even though that service never sent its own segment. This gives visibility into downstream service calls without requiring those services to be instrumented.

X-Ray Sampling Rules control what percentage of requests are traced. The default rule captures 5% of requests with a reservoir of 1 req/sec. You can create custom rules by service, URL path, or HTTP method to fine-tune sampling rates. This significantly reduces data ingestion costs while maintaining statistical visibility. You can trace 100% of error responses even while sampling normal traffic.

Elastic Beanstalk has native X-Ray integration. In the console, under "Software Configuration", enable X-Ray Daemon. Alternatively, set XRayEnabled: true in the Beanstalk configuration. Beanstalk will automatically install, configure, and run the X-Ray daemon on the underlying EC2 instances. No manual installation required — this is the managed Beanstalk way.

Hierarchy: Trace → contains Segments (one per service) → contains Subsegments (granular breakdown within a service). Example: a Lambda function's Segment might contain subsegments for: DynamoDB GetItem call (20ms), S3 PutObject call (45ms), and a custom code block (5ms). Subsegments let you pinpoint exactly which operation within a service is slow.

X-Ray Filter Expressions are queries that filter the list of traces. You can filter by URL, duration, HTTP status code, response time, annotations, service name, and more. The expression syntax: http.url CONTAINS "/api/orders" AND duration > 2 would return only traces matching that URL that exceeded 2 seconds. Use this in the X-Ray console or via the GetTraceSummaries API.

The three pillars of observability are: Metrics (numerical time-series data — what is happening), Logs (timestamped event records — what happened in detail), and Traces (request journey across services — where is it slow). AWS: CloudWatch = Metrics + Logs, X-Ray = Traces, CloudTrail = API audit log (a special form of Logs).

AWS Lambda Powertools (available for Python, TypeScript, Java, .NET) provides three utilities: Logger (structured JSON logging with correlation IDs), Metrics (EMF-based custom metrics), and Tracer (X-Ray tracing with decorators). It's the standard "batteries-included" toolkit for Lambda observability — reduces dozens of lines of instrumentation code to a few decorator annotations.

Structured logging writes logs as JSON with consistent, defined fields: timestamp, level, service, correlationId, userId, message, etc. Unlike unstructured logs ("Error processing order 123 for user abc"), structured logs let you easily filter, aggregate, and search — e.g., find all logs where userId = "abc" AND level = "ERROR" using CloudWatch Logs Insights or Athena.

Centralized logging with correlation IDs is the proper pattern. Each service includes the same correlationId (generated at API Gateway entry, propagated via headers) in its log messages. When searching for a failed request, filter all logs by that correlationId — instantly see the complete timeline across all 10 services. CloudWatch Logs Insights can also query multiple log groups: fields @timestamp | filter correlationId = "abc-123".

EMF (Embedded Metric Format) writes structured JSON to stdout. CloudWatch picks up metrics from logs asynchronously after the function returns — zero added latency. PutMetricData makes a synchronous HTTPS call to CloudWatch during function execution — this adds network latency (typically 5-50ms) to every invocation, increasing both function duration and cost. For high-volume Lambda functions, this difference is significant.

When individual service metrics look normal but the end-to-end request is slow, the bottleneck is often in the communication between services or in a downstream service not instrumented with basic metrics (like DynamoDB read latency, network hops). The X-Ray Service Map shows the complete trace timeline with latency at each hop, revealing DynamoDB throttling, slow table reads, or high serialization overhead that wouldn't show up in Lambda duration metrics alone.

Amazon Athena is serverless SQL directly on S3 — no cluster, no data loading, no ongoing infrastructure cost. You pay only for the bytes scanned. For weekly ad-hoc queries on 3TB of logs, Athena is far more cost-effective than OpenSearch (which requires persistent cluster costs) or EMR (cluster provisioning time + cost). Define a table schema over the S3 location and query with standard SQL.

A Correlation ID (also called request ID or trace ID) is generated when a request enters your system (at API Gateway or the first service), then passed as an HTTP header (X-Correlation-ID) to every downstream service, and included in every log line. When debugging, search for the correlationId across all log groups to see the complete request history across 10 services in chronological order. Without it, connecting related log entries across services is nearly impossible.

CloudTrail records all AWS API calls, including who (IAM ARN), when (timestamp), from where (source IP), and what was called (DeleteBucket on which bucket). The default 90-day trail in Event History is searchable. For 45 days ago, the event would still be within the 90-day retention window. Search by event name "DeleteBucket" and resource name to find the exact API call with the caller's identity.

By default, CloudTrail records Management Events only — control plane operations like CreateBucket, TerminateInstances, CreateRole, DeleteTable. Data Events (S3 GetObject, PutObject, Lambda InvokeFunction) are NOT enabled by default due to high volume and additional cost. You must explicitly enable them. Insights Events (anomaly detection) are also optional and cost extra.

CloudTrail Insights analyzes your normal write API call patterns (your "baseline") and generates Insights Events when it detects anomalies. Examples: "S3 DeleteObject calls spiked from 5/min to 500/min" (possible data deletion attack), "IAM CreateUser called 50 times in 1 minute" (possible account compromise). Insights events are additional and help detect security incidents automatically.

CloudTrail's Event History retains management events for 90 days at no additional charge. This is searchable in the AWS Console. For longer retention (compliance requirements often demand 1-7 years), you must create a Trail that delivers logs to an S3 bucket — S3 then provides indefinite storage with lifecycle rules for cost management.

S3 Data Events in CloudTrail specifically capture object-level operations (GetObject, PutObject, DeleteObject, etc.) that are NOT captured by management events. You configure this by creating a Trail and enabling Data Events with an Event Selector pointing to the specific S3 bucket ARN. Note: this increases CloudTrail cost significantly for high-traffic buckets due to high event volume.

Provisioned Concurrency pre-warms N execution environments — Lambda runs the init code (module imports, DB connections) for each environment in advance. When a request arrives, it's handled by an already-warm environment with no cold start delay. Billing includes the provisioned capacity even when idle. Use it for latency-sensitive applications with predictable traffic patterns.

Reserved Concurrency does two things simultaneously: (1) Caps the function at N concurrent executions (throttles beyond that). (2) Guarantees those N slots are always available for this function (the account's other functions can't use those slots). Example: reserve 100 concurrency for Function A → A can have at most 100 concurrent executions, and always has 100 slots available.

By setting Reserved Concurrency on the report Lambda (e.g., cap at 200), you prevent it from consuming the entire 1000-execution account pool. This leaves enough concurrency available for the payment function from the shared unreserved pool. You could also set Reserved Concurrency on the payment function to guarantee it always has a dedicated slice of concurrency, but capping the greedy function (C) is the cleaner fix.

AWS Lambda Power Tuning is a Step Functions state machine that automatically invokes your function at multiple memory settings (e.g., 128MB, 256MB, 512MB, 1024MB, 2048MB), measures real execution time and cost for each, then outputs a visualization. You choose the "strategy" (cheapest, fastest, or balanced). This is the standard community tool for Lambda right-sizing (installed via SAR — Serverless Application Repository).

For latency-critical APIs where cold starts violate SLAs, Provisioned Concurrency is the correct solution — it guarantees pre-warmed execution environments with zero cold start time. More memory (A) reduces cold start duration but doesn't eliminate it. Reserved Concurrency (B) doesn't prevent cold starts. SnapStart (D) is only for Java runtimes and works differently.

Lambda cost = memory (GB) × duration (seconds) = GB-seconds. Original: 0.128 GB × 0.8s = 0.1024 GB-s. New: 0.512 GB × 0.2s = 0.1024 GB-s. Same cost! Often, increasing memory proportionally reduces duration (CPU scales with memory), keeping cost neutral while improving performance. Sometimes you can increase memory significantly, get much better performance, AND reduce cost if duration drops more than proportionally.

Lambda burst concurrency: When traffic suddenly spikes, Lambda can immediately scale to 3,000 concurrent executions (in us-east-1, us-west-2, eu-west-1 — smaller in other regions). After that initial burst, Lambda scales at +500 executions per minute until reaching the account limit (default 1,000). If traffic spikes faster than scaling, requests are throttled (429 error). Design with SQS or retry logic for bursty workloads.

CodeGuru Profiler continuously samples your application's CPU usage at runtime (using agents for Java, Python, Node.js) and builds flame graphs showing which methods consume the most CPU time. It identifies expensive methods, wasteful code paths, and latency bottlenecks in live production or staging environments. CodeGuru Reviewer is for static code analysis of source code in PRs (security, best practices). They are complementary tools.

SNS Subscription Filter Policies are JSON policies attached to each individual SQS queue subscription. When a message is published to SNS with message attributes (e.g., eventType="purchase", amount=1500), SNS evaluates each subscription's filter policy and only delivers the message to matching subscribers. No Lambda routing needed — this is native SNS functionality for the Fanout + Filter pattern.

Amazon CloudFront caches S3 objects at 450+ edge locations worldwide. During a sale event, CloudFront serves cached images from the nearest edge location — most requests never reach the S3 origin. This eliminates S3 throttling (S3 limit is 5,500 GET requests/second per prefix; CloudFront serves millions from cache). CloudFront also reduces latency (users connect to nearby edge) and S3 data transfer costs (only cache misses hit S3).

DynamoDB Streams captures a time-ordered sequence of item-level modifications (writes, updates, deletes) in near-real-time. Each stream record contains: what happened (INSERT/MODIFY/REMOVE) and optionally the old image, new image, or both (configured via StreamViewType). Records are retained for 24 hours. Lambda can be configured to trigger on new stream records for event-driven processing (replication, notifications, search indexing).

Compute Optimizer supports exactly 5 resource types: EC2 instances (instance type), EC2 Auto Scaling Groups (instance type), EBS volumes (volume type, IOPS, throughput), Lambda functions (memory setting), and ECS services on Fargate (CPU/memory). Note: RDS is NOT supported by Compute Optimizer — use Performance Insights for RDS right-sizing.

ElastiCache is the canonical answer for "improve RDS read performance". For static reference data (rarely changes), Lazy Loading works well: first read goes to RDS, result cached in ElastiCache with a long TTL. Subsequent reads (which are the majority) return in sub-millisecond from memory without hitting the database. Read Replicas (D) also help but only shift reads to replica DB — still slower than in-memory cache.

Kinesis Data Streams is built for high-throughput, real-time data ingestion at massive scale. Key advantages over SQS: data is ordered within shards, retained for up to 365 days (vs SQS max 14 days), multiple consumers can read the same records independently (replay capability), and designed for streaming data patterns. At 50,000 records/sec, Kinesis with appropriate shard count handles this natively.

VPC Flow Logs capture ACCEPT/REJECT decisions at the network interface level. If security groups are fine, the next suspects are: Network ACLs (NACLs — stateless, can block return traffic), routing tables (missing route between subnets), or misconfigured VPC peering. VPC Flow Logs will show REJECT records if traffic is being blocked by NACLs. Check for asymmetric NACL rules (NACLs are stateless; both inbound and outbound rules must be configured).