Domain 4 › Topic 12
⭐⭐⭐⭐ Log Analytics Trio

🔎 OpenSearch · Athena · Kinesis

Three powerful services for log analytics and data processing. OpenSearch for log search and visualization. Athena for SQL queries on S3. Kinesis for real-time streaming data ingestion. They often work together in log analytics pipelines.

🔑 Covers: OpenSearch Search/Visualization · Athena SQL on S3 · Kinesis Streaming · Pipeline Architecture

🔎 Amazon OpenSearch Service — Full-Text Log Search

Amazon OpenSearch Service (formerly Elasticsearch Service)
A managed service based on the open-source OpenSearch/Elasticsearch engine. Provides powerful full-text search capabilities and the OpenSearch Dashboards (formerly Kibana) visualization tool. You index your documents (logs, records) and can search them with complex queries in milliseconds — even across billions of records.

🎯 Key Use Cases

📝 Log Analytics

Ingest application logs. Search across billions of log entries instantly. Create dashboards showing error rates, top endpoints, slow queries. Used as the "E" in ELK Stack (Elasticsearch-Logstash-Kibana).

🔍 Application Search

Add full-text search to your application. Users can search products, articles, documents. OpenSearch understands context and relevance — not just exact string matching.

📊 Visualization

OpenSearch Dashboards (based on Kibana) provides rich visualization: bar charts, pie charts, time-series graphs, heat maps, maps. Create operations dashboards for your team.

📥 How to Get Data Into OpenSearch

SourceHow to Send Data
CloudWatch LogsSubscription Filter → Lambda → OpenSearch OR Kinesis Firehose → OpenSearch
S3Lambda triggered on S3 event → index to OpenSearch
Kinesis Data StreamsKinesis → Lambda → OpenSearch OR Kinesis → Firehose → OpenSearch
DynamoDB StreamsDynamoDB Streams → Lambda → OpenSearch (for search over DynamoDB data)
🎯 Exam Tip
OpenSearch is the answer for: "full-text search over logs", "search for specific patterns across billions of log entries", "need Kibana-style dashboards", "ELK-style log analysis". CloudWatch Logs Insights is better for ad-hoc queries; OpenSearch is better for ongoing operational dashboards and full-text search.

🦉 Amazon Athena — SQL Queries on S3

Amazon Athena
A serverless query service that lets you run standard SQL queries directly against data stored in Amazon S3. You don't load data into a database — you query it where it sits in S3. Pay per query (per data scanned). Uses a schema-on-read approach: you define the structure when querying, not when storing.

🎯 When to Use Athena

  • Query log files stored in S3 — CloudTrail logs, ALB access logs, VPC Flow Logs, S3 access logs
  • Query large CSV/JSON/Parquet/ORC files without loading them into a database
  • Run ad-hoc analysis on historical data that's been exported to S3
  • Query AWS service logs that automatically go to S3 (CloudTrail, Config, etc.)
📌 Example — Query CloudTrail Logs with Athena
CloudTrail logs are stored in S3 as JSON files. With Athena, you can run:

SELECT userIdentity.userName, eventName, eventTime, sourceIPAddress
FROM cloudtrail_logs
WHERE eventName = 'DeleteBucket' AND eventTime > '2024-01-01'
ORDER BY eventTime DESC;

This instantly finds all S3 bucket deletions in 2024 — without loading data into any database!

💰 Athena Cost Optimization — Use Partitioning and Columnar Formats

Athena charges $5 per TB scanned. To reduce costs:

  • Partitioning: Organize S3 data by date/region/etc. Query only the partitions you need. Example: WHERE year='2024' AND month='01' only scans January data.
  • Columnar formats: Use Parquet or ORC instead of CSV/JSON. Queries only read the columns needed, not the entire file. 10-100x cost reduction for analytical queries.
  • Compression: Compress data with gzip/snappy. Less data = cheaper scans.

🌊 Amazon Kinesis — Real-Time Streaming

Amazon Kinesis
A family of services for collecting, processing, and analyzing real-time streaming data. Unlike SQS (which handles discrete messages), Kinesis is designed for continuous, high-throughput data streams — like clickstream data, IoT sensor readings, application metrics, log data. Think of it as a conveyor belt for data.

📦 The Kinesis Family — Four Services

🌊 Kinesis Data Streams

The core streaming service. Data producers send records to a stream divided into shards. Consumers (Lambda, Kinesis Data Analytics, custom apps) read from the shards. Data is retained 24 hours to 365 days. Multiple consumers can read the same data independently. Use for: custom real-time processing, when you need multiple consumers.

🚿 Kinesis Data Firehose

A delivery service that automatically loads streaming data into a destination: S3, Redshift, OpenSearch, Splunk, or HTTP endpoint. No custom code needed for delivery. Can transform data with Lambda. Use for: getting streaming data into S3/OpenSearch/Redshift without writing consumer code.

📊 Kinesis Data Analytics

Run SQL queries or Apache Flink jobs on streaming data in real-time. Example: "What is the average order value in the last 5 minutes?" without storing data first. Use for: real-time analytics, anomaly detection, aggregation on live streams.

📹 Kinesis Video Streams

Ingests video streams from cameras. Process or store video data. Use for: security cameras, live video processing, ML on video frames.

🔄 Common Log Analytics Pipeline — The Full Stack

Real-World Log Analytics Architecture
📱ApplicationWeb, mobile, IoT
λLambdaServerless
🖥️EC2/ECSServers
↓ all emit logs/events ↓
🌊Kinesis Data StreamsReal-time ingestion
↓ fan out to multiple consumers ↓
🚿Kinesis Firehose → S3Long-term storage
🔎Firehose → OpenSearchReal-time search
📊Kinesis AnalyticsReal-time aggregation
↓ query historical data in S3 ↓
🦉AthenaSQL queries on S3 logs
🎯 Exam Comparison — Kinesis vs SQS vs SNS
SQS: Message queue — one consumer processes each message. Decoupling async tasks. Not for high-volume streams.
SNS: Pub/sub — one message, many subscribers. Notification broadcasting.
Kinesis Data Streams: Ordered, replayable, multiple consumers, massive throughput. Clickstreams, IoT, logs.
Kinesis Firehose: Zero-code delivery of streams to S3/OpenSearch/Redshift.
← DynamoDB Optimization