Domain 4 › Topic 19
⭐⭐⭐ Big Data Processing

🐘 Amazon EMR

Amazon EMR (Elastic MapReduce) is a managed big data platform that runs Apache Spark, Hadoop, HBase, Presto, and other frameworks for processing massive datasets. It's the AWS service for large-scale data processing and analytics.

🔑 Covers: EMR Basics · Apache Spark/Hadoop · S3 Integration · Use Cases · Cost Optimization

🐘 What is Amazon EMR?

Amazon EMR (Elastic MapReduce)
A cloud big data platform that simplifies running open-source big data frameworks like Apache Spark, Hadoop, HBase, Presto, and Hive on AWS. EMR provisions and manages the cluster of EC2 instances for you. You provide the code and data (usually in S3), and EMR runs your processing jobs. Used for transforming, analyzing, and processing terabytes to petabytes of data.

🔧 Core Frameworks EMR Supports

⚡ Apache Spark

The most popular big data processing engine. In-memory processing makes it 100x faster than Hadoop MapReduce for iterative algorithms. Used for: ETL, machine learning (MLlib), graph processing, streaming (Spark Streaming).

🐘 Apache Hadoop MapReduce

The original big data framework. Processes data in two phases: Map (transform each record) and Reduce (aggregate results). Disk-based processing — slower than Spark but handles datasets larger than memory.

🔍 Apache Hive / Presto

SQL-like query interfaces on top of big data. Hive translates SQL to MapReduce/Spark jobs. Presto is a fast SQL query engine for interactive analytics on large datasets in S3.

🏗️ EMR Cluster Architecture

EMR Cluster — Three Node Types
👑 Master Node Manages cluster, tracks job status, coordinates work. One per cluster. Run resource manager (YARN). EC2 instance you SSH into.
↓ distributes work ↓
⚙️ Core Nodes Run tasks AND store data (HDFS). If a core node fails, data might be lost. At least one required.
🔄 Task Nodes Run tasks only (no HDFS storage). Can use Spot instances (cheap!). Can be added/removed while cluster runs. Optional.

📦 S3 Integration — The Key Optimization Pattern

Instead of storing data in HDFS (on EC2 disks), store data in S3. This is the most important EMR optimization pattern for cost:

✅ With S3 (Recommended)

  • Data persists after cluster terminates
  • No storage cost during idle time
  • Spin up/down clusters as needed
  • Multiple clusters can share the same data
  • S3 scales infinitely

HDFS (Only for temp data)

  • Data stored on EC2 disks (lost when cluster stops)
  • Faster for iterative algorithms (data is local)
  • Use HDFS only for intermediate processing results
  • Store final results and source data in S3

💰 EMR Cost Optimization

  • Spot Instances for Task Nodes: Task nodes don't store data → safe to use Spot (cheap). Can reduce cost by 70-90%.
  • Transient clusters: Spin up only when needed, terminate after job completes. No cost when idle.
  • Right-sizing: Use Compute Optimizer recommendations for EC2 instance types.
  • EMR Serverless: Newer option — no cluster management, pay only for compute used during job runs.
🎯 Exam Tip
EMR is the answer for: "process terabytes of data with Spark/Hadoop", "run big data analytics jobs", "large-scale ETL processing". For optimization: use S3 for storage + Spot instances for task nodes. Don't confuse with Athena (serverless SQL on S3 — no cluster) or Kinesis Data Analytics (real-time streaming analytics).
← Compute Optimizer