Simple Hadoop with EMR

Set up a java maven project in an ide, configure group id and artifact, add hadoop mapreduce dependencies, and implement mapper, reducer, and runner with suitable imports.

Explore how the mapper processes map value pairs with input types like longWritable and text and output types like text and intWritable, using wrappers that enable serialization and sorting.

Write the mapper to emit consonant and vowel counts per line via the context. Convert lines to strings, count vowels and consonants, and output to two buckets for the reducer.

Implement a Hadoop reducer that sums the mapper-provided counts for each key (consonants or vowels) by iterating over values and emitting the total with the same key.

Create a Hadoop runner main method, configure job with mapper and reducer, set end line input format, and specify input/output paths before building and running the jar on EMR.

Learn how to package a Hadoop application into a single runnable jar by using the Maven shade plugin to include dependencies, enabling deployment to Hadoop/EMR without separate jars.

Explore AWS basics with S3 and EMR, upload your jar to S3, set up buckets and input/output folders, and run a vowel-consonant count on Hadoop.

Set up an EMR cluster, add a step with a custom jar, and configure the main class name plus input and output arguments, ensuring unique output folders.

Learn to run a Hadoop job on AWS EMR by cloning clusters, configuring a jar and main class, setting input/output paths, and counting vowels and consonants in results.

Analyze stock market data with mapper and reducer on EMR to compute monthly deltas by subtracting open from close, outputting month-level results.

Running the same job on a 10-node Hadoop cluster (1 master, 9 core) reduces time from about an hour to 15 minutes. Terminate cluster after the job to avoid charges.

Learn to run unconventional EMR Hadoop jobs without a map or reducer, scale across many nodes, and set up a Maven project with EMR dependencies to interact with external services.

Build a map-only Hadoop job on EMR that reads URLs, calls an API with HTTP GET, parses JSON with Jackson, and saves the body to S3 using the AWS SDK.

Run a Hadoop job on Amazon EMR with a custom jar to fetch JSON API data, extract the body, and save results to S3, including setup, input creation, and troubleshooting.

learn to apply EMR and Hadoop to real problems, explore map-only versus reducing workflows, and build hands-on experience with a hello world example.