Hadoop MapReduce Example

Hadoop MapReduce is the heavy-lifter framework for processing big data on distributed systems. It lets you break down massive datasets into smaller chunks, process them in parallel, and then combine the results.

Understanding how to run MapReduce jobs is essential for leveraging Hadoop's full potential. In this post, we’ll explore a classic use case: the WordCount application.

Key Takeaways

• MapReduce jobs in Hadoop process data by mapping tasks to smaller chunks and reducing them to a final result.
• WordCount is a quintessential MapReduce application demonstrating the mapper and reducer functions.
• Setting up WordCount includes configuring input/output paths, creating necessary input files, and executing the job through Hadoop CLI.

MapReduce Basic Example

Hadoop provides built-in examples, including the WordCount application. Navigate to /hadoop/share/hadoop/mapreduce/ where you'll find hadoop-mapreduce-examples.jar containing sample classes like WordCount.

Here's the source code for the WordCount class:

import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount {

    public static class TokenizerMapper extends Mapper{

        private final static IntWritable one = new IntWritable(1);
        private Text word = new Text();

        public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
            StringTokenizer itr = new StringTokenizer(value.toString());
            while (itr.hasMoreTokens()) {
                word.set(itr.nextToken());
                context.write(word, one);
            }
        }
    }

    public static class IntSumReducer extends Reducer {
        private IntWritable result = new IntWritable();

        public void reduce(Text key, Iterable values, Context context) throws IOException, InterruptedException {
            int sum = 0;
            for (IntWritable val : values) {
                sum += val.get();
            }
            result.set(sum);
            context.write(key, result);
        }
    }

    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        Job job = Job.getInstance(conf, "word count");
        job.setJarByClass(WordCount.class);
        job.setMapperClass(TokenizerMapper.class);
        job.setCombinerClass(IntSumReducer.class);
        job.setReducerClass(IntSumReducer.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));
        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
}

The WordCount example highlights key components: TokenizerMapper for mapping and IntSumReducer for reducing.

The Mapper Function

In TokenizerMapper, the map function takes input text and outputs key/value pairs for each word. It processes each word as a unique key, paired with a count of 1.

The Reducer Function

The IntSumReducer class implements the reduce function to sum these counts, generating a total count for each word.

The main method sets everything in motion by configuring the job, setting paths, and executing the function.

Running the WordCount MapReduce Application

To run WordCount, you'll need to configure your input/output directories in HDFS and create the necessary input files.

1. Create Input Directory

Create an input directory in HDFS using:

$ hdfs dfs -mkdir -p /example/wordcount/input

2. Create Input Files

Create two text files locally:

echo 'Hello MapReduce Hello' >> input1.txt

echo 'Goodbye MapReduce Goodbye' >> input2.txt

3. Add Files to HDFS

Use these commands to upload your files to HDFS:

$ hdfs dfs -put input1.txt /example/wordcount/input

$ hdfs dfs -put input2.txt /example/wordcount/input

4. Run the MapReduce Application

Execute the WordCount job with:

$ hadoop jar hadoop-mapreduce-examples.jar wordcount /example/wordcount/input /example/wordcount/output

Inputs and outputs are specified via arguments, with results in /example/wordcount/output.

5. Verify Results

After completion, confirm the results using:

$ hdfs dfs -cat /example/wordcount/output/part-r-0000
goodbye	2
hello	2
mapreduce	2

The result validates our inputs' total word counts.

Conclusion

The WordCount example is a gateway to running MapReduce effectively on Hadoop. With this basic understanding, you're poised to delve into more complex MapReduce tasks.

FAQ

What is the role of a combiner?

Combiners optimize job processing by locally combining map outputs with the same key before sending them to the reducer, reducing data shuffled across the network.

Can I run multiple MapReduce jobs in sequence?

Yes, you can chain MapReduce jobs together by taking the output of one as the input to another, using scripting or workflow management tools like Apache Oozie.

What's the best way to manage error handling in MapReduce jobs?

Handle errors by configuring logging, using counters to track exceptions, and implementing robust default handling in your mapper and reducer classes.