Tutorials

Hadoop MapReduce Example

Key Takeaways

  • WordCount emits a separate (word, 1) record for every token, then sums the counts for each word.
  • Hadoop's shuffle and sort connect the mapper and reducer by grouping intermediate records with the same key.
  • You can run Hadoop's bundled examples JAR without compiling Java, or build the complete Java example below.
  • The HDFS input directory must exist; the job's output directory must not already exist.
  • This example preserves capitalization and explicitly uses one reducer, producing part-r-00000 with standard uncompressed text output.

A Hadoop MapReduce WordCount example counts words by emitting (word, 1) pairs from a mapper and summing the values for each word in a reducer. It's a small job that teaches the complete path from HDFS input to distributed processing and output files.

MapReduce is a batch-processing framework, not the automatic choice for every analytics workload. For iterative or interactive work, consider the trade-offs in Hadoop MapReduce vs Spark; here, the goal is to understand MapReduce itself.

Environment and Prerequisites

This tutorial targets the Apache Hadoop 3.5.0 binary distribution on Linux, using Bash and JDK 17. Hadoop 3.5.0 requires Java 17 on servers and supports Java 17 or Java 21 on clients; use JDK 17 throughout this exercise. These requirements are documented in the Hadoop 3.5.0 overview.

You should already have a working installation. Put its command-line tools on PATH, set JAVA_HOME, and use your platform's client configuration. If necessary, point HADOOP_CONF_DIR at the supplied configuration directory rather than an unrelated local installation.

Execution modeProcessingStorage in this tutorial
Standalone/localLocal runner; no YARN daemonsLocal files, explicitly selected with file:///
YARN with HDFSTasks scheduled through YARNHDFS input and output directories

The main walkthrough assumes fs.defaultFS points to your HDFS cluster and mapreduce.framework.name is yarn. A single-node installation can use HDFS while still running MapReduce locally, so “pseudo-distributed” doesn't automatically mean YARN execution. Review Hadoop configuration and YARN architecture if those distinctions are unclear.

Check the client before starting:

hadoop version
hdfs getconf -confKey fs.defaultFS
hdfs dfs -ls /
yarn application -list

These commands report the installed version and filesystem configuration, list the filesystem root, and contact YARN to list applications. You also need permission to create or use /example/wordcount; otherwise replace that prefix throughout with a directory you're authorized to write.

How the WordCount MapReduce Job Works

HDFS input files
       |
   input splits
       |
   mapper records: (word, 1)
       |
   optional local combiner
       |
   shuffle + sort: group matching words
       |
   reducer: sum each word's values
       |
HDFS output: part-r-00000

An input split describes a portion of the input assigned to one map task. Hadoop creates one map task per split; a split isn't simply another name for a file or an HDFS block. The input format determines how files are split and turned into records. For a shorter conceptual introduction, see what MapReduce does.

The Mapper Function

TextInputFormat supplies each line as a Text value and its position in the file as a LongWritable key. Our mapper ignores that position and tokenizes the line. Hello MapReduce Hello produces (Hello, 1), (MapReduce, 1), and another (Hello, 1)—mapping does not make the keys unique.

Shuffle, Sort, and the Reducer Function

The framework partitions mapper output among reducers, transfers it during the shuffle, and sorts and groups matching keys. The reducer receives a word and its associated values, then emits one total. For example, Hello with values [1, 1] becomes (Hello, 2); a combiner might instead supply the partial total [2].

Why Summation Is a Safe Combiner

A combiner performs optional partial aggregation of intermediate records. It may run zero, one, or multiple times, so correctness cannot depend on it running. Summation works because adding partial sums gives the same total regardless of grouping or order; calculating an average of partial averages does not generally work.

MapReduce Java Example

Save this as WordCount.java in a new local working directory. It retains the classic mapper, sum reducer, combiner, and job configuration, while adding argument validation and explicit input/output formats. The code selects one reducer and disables output compression so the result-reading command is predictable.

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;

public class WordCount {
    public static class TokenizerMapper
            extends Mapper<LongWritable, Text, Text, IntWritable> {
        private static final IntWritable ONE = new IntWritable(1);
        private final Text word = new Text();

        @Override
        protected void map(LongWritable offset, Text line, Context context)
                throws IOException, InterruptedException {
            StringTokenizer tokens = new StringTokenizer(line.toString());
            while (tokens.hasMoreTokens()) {
                word.set(tokens.nextToken());
                context.write(word, ONE);
            }
        }
    }

    public static class IntSumReducer
            extends Reducer<Text, IntWritable, Text, IntWritable> {
        private final IntWritable result = new IntWritable();

        @Override
        protected void reduce(Text word, Iterable<IntWritable> values,
                              Context context)
                throws IOException, InterruptedException {
            int sum = 0;
            for (IntWritable value : values) {
                sum += value.get();
            }
            result.set(sum);
            context.write(word, result);
        }
    }

    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        String[] paths = new GenericOptionsParser(conf, args)
                .getRemainingArgs();
        if (paths.length != 2) {
            System.err.println(
                    "Usage: WordCount [generic options] <input> <output>");
            System.exit(2);
        }

        Job job = Job.getInstance(conf, "word count");
        job.setJarByClass(WordCount.class);
        job.setMapperClass(TokenizerMapper.class);
        job.setCombinerClass(IntSumReducer.class);
        job.setReducerClass(IntSumReducer.class);
        job.setNumReduceTasks(1);
        job.setInputFormatClass(TextInputFormat.class);
        job.setOutputFormatClass(TextOutputFormat.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        FileOutputFormat.setCompressOutput(job, false);
        FileInputFormat.addInputPath(job, new Path(paths[0]));
        FileOutputFormat.setOutputPath(job, new Path(paths[1]));

        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
}

GenericOptionsParser separates Hadoop options such as -Dname=value from the two application paths. waitForCompletion(true) submits the job, reports progress, and returns whether it succeeded; this program converts that result into its process exit status.

Running the WordCount MapReduce Application

Locate the Hadoop Examples JAR

The bundled application lets you try WordCount before compiling anything. For the Apache Hadoop 3.5.0 binary distribution, use the versioned JAR under your installation directory:

: "${HADOOP_HOME:?Set HADOOP_HOME to your Hadoop installation}"
ls "$HADOOP_HOME"/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar
EXAMPLES_JAR="$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.5.0.jar"

If the listing shows a different filename, set EXAMPLES_JAR to that actual file. For packaged installations or platform images, use the examples-JAR location documented by your platform; don't assume /hadoop is the installation root.

1. Create the HDFS Input Directory

hdfs dfs -mkdir -p /example/wordcount/input

This directory must contain only the intended sample inputs to reproduce the counts below. For more filesystem operations, see the guide to HDFS commands.

2. Create Two Local Input Files

printf '%s\n' 'Hello MapReduce Hello' > input1.txt
printf '%s\n' 'Goodbye MapReduce Goodbye' > input2.txt

Use these filenames only in your tutorial working directory: the commands overwrite existing files rather than appending another copy of the sample text.

3. Upload the Files to HDFS

hdfs dfs -put -f input1.txt input2.txt /example/wordcount/input/
hdfs dfs -ls /example/wordcount/input

-put copies local files into the configured filesystem; -f replaces existing destination files with the same names.

4. Submit the Bundled WordCount Job

Choose an output path that doesn't exist. The command explicitly selects one reducer and uncompressed output rather than relying on your cluster's settings.

hadoop jar "$EXAMPLES_JAR" wordcount \
  -Dmapreduce.job.reduces=1 \
  -Dmapreduce.output.fileoutputformat.compress=false \
  /example/wordcount/input \
  /example/wordcount/output
# Optional cleanup: deletes the previous tutorial output directory.
hdfs dfs -rm -r /example/wordcount/output

If you use that cleanup command, run it before resubmitting the job, not before reading results you want to keep.

5. Read the Results

hdfs dfs -cat /example/wordcount/output/part-r-00000
Goodbye	2
Hello	2
MapReduce	2

The filename contains five zeroes. Standard text output separates each word and count with a tab, and this sample preserves capitalization: the input contains Hello, not hello.

Reducer count is job configuration, not a universal constant. With multiple reducers, standard file output creates multiple reducer part files; each reducer receives sorted keys, but concatenating those files doesn't guarantee a globally sorted result.

Build and Run Your Java Version

From the directory containing WordCount.java, compile against your installed Hadoop libraries and package the generated classes. This avoids selecting a separate dependency version that might not match your cluster.

mkdir -p build/classes
javac -cp "$(hadoop classpath --glob)" -d build/classes WordCount.java
jar --create --file build/wordcount.jar -C build/classes .

hadoop jar build/wordcount.jar WordCount \
  /example/wordcount/input \
  /example/wordcount/output-java

hdfs dfs -cat /example/wordcount/output-java/part-r-00000

hadoop classpath --glob expands the installation's Hadoop classpath. Notice the invocation difference: the examples JAR uses the registered command wordcount, while your JAR uses your Java main class, WordCount. Keep output-java absent before submission.

Optional: Run Without HDFS or YARN

After creating the local input files above, you can run your compiled job locally. Explicitly select both the local execution framework and local filesystem; changing one doesn't automatically change the other. Use a fresh local-output directory name.

mkdir -p local-input
cp input1.txt input2.txt local-input/
hadoop jar build/wordcount.jar WordCount \
  -Dmapreduce.framework.name=local \
  -Dfs.defaultFS=file:/// \
  local-input local-output
cat local-output/part-r-00000

Case, Punctuation, and Unicode Semantics

The classic mapper uses StringTokenizer's default delimiters: space, tab, newline, carriage return, and form feed. It neither lowercases tokens nor removes punctuation, and that delimiter set does not cover every Unicode whitespace character. Consequently, Hello, hello, and Hello! are different keys.

To change those semantics, add this field inside TokenizerMapper and replace its map method. This fragment uses fully qualified Java names, so no additional imports are needed.

// Fragment: inside TokenizerMapper; replace its existing map() method.
private static final java.util.regex.Pattern TOKEN =
        java.util.regex.Pattern.compile("[\\p{L}\\p{M}\\p{N}]+");

@Override
protected void map(LongWritable offset, Text line, Context context)
        throws IOException, InterruptedException {
    java.util.regex.Matcher tokens = TOKEN.matcher(
            line.toString().toLowerCase(java.util.Locale.ROOT));
    while (tokens.find()) {
        word.set(tokens.group());
        context.write(word, ONE);
    }
}

This version lowercases using locale-independent rules and counts runs of Unicode letters, combining marks, and numbers. Punctuation separates tokens, so don't becomes don and t. With the original two files, the totals remain two each, but the output keys become goodbye, hello, and mapreduce. Rebuild your JAR before running the changed mapper.

This is a deliberate counting policy, not language-aware word segmentation. Decide how your application should handle apostrophes, accents, and languages without spaces before treating these counts as business data.

Common Mistakes and First-Run Failures

  • Output already exists: choose another output directory or deliberately remove the previous tutorial output. Creating an empty destination directory still causes rejection.
  • Input path missing: inspect the exact path with hdfs dfs -ls. Creating local files isn't the same as uploading them into HDFS.
  • Wrong JAR or class: verify the full JAR path and distinguish the bundled wordcount command from your WordCount class.
  • NameNode or ResourceManager connection failure: check the client configuration, service availability, network access, and credentials. Don't reformat an existing HDFS installation as a troubleshooting shortcut.

Validation, Diagnostics, and Job Chaining

Count Deliberately Skipped Records

Decide whether invalid input should fail the job or be skipped with an observable count. For example, this fragment records lines with no tokens; insert it immediately after constructing tokens in the original mapper. Hadoop exposes named counters through the task context.

// Fragment: after constructing tokens in the original TokenizerMapper.map().
if (!tokens.hasMoreTokens()) {
    context.getCounter("WordCount", "EMPTY_LINES").increment(1);
    return;
}

For structured input, validate required fields explicitly and count rejected records by reason. Use task logs for bounded diagnostics, not a log entry for every token or sensitive record. Task stderr is captured in YARN task logs; don't swallow unexpected exceptions just to make the job appear successful.

Run Dependent Jobs Only After Success

You can chain jobs by using a successful job's output directory as the next job's input, with a new output destination for each stage. A shell driver that checks exit status is enough for a small sequence; use the orchestration service supported by your platform when dependencies and recovery become more involved.

Don't adopt Oozie as an unqualified new-deployment recommendation: Apache Oozie retired in February 2025. Treat it as legacy infrastructure where your existing platform still operates it.

FAQ

How do I run the Hadoop WordCount example?

Use hadoop jar with the full examples-JAR path, the wordcount command, and your HDFS input and output paths. The input directory must exist, and the output directory must not exist.

Where is hadoop-mapreduce-examples.jar located?

In Apache's binary distribution, look under $HADOOP_HOME/share/hadoop/mapreduce/ for a versioned hadoop-mapreduce-examples-*.jar. If your platform packages Hadoop differently, use its documented JAR location and actual filename.

Why does a Hadoop MapReduce job fail when the output directory already exists?

File-based MapReduce output rejects an existing destination to avoid overwriting previous results, even if the directory is empty. Choose a new output path or deliberately remove the old tutorial output before resubmitting.

What is the difference between a combiner and a reducer?

A combiner optionally aggregates intermediate records, while the reducer processes the grouped values assigned to it to produce final output. A combiner may run zero, one, or multiple times, so it must preserve the result under repeated partial aggregation.

Why does Hadoop create part-r-00000 output files?

The standard file-based output names each reducer's part using an r marker and a zero-padded partition number; part-r-00000 is partition zero. This tutorial explicitly selects one reducer, so it produces one reducer part file.

Further Reading

Written by Sam Erickson

Software engineer. I write every article on StackChief myself. Author of Mastering the Tech Interview. More about me

Related reading