Hadoop Configuration

Key Takeaways

• Understand the three modes of Hadoop: standalone, pseudo-distributed, and fully-distributed.
• Configurations are crucial for running Hadoop effectively in pseudo-distributed mode.
• Familiarize yourself with core-site.xml, hdfs-site.xml, yarn-site.xml, and mapred-site.xml.
• Ensure that Hadoop user and Java environments are correctly configured for smooth operation.
• Verification commands help confirm the accuracy of your configuration.

Hadoop operates in three primary modes, each suited for different use cases and stages of deployment.

Standalone Mode

The default configuration of Hadoop, standalone mode, runs as a single process on your system. It's suitable for debugging and exploring Hadoop features without any setup complexity.

Pseudo-Distributed Mode

In this mode, Hadoop runs each daemon as a distinct Java process on a single machine. It simulates a distributed environment, providing a stepping stone toward fully-distributed setups.

Fully Distributed Mode

This production-level setup spreads Hadoop processes across multiple machines, requiring at least two nodes. It's used in production environments for handling large-scale data processing.

Here, we'll configure Hadoop in pseudo-distributed mode to provide a realistic distributed computing experience using minimal hardware.

Configuring Hadoop

If Java and Hadoop are installed as detailed in our Hadoop Environment Setup, follow these steps. You'll need to adjust configuration files in /usr/local/hadoop/etc/hadoop:

core-site.xml

This file sets Hadoop's Hadoop Distributed File System (HDFS) communication. Update the file:

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>

This configuration establishes the URI for all HDFS requests, centralizing file system communication.

hdfs-site.xml

The main configuration for HDFS includes setting the replication factor and defining paths for the namenode and datanodes:

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:///home/hadoop/hdfs/namenode</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>file:///home/hadoop/hdfs/datanode</value>
  </property>
</configuration>

Ensure these directories are under the hadoop user's ownership, confirming the user's read/write access to necessary paths.

yarn-site.xml

Yarn, the resource manager, requires configuration to manage Hadoop resources:

<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
</configuration>

mapred-site.xml

To prepare MapReduce, first copy and rename the template file, then configure:

cp mapred-site.xml.template mapred-site.xml

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>

Configuring the Hadoop User Environment

Switch to the hadoop user and amend your ~/.bashrc file:

su hadoop

export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

Apply changes using:

source ~/.bashrc

Configuring Java for Hadoop

Specify the Java path in hadoop-env.sh:

export JAVA_HOME=/usr/local/jdk

Point to your Java location to ensure Hadoop can access necessary Java classes.

Verify Hadoop Configuration

Initialize HDFS by formatting the namenode. Execute:

hdfs namenode -format

If set up correctly, you'll receive a confirmation message indicating success.

Verify Yarn

Start Yarn with:

start-yarn.sh

A successful startup will output running daemons and their logging details.

Verify HDFS

To ensure HDFS services are correctly deployed, run:

start-dfs.sh

Look for logging confirmations without error traces to verify healthy namenode and datanode processes.

Conclusion

With your configuration set, access Hadoop's services through http://localhost:50070/ and Yarn applications at http://localhost:8088/. Next steps involve exploring HDFS architecture and operations.

FAQ

How do I switch from pseudo-distributed to fully-distributed mode?

Transition involves adding real hardware nodes. Update configuration files to specify the new nodes' addresses and adjust dfs.replication as needed.

What are the minimum requirements for running Hadoop in fully-distributed mode?

You'll need at least two nodes with adequate CPU, memory, and disk space. Each node should ideally have a dual-core processor, 8GB RAM, and a few hundred GBs of disk space.

Why am I encountering a permissions error on startup?

Verify that the directories specified in your configuration files have the correct ownership by the hadoop user. Common issues arise from misconfigured file permissions.

What Java version is supported with Hadoop?

Recent Hadoop distributions are compatible with Java 11 and above. Check Hadoop's official documentation for precise requirements.