Hadoop Environment Setup

Key Takeaways

• Setting up Hadoop on Linux involves installing Java, configuring SSH, and downloading Hadoop.
• It's crucial to create a separate user for Hadoop for better isolation and security.
• Both Java and Hadoop installations require proper environment variable configuration.

Hadoop Installation on Linux

Here's how you can get Hadoop running on a Linux system. We recommend using a Linux-based environment for Hadoop due to its configuration flexibility and extensive documentation. If you're not on Linux, consider creating a Linux VM for ease of compatibility with this guide.

Setting Up a Dedicated Hadoop User

It’s best practice to create a dedicated user for Hadoop. This separation ensures Hadoop's operations don't inadvertently affect your main system configuration. Execute the following commands to create a new user:

$ sudo su
# useradd hadoop
# passwd hadoop

Configuring SSH for Hadoop

Hadoop requires SSH with password-less login to manage operations across nodes in a cluster. Set up SSH keys as follows:

$ ssh-keygen -t rsa
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys

This generates an SSH key pair and updates your authorized_keys file to allow password-less SSH access.

Installing Java

Java is an essential component for running Hadoop. Ensure you have the latest Java JDK installed. Here's how you set it up:

1. Download and Install Java

Visit the official Java website to download the latest stable JDK version. Once downloaded, extract it to a directory of your choice.

2. Move JDK to /usr/local

After extraction, relocate the jdk directory to /usr/local to make it accessible to all users.

3. Set JAVA_HOME and Update PATH

$ echo "export JAVA_HOME=/usr/local/jdk" >> ~/.bashrc
$ echo "export PATH=$PATH:$JAVA_HOME/bin" >> ~/.bashrc

Modify your environment variables to include Java.

4. Apply and Test

$ source ~/.bashrc
$ java -version

Verify your Java installation with java -version.

Downloading and Installing Hadoop

Now, let's download Hadoop and get it running.

1. Prepare Directory

$ cd /usr/local

Create a directory structure for Hadoop similar to Java.

2. Download Latest Hadoop

# wget https://downloads.apache.org/hadoop/common/stable2/hadoop-3.3.1.tar.gz
# tar xzf hadoop-3.3.1.tar.gz
# mv hadoop-3.3.1 hadoop

Download the latest stable Hadoop release and extract it.

3. Set HADOOP_HOME and Update PATH

$ echo "export HADOOP_HOME=/usr/local/hadoop" >> ~/.bashrc
$ echo "export PATH=$PATH:$HADOOP_HOME/bin" >> ~/.bashrc

Make sure the Hadoop binaries are accessible through your path.

4. Apply and Verify

$ source ~/.bashrc
$ hadoop version

Check the Hadoop installation with hadoop version.

FAQ

Why create a separate user for Hadoop?

Creating a separate user for Hadoop helps isolate it from your main system, reducing the risk of accidental changes or conflicts with system files.

Why use SSH for Hadoop setup?

SSH enables secure and password-less access to nodes in a Hadoop cluster, which is critical for automating cluster interactions and maintaining security.

Can I use JDK versions other than the latest?

While it’s generally safe to use newer versions of the JDK, it's always best to verify compatibility with the specific Hadoop version you are using to avoid unexpected issues.

What to do if I encounter permission issues?

Make sure all paths and directories have the correct read/write permissions for the Hadoop user, ensuring smooth installation and operation.