Running Hadoop Locally

Follow these instructions to set up Hadoop locally.

  1. Download Hadoop

    wget http://mirror.its.uidaho.edu/pub/apache/hadoop/core/hadoop-0.18.1/hadoop-0.18.1.tar.gz

    Unzip this file

    tar xvzf hadoop-0.18.1.tar.gz

  2. Edit file hadoop-0.18.1/conf/hadoop-env.sh. Set variable JAVA_HOME. It is likely that JAVA_HOME is already set in your shell (note that you still have to adjust the config file). You can see the current value by running echo $JAVA_HOME.

    Also, edit file hadoop-0.18.1/conf/hadoop-site.xml. Replace it with the content provided here.

    If running ssh localhost prompts you for a password, you will also need to run
    ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
    cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

  3. Format your Hadoop file system using bin/hadoop namenode -format and then start Hadoop

    bin/start-all.sh &

    You can create your user directory in the Hadoop file system by running bin/hadoop fs -mkdir /users/YOUR_USERNAME.