AWS Setup

Setting up your AWS account

After you have signed up for your AWS account and received the credit code, follow the instructions in the email to set up your account online. After that, go to http://aws.amazon.com/ and sign in. You need to double-check that your account is signed up for three of their services: Simple Storage Service (S3), Elastic Compute Cloud (EC2), and Amazon Elastic MapReduce by clicking here -- you should see the Services tab at the top of the page.

Important: You must remember to terminate manually the AWS cluster when you are done: if you just close the browser, the job flows continue to run, and amazon will continue to charge you for days and weeks, exhausting your credit and charging you huge amount on your credit card. Remember to terminate the AWS cluster!!

Setting up an EC2 key pair

Note: Some students were having problem running job flows because of no active key found, go to AWS security credentials page and make sure that you see a key under the access key, if not just click Create a new Access Key.

To connect to an Amazon EC2 node, such as the master nodes for the Hadoop clusters you will be creating, you need an SSH key pair. To create and install one, do the following:

  1. After setting up your account, follow Amazon's instructions to create a key pair. Follow the instructions in section "Having AWS create the key pair for you," subsection "AWS Management Console." (Don't do this in Internet Explorer, or you might not be able to download the .pem private key file.)
  2. Download and save the .pem private key file to disk. We will reference the .pem file as </path/to/saved/keypair/file.pem> in the following instructions.
  3. Make sure only you can access the .pem file. If you do not change the permissions, you will get an error message later: 
    $ chmod 600 </path/to/saved/keypair/file.pem>
  4. Note: This step will NOT work on Windows 7 with cygwin. Windows 7 does not allow file permissions to be changed through this mechanism, and they must be changed for ssh to work. So if you must use Windows, you should use PuTTY as your ssh client. In this case, you will further have to transform this key file into PuTTY format. For more information go to http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html and look under "Private Key Format."

Checking your Balance

Please check your balance regularly!!!

  1. Go to the Management Console.
  2. Click on your name in the top right corner and select "Account Activity".
  3. Now click on "detail" to see any charges < $1.

To avoid unnecessary charges, terminate your job flows when you are not using them.

USEFUL: AWS customers can now use billing alerts to help monitor the charges on their AWS bill. You can get started today by visiting your Account Activity page to enable monitoring of your charges. Then, you can set up a billing alert by simply specifying a bill threshold and an e-mail address to be notified as soon as your estimated charges reach the threshold.