IMPORT YOUR OWN RSA SSH KEY INTO AMAZON EC2

Recently, I needed to import my own RSA SSH key to EC2. Here is a quick way to import to all regions

NB: You need the ec2 tools set up before you can run this. You will also need to have setup an x509 certificate pair.

You can read more about the ec2-import-keypair command in the EC2 documentation.

Assign multiple security groups on Amazon EC2 Run Instances (ec2-run-instances)

Recently, I needed to deploy new servers with multiple defined security groups on AWS EC2 using CLI.

Note: Security groups can only be defined on instance launch unless using VPC.

Logging the client IP behind Amazon ELB with Apache

When you place your Apache Web Server behind an Amazon Elastic Load Balancer, Apache receives all requests from the ELB’s IP address.

Therefore, if you wish to do anything with the real client IP address, such as logging or whitelisting, you need to make use of the X-Forwarded-For HTTP Header Amazon ELB includes in each request which contains the IP address of the original host.

Solution for logging the true client IP

Before:

After:

The one downside is that depending on how ELB treats X-Forwarded-For, it may allow clients to spoof their source IP.

Hopefully this helps out anyone experiencing this issue.

Install s3cmd/s3tools Debian/Ubuntu

1. Register for Amazon AWS (yes, it asks for credit card)

2. Install s3cmd (following commands are for debian/ubuntu, but you can find how-to for other Linux distributions on s3tools.org/repositories)

3. Get your key and secret key at this link

4. Configure s3cmd to work with your account

5. Make a bucket (must be an original name, s3cmd will tell you if it’s already used)

Simple Shell Script to Send Email Ubuntu

We run a local SVN sever that syncs to Amazon S3 + an EC2 instance. Syncing locally to EC2 can take anywhere from 10-20 minutes and instead of waiting for the script to finish I decided to write a quick little email notify script. Enjoy.

Remotely Connect to Your EC2 Server Using Coda FTP

Getting Coda to Remotely Connect to Your EC2 Server

Okay, so let’s jump to it. How do you get Coda to connect to your EC2 server?

First thing you will need is your private key (the file that you should have received from Amazon that ends in PEM). If you don’t have one of these babies then you’re going to need to go through Amazon’s getting started guide paying specific attention to section 3 & 4 on the “Launch Instance” section and the “Connect to Linux/Unix/Windows Instance”.

For the purposes of this walk through we’ll assume that the name of your private key is: my_key.pem

Now that you have your key you will need to move it to your ssh directory. Opening up Terminal, go to your ssh directory by typing:

You may either choose to copy your key into this folder, or, you can leave your private key wherever it so provided you know the path to it. Copying the file to the ssh directory would require you to do this:

Once you have either copied your key or noted its exact location in the ssh directory that you are actively in edit the config

Even if you don’t have that file, by doing the above you will be creating it. You may also need to be super-user if no permission is given:

In this file add the following lines (instructions of each line are after the # mark so you may remove information after these):

To save this document in vim you will need to exit by hitting :wq and then hitenter to save the document.

When this done, if you private key hasn’t been added you should do that now by entering:

or

To test that there are no problems with connecting to your server, enter in terminal the next command:

(instead of root you may be asked to use ubuntu instead, i.e.

If you receive any

errors at the bottom of the text you will have to change the permission of your key file, to do this enter:

or

Then try running the previous command again.

If you find upon running the

that you are faced with a prompt that asks you a yes/no question, enter “yes”.

Once you have successfully gained access to your EC2 server from the terminal window you are now almost ready to interface your server with Coda!

Open up Coda, click on the Sites tab button, click Add Site…and in all the text-boxes add the appropriate information according to your settings from the image above. A little more detail is provided here:

  • Nickname: can be anything you want to help you identify the site in Coda.
  • Root URL: URL to access the site.
  • Local URL: local location of the site (I use MAMP to test my sites)
  • Remote Root: where the server files are located that are directly correlated to the local structure
  • Local Root: where the local file/folder structure can be found
  • Server: the “Host” name you inserted in the config file above, i.e. 1.2.3.4 (or the public DNS address)
  • User Name: either “root” or “ubuntu” or whatever the server requests as a user to log in to the server
  • Password: LEAVE BLANK!
  • Port: leave on the default 22
  • Protocal: SFTP

When all this has been done you can click on “Save”. When you’re sure that all the details match up correctly, click connect in the “Remote” tab area and hopefully everything should work as it should! If you do get password prompts check that your key file has been chmod to 600 (see above) and that your ~/.ssh directory has been chmod to 700.

If you still run into problems PLEASE check that all the server details and username details are correct. I had my server IP address round the wrong way and that threw out a few attempts.

Anyway, hopefully this helps you integrate Coda and your EC2 server together quicker.

PS: make sure your folder has permissions for the user you are uploading the data to, I had to run this command in terminal before it allowed me to upload files from within Coda:

Monitoring and Scaling Production Setup using Scalr

Scalr is a fully redundant, self-curing and self-scaling hosting environment utilizing Amazon’s EC2. It is an initiative delivered by Intridea. It allows you to create server farms through a web-based interface using prebuilt AMI’s for load balancers (pound or nginx), app servers (apache, others), databases (mysql master-slave, others), and a generic AMI to build on top of. The health of the farm is continuously monitored and maintained. When the Load Average on a type of node goes above a configurable threshold a new node is inserted into the farm to spread the load and the cluster is reconfigured. When a node crashes a new machine of that type is inserted into the farm to replace it. 4 AMI’s are provided for load balancers, mysql databases, application servers, and a generic base image to customize. Scalr allows you to further customize each image, bundle the image and use that for future nodes that are inserted into the farm. You can make changes to one machine and use that for a specific type of node. New machines of this type will be brought online to meet current levels and the old machines are terminated one by one. The project is still very young, but we’re hoping that by open sourcing it the AWS development community can turn this into a robust hosting platform and give users an alternative to the current fee based services available.

Run a Hadoop Cluster on EC2 the easy way using Apache Whirr

To set up the cluster, you need two things-
1.    An AWS account
2.    A local machine running Ubuntu (Mine was running lucid)
The following steps should do the trick-
Step 1 – Add the JDK repository to apt and install JDK (replace lucid with your Ubuntu version, check using lsb_release – c in the terminal) –

Step 2 – Create a file named cloudera.list in /etc/apt/sources.list.d/ and paste the following content in it (again, replace lucid with your version)-

Step 3 – Add the Cloudera Public Key to your repository, update apt,  install Hadoop and Whirr-

Step 4 – Create a file hadoop.properties in your $HOME folder and paste the following content in it.

Step 5 – Replace [AWS ID] and [AWS KEY] with your own AWS Access Identifier and Key. You can find them in the Access Credentials section of your Account. Notice the third line, you can use it to define the nodes that will run on your cluster. This cluster will run a node as combined namenode (nn) and jobtracker (jt) and another node as combined datanode (dn) and tasktracker (tt).

Step 6 – Generate a RSA keypair on your machine. Do not enter any passphrase.

Step 7 – Launch the cluster! Navigate to your home directory and run-

This step will take some time as Whirr creates instances and configures Hadoop on them.

Step 8 – Run a Whirr Proxy. The proxy is required for secure communication between master node of the cluster and the client machine (your Ubuntu machine). Run the following command in a new terminal window-

Step 9 – Configure the local Hadoop installation to use Whirr for running jobs.

Step 10 – Add $HADOOP_HOME to ~/.bashrc file by placing the following line at the end-

Step 11 – Test run a MapReduce job-

Step 12 (Optional) – Destroy the cluster-

Note: This tutorial was prepared using material from the CDH3 Installation Guide

Deploy Hadoop cluster with Ubuntu Juju

Here I’m using new features of Ubuntu Server (namely Ubuntu Juju) to easily deploy a small Hadoop cluster.

Add juju apt-repository:

Add charms NON-local:

Add charms local:

Generate your environment settings with

Then edit ~/.juju/environments.yaml to use your EC2 keys. It’ll look something like:

Bootstrap and then deploy charms NON-local

Bootstrap and then deploy charms local

Check Juju status:

Everything is finished and happy when

Optionally scale horizontally with datanode:

When all said and done: (SSH into namenode/datanode)