Back in 2012, Amazon Web Services launched support for multiple private IP addresses for EC2 instances, within a VPC.

This is particularly useful if you host several SSL websites on a single EC2 instance, as each SSL certificate must be hosted on it’s own (private) IP address. Then you can associate the private IP address with an Elastic IP address to make the SSL website accessible from the internet.

Multiple IPs and Limits

This AWS blog entry briefly describe the multiple IPs management: http://aws.typepad.com/aws/2012/07/multiple-ip-addresses-for-ec2-instances-in-a-virtual-private-cloud.html

When you create a VPC, you are by default limited to 5 elastic IP addresses. However it is easy to request for an increase by completing this form https://aws.amazon.com/support/createCase?type=service_limit_increase&serviceLimitIncreaseType=elastic-ips

Note that a single Elastic Network Interfaces (ENI) can have multiple secondary IP addresses, for example on a m1.small instance type, you can have up to 4 IPs, which in Linux would be the eth0, eth0:0, eth0:1 and eth0:2 interfaces.

There is also a limit on the number of ENIs and IPs for each instance type, see the documentation at:

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI

Asymmetric Routing

When you add a second ENI, the AWS documentation is missing a fundamental note on how to configure the instance O.S. for handling the network routes.

If you attach the second ENI, associate it with an Elastic IP and bring it up (with ifup) in Linux after adding to /etc/network/interfaces, your network will very likely be performing asymmetric routing. Try and ping the Elastic IP of eth1, you get no response. This is because the response packets leaving the instance do not get sent out via the correct gateway.

Asymmetric routing is explained in depth in this article http://www.linuxjournal.com/article/7291

Route configuration with additional ENIs

The fix is to add additional routes for the new ENIs. This guide assumes that so far you have followed this documentation for adding a second ENI http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#attach_eni_launch

We’re assuming the instance has an interface eth0 with the private address 10.0.1.10 from a 10.0.1.0/24 subnet and we want to add an ENI using a different subnet 10.0.2.0/24 with an IP address of 10.0.2.10

The /etc/network/interfaces file should look like this after adding eth1:

Then bring up eth1 interface:

Let’s check the route:

There is one default gateway at 10.0.1.1 (which is bound to VPC the internet gateway) and will route any traffic from eth0. However any traffic from eth1 with a destination outside of 10.0.2.0/24 will be dropped, so we need to re-configure the routing to the default gateway for the 10.0.2.0/24 subnet.

Firstly, add an entry “2 eth1_rt” to the route table:

Next we need to add a default route to the gateway for eth1:

Verify that the route is added:

Finally we need to add a rule which will tell the route table to route traffic with a source of 10.0.2.0/24 via the rt_eth1 table:

Verify that the rule is added:

Now from your machine, try and ping the Elastic IP associated with eth1 and it should now work, asymmetrical routing has been fixed !

To make the route changes permanent so that they can survive a reboot, add them to the interfaces file:

If you wish to associate an private IP from the 10.0.1.0/24 subnet to eth1 (same subnet as eth0 network), just replace the gateway and subnet values to 10.0.1.1 and 10.0.1.0/24 respectively.