Multiple IPs and ENIs on EC2 in a VPC

aws-logoBack in 2012, Amazon Web Services launched support for multiple private IP addresses for EC2 instances, within a VPC.

This is particularly useful if you host several SSL websites on a single EC2 instance, as each SSL certificate must be hosted on it’s own (private) IP address. Then you can associate the private IP address with an Elastic IP address to make the SSL website accessible from the internet.

Multiple IPs and Limits

This AWS blog entry briefly describe the multiple IPs management: http://aws.typepad.com/aws/2012/07/multiple-ip-addresses-for-ec2-instances-in-a-virtual-private-cloud.html

When you create a VPC, you are by default limited to 5 elastic IP addresses. However it is easy to request for an increase by completing this form https://aws.amazon.com/support/createCase?type=service_limit_increase&serviceLimitIncreaseType=elastic-ips

Note that a single Elastic Network Interfaces (ENI) can have multiple secondary IP addresses, for example on a m1.small instance type, you can have up to 4 IPs, which in Linux would be the eth0, eth0:0, eth0:1 and eth0:2 interfaces.

There is also a limit on the number of ENIs and IPs for each instance type, see the documentation at:

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI

Asymmetric Routing

When you add a second ENI, the AWS documentation is missing a fundamental note on how to configure the instance O.S. for handling the network routes.

If you attach the second ENI, associate it with an Elastic IP and bring it up (with ifup) in Linux after adding to /etc/network/interfaces, your network will very likely be performing asymmetric routing. Try and ping the Elastic IP of eth1, you get no response. This is because the response packets leaving the instance do not get sent out via the correct gateway.

Asymmetric routing is explained in depth in this article http://www.linuxjournal.com/article/7291

Route configuration with additional ENIs

The fix is to add additional routes for the new ENIs. This guide assumes that so far you have followed this documentation for adding a second ENI http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#attach_eni_launch

We’re assuming the instance has an interface eth0 with the private address 10.0.1.10 from a 10.0.1.0/24 subnet and we want to add an ENI using a different subnet 10.0.2.0/24 with an IP address of 10.0.2.10

The /etc/network/interfaces file should look like this after adding eth1:

Then bring up eth1 interface:

Let’s check the route:

There is one default gateway at 10.0.1.1 (which is bound to VPC the internet gateway) and will route any traffic from eth0. However any traffic from eth1 with a destination outside of 10.0.2.0/24 will be dropped, so we need to re-configure the routing to the default gateway for the 10.0.2.0/24 subnet.

Firstly, add an entry “2 eth1_rt” to the route table:

Next we need to add a default route to the gateway for eth1:

Verify that the route is added:

Finally we need to add a rule which will tell the route table to route traffic with a source of 10.0.2.0/24 via the rt_eth1 table:

Verify that the rule is added:

Now from your machine, try and ping the Elastic IP associated with eth1 and it should now work, asymmetrical routing has been fixed !

To make the route changes permanent so that they can survive a reboot, add them to the interfaces file:

If you wish to associate an private IP from the 10.0.1.0/24 subnet to eth1 (same subnet as eth0 network), just replace the gateway and subnet values to 10.0.1.1 and 10.0.1.0/24 respectively.

9 thoughts on “Multiple IPs and ENIs on EC2 in a VPC

  1. Nice article explaining the secondary interface routing. Another important thing that Amazon web site doesn’t mention, at least none that I can find, is that configuring the secondary interface (on Linux) as static is a must and that is before you go and attach it to the instance. And that is because of the fact that without this preparation the new eth1 interface will get configured via dhcp as well, overwriting the default gateway of eth0. If eth1 has an ip of a private network, one without internet gateway, then it’s game over. You are left without ssh access to the instance even if you have EIP attached to both instances.
    I have witnessed this even in a situation when launching a new instance from the wizard and choosing to create eth0 on public and eth1 on private network.

  2. Ugh! I had seen you post on this earlier today when having problems attaching an additional ENI and then AWS’s support had me do a reboot and then WHAM! Now no ssh or anything. How did you go about resolving this?

    • Try a force dis associating of the second ENI from the instance then reboot it. Keep an eye on the system log via the console.

      If still no luck then you may need to remove any network configuration you added by mounting the root EBS on another running instance and edit the network interfaces then remount it on the instance which is stuck.

      Good luck!

  3. Thank alot !! ….It really worked, i wasted so much time on other portals but something or the other thing was missing there.

  4. I was curious if you ever thought of changing the page layout of your
    website? Its very well written; I love what youve got to say.
    But maybe you could a little more in the way of content so
    people could connect with it better. Youve
    got an awful lot of text for only having 1 or two
    images. Maybe you could space it out better?

Leave a Reply to Anabolika Cancel reply

Your email address will not be published. Required fields are marked *