Back in 2012, Amazon Web Services launched support for multiple private IP addresses for EC2 instances, within a VPC.
This is particularly useful if you host several SSL websites on a single EC2 instance, as each SSL certificate must be hosted on it’s own (private) IP address. Then you can associate the private IP address with an Elastic IP address to make the SSL website accessible from the internet.
Multiple IPs and Limits
This AWS blog entry briefly describe the multiple IPs management: http://aws.typepad.com/aws/2012/07/multiple-ip-addresses-for-ec2-instances-in-a-virtual-private-cloud.html
When you create a VPC, you are by default limited to 5 elastic IP addresses. However it is easy to request for an increase by completing this form https://aws.amazon.com/support/createCase?type=service_limit_increase&serviceLimitIncreaseType=elastic-ips
Note that a single Elastic Network Interfaces (ENI) can have multiple secondary IP addresses, for example on a m1.small instance type, you can have up to 4 IPs, which in Linux would be the eth0, eth0:0, eth0:1 and eth0:2 interfaces.
There is also a limit on the number of ENIs and IPs for each instance type, see the documentation at:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI
Asymmetric Routing
When you add a second ENI, the AWS documentation is missing a fundamental note on how to configure the instance O.S. for handling the network routes.
If you attach the second ENI, associate it with an Elastic IP and bring it up (with ifup) in Linux after adding to /etc/network/interfaces, your network will very likely be performing asymmetric routing. Try and ping the Elastic IP of eth1, you get no response. This is because the response packets leaving the instance do not get sent out via the correct gateway.
Asymmetric routing is explained in depth in this article http://www.linuxjournal.com/article/7291
Route configuration with additional ENIs
The fix is to add additional routes for the new ENIs. This guide assumes that so far you have followed this documentation for adding a second ENI http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#attach_eni_launch
We’re assuming the instance has an interface eth0 with the private address 10.0.1.10 from a 10.0.1.0/24 subnet and we want to add an ENI using a different subnet 10.0.2.0/24 with an IP address of 10.0.2.10
The /etc/network/interfaces file should look like this after adding eth1:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# The loopback network interface auto lo iface lo inet loopback # The primary network interface auto eth0 iface eth0 inet dhcp auto eth1 iface eth1 inet static address 10.0.2.10 netmask 255.255.255.0 network 10.0.2.0 broadcast 10.0.2.255 |
Then bring up eth1 interface:
|
1 |
$ sudo ifup eth1 |
Let’s check the route:
|
1 2 3 4 |
$ ip route show default via 10.0.1.1 dev eth0 metric 100 10.0.1.0/24 dev eth0 proto kernel scope link src 10.0.1.10 10.0.2.0/24 dev eth1 proto kernel scope link src 10.0.2.10 |
There is one default gateway at 10.0.1.1 (which is bound to VPC the internet gateway) and will route any traffic from eth0. However any traffic from eth1 with a destination outside of 10.0.2.0/24 will be dropped, so we need to re-configure the routing to the default gateway for the 10.0.2.0/24 subnet.
Firstly, add an entry “2 eth1_rt” to the route table:
|
1 |
$ sudo bash -c "echo '2 eth1_rt' >> /etc/iproute2/rt_tables" |
Next we need to add a default route to the gateway for eth1:
|
1 |
$ sudo ip route add default via 10.0.2.1 dev eth1 table eth1_rt |
Verify that the route is added:
|
1 2 |
$ ip route show table eth1_rt default via 10.0.2.1 dev eth1 |
Finally we need to add a rule which will tell the route table to route traffic with a source of 10.0.2.0/24 via the rt_eth1 table:
|
1 |
$ sudo ip rule add from 10.0.2.0/24 lookup eth1_rt prio 1000 |
Verify that the rule is added:
|
1 2 3 4 5 |
$ ip rule show 0: from all lookup local 1000: from 10.0.2.0/24 lookup eth1_rt 32766: from all lookup main 32767: from all lookup default |
Now from your machine, try and ping the Elastic IP associated with eth1 and it should now work, asymmetrical routing has been fixed !
To make the route changes permanent so that they can survive a reboot, add them to the interfaces file:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# The loopback network interface auto lo iface lo inet loopback # The primary network interface auto eth0 iface eth0 inet dhcp # The second network interface auto eth1 iface eth1 inet static address 10.0.2.10 netmask 255.255.255.0 network 10.0.2.0 broadcast 10.0.2.255 up ip route add default via 10.0.2.1 dev eth1 table eth1_rt up ip rule add from 10.0.2.0/24 lookup eth1_rt prio 1000 |
If you wish to associate an private IP from the 10.0.1.0/24 subnet to eth1 (same subnet as eth0 network), just replace the gateway and subnet values to 10.0.1.1 and 10.0.1.0/24 respectively.
