Back in 2012, Amazon Web Services launched support for multiple private IP addresses for EC2 instances, within a VPC.
This is particularly useful if you host several SSL websites on a single EC2 instance, as each SSL certificate must be hosted on it’s own (private) IP address. Then you can associate the private IP address with an Elastic IP address to make the SSL website accessible from the internet.
Multiple IPs and Limits
This AWS blog entry briefly describe the multiple IPs management: http://aws.typepad.com/aws/2012/07/multiple-ip-addresses-for-ec2-instances-in-a-virtual-private-cloud.html
When you create a VPC, you are by default limited to 5 elastic IP addresses. However it is easy to request for an increase by completing this form https://aws.amazon.com/support/createCase?type=service_limit_increase&serviceLimitIncreaseType=elastic-ips
Note that a single Elastic Network Interfaces (ENI) can have multiple secondary IP addresses, for example on a m1.small instance type, you can have up to 4 IPs, which in Linux would be the eth0, eth0:0, eth0:1 and eth0:2 interfaces.
There is also a limit on the number of ENIs and IPs for each instance type, see the documentation at:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI
Asymmetric Routing
When you add a second ENI, the AWS documentation is missing a fundamental note on how to configure the instance O.S. for handling the network routes.
If you attach the second ENI, associate it with an Elastic IP and bring it up (with ifup) in Linux after adding to /etc/network/interfaces, your network will very likely be performing asymmetric routing. Try and ping the Elastic IP of eth1, you get no response. This is because the response packets leaving the instance do not get sent out via the correct gateway.
Asymmetric routing is explained in depth in this article http://www.linuxjournal.com/article/7291
Route configuration with additional ENIs
The fix is to add additional routes for the new ENIs. This guide assumes that so far you have followed this documentation for adding a second ENI http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#attach_eni_launch
We’re assuming the instance has an interface eth0 with the private address 10.0.1.10 from a 10.0.1.0/24 subnet and we want to add an ENI using a different subnet 10.0.2.0/24 with an IP address of 10.0.2.10
The /etc/network/interfaces file should look like this after adding eth1:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# The loopback network interface auto lo iface lo inet loopback # The primary network interface auto eth0 iface eth0 inet dhcp auto eth1 iface eth1 inet static address 10.0.2.10 netmask 255.255.255.0 network 10.0.2.0 broadcast 10.0.2.255 |
Then bring up eth1 interface:
|
1 |
$ sudo ifup eth1 |
Let’s check the route:
|
1 2 3 4 |
$ ip route show default via 10.0.1.1 dev eth0 metric 100 10.0.1.0/24 dev eth0 proto kernel scope link src 10.0.1.10 10.0.2.0/24 dev eth1 proto kernel scope link src 10.0.2.10 |
There is one default gateway at 10.0.1.1 (which is bound to VPC the internet gateway) and will route any traffic from eth0. However any traffic from eth1 with a destination outside of 10.0.2.0/24 will be dropped, so we need to re-configure the routing to the default gateway for the 10.0.2.0/24 subnet.
Firstly, add an entry “2 eth1_rt” to the route table:
|
1 |
$ sudo bash -c "echo '2 eth1_rt' >> /etc/iproute2/rt_tables" |
Next we need to add a default route to the gateway for eth1:
|
1 |
$ sudo ip route add default via 10.0.2.1 dev eth1 table eth1_rt |
Verify that the route is added:
|
1 2 |
$ ip route show table eth1_rt default via 10.0.2.1 dev eth1 |
Finally we need to add a rule which will tell the route table to route traffic with a source of 10.0.2.0/24 via the rt_eth1 table:
|
1 |
$ sudo ip rule add from 10.0.2.0/24 lookup eth1_rt prio 1000 |
Verify that the rule is added:
|
1 2 3 4 5 |
$ ip rule show 0: from all lookup local 1000: from 10.0.2.0/24 lookup eth1_rt 32766: from all lookup main 32767: from all lookup default |
Now from your machine, try and ping the Elastic IP associated with eth1 and it should now work, asymmetrical routing has been fixed !
To make the route changes permanent so that they can survive a reboot, add them to the interfaces file:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# The loopback network interface auto lo iface lo inet loopback # The primary network interface auto eth0 iface eth0 inet dhcp # The second network interface auto eth1 iface eth1 inet static address 10.0.2.10 netmask 255.255.255.0 network 10.0.2.0 broadcast 10.0.2.255 up ip route add default via 10.0.2.1 dev eth1 table eth1_rt up ip rule add from 10.0.2.0/24 lookup eth1_rt prio 1000 |
If you wish to associate an private IP from the 10.0.1.0/24 subnet to eth1 (same subnet as eth0 network), just replace the gateway and subnet values to 10.0.1.1 and 10.0.1.0/24 respectively.
Nice article explaining the secondary interface routing. Another important thing that Amazon web site doesn’t mention, at least none that I can find, is that configuring the secondary interface (on Linux) as static is a must and that is before you go and attach it to the instance. And that is because of the fact that without this preparation the new eth1 interface will get configured via dhcp as well, overwriting the default gateway of eth0. If eth1 has an ip of a private network, one without internet gateway, then it’s game over. You are left without ssh access to the instance even if you have EIP attached to both instances.
I have witnessed this even in a situation when launching a new instance from the wizard and choosing to create eth0 on public and eth1 on private network.
Ugh! I had seen you post on this earlier today when having problems attaching an additional ENI and then AWS’s support had me do a reboot and then WHAM! Now no ssh or anything. How did you go about resolving this?
Try a force dis associating of the second ENI from the instance then reboot it. Keep an eye on the system log via the console.
If still no luck then you may need to remove any network configuration you added by mounting the root EBS on another running instance and edit the network interfaces then remount it on the instance which is stuck.
Good luck!
Will these instruction work for RedHat 5.9 /etc/network/interfaces does not exist.
Nice article explaining the secondary interface routing.
Finally a working solution.
Than you, good man!
Thank alot !! ….It really worked, i wasted so much time on other portals but something or the other thing was missing there.
It’s an remarkable post designed for all the web users; they will obtain advantage from it I
am sure.
I was curious if you ever thought of changing the page layout of your
website? Its very well written; I love what youve got to say.
But maybe you could a little more in the way of content so
people could connect with it better. Youve
got an awful lot of text for only having 1 or two
images. Maybe you could space it out better?
If you desire to get much from this article then you have to apply
such methods to your won blog.
This article offers clear idea for the new viewers of blogging, that in fact how
to do blogging and site-building.
Pretty component of content. I simply stumbled upon your
website and in accession capital to say that I get
actually enjoyed account your blog posts. Any way I will be subscribing to your augment or even I
success you get entry to consistently fast.
Hi there to every body, it’s my first pay a visit of this web site; this weblog
contains remarkable and truly good stuff for visitors.
Wedding venues play a pivotal role in the vibrant city
of Las Vegas, Nevada, where couples flock from around
the world to tie the knot. From extravagant
ceremonies to intimate gatherings, the choice of wedding location sets the
tone for one of life’s most memorable events. With a plethora of options ranging from outdoor garden settings to
elegant banquet halls, selecting the perfect venue
is essential for creating the wedding of your dreams.
Nestled in the heart of Las Vegas, Lotus House Events offers couples a picturesque backdrop for their
special day. Founded in the same year as the city itself, Lotus House Events is steeped
in history and tradition, mirroring the dynamic spirit of Las Vegas.
With a population of 646,790 residents and over
832,367 households, Las Vegas is a melting pot of diverse
cultures and communities. Interstate 11 traverses the city, providing convenient
access to neighboring areas and attractions.
In a city known for its extreme temperatures, ranging from
scorching summers to mild winters, home repairs
are a constant consideration for residents. Whether it’s air conditioning maintenance to beat the summer heat or roofing repairs to
withstand occasional rainfall, homeowners understand the importance of
budgeting for these expenses. On average, repairs typically
range from a few hundred to several thousand dollars, depending
on the nature of the work required and the contractor hired.
Exploring the vibrant tapestry of Las Vegas’s attractions, residents and visitors alike are spoiled for choice.
From the whimsical wonders of AREA15 to the serene
beauty of Aliante Nature Discovery Park, there’s something for everyone to enjoy.
Thrill-seekers can brave the Asylum-Hotel Fear Haunted House, while art enthusiasts can marvel at the exhibits in the Arts District.
History buffs can delve into the Atomic Museum’s intriguing displays, while families
can create lasting memories at the Discovery Children’s Museum.
Choosing Lotus House Events as your wedding venue in Las Vegas ensures a seamless and
unforgettable experience for you and your guests.
With a variety of indoor and outdoor spaces to accommodate weddings
of all sizes and styles, Lotus House Events offers unparalleled flexibility and customization options.
From expert wedding planning services to exquisite catering and decor, every detail is meticulously curated to bring your vision to life.
With convenient packages and availability, Lotus House Events takes the stress out of wedding
planning, allowing you to focus on creating cherished memories that will last a
lifetime.