IP Multipathing (IPMP) in Sun Solaris enables the load balancing capabilities and resilience for Network connections with multple Network Interface Cards (NIC).
I discussed here about providing resilience for network connections with multiple NICs on the system. Now, we take it to the next step and make the network connections not only resilient but also load balance the network connections such that both the NICs participating in IPMP are active and forwards traffic. This improves the network throughput and thereby efficiency of the server especially if it is a critical system serving multiple connections.
To configure IPMP for load balancing we need to have
1. Two Virtual IP Addresses. These IPs are used by the Applications for data.
2. Test IP Address for each NIC. These IPs are not used by applications and are only used to prode a remote target device to check connectivity.
3. Each Interface has unique MAC-Address. By default in SPARC platforms, all NICs have a System-wide MAC-Address assigned and so they share a single MAC-Address. To change this behaviour click here
The NICs doesn’t have to be of the same kind but have to be of the same speed (10/100/1000Mbps).
In our configuration,
192.168.1.99 – Virtual IP1
192.168.1.100 – Virtual IP2
192.168.1.101 – Test IP for ce0 (NIC1)
192.168.1.102 – Test IP for ce1 (NIC2)
appserver – Actual hostname
appserver-1 – Hostname for Data IP2
appserver-2 – Hostname for Data IP2
appserver-ce0 – Hostname for test IP on ce0 interface
appserver-ce1 – Hostname for test IP on ce1 interface
Add Host Entries in /etc/hosts
Let’s start with adding the hosts entries for the IP addresses in the /etc/hosts file.
# IPMP group appserver-ipmp
127.0.0.1 localhost
192.168.1.99 appserver-1 loghost
192.168.1.100 appserver-2 appserver loghost
192.168.1.101 appserver-ce0 loghost
192.168.1.102 appserver-ce1 loghost
We have configured a hostname for each of the Virtual IPs and the Test IPs. However, the Test IPs should not be used by applications for any network connections.
Create hostname.ce* files
For every interface on the system create a hostname.ce* file. For us, create the files
hostname.ce0 & hostname.ce1
Edit hostname.ce0
Add the following on the hostname.ce0 file. This is the primary or master interface of the IPMP Pair
appserver-ce0 netmask + broadcast + group appserver-ipmp deprecated -failover up \
addif appserver netmask + broadcast + failover up
Edit hostname.ce1
Add the following on the hostname.ce1 file. This is the secondary or slave interface of the IPMP Pair
appserver-ce1 netmask + broadcast + group appserver-ipmp deprecated -failover up \
addif appserver netmask + broadcast + failover up
where
netmask – assigns the default netmask
broadcast – assigns the default broadcast value
group – specifies the IPMP group
deprecated – indicate test Interface not be used for data transfer
-failover – makes the test interface not to failover
Now, the configuration is complete and an ifconfig output should look like as follows:
root@ appserver:/$
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
ce0: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
inet 192.168.1.99 netmask ffffff00 broadcast 192.168.1.255
groupname appserver-ipmp
ether 0:xx:xx:xx:xx:x
ce0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 192.168.1.101 netmask ffffff00 broadcast 192.168.1.255
ce1: flags=69040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
inet 192.168.1.100 netmask ffffff00 broadcast 192.168.1.255
groupname appserver-ipmp
ether 0:xx:xx:xx:xx:x
ce1:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4inet 192.168.1.102 netmask ffffff00 broadcast 192.168.1.255
Now, both the NICs will forward traffic and when one of the inerface fails, it transparently failover the virtual IP address onto the other active interface and you can see an interface “ce1:2” would be created for the failed over IP. When the link is restored, this will be failed back to the ce0 interface. There should be no disruption to the network connections.
We written; very informative.
I’m trying to setup IPMP with load balancing across 4 bge interfaces on my jumpstart server to provide better throughput across my networks; We routinely jumpstart 2-4 different systems compised of 30-70 servers and workstations each. It get s bit slow at times. So, I tried this today, and much to my suprise, it didn’t work; I tried it as specified above using only two interfacers (bge0 and bge1), configured as depicted. On reboot, I could ping my gateway address and external systems (2 different subnets), but couldn’t ping or ssh/telnet to any systems on the same subnet. When I detached the bge1 network cable, I could still ping the router and ping/ssh/telnet to external systems, but not to systems within the same subnet; when I detached bge0 I could not connect to anything, and I got an error both internal to my subnet and external indicating the system wasn’t reacable though the bge1 interface. I verified the /etc/defaultrouter was correct, verified the routing tables (netstat -r), and created /etc/notrouter (as suggested by some), but test results made the same.
Your hostname.ce0 & ce1 files both attempt to add the exact same virtual interface:
“addif appserver netmask + broadcast + failover up”
How can that work?
Also you don’t show any running of the ifconfig command to actually configuer the interfaces, so are we to assume you reboot after creating the hostname.ce0 & ce1 files?
Hi
I haven’t understood how to form a group , If I have 4 NIC with single Mac Address i.e. Hme 0 ,1, 2, 3.
If possible please reply me How to form a group to achieve Fault tolerance and Load Spreading
whatever if fail over is ok for me but major requirement is load spreading in solaris.
I haven’t try it and waiting for forming group first then I can test load spreading and fault tolerance.
waiting for reply
I really doubt this configuration. I have implemented IPMP on several hundrads, but the configuration shown above for loadbancing is not convincing.
Nicely done. There are two typos in your config. The hostname.[interfaces] file should read addif appserver-0[1-2]. The ifconfig output is correct though.
This config would provide outbound balancing and inbound failover. For inbound failover, you need to look at Sun trunking (aka etherchannel). You will lose failover with trunking unless you setup another interface with the trunk in an IPMP group (aka 3 interfaces, 2 trunked, 1 not).
Thanks,
Eric
I want to use IPMP for failover but do not want traffic load balanced between the interfaces. I want an “all or nothing” failover of all traffic from the primary to the secondary using link-based IPMP on Solaris 10 to different switches for different interfaces. Etherchannel trunking seems to want to work properly only when both interfaces go to the same switch.
Maybe IPMP isn’t the best way to achieve this and an alternative is available?
I found what I was looking for. For example, in the configuration above:
appserver-ce1 netmask + broadcast + group appserver-ipmp deprecated -failover up
becomes
appserver-ce1 netmask + broadcast + group appserver-ipmp deprecated -failover standby up
The addition of the “standby” keyword causes Solaris not to use the second interface for outbound traffic unless the primary interface fails. Same is true for probe-based or link-based failover.
If your servers are directly connected to (for example) a L2/L3 switch doing NAT, having traffic egress from both interfaces can cause problems with CPU and CAM on the switch. It can greatly increase the CPU consumption when doing NAT as ingress traffic will cause CPU to store a flow state for NAT in CAM for the switch egress port to the server and return traffic on a different port causes the CPU to create a new flow state in CAM for doing the NAT of the return traffic on a different port. So now you have one flow consuming two CAM entries.
Also, systems, application, *and* network people must be aware that both paths are being used for outbound traffic. Else troubleshooting can be quite an “interesting” experience for everyone involved.
The above solution is good.i have similar requirement. i want to make trunking with 4 interface cards on solaris10 in sun machine.
Could guys please help me on this.
anyone can tell me about IPMP in Solaris 10
Nice Artilce, Keep Update Bro
Two suggestions for clarity on this article.
1. You show “+” symbols in your config file, but I think you aren’t supposed to enter “+” symbols at all.
2. You have a line
deprecated -failover up \
I suppose the \ means a line continuation. I kind of suspect it’s wrong, but regardless if it is or not:
Please, just show the config file with 2 whole lines, and no line continuations. There is room for it. So it should be done.
Regards,
Sam
Well described, … but how the load balancing logic believed to came? I mean SOMEONE can explain to me the trick behind these configuration. In what CONCEPT it does LOAD-BALANCING…thx for ur comming assistance. ANYWAY GOOD GUIDE Keep it up bro.
Nice stuff bro,i am going to configure the same on my test environment.