Friday, October 30, 2009

wansimulator

About:
Are you thinking to setup a "wan-simulator" for testing application functionality and performance over-head of the wan(wide area network) within the lab environment, than this might be of interest to you .

Overview:

Any x86 (64/32bit) system can be used to host this. This system could run latest OpenSolaris, to download OpenSolaris visit http://genunix.org/ . Wan-simulator leverages on to project crossbow, it's a streams module/driver which emulates WAN environment. It captures packets from IP and manipulates the packets according to the WAN setup.

There are five parameters to control the environment.
  1. network propagation delay,
  2. bandwidth,
  3. drop rate,
  4. re-ordering, and
  5. corruption.
Hxbt acts like a "pipe" between two hosts. And the pipe is the emulated WAN.

Note that the current hxbt implementation only operates on the output side. Hxbt can be visualized as a emulated router on the outgoing path. And the returning path is a "direct" link. In normal circumstances, this asymmetry should not be a problem as long as the parameters are set properly to reflect WAN condition on BOTH paths. For example, if outgoing and incoming paths both introduce a 10ms delay, set hxbt to emulate a 20ms delay. If incoming and outgoing paths need to be emulated, set up hxbt in machines on both sides of the path.
For more details about the wan-simulator please visit OpenSolaris networking.

This case study is a typical example where one need to have simulated wan or have systems geographically apart to test the replication over long distance for disaster recovery testing. There could be other case studies like if we have to understand the impact of the AJAX/Javascripts running on the web-client or hand-held devices over long distance, we could setup such an environment to do the required testing within the comfort of one location, this eases out some of the challenges one would face to do such activity, like co-ordination over the long distance to ensure the required environment is sane as per the requirement etc.

In this example of disaster recovery scenario, where 2 sites are geographically apart around 1500 miles,bandwidth is 8Mbps, link is bad, simulating 5% packet drops and 5% corruption of data. There are multiple IP addresses on each side talking to each other over this simulated wan to do the data replication and monitor the health of each site and management of Inter Cluster Resource Management(ICRM) etc. Since these 2 sites are geographically apart, for these 2 sites on 2 different TCP/IP network this wan-simulator acts as router among both the sites, as well as provides the required functionality of wide area network by using hit-box/hxbt (wan-simulator). The above diagram attempts to show this environment.

If need to compute the latency etc, visit netqos or compute the required latency and understand packet drops and understand the data corruptions by manually observing it using ping and other sniffing tools.

Installation of wansimulator:


All you have to do is download OpenSolaris from http://www.genunix.org/. I had tried this on OpenSolaris b122, however you could download and try it out on the latest build too.

1. Download the pre-compiled binaries tar ball from this blog, or download the latest source code from http://opensolaris.org/os/community/networking/. Here I have used tar ball from this blog.

2. Unpack the tar ball, copy the pre-compiled binaries to the appropriate locations,
cd /extracted_directory
cp onnv/usr/src/uts/intel/hxbt/obj32/hxbt /kernel/drv
cp onnv/usr/src/uts/intel/hxbt/obj64/hxbt /kernel/drv/amd64
cp onnv/usr/src/uts/common/inet/hxbt/hxbt.conf /kernel/drv
cp onnv/usr/src/cmd/cmd-inet/usr.sbin/ifhit/ifhit /usr/sbin
add_drv -m"* 0666 root root" hxbt
How to verify it is installed?

Insert the hxbt module to verify the successful installation and configuration.
ifconfig nge0 modlist
ifconfig nge0 modinsert hxbt@2
ifconfig nge0 modlist

Un-install:

This is you would do, once you are done with wansimulator.

root@wansimulator:~#ifconfig nge0 modlist
0 arp
1 ip
2 hxbt
3 nge
root@wansimulator:~#ifconfig nge0 modremove hxbt@2
root@wansimulator:~#modinfo | grep hxbt
252 fffffffff7dc3000 2460 - 1 hxbt (hxbt stream module v1.1)
252 fffffffff7dc3000 2460 291 1 hxbt (hxbt stream driver v1.1)
root@wansimulator:~#modunload -i 252
root@wansimulator:~#pfexec rem_drv hxbt

Configure wan-simulator:

Pre-requisite
[optional]: Configure this OpenSolais wansimulator system as router. For this case study I did configure here are the steps to configure router.

Here are the network interfaces that this system hosts, which takes part in routing.

root@wansimulator:~# ifconfig nge0
nge0: flags=1100843 mtu 1500 index 2
inet 192.168.50.250 netmask ffffff00 broadcast 192.168.50.255
ether 0:21:28:44:67:a
root@wansimulator:~# ifconfig nge1
nge1: flags=1100943 mtu 1500 index 3
inet 10.10.50.250 netmask ffffff00 broadcast 10.10.50.255
ether 0:21:28:44:67:b
root@wansimulator:~#


---> ifhit without options lists all the targets that are configured to take a hit.

root@wansimulator:~# ifhit
hxbt target list:

root@wansimulator:~#


----> Lets look at the time to reach destination before configuring hit-box(wansimulator).
===> ping from primary network

root@pnode1:~# ping -s pnode1
PING pnode1: 56 data bytes
64 bytes from pnode1 (10.10.50.105): icmp_seq=0. time=0.178 ms
64 bytes from pnode1 (10.10.50.105): icmp_seq=1. time=0.0670 ms
^C
----pnode1 PING Statistics----
2 packets transmitted, 2 packets received, 0% packet loss
round-trip (ms) min/avg/max/stddev = 0.0670/0.122/0.178/0.078
root@pnode1:~# ping -s drnode1
PING drnode1: 56 data bytes
64 bytes from drnode1 (192.168.50.103): icmp_seq=0. time=0.443 ms
64 bytes from drnode1 (192.168.50.103): icmp_seq=1. time=0.389 ms
^C
----drnode1 PING Statistics----
2 packets transmitted, 2 packets received, 0% packet loss
round-trip (ms) min/avg/max/stddev = 0.389/0.416/0.443/0.038
root@pnode1:~#


===> ping from DR network

root@drnode1:~# ping -s drnode1
PING drnode1: 56 data bytes
64 bytes from drnode1 (192.168.50.103): icmp_seq=0. time=0.172 ms
64 bytes from drnode1 (192.168.50.103): icmp_seq=1. time=0.154 ms
^C
----drnode1 PING Statistics----
2 packets transmitted, 2 packets received, 0% packet loss
round-trip (ms) min/avg/max/stddev = 0.154/0.163/0.172/0.013
root@drnode1:~# ping -s pnode1
PING pnode1: 56 data bytes
64 bytes from pnode1 (10.10.50.105): icmp_seq=0. time=0.453 ms
64 bytes from pnode1 (10.10.50.105): icmp_seq=1. time=0.392 ms
^C
----pnode1 PING Statistics----
2 packets transmitted, 2 packets received, 0% packet loss
round-trip (ms) min/avg/max/stddev = 0.392/0.422/0.453/0.043
root@drnode1:~#


---> Following are the IP of the wan-simulator(hit-box) configured:
192.168.50.250 drrouter
10.10.50.250 prouter
root@wansimulator:~# ifconfig nge0
nge0: flags=1100843 mtu 1500 index 2
inet 192.168.50.250 netmask ffffff00 broadcast 192.168.50.255
ether 0:21:28:44:67:a
root@wansimulator:~# ifconfig nge1
nge1: flags=1100943 mtu 1500 index 3
inet 10.10.50.250 netmask ffffff00 broadcast 10.10.50.255
ether 0:21:28:44:67:b
root@wansimulator:~#

---> Following is the set IP addresses of both the networks which will be communicating among each other:

192.168.50.103 drnode1
192.168.50.104 drnode2
192.168.50.111 drcluster
192.168.50.112 dapprep
192.168.50.113 ddbrep
10.10.50.105 pnode1
10.10.50.106 pnode2
10.10.50.111 primarycluster
10.10.50.112 papprep
10.10.50.113 pdbrep

----> Configure all the above IP addresses as follows, to simulate 4Mbps link over a distance of 1000 miles which is would have 250millisecond delay and drops approx 5% packet, corrupt 5% of packets going to such that there are 2bytes in each packet are corrupted.

root@wansimulator:~# ifhit 10.10.50.105 -b 4096 -d 5 -c 5 -C 2 -l 250
(::ffff:10.10.50.105)
Bandwidth 524288 (bytes/s) (4096.000000 Kbps)
Drop rate = 5.00 %
Delay = 250 ms
Corruption rate = 5.00 %, corruption count = 2 bytes/packet
root@wansimulator:~#

====> Test the above by pinging to pnode1 (10.10.50.105) from drnode1.
root@drnode1:~# ping -s pnode1
PING pnode1: 56 data bytes
64 bytes from pnode1 (10.10.50.105): icmp_seq=0. time=247. ms
64 bytes from pnode1 (10.10.50.105): icmp_seq=1. time=248. ms
64 bytes from pnode1 (10.10.50.105): icmp_seq=2. time=248. ms
64 bytes from pnode1 (10.10.50.105): icmp_seq=3. time=248. ms
64 bytes from pnode1 (10.10.50.105): icmp_seq=4. time=248. ms
64 bytes from pnode1 (10.10.50.105): icmp_seq=6. time=248. ms
64 bytes from pnode1 (10.10.50.105): icmp_seq=7. time=248. ms
64 bytes from pnode1 (10.10.50.105): icmp_seq=8. time=248. ms
64 bytes from pnode1 (10.10.50.105): icmp_seq=9. time=248. ms
64 bytes from pnode1 (10.10.50.105): icmp_seq=10. time=248. ms
64 bytes from pnode1 (10.10.50.105): icmp_seq=11. time=248. ms
64 bytes from pnode1 (10.10.50.105): icmp_seq=12. time=248. ms
64 bytes from pnode1 (10.10.50.105): icmp_seq=13. time=248. ms
64 bytes from pnode1 (10.10.50.105): icmp_seq=14. time=248. ms
64 bytes from pnode1 (10.10.50.105): icmp_seq=15. time=248. ms
64 bytes from pnode1 (10.10.50.105): icmp_seq=16. time=248. ms
64 bytes from pnode1 (10.10.50.105): icmp_seq=17. time=248. ms
64 bytes from pnode1 (10.10.50.105): icmp_seq=18. time=248. ms
64 bytes from pnode1 (10.10.50.105): icmp_seq=19. time=248. ms
64 bytes from pnode1 (10.10.50.105): icmp_seq=20. time=248. ms
64 bytes from pnode1 (10.10.50.105): icmp_seq=21. time=248. ms
^C
----pnode1 PING Statistics----
23 packets transmitted, 21 packets received, 8% packet loss
round-trip (ms) min/avg/max/stddev = 247./247.9/248./0.262

root@drnode1:~# ping -s pnode1
PING pnode1: 56 data bytes
64 bytes from pnode1 (10.10.50.105): icmp_seq=0. time=250. ms
64 bytes from pnode1 (10.10.50.105): icmp_seq=1. time=248. ms
64 bytes from pnode1 (10.10.50.105): icmp_seq=2. time=248. ms
64 bytes from pnode1 (10.10.50.105): icmp_seq=3. time=248. ms
^C
----pnode1 PING Statistics----
4 packets transmitted, 4 packets received, 0% packet loss
round-trip (ms) min/avg/max/stddev = 248./248./250./1.3
root@drnode1:~#


===> Do the same thing to all the IP addresses.

root@wansimulator:~# ifhit 10.10.50.106 -b 4096 -d 5 -c 5 -C 2 -l 250
(::ffff:10.10.50.106)
Bandwidth 524288 (bytes/s) (4096.000000 Kbps)
Drop rate = 5.00 %
Delay = 250 ms
Corruption rate = 5.00 %, corruption count = 2 bytes/packet
root@wansimulator:~# ifhit 10.10.50.111 -b 4096 -d 5 -c 5 -C 2 -l 250
(::ffff:10.10.50.111)
Bandwidth 524288 (bytes/s) (4096.000000 Kbps)
Drop rate = 5.00 %
Delay = 250 ms
Corruption rate = 5.00 %, corruption count = 2 bytes/packet
root@wansimulator:~# ifhit 10.10.50.112 -b 4096 -d 5 -c 5 -C 2 -l 250
(::ffff:10.10.50.112)
Bandwidth 524288 (bytes/s) (4096.000000 Kbps)
Drop rate = 5.00 %
Delay = 250 ms
Corruption rate = 5.00 %, corruption count = 2 bytes/packet
root@wansimulator:~# ifhit 10.10.50.113 -b 4096 -d 5 -c 5 -C 2 -l 250
(::ffff:10.10.50.113)
Bandwidth 524288 (bytes/s) (4096.000000 Kbps)
Drop rate = 5.00 %
Delay = 250 ms
Corruption rate = 5.00 %, corruption count = 2 bytes/packet
root@wansimulator:~#
root@wansimulator:~# ifhit 192.168.50.103 -b 4096 -d 5 -c 5 -C 2 -l 250
(::ffff:192.168.50.103)
Bandwidth 524288 (bytes/s) (4096.000000 Kbps)
Drop rate = 5.00 %
Delay = 250 ms
Corruption rate = 5.00 %, corruption count = 2 bytes/packet
root@wansimulator:~# ifhit 192.168.50.104 -b 4096 -d 5 -c 5 -C 2 -l 250
(::ffff:192.168.50.104)
Bandwidth 524288 (bytes/s) (4096.000000 Kbps)
Drop rate = 5.00 %
Delay = 250 ms
Corruption rate = 5.00 %, corruption count = 2 bytes/packet
root@wansimulator:~# ifhit 192.168.50.111 -b 4096 -d 5 -c 5 -C 2 -l 250
(::ffff:192.168.50.111)
Bandwidth 524288 (bytes/s) (4096.000000 Kbps)
Drop rate = 5.00 %
Delay = 250 ms
Corruption rate = 5.00 %, corruption count = 2 bytes/packet
root@wansimulator:~# ifhit 192.168.50.112 -b 4096 -d 5 -c 5 -C 2 -l 250
(::ffff:192.168.50.112)
Bandwidth 524288 (bytes/s) (4096.000000 Kbps)
Drop rate = 5.00 %
Delay = 250 ms
Corruption rate = 5.00 %, corruption count = 2 bytes/packet
root@wansimulator:~# ifhit 192.168.50.113 -b 4096 -d 5 -c 5 -C 2 -l 250
(::ffff:192.168.50.113)
Bandwidth 524288 (bytes/s) (4096.000000 Kbps)
Drop rate = 5.00 %
Delay = 250 ms
Corruption rate = 5.00 %, corruption count = 2 bytes/packet
root@wansimulator:~#
root@wansimulator:~# ifhit
hxbt target list:
::ffff:10.10.50.105
::ffff:10.10.50.106
::ffff:192.168.50.103
::ffff:192.168.50.104
::ffff:10.10.50.111
::ffff:10.10.50.112
::ffff:10.10.50.113
::ffff:192.168.50.111
::ffff:192.168.50.112
::ffff:192.168.50.113

root@wansimulator:~#

===> After configuring all the IP address the delay is double

root@pnode1:~# ping -s drnode1
PING drnode1: 56 data bytes
64 bytes from drnode1 (192.168.50.103): icmp_seq=0. time=495. ms
64 bytes from drnode1 (192.168.50.103): icmp_seq=1. time=492. ms
64 bytes from drnode1 (192.168.50.103): icmp_seq=3. time=492. ms
64 bytes from drnode1 (192.168.50.103): icmp_seq=4. time=493. ms
64 bytes from drnode1 (192.168.50.103): icmp_seq=5. time=492. ms
64 bytes from drnode1 (192.168.50.103): icmp_seq=6. time=492. ms
64 bytes from drnode1 (192.168.50.103): icmp_seq=8. time=492. ms
^C
----drnode1 PING Statistics----
9 packets transmitted, 7 packets received, 22% packet loss
round-trip (ms) min/avg/max/stddev = 492./493./495./0.95
root@pnode1:~#
===> ping on the same network no packet drops.
root@pnode1:~# ping -s pnode2
PING pnode2: 56 data bytes
64 bytes from pnode2 (10.10.50.106): icmp_seq=0. time=0.991 ms
64 bytes from pnode2 (10.10.50.106): icmp_seq=1. time=0.143 ms
64 bytes from pnode2 (10.10.50.106): icmp_seq=2. time=0.166 ms
64 bytes from pnode2 (10.10.50.106): ^C
----pnode2 PING Statistics----
4 packets transmitted, 4 packets received, 0% packet loss
round-trip (ms) min/avg/max/stddev = 0.143/0.363/0.991/0.42
root@pnode1:~#

Conclusion:
Since we configure 4Mbps one way, it's like send & receive will get 8Mbps. Hence one should do appropriate computation to simulate the required network.
For instance, if you want to configure 8 Mbps link, split that into two halves for each way.

References:
OpenSolaris
Project crossbow
readme on hit-box (wan-simulator from project page)
Network community has reference to hxbt.

Acknowledgment:
I would like to thank Sunay Tripathi and Kais Belgaied for their support to help me understand the working of the wan-simualtor and creating it.

FreeRADIUS with MySQL cluster

About: This is all about deploying FreeRADIUS with MySQL cluster , understand about FreeRADIUS deployment options with MySQL cluster for h...