netSLS combines the Hadoop Scheduler Load Simulator (SLS) and the network emulator MaxiNet to emulate Hadoop network traffic based on artificial or real world job traces.
Download and start the MaxiNet virtual machine images from http://maxinet.github.io. The VMs subsequently will be referred to by their hostnames, "worker1" and "worker2".
First, upgrade Pyro4 on both VMs:
sudo pip install --upgrade Pyro4
Before proceeding, follow the instructions on http://maxinet.github.io/#quickstart to start and test your MaxiNet setup.
Now install netSLS on worker1:
sudo apt-get update
sudo apt-get install openjdk-7-jdk maven virtualenv libzmq-dev python-dev
git clone https://github.com/MaxiNet/netSLS.git
cd netSLS
./setup.sh
The netSLS network emulator requires ssh access to all MaxiNet workers. Add the following content to ~/.ssh/config
Host worker2
HostName 192.168.0.2
and distribute an ssh key to all workers:
ssh-keygen -f ~/.ssh/id_rsa -N ""
ssh-copy-id worker1
ssh-copy-id worker2
Start the network emulator:
cd ~/netSLS
./network_emulator.sh
You might want to check if the network emulator works properly first:
cd ~/netSLS/network_emulator
env/bin/python test/test_all_rpc.py
Now start sls with a demo trace:
cd ~/netSLS
./sls.sh generator/medium.json
The MaxiNet API allows to run the network emulator and sls on hosts different from the MaxiNet workers. Assuming the MaxiNet setup from above and the netSLS repository is cloned into ~/netSLS on the hosts running the network emulator and sls.
- Enable ssh access to all workers. Add the following entries to ~/.ssh/config (adapt addresses accordingly)
Host worker1
User maxinet
HostName a.b.c.d
Host worker2
User maxinet
HostName a.b.c.e
and copy the ssh key to all workers.
- Copy the MaxiNet configuration:
scp worker1:/etc/MaxiNet.cfg ~/netSLS/network_emulator/MaxiNet.cfg
Make sure, that the addresses in MaxiNet.cfg match the addresses in ~/.ssh/config. Otherwise modify the workers' and the copied MaxiNet.cfg file to match these addresses. MaxiNet needs to be restarted at this point.
- Create a python virtualenv and install dependencies (might require additional packages to be installed, cf. Quickstart):
cd ~/netSLS/network_emulator
./setup_virtualenv.sh
- Start the network emulator
cd ~/netSLS
./network_emulator.sh
- Build sls:
cd ~/netSLS/sls
mvn compile
-
If sls runs on a host different from the host running the network emulator the configuration needs to be changed. In ~/netSLS/sls/hadoop/etc/hadoop/sls-runner.xml change the properties
yarn.sls.networkemulator.zmq.url
andyarn.sls.networkemulator.rpc.url
to match the network emulator's address. -
Start sls
cd ~/netSLS
./sls.sh generator/medium.json
When modifying sls using an IDE, it is recommended to use IntelliJ IDEA rather than Eclipse (due to problems with the Eclipse Maven plugin).
Simply import the sls folder as Maven project and add the folder hadoop/etc/hadoop to the project's classpath.
netSLS may fail due to various reasons. Most likely there is a version mismatch between the system's and network emulator's versions of MaxiNet or Pyro4.
- Update system's MaxiNet:
cd ~/MaxiNet
git pull
sudo make install
- Update system's Pyro4:
sudo pip install --upgrade Pyro4
- Update network emulator dependencies:
cd ~/netSLS/network_emulator
./setup_virtualenv.sh