Setting Up Your Ubuntu Environment for Wallaroo
These instructions have been tested for Ubuntu Artful, Bionic, Trusty, and Xenial releases.
There are a few applications/tools which are required to be installed before you can proceed with the setup of the Wallaroo environment.
In order to compile the Wallaroo example applications, your system will need to have approximately 3 GB working memory (this can be RAM or swap). If you don't have enough memory, you are likely to see that the compile process is
Killed by the OS.
Ensures you'll have the latest available packages:
sudo apt-get update
If you do not already have Git installed, install it:
sudo apt-get install git
Set up Environment for the Wallaroo Tutorial
If you haven't already done so, create a directory called
~/wallaroo-tutorial and navigate there by running:
cd ~/ mkdir ~/wallaroo-tutorial cd ~/wallaroo-tutorial
This will be our base directory in what follows. Download the Wallaroo sources (this will create a subdirectory called
curl -L -o wallaroo-0.6.0.tar.gz 'https://wallaroo-labs.bintray.com/wallaroolabs-ftp/wallaroo/0.6.0/wallaroo-0.6.0.tar.gz' mkdir wallaroo-0.6.0 tar -C wallaroo-0.6.0 --strip-components=1 -xzf wallaroo-0.6.0.tar.gz rm wallaroo-0.6.0.tar.gz cd wallaroo-0.6.0
sudo apt-get update sudo apt-get install -y build-essential
sudo apt-get install -y libssl-dev
Install GCC 4.7 or Higher
If you have at least GCC 4.7 installed on your machine, you don't need to do anything. If you have gcc 4.6 or lower, you'll need to upgrade. You can check your
gcc version by running:
To upgrade to GCC 5 if you don't have at least GCC 4.7:
sudo apt-get install software-properties-common
sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
sudo apt-get update sudo apt-get install -y gcc-5 g++-5
gcc-5 as the default option:
sudo update-alternatives --install /usr/bin/gcc gcc \ /usr/bin/gcc-5 60 --slave /usr/bin/g++ g++ /usr/bin/g++-5
Add ponyc and pony-stable apt-key keyserver
In order to install
apt-get the following keyserver must be added to the APT key management utility.
sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys E04F0923 B3B48BDA
The following packages need to be installed to allow
apt to use a repository over HTTPS:
sudo apt-get install \ apt-transport-https \ ca-certificates \ curl \ gnupg2 \ software-properties-common
Now you need to install Pony compiler
sudo add-apt-repository "deb https://dl.bintray.com/pony-language/ponylang-debian $(lsb_release -cs) main" sudo apt-get update sudo apt-get -V install ponyc=0.25.0
Next, you need to install
pony-stable, a Pony dependency management library. Navigate to a directory where you will put the
pony-stable repo and execute the following commands:
sudo add-apt-repository "deb https://dl.bintray.com/pony-language/ponylang-debian $(lsb_release -cs) main" sudo apt-get update sudo apt-get -V install pony-stable
Install Compression Development Libraries
Wallaroo's Kakfa support requires
liblz to be installed.
Xenial Ubuntu and later:
sudo apt-get install -y libsnappy-dev liblz4-dev
sudo apt-get install -y libsnappy-dev
Trusty Ubuntu has an outdated
liblz4 package. You will need to install from source like this:
cd ~/ curl -L -o liblz4-1.7.5.tar.gz https://github.com/lz4/lz4/archive/v1.7.5.tar.gz tar zxvf liblz4-1.7.5.tar.gz cd lz4-1.7.5 make sudo make install
Install Python Development Libraries
sudo apt-get install -y python python-dev
sudo apt-get install -y python3 python3-dev
Download and configure the Metrics UI
cd ~/wallaroo-tutorial/wallaroo-0.6.0/ mkdir bin curl -L -o Wallaroo_Metrics_UI-0.6.0-x86_64.AppImage 'https://wallaroo-labs.bintray.com/wallaroolabs-ftp/wallaroo/0.6.0/Wallaroo_Metrics_UI-0.6.0-x86_64.AppImage' chmod +x Wallaroo_Metrics_UI-0.6.0-x86_64.AppImage ./Wallaroo_Metrics_UI-0.6.0-x86_64.AppImage --appimage-extract mv squashfs-root bin/metrics_ui sed -i 's/sleep 4/sleep 0/' bin/metrics_ui/AppRun rm Wallaroo_Metrics_UI-0.6.0-x86_64.AppImage ln -s metrics_ui/AppRun bin/metrics_reporter_ui
Machida is the program that runs Wallaroo Python applications. Machida supports resiliency, but it comes with a performance hit. If you don't need resiliency then we recommend to build Machida without the flag. Here we build Machida once with resiliency and once without resiliency. Change to the Wallaroo directory. If you are using Python 2.7 you should run the following commands:
cd ~/wallaroo-tutorial/wallaroo-0.6.0/ make build-machida-all resilience=on cp machida/build/machida bin/machida-resilience make clean-machida-all make build-machida-all cp machida/build/machida bin cp -r machida/lib/ bin/pylib
If you are using Python 3.X you should run the following commands:
cd ~/wallaroo-tutorial/wallaroo-0.6.0/ make build-machida3-all resilience=on cp machida3/build/machida3 bin/machida3-resilience make clean-machida3-all make build-machida3-all cp machida3/build/machida3 bin cp -r machida/lib/ bin/pylib
Compiling Giles Sender, Data Receiver, Cluster Shutdown, and Cluster Shrinker tools
Giles Sender is used to supply data to Wallaroo applications over TCP, and Data Receiver is used as a fast TCP Sink that optionally writes the messages it receives to the console. The two together are useful when developing and testing applications that use TCP Sources and a TCP Sink.
The Cluster Shutdown tool is used to instruct the cluster to shutdown cleanly, clearing away any resilience and recovery files it may have created.
The Cluster Shrinker tool is used to tell a running cluster to reduce the number of workers in the cluster and to query the cluster for information about how many workers are eligible for removal.
To compile all of the tools, change to the root Wallaroo directory:
cd ~/wallaroo-tutorial/wallaroo-0.6.0/ make build-giles-sender-all build-utils-all cp utils/data_receiver/data_receiver bin cp utils/cluster_shrinker/cluster_shrinker bin cp utils/cluster_shutdown/cluster_shutdown bin cp giles/sender/sender bin
Set up activate file for setting environment variables
activate file sets up the environment for running Wallaroo examples when it is sourced
The following command copies it into the correct location:
cp misc/activate bin/
Awesome! All set. Time to try running your first application.