Giles components act as external testers for Wallaroo. Giles Receiver is used to mimic the behavior of an external sink for outgoing data from Wallaroo. Giles Receiver has the option to run in one of two ways:
- With no other commandline options given beside the listen address, it will write incoming binary data to a file named
received.txt, with each message preceded by a timestamp of when the message was received.
- With the
--no-write/-wflag given as a commandline argument, it will drop all incoming binary data and not write to file. This option is useful when benchmarking application speed, where the sink could potentially be a bottleneck.
If you do not already have the
wallaroo repo, create a local copy of the repo:
git clone https://github.com/WallarooLabs/wallaroo
cd wallaroo/giles/receiver make
Command Line Arguments
giles/receiver takes several command line arguments, below are a list with accompanying notes:
--listen/-laddress to listen on for incoming data from Wallaroo. Must be given in the
--no-write/-wflag to drop receive and drop incoming data.
--expect/-eterminate the receiver after an expected number of messages is received.
--metrics/-mCompute throughput metrics at the end of a run (requires
Listen for Wallaroo output and save to
receiver --listen 127.0.0.1:5555 --ponythreads=1 --ponynoblock
Listen for Wallaroo output, but don't save anything (e.g. "A Very Fast Sink Receiver")
If you just want your application to run as fast as possible without spending any resources on saving output data, use
receiver --listen 127.0.0.1:5555 --no-write --ponythreads=1 --ponynoblock
Pony Runtime Options
The following pony runtime parameters may also be useful in some situations
--ponythreadslimits the number of threads the pony runtime will use.
A common usecase is limiting
giles/receiver to one thread, noblock, and pinning asio:
receiver --listen 127.0.0.1:5555 --ponythreads=1 --ponynoblock --ponypinasio
giles/receiver vs. netcat
One question that comes up often is why not just use netcat for a output receiver. The reason is two-fold:
- If the giles sender and receiver are run on the same physical machine, then their monotonic unadjusted clock is synchronised, and therefore the timestamps from
giles/receivercan be used to compute total latencies, assuming one can relate a message in
sent.txtto a message in
- While doing performance tuning, we noticed that netcat sometimes struggles to handle heavy loads. This is understandable, as it was never designed with that purpose.
giles/receiver, on the other hand, is specifically designed to not be bottoleneck when it is used in performance tuning a Wallaroo application.
Output File Format
Giles Receiver saves incoming messages in a binary format of the following specification:
- Message Length: A 32-bit (4-byte) unsigned integer specifying the length of the binary message.
- Timestamp: a 64-bit (8-byte) unsigned integer specificying the unadjusted monotonic time (in nanoseconds) at which the the message was received. This timestamp is specific to the machine that is performing this encoding and cannot be translated across machines.
- A binary blob length
An example for such an entry, with a binary message of 10 bytes and the timestamp 10 is
Note that you cannot trust newlines as separators in this encoding. As you can see in the encoded entry above, the character
'\n' appears in both the
'\x00\x00\x00\n' and in the
So care must be taken when decoding giles-receiver output files to not assume that any character separator is safe for use. Instead, the recommended way to decode these files is to treat their contents as a bytestream, and decode them one message at at time in the following manner:
- Read the first 4 bytes and convert them from a big-endian 32-bit unsigned int to a number in your decoder.
- Read the next 8 bytes, and if you have a use for the monotonic unadjusted nanoseconds timestamp, convert it to a number from a big-endian 64-bit unsigned integer.
message_lengthbytes and decode according to your application's
SinkEncoderencoding scheme, if desired.
- Save or process the message
- GOTO 1
For decoding the output, please see Decoding Giles Receiver Output.