Giles Sender


Giles components act as external testers for Wallaroo. Giles Sender is used to mimic the behavior of an incoming data source. Giles Sender sends data to Wallaroo in one of four ways:

  • With no other commandline options given, it will send string representations of integers, starting with 1 and increasing by 1 with each new message.
  • With a file name given as the --file/-f argument it will send each line of the given file as a message.
  • With a file name given as the --file/-f argument and binary format specified with --binary/-y and a fixed message size, specified with --msg-size/-g, it will send every SIZE bytes of the given file as a message.
  • With a file name given as the --file/-f argument and binary format specified with --binary/-y and variable message size specified with --variable-size/-z, it will read the first 4 bytes to get the message size, send that message and repeat the process.

Building giles/sender

If you do not already have the wallaroo repo, create a local copy:

git clone

Compile the giles-sender binary:

cd wallaroo/giles/sender

Command Line Arguments

giles/sender takes several command line arguments, below is a list with accompanying notes:

  • --host/-h address and port combination that Wallaroo is listening on for incoming source data. Must be provided in the format.
  • --messages/-m number of messages to send before terminating the sender.
  • --file/-f a file with newline-separated entries from which to read entries to send. If multiple files are provided, the first file is read to the end before the next file is open and read, and so on.
  • --batch-size/-s specifies the number of messages sent at every interval.
  • --interval/-i interval at which to send a batch of messages to Wallaroo. Defaults to 5_000_000. The value is in nanoseconds.
  • --repeat/-r tells the sender to repeat reading the file/s as long as necessary to send the number of messages specified by -m.
  • --binary/-y a flag to pass if the data source file is in binary format.
  • --u64/-u send binary-encoded U64 values, starting from offset and incrementing by 1. The first value sent is offset+1.
  • --start-from/-v Set the offset for use with --u64/-u or when --file/-f is not used.
  • --variable-size/-z a flag to pass if using binary format and the binary messages have varying lengths. Cannot be used with --msg-size/-g.
  • --msg-size/-g the size of the binary message in bytes. Cannot be used with --variable-size/-z.
  • --no-write/-w flag to stop the sender from writing the messages sent to file.


Send words from a file

Each line contains a word

# words.txt


sender --host --file words.txt --messages 100 --repeat

Send binary data from a file

Fixed Length Data

In this case, the data in the file is already framed, and the --msg-length parameter is the number of bytes for the message and the length header.

With a 4-byte header and 6-bytes of text, coming to a total of 10-bytes:

# my_binary_data_file


sender --host --file=my_binary_data_file --messages=10 \
  --repeat --batch-size=1 --interval=10_000_000 --binary --msg-size 10 \

Variable Length Data

In this case, the data in the file is frame encoded, but the message length is not constant.

# my_variable_binary_data_file
\x00\x00\x00\x0bhello world


sender --host --file=my_variable_binary_data_file --messages=10 \
  --repeat --batch-size=1 --interval=10_000_000 --binary --variable-size \

Multiple Input Files

When using multiple input files, sender effectively concatenates the files by reading the first file to the end, then reading the next file, and so on. You can use this to concatenate multiple files, or to repeat the same file (although in this case, the --repeat option is better suited).

With the files

# file1

# file2

To concatenate the files, in order, use

sender --host --file=file1,file2 --messages=100 --batch-size=1 \
  --interval=10_000_000 --no-write --ponythreads=1 --ponynoblock

To send the contents of the first file twice, then the contents of the second file once, use

sender --host --file=file1,file1,file2 --messages=100 \
  --batch-size=1 --interval=10_000_000 --no-write \
  --ponythreads=1 --ponynoblock

Multiple input files are also supported for binary input, but they must all be either fixed length or variable length.

Send a sequence of numbers, encoded as strings

To send the sequence '1', '2', '3', ..., '100'


sender --host --messages 100 --ponythreads=1 --ponynoblock

Send a sequence of numbers, encoded as big-endian U64

To send the sequence 1,2,3,...,100


sender --host --messages 100 --u64 \
  --ponythreads=1 --ponynoblock

To send the sequence 101,102,...,200


sender --host --messages 100 --u64 --start-from 100 \
  --ponythreads=1 --ponynoblock

Preparing Data Files for Giles Sender

sender has three modes for file input:

  1. newline delimited text
  2. fixed-length binary
  3. variable-lengh binary

To create a data file for the first case, store your records separated by newlines in a file. To create data for the binary cases, follow the framing protocol.

Pony Runtime Options

The following pony runtime parameters may also be useful in some situations

  • --ponythreads limits the number of threads the pony runtime will use.
  • --ponynoblock
  • --ponypinasio

results matching ""

    No results matching ""