Network Performance Framework (NPF) documentation

NPF runs performance tests on network software by running snippets of bash scripts on a cluster following a simple definition script. For instance, the following test script for iPerf3 throughput (omitting graph styling options):

%info
IPerf 2 Throughput Experiment

%config
n_runs=5
var_names={PARALLEL:Number of parallel connections,WINDOW:Window size (kB),THROUGHPUT:Throughput}

%variables
PARALLEL=[1-8]
WINDOW={16,512}
TIME=2

%script@server
iperf -s

%script@client delay=1
//Launch the program, copy the output to a log
iperf -c ${server:0:ip} -w ${WINDOW}k -t $TIME -P $PARALLEL 2>&1 | tee iperf.log
//Parse the log to find the throughput
result=$(cat iperf.log | grep -ioE "[0-9.]+ [kmg]bits" | tail -n 1)
//Give the throughput to NPF through stdout
echo "RESULT-THROUGHPUT $result"

Can be run with the following command line:

npf --test tests/tcp/01-iperf.npf --cluster client=machine01.cluster.com server=machine02.cluster.com

To produce the following graph:

This is the default graph, with almost no configuration, generated out of the data automatically. The default is not always a lineplot. NPF selected a lineplot because of the number of variables and points.

Result for tests/tcp/01-iperf.npf, by default it gives a lineplot

Test scripts defines variables for which NPF runs tests to evaluate their combinations and report performance results.

NPF will output the results in configurable CSV files. Below, two example of CSV outputs:

The format given with --single-output is inteded to use Pandas and post-process the data by yourself. It gives you the result of every runs, for every variables.

CSV example

index

build

test_index

PARALLEL

WINDOW

TIME

THROUGHPUT

run_index

0

local

0

1

16

2

6743098654.72

0

1

local

0

1

16

2

9899899617.28

1

2

local

0

1

16

2

9320079032.32

2

3

local

0

1

16

2

10350871183.36

3

4

local

0

1

16

2

9212704849.92

4

5

local

1

2

16

2

6506875453.44

0

6

local

1

2

16

2

16965120819.2

1

7

local

1

2

16

2

16857746636.8

2

8

local

1

2

16

2

17609365913.6

3

9

local

1

2

16

2

16642998272.0

4

10

local

2

3

16

2

20937965568.0

0

11

local

2

3

16

2

19864223744.0

1

…

NPF automatically builds graphs, computes statistics result for each test showing the importance of each factor, the min/max values, or average for each factor levels.

Feature importance:
   PARALLEL : 0.3966
   WINDOW : 0.6034

Max:
   PARALLEL = 7, WINDOW = 512, TIME = 2 : 99514392248.32
Min:
   PARALLEL = 1, WINDOW = 16, TIME = 2 : 9105330667.52

Means per variables :
PARALLEL:
   1 : 17244293693.44
   2 : 35278861369.34
   3 : 45354854645.76
   4 : 60161754398.72
   5 : 64993592606.72
   6 : 68193343242.24
   7 : 72037338972.16
   8 : 73025181450.24

WINDOW:
   16 : 29615410118.66
   512 : 79456894976.00

NPF can be used to compare different softwares, or the evolution of performances through commits.

TODO

NPF can also generate a website that will allow to explore the data, and share an interactive graph that can be directly linked from a paper.

TODO : demo of website

With the --notebook <path> parameter, NPF can output a Jupyter Notebook that produces the graphs and gives a starting point for further analysis or fine tuning of the graphs. The following image shows the Jupyter Notebook generated by NPF for the iPerf3 throughput test.

Jupyter Notebook for tests/tcp/01-iperf.npf

NPF can execute tests on a computer cluster, running your tests in multiple configurations rapidly and on relevant hardware.

TODO

Test scripts are easy to write and share. We encourage users to share their .npf scripts along with their code to allow other users to reproduce their results and graphs.

The documentation describes first the basic usage and installation of NPF. It then elaborates on writing .npf tests scripts, with the definition of variables and advanced configuration for graphs generation. NPF can also compile and deploy code as specified in repositories. Computer clusters can be specified to NPF in cluster files. A library of reusable modules is available for common tasks such as traffic generation, system tuning, and performance monitoring.

Contents