# Measuring performance with parsl-perf

`parsl-perf` is tool for making basic performance measurements of Parsl configurations.

It runs repeated batches of no-op apps, giving a measurement of tasks per second. By default, `parsl-perf` will attempt to estimate a batch size that will take 2 minutes to execute.

This can give a basic measurement of some of the overheads in task execution.

`parsl-perf` must be invoked with a configuration file, which is a Python file containing a variable `config` which contains a `Config` object, or a function `fresh_config` which returns a `Config` object. The `fresh_config` format is the same as used with the pytest test suite.

To specify a `parsl_resource_specification` for tasks, add a `--resources` argument.

To change the target runtime from the default of 120 seconds, add a `--time` parameter.

To change the iteration algorithm used to pick the next batch size and to decide when to stop, use the `--iterate` parameter:

> - `--iterate=estimate` (the default) will run a small batch, then repeatedly re-estimate a new batch size until it finds one that takes at least 75% of the target run time. This will rapidly find a batch size of around the target size.

> - `--iterate=exponential` will start with a small batch, then double the size in each subsequent batch until reaching the target run time. This is useful for understanding how a configuration scales down as batch sizes increase, which is sometimes sub-linear, and to get more consistent batch sizes across runs of `parsl-perf`.

> - `--iterate=10,10000,10000` (or any other sequence of batch sizes) will run a fixed sequence of batches: in this example, first a batch of 10 tasks, then two batches of 10000 tasks. This is useful for making tests that compare fixed batch sizes along with warmup steps between different configurations, without the additional batches of `estimate` and `exponential` to discover time-sized (rather than count-sized) batch sizes.

For example:

    $ python -m parsl.benchmark.perf --config parsl/tests/configs/workqueue_ex.py --resources '{"cores":1, "memory":0, "disk":0}'
    ==== Iteration 1 ====
    Will run 10 tasks to target 120 seconds runtime
    Submitting tasks / invoking apps
    warning: using plain-text when communicating with workers.
    warning: use encryption with a key and cert when creating the manager.
    All 10 tasks submitted ... waiting for completion
    Submission took 0.008 seconds = 1248.676 tasks/second
    Runtime: actual 3.668s vs target 120s
    Tasks per second: 2.726

    [...]

    ==== Iteration 4 ====
    Will run 57640 tasks to target 120 seconds runtime
    Submitting tasks / invoking apps
    All 57640 tasks submitted ... waiting for completion
    Submission took 34.839 seconds = 1654.487 tasks/second
    Runtime: actual 364.387s vs target 120s
    Tasks per second: 158.184
    Cleaning up DFK
    The end