benchrun API documentation

class benchrun.Benchmark(iteration, case_list, result_fields_append=None, drop_caches=False, gc_collect=True, gc_disable=True, subprocess=True, error_handling='stop')

Bases: object

A class that takes an Iteration instance and runs it correctly for each case

Parameters:

• iteration (Iteration) – An instance of Iteration defining code to benchmark

• case_list (CaseList) – An instance of CaseList defining a list of valid cases for which to run this benchmark.

• results_fields_append (Dict[str, Any]) –

  A dict of values to be appended to BenchmarkResult values after instantiation. Useful for appending extra tags or other metadata in addition to that gathered elsewhere. Only applicable for dict attributes. For each element, will override any keys that already exist, i.e. it does not append recursively.

  In most cases, should be set on the adapter, not here, but can be set here to specify benchmark-dependent values like appending a tag for suite.

• drop_caches (bool) – Try to drop disk caches?

• gc_collect (bool) – Run garbage collection before timing code?

• gc_disable (bool) – Disable garbage collection during timing?

• subprocess (bool) – Run all benchmarks in a subprocess?

• error_handling (str) – What should happen if an iteration errors out? Options: "stop" (skip future iterations and report only error), "break" (skip future iterations and report everything so far), "continue" (run all iterations even if they fail and report everything). As we currently can’t report both metrics and errors for the same benchmark, "stop" and "break" are currently identical, and "continue" may run longer, but will report the same thing.

class benchrun.BenchmarkList(benchmarks)

Bases: object

A list of benchmarks in a Callable class suitable for adapting with benchadapt’s generic CallableAdapter

benchmarks

A list of instances of Benchmark to run together

Type:

List[Benchmark]

class benchrun.CaseList(params)

Bases: object

A class to define a valid list of cases on which to run a benchmark from a set of valid values for each parameter.

params

A dict where keys are the parameter for a benchmark and values are a list of valid arguments for that parameter

Type:

Dict[str, list]

case_list

A list of cases, where each case is a dict where keys are parameters and values are an argument for each respective parameter

Type:

List[Dict[str, Any]]

filter_cases(case_list)

Filter out invalid cases. This method takes a list of cases (dicts where keys are params and values are arguments), and returns a list of cases, possibly filtered down to valid combinations.

By default, does not filter out any cases. Override this method to implement filtering. To specify only a specific set of cases which should be run, simply ignore the input and return that list of cases.

Parameters:

case_list (List[Dict[str, Any]]) – A list of cases, where each case is a dict where keys are parameters and values are arguments for a benchmark.

Return type:

List[Dict[str, Any]]

Returns:

• A case list structured the same as the input, but possibly with invalid

• combinations removed.

class benchrun.Iteration

Bases: ABC

Abstract class defining how to run one iteration of a benchmark.

name

A name for the benchmark. Should be specified when inheriting.

Type:

str

cache

A CacheManager instance for clearing the disk cache when specified. Do not mess with this.

Type:

CacheManager

env

A dict stages can use to pass data between them.

Type:

dict

after_each(case)

Code to run in each iteration after timing.

Parameters:

case (dict) – A dict where keys are parameters and values are scalar arguments for a benchmark

Return type:

None

before_each(case)

Code to run in each iteration before timing.

Parameters:

case (dict) – A dict where keys are parameters and values are scalar arguments for a benchmark

Return type:

None

abstract run(case)

Code to time.

Parameters:

case (dict) – A dict where keys are parameters and values are scalar arguments for a benchmark

Return type:

None

run_iteration(case, settings)

Run a single iteration, without setup or teardown

Return type:

dict

setup(case)

Global setup that runs once before any iteration.

Return type:

None

teardown(case)

Global teardown that runs once after all iterations

Return type:

None