benchalerts API documentation

class benchalerts.AlertPipeline(steps, error_handlers=None)

Bases: object

A structure for running a sequence of configurable AlertPipelineStep instances.

Parameters:

  • steps (List[AlertPipelineStep]) – A list of AlertPipelineStep instances.

  • error_handlers (Optional[List[AlertPipelineErrorHandler]]) – An optional list of AlertPipelineErrorHandler instances to handle any errors that may arise before raising them.

run_pipeline()

Run the pipeline.

Returns:

All steps’ outputs, keyed by step name.

Return type:

Dict[str, Any]

class benchalerts.Alerter

Bases: object

A class to generate default messages and statuses for alerting. You can override any methods for your project’s custom settings.

static clean(text)

A small helper: clean block text so it displays nicely as GitHub Markdown.

Return type:

str

github_check_details(full_comparison)

Generate Markdown details of what happened regarding regressions.

Return type:

Optional[str]

github_check_status(full_comparison)

Return a different status based on errors and regressions.

Return type:

CheckStatus

github_check_summary(full_comparison, build_url)

Generate a Markdown summary of what happened regarding errors and regressions.

Return type:

str

github_pr_comment(full_comparison, check_link)

Generate a GitHub PR comment that summarizes and links to a GitHub Check.

Return type:

str

intro_sentence(full_comparison)

Generate a Markdown sentence to introduce a message.

Return type:

str

slack_message(full_comparison, check_details, comment_details)

Generate a Slack message that links to a GitHub Check.

Return type:

str

benchalerts.pipeline_steps subpackage

class benchalerts.pipeline_steps.BaselineRunCandidates(value)

Bases: Enum

Types of baselines available from /api/runs/{run_id} in the candidate_baseline_runs field.

class benchalerts.pipeline_steps.GetConbenchZComparisonForRunsStep(run_ids, baseline_run_type, z_score_threshold=None, conbench_client=None, step_name=None)

Bases: AlertPipelineStep

An AlertPipeline step to get information from Conbench comparing run(s) to their baselines, using a z-score threshold. This is always the first step of the pipeline.

Parameters:

  • run_ids (List[str]) – A list of Conbench run IDs of the runs to compare. There must be at least one, and they must all be from the same commit.

  • baseline_run_type (BaselineRunCandidates) – The type of baseline to use. See BaselineRunCandidates for options.

  • z_score_threshold (Optional[float]) – The (positive) z-score threshold to send to the Conbench compare endpoint. Benchmarks with a z-score more extreme than this threshold will be marked as regressions or improvements in the result. Default is to use whatever Conbench uses for default (at the time of writing, this is 5).

  • conbench_client (Optional[ConbenchClient]) – A ConbenchClient instance. If not given, one will be created using the standard environment variables.

  • step_name (Optional[str]) – The name for this step. If not given, will default to this class’s name.

Returns:

Information about each run associated with the contender commit, and a comparison to its baseline run if that exists.

Return type:

FullComparisonInfo

Notes

Environment variables

CONBENCH_URL

The URL of the Conbench server. Only required if conbench_client is not provided.

CONBENCH_EMAIL

The email to use for Conbench login. Only required if conbench_client is not provided and the server is private.

CONBENCH_PASSWORD

The password to use for Conbench login. Only required if conbench_client is not provided and the server is private.

run_step(previous_outputs)

Run this step.

Parameters:

previous_outputs (Dict[str, Any]) – A dict of previous steps’ outputs in the pipeline, keyed by step name.

Returns:

Anything (sent to subsequent steps via the previous_outputs dict).

Return type:

Any

class benchalerts.pipeline_steps.GetConbenchZComparisonStep(commit_hash, baseline_run_type, z_score_threshold=None, conbench_client=None, step_name=None)

Bases: GetConbenchZComparisonForRunsStep

An AlertPipeline step to get information from Conbench comparing the runs on a contender commit to their baselines, using a z-score threshold. This is always the first step of the pipeline.

Parameters:

  • commit_hash (str) – The commit hash of the contender commit to compare. Needs to match EXACTLY what Conbench has stored; typically 40 characters. It can’t be a shortened version of the hash.

  • baseline_run_type (BaselineRunCandidates) – The type of baseline to use. See BaselineCandidates for options.

  • z_score_threshold (Optional[float]) – The (positive) z-score threshold to send to the Conbench compare endpoint. Benchmarks with a z-score more extreme than this threshold will be marked as regressions or improvements in the result. Default is to use whatever Conbench uses for default (at the time of writing, this is 5).

  • conbench_client (Optional[ConbenchClient]) – A ConbenchClient instance. If not given, one will be created using the standard environment variables.

  • step_name (Optional[str]) – The name for this step. If not given, will default to this class’s name.

Returns:

Information about each run associated with the contender commit, and a comparison to its baseline run if that exists.

Return type:

FullComparisonInfo

Notes

Environment variables

CONBENCH_URL

The URL of the Conbench server. Only required if conbench_client is not provided.

CONBENCH_EMAIL

The email to use for Conbench login. Only required if conbench_client is not provided and the server is private.

CONBENCH_PASSWORD

The password to use for Conbench login. Only required if conbench_client is not provided and the server is private.

run_step(previous_outputs)

Run this step.

Parameters:

previous_outputs (Dict[str, Any]) – A dict of previous steps’ outputs in the pipeline, keyed by step name.

Returns:

Anything (sent to subsequent steps via the previous_outputs dict).

Return type:

Any

class benchalerts.pipeline_steps.GitHubCheckErrorHandler(commit_hash, repo=None, github_client=None, build_url=None)

Bases: AlertPipelineErrorHandler

Handle errors in a pipeline by updating a GitHub Check status.

Parameters:

  • commit_hash (str) – The commit hash to update.

  • repo (Optional[str]) – The repo name to post the status to, in the form “owner/repo”. Either provide this or github_client.

  • github_client (Optional[GitHubRepoClient]) – A GitHubRepoClient instance. Either provide this or repo.

  • build_url (Optional[str]) – An optional build URL to include in the GitHub Check.

Notes

Environment variables

GITHUB_APP_ID

The ID of a GitHub App that has been set up according to this package’s instructions and installed to your repo. Only required if repo is provided instead of github_client.

GITHUB_APP_PRIVATE_KEY

The private key file contents of a GitHub App that has been set up according to this package’s instructions and installed to your repo. Only required if repo is provided instead of github_client.

handle_error(exc, traceback)

Handle an error that may have happened during a pipeline run.

Parameters:

  • exc (BaseException) – The exception that was raised.

  • traceback (str) – A string of the traceback.

Return type:

None

class benchalerts.pipeline_steps.GitHubCheckStep(commit_hash, comparison_step_name, repo=None, github_client=None, step_name=None, external_id=None, build_url=None, alerter=None)

Bases: AlertPipelineStep

An AlertPipeline step to update a GitHub Check on a commit on GitHub, based on information collected in a previously-run GetConbenchZComparisonStep or GetConbenchZComparisonForRunsStep.

You must use GitHub App authentication to use this step.

Parameters:

  • commit_hash (str) – The commit hash to update.

  • comparison_step_name (str) – The name of the GetConbenchZComparisonStep or GetConbenchZComparisonForRunsStep that ran earlier in the pipeline. If unspecified in that step, it defaults to the name of the step class, e.g. "GetConbenchZComparisonStep".

  • repo (Optional[str]) – The repo name to post the status to, in the form “owner/repo”. Either provide this or github_client.

  • github_client (Optional[GitHubRepoClient]) – A GitHubRepoClient instance. Either provide this or repo.

  • step_name (Optional[str]) – The name for this step. If not given, will default to this class’s name.

  • external_id (Optional[str]) – An optional reference to another system. This argument will set the “external_id” property of the posted GitHub Check, but do nothing else.

  • build_url (Optional[str]) – An optional build URL to include in the GitHub Check.

  • alerter (Optional[Alerter]) – Advanced usage; should not be necessary in most cases. An optional Alerter instance to use to format the GitHub Check’s status, title, summary, and details. If not provided, will default to Alerter().

Returns:

  • dict – The response body from the GitHub HTTP API as a dict.

  • FullComparisonInfo – The FullComparisonInfo object that the GetConbenchZComparisonStep or GetConbenchZComparisonForRunsStep output.

Notes

Environment variables

GITHUB_APP_ID

The ID of a GitHub App that has been set up according to this package’s instructions and installed to your repo. Only required if repo is provided instead of github_client.

GITHUB_APP_PRIVATE_KEY

The private key file contents of a GitHub App that has been set up according to this package’s instructions and installed to your repo. Only required if repo is provided instead of github_client.

run_step(previous_outputs)

Run this step.

Parameters:

previous_outputs (Dict[str, Any]) – A dict of previous steps’ outputs in the pipeline, keyed by step name.

Returns:

Anything (sent to subsequent steps via the previous_outputs dict).

Return type:

Any

class benchalerts.pipeline_steps.GitHubPRCommentAboutCheckStep(pr_number, repo=None, github_client=None, check_step_name='GitHubCheckStep', step_name=None, alerter=None)

Bases: AlertPipelineStep

An AlertPipeline step to make a comment on a PR about a GitHub Check that was created by a previously-run GitHubCheckStep. This is useful if you’re running benchmarks on a merge-commit, and no one is necessarily monitoring the Checks on the default branch. It should be set up to notify the PR that caused the merge-commit, so that the relevant people can take action if necessary.

Parameters:

  • pr_number (int) – The number of the PR to make the comment on.

  • repo (Optional[str]) – The repo name to make the comment on, in the form “owner/repo”. Either provide this or github_client.

  • github_client (Optional[GitHubRepoClient]) – A GitHubRepoClient instance. Either provide this or repo.

  • check_step_name (str) – The name of the GitHubCheckStep that ran earlier in the pipeline. Defaults to “GitHubCheckStep” (which was the default if no name was given to that step).

  • step_name (Optional[str]) – The name for this step. If not given, will default to this class’s name.

  • alerter (Optional[Alerter]) – Advanced usage; should not be necessary in most cases. An optional Alerter instance to use to format the comment. If not provided, will default to Alerter().

Returns:

The response body from the GitHub HTTP API as a dict.

Return type:

dict

Notes

Environment variables

GITHUB_APP_ID

The ID of a GitHub App that has been set up according to this package’s instructions and installed to your repo. Recommended over GITHUB_API_TOKEN. Only required if repo is provided instead of github_client.

GITHUB_APP_PRIVATE_KEY

The private key file contents of a GitHub App that has been set up according to this package’s instructions and installed to your repo. Recommended over GITHUB_API_TOKEN. Only required if repo is provided instead of github_client.

GITHUB_API_TOKEN

A GitHub Personal Access Token with the repo:status permission. Only required if not authenticating with a GitHub App, and if repo is provided instead of github_client.

run_step(previous_outputs)

Run this step.

Parameters:

previous_outputs (Dict[str, Any]) – A dict of previous steps’ outputs in the pipeline, keyed by step name.

Returns:

Anything (sent to subsequent steps via the previous_outputs dict).

Return type:

Any

class benchalerts.pipeline_steps.SlackErrorHandler(channel_id, slack_client=None, build_url=None)

Bases: AlertPipelineErrorHandler

Handle errors in a pipeline by posting a Slack message.

Parameters:

  • channel_id (str) – The ID of the Slack channel to post to.

  • slack_client (Optional[SlackClient]) – A SlackClient instance. If not provided, will default to SlackClient().

  • build_url (Optional[str]) – An optional build URL to include in the message.

Notes

Environment variables

SLACK_TOKEN

A Slack token; see https://api.slack.com/authentication/token-types. Tokens look like xoxb-... if they’re bot tokens. Only required if slack_client is not provided.

handle_error(exc, traceback)

Handle an error that may have happened during a pipeline run.

Parameters:

  • exc (BaseException) – The exception that was raised.

  • traceback (str) – A string of the traceback.

Return type:

None

class benchalerts.pipeline_steps.SlackMessageAboutBadCheckStep(channel_id, slack_client=None, check_step_name='GitHubCheckStep', pr_comment_step_name=None, step_name=None, alerter=None)

Bases: AlertPipelineStep

An AlertPipeline step to post to Slack about a failing GitHub Check that was created by a previously-run GitHubCheckStep. This is useful if you’re running benchmarks on a merge-commit, and no one is necessarily monitoring the Checks on the default branch.

Parameters:

  • channel_id (str) – The ID of the Slack channel to post to.

  • slack_client (Optional[SlackClient]) – A SlackClient instance. If not provided, will default to SlackClient().

  • check_step_name (str) – The name of the GitHubCheckStep that ran earlier in the pipeline. Defaults to “GitHubCheckStep” (which was the default if no name was given to that step).

  • pr_comment_step_name (Optional[str]) – [Optional] The name of the GitHubPRCommentStep that ran earlier in the pipeline. If provided, will include a link to the comment in the Slack message.

  • step_name (Optional[str]) – The name for this step. If not given, will default to this class’s name.

  • alerter (Optional[Alerter]) – Advanced usage; should not be necessary in most cases. An optional Alerter instance to use to format the message. If not provided, will default to Alerter().

Returns:

The response body from the Slack HTTP API as a dict, or None if no message was posted (e.g. if the check was successful).

Return type:

dict

Notes

Environment variables

SLACK_TOKEN

A Slack token; see https://api.slack.com/authentication/token-types. Tokens look like xoxb-... if they’re bot tokens. Only required if slack_client is not provided.

run_step(previous_outputs)

Run this step.

Parameters:

previous_outputs (Dict[str, Any]) – A dict of previous steps’ outputs in the pipeline, keyed by step name.

Returns:

Anything (sent to subsequent steps via the previous_outputs dict).

Return type:

Any