qa

# Workflow `qa`

- `task_data`:

  - `prefix` (string, optional): prefix this string to the item names provided in the internal collection

  - `reference_prefix` (string, optional unless `enable_regression_tracking` is set): prefix for the item names provided in the internal collection in the corresponding workflow run for reference tests

  - `source_artifact` (Single lookup, required): the debian:source-package or debian:upload artifact representing the source package to test

  - `binary_artifacts` (Multiple lookup, required): the debian:binary-package or debian:upload artifacts representing the binary packages to test

  - `qa_suite` (Single lookup, optional unless `enable_regression_tracking`, `enable_reverse_dependencies_autopkgtest`, `update_qa_results`, or `enable_debdiff` is True): the debian:suite collection that reference tests are being run against to detect regressions, to search for reverse-dependencies, and to debdiff against

  - `reference_qa_results` (Single lookup, optional unless `enable_regression_tracking` or `update_qa_results` is True): the debian:qa-results collection that contains the reference results of QA tasks to use to detect regressions

  - `enable_regression_tracking` (boolean, defaults to False): configure the workflow to detect and display regressions in QA results

  - `update_qa_results` (boolean, defaults to False): when set to True, the workflow runs QA tasks and updates the collection passed in `reference_qa_results` with the results.

  - `vendor` (string, required): the distribution vendor on which to run tests

  - `codename` (string, required): the distribution codename on which to run tests

  - `extra_repositories` (optional): see PackageBuild

  - `architectures` (list of strings, optional): if set, only run on any of these architecture names

  - `architectures_allowlist` (list of strings, optional, either concrete architecture names or `all`): if set, only run on any of these architecture names; while `architectures` is intended to be supplied by users or passed down from a higher-level workflow, this field is intended to be provided via Task configuration

  - `architectures_denylist` (list of strings, optional, either concrete architecture names or `all`): if set, do not run on any of these architecture names; this field is intended to be provided via Task configuration

  - `arch_all_build_architecture` (string, defaults to `amd64`): concrete architecture on which to run tasks for `Architecture: all` packages

  - `enable_check_installability` (boolean, defaults to True): whether to include installability-checking tasks

  - `check_installability_suite` (Single lookup, required if `enable_check_installability` is True): the debian:suite collection to check installability against; once we have a good way to look up the primary suite for a vendor and codename, this could default to doing so

  - `enable_autopkgtest` (boolean, defaults to True): whether to include autopkgtest tasks

  - `autopkgtest_backend` (string, optional): see Autopkgtest

  - `enable_reverse_dependencies_autopkgtest` (boolean, defaults to False): whether to include autopkgtest tasks for reverse-dependencies

  - `enable_lintian` (boolean, defaults to True): whether to include lintian tasks

  - `lintian_backend` (string, optional): see Lintian

  - `lintian_fail_on_severity` (string, optional): see Lintian

  - `enable_piuparts` (boolean, defaults to True): whether to include piuparts tasks

  - `piuparts_backend` (string, optional): see Piuparts

  - `piuparts_environment` (string, optional): the environment to run piuparts in

  - `enable_debdiff` (boolean, defaults to False): whether to include debdiff tasks for source and binary packages. Compares the supplied source package and the binary packages against the packages available in the distribution identified by `qa_suite`.

  - `enable_blhc` (boolean, defaults to False): whether to include `blhc` tasks for the build logs

  - `fail_on` (string, optional): indicate the conditions to trigger a failure of the whole workflow. Allowed values are `failure`, `regression`, `never`. With `failure`, the workflow is marked as failed if one of the QA task fails. With `regression`, the workflow fails only if one of the QA result is a regression compared to the former result. With `never`, the workflow always succeeds. The default value is `regression` if `enable_regression_tracking` is True, otherwise it is `failure`.

The workflow computes dynamic metadata as:

- subject: package name of `source_artifact`

- parameter_summary: package name_version

- source_artifact_id: id of the source artifact

- binary_artifacts_ids: ids of the binary artifacts or None

- reference_source_artifact_id: id of the reference artifact or None

Any of the lookups in `source_artifact` or `binary_artifacts` may result in promises, and in that case the workflow adds corresponding dependencies. Binary promises must include an `architecture` field in their data.

The effective set of architectures is `{architectures}` (defaulting to all architectures supported by this Debusine instance and the `{vendor}:{codename}` suite, plus `all`), intersecting `{architectures_allowlist}` if set, and subtracting `{architectures_denylist}` if set.

The workflow creates sub-workflows and tasks as follows, with substitutions based on its own task data:

- if `enable_check_installability` is set, a single CheckInstallability, with task data:

  - `suite`: `{check_installability_suite}`

  - `binary_artifacts`: the subset of the lookup in this workflow’s `binary_artifacts` for each available architecture

- if `enable_autopkgtest` is set, an autopkgtest sub-workflow, with task data:

  - `source_artifact`: `{source_artifact}`

  - `binary_artifacts`: the subset of the lookup in this workflow’s `binary_artifacts` for each of `all` and the concrete architecture in question that exist

  - `vendor`: `{vendor}`

  - `codename`: `{codename}`

  - `backend`: `{autopkgtest_backend}`

  - `architectures`: the effective set of architectures

  - `arch_all_build_architecture`: `{arch_all_build_architecture}`

- if `enable_reverse_dependencies_autopkgtest` is set, a reverse_dependencies_autopkgtest sub-workflow, with task data:

  - `source_artifact`: `{source_artifact}`

  - `binary_artifacts`: the subset of the lookup in this workflow’s `binary_artifacts` for each of `all` and the concrete architecture in question that exist

  - `qa_suite`: `{qa_suite}`

  - `vendor`: `{vendor}`

  - `codename`: `{codename}`

  - `backend`: `{autopkgtest_backend}`

  - `architectures`: the effective set of architectures

  - `arch_all_build_architecture`: `{arch_all_build_architecture}`

- if `enable_lintian` is set, a lintian sub-workflow, with task data:

  - `source_artifact`: `{source_artifact}`

  - `binary_artifacts`: the subset of the lookup in this workflow’s `binary_artifacts` for each of `all` and the concrete architecture in question that exist

  - `vendor`: `{vendor}`

  - `codename`: `{codename}`

  - `backend`: `{lintian_backend}`

  - `architectures`: the effective set of architectures

  - `arch_all_build_architecture`: `{arch_all_build_architecture}`

  - `fail_on_severity`: `{lintian_fail_on_severity}`

- if `enable_piuparts` is set, a piuparts sub-workflow, with task data:

  - `binary_artifacts`: the subset of the lookup in this workflow’s `binary_artifacts` for each of `all` and the concrete architecture in question that exist

  - `vendor`: `{vendor}`

  - `codename`: `{codename}`

  - `backend`: `{piuparts_backend}`

  - `architectures`: the effective set of architectures

  - `arch_all_build_architecture`: `{arch_all_build_architecture}`

Todo

Not implemented: `enable_check_installability` and `check_installability_suite`.

## Behavior with `update_qa_results` set to True

When `update_qa_results` is set to True, the goal of the workflow is modified: its only purpose is to provide reference results to be stored in a debian:qa-results collection. Task failures are never fatal for the parent workflow or for dependent tasks.

During orchestration, the workflow compares the data available in the debian:qa-results collection together with information about the submitted `source_artifact` and `binary_artifacts`.

When a missing or outdated QA result is detected, it schedules the appropriate QA task, and it creates a corresponding promise in the internal collection (the name of the promise is the prefix followed by the expected name of the collection entry). The QA task has the following event reactions:

- `on_assignment`: an action to skip the work request if the latest relevant item in the debian:qa-results collection has changed since the work request has created; this avoids wasting resources if multiple parallel workflows trigger an update of the same QA results

- `on_success`: an action to add the result to the debian:qa-results collection

- `on_failure`: same as `on_success`

Note that when `enable_reverse_dependencies_autopkgtest` is set to True, it must also update the autopkgtest results of the reverse dependencies and thus compute the same list of packages as the `reverse_dependencies_autopkgtest` workflow (using the same `qa_suite` collection).

## Behavior with `enable_regression_tracking` set to True

When `enable_regression_tracking` is set to True, the orchestrator of the `qa` workflow schedules workflow callbacks that will perform the regression analysis. In order to wait for the availability of the QA result(s), those callbacks have dependencies against:

- the promises associated to the QA result(s) that are required from the additional `qa` workflow building reference results

- the promises associated to the QA result(s) that are required from the sub-workflows

The `workflow_data` field for those workflow callbacks have:

- `visible` set to False so that they do not show up in the workflow hierarchy (new feature to implement)

- `step` set to unique values starting with `regression-analysis` (typically `regression-analysis-{architecture}`)

As part of the callback, the analysis is performed and the result of the analysis is stored in the `output_data` field of the workflow.

Note

We use simple workflow callbacks instead of full-fledged worker tasks or server tasks because we assume that regression analysis can be completed just by comparing the artifact metadata and/or the collection item. Workflow callbacks are already dealt through celery tasks so they are relatively cheap. Note however that the large number of callbacks requires use of careful locking to serialize the operations between concurrent runs trying to update the same workflow.

## Handling of `fail_on`

With `fail_on: never` or `fail_on: regression`, all the sub-workflows are run with `workflow_data.allow_failure: true`.

With `fail_on: regression`, a final orchestrator callback is scheduled:

- it depends on all the `regression-analysis` callbacks

- `workflow_data.visible` is set to True

- `workflow_data.step` is `final-regression-analysis`

- `workflow_data.display_name` is `Regression analysis`

The callback reviews the data in `output_data.regression_analysis` and sets its own result to FAILURE in case of regression, or SUCCESS otherwise.