debian_pipeline

# Workflow `debian_pipeline`

We want to provide a workflow coordinating all the steps that are typically run to build and test an upload to Debian, similar to the Salsa CI pipeline but (eventually) with more distribution-wide testing and the ability to handle the task of performing the upload.

This builds on the existing sbuild workflow.

- `task_data`:

  - `source_artifact` (Single lookup, required): the debian:source-package or debian:upload artifact representing the source package to test

  - `suite` (Single lookup, optional): a debian:suite collection to publish packages to; if set, this suite is also added to the start of `extra_repositories`

  - `vendor` (string, required): the distribution vendor on which to run tests

  - `codename` (string, required): the distribution codename on which to run tests

  - `extra_repositories` (optional): see PackageBuild

  - `architectures` (list of strings, optional): if set, only run on any of these architecture names

  - `architectures_allowlist` (list of strings, optional, either concrete architecture names or `all`): if set, only run on any of these architecture names; while `architectures` is intended to be supplied by users, this field is intended to be provided via Task configuration

  - `architectures_denylist` (list of strings, optional, either concrete architecture names or `all`): if set, do not run on any of these architecture names; this field is intended to be provided via Task configuration

  - `arch_all_build_architecture` (string, defaults to `amd64`): concrete architecture on which to run tasks for `Architecture: all` packages

  - `signing_template_names` (dictionary, optional): mapping from architecture to list of names of binary packages that should be used as signing templates by the make_signed_source sub-workflow

  - `sbuild_backend` (string, optional): see PackageBuild

  - `sbuild_environment_variant` (string, optional): variant of the environment to build on, e.g. `buildd`

  - `qa_suite` (Single lookup, required if `enable_reverse_dependencies_autopkgtest` or `enable_debdiff` is True): the debian:suite collection to search for reverse-dependencies or debdiff original packages, or to use to generate reference QA results to detect regressions. Once we have a good way to look up the primary suite for a vendor and codename, this could default to doing so

  - `enable_regression_tracking` (boolean, defaults to False): configure the QA workflow to detect and display regressions in QA results

  - `regression_tracking_qa_results` (Single lookup, required if `enable_regression_tracking` is True): the debian:qa-results collection that contains the reference results of QA tasks to use to detect regressions

  - `qa_failure_policy` (string, defaults to `ignore`): the policy to apply if the `qa` workflow fails. Allowed values are `ignore`, `fail`, `confirm`.

  - `enable_check_installability` (boolean, defaults to True): whether to include installability-checking tasks

  - `check_installability_suite` (Single lookup, required if `enable_check_installability` is True): the debian:suite collection to check installability against; once we have a good way to look up the primary suite for a vendor and codename, this could default to doing so

  - `enable_autopkgtest` (boolean, defaults to True): whether to include autopkgtest tasks

  - `autopkgtest_backend` (string, optional): see Autopkgtest

  - `enable_reverse_dependencies_autopkgtest` (boolean, defaults to False): whether to include autopkgtest tasks for reverse-dependencies

  - `enable_lintian` (boolean, defaults to True): whether to include lintian tasks

  - `lintian_backend` (string, optional): see Lintian

  - `lintian_fail_on_severity` (string, optional): see Lintian

  - `enable_piuparts` (boolean, defaults to True): whether to include piuparts tasks

  - `piuparts_backend` (string, optional): see Piuparts

  - `piuparts_environment` (string, optional): the environment to run piuparts in

  - `enable_debdiff` (boolean, defaults to False): whether to include debdiff for source and binary packages, comparing the supplied source package and the built binary packages against the packages available in the distribution identified by `qa_suite`.

  - `enable_blhc` (boolean, defaults to True): whether to include `blhc` tasks for the logs associated to the builds

  - `enable_confirmation` (boolean, defaults to False): whether the generated workflow includes a confirmation step asking the user to double check what was built before the upload

  - `enable_make_signed_source` (boolean, defaults to False): whether to sign the contents of builds and make a signed source package

  - `make_signed_source_purpose` (string, required only if `enable_make_signed_source` is True): the purpose of the key to sign with; see Sign

  - `make_signed_source_key` (string, required only if `enable_make_signed_source` is True): the fingerprint to sign with; must match `purpose`

  - `enable_upload` (boolean, defaults to False): whether to upload to an upload queue

  - `upload_key` (Single lookup, optional): key used to sign uploads

  - `upload_binary_key` (Single lookup, optional): key used to sign binary uploads

  - `upload_require_signature` (boolean, defaults to True): whether uploads must be signed; if this is set and neither `upload_key` nor (if applicable) `upload_binary_key` is set, then the user will have to remotely sign the files

  - `upload_include_source` (boolean, defaults to True): include source with the upload

  - `upload_include_binaries` (boolean, defaults to True): include binaries with the upload

  - `upload_merge_uploads` (boolean, defaults to True): if True, merge the uploads for each source package and its binaries that are being uploaded, and create one PackageUpload task per source package to upload them all together; if False, create a separate PackageUpload task for each source and binary upload

  - `upload_since_version` (string, optional): if `source_artifact` is a debian:source-package, include changelog information from all versions strictly later than this version in the `.changes` file; the default is to include only the topmost changelog entry

  - `upload_target_distribution` (string, optional): if `source_artifact` is a debian:source-package, override the target `Distribution` field in the `.changes` file to this value; the default is to use the distribution from the topmost changelog entry

  - `upload_target` (string, defaults to `ftp://anonymous@ftp.upload.debian.org/pub/UploadQueue/`): the upload queue, as an `ftp://` or `sftp://` URL

  - `upload_delayed_days` (integer, optional): the number of days to delay the upload; this assumes that the upload queue implements Debian’s convention of uploading delayed uploads to a `DELAYED/{n}-day` queue

  - `publish_replace` (boolean, defaults to False): if True, replace existing similar items, if the target suite allows it

The workflow computes dynamic metadata as:

- subject: package name of `source_artifact`

- source_artifact_id: id of the source artifact

The effective set of architectures is `{architectures}` (defaulting to all architectures supported by this Debusine instance and the `{vendor}:{codename}` suite, plus `all`), intersecting `{architectures_allowlist}` if set, and subtracting `{architectures_denylist}` if set.

The workflow creates sub-workflows and tasks as follows, with substitutions based on its own task data:

- an sbuild sub-workflow, with task data:

  - `input.source_artifact`: `{source_artifact}`

  - `target_distribution`: `{vendor}:{codename}`

  - `backend`: `{sbuild_backend}`

  - `architectures`: the effective set of architectures

  - `arch_all_build_architecture`: `{arch_all_build_architecture}`, if set

  - `environment_variant`: `{sbuild_environment_variant}`, if set

  - `signing_template_names`: `{signing_template_names}`, if set

- if any of `enable_check_installability`, `enable_autopkgtest`, `enable_lintian`, `enable_piuparts` and `enable_debdiff` are True, a qa sub-workflow, with task data copied from the items of the same name in this workflow’s task data, plus:

  - `binary_artifacts`: `internal@collections/name:build-{architecture}`, for each available architecture

  - `architectures`: the effective set of architectures

- if `enable_confirmation` is set, a Confirm

- if `enable_make_signed_source` and `signing_template_names` are set, a make_signed_source sub-workflow, with task data:

  - `binary_artifacts`: `internal@collections/name:build-{architecture}`, for each available architecture

  - `signing_template_artifacts`: `internal@collections/name:signing-template-{architecture}-{binary_package_name}`, for each architecture and binary package name from `signing_template_names`

  - `vendor`: `{vendor}`

  - `codename`: `{codename}`

  - `architectures`: the effective set of architectures

  - `purpose`: `{make_signed_source_purpose}`

  - `key`: `{make_signed_source_key}`

  - `sbuild_backend`: `{sbuild_backend}`

- if `enable_upload` is set, a package_upload sub-workflow for each source package being uploaded (at least the top-level `source_artifact`, but also each assembled signed source package from the make_signed_source sub-workflow if one exists), configured to require a signature from the developer, with task data:

  - `source_artifact`: the source artifact to upload (or unset if this upload is for the top-level source artifact and `upload_include_source` is False)

  - `binary_artifacts`: `internal@collections/name:{prefix}build-{architecture}`, for each available architecture (or empty if `upload_include_binaries` is False), where `prefix` is empty if this upload is for the top-level source artifact or `signed-source-{architecture}-{binary_package_name}|` if this upload is for an assembled signed source package

  - `merge_uploads`: `{upload_merge_uploads}`

  - `since_version`: `{upload_since_version}`

  - `target_distribution`: `{upload_target_distribution}`

  - `key`: `{upload_key}`

  - `require_signature`: `{upload_require_signature}`

  - `target`: `{upload_target}`

  - `vendor`: `{vendor}`

  - `codename`: `{codename}`

  - `arch_all_build_architecture`: `{arch_all_build_architecture}`

  - `delayed_days`: `{upload_delayed_days}`

- if `suite` is set, a package_publish sub-workflow for each source package being published (at least the top-level `source_artifact`, but also each assembled signed source package from the make_signed_source sub-workflow if one exists), with task data:

  - `source_artifact`: the source artifact to publish

  - `binary_artifacts`: `internal@collections/name:{prefix}build-{architecture}`, for each available architecture, where `prefix` is empty if publishing the top-level source artifact or `signed-source-{architecture}-{binary_package_name}|` if publishing an assembled signed source package

  - `target_suite`: `{suite}`

  - `replace`: `{publish_replace}`

The dependencies between these sub-workflows differ based on the value of `qa_failure_policy`:

- `ignore` (default): The qa sub-workflow is scheduled but nothing depends on it. If `enable_confirmation` is set, then the make_signed_source, package_upload, and package_publish sub-workflows depend on the Confirm task.

- `fail`: The make_signed_source, package_upload, and package_publish sub-workflows depend on the qa sub-workflow, so they are not executed before it finishes and are aborted if anything fails.

- `confirm`: The qa sub-workflow is scheduled with `allow_failure: true` in its workflow data, and a Confirm task is scheduled between it and the remaining parts of the workflow. If the qa sub-workflow succeeds, then that task is confirmed automatically; otherwise, it waits for the user to decide whether to continue the workflow despite the QA failures.

## When `enable_regression_tracking` is set

- the normal qa workflow is run with:

  - `enable_regression_tracking` set to True

  - `reference_qa_results` set with the value of `regression_tracking_qa_results`

  - `reference_prefix` set to `reference-qa-result|`

- an extra qa workflow is run with:

  - `enable_regression_tracking` set to False

  - `reference_qa_results` set with the value of `regression_tracking_qa_results`

  - `update_qa_results` set to True

  - `prefix` set to `reference-qa-result|`

  - `source_artifact` and `binary_artifacts` pointing to the corresponding artifacts in the debian:suite collection listed by `qa_suite`

  - `fail_on` set to `never`

  - `enable_debdiff` set to False (since debdiff doesn’t contribute any QA results)

Todo

Not implemented:

- `enable_check_installability`, `check_installability_suite` (see CheckInstallability)