# Contribute to Debusine

Contributions to Debusine are greatly appreciated! You can contribute in many ways, be it writing documentation and blog posts, or fixing bugs and implementing new features.

## Join the community

You can reach us using IRC on the #debusine channel at irc.debian.org. That channel is also available as a (bridged) Matrix room. There’s also a #debusine-notifications channel where you can see all activities happening in the GitLab project (it’s also bridged in a Matrix room).

You can also file issues in GitLab’s interface. There are two templates to help you file bug reports and feature requests.

While Freexian is driving the development forward by paying a few developers, rest assured that external contributions are welcome.

## Where to start?

### Starting small

Before starting to work on bigger projects, it probably makes sense to get used to our development workflows by tackling a few small issues (they have the Quick fix label).

### Looking for issues

If you are looking for bigger issues to tackle, you will find plenty in the issue tracker.

The paid developers are usually focused on all the issues planned in the current milestone, working through the list by order of priority (from labels P1 to P3). If you want to help with the current milestone, you should probably start with a low-priority issue.

Otherwise you can also find issues in the Backlog milestone that are not part of the current development plan and that paid developers will not handle at this point.

### Add a new feature

If you want to work on a new feature that is not yet planned, then you should first create a new issue using the `Feature Request` template (see Design new features with issues).

For larger features that require significant design work, it is recommended to first write a development blueprint (see Design new features with development blueprints).

A particular case where we expect and invite contributions is that of new workflows. See Contribute a new workflow for more details of how to go about doing that.

## How to contribute

Note

See runtime environment to understand the runtime environment.

### Set up your environment

Ready to contribute? Here’s the quickest way to set up Debusine for local development, with everything in a single container so that your host system isn’t affected:

> 1. Set up Incus, then start a development container:

>        $ incus launch images:debian/trixie/cloud debusine-dev
>        $ incus exec debusine-dev -- cloud-init status --wait

> 2. Install `git` in the container:

>        $ incus exec debusine-dev -- apt --update install git

> 3. Optionally, share a directory from your host system with the container so that you can conveniently edit Debusine source code from your host. For example, if you keep source code under the `src` subdirectory of your home directory:

>        $ incus config device add debusine-dev src \
>            disk source=$HOME/src path=/home/debian/src shift=true

>    `shift=true` may require some configuration to get ID mapping to work properly; see the upstream docs.

> 4. Enter the container:

>        $ incus exec debusine-dev -- su - debian

>    Follow the rest of these instructions from inside the container.

> 5. Clone Debusine locally (if you shared a source directory from your host system earlier, you should `cd` to somewhere under that source directory first):

>        $ git clone https://salsa.debian.org/freexian-team/debusine.git
>        $ cd debusine

> 6. For a quick startup, run this command:

>        $ bin/quick-setup.sh

>    It will install required packages with apt and put a sample configuration file in place. It will also create a user and a database in PostgreSQL. The user will be named after your current Unix user and will own the new database so that it can connect to the newly created database without any other authentication.

> 7. Populate the database with:

>        $ ./manage.py migrate

### Make and test your changes

> 1. Install `pre-commit`:

>        $ sudo apt install pre-commit

> 2. Set up `pre-commit` to run automatically when you commit (optional):

>        $ pre-commit install

>    This runs some checks (equivalent to `make check`) every time you run `git commit`. This downloads linters from upstream repositories; if you prefer to avoid that on your development machine, you can skip this step and rely on CI running it for you instead (but with longer latency). See Pre-commit hooks for more details.

>    Make sure you run this in the same environment where you’re going to run `git commit`. If you commit inside the container, then you should install `pre-commit` inside the container; if you commit directly on your host system (perhaps via your editor or IDE), then you should install `pre-commit` directly on your host system.

> 3. Switch to a new branch:

>        $ git checkout -b name-of-your-bugfix-or-feature

> 4. Develop your new feature, ideally following the rules of Test-Driven Development. Remember to update the release notes as needed.

> 5. Run quick static analysis checks on your code (optional):

>        $ make check

>    As above, this downloads linters from upstream repositories, so you can skip it if you’re very cautious about what code runs on your machine, but it spots many problems more quickly than the full unit test suite does.

> 6. Check that all unit tests pass:

>        $ make coverage

>    This runs tests using `pytest` and ensures that you have maintained 100% test coverage. If you get errors, make sure to fix them. You can also run `pytest` manually if you need to use more specific options; for example, `-k` is useful to run specific tests, and `--pdb` is useful for interactive debugging.

>    Note

>    If you get errors like `OSError: [Errno 38] Function not implemented`, then it means that you are lacking /dev/shm with proper permissions.

### Run Debusine components

Note

Most Debusine changes can be tested much more easily and comprehensively by running the test suite, and if that’s all you need to do then you can safely skip these steps. However, if you need to look at the effects of user interface changes in a web browser, or if you need to experiment with task changes in a real worker, then it can be useful to have something closer to a full Debusine deployment.

Each of these commands should be run from a separate shell inside your container (so `incus shell debusine-dev -- su - debian` and change to the directory where you cloned Debusine, as above).

> 1. Start a local test server:

>        $ ./manage.py runserver
>        [...]
>        Starting development server at http://127.0.0.1:8000/
>        Quit the server with CONTROL-C.

>    Visit the URL returned to have access to the test website.

> 2. If you want to serve apt repositories as well, start a second server on port `8081`:

>        $ ./manage.py runserver 8081
>        [...]
>        Starting development server at http://127.0.0.1:8001/
>        Quit the server with CONTROL-C.

> 3. If you want to run tasks, then start the Celery workers:

>        $ python3 -m celery -A debusine.project worker -l INFO -Q scheduler --concurrency 1

>        $ python3 -m celery -A debusine.project worker -l INFO

> 4. Configure the worker:

>        $ mkdir -p ~/.config/debusine/worker
>        $ cat <<EOF > ~/.config/debusine/worker/config.ini
>        [General]
>        api-url = http://localhost:8000/api
>        log-level = DEBUG
>        EOF

> 5. Run the worker with `sbin` dirs in `PATH`, e.g.:

>        $ PATH="/usr/sbin:/sbin:$PATH" LANG=C.UTF-8 python3 -m debusine.worker

> 6. Enable the worker. You can find the name of the worker (by default is the hostname):

>        $ ./manage.py list_workers

>    And then enable it:

>        $ ./manage.py manage_worker enable NAME

> 7. Create a local Django superuser on your test instance:

>        $ ./manage.py createsuperuser

> 8. Login as that user using the link in the top right of the initial REST framework site.

> 9. If you want to use the debusine client, set it up via:

>        $ python3 -m debusine.client setup

>    Set the default workspace to `System`.

> 10. Many of the Debusine worker tasks require an environment, which is a tarball or system image to pass to an executor such as `unshare` or `incus`). You can create one via:

>         $ python3 -m debusine.client work-request create mmdebstrap <<EOF
>           bootstrap_options:
>             architecture: amd64
>             extra_packages:
>             - devscripts
>             variant: minbase
>           bootstrap_repositories:
>           - components:
>             - main
>             mirror: http://deb.debian.org/debian
>             suite: trixie
>         EOF

>     Note that the worker needs the package `mmdebstrap` installed. If the worker does not have the package installed the work request will be in `Pending` and never run. If needed install the package and restart the worker.

>     You can follow the progress (e.g. `Pending`, `Running`, or `Completed`) via the web interface (browse to the scope, then click on the navigation bar for “Work Requests” to see it).

> 11. In order to create workflows or tasks (and also other operations such as creating workflow templates) you will need to give permissions to the user:

> >     $ ./manage.py group create debusine/SystemAdmins
> >     $ ./manage.py workspace grant_role debusine/System owner SystemAdmins
> >     $ ./manage.py group members debusine/SystemAdmins --add YOUR-USER-NAME

### Run integration tests locally

Debusine has a small number of integration tests that check end-to-end functioning of important major features. While these are much slower than unit tests, they run with a full server and worker, so can do things that unit tests can’t: for example, they can run full workflows on real packages.

In most cases, it is easiest to let GitLab CI run integration tests for you. However, if you need to debug failures there, or if you are specifically working on changes to the integration tests, it can be useful to run them locally. You can do this using Incus. It is normally best to use a virtual machine of Debian’s current testing suite; other combinations are possible, but involve more setup.

These instructions assume that your host system has at least autopkgtest 5.53, or a backport of it. This is available in both trixie-backports and forky.

> 1. Create a base image optimized for use by `autopkgtest`:

>        $ autopkgtest-build-incus --vm images:debian/forky

>    You only need to do this occasionally; once a week or so to keep the image reasonably fresh is plenty.

> 2. Decide which integration test you want to run. You can find the available list using:

>        $ grep '^Tests:' debian/tests/control

> 3. In the branch you want to test, run this command, replacing `TEST-NAME` with the name of the integration test you want to run:

>        $ autopkgtest -Us --test-name=TEST-NAME -- \
>            incus autopkgtest/debian/forky/amd64/vm -- \
>            -c limits.cpu=2 -c limits.memory=4GiB

>    This will first build packages from your working tree, then run the given integration test using those packages.

> 4. If you want to look at what’s going on in more detail, you can run commands in the running VM from another terminal. The VM will have a name beginning with `autopkgtest-incus-` (or `autopkgtest-lxd-` if you are using a version of autopkgtest earlier than 5.54), and you can find it using `incus list` or by tab-completion after `incus shell` or `incus exec`. For example, monitoring the systemd journal of a running integration test will show when individual tasks are run by the worker:

>        $ incus exec autopkgtest-incus-XXXXXX -- journalctl --follow

> 5. The `autopkgtest` command will automatically delete the VM once the test has finished. You can use its `--output-dir` option to save artifacts to a given directory, which must either not exist yet or be empty. Alternatively, you can edit an integration test to pause before exiting (perhaps by adding `sleep infinity` to the `copy_logs` function in `debian/tests/utils/add-debusine-log-files-to-artifacts.sh`) so that you can inspect the state of the VM at your leisure.

### Contribute your changes

It is usually easiest to run these steps directly on your host system, to avoid needing to copy SSH keys around.

> 1. If you haven’t done so already, create a guest account on Salsa (a GitLab instance run by the Debian project) by visiting this page: https://salsa.debian.org/users/sign_up

>    Follow all the steps to confirm your email, fill in your profile, and set up your SSH keys.

>    You might want to have a look at Salsa’s documentation and GitLab’s documentation if you have questions about something.

>    Debian Developers can skip this step, as they already have an account on this service.

> 2. If you haven’t done so already, visit the Debusine project’s page and fork it to your own account. See GitLab’s help on how to fork a project. Then, add your fork as a remote, replacing `your-account` with your Salsa username:

>        $ git remote add -f $USER git@salsa.debian.org/your-account/debusine.git

> 3. Push your branch to your repository:

>        $ git push -u $USER name-of-your-bugfix-or-feature

> 4. Submit your work to us by creating a merge request. The previous `git push` step will give you a URL you can use for this. Alternatively, you can visit the Debusine project fork hosted in your own account and create a merge request from there (either through the “Branches” page, or through the “Merge requests” page). See GitLab’s help on merge requests if needed.

>    Make sure to address any issues identified by the continuous integration system. The result of its “pipeline” can be directly seen in the merge request (and in the commits pushed to your own repository). You can push additional changes to your branch as needed for this (`git push name-of-your-bugfix-or-feature`). If you need to amend or rebase commits, then use `git push --force-with-lease name-of-your-bugfix-or-feature`.

## Test latest `devel` branch code

If you want to test the code in the `devel` branch, you can install packages from a repository that is automatically generated by our GitLab CI pipeline. To avoid confusion, we recommend doing this in a separate container from your normal development container.

    $ sudo tee /etc/apt/sources.list.d/debusine.sources <<END
    Types: deb
    URIs: https://deb.debusine.freexian.com/freexian/debusine
    Suites: trixie-devel
    Components: main
    Signed-By:
      -----BEGIN PGP PUBLIC KEY BLOCK-----
      .
      mDMEaaX1bBYJKwYBBAHaRw8BAQdAfpkAx9/TxpbnGHezlUiruo7YT0EcFHEQ9Swt
      65r8wEa0KUFyY2hpdmUgc2lnbmluZyBrZXkgZm9yIGZyZWV4aWFuL2RlYnVzaW5l
      iJAEExYKADgWIQT8mEBDdxrXXLXVVA1rNLmlWenTHAUCaaX1bAIbAwULCQgHAgYV
      CgkICwIEFgIDAQIeAQIXgAAKCRBrNLmlWenTHDBIAP0S1OwE1zTfV+0jFXYZW6J8
      dcS+fKF1S1YCi6cXLuWzZwEApcVXixlRDzOmQQBymyG35nf2ZRYFuhLOlkD/Ad/W
      Xg8=
      =VWZC
      -----END PGP PUBLIC KEY BLOCK-----
    END
    $ sudo apt update

Then run:

    $ sudo apt update
    $ sudo apt install debusine-server debusine-worker debusine-client

And follow the instructions to set up Set up a Debusine server, Set up a Debusine worker or Set up debusine-client.

## Write access to the git repository

Project members have write access to the main git repository. They can thus clone the repository with this URL:

    $ git clone git@salsa.debian.org:freexian-team/debusine.git

From there they can push their changes directly. They are however free to use a fork and request review anyway when they prefer.

## Django Debug Toolbar

Django Debug Toolbar can be used but only on development branches. There are commented out lines in `debusine/project/settings/development.py` and `debusine/project/urls.py` which can be used to add the support.