Prev |  | Next

## 5. Continuous Integration (NUT CI farm) build agent preparation

### 5.1. Custom NUT CI farm build agents: VMs on DigitalOcean

This section details Installation of VMs on Digital Ocean.

### 5.2. Setting up the non-standard VM farm for NUT CI on DigitalOcean

Since 2023 the Network UPS Tools project employs virtual machines, hosted and courteously sponsored as part of FOSS support program by DigitalOcean, for a significant part of the NUT CI farm based on a custom Jenkins setup.

Use of complete machines, virtual or not, in the NUT CI farm allows our compatibility and non-regression testing to be truly multi-platform, spanning various operating system technologies and even (sometimes emulated) CPU architectures.

To that extent, while it is easy to deploy common OS images and manage the resulting VMs, there are only so many images and platforms that are officially supported by the hosting as general-purpose "DigitalOcean VPS Droplets", and work with other operating systems is not easy. But not impossible, either.

In particular, while there is half a dozen Linux distributions offered out of the box, official FreeBSD support was present earlier but abandoned shortly before NUT CI farm migration from the defunct Fosshost.org considered this hosting option.

Still, there were community reports of various platforms including *BSD and illumos working in practice (sometimes with caveats), which just needed some special tinkering to run and to manage. This chapter details how the NUT CI farm VMs were set up on DigitalOcean.

#### Design trade-offs

Note that some design choices were made because equivalent machines existed earlier on Fosshost.org hosting, and filesystem content copies or ZFS snapshot transfers were the least disruptive approach (using ZFS wherever possible also allows to keep the history of system changes as snapshots, easily replicated to offline storage).

It is further important to note that DigitalOcean VMs in recovery mode apparently must use the one ISO image provided by DigitalOcean. At the time of this writing it was based on Ubuntu 18.04 LTS with ZFS support — so the ZFS pools and datasets on VMs that use them should be created AND kept with options supported by that version of the filesystem implementation.

Another note regards pricing: resources that "exist" are billed, whether they run or not (e.g. turned-off VMs still reserve CPU/RAM to be able to run on demand, dormant storage for custom images is used even if they are not active filesystems, etc.)

As of this writing, the hourly prices are applied for resources spawned and destroyed within a calendar month. After a monthly-rate total price for the item is reached, that is applied instead.

##### OS images

Some links will be in OS-specific chapters below; further reading for this effort included:

- https://www.digitalocean.com/blog/custom-images
- https://ptribble.blogspot.com/2021/04/running-tribblix-on-digital-ocean.html — notes on custom image creation, may involve https://github.com/illumos/metadata-agent
- https://bsd-cloud-image.org/— A collection of pre-built *BSD cloud images

According to the fine print in the scary official docs, DigitalOcean VMs can only use "custom images" in one of a number of virtual HDD formats, which should carry an ext3/ext4 filesystem for DigitalOcean addons to barge into for management.

In practice, uploading other images (OpenIndiana Hipster "cloud" image, OmniOS, FreeBSD) from your workstation or by providing an URL to an image file on the Internet (see links in this document for some collections) sort of works. While the upload status remained "pending", a VM could often be made with it soon… but in other cases you have to wait a surprisingly long time, some 15-20 minutes, and additional images suddenly become "Uploaded".

- The initial theory was that we exceeded some limit and after ending the setups with one custom image, it can be nuked and then another used in its place; in practice this seems to be not true — just the storage information refresh (perhaps propagation from cache to committed) can lag.
- Note that your budget would be invoiced for storage of custom images too. If you use stock ISOs once, it makes sense to remove them later.
- There is also an option to use pre-installed operating systems, so you can dynamically create and destroy VMs with minimal work after creation (e.g. with Jenkins cloud plugins to spawn workers according to labels); in this case you may want to retain (eventually update) the golden image.
- It may be that not all non-standard images are supported, but those with `cloud-init` or similar tools (see https://www.digitalocean.com/blog/custom-images for details).

##### Networking

FIXME: Private net, DO-given IPs

One limitation seen with "custom images" is that IPv6 is not offered to those VMs.

Generally all VMs get random (hopefully persistent) public IPv4 addresses from various subnets. It is possible to also request an interconnect VLAN for one project’s VMs co-located in same data center and have it attached (with virtual IP addresses) to an additional network interface on each of your VMs: it is supposed to be faster and free (regarding traffic quotas).

- For the Jenkins controller which talks to the world (and enjoys an off-hosting backup at a maintainer’s home server) having substantial monthly traffic quota is important.
- For the set of builders hosted on DigitalOcean, which would primarily talk to the controller in the common VLAN — not so much (just OS upgrades? maybe GitHub?)

One more potential caveat: while DigitalOcean provides VPC network segments for free inter-communications of a group of droplets, it assigns IP addresses to those and does not let any others be used by the guest. This causes some hassle when importing a set of VMs which used different IP addresses on their inter-communications VLAN originally (on another hosting).

##### Common notes for illumos VMs

The original OpenIndiana Hipster and OmniOS VMs were configured with the https://github.com/jimklimov/illumos-splitroot-scripts methodology and scripting, so there are quite a few datasets dedicated to their purposes instead of a large one.

There are known issues about VM reboot:

- Per https://www.illumos.org/issues/14526 and personal and community practice, it seems that "slow reboot" for illumos VMs on QEMU-6.x (and on DigitalOcean) misbehaves and hangs, ultimately the virtual hardware is not power-cycled.
- A power-off/on cycle through UI (and probably REST API) does work.
- It took about 2 hours for `rebooting...` to take place in fact. At least, the machine would not be stuck for eternity in case of unattended crashes.
- Other kernels (Linux, BSD, …) are not impacted by this, it seems.

Wondering if there are QEMU HW watchdogs on DigitalOcean that we could use…

#### Using the DigitalOcean Recovery ISO

As noted above, for installation and subsequent management DigitalOcean’s recovery ISO must be used when booting the VM, which is based on Ubuntu and includes ZFS support. It was used a lot both initially and over the years, so deserves a dedicated chapter.

To boot into the recovery environment, you should power off the VM (see the Power left-menu item in DigitalOcean web dashboard, and "Turn off Droplet"), then go into the Recovery menu and select "Boot from Recovery ISO" and power on the Droplet. When you are finished with recovery mode operations, repeat this routine but select "Boot from Hard Drive" instead.

### Note

Sometimes you might be able to change the boot device in advance (it takes time to apply the setting change) and power-cycle the VM later.

The recovery live image allows to install APT packages, such as `mc` (file manager and editor) and `mbuffer` (to optimize `zfs-send`/`zfs-recv` traffic). When the image boots, it offers a menu which walks through adding SSH public keys (can import ones from e.g. GitHub by username).

Note that if your client system uses `screen`, `tmux` or `byobu`, the new SSH connections would get the menu again. To get a shell right away, interactive or for scripting like `rsync` and `zfs recv` counterparts, you should `export TERM=vt220` from your `screen` session (the latter proved useful in any case for independence of the long replication run from connectivity of my laptop to Fosshost/DigitalOcean VMs).

- SSH keys can be imported with a `ssh-import-id-gh` helper script provided in the image:

      #recovery# ssh-import-id-gh jimklimov
      2023-12-10 21:32:18,069 INFO Already authorized ['2048',
          'SHA256:Q/ouGDQn0HUZKVEIkHnC3c+POG1r03EVeRr81yP/TEoQ',
          'jimklimov@github/10826393', '[RSA]']
      ...

- More can be pasted into `~/.ssh/authorized_keys` later;
- The real SSH session is better than the (VNC-based web-wrapped) Rescue Console, which is much less responsive and also lacks mouse and copy-paste integration with your browser;

- On your SSH client side (e.g. in the `screen` session on original VM which would send a lot of data), you can add non-default (e.g. one-time) keys of the SSH server of the recovery environment with:

      #origin# eval `ssh-agent`
      #origin# ssh-add ~/.ssh/id_rsa_custom_key

Make the recovery userland convenient:

    #recovery# apt install mc mbuffer

- `mc`, `mcview` and `mcedit` are just very convenient to manage systems and to manipulate files;

- ZFS send/receive traffic is quite bursty, with long quiet times as it investigates the source or target pools respectively, and busy streaming times with data.

  Using an `mbuffer` on at least one side (ideally both to smooth out network latency) is recommended to have something useful happen when at least one of the sides has the bulk data streaming phase.

#### OpenIndiana

Helpful links for this part of the quest:

- https://openindiana.org/downloads/ ⇒ see https://dlc.openindiana.org/isos/hipster/20231027/OI-hipster-cloudimage.img.gz — OI distro-provided cloud images (detailed at https://www.openindiana.org/announcements/openindiana-hipster-2023-04-announcement/ release notes, though not at later ones)

##### DO-NUT-CI-OI VM creation

Initial attempt, using the OpenIndiana cloud image ISO:

The OI image could be loaded… but that’s it — the logo is visible on the DigitalOcean Recovery Console, as well as some early boot-loader lines ending with a list of supported consoles. I assume it went into the `ttya` (serial) console as one is present in the hardware list, but DigitalOcean UI does not make it accessible and I did not find quickly if there are any REST API or SSH tunnel into serial ports.

### Note

The web console did not come up quickly enough after a VM (re-)boot for any interaction with the early seconds of ISO image loader’s uptime, if it even offers any.

It probably booted and auto-installed, since I could see an `rpool/swap` twice the size of VM RAM later on, and the `rpool` occupied the whole VM disk (created with auto-sizing).

The VM can however be rebooted with a (DO-provided) Recovery ISO, based at that time on Ubuntu 18.04 LTS with ZFS support — which was sufficient to send over the existing VM contents from original OI VM on Fosshost. See above about booting and preparing that environment.

##### DO-NUT-CI-OI VM OS transfer

As the practically useful VM already existed at Fosshost.org, and a quick shot failed at making a new one from scratch, in order to only transfer local zones (containers), a decision was made to transfer the whole ZFS pool via snapshots using the Recovery ISO.

First, following up from the first experiment above: I can import the ZFS pool created by cloud-OI image into the Linux Recovery CD session:

- Check known pools:

      #recovery# zpool import
         pool: rpool
             id: 7186602345686254327
        state: ONLINE
       status: The pool was last accessed by another system.
       action: The pool can be imported using its name or numeric identifier and the `-f' flag.
           see: http://zfsonlinux.org/msg/ZFS-8000-EY
       config:
              rpool ONLINE
                 vda ONLINE

- Import without mounting (`-N`), using an alternate root if we decide to mount something later (`-R /a`), and ignoring possible markers that the pool was not unmounted so might be used by another storage user (`-f`):

      #recovery# zpool import -R /a -N -f rpool

- List what we see here:

      #recovery# zfs list
      NAME                  USED  AVAIL  REFER  MOUNTPOINT
      rpool                34.1G   276G   204K  /rpool
      rpool/ROOT           1.13G   276G   184K  legacy
      rpool/ROOT/c936500e  1.13G   276G  1.13G  legacy
      rpool/export          384K   276G   200K  /export
      rpool/export/home     184K   276G   184K  /export/home
      rpool/swap           33.0G   309G   104K  -

The import and subsequent inspection above showed that the kernel core-dump area was missing, compared to the original VM… so adding per best practice:

- Check settings wanted by the installed machine for the `rpool/dump` dataset:

      #origin# zfs get -s local all rpool/dump
      NAME        PROPERTY                        VALUE                           SOURCE
      rpool/dump  volsize                         1.46G                           local
      rpool/dump  checksum                        off                             local
      rpool/dump  compression                     off                             local
      rpool/dump  refreservation                  none                            local
      rpool/dump  dedup                           off                             local

- Apply to the new VM:

      #recovery# zfs create -V 2G -o checksum=off -o compression=off \
          -o refreservation=none -o dedup=off rpool/dump

To receive ZFS streams from the running OI into the freshly prepared cloud-OI image, it wanted the ZFS features to be enabled (all were disabled by default) since some are used in the replication stream:

- Check what is there initially (on the new VM):

      #recovery# zpool get all
      NAME   PROPERTY                       VALUE                          SOURCE
      rpool  size                           320G                           -
      rpool  capacity                       0%                             -
      rpool  altroot                        -                              default
      rpool  health                         ONLINE                         -
      rpool  guid                           7186602345686254327            -
      rpool  version                        -                              default
      rpool  bootfs                         rpool/ROOT/c936500e            local
      rpool  delegation                     on                             default
      rpool  autoreplace                    off                            default
      rpool  cachefile                      -                              default
      rpool  failmode                       wait                           default
      rpool  listsnapshots                  off                            default
      rpool  autoexpand                     off                            default
      rpool  dedupditto                     0                              default
      rpool  dedupratio                     1.00x                          -
      rpool  free                           318G                           -
      rpool  allocated                      1.13G                          -
      rpool  readonly                       off                            -
      rpool  ashift                         12                             local
      rpool  comment                        -                              default
      rpool  expandsize                     -                              -
      rpool  freeing                        0                              -
      rpool  fragmentation                  -                              -
      rpool  leaked                         0                              -
      rpool  multihost                      off                            default
      rpool  feature@async_destroy          disabled                       local
      rpool  feature@empty_bpobj            disabled                       local
      rpool  feature@lz4_compress           disabled                       local
      rpool  feature@multi_vdev_crash_dump  disabled                       local
      rpool  feature@spacemap_histogram     disabled                       local
      rpool  feature@enabled_txg            disabled                       local
      rpool  feature@hole_birth             disabled                       local
      rpool  feature@extensible_dataset     disabled                       local
      rpool  feature@embedded_data          disabled                       local
      rpool  feature@bookmarks              disabled                       local
      rpool  feature@filesystem_limits      disabled                       local
      rpool  feature@large_blocks           disabled                       local
      rpool  feature@large_dnode            disabled                       local
      rpool  feature@sha512                 disabled                       local
      rpool  feature@skein                  disabled                       local
      rpool  feature@edonr                  disabled                       local
      rpool  feature@userobj_accounting     disabled                       local

- Enable all features this pool knows about (list depends on both ZFS module versions which created the pool and which are running now):

      #recovery# zpool get all | grep feature@ | awk '{print $2}' | \
          while read F ; do zpool set $F=enabled rpool ; done

On the original VM, stop any automatic snapshot services like ZnapZend or `zfs-auto-snapshot`, and manually snapshot all datasets recursively so that whole data trees can be easily sent over (note that we then remove some snaps like for `swap`/`dump` areas which otherwise waste a lot of space over time with blocks of obsolete swap data held by the pool for possible dataset rollback):

    #origin# zfs snapshot -r rpool@20231210-01
    #origin# zfs destroy rpool/swap@20231210-01&
    #origin# zfs destroy rpool/dump@20231210-01&

On the receiving VM, move existing cloudy `rpool/ROOT` out of the way, if we would not use it anyway, so the new one from the original VM can land (for kicks, we can `zfs rename` the cloud-image’s boot environment back into the fold after replication is complete). Also prepare to maximally compress the received root filesystem data, so it does not occupy too much in the new home (this is not something we write too often, so slower `gzip-9` writes can be tolerated):

    #recovery# zfs rename rpool/ROOT{,x} ; \
        while ! zfs set compression=gzip-9 rpool/ROOT ; do sleep 0.2 || break ; done

Send over the data (from the prepared `screen` session on the origin server); first make sure all options are correct while using a dry-run mode, e.g.:

    ### Do not let other work of the origin server preempt the replication
    #origin# renice -n -20 $$

    #origin# zfs send -Lce -R rpool/ROOT@20231210-01 | mbuffer | \
        ssh root@recovery "mbuffer | zfs recv -vFnd rpool"

- Then remove `-n` from `zfs recv` after initial experiments confirm it would receive what you want and where you want it, and re-run.

With sufficiently large machines and slow source hosting, expect some hours for the transfer.

- I saw 4-8Mb/s in the streaming phase for large increments, and quite a bit of quiet time during enumeration of even almost-empty regular snapshots made by ZnapZend— low-level work with ZFS metadata has a cost.

Note that one of the benefits of ZFS (and the non-automatic snapshots used here) is that it is easy to catch-up later to send the data which the original server would generate and write during the replication. You can keep it actually working until the last minutes of the migration.

After the large initial transfers complete, follow-up with a pass to stop the original services (e.g. whole `zones` either from OS default grouping or as wrapped by https://github.com/jimklimov/illumos-smf-zones scripting) and replicate any new information created on origin server during this transfer (and/or human outage for the time it would take you to focus on this task again, after the computers were busy for many hours…)

### Note

The original VM had ZnapZend managing regular ZFS snapshots and their off-site backups. As the old machine would no longer be doing anything of consequence, keep the service there disable and also turn off the tunnel to off-site backup — this serves to not confuse your remote systems as an admin. The new VM clone would just resume the same snapshot history, poured to the same off-site backup target.

- `rsync` the `rpool/boot/` from old machine to new, which is a directory right in the `rpool` dataset and has boot-loader configs; update `menu.lst` for GRUB boot-loader settings;
- run `zpool set bootfs=...` to enable the transplanted root file system;
- `touch reconfigure` in the new rootfs (to pick up changed hardware on boot);
- be ready to fiddle with `/etc/dladm/datalink.conf` (if using virtual links, etherstubs, etc.), as well as `/etc/hostname*`, `/etc/defaultrouter` etc.
- revise the loader settings regarding the console to use (should be `text` first here on DigitalOcean) — see in `/boot/solaris/bootenv.rc` and/or `/boot/defaults/loader.conf`
- reboot into production mode to see if it all actually "works" :)

If the new VM does boot correctly, log into it and:

- Revive the `znapzend` retention schedules: they have a configuration source value of `received` in ZFS properties of the replica, so are ignored by the tool. See `znapzendzetup list` on the original machine to get a list of datasets to check on the replica, e.g.:

      :; zfs get -s received all rpool/{ROOT,export,export/home/abuild/.ccache,zones{,-nosnap}} \
          | grep znapzend | while read P K V S ; do zfs set $K="$V" $P & done

- re-enable `znapzend` and `zones` SMF services on the new VM;
- check about `cloud-init` integration services; the `metadata-agent` seems buildable and installable, it logged the SSH keys on console after service manifest import (details elaborated in links above).

##### DO-NUT-CI-OI VM preparation as build agent

As of this writing, the NUT CI Jenkins controller runs on DigitalOcean — and feels a lot snappier in browsing and SSH management than the older Fosshost.org VMs. Despite the official demise of the platform, they were alive and used as build agents for the newly re-hosted Jenkins controller for over a year until somebody or something put them to rest: the container with the old production Jenkins controller was set to not-auto-booting, and container with worker was attached to the new controller.

The Jenkins SSH Build Agent setups involved here were copied on the controller (as XML files) and then updated to tap into the different "host" and "port" (so that the original definitions can in time be used for replicas on DO), and due to trust settings — the `~jenkins/.ssh/known_hosts` file on the new controller had to be updated with the "new" remote system fingerprints. Otherwise, the migration went smooth.

Similarly, existing Jenkins swarm agents from community PCs had to be taught the new DNS name (some had it in `/etc/hosts`), but otherwise connected OK.

#### OmniOS

Helpful links for this part of the quest:

- https://omnios.org/download ⇒ see https://downloads.omnios.org/media/lts/omnios-r151046.cloud.vmdk (LTS) or https://downloads.omnios.org/media/stable/omnios-r151048.cloud.vmdk (recent stable) or daily "bloody" images like https://downloads.omnios.org/media/bloody/omnios-bloody-20231209.cloud.vmdk

##### DO-NUT-CI-OO VM preparation

Added replicas of more existing VMs: OmniOS (relatively straightforward with the OI image).

The original OmniOS VM used ZFS, so its contents were sent-received similarly to the OI VM explained above.

#### FreeBSD

Helpful links for this part of the quest:

- https://www.adminbyaccident.com/freebsd/how-to-upload-a-freebsd-custom-image-on-digitalocean/

##### DO-NUT-CI-FREEBSD VM preparation

Added replicas of more existing VMs: FreeBSD 12 (needed to use a seed image, tried an OpenIndiana image first but did not cut it — the ZFS options in its `rpool` were too new, so the older build of the BSD loader was not too eager to find the pool).

The original FreeBSD VM used ZFS, so its contents were sent-received similarly to the OI VM explained above.

- The (older version of?) FreeBSD loader rejected a `gzip-9` compressed `zroot/ROOT` location, so care had to be taken to first disable compression (only on the original system’s tree of root filesystem datasets). The last applied ZFS properties are used for the replication stream.

#### OpenBSD

Helpful links for this part of the quest:

- https://dev.to/nabbisen/custom-openbsd-droplet-on-digitalocean-4a9o—  how to piggyback OpenBSD via FreeBSD images (no longer offered by default on DO)

##### DO-NUT-CI-OPENBSD VM creation

Added a replica of OpenBSD 6.5 VM as an example of relatively dated system in the CI farm, which went decently well as a `dd` stream of the local VM’s vHDD into DO recovery console session:

    #tgt-recovery# mbuffer -4 -I 12340 > /dev/vda

    #src# dd if=/dev/rsd0c | time nc myHostingIP 12340

…followed by a reboot and subsequent adaptation of `/etc/myname` and `/etc/hostname.vio*` files.

I did not check if the DigitalOcean recovery image can directly mount BSD UFS partitions, as it sufficed to log into the pre-configured system.

One caveat was that it was originally installed with X11, but DigitalOcean web-console did not pass through the mouse nor advanced keyboard shortcuts. So `rcctl disable xenodm` (to reduce the attack surface and resource waste).

FWIW, `openbsd-7.3-2023-04-22.qcow2` "custom image" did not seem to boot. At least, no activity on display and the IP address did not go up.

#### Linux

Helpful links for this part of the quest:

- https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/Debian%20Bookworm%20Root%20on%20ZFS.html — first steps for moving our older Linux VM onto ZFS root

##### DO-NUT-CI-LINUX VM creation

Spinning up the Debian-based Linux builder (with many containers for various Linux systems) with ZFS, to be consistent across the board, was an adventure.

- DigitalOcean rescue CD is Ubuntu 18.04 based, it has an older ZFS version so instructions from https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/Debian%20Stretch%20Root%20on%20ZFS.html have to be used particularly to `zpool create bpool` (with the dumbed-down options for GRUB to be able to read that boot-pool);
- For the rest of the system, https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/Debian%20Bookworm%20Root%20on%20ZFS.html is relevant for current distro (Debian 12) and is well-written;
- Note that while in many portions the "MBR or (U)EFI" boot is a choice of either one command to copy-paste or another, the spot about installing GRUB actually requires both (MBR for disk to be generally bootable, and EFI to proceed with that implementation);
- If the (recovery) console with the final OS is too "tall" in the Web-UI, so the lower rows are hidden by the DO banner with IP address, and you can’t see the commands you are typing, try `clear ; stty size` to check the current display size (was 128x48 for me) and `stty rows 45` to reduce it a bit. Running a full-screen program like `mc` helps gauge if you got it right.

##### DO-NUT-CI-LINUX VM OS transfer

After the root pool was prepared and the large tree of datasets defined to handle the numerous LXC containers, `abuild` home directory, and other important locations of the original system, `rsync -avPHK` worked well to transfer the data.

##### DO-NUT-CI-LINUX VM preparation as build agent

Numerous containers with an array of Linux distributions are used as either Jenkins SSH build agents or swarm agents, as documented in chapters about LXC containers.

### 5.3. Custom NUT CI farm build agents: LXC multi-arch containers

This section details configuration of LXC containers as build environments for NUT CI farm; this approach can also be used on developer workstations.

### 5.4. Setting up the multi-arch Linux LXC container farm for NUT CI

Due to some historical reasons including earlier personal experience, the Linux container setup implemented as described below was done with persistent LXC containers wrapped by LIBVIRT for management. There was no particular use-case for systems like Docker (and no firepower for a Kubernetes cluster) in that the build environment intended for testing non-regression against a certain release does not need to be regularly updated — its purpose is to be stale and represent what users still running that system for whatever reason (e.g. embedded, IoT, corporate) have in their environments.

#### Common preparations

- Example list of packages for Debian-based systems may include (not necessarily is limited to):

      :; apt install lxc lxcfs lxc-templates \
          ipxe-qemu qemu-kvm qemu-system-common qemu-system-data \
          qemu-system-sparc qemu-system-x86 qemu-user-static qemu-utils \
          virt-manager virt-viewer virtinst ovmf \
          libvirt-daemon-system-systemd libvirt-daemon-system \
          libvirt-daemon-driver-lxc libvirt-daemon-driver-qemu \
          libvirt-daemon-config-network libvirt-daemon-config-nwfilter \
          libvirt-daemon libvirt-clients

      # TODO: Where to find virt-top - present in some but not all releases?
      # Can fetch sources from https://packages.debian.org/sid/virt-top and follow
      # https://www.linuxfordevices.com/tutorials/debian/build-packages-from-source
      # Be sure to use 1.0.x versions, since 1.1.x uses a "better-optimized API"
      # which is not implemented by libvirt/LXC backend.

  ### Note

  This claims a footprint of over a gigabyte of new packages when unpacked and installed to a minimally prepared OS. Much of that would be the graphical environment dependencies required by several engines and tools.

- Prepare LXC and LIBVIRT-LXC integration, including an "independent" (aka "masqueraded) bridge for NAT, following https://wiki.debian.org/LXC and https://wiki.debian.org/LXC/SimpleBridge

  - For dnsmasq integration on the independent bridge (`lxcbr0` following the documentation examples), be sure to mention:

    - `LXC_DHCP_CONFILE="/etc/lxc/dnsmasq.conf"` in `/etc/default/lxc-net`
    - `dhcp-hostsfile=/etc/lxc/dnsmasq-hosts.conf` in/as the content of `/etc/lxc/dnsmasq.conf`
    - `touch /etc/lxc/dnsmasq-hosts.conf` which would list simple `name,IP` pairs, one per line (so one per container)
    - `systemctl restart lxc-net` to apply config (is this needed after setup of containers too, to apply new items before booting them?)
    - For troubleshooting, see `/var/lib/misc/dnsmasq.lxcbr0.leases` (in some cases you may have to rename it away and reboot host to fix IP address delegation)
- Install qemu with its `/usr/bin/qemu-*-static` and registration in `/var/lib/binfmt`
- Prepare an LVM partition (or preferably some other tech like ZFS) as `/srv/libvirt` and create a `/srv/libvirt/rootfs` to hold the containers

- Prepare `/home/abuild` on the host system (preferably in ZFS with lightweight compression like lz4 — and optionally, only if the amount of available system RAM permits, with deduplication; otherwise avoid it); account user and group ID numbers are `399` as on the rest of the CI farm (historically, inherited from OBS workers)

  - It may help to generate an ssh key without a passphrase for `abuild` that it would trust, to sub-login from CI agent sessions into the container. Then again, it may be not required if CI logs into the host by SSH using `authorized_keys` and an SSH Agent, and the inner ssh client would forward that auth channel to the original agent.

        abuild$ ssh-keygen
        # accept defaults

        abuild$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
        abuild$ chmod 640 ~/.ssh/authorized_keys

- Edit the root (or whoever manages libvirt) `~/.profile` to default the virsh provider with:

      LIBVIRT_DEFAULT_URI=lxc:///system
      export LIBVIRT_DEFAULT_URI

- If host root filesystem is small, relocate the LXC download cache to the (larger) `/srv/libvirt` partition:

      :; mkdir -p /srv/libvirt/cache-lxc
      :; rm -rf /var/cache/lxc
      :; ln -sfr /srv/libvirt/cache-lxc /var/cache/lxc

  - Maybe similarly relocate shared `/home/abuild` to reduce strain on rootfs?

#### Setup a container

Note that completeness of qemu CPU emulation varies, so not all distros can be installed, e.g. "s390x" failed for both debian10 and debian11 to set up the `openssh-server` package, or once even to run `/bin/true` (seems to have installed an older release though, to match the outdated emulation?)

While the `lxc-create` tool does not really specify the error cause and deletes the directories after failure, it shows the pathname where it writes the log (also deleted). Before re-trying the container creation, this file can be watched with e.g. `tail -F /var/cache/lxc/.../debootstrap.log`

### Note

You can find the list of LXC "template" definitions on your system by looking at the contents of the `/usr/share/lxc/templates/` directory, e.g. a script named `lxc-debian` for the "debian" template. You can see further options for each "template" by invoking its help action, e.g.:

    :; lxc-create -t debian -h

##### Initial container installation (for various guest OSes)

- Install containers like this:

      :; lxc-create -P /srv/libvirt/rootfs \
          -n jenkins-debian11-mips64el -t debian -- \
          -r bullseye -a mips64el

  - to specify a particular mirror (not everyone hosts everything —  so if you get something like `"E: Invalid Release file, no entry for main/binary-mips/Packages"` then see https://www.debian.org/mirror/list for details, and double-check the chosen site to verify if the distro version of choice is hosted with your arch of choice):

        :; MIRROR="http://ftp.br.debian.org/debian/" \
           lxc-create -P /srv/libvirt/rootfs \
            -n jenkins-debian10-mips -t debian -- \
            -r buster -a mips

  - …or for EOLed distros, use the Debian Archive server.

    - Install the container with Debian Archive as the mirror like this:

          :; MIRROR="http://archive.debian.org/debian-archive/debian/" \
             lxc-create -P /srv/libvirt/rootfs \
              -n jenkins-debian8-s390x -t debian -- \
              -r jessie -a s390x

    - Note you may have to add trust to their (now expired) GPG keys for packaging to verify signatures made at the time the key was valid, by un-symlinking (if appropriate) the debootstrap script such as `/usr/share/debootstrap/scripts/jessie`, commenting away the `keyring /usr/share/keyrings/debian-archive-keyring.gpg` line and setting `keyring /usr/share/keyrings/debian-archive-removed-keys.gpg` instead. You may further have to edit `/usr/share/debootstrap/functions` and/or `/usr/share/lxc/templates/lxc-debian` to honor that setting (the `releasekeyring` was hard-coded in version I had installed), e.g. in the latter file ensure such logic as below, and re-run the installation:

          ...
              # If debian-archive-keyring isn't installed, fetch GPG keys directly
              releasekeyring="`grep -E '^keyring ' "/usr/share/debootstrap/scripts/$release" | sed -e 's,^keyring ,,' -e 's,[ #].*$,,'`" 2>/dev/null
              if [ -z $releasekeyring ]; then
                  releasekeyring=/usr/share/keyrings/debian-archive-keyring.gpg
              fi
              if [ ! -f $releasekeyring ]; then
          ...

  - …Alternatively, other distributions can be used (as supported by your LXC scripts, typically in `/usr/share/debootstrap/scripts`), e.g. Ubuntu:

        :; lxc-create -P /srv/libvirt/rootfs \
            -n jenkins-ubuntu1804-s390x -t ubuntu -- \
            -r bionic -a s390x

  - For distributions with a different packaging mechanism from that on the LXC host system, you may need to install corresponding tools (e.g. `yum4`, `rpm` and `dnf` on Debian hosts for installing CentOS and related guests). You may also need to pepper with symlinks to taste (e.g. `yum => yum4`), or find a `pacman` build to install Arch Linux or derivative, etc. Otherwise, you risk seeing something like this:

        root@debian:~# lxc-create -P /srv/libvirt/rootfs \
            -n jenkins-centos7-x86-64 -t centos -- \
            -R 7 -a x86_64

        Host CPE ID from /etc/os-release:
        'yum' command is missing
        lxc-create: jenkins-centos7-x86-64: lxccontainer.c:
          create_run_template: 1616 Failed to create container from template
        lxc-create: jenkins-centos7-x86-64: tools/lxc_create.c:
          main: 319 Failed to create container jenkins-centos7-x86-64

    Note also that with such "third-party" distributions you may face other issues; for example, the CentOS helper did not generate some fields in the `config` file that were needed for conversion into libvirt "domxml" (as found by trial and error, and comparison to other `config` files):

        lxc.uts.name = jenkins-centos7-x86-64
        lxc.arch = x86_64

    Also note the container/system naming without underscore in "x86_64" —  the deployed system discards the character when assigning its hostname. Using "amd64" is another reasonable choice here.

  - For Arch Linux you would need `pacman` tools on the host system, so see https://wiki.archlinux.org/title/Install_Arch_Linux_from_existing_Linux#Using_pacman_from_the_host_system for details. On a Debian/Ubuntu host, assumed ready for NUT builds per Prerequisites for building NUT on different OSes (or `docs/config-prereqs.txt` in NUT sources for up-to-date information), it would start like this:

        :; apt-get update
        :; apt-get install meson ninja-build cmake

        # Some dependencies for pacman itself; note there are several libcurl builds;
        # pick another if your system constraints require you to:
        :; apt-get install libarchive-dev libcurl4-nss-dev gpg libgpgme-dev

        :; git clone https://gitlab.archlinux.org/pacman/pacman.git
        :; cd pacman

        # Iterate something like this until all needed dependencies fall
        # into line (note libdir for your host architecture):
        :; rm -rf build; mkdir build && meson build --libdir=/usr/lib/x86_64-linux-gnu

        :; ninja -C build
        # Depending on your asciidoc version, it may require that `--asciidoc-opts` are
        # passed as equation to a vale (not space-separated from it). Then apply this:
        #   diff --git a/doc/meson.build b/doc/meson.build
        #   -      '--asciidoc-opts', ' '.join(asciidoc_opts),
        #   +      '--asciidoc-opts='+' '.join(asciidoc_opts),
        # and re-run (meson and) ninja.

        # Finally when all succeeded:
        :; sudo ninja -C build install
        :; cd

    You will also need `pacstrap` and Debian `arch-install-scripts` package does not deliver it. It is however simply achieved:

        :; git clone https://github.com/archlinux/arch-install-scripts
        :; cd arch-install-scripts
        :; make && sudo make PREFIX=/usr install
        :; cd

    It will also want an `/etc/pacman.d/mirrorlist` which you can populate for your geographic location from https://archlinux.org/mirrorlist/ service, or just fetch them all (don’t forget to uncomment some `Server =` lines):

        :; mkdir -p /etc/pacman.d/
        :; curl https://archlinux.org/mirrorlist/all/ > /etc/pacman.d/mirrorlist

    And to reference it from your host `/etc/pacman.conf` by un-commenting the `[core]` section and `Include` instruction, as well as adding `[community]` and `[extra]` sections with same reference, e.g.:

        [core]
        ### SigLevel = Never
        SigLevel = PackageRequired
        Include = /etc/pacman.d/mirrorlist

        [extra]
        ### SigLevel = Never
        SigLevel = PackageRequired
        Include = /etc/pacman.d/mirrorlist

        [community]
        ### SigLevel = Never
        SigLevel = PackageRequired
        Include = /etc/pacman.d/mirrorlist

    And just then you can proceed with LXC:

        :; lxc-create -P /srv/libvirt/rootfs \
            -n jenkins-archlinux-amd64 -t archlinux -- \
            -a x86_64 -P openssh,sudo

    In my case, it had problems with GPG keyring missing (using one in host system, as well as the package cache outside the container, it seems) so I had to run `pacman-key --init; pacman-key --refresh-keys` on the host itself. Even so, `lxc-create` complained about updating some keyring entries and I had to go one by one picking key servers (serving different metadata) like this:

        :; pacman-key --keyserver keyserver.ubuntu.com --recv-key 6D1655C14CE1C13E

    In the worst case, see `SigLevel = Never` for `pacman.conf` to not check package integrity (seems too tied into thinking that host OS is Arch)…

    It seems that pre-fetching the package databases with `pacman -Sy` on the host was also important.

##### Initial container-related setup

- Add the "name,IP" line for this container to `/etc/lxc/dnsmasq-hosts.conf` on the host, e.g.:

      jenkins-debian11-mips,10.0.3.245

  ### Note

  Don’t forget to eventually `systemctl restart lxc-net` to apply the new host reservation!

- Convert a pure LXC container to be managed by LIBVIRT-LXC (and edit config markup on the fly — e.g. fix the LXC `dir:/` URL schema):

      :; virsh -c lxc:///system domxml-from-native lxc-tools \
          /srv/libvirt/rootfs/jenkins-debian11-armhf/config \
          | sed -e 's,dir:/srv,/srv,' \
          > /tmp/x && virsh define /tmp/x

  ### Note

  You may want to tune the default generic 64MB RAM allocation, so your launched QEMU containers are not OOM-killed as they exceeded their memory `cgroup` limit. In practice they do not eat that much resident memory, just want to have it addressable by VMM, I guess (swap is not very used either), at least not until active builds start (and then it depends on compiler appetite and `make` program parallelism level you allow, e.g. by pre-exporting `MAXPARMAKES` environment variable for `ci_build.sh`, and on the number of Jenkins "executors" assigned to the build agent).

  - It may be needed to revert the generated "os/arch" to `x86_64` (and let QEMU handle the rest) in the `/tmp/x` file, and re-try the definition:

        :; virsh define /tmp/x

- Then execute `virsh edit jenkins-debian11-armhf` (and same for other containers) to bind-mount the common `/home/abuild` location, adding this tag to their "devices":

          <filesystem type='mount' accessmode='passthrough'>
            <source dir='/home/abuild'/>
            <target dir='/home/abuild'/>
          </filesystem>

  - Note that generated XML might not conform to current LXC schema, so it fails validation during save; this can be bypassed with `i` when it asks. One such case was however with indeed invalid contents, the "dir:" schema removed by example above.

#### Shepherd the herd

- Monitor deployed container rootfs’es with:

      :; du -ks /srv/libvirt/rootfs/*

  (should have non-trivial size for deployments without fatal infant errors)

- Mass-edit/review libvirt configurations with:

      :; virsh list --all | awk '{print $2}' \
         | grep jenkins | while read X ; do \
           virsh edit --skip-validate $X ; done

  - …or avoid `--skip-validate` when markup is initially good :)

- Mass-define network interfaces:

      :; virsh list --all | awk '{print $2}' \
         | grep jenkins | while read X ; do \
           virsh dumpxml "$X" | grep "bridge='lxcbr0'" \
           || virsh attach-interface --domain "$X" --config \
              --type bridge --source lxcbr0 ; \
         done

- Verify that unique MAC addresses were defined (e.g. `00:16:3e:00:00:01` tends to pop up often, while `52:54:00:xx:xx:xx` are assigned to other containers); edit the domain definitions to randomize, if needed:

      :; grep 'mac add' /etc/libvirt/lxc/*.xml | awk '{print $NF" "$1}' | sort

- Make sure at least one console device exists (end of file, under the network interface definition tags), e.g.:

          <console type='pty'>
            <target type='lxc' port='0'/>
          </console>

- Populate with `abuild` account, as well as with the `bash` shell and `sudo` ability, reporting of assigned IP addresses on the console, and SSH server access complete with envvar passing from CI clients by virtue of `ssh -o SendEnv='*' container-name`:

      :; for ALTROOT in /srv/libvirt/rootfs/*/rootfs/ ; do \
          echo "=== $ALTROOT :" >&2; \
          grep eth0 "$ALTROOT/etc/issue" || ( printf '%s %s\n' \
              '\S{NAME} \S{VERSION_ID} \n \l@\b ;' \
              'Current IP(s): \4{eth0} \4{eth1} \4{eth2} \4{eth3}' \
              >> "$ALTROOT/etc/issue" ) ; \
          grep eth0 "$ALTROOT/etc/issue.net" || ( printf '%s %s\n' \
              '\S{NAME} \S{VERSION_ID} \n \l@\b ;' \
              'Current IP(s): \4{eth0} \4{eth1} \4{eth2} \4{eth3}' \
              >> "$ALTROOT/etc/issue.net" ) ; \
          groupadd -R "$ALTROOT" -g 399 abuild ; \
          useradd -R "$ALTROOT" -u 399 -g abuild -M -N -s /bin/bash abuild \
          || useradd -R "$ALTROOT" -u 399 -g 399 -M -N -s /bin/bash abuild \
          || { if ! grep -w abuild "$ALTROOT/etc/passwd" ; then \
                  echo 'abuild:x:399:399::/home/abuild:/bin/bash' \
                  >> "$ALTROOT/etc/passwd" ; \
                  echo "USERADDed manually: passwd" >&2 ; \
               fi ; \
               if ! grep -w abuild "$ALTROOT/etc/shadow" ; then \
                  echo 'abuild:!:18889:0:99999:7:::' >> "$ALTROOT/etc/shadow" ; \
                  echo "USERADDed manually: shadow" >&2 ; \
               fi ; \
             } ; \
          if [ -s "$ALTROOT/etc/ssh/sshd_config" ]; then \
              grep 'AcceptEnv \*' "$ALTROOT/etc/ssh/sshd_config" || ( \
                  ( echo "" ; \
                    echo "# For CI: Allow passing any envvars:"; \
                    echo 'AcceptEnv *' ) \
                  >> "$ALTROOT/etc/ssh/sshd_config" \
              ) ; \
          fi ; \
         done

  Note that for some reason, in some of those other-arch distros `useradd` fails to find the group anyway; then we have to "manually" add them.

- Let the host know and resolve the names/IPs of containers you assigned:

      :; grep -v '#' /etc/lxc/dnsmasq-hosts.conf \
         | while IFS=, read N I ; do \
          getent hosts "$N" >&2 || echo "$I $N" ; \
         done >> /etc/hosts

#### Further setup of the containers

See Prerequisites for building NUT on different OSes (or `docs/config-prereqs.txt` in NUT sources for up-to-date information) about dependency package installation for Debian-based Linux systems.

It may be wise to not install e.g. documentation generation tools (or at least not the full set for HTML/PDF generation) in each environment, in order to conserve space and run-time stress.

Still, if there are significant version outliers (such as using an older distribution due to vCPU requirements), it can be installed fully just to ensure non-regression — e.g. that when adapting `Makefile` rule definitions or compiler arguments to modern toolkits, we do not lose the ability to build with older ones.

For this, `chroot` from the host system can be used, e.g. to improve the interactive usability for a population of Debian(-compatible) containers (and to use its networking, while the operating environment in containers may be not yet configured or still struggling to access the Internet):

    :; for ALTROOT in /srv/libvirt/rootfs/*/rootfs/ ; do \
        echo "=== $ALTROOT :" ; \
        chroot "$ALTROOT" apt-get install \
            sudo bash vim mc p7zip p7zip-full pigz pbzip2 git \
       ; done

Similarly for `yum`-managed systems (CentOS and relatives), though specific package names can differ, and additional package repositories may need to be enabled first (see Prerequisites for building NUT on different OSes (or `docs/config-prereqs.txt` in NUT sources for up-to-date information) for more details such as recommended package names).

Note that technically `(sudo) chroot ...` can also be used from the CI worker account on the host system to build in the prepared filesystems without the overhead of running containers as complete operating environments with any standard services and several copies of Jenkins `agent.jar` in them.

Also note that externally-driven set-up of some packages, including the `ca-certificates` and the JDK/JRE, require that the `/proc` filesystem is usable in the chroot environment. This can be achieved with e.g.:

    :; for ALTROOT in /srv/libvirt/rootfs/*/rootfs/ ; do \
        for D in proc ; do \
          echo "=== $ALTROOT/$D :" ; \
          mkdir -p "$ALTROOT/$D" ; \
          mount -o bind,rw "/$D" "$ALTROOT/$D" ; \
        done ; \
       done

TODO: Test and document a working NAT and firewall setup for this, to allow SSH access to the containers via dedicated TCP ports exposed on the host.

##### Arch Linux containers

Arch Linux containers prepared by procedure above include only a minimal footprint, and if you missed the `-P pkg,list` argument, they can lack even an SSH server. Suggestions below assume this path to container:

    :; ALTROOT=/srv/libvirt/rootfs/jenkins-archlinux-amd64/rootfs/

Let `pacman` know current package database:

    :; grep 8.8.8.8 $ALTROOT/etc/resolv.conf || (echo 'nameserver 8.8.8.8' > $ALTROOT/etc/resolv.conf)
    :; chroot $ALTROOT pacman -Syu
    :; chroot $ALTROOT pacman -S openssh sudo
    :; chroot $ALTROOT systemctl enable sshd
    :; chroot $ALTROOT systemctl start sshd

This may require that you perform bind-mounts above, as well as "passthrough" the `/var/cache/pacman/pkg` from host to guest environment (in `virsh edit`, and bind-mount for `chroot` like for `/proc` et al above).

It is possible that `virsh console` would serve you better than `chroot`. Note you may have to first `chroot` to set the `root` password anyhow.

#### Troubleshooting

- Q: Container won’t start, its `virsh console` says something like:

      Failed to create symlink /sys/fs/cgroup/net_cls: Operation not permitted

  A: According to https://bugzilla.redhat.com/show_bug.cgi?id=1770763 (skip to the end for summary) this can happen when a newer Linux host system with `cgroupsv2` capabilities runs an older guest distro which only knows about `cgroupsv1`, such as when hosting a CentOS 7 container on a Debian 11 server.

  - One workaround is to ensure that the guest `systemd` does not try to "join" host facilities, by setting an explicit empty list for that:

        :; echo 'JoinControllers=' >> "$ALTROOT/etc/systemd/system.conf"

  - Another approach is to upgrade `systemd` related packages in the guest container. This may require additional "backport" repositories or similar means, possibly maintained not by distribution itself but by other community members, and arguably would logically compromise the idea of non-regression builds in the old environment "as is".

- Q: Server was set up with ZFS as recommended, and lots of I/O hit the disk even when application writes are negligible

  A: This was seen on some servers and generally derives from data layout and how ZFS maintains the tree structure of blocks. A small application write (such as a new log line) means a new empty data block allocation, an old block release, and bubble up through the whole metadata tree to complete the transaction (grouped as TXG to flush to disk).

  - One solution is to use discardable build workspaces in RAM-backed storage like `/dev/shm` (`tmpfs`) on Linux, or `/tmp` (`swap`) on illumos hosting systems, and only use persistent storage for the home directory with `.ccache` and `.gitcache-dynamatrix` directories.
  - Another solution is to reduce the frequency of TXG sync from modern default of 5 sec to conservative 30-60 sec. Check how to set the `zfs_txg_timeout` on your platform.

### 5.5. Connecting Jenkins to the containers

To properly cooperate with the jenkins-dynamatrix project driving regular NUT CI builds, each build environment should be exposed as an individual agent with labels describing its capabilities.

#### Agent Labels

With the `jenkins-dynamatrix`, agent labels are used to calculate a large "slow build" matrix to cover numerous scenarios for what can be tested with the current population of the CI farm, across operating systems, `make`, shell and compiler implementations and versions, and C/C++ language revisions, to name a few common "axes" involved.

##### Labels for QEMU

Emulated-CPU container builds are CPU-intensive, so for them we define as few capabilities as possible: here CI is more interested in checking how binaries behave on those CPUs, not in checking the quality of recipes (distcheck, Make implementations, etc.), shell scripts or documentation, which is more efficient to test on native platforms.

Still, we are interested in results from different compiler suites, so specify at least one version of each.

### Note

Currently the NUT `Jenkinsfile-dynamatrix` only looks at various `COMPILER` variants for `qemu-nut-builder` use-cases, disregarding the versions and just using one that the environment defaults to.

The reduced set of labels for QEMU workers looks like:

    qemu-nut-builder qemu-nut-builder:alldrv
    NUT_BUILD_CAPS=drivers:all NUT_BUILD_CAPS=cppunit
    OS_FAMILY=linux OS_DISTRO=debian11 GCCVER=10 CLANGVER=11
    COMPILER=GCC COMPILER=CLANG
    ARCH64=ppc64le ARCH_BITS=64

##### Labels for native builds

For contrast, a "real" build agent’s set of labels, depending on presence or known lack of some capabilities, looks something like this:

    doc-builder nut-builder nut-builder:alldrv
    NUT_BUILD_CAPS=docs:man NUT_BUILD_CAPS=docs:all
    NUT_BUILD_CAPS=drivers:all NUT_BUILD_CAPS=cppunit=no
    OS_FAMILY=bsd OS_DISTRO=freebsd12 GCCVER=10 CLANGVER=10
    COMPILER=GCC COMPILER=CLANG
    ARCH64=amd64 ARCH_BITS=64
    SHELL_PROGS=sh SHELL_PROGS=dash SHELL_PROGS=zsh SHELL_PROGS=bash
    SHELL_PROGS=csh SHELL_PROGS=tcsh SHELL_PROGS=busybox
    MAKE=make MAKE=gmake
    PYTHON=python2.7 PYTHON=python3.8

#### Generic agent attributes

- Name: e.g. `ci-debian-altroot--jenkins-debian10-arm64` (note the pattern for "Conflicts With" detailed below)

- Remote root directory: preferably unique per agent, to avoid surprises; e.g.: `/home/abuild/jenkins-nut-altroots/jenkins-debian10-armel`

  - Note it may help that the system home directory itself is shared between co-located containers, so that the `.ccache` or `.gitcache-dynamatrix` are available to all builders with identical contents
  - If RAM permits, the Jenkins Agent working directory may be placed in a temporary filesystem not backed by disk (e.g. `/dev/shm` on modern Linux distributions); roughly estimate 300Mb per executor for NUT builds.
- Usage: "Only build jobs with label expressions matching this node"

- Node properties / Environment variables:

  - `PATH+LOCAL` ⇒ `/usr/lib/ccache`

#### Where to run agent.jar

Depending on circumstances of the container, there are several options available to the NUT CI farm:

- Java can run in the container, efficiently (native CPU, different distro) ⇒ the container may be exposed as a standalone host for direct SSH access (usually by NAT, exposing SSH on a dedicated port of the host; or by first connecting the Jenkins controller with the host as an SSH Build Agent, and then calling SSH to the container as a prefix for running the agent; or by using Jenkins Swarm agents), so ultimately the build `agent.jar` JVM would run in the container. Filesystem for the `abuild` account may be or not be shared with the host.
- Java can not run in the container (crashes on emulated CPU, or is too old in the agent container’s distro — currently Jenkins requires JRE 17+, but eventually will require 21+) ⇒ the agent would run on the host, and then the host would `ssh` or `chroot` (networking not required, but bind-mount of `/home/abuild` and maybe other paths from host would be needed) called for executing `sh` steps in the container environment. Either way, home directory of the `abuild` account is maintained on the host and shared with the guest environment, user and group IDs should match.
- Java is inefficient in the container (operations like un-stashing the source succeed but take minutes instead of seconds) ⇒ either of the above

### Note

As time moves on and Jenkins core and its plugins get updated, support for some older run-time features of the build agents can get removed (e.g. older Java releases, older Git tooling). While there are projects like Temurin that provide Java builds for older systems, at some point a switch to "Jenkins agent on new host going into older build container" approach can become unavoidable. One clue to look at in build logs is failure messages like:

    Caused by: java.lang.UnsupportedClassVersionError:
      hudson/slaves/SlaveComputer$SlaveVersion has been compiled by a more
      recent version of the Java Runtime (class file version 61.0), this version
      of the Java Runtime only recognizes class file versions up to 55.0

##### Using Jenkins SSH Build Agents

This is a typical use-case for tightly integrated build farms under common management, where the Jenkins controller can log by SSH into systems which act as its build agents. It injects and launches the `agent.jar` to execute child processes for the builds, and maintains a tunnel to communicate.

Methods below involving SSH assume that you have configured a password-less key authentication from the host machine to the `abuild` account in each guest build environment container. This can be an `ssh-keygen` result posted into `authorized_keys`, or a trusted key passed by a chain of ssh agents from a Jenkins Credential for connection to the container-hoster into the container. The private SSH key involved may be secured by a pass-phrase, as long as your Jenkins Credential storage knows it too. Note that for the approaches explored below, the containers are not directly exposed for log-in from any external network.

- For passing the agent through an SSH connection from host to container, so that the `agent.jar` runs inside the container environment, configure:

  - Launch method: "Agents via SSH"

  - Host, Credentials, Port: as suitable for accessing the container-hoster

    ### Note

    The container-hoster should have accessed the guest container from the account used for intermediate access, e.g. `abuild`, so that its `.ssh/known_hosts` file would trust the SSH server on the container.

  - Prefix Start Agent Command: content depends on the container name, but generally looks like the example below to report some info about the final target platform (and make sure `java` is usable) in the agent’s log. Note that it ends with un-closed quote and a space char:

        ssh jenkins-debian10-amd64 '( java -version & uname -a ; getconf LONG_BIT; getconf WORD_BIT; wait ) &&

  - Suffix Start Agent Command: a single quote to close the text opened above:

    '

- The other option is to run the `agent.jar` on the host, for all the network and filesystem magic the agent does, and only execute shell steps in the container. The solution relies on overridden `sh` step implementation in the `jenkins-dynamatrix` shared library that uses a magic `CI_WRAP_SH` environment variable to execute a pipe into the container. Such pipes can be `ssh` or `chroot` with appropriate host setup described above.

  ### Note

  In case of ssh piping, remember that the container’s `/etc/ssh/sshd_config` should `AcceptEnv *` and the SSH server should be restarted after such configuration change.

  - Launch method: "Agents via SSH"
  - Host, Credentials, Port: as suitable for accessing the container-hoster

  - Prefix Start Agent Command: content depends on the container name, but generally looks like the example below to report some info about the final target platform (and make sure it is accessible) in the agent’s log. Note that it ends with a space char, and that the command here should not normally print anything into stderr/stdout (this tends to confuse the Jenkins Remoting protocol):

        echo PING > /dev/tcp/jenkins-debian11-ppc64el/22 &&

  - Suffix Start Agent Command: empty

- Node properties / Environment variables:

  - `CI_WRAP_SH` ⇒

        ssh -o SendEnv='*' "jenkins-debian11-ppc64el" /bin/sh -xe

##### Using Jenkins Swarm Agents

This approach allows remote systems to participate in the NUT CI farm by dialing in and so defining an agent. A single contributing system may be running a number of containers or virtual machines set up following the instructions above, and each of those would be a separate build agent.

Such systems should be "dedicated" to contribution in the sense that they should be up and connected for days, and sometimes tasks would land.

Configuration files maintained on the Swarm Agent system dictate which labels or how many executors it would expose, etc. Credentials to access the NUT CI farm Jenkins controller to register as an agent should be arranged with the farm maintainers, and currently involve a GitHub account with Jenkins role assignment for such access, and a token for authentication.

The jenkins-swarm-nutci repository contains example code from such setup with a back-up server experiment for the NUT CI farm, including auto-start method scripts for Linux systemd and upstart, illumos SMF, and OpenBSD rcctl.

#### Sequentializing the stress

##### Running one agent at a time

Another aspect of farm management is that emulation is a slow and intensive operation, so we can not run all agents and execute builds at the same time.

The current solution relies on the Conflict-Aware On Demand Retention Strategy plugin to allow co-located build agents to "conflict" with each other — when one picks up a job from the queue, it blocks neighbors from starting; when it is done, another may start.

Containers can be configured with "Availability ⇒ On demand", with shorter cycle to switch over faster (the core code sleeps a minute between attempts):

- In demand delay: `0`;
- Idle delay: `0` (Jenkins may change it to `1`);
- Conflicts with: `^ci-debian-altroot--.*$` assuming that is the pattern for agent definitions in Jenkins — not necessarily linked to hostnames.

Also, the "executors" count should be reduced to the amount of compilers in that system (usually 2) and so avoid extra stress of scheduling too many emulated-CPU builds at once.

##### Sequentializing the git cache access

As part of the `jenkins-dynamatrix` optional optimizations, the NUT CI recipe invoked via `Jenkinsfile-dynamatrix` maintains persistent git reference repositories that can be used to cache NUT codebase (including the tested commits) and so considerably speed up workspace preparation when running numerous build scenarios on the same agent.

Such `.gitcache-dynamatrix` cache directories are located in the build workspace location (unique for each agent), but on a system with numerous containers these names can be symlinks pointing to a shared location.

To avoid collisions with several executors updating the same cache with new commits, critical access windows are sequentialized with the use of Lockable Resources plugin. On the `jenkins-dynamatrix` side this is facilitated by labels:

    DYNAMATRIX_UNSTASH_PREFERENCE=scm-ws:nut-ci-src
    DYNAMATRIX_REFREPO_WORKSPACE_LOCKNAME=gitcache-dynamatrix:SHARED_HYPERVISOR_NAME

- The `DYNAMATRIX_UNSTASH_PREFERENCE` tells the `jenkins-dynamatrix` library code which checkout/unstash strategy to use on a particular build agent (following values defined in the library; `scm-ws` means SCM caching under the agent workspace location, `nut-ci-src` names the cache for this project);
- The `DYNAMATRIX_REFREPO_WORKSPACE_LOCKNAME` specifies a semi-unique string: it should be same for all co-located agents which use the same shared cache location, e.g. guests on the same hypervisor; and it should be different for unrelated cache locations, e.g. different hypervisors and stand-alone machines.

Prev |  | Next
| Home |