Prev |  | Next

## 4. Continuous Integration (NUT CI farm) technologies

### 4.1. CI Farm configuration notes

### Note

This chapter contains information about NUT CI farm setup tricks that were applied at different times by the maintainer team to ensure regular builds and tests of the codebase. Whether these are used in daily production today or not, similar setup should be possible locally on developer and contributor machines.

### 4.2. Multiple FOSS CI providers and technologies

While there are many FOSS-friendly CI offerings, they are usually (and reasonably) focused on the OS market leaders — offering recent releases of Linux, Windows and MacOS build agents, and sometimes a way to "bring your own device" to cover other systems. The NUT CI farm does benefit from those offerings as well, using GitHub Actions with CodeQL for Linux code quality inspection, AppVeyor CI for Windows, and CircleCI for MacOS, to name a few.

But on the other hand, being a massively multi-platform effort (and aiming to support older boxes that are still alive even if their vendors and/or distro versions are not), a comprehensive NUT CI approach requires many machines running uncommon operating systems. This is where custom virtual machines help, and more so — a core set of those hosted in the cloud and dedicated to the project, rather than only some resources intermittently contributed by community members which come and go.

### Note

Community-provided builders running on further systems are also welcome, and the option is of course supported, as managed by the Jenkins-Dynamatrix effort which appeared due to such need, and runs the core NUT CI farm.

We have also had historic experience with FOSS CI providers (and community members' machines) disappearing, so having NUT CI farm goals covered by multiple independent implementations is also a feature beyond having yet another set of digital eyes looking at our code quality (which is also a goal in itself).

### 4.3. Jenkins is the way

- https://stories.jenkins.io/user-story/jenkins-is-the-way-for-networkupstools/
- https://github.com/jenkins-infra/stories/blob/main/src/user-story/jenkins-is-the-way-for-networkupstools/index.yaml

#### The jenkins-dynamatrix library

FIXME: Write the chapter text.

For now, see https://github.com/networkupstools/jenkins-dynamatrix sources (note the README and large comments at start of files may be obsolete, as of this writing — documenting the initial ideas, but the implementation might differ from that over time).

#### Jenkinsfile-dynamatrix cases in NUT sources

FIXME: Write the chapter text.

For now, see the `Jenkinsfile-dynamatrix` in the NUT sources (maybe only git), e.g. https://github.com/networkupstools/nut/blob/master/Jenkinsfile-dynamatrix for the practical pipeline preparation and hand-off to library implementation.

#### Jenkins CI

Since mid-2021, the NUT CI farm is implemented by several virtual servers courteously provided originally by Fosshost and later by DigitalOcean.

These run various operating systems as build agents, and a Jenkins instance to orchestrate the builds of NUT branches and pull requests on those agents.

This is driven by `Jenkinsfile-dynamatrix` and a Jenkins Shared Library called jenkins-dynamatrix which prepares a matrix of builds across as many operating systems, bitnesses/architectures, compilers, make programs and C/C++ revisions as it can — based on the population of currently available build agents and capabilities which they expose as agent labels.

This hopefully means that people interested in NUT can contribute to the build farm (and ensure NUT is and remains compatible with their platform) by running a Jenkins Swarm agent with certain labels, which would dial into https://ci.networkupstools.org/ controller. Please contact the NUT maintainer if you want to participate in this manner.

The `Jenkinsfile-dynamatrix` recipe allows NUT CI farm to run different sets of build scenarios based on various conditions, such as the name of branch being built (or PR’ed against), changed files (e.g. C/C++ sources vs. just docs), and some build combinations may be not required to succeed.

For example, the main development branch and pull requests against it must cleanly pass all specified builds and tests on various platforms with the default level of warnings specified in the `configure` script. These are balanced to not run too many build scenarios overall, but just a quick and sufficiently representative set.

As another example, there is special handling for "fightwarn" pattern in the branch names to run many more builds with varying warning levels and more variants of intermediate language revisions, and so expose concerns deliberately missed by default warnings levels in "master" branch builds (the bar moves over time, as some classes of warnings become extinct from our codebase).

Further special handling for branches named like `fightwarn.*89.*` regex enables more intensive warning levels for a GNU89 build specifically (which are otherwise disabled as noisy yet not useful for supported C99+ builds), and is intended to help develop fixes for support of this older language revision, if anyone would dare.

Many of those unsuccessful build stages are precisely the focus of the "fightwarn" effort, and are currently marked as "may fail", so they end up as "UNSTABLE" (seen as orange bubbles in the Jenkins BlueOcean UI, or orange cells in the tabular list of stages in the legacy UI), rather than as "FAILURE" (red bubbles) for build scenarios that were not expected to fail and usually represent higher-priority problems that would block a PR.

Developers whose PR builds (or attempts to fix warnings) did not succeed in some cell of such build matrix, can look at the individual logs of that cell. Beside indication from the compiler about the failure, the end of log text includes the command which was executed by CI worker and can be reproduced locally by the developer, e.g.:

    22:26:01  FINISHED with exit-code 2 cmd:  (
    22:26:01  [ -x ./ci_build.sh ] || exit
    22:26:01
    22:26:01  eval BUILD_TYPE="default-alldrv" BUILD_WARNOPT="hard" \
        BUILD_WARNFATAL="yes" MAKE="make"  CC=gcc-10 CXX=g++-10 \
        CPP=cpp-10 CFLAGS='-std=gnu99 -m64' CXXFLAGS='-std=gnu++11 -m64' \
        LDFLAGS='-m64' ./ci_build.sh
    22:26:01  )

or for autotools-driven scenarios (which prep, configure, build and test in separate stages — so for reproducing a failed build you should also look at its configuration step separately):

    22:28:18  FINISHED with exit-code 0 cmd:  ( [ -x configure ] || exit; \
        eval  CC=clang-9 CXX=clang++-9 CPP=clang-cpp-9 CFLAGS='-std=c11 -m64' \
        CXXFLAGS='-std=c++11 -m64' LDFLAGS='-m64' time ./configure )

To re-run such scenario locally, you can copy the line from `eval` (but without the `eval` keyword itself) up to and including the executed script or tool, into your shell. Depending on locally available compilers, you may have to tweak the `CC`, `CXX` and `CPP` arguments; note that a `CPP` may be specified as `/path/to/CC -E` for GCC and CLANG based toolkits at least, if they lack a standalone preprocessor program (e.g. IntelCC).

### Note

While NUT recipes do not currently recognize a separate `CXXCPP`, it would follow similar semantics.

Some further details about the NUT CI farm workers are available in Prerequisites for building NUT on different OSes (or `docs/config-prereqs.txt` in NUT sources for up-to-date information) and Custom NUT CI farm build agents: LXC multi-arch containers (or `docs/ci-farm-lxc-setup.txt` in NUT sources for up-to-date information) documentation.

### 4.4. AppVeyor CI

Primarily used for building NUT for Windows on Windows instances provided in the cloud — and so ensure non-regression as well as downloadable archives with binary installation prototype area, intended for enthusiastic testing (proper packaging to follow). NUT for Windows build-ability was re-introduced soon after NUT 2.8.0 release.

This relies on a few prerequisite packages and a common NUT configuration, as coded in the `appveyor.yml` file in the NUT codebase.

### 4.5. CircleCI

Primarily used for building NUT for MacOS on instances provided in the cloud, and so ensure non-regression across several Xcode releases.

This relies on a few prerequisite packages and a common NUT configuration, as coded in the `.circleci/config.yml` file in the NUT codebase.

### 4.6. Travis CI

See the `.travis.yml` file in project sources for a detailed list of third party dependencies and a large matrix of `CFLAGS` and compiler versions last known to work or to not (yet) work on operating systems available to that CI solution.

### Note

The cloud Travis CI offering became effectively defunct for open-source projects in mid-2021, so the `.travis.yml` file in NUT codebase is not actively maintained.

Local private deployments of Travis CI are possible, so if anybody does use it and has updated markup to share, they are welcome to post PRs.

The NUT project on GitHub had integration with Travis CI to test a large set of compiler and option combinations, covering different versions of gcc and clang, C standards, and requiring to pass builds at least in a mode without warnings (and checking the other cases where any warnings are made fatal).

### 4.7. CodeQL

(Earlier this role was performed by LGTM.com) Run GitHub Actions for static analysis of C, C++ and Python code and recipes, to produce suggestions based on common coding flaws and best-practice security patterns.

Prev |  | Next
| Home |