- Table of Contents
- Previous Chapter
- Next Chapter

# Chapter 5: Running `stealth'

Now that stealth has been compiled, the construction of a policy file has been covered, and a service-account on the client has been defined, what must be done to run stealth in practice?

Here's what remains to be done:

- Install stealth at a proper location
- Construct one or more policy files
- Learn to interpret stealth's output.
- Optionally, automate the removal of old log-files.
- Determine a schedule for running stealth automatically, e.g. using cron(1) or ssh-cron(1)

In this chapter, these topics are discussed.

## 5.1: Installing `stealth'

As stealth is mainly a system administrator's tool, it could be installed in `/usr/bin`. In that case, do (as root) in the directory where stealth was compiled/unpacked:

        ./build install program

Alternatively, another default location may be specified in the `INSTALL.im` file or may be provided to the `build` script. E.g.,

        ./build install program /usr/local/bin/stealth

installing the binary program as `/usr/local/bin/stealth`.

The provided `icmake build` script can be started without arguments for an overview of possible commands.

## 5.2: Stealth command-line and policy file options

Short options are provided between parentheses, immediately following their long option equivalents.

Option descriptions showing (C) can only be used on the command-line, and are ignored when specified in the second section of the policy file.

In the overview of options ``<uds>`' represents the name of the Unix Domain Socket to use, and ``<file-spec>`' refers to a (relative or absolute) specification of a file location.

With the first and second synopses relative locations (of the Unix Domain Socket and of other file-specifications) are interpreted relative to the current working directory.

Command-line options overrule options defined in the policy-file.

- `--daemon (-d) <uds>`: (C) run as background (daemon) process. When the Stealth daemon process is started, the Unix Domain Socket (tt<uds>) may not already exist.
- `--dry-run`: (C) no integrity scans or reloads are performed, but are assumed OK. Remaining tasks are normally performed;
- `--help (-h)`: (C) Display help information and exit;
- `--log (-L) <file-spec>`: log messages are appended to `file-spec'. If file-spec does not exist, it is first created;
- `--logmail`: mail sent by stealth is logged (requires `--log` or `--syslog`);
- `--max-size <size>[BKMG]`: files retrieved by `GET` commands may at most have `<size>` bytes (B), KBytes (K), MBytes (M), GBytes (G). The default size is 10M, the default unit is B.
- `--no-mail`: mail is not sent. By default mail is sent as configured in the policy-file (`--logmail` can be specified independently from `--no-mail`);
- `--parse-policy-file (-p)`: (C) parse the policy file, after which stealth ends.
  Specify once to see the numbered commands;
  twice to see the policy file parsing steps as well.
  Results are written to the std. output.
- `--ping <uds>`: (C) performs no actions, but is used to verify that a stealth daemon can be reached via its Unix Domain Socket (`<uds>`). The daemon will respond even if it's currently performing an integrity scan. It is used by the `/usr/bin/stealthcron` script to verify that a stealth daemon is alive.
- `--random-interval (-i) <interval>[m]>`: start the scan a random interval of <interval> seconds (or minutes if an `m' is appended (no blanks) to <interval>) following the delay specified at `--repeat` (see below). This option requires specification of the `--repeat` and `--daemon` options;
- `--reload <uds>`: (C) reloads the configuration and skip-files and restarts the scan of the stealth daemon process. Options defined in the policy file are also reloaded. However, command-line options always take priority over options defined in the policy file, so when command-line options were used when starting stealth in daemon mode, they cannot be modified by reloading the policy file.
- `--repeat <seconds>`: wake up and perform an integrity scan at interrupts or after `<seconds>` seconds (or minutes if an `m' is appended (no blanks) to <seconds>) after completing the previous integrity scan. The option `--random-interval` can be used to add a random delay to `<seconds>` until the next integrity scan is performed. This option requires specification of the and `--daemon` option;
- `--rerun <uds>`: (C) start executing the integrity scan commands that are specified in the stealth daemon process's policy file;
- `--resume <uds>`: (C) resume a suspended stealth process, implies `--rerun`;
- `--run-command (-r) <nr>`: (C) Only execute command number <nr> (natural number). Command numbers are shown by stealth `---parse-policy-file`. This option can only be specified using the second synopsis;
- `--skip-files (-s) <file-spec>`: all entries in `<file-spec>` are skipped. Their integrity is not monitored. If an entry is already present in a log file then stealth once generates an `IGNORING` message in the mail sent to the address specified at `EMAIL` in the policy file. Each entry mentioned in `file-spec` must be on a line of its own and must be specified using absolute file paths. Entries ending in a slash are assumed to be directories whose full contents must be skipped. Other entries are interpreted as the names of files to skip. Initial and trailing blanks, empty lines and lines having a `#` as their 1st non blank character are ignored. Here are some examples:

              # skip all files in user's Mail directory
          /home/user/Mail/
              # skip user's .history file
          /home/user/.history

- `--stdout (-o)`: messages are (also) written to the std. output stream (only available with the second synopsis);
- `--suspend <uds>`: (C) suspends a currently active stealth process. Following `--suspend` use `--resume` to re-activate an stealth daemon or `--terminate` to end an stealth daemon;
- `--syslog`: write syslog messages;
- `--syslog-facility <facility>`: syslog facility to use. By default facility DAEMON is used;
- `--syslog-priority <priority>`: syslog priority to use. By default priority NOTICE is used;
- `--syslog-tag <tag>`: `<tag>` specifies the identifier that is prefixed to syslog messages. By default the tag `STEALTH' is used, see also the next section;
- `--terminate <uds>`: (C) terminate a currently active stealth process;
- `--time-stamp (-t) <type>`: the time-stamps to use. By default UTC. To use the local time specify `--time-stamp LT`. The `--time-stamp` option does not apply to time-stamps generated by syslog (see also the next section);
- `--usage`: (C) Display help information and exit;
- `--verbosity <value>`: determines the amount of logged information. Requires options `--log` or `--syslog`. Possible values are:
  0: nothing is logged
  1: (default) mode reports and policy commands
  2: also: ipc commands and actions
  3: also: integrity scan informative messages
- `--version (-v)`: (C) Display stealth's version information and terminate;

- `policy`: file specification of the policy file. If a relative location is specified then this location is interpreted relative to the current working directory. Stealth converts this relative specification to an absolute file location, and an option like `--reload` will reload the policy file from the thus determined absolute file path.

Only one of the options `--daemon, --reload, --resume, --suspend` or `--terminate` can be specified. The options `--reload, --rerun, --resume, --suspend,` and `--terminate` ignore any other options.

The following options are still recognized for backward compatibility with stealth pre-3.00 versions and will be removed in a future stealth version. They generate error messages suggesting alternatives:

- `--echo-commands (-e)`: echo commands to std error when they are processed; use `--log` instead.
- `--keep-alive`: run as a daemon; use `--daemon` instead.
- `--only-stdout`: scan report is written to stdout; use `--stdout` instead.
- `--quiet (-q)`: suppresses progress messages written to stderr; use `--verbosity 0` instead.
- `--suppress <uds>`: suppresses a currently active stealth process; use `--suspend` instead.

The following options were discontinued starting since stealth version 3.00.00:

- `--debug` (option `--verbosity` or `--dry-run` could be used instead);
- `--no-child-processes`;
- `--parse-config-file`.

When specifying long options in policy files initial hyphens should be omitted. Here are some examples:

        %%
        log /tmp/stealth.log
        verbosity 3

### 5.2.1: Rsyslog filtering

When using rsyslogd(1) property based filters may be used to filter syslog messages and write them to a file of your choice. E.g., to filter messages starting with the syslog message tag (e.g., `STEALTH`) use

    :syslogtag, isequal, "STEALTH:"   /var/log/stealth.log
    :syslogtag, isequal, "STEALTH:"   stop

Note that the colon is part of the tag, but is not specified with the `syslog-tag` option.

This causes all messages having the `STEALTH:` tag to be written on `/var/log/stealth.log` after which they are discarded. More extensive filtering is also supported, see, e.g., `http://www.rsyslog.com/doc/rsyslog_conf_filter.html` and `http://www.rsyslog.com/doc/property_replacer.html`

Time stamps written by `rsyslogd` are not controlled by stealth's `--time-stamp` option, but, e.g., by a `TZ` specification in `/etc/default/rsyslog`. Simply add the line

        export TZ=UTC

to `/etc/default/rsyslog`, followed by restarting `rsyslogd` configures `rsyslogd` to generate time stamps using UTC.

## 5.3: Construct one or more policy files

Here we assume that stealth is run by root, and that root wants to store information about the host `client` under the subdirectory `/root/stealth/client`.

Furthermore, we assume that reports of stealth integrity-scans should be sent to the user `admin@elsewhere`, who is only interested in receiving a short summary of changes, as the full report can always be read at the monitor itself. To accomplish this a small support-script was developed, filtering the report generated by stealth down to its essentials.

As the `sha1sum` program on the client may be compromised, it is a good idea to start the integrity scan by transferring the client's `sha1sum` program to the monitor first, to verify the integrity of that program locally (i.e., at the monitor), before trusting it to compute sha1sums of the client's files. The same holds true for any libraries and support programs (like `find`) that are used intensively during integrity scans.

Sha1sum checks should be performed on all setuid and setgid files on the `client`, and in order to be able reach all files on `client`, `root@monitor` is allowed to login to the `root@client` account using an `ssh` connection.

Furthermore, sha1sum checks should be performed on all configuration files, living under `/etc` and on the file `/usr/bin/find` (which is used intensively when performing the integrity checks).

Next, the construction of the required `policy` file, implementing the abovementioned requirements, is described in the following subsections.

### 5.3.1: DEFINE directives

First we write some `DEFINE` directives simplifying complex command specifications:

        DEFINE  SSHCMD  /usr/bin/ssh root@client -T -q exec /bin/bash --noprofile
        DEFINE  EXECSHA1 -xdev -perm +u+s,g+s \( -user root -or -group root \) \
                        -type f -exec /usr/bin/sha1sum {} \;

The first `DEFINE` defines the `ssh` command to use: an ssh-connection will be made to the root account at the client.

The second `DEFINE` shows the arguments for find(1) when looking for all root setuid or setgid normal files. For all these files the sha1sum(1) program should be run.

### 5.3.2: USE directives

Next some `USE` directives, matching our specific, local, situation, are defined:

        USE BASE        /root/stealth/client
        USE EMAIL       admin@elswhere
        USE MAILER      /root/bin/stealthmail
        USE MAILARGS    "Client STEALTH report"
        USE SSH         ${SSHCMD}

- All output is written under the `/root/stealth/client` directory;
- Mail is sent to the user `admin@elsewhere`;
- As mail program we use a filtering script (`stealthmail`), which is installed in `/root/bin`;
- The script handles its own argument. As it can be used by stealth performing integrity scans on other clients as well, it is given an argument which can be used as e-mail subject, identifying the client-computer that has been integrity-scanned;
- The ssh-command is defined by the `SSHCMD`. It's definition is used at the `USE SSH` specification;
- Default values of all remaining `USE` directives are OK, and thus were not explicitly specified. They are:

          USE DD          /bin/dd
          USE DIFF        /usr/bin/diff
          USE REPORT      report
          USE SH          /bin/sh

### 5.3.3: Commands

The following commands are now defined:

- first, we copy the client's `sha1sum` program to the monitor. In practice, this should also include the shared object libraries that are used by `sha1sum`, as they might have become corrupted as well;
- Once `sha1sum` is locally available its integrity is verified;
- Once the integrity of the client's `sha1sum` has been verified, it is used to verify the integrity of the client's `/usr/bin/find` program;
- Following this, all setuid and setgid root files are located and checked for integrity;
- Finally, the integrity of the client configuration files under `/etc` is verified;

In this manual the sha1sum(1) program is frequently used when checking hash values. Stronger hash functions (like sha256sum(1)) might be preferred in practice. `sha256sum's` hash values are remarkably longer than `sha1sum's` hash values. When using these longer hash values in the manual it often clobbers the layout of examples. Therefore in this manual sha1sum(1) is continued to be used.

Realize, however, that when updating existing policy files to use sha256sum(1) instead of sha1sum(1), that previously generated log files (that used sha1sum(1)) are incompatible with log files obtained when using sha256sum(1). In practice this means that new log files need to be generated, and any previously geneerated log files must be disregarded.

#### 5.3.3.1: Obtaining the client's sha1sum program

To copy the client's `sha1sum` program to a local directory we specify:

        GET /usr/bin/sha1sum /root/tmp

This command must succeed.

#### 5.3.3.2: Checking the integrity of the client's sha1sum program

Next, we check the integrity of the received `sha1sum` program. For this, we use the monitor's `sha1sum` program:

        LABEL \nCheck the client's sha1sum program
        LOCAL CHECK LOG = local/sha1 /usr/bin/sha1sum /root/tmp/sha1sum

The `LABEL` command writes the label to the report file just before writing the `sha1sum` program's output.

The `LOCAL` command checks the sha1sum of the program copied from the client. The report is written on the file `/root/stealth/client/local/sha1`. If this fails, stealth terminates, alerting `admin@elsewhere` that the check failed. This is a serious event, as it indicates that either the monitor's `sha1sum` is behaving unexpectedly or that the client's `sha1sum` program has unexpectedly changed.

The `sha1sum` program may have changed due to a normal upgrade. If so, `admin@elsewhere` will know this, and can (probably) ignore the warning. The next time stealth is run, the (now updated) SHA1 value is used, and it again compares the obtained `SHA1` value to the one obtained for the downloaded `sha1sum` program.

#### 5.3.3.3: Checking the client's /usr/bin/find program

The client normally uses its `find` command intensively: `find` is a great tool for producing reports about almost any conceivable combination of characteristics of sets of files. Of course, the client's `find` command must itself be OK, as well as the client's `sha1sum` program. Now that we know that the client's `sha1sum` program is OK, we can use it to check the client's `/usr/bin/find` program.

Note that the monitor itself no longer needs to invest any significant processing load: only the client itself is taxed for checking the integrity of its own files:

        LABEL \nchecking the client's /usr/bin/find program
        CHECK LOG = remote/binfind /usr/bin/sha1sum /usr/bin/find

#### 5.3.3.4: Checking the client's setuid/setgid files

Having checked the client's `sha1sum` and `find` programs, sha1 checksum checks should be performed on all setuid and setgid files on the client. For this we activate the `sha1sum` program on the client. In order to check the setuid/setgid files, the following command is added to the policy file:

        LABEL \nsuid/sgid/executable files uid or gid root on the / partition
        CHECK LOG = remote/setuidgid /usr/bin/find / ${EXECSHA1}

#### 5.3.3.5: Checking the configuration files in the client's /etc/ directory

Finally, the client's configuration files are checked. Some of these files change so frequently that we don't want them to be checked. E.g., `/etc/adjtime, /etc/mtab`. To check the configuration file, do:

        LABEL \nconfiguration files under /etc
        CHECK LOG = remote/etcfiles                         \
              /usr/bin/find /etc -type f -not -perm /6111   \
                -not -regex "/etc/\(adjtime\|mtab\)"        \
                -exec /usr/bin/sha1sum {} \;

### 5.3.4: The complete `policy' file

Here is the complete policy file we've constructed so far:

        DEFINE  SSHCMD  /usr/bin/ssh root@client -T -q exec /bin/bash --noprofile
        DEFINE  EXECSHA1 -xdev -perm +u+s,g+s \( -user root -or -group root \) \
                        -type f -exec /usr/bin/sha1sum {} \;

        USE BASE        /root/stealth/client
        USE EMAIL       admin@elswhere
        USE MAILER      /root/bin/stealthmail
        USE MAILARGS    "Client STEALTH report"
        USE SSH         ${SSHCMD}

        USE DD          /bin/dd
        USE DIFF        /usr/bin/diff
        USE REPORT      report
        USE SH          /bin/sh

        GET /usr/bin/sha1sum /root/tmp

        LABEL \nCheck the client's sha1sum program
        LOCAL CHECK LOG = local/sha1 /usr/bin/sha1sum /root/tmp/sha1sum

        LABEL \nchecking the client's /usr/bin/find program
        CHECK LOG = remote/binfind /usr/bin/sha1sum /usr/bin/find

        LABEL \nsuid/sgid/executable files uid or gid root on the / partition
        CHECK LOG = remote/setuidgid /usr/bin/find / ${EXECSHA1}

        LABEL \nconfiguration files under /etc
        CHECK LOG = remote/etcfiles                         \
              /usr/bin/find /etc -type f -not -perm /6111   \
                -not -regex "/etc/\(adjtime\|mtab\)"        \
                -exec /usr/bin/sha1sum {} \;

## 5.4: Running `stealth' for the first time

When stealth is now run, it creates its initial report files under `root/stealth/client`.

The first time stealth is run, it is usually run `by hand'. The initial run by hand probably benefits from the `--stdout` option, as it shows all executed commands on the standard output:

        stealth --stdout policy

Furthermore, the reports are initialized. Running stealth this way for the `policy` file constructed in the previous sections produces the following output (lines were wrapped to improve readability):

        GET /usr/bin/sha1sum /root/tmp
        LABEL \nCheck the client's sha1sum program
        LOCAL CHECK LOG = local/sha1 /usr/bin/sha1sum /root/tmp/sha1sum
        LABEL \nchecking the client's /usr/bin/find program
        CHECK LOG = remote/binfind /usr/bin/sha1sum /usr/bin/find
        LABEL \nsuid/sgid/executable files uid or gid root on the / partition
        CHECK LOG = remote/setuidgid /usr/bin/find / -xdev -perm +u+s,g+s
                \( -user root -or -group root \) -type f
                -exec /usr/bin/sha1sum {} \;
        LABEL \nconfiguration files under /etc
        CHECK LOG = remote/etcfiles                         /usr/bin/find /etc
                -type f -not -perm /6111   -not -regex "/etc/\(adjtime\|mtab\)"
                -exec /usr/bin/sha1sum {} \;
        LOCAL /usr/bin/scp -q root@client:/usr/bin/sha1sum /root/tmp
        LABEL \nCheck the client's sha1sum program
        LOCAL CHECK LOG = local/sha1 /usr/bin/sha1sum /root/tmp/sha1sum
        LABEL \nchecking the client's /usr/bin/find program
        CHECK LOG = remote/binfind /usr/bin/sha1sum /usr/bin/find
        LABEL \nsuid/sgid/executable files uid or gid root on the / partition
        CHECK LOG = remote/setuidgid /usr/bin/find / -xdev -perm +u+s,g+s
                \( -user root -or -group root \) -type f
                -exec /usr/bin/sha1sum {} \;
        LABEL \nconfiguration files under /etc
        CHECK LOG = remote/etcfiles                        /usr/bin/find /etc
                -type f -not -perm /6111   -not -regex "/etc/\(adjtime\|mtab\)"
                -exec /usr/bin/sha1sum {} \;

### 5.4.1: The mailed report

The `/root/bin/stealthmail` script is called with the following arguments:

        "Client STEALTH report" admin@elswhere

The mailed report contains information comparable to this:

        STEALTH (4.00.00) started at Sat, 07 Feb 2015 22:10:56 +0100

        Check the client's sha1sum program
        Initialized report on local/sha1

        checking the client's /usr/bin/find program
        Initialized report on remote/binfind

        suid/sgid/executable files uid or gid root on the / partition
        Initialized report on remote/setuidgid

        configuration files under /etc
        Initialized report on remote/etcfiles

### 5.4.2: Files under /root/stealth/client

Under `/root/stealth/client` the following entries are now available:

- `local`: below this directory the reports of the locally performed checks are found. Using our demo `policy` file, only one logfile is found here: `sha1`, containing the client's SHA1 checksum of its `/usr/bin/sha1sum` program:

          45251e259bfaf1951658a7b66c328c52  /root/tmp/sha1sum

- `remote`: at this directory the reports of the remotely performed checks are found. Using our demo `policy` file, three files were created:

  The file `binfind`, containing the checksum of the client's `/usr/bin/find` program:

          fc62fc774999584f1e29e0f94279a652  /usr/bin/find

  The file `etcfiles`, containing the checksums of the client's configuration files under `/etc` (shown only partially):

          ced739ecb2c43a20053a9f0eb308b2b0  /etc/modutils/aliases
          a2322d7e2f95317b2ddf3543eb4c74c0  /etc/modutils/paths
          f9e3eac60200d41dd5569eeabb4eddff  /etc/modutils/arch/i386
          f07da2ebf00c6ed6649bae5501b84c4f  /etc/modutils/arch/m68k.amiga
          2893201cc7f7556160fa9cd1fb5ba56a  /etc/modutils/arch/m68k.atari
              ...
          bf73b4e76066381cd3caf80369ce1d0e  /etc/deluser.conf
          4cd70d9aee333307a09caa4ef003501d  /etc/adduser.conf.dpkg-save
          8c749353c5027d0065359562d4383b8d  /etc/gimp/1.2/gtkrc_user
          3ec404ec597ef5460600cccf0192f4d6  /etc/gimp/1.2/unitrc
          8c740345b891179228e3d1066291167b  /etc/gimp/1.2/gtkrc

  The file `setuidgid`, containing the checksums of the client's setuid/setgid root files (shown only partially):

          030f3f84ec76a8181cca087c4ba655ea  /bin/login
          b6c0209547d88928f391d2bf88af34aa  /bin/ping
          5d324ad212b2ff8f767637ac1a8071ec  /bin/su
          344dbedc398d5114966914419ef53fcc  /usr/bin/wall
          27b045bd7306001f9ea31bc18712d8b7  /usr/bin/rxvt-xpm
          ...
          3567b18ffc39c2dc6ec0c0d0fc483f4f  /usr/lib/ssh-keysign
          3383a7955ac2406311e9aa51c6ac9c2c  /usr/X11R6/bin/X
          3c99ea0425c6e0278039e16478d2fb57  /usr/X11R6/bin/xterm
          d590f7f5b4d6ae61680692a52235d342  /usr/local/bin/setuidcall
          4c17203d7d91ec4946dea2f0ae365d5b  /sbin/unix_chkpwd

  Of course, the checksums and the filenames shown are only for documentation purposes. At other systems different files and/or checksums will be reported.

- The file `/root/client/report` New lines are always appended to the `/root/client/report` file. It will never shorten, unless shorten by the systems administrator at `monitor'.

  This file contains the following:

          STEALTH (3.00.00) started at Wed, 20 Aug 2014 11:06:50 +0000

          Check the client's sha1sum program
          Initialized report on local/sha1

          checking the client's /usr/bin/find program
          Initialized report on remote/binfind

          suid/sgid/executable files uid or gid root on the / partition
          Initialized report on remote/setuidgid

          configuration files under /etc
          Initialized report on remote/etcfiles

This completes the information generated by stealth during its first run.

## 5.5: Subsequent `stealth' runs

### 5.5.1: All files unaltered

When stealth is subsequently run, it updates its report files under `root/stealth/client`. If nothing has changed, the log-files remain unaltered. Subsequent runs will, however, add some new info to the file `/root/client/report`:

        STEALTH (4.00.00) started at Sat, 07 Feb 2015 22:10:56 +0100

        Check the client's sha1sum program
        Initialized report on local/sha1

        checking the client's /usr/bin/find program
        Initialized report on remote/binfind

        suid/sgid/executable files uid or gid root on the / partition
        Initialized report on remote/setuidgid

        configuration files under /etc
        Initialized report on remote/etcfiles

        STEALTH (4.00.00) started at Sat, 07 Feb 2015 22:22:15 +0100

Note that just one extra line was added: a timestamp showing the date/time of the last run. The systems administrator may rotate the report file every once in a while to reclaim some disk space.

### 5.5.2: Modifications occurred

Basically, three kinds of modifications are possible: additions, modifications, and removals. Here we'll show the effects all these changes have on stealth's output.

For illustrative purposes, the following changes were made to the `client`'s files:

- `/etc/motd` was changed
- the file `timezone~` was removed
- the file `/etc/motd.org` was created

Next, stealth was again run, producing the following output:

- The following new info is now added to file `/root/client/report`:

          STEALTH (3.00.00) started at Wed, 20 Aug 2014 11:13:38 +0000

          configuration files under /etc
          ADDED: /etc/motd.org
              < 945d0b8208e9861b8f9f2de155e619f9  /etc/motd.org
          MODIFIED: /etc/motd
              < 7f96195d5f051375fe7b523d29e379c1  /etc/motd
              > 945d0b8208e9861b8f9f2de155e619f9  /etc/motd
          REMOVED: /etc/timezone~
              > 6322bc8cb3ec53f5eea33201b434b74b  /etc/timezone~

  Note that all changes were properly detected and logged in the file `/root/client/report`.

- Furthermore, a matching report was sent by mail:

          STEALTH (3.00.00) started at Wed, 20 Aug 2014 11:13:38 +0000

          configuration files under /etc
          ADDED: /etc/motd.org
              < 945d0b8208e9861b8f9f2de155e619f9  /etc/motd.org
          MODIFIED: /etc/motd
              < 7f96195d5f051375fe7b523d29e379c1  /etc/motd
              > 945d0b8208e9861b8f9f2de155e619f9  /etc/motd
          REMOVED: /etc/timezone~
              > 6322bc8cb3ec53f5eea33201b434b74b  /etc/timezone~

  Note that the report only shows the info that was added to the /root/client/report file.

  The report itself could be beautified further. E.g., I use the following script to mail the report to the addressee:

          #!/bin/bash

          NAME=`basename $0`

          tee /root/stealth/lastreport/$NAME | egrep -v '^([[:space:]]|[[:space:]]*$)' |
              sort | uniq | mail -s $1 $2

  For the `client` computer, this little script writes the mailed report on a file `/root/stealth/lastreport/client`, overwriting its previous contents, removes all lines beginning with blanks (thus trimming away the `diff`-generated lines), and e-mails the `sort`ed and `uniq`ed lines using `mail`. The addressee (`admin@elsewhere`) then receives the following information:

          ADDED: /etc/motd.org
          MODIFIED: /etc/motd
          REMOVED: /etc/timezone~
          STEALTH (3.00.00) started at Wed, 20 Aug 2014 11:13:38 +0000
          configuration files under /etc

  In practice this provides me with all the information I need if something out of the ordinary has happened.

- Finally, the file

          /root/stealth/client/remote/etcfiles

  was recreated, saving the old file as

          /root/stealth/client/remote/etcfiles.20021028-112851

  As remarked earlier (see section 4.3), many `logfile.YYMMDD-HHMMSS` files could eventually accumulate. As discussed in section 4.3, it might be considered to remove old log files every now and then.

### 5.5.3: Failing LOCAL commands

If the client's `sha1sum` program itself is altered, a serious situation has developed. In that case, further actions by stealth would be suspect, as their results might easily be currupted. Additional checks will be performed, but a warning is generated on the `report` file (and in the mail sent to `admin@elsewhere`):

        STEALTH (4.00.00) started at Sat, 07 Feb 2015 22:27:15 +0100

        Check the client's sha1sum program
        MODIFIED: /root/tmp/sha1sum
            < fc62fc774999584f1e29e0f94279a652  /root/tmp/sha1sum
            > 45251e259bfaf1951658a7b66c328c52  /root/tmp/sha1sum

        *** BE CAREFUL *** REMAINING RESULTS MAY BE FORGED

        configuration files under /etc
        REMOVED: /etc/motd.org
            > 945d0b8208e9861b8f9f2de155e619f9  /etc/motd.org
        MODIFIED: /etc/motd
            < 945d0b8208e9861b8f9f2de155e619f9  /etc/motd
            > 7f96195d5f051375fe7b523d29e379c1  /etc/motd

(The report shows the removal of the previously added file `motd.org`, and the modification of `motd`. These are real, as the original `motd` file, modified earlier, was restored at this point).

### 5.5.4: Skipping (some) integrity checks

Some files or directories may not require integrity checks. Automated processes may modify files which are not threatening the proper functioning of running programs or processes. In those cases a file can be prepared holding the absolute paths of entries to be skipped. Each entry should appear on a line of its own without any additional information.

Stealth can be informed about this file using the `--skip-files <file-spec>` option. When a relative file specification is used with `--skip-files` it is interpreted a location relative to the current working directory. The skip-files file itself must contain the paths of the entries to be skipped during file integrity scans. If an entry is already present in a log file then stealth once generates an `IGNORING` message in the mail sent to the address specified at `EMAIL` in the policy file. Each entry in a skip-file must be on a line of its own and must be specified using absolute file paths. Entries ending in a slash are assumed to be directories whose full contents must be skipped. Other entries are interpreted as the names of files to skip. Initial and trailing blanks, empty lines and lines having a `#` as their 1st non blank character are ignored.

Here is an example:

        stealth -e --skip-files /root/stealth/remote/skipping remote.pol

If an entry `/etc/skipme` appears in the current logs which is thereafter added to the `skippath` file then the mail generated by stealth once contains a line like the following:

        SKIPPING: /etc/skipme
            > a7695bb2d019e60988e757a4b692acfe  /etc/skipme

The reported hash-value is the hash-value at the time of the stealth-run reporting the `SKIPPING` message.

Entries ending in a slash are assumed to be directories whose contents must (recursively) be skipped.

## 5.6: Automating repeated `stealth' runs

To automate stealth's integrity scans, a file `/etc/cron.d/stealth` could be created, containing a line like

        2,17,32,47 * * * *  root    test -x /usr/bin/stealth && \
                               /usr/bin/stealth /root/stealth/client.pol

This starts stealth 2 minutes after every hour. In this example the ssh-key must not require a passphrase, as crontab(1) cannot provide passphrases of ssh-keys. Ssh-keys requiring passphrases can, however, be used if repeated stealth runs are controlled by a program like ssh-cron(1).

In general, randomized events are harder to notice. For this stealth offers the `--repeat` and `--random-interval` options. Both options expect an argument in seconds (or in minutes, if an `m` is appended to the specification). After each integrity scan the next integrity scan starts after the time interval specified by the `--repeat` option plus a random time value selected from the time interval specified by the `--random-interval` option. For example, the stealth daemon started by the following command repeatedly performs integrity scans between two and five minutes after the last integrity scan completed:

        stealth --daemon /root/stealth/client.uds \
                --repeat 2m --random-interval 3m /root/stealth/client.pol

Once this program has started its ssh(1) connection to the client host persists, and a possibly required ssh-passphrase is no longer required. Additional (intermediate) integrity scans can always be requested (also without requiring ssh-passphrase specifications) using the command

        stealth --rerun /root/stealth/client.uds

## 5.7: Report File Rotation

When stealth performs integrity scans it appends information to the report file. This file therefore eventually grows to a large size, and the systems manager controlling stealth might want to rotate the report file every once in a while (e.g., using a program like logrotate(1), also see the upcoming section 5.7.2). To ensure that no log-rotation takes place while stealth is busy performing integrity scans (thus modifying the report file) the options --suspend and `--resume` were implemented. Both options require the process-ID file of currently active stealth process as their argument.

For example, if a stealth process was once started using the command

        stealth --daemon /root/stealth/small.uds --repeat 900 \
                        /root/stealth/small.pol

then the `--suspend` and `--resume` commands for this process should be called as:

        stealth --suspend /root/stealth/small.uds
        stealth --resume /root/stealth/small.uds

The stealth process identified in the files provided as arguments to the `--suspend` and `--resume` options is called the daemon stealth process below.

The `--suspend` option has the following effect:

- If the daemon stealth process is currently processing its policy file, performing an integrity scan, then the currently executing policy file command is completed, whereafter further commands are ignored, except for `--resume` (see below) and `--terminate`.
- Any scheduled integrity scans following the `--suspend` command are ignored by the daemon stealth process;
- The daemon stealth process writes a message that it is being suspended to the report file and then processes the report file as usual.

Now that the report file is no longer modified by the daemon stealth process, log-rotation may take place. E.g., a program like logrotate(1) allows its users to specify a command or script just before log-rotation takes place, and `stealth `--suspend udsfile`' could be specified nicely in such a pre-rotation section.

The `--resume` option has the following effect:

- The daemon stealth process resumes its activities by performing another integrity scan. Thus, `--resume` implies `--rerun`.
- Any scheduled integrity scans following the `--resume` command are again honored by the daemon stealth process.

Note that, once `--suspend` has been issued, all commands except `--resume` and `--terminate` are ignored by the daemon stealth process. While suspended, the `--terminate` command is acknowledged as a `emergency exit' which may or may not interfere with, e.g., an ongoing log-rotation process. The daemon stealth process should not normally be terminated while it is in its suspended mode. The normal way to terminate a stealth process running in the background is:

- Wait for the daemon stealth process to complete an ongoing series of integrity scan commands;
- Issue the `stealth `--terminate udsfile`' command.

### 5.7.1: Status file cleanup

Whenever stealth is run and it encounters a modified situation the already existing status file summarizing that particular situation is saved and a new status file is created. Eventually, this will result in many status files. While report files can be rotated, it is pointless to rotate old status files, since they are never modified. Instead, status files exceeding a certain age could be removed and more recent files might be zipped to conserve space. In stealth's binary distribution the file `/usr/share/doc/stealth/usr/bin/stealthcleanup` is provided which can be used to perform such a cleanup. The script expects one argument: a resource file defining the following shell variables:

- `directories`: the directories below which the status files are found;
- `gzdays`: the number of days a status file must exist before it is compressed using gzip(1);
- `rmdays`: the maximum age (in days) of compressed status files. Files exceeding this age are removed using rm(1).

Here is the `stealthcleanup` script as contained in the binary distribution's `/usr/share/doc/stealth/additional` directory:

    #!/bin/bash

    usage()
    {
        echo "
    Usage: $0 [-v] rc-file
    Where:
        -v: Show the actions that are performed
        rc-file: resource file defining:
                \`directories' - one or more directories containing status files
                \`gzdays'      - number of days status files may exist before they
                                 are compressed
                \`rmdays'      - number of days gzipped status files may exist
                                 before they are removed.
    "
        exit 1
    }

    error()
    {
        echo "$*" >&2
        exit 1
    }

    if [ "$1" == "-v" ]
    then
        verbose=1
        shift 1
    else
        verbose=0
    fi

    [ $# == 1 ] || usage

    # now source the configuration file
    . $1

    for x in $directories
    do
        cd $x || error "\`$x' must be a directory"
        if [ $verbose -eq 1 ]
        then
            echo "
    cd $x"
        fi

        if [ $verbose -eq 1 ]
        then
            echo \
        /usr/bin/find ./ -mtime +$rmdays -type f -regex '.*[0-9]+-[0-9]+\.gz' \
            -exec /bin/rm {} \;
        fi
        /usr/bin/find ./ -mtime +$rmdays -type f -regex '.*[0-9]+-[0-9]+\.gz' \
            -exec /bin/rm {} \;

        if [ $verbose -eq 1 ]
        then
            echo \
        /usr/bin/find ./ -mtime +$gzdays -type f -regex '.*[0-9]+-[0-9]+' \
            -exec /bin/gzip {} \;
        fi
        /usr/bin/find ./ -mtime +$gzdays -type f -regex '.*[0-9]+-[0-9]+' \
            -exec /bin/gzip {} \;
    done

    exit 0

Assuming that the status files are written in `/var/stealth/target/local` and `/var/stealth/target/remote`; that status file should be compressed when older than 2 days and removed after 30 days, the resource file is:

    directories="
        /var/stealth/target/local
        /var/stealth/target/remote
        "

    rmdays=30
    gzdays=3

Furthermore assuming that the resourcefile is installed in `/etc/stealth/cleanup.rc` and the `stealthcleanup` script itself in `/usr/bin/stealthcleanup`, the `stealthcleanup` script could be called as follows:

        /usr/bin/stealthcleanup /etc/stealth/cleanup.rc

Note that `stealthcleanup` may be called whether or not there are active stealth processes.

### 5.7.2: Using `logrotate' to control report- and status files

A program like logrotate(1) allows its users to specify a command or script immediately following log-rotation, and `stealth `--resume pidfile`' could be specified nicely in such a post-rotation section.

Here is an example of a specification that can be used with logrotate(1). Logrotate (on Debian systems) keeps its configuration files in `/etc/logrotate.d`, and assuming there is a host `target`, whose report file is `/var/stealth/target/report`, the required logrotate(1) specification file (e.g., `/etc/logrotate.d/target`) could be:

    /root/stealth/report /var/log/stealth/client-small.log {
        weekly
        rotate 12
        compress
        missingok
        copytruncate
        sharedscripts
        prerotate
            /usr/bin/stealth --suspend /root/stealth/small.uds
        endscript
        postrotate
            /usr/bin/stealth --resume /root/stealth/small.uds
        endscript
    }

Using this specification file, logrotate(1) will

- perform weekly rotations of the report file;
- keep up to 12 rotated files, compressing them using gzip(1);
- suspend the stealth daemon, before rotating its report file; suspended;
- following the rotation, stealth's actions are resumed.

Note thet `stealth --resume xxx` always initiates another file integrity scan.

- Table of Contents
- Previous Chapter
- Next Chapter