# omsendertrack: Sender Tracking Output Module

Module Name:

  omsendertrack

Author:

  Rainer Gerhards <rgerhards@adiscon.com>

Available since:

  8.2506.0 (Proof-of-Concept)

Status: Proof-of-concept implementation. This module is currently in an experimental stage. Further details and discussion regarding its development and progress can be found in issue #5599.

## Purpose

The `omsendertrack` output module is designed to collect and maintain real-time statistics about message senders across all configured rsyslog inputs. Its primary goal is to provide a flexible and persistent mechanism for tracking message flow from various sources.

Key uses for `omsendertrack` include:

- Identifying Top Senders: Quickly pinpoint which hosts or applications

    are generating the most log traffic.

- Monitoring Sender Behavior: Detect changes in message rates or patterns

    from specific senders, which can indicate issues or unusual activity.

- Understanding Message Distribution: Gain insights into the overall

    distribution of messages within your logging infrastructure.

- Persistency: Message counts and last event times persist across rsyslog

    daemon restarts, ensuring continuous tracking.

The module achieves this by periodically writing these statistics to a JSON statefile.

## Functionality

The `omsendertrack` module operates through several key stages and mechanisms to ensure accurate and persistent sender tracking.

### Initialization

Upon rsyslog startup, the `omsendertrack` module attempts to load its previously saved state from the configured statefile. This data, which includes sender identifiers, message counts, and last event times, is loaded into an in-memory hash table. This ensures that message statistics are restored and tracking continues seamlessly across daemon restarts. A background task is then spawned to handle periodic state persistence.

### OnAction Call (Message Processing)

When a message is routed to an `omsendertrack` action:

1. Sender Identification: The module uses the configured senderid template to derive a unique identifier for the message sender.

2. Statistic Update: It then updates or inserts an entry for this sender in its internal hash table.

3. Timestamp Recording: The last-event-time for the sender is updated with the current UTC timestamp of the received message.

4. Message Counting: The message-count for that sender is incremented.

5. Average Rate (Optional): If configured, the avg-message-count (average message rate) may also be recalculated.

### HUP Signal Handling

When rsyslog receives a HUP signal (typically used for configuration reloads), the `omsendertrack` module is designed to check for the existence of a cmdfile. If a cmdfile is specified and found, it would be read and its commands processed. After processing, the cmdfile would be deleted to prevent re-execution on subsequent HUP signals.

Note: Command file support is currently not implemented in this proof-of-concept version of the module.

### Background Task

A dedicated background task is responsible for persisting the module’s current state to the configured statefile. This task wakes up at the interval specified in the configuration. It performs atomic writes to the statefile to prevent data corruption, even if rsyslog unexpectedly terminates during a write operation.

### Shutdown

During rsyslog shutdown, the `omsendertrack` module ensures that the most current sender statistics are saved to the statefile. This critical step guarantees data persistence and allows for an accurate resumption of tracking when rsyslog restarts.

## Configuration

The `omsendertrack` module supports the following action parameters.

Note

Parameter names are case-insensitive; camelCase is recommended for readability.

## Action Parameters

Parameter | Summary
--- | ---
senderid | Sets the template used to derive the unique sender identifier that omsendertrack tracks.
interval | Sets how many seconds elapse between each write of sender statistics to the state file.
statefile | Specifies the absolute path to the JSON file where omsendertrack persists sender statistics.
cmdfile | Defines the absolute path to the command file that omsendertrack reads when rsyslog receives a HUP signal.

## Statistic Counter

The `omsendertrack` module is designed to maintain a set of statistics for each unique sender identifier it tracks. These statistics are intended to be periodically serialized and written to the configured statefile in JSON format.

Important: This module does not offer statistics counters in the typical sense that are consumable by other rsyslog modules like impstats. The collected data is primarily intended for direct consumption from the generated state file.

Note: There are currently no statistics counters available in this proof-of-concept version of the module.

The JSON structure for each sender entry is envisioned to look like this:

    {
      "senderid": "value_from_template",
      "last-event-time": "YYYY-MM-DDTHH:MM:SS.sssZ",
      "message-count": "N_VALUE",
      "avg-message-count": "M_POINT_M_VALUE"
    }

Where:

- `senderid`: The unique identifier for the sender, as determined by the

    senderid template.

- `last-event-time`: A UTC timestamp (ISO 8601 format) indicating when the

    last message from this sender was received.

- `message-count`: The total number of messages received from this sender

    since tracking began (or since the last reset).

- `avg-message-count`: (Optional) The average message rate from this sender

    since tracking began, calculated over the total elapsed time. This field’s presence depends on future module configuration and implementation details.

## Usage within Rsyslog Configuration

The `omsendertrack` module functions as an output module (OM), meaning you integrate its action where you want sender statistics to be collected within your rsyslog configuration. Each instance of `omsendertrack` counts its senders independently.

Queue Considerations:

It’s technically possible to place the `omsendertrack` action within a dedicated ruleset that has a queue, or to add a queue directly to the action itself. However, `omsendertrack` processing is extremely fast, with the overhead of a queue often being multiple times greater than the actual call to the module. For this reason, adding a queue generally does not make sense for `omsendertrack` and is not recommended as it would introduce unnecessary complexity and potential latency without significant benefit.

For optimal performance, always consider calling `omsendertrack` actions synchronously. This can be done within an existing ruleset, or by a synchronous `call` statement to a dedicated ruleset that has no queue.

## Best Practices

To ensure efficient and correct operation of the `omsendertrack` module, adhere to the following best practices:

- Prioritize Synchronous Calls: Always call `omsendertrack` actions

    synchronously. The module is highly optimized for quick processing, and asynchronous calls with queues are generally unnecessary and can introduce overhead without benefit.

- Avoid Queues on Dedicated Rulesets: If you use a dedicated ruleset to

    house the `omsendertrack` action (as shown in Example 2), ensure that this specific ruleset does not have a queue configured. The module’s fast execution makes queues redundant here.

- Efficient Sender Identification: Choose your senderid template carefully. Simpler templates (e.g., `"%hostname%"`, `"%fromhost-ip%"`) result in better performance, as template processing occurs for every message.

- Appropriate `interval` for State File Writes: Balance your need for

    up-to-date statistics against disk I/O. A very small interval can lead to increased disk writes, while a larger one might mean slightly older data on disk in case of an unexpected shutdown.

- Ensure State File Write Permissions: Verify that the rsyslog user has

    proper write permissions to the directory specified in the statefile parameter. Without this, statistics cannot be persisted.

- Dedicated Ruleset for Unified Stats: Use a dedicated ruleset that is

    called from multiple input-bound rulesets (Example 2) only when you need to collect statistics from those diverse inputs into a single, unified sender statistics file.

- Multiple Instances for Separate Stats: Deploy multiple `omsendertrack`

    action instances (Example 3) only when you explicitly desire to generate separate sender statistics files based on different filtering criteria or input sources. Do not create multiple instances if a single, aggregated statistic file is your goal.

## Examples

Let’s look at some examples of how to configure the `omsendertrack` module.

### Example 1: Basic Configuration

This is the simplest way to use `omsendertrack`. It loads the module and configures it to track senders based on their hostname, updating statistics every 60 seconds and storing them in a state file. This approach is suitable when all messages you wish to track are processed within a single ruleset or when the overall volume is low.

    module(load="omsendertrack")
    action(type="omsendertrack"
           senderId="%hostname%"
           interval="60"
           stateFile="/var/lib/rsyslog/senderstats.json")

### Example 2: Usage with Dedicated Ruleset

A dedicated ruleset for `omsendertrack` is suggested specifically when you need to count senders where the incoming messages are bound to different rulesets, and you want all those messages to contribute to a single, unified sender statistics file.

This example shows how to set up `omsendertrack` within a dedicated ruleset, which is then called synchronously from multiple input-bound rulesets. This allows you to centralize sender tracking while maintaining separate message processing flows for other actions.

    # Define the template for senderId in omsendertrack
    template(name="id-template" type="list") {
        property(name="hostname")
    }

    # Ruleset omsendertrack-ruleset: Must only contain the omsendertrack action
    # This ruleset should NOT have a queue.
    ruleset(name="omsendertrack-ruleset") {
        action(
            type="omsendertrack"
            senderId="id-template"
            interval="60"
            stateFile="/var/lib/rsyslog/senderstats.json"
            cmdFile="/var/lib/rsyslog/sendercommands.txt"
        )
    }

    # Ruleset a: Calls omsendertrack-ruleset synchronously, then forwards messages
    ruleset(name="a"
        queue.type="LinkedList"
        queue.spoolDirectory="/var/lib/rsyslog/queue_a"
        queue.fileName="q_a"
        queue.maxDiskSpace="1g"
        queue.saveOnShutdown="on"
        queue.discardSeverity="8"
        queue.discardMark="1"
    ) {
        call omsendertrack-ruleset
        action(
            type="omfwd"
            target="192.0.2.1"
            port="10514"
            protocol="udp"
        )
        action(
            type="omfwd"
            target="192.0.2.2"
            port="10514"
            protocol="tcp"
        )
    }

    # Ruleset b: Calls omsendertrack-ruleset synchronously, then forwards messages
    ruleset(name="b"
        queue.type="LinkedList"
        queue.spoolDirectory="/var/lib/rsyslog/queue_b"
        queue.fileName="q_b"
        queue.maxDiskSpace="1g"
        queue.saveOnShutdown="on"
        queue.discardSeverity="8"
        queue.discardMark="1"
    ) {
        call omsendertrack-ruleset
        action(
            type="omfwd"
            target="192.0.2.3"
            port="514"
            protocol="udp"
        )
        action(
            type="omfwd"
            target="192.0.2.4"
            port="514"
            protocol="tcp"
        )
    }

    # Input for ruleset a (example: UDP input)
    input(type="imudp" port="5140" ruleset="a")

    # Input for ruleset b (example: TCP input)
    input(type="imtcp" port="5141" ruleset="b")

    # Default ruleset (if messages don't match other inputs)
    # This is here for completeness, you can remove or modify it as needed.
    ruleset(name="RSYSLOG_DefaultRuleset") {
        stop
    }

### Example 3: Multiple Instances for Separate Statistics

It is possible to use multiple instances of `omsendertrack` if it is desired to create separate sender statistics files based on different criteria. For example, you might want to track UDP senders and TCP senders in distinct state files.

    # Track UDP senders in a separate state file
    ruleset(name="udp-sender-tracking") {
        action(
            type="omsendertrack"
            senderId="%fromhost-ip%"
            interval="300"
            stateFile="/var/lib/rsyslog/udp_sender_stats.json"
        )
        # Add other actions for UDP messages here (e.g., forwarding, writing to file)
    }

    # Track TCP senders in another state file
    ruleset(name="tcp-sender-tracking") {
        action(
            type="omsendertrack"
            senderId="%fromhost-ip%"
            interval="300"
            stateFile="/var/lib/rsyslog/tcp_sender_stats.json"
        )
        # Add other actions for TCP messages here
    }

    # Bind inputs to the respective sender tracking rulesets
    input(type="imudp" port="514" ruleset="udp-sender-tracking")
    input(type="imtcp" port="514" ruleset="tcp-sender-tracking")

    # Further processing for all messages (e.g., default ruleset)
    ruleset(name="RSYSLOG_DefaultRuleset") {
        stop
    }

Support: rsyslog Assistant | GitHub Discussions | GitHub Issues: rsyslog source project

Contributing: Source & docs: rsyslog source project

© 2008–2025 Rainer Gerhards and others. Licensed under the Apache License 2.0.