# omazureeventhubs: Microsoft Azure Event Hubs Output Module

Module Name: | omazureeventhubs
Author: | Andre Lorbach <alorbach@adiscon.com>
Available since: | v8.2304

## Purpose

The purpose of the rsyslog output plugin omazureeventhubs is to provide a fast and reliable way to send log data from rsyslog to Microsoft Azure Event Hubs. This plugin uses the Advanced Message Queuing Protocol (AMQP) to securely transmit log data from rsyslog to Microsoft Azure, where it can be centralized, analyzed, and stored. The plugin uses the “Qpid Proton C API” library to implement the AMQP protocol, providing a flexible and efficient solution for sending log data to Microsoft Azure Event Hubs.

AMQP is a reliable and secure binary protocol for exchanging messages between applications, and it is widely used in the cloud and enterprise messaging systems. The use of AMQP in the omazureeventhubs plugin, in combination with the Qpid Proton C API library, ensures that log data is transmitted in a robust and reliable manner, even in the presence of network outages or other disruptions.

The omazureeventhubs plugin supports various configuration options, allowing organizations to customize their log data pipeline to meet their specific requirements. This includes options for specifying the Event Hubs endpoint, port, and authentication credentials. With this plugin, organizations can easily integrate their rsyslog infrastructure with Microsoft Azure Event Hubs, providing a scalable and secure solution for log management. The plugin is designed to work with the latest versions of rsyslog and Microsoft Azure, ensuring compatibility and reliability.

## Requirements

To output logs from rsyslog to Microsoft Azure Event Hubs, you will need to fulfill the following requirements:

- Qpid Proton C Library Version 0.13 or higher including Qpid Proton ProActor

- The AMQP Protocol needs to have firewall Ports 5671 and 443 TCP to be open for outgoing connections.

## Configuration Parameters

Note

Parameter names are case-insensitive; camelCase is recommended for readability.

Warning

Older guidance mentioned a `closeTimeout` parameter, but the module does not implement it. Configurations that reference `closeTimeout` will fail to load.

### Input Parameters

Parameter | Summary
--- | ---
azurehost | Sets the fully qualified Event Hubs namespace host the action connects to.
azureport | Defines the TCP port used when connecting to the Event Hubs namespace.
azure_key_name | Specifies the shared access key name used to authenticate with Event Hubs.
azure_key | Provides the shared access key value used to sign Event Hubs requests.
container | Identifies the Event Hubs instance that receives the formatted log messages.
template | Selects the rsyslog template that formats messages sent to Event Hubs.
amqp_address | Provides a full AMQPS connection string that overrides individual Azure settings.
eventproperties | Adds custom key-value properties to each AMQP message emitted by the action.
statsname | Names the statistics counters that track this action instance.

## Statistic Counter

This plugin maintains global statistics for omazureeventhubs that accumulate all action instances. The statistic origin is named “omazureeventhubs” with following counters:

- submitted - This counter tracks the number of log messages that have been submitted by the rsyslog process to the output module for delivery to Microsoft Azure Event Hubs.

- accepted - This counter tracks the number of log messages that have been successfully delivered to Microsoft Azure Event Hubs by the output module.

- failures - This counter tracks the number of log messages that have failed to be delivered to Microsoft Azure Event Hubs due to various error conditions, such as network connectivity issues, incorrect configuration settings, or other technical problems. This counter provides important information about any issues that may be affecting the delivery of log data to Microsoft Azure Event Hubs.

- failures_other - This counter tracks the number of log messages that have failed to be delivered due to other error conditions, such as incorrect payload format or unexpected data.

These statistics counters are updated in real-time by the rsyslog output module as log data is processed, and they provide valuable information about the performance and operation of the output module.

For multiple actions using statistics callback, there will be one record for each action.

## Examples

### Example 1: Use AMQP URL

The following sample does the following:

- loads the omazureeventhubs module

- outputs all logs to Microsoft Azure Event Hubs with standard template

- Uses amqp_address parameter

    module(load="omazureeventhubs")
    action(type="omazureeventhubs" amqp_address="amqps://<AccessKeyName>:<AccessKey>@<EventHubsNamespace>.servicebus.windows.net/<EventHubsInstance>")

### Example 2: RAW Format

The following sample does the following:

- loads the omazureeventhubs module

- outputs all logs to Microsoft Azure Event Hubs with simple custom template

- Uses azurehost, azureport, azure_key_name and azure_key parameters instead of amqp_address parameter

    module(load="omazureeventhubs")
    template(name="outfmt" type="string" string="%msg%\n")

    action(type="omazureeventhubs"
         azurehost="<EventHubsNamespace>.servicebus.windows.net"
         azureport="5671"
         azure_key_name="<AccessKeyName>"
         azure_key="<AccessKey>"
         container="<EventHubsInstance>"
         template="outfmt"
    )

### Example 3: JSON Format

The following sample does the following:

- loads the omazureeventhubs module

- outputs all logs to Microsoft Azure Event Hubs with JSON custom template

- Uses azurehost, azureport, azure_key_name and azure_key parameters instead of amqp_address parameter

- Uses eventproperties array parameter to set additional message properties

    module(load="omazureeventhubs")
    template(name="outfmtjson" type="list" option.jsonf="on") {
         property(outname="timestamp" name="timereported" dateFormat="rfc3339" format="jsonf")
         constant(value="\"source\": \"EventHubMessage\", ")
         property(outname="host" name="hostname" format="jsonf")
         property(outname="severity" name="syslogseverity" caseConversion="upper" format="jsonf" datatype="number")
         property(outname="facility" name="syslogfacility" format="jsonf" datatype="number")
         property(outname="appname" name="syslogtag" format="jsonf")
         property(outname="message" name="msg" format="jsonf" )
         property(outname="etlsource" name="$myhostname" format="jsonf")
    }

    action(type="omazureeventhubs"
         azurehost="<EventHubsNamespace>.servicebus.windows.net"
         azureport="5671"
         azure_key_name="<AccessKeyName>"
         azure_key="<AccessKey>"
         container="<EventHubsInstance>"
         template="outfmtjson"
         eventproperties=[       "Table=CustomTable",
                                 "Format=JSON"]
    )

### Example 4: High Performance

To achieve high performance when sending syslog data to Azure Event Hubs, you should consider configuring your output module to use multiple worker instances. This can be done by setting the “workerthreads” parameter in the configuration file.

The following example is for high performance (Azure Premium Tier) and does the following:

- loads the omazureeventhubs module

- outputs all logs to Microsoft Azure Event Hubs with JSON custom template

- Uses azurehost, azureport, azure_key_name and azure_key parameters instead of amqp_address parameter

- Uses eventproperties array parameter to set additional message properties

- Uses Linkedlist In-Memory Queue which enables multiple omazureeventhubs workers running at the same time. Using a dequeue size of 2000 and a dequeue timeout of 1000 has shown very good results in performance tests.

- Uses 8 workerthreads in this example, which will be spawn automatically if more than 2000 messages are waiting in the Queue. To achieve more performance, the number can be incremented.

    module(load="omazureeventhubs")
    template(name="outfmtjson" type="list" option.jsonf="on") {
         property(outname="timestamp" name="timereported" dateFormat="rfc3339" format="jsonf")
         constant(value="\"source\": \"EventHubMessage\", ")
         property(outname="host" name="hostname" format="jsonf")
         property(outname="severity" name="syslogseverity" caseConversion="upper" format="jsonf" datatype="number")
         property(outname="facility" name="syslogfacility" format="jsonf" datatype="number")
         property(outname="appname" name="syslogtag" format="jsonf")
         property(outname="message" name="msg" format="jsonf" )
         property(outname="etlsource" name="$myhostname" format="jsonf")
    }

    action(type="omazureeventhubs"
         azurehost="<EventHubsNamespace>.servicebus.windows.net"
         azureport="5671"
         azure_key_name="<AccessKeyName>"
         azure_key="<AccessKey>"
         container="<EventHubsInstance>"
         template="outfmtjson"
         eventproperties=[       "Table=CustomTable",
                                 "Format=JSON"]
         queue.type="linkedList"
         queue.size="200000"
         queue.saveonshutdown="on"
         queue.dequeueBatchSize="2000"
         queue.minDequeueBatchSize.timeout="1000"
         queue.workerThreads="8"
         queue.workerThreadMinimumMessages="2000"
         queue.timeoutWorkerthreadShutdown="10000"
         queue.timeoutshutdown="1000"
    )

Support: rsyslog Assistant | GitHub Discussions | GitHub Issues: rsyslog source project

Contributing: Source & docs: rsyslog source project

© 2008–2025 Rainer Gerhards and others. Licensed under the Apache License 2.0.