# omelasticsearch: Elasticsearch Output Module

Module Name: | omelasticsearch
Author: | Rainer Gerhards <rgerhards@adiscon.com>

## Purpose

This module provides native support for logging to Elasticsearch.

## Notable Features

- Statistic Counter

## Target platform detection

Starting with release 8.2510.0 the module probes the configured servers during configuration processing to determine whether they are running Elasticsearch or OpenSearch and to capture the version number that is exposed by the cluster. The probe happens once at startup, before the action begins to process any messages. When the detection succeeds the module will automatically adapt internal defaults (for example, legacy clusters continue to receive the `system` index) and it will override `esVersion.major` with the detected major version. The configured action keeps running even if the probe cannot reach the servers; in that case rsyslog falls back to the provided configuration values.

## Configuration Parameters

Note

Parameter names are case-insensitive; CamelCase is recommended for readability.

Note

This module supports action parameters, only.

### Action Parameters

Parameter | Summary
--- | ---
Server | List of Elasticsearch servers to send events to.
Serverport | Default port used when server URLs omit a port.
HealthCheckTimeout | Milliseconds to wait for an HTTP health check before sending events.
esVersion.major | Deprecated manual override for the detected Elasticsearch/OpenSearch major version.
searchIndex | Elasticsearch index where events are written.
dynSearchIndex | Treat searchIndex as a template name instead of a literal.
searchType | Elasticsearch type to use; empty string omits the type.
dynSearchType | Treat searchType as a template name.
pipelineName | Ingest pipeline to run before indexing.
dynPipelineName | Treat pipelineName as a template name.
skipPipelineIfEmpty | Omit the pipeline parameter when pipelineName expands to an empty string.
asyncrepl | Deprecated option formerly enabling asynchronous replication.
usehttps | Default scheme for servers missing one.
timeout | How long Elasticsearch waits for a primary shard before failing.
indexTimeout | Milliseconds to wait for an indexing request to complete.
template | Template used to render the JSON document sent to Elasticsearch.
bulkmode | Use the Elasticsearch Bulk API to send batched events.
maxbytes | Maximum size of a bulk request body when bulkmode is enabled.
parent | Parent document ID assigned to indexed events.
dynParent | Treat parent as a template for per-record parent IDs.
uid | User name for basic HTTP authentication.
pwd | Password for basic HTTP authentication.
apikey | Supplies the value for the `Authorization: ApiKey` HTTP header.
errorFile | File that receives records rejected during bulk mode.
tls.cacert | CA certificate file used to verify the Elasticsearch server.
tls.mycert | Client certificate for mutual TLS authentication.
tls.myprivkey | Unencrypted private key for tls.mycert.
allowunsignedcerts | Disable TLS peer verification (insecure, for testing only).
skipverifyhost | Disable TLS host name verification (insecure, for testing).
bulkid | Unique identifier assigned to each record.
dynbulkid | Treat bulkid as a template that generates per-record IDs.
writeoperation | Bulk action type: index (default) or create to avoid overwrites.
retryfailures | Resubmit failed bulk items back into rsyslog for retry.
retryruleset | Ruleset used when processing retried records.
ratelimit.interval | Seconds over which retry rate limiting is calculated.
ratelimit.burst | Maximum messages allowed in a rate-limit interval.
rebindinterval | Operations after which to reconnect; -1 disables periodic reconnect.

## Statistic Counter

This plugin maintains global statistics, which accumulate all action instances. The statistic is named “omelasticsearch”. Parameters are:

- submitted - number of messages submitted for processing (with both success and error result)

- fail.httprequests - the number of times an HTTP request failed. Note that a single HTTP request may be used to submit multiple messages, so this number may be (much) lower than fail.http.

- fail.http - number of message failures due to connection like-problems (things like remote server down, broken link etc)

- fail.es - number of failures due to elasticsearch error reply; Note that this counter does NOT count the number of failed messages but the number of times a failure occurred (a potentially much smaller number). Counting messages would be quite performance-intense and is thus not done.

The following counters are available when retryfailures=”on” is used:

- response.success - number of records successfully sent in bulk index requests - counts the number of successful responses

- response.bad - number of times omelasticsearch received a response in a bulk index response that was unrecognized or unable to be parsed. This may indicate that omelasticsearch is attempting to communicate with a version of Elasticsearch that is incompatible, or is otherwise sending back data in the response that cannot be handled

- response.duplicate - number of records in the bulk index request that were duplicates of already existing records - this will only be reported if using writeoperation=”create” and bulkid to assign each record a unique ID

- response.badargument - number of times omelasticsearch received a response that had a status indicating omelasticsearch sent bad data to Elasticsearch. For example, status 400 and an error message indicating omelasticsearch attempted to store a non-numeric string value in a numeric field.

- response.bulkrejection - number of times omelasticsearch received a response that had a status indicating Elasticsearch was unable to process the record at this time - status 429. The record can be retried.

- response.other - number of times omelasticsearch received a response not recognized as one of the above responses, typically some other 4xx or 5xx HTTP status.

- rebinds - if using rebindinterval this will be the number of times omelasticsearch has reconnected to Elasticsearch

The fail.httprequests and fail.http counters reflect only failures that omelasticsearch detected. Once it detects problems, it (usually, depends on circumstances) tell the rsyslog core that it wants to be suspended until the situation clears (this is a requirement for rsyslog output modules). Once it is suspended, it does NOT receive any further messages. Depending on the user configuration, messages will be lost during this period. Those lost messages will NOT be counted by impstats (as it does not see them).

Note that some previous (pre 7.4.5) versions of this plugin had different counters. These were experimental and confusing. The only ones really used were “submits”, which were the number of successfully processed messages and “connfail” which were equivalent to “failed.http”.

## How Retries Are Handled

When using retryfailures=”on” (retryfailures), the original Message object (that is, the original smsg_t *msg object) is not available. This means none of the metadata associated with that object, such as various timestamps, hosts/ip addresses, etc. are not available for the retry operation. The only thing available are the metadata header (_index, _type, _id, pipeline, _parent) and original JSON string sent in the original request, and whatever data is returned in the error response. All of these are made available in the $.omes fields. If the same field name exists in the request metadata and the response, the field from the request will be used, in order to facilitate retrying the exact same request. For the message to retry, the code will take the original JSON string and parse it back into an internal Message object. This means you may need to use a different template to output messages for your retry ruleset. For example, if you used the following template to format the Elasticsearch message for the initial submission:

    template(name="es_output_template"
             type="list"
             option.json="on") {
               constant(value="{")
                 constant(value="\"timestamp\":\"")      property(name="timereported" dateFormat="rfc3339")
                 constant(value="\",\"message\":\"")     property(name="msg")
                 constant(value="\",\"host\":\"")        property(name="hostname")
                 constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
                 constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
                 constant(value="\",\"syslogtag\":\"")   property(name="syslogtag")
               constant(value="\"}")
             }

You would have to use a different template for the retry, since none of the timereported, msg, etc. fields will have the same values for the retry as for the initial try.

Same with the other omelasticsearch parameters which can be constructed with templates, such as “dynpipelinename”, “dynsearchindex”, “dynsearchtype”, “dynparent”, and “dynbulkid”. For example, if you generate the _id to use in the request, you will want to reuse the same _id for each subsequent retry:

    template(name="id-template" type="string" string="%$.es_msg_id%")
    if strlen($.omes!_id) > 0 then {
        set $.es_msg_id = $.omes!_id;
    } else {
        # NOTE: depends on rsyslog being compiled with --enable-uuid
        set $.es_msg_id = $uuid;
    }
    action(type="omelasticsearch" bulkid="id-template" ...)

That is, if this is a retry, $.omes!_id will be set, so use that value for the bulk id for this record, otherwise, generate a new one with $uuid. Note that the template uses the temporary variable $.es_msg_id which must be set each time, to either $.omes!_id or $uuid.

The rawmsg field is a special case. If the original request had a field called message, then when constructing the new message from the original to retry, the rawmsg message property will be set to the value of the message field. Otherwise, the rawmsg property value will be set to the entire original request - the data part, not the metadata. In previous versions, without the message field, the rawmsg property was set to the value of the data plus the Elasticsearch metadata, which caused problems with retries. See rsyslog issue 3573

## Examples

### Example 1

The following sample does the following:

- loads the omelasticsearch module

- outputs all logs to Elasticsearch using the default settings

    module(load="omelasticsearch")
    action(type="omelasticsearch")

### Example 2

The following sample does the following:

- loads the omelasticsearch module

- outputs all logs to Elasticsearch using the full JSON logging template including program name

    module(load="omelasticsearch")
    action(type="omelasticsearch" template="FullJSONFmt")

### Example 3

The following sample does the following:

- loads the omelasticsearch module

- defines a template that will make the JSON contain the following properties

  - RFC-3339 timestamp when the event was generated

  - the message part of the event

  - hostname of the system that generated the message

  - severity of the event, as a string

  - facility, as a string

  - the tag of the event

- outputs to Elasticsearch with the following settings

  - host name of the server is myserver.local

  - port is 9200

  - JSON docs will look as defined in the template above

  - index will be “test-index”

  - type will be “test-type”

  - activate bulk mode. For that to work effectively, we use an in-memory queue that can hold up to 5000 events. The maximum bulk size will be 300

  - retry indefinitely if the HTTP request failed (eg: if the target server is down)

    module(load="omelasticsearch")
    template(name="testTemplate"
             type="list"
             option.json="on") {
               constant(value="{")
                 constant(value="\"timestamp\":\"")      property(name="timereported" dateFormat="rfc3339")
                 constant(value="\",\"message\":\"")     property(name="msg")
                 constant(value="\",\"host\":\"")        property(name="hostname")
                 constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
                 constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
                 constant(value="\",\"syslogtag\":\"")   property(name="syslogtag")
               constant(value="\"}")
             }
    action(type="omelasticsearch"
           server="myserver.local"
           serverport="9200"
           template="testTemplate"
           searchIndex="test-index"
           searchType="test-type"
           bulkmode="on"
           maxbytes="100m"
           queue.type="linkedlist"
           queue.size="5000"
           queue.dequeuebatchsize="300"
           action.resumeretrycount="-1")

### Example 4

The following sample shows how to use writeoperation with dynbulkid and bulkid. For simplicity, it assumes rsyslog has been built with –enable-libuuid which provides the uuid property for each record:

    module(load="omelasticsearch")
    set $!es_record_id = $uuid;
    template(name="bulkid-template" type="list") { property(name="$!es_record_id") }
    action(type="omelasticsearch"
           ...
           bulkmode="on"
           bulkid="bulkid-template"
           dynbulkid="on"
           writeoperation="create")

### Example 5

The following sample shows how to use retryfailures to process, discard, or retry failed operations. This uses writeoperation=”create” with a unique bulkid so that we can check for and discard duplicate messages as successful. The try_es ruleset is used both for the initial attempt and any subsequent retries. The code in the ruleset assumes that if $.omes!status is set and is non-zero, this is a retry for a previously failed operation. If the status was successful, or Elasticsearch said this was a duplicate, the record is already in Elasticsearch, so we can drop the record. If there was some error processing the response e.g. Elasticsearch sent a response formatted in some way that we did not know how to process, then submit the record to the error_es ruleset. If the response was a “hard” error like 400, then submit the record to the error_es ruleset. In any other case, such as a status 429 or 5xx, the record will be resubmitted to Elasticsearch. In the example, the error_es ruleset just dumps the records to a file.

    module(load="omelasticsearch")
    module(load="omfile")
    template(name="bulkid-template" type="list") { property(name="$.es_record_id") }

    ruleset(name="error_es") {
            action(type="omfile" template="RSYSLOG_DebugFormat" file="es-bulk-errors.log")
    }

    ruleset(name="try_es") {
        if strlen($.omes!status) > 0 then {
            # retry case
            if ($.omes!status == 200) or ($.omes!status == 201) or (($.omes!status == 409) and ($.omes!writeoperation == "create")) then {
                stop # successful
            }
            if ($.omes!writeoperation == "unknown") or (strlen($.omes!error!type) == 0) or (strlen($.omes!error!reason) == 0) then {
                call error_es
                stop
            }
            if ($.omes!status == 400) or ($.omes!status < 200) then {
                call error_es
                stop
            }
            # else fall through to retry operation
        }
        if strlen($.omes!_id) > 0 then {
            set $.es_record_id = $.omes!_id;
        } else {
            # NOTE: depends on rsyslog being compiled with --enable-uuid
            set $.es_record_id = $uuid;
        }
        action(type="omelasticsearch"
                  ...
                  bulkmode="on"
                  bulkid="bulkid-template"
                  dynbulkid="on"
                  writeoperation="create"
                  retryfailures="on"
                  retryruleset="try_es")
    }
    call try_es

Support: rsyslog Assistant | GitHub Discussions | GitHub Issues: rsyslog source project

Contributing: Source & docs: rsyslog source project

© 2008–2025 Rainer Gerhards and others. Licensed under the Apache License 2.0.