# JSON Structured Content Extraction Module (mmjsonparse)

Module name:

  mmjsonparse

Introduced:

  6.6.0 (find-json mode added in 8.2510)

Author:

  Rainer Gerhards <rgerhards@adiscon.com>

Parses JSON-structured content in log messages and exposes fields as message properties. Supports legacy cookie-prefixed parsing and a flexible find-json scan mode. Provides scan counters to assess success, failure, and truncation.

## Overview

`mmjsonparse` extracts JSON content from incoming messages and exposes parsed fields beneath the structured data container (for example, `$!field`). It supports two operating modes:

- find-json (recommended): scans the message for the first valid top-level JSON object and parses it.

- cookie (legacy): parses JSON only when the message starts with the legacy `@cee:` prefix.

Use `mmjsonparse` early in the processing pipeline to make JSON fields available for filtering, normalization, or routing. The cookie mode is retained for backward compatibility and should not be used in new deployments.

### Introduced and Compatibility

The module has been available since rsyslog 6.6.0. The enhanced find-json scanning mode was introduced in version 8.2510 and is present in all later releases.

### Behavior Notes

- This is not a generic validator. If messages contain trailing non-JSON after the parsed object and trailing content is not allowed, parsing is considered failed according to the configured policy.

- Parameter names are case-insensitive. For readability, camelCase is recommended.

## Notable Features

- Check Parsing Result

- Statistics Counters

- Handling Parsing Failures

- Examples

## Parsing Modes

Mode | Description
--- | ---
`find-json` | Scans for the first top-level `{...}` anywhere in the message and parses it. Suited for modern app logs that embed JSON within other text.
`cookie` (legacy) | Parses only when the message begins with `@cee:` and is immediately followed by valid JSON. Kept for backward compatibility with historic CEE/lumberjack emitters.

## Configuration Parameters

### Action Parameters

Parameter | Summary
--- | ---
mode | Specifies the parsing mode for JSON content detection.
cookie | Defines the cookie string that must appear before the JSON content of a message.
max_scan_bytes | Maximum number of bytes to scan when searching for JSON content in find-json mode.
allow_trailing | Whether to allow non-whitespace data after the JSON object in find-json mode.
useRawMsg | Controls whether parsing operates on the raw message or only the MSG part.
container | Sets the JSON container path where parsed properties are stored.

## Check Parsing Result

Use the `$parsesuccess` variable to check whether JSON parsing succeeded.

    action(type="mmjsonparse" mode="find-json")

    if $parsesuccess == "OK" then {
        # downstream processing for structured messages
        action(type="omfile" file="/var/log/rsyslog/structured.log")
    } else if $parsesuccess == "FAIL" then {
        # handle failures (see dedicated section below)
        action(type="omfile" file="/var/log/rsyslog/nonconforming.log")
        stop
    }

## Handling Parsing Failures

If JSON extraction fails, `$parsesuccess` is set to `"FAIL"`. Best practice is to keep such messages separate for inspection. Two common patterns are shown below.

### A) Direct to non-conforming log with stop

    action(type="mmjsonparse" mode="find-json")

    if $parsesuccess == "FAIL" then {
        action(
          type="omfile"
          file="/var/log/rsyslog/nonconforming.log"
          template="RSYSLOG_TraditionalFileFormat"
        )
        stop    # prevent accidental downstream processing
    }

    # OK path continues as usual
    if $parsesuccess == "OK" then {
        action(type="omfile" file="/var/log/rsyslog/structured.log")
    }

### B) Handoff to a dedicated inspection ruleset (with stop)

    ruleset(name="inspectNonConforming"){
        # capture raw for triage
        template(name="rawmsg" type="string" string="%rawmsg%\\n")
        action(type="omfile" file="/var/log/rsyslog/nonconforming.raw" template="rawmsg")
        # optional: richer snapshot for short-term debugging
        # action(type="omfile" file="/var/log/rsyslog/nonconforming.meta" template="RSYSLOG_DebugFormat")
        stop
    }

    action(type="mmjsonparse" mode="find-json")

    if $parsesuccess == "FAIL" then {
        call inspectNonConforming
    } else {
        action(type="omfile" file="/var/log/rsyslog/structured.log")
    }

### Operational recommendations

- Use `stop` on the failure path if you must ensure the message does not reach later actions unintentionally. Omit `stop` if you intentionally want both paths.

- Apply a distinct retention policy to non-conforming logs; they can be bursty.

- For transient debugging, log `%rawmsg%` (and, if supported, an all-JSON snapshot) to aid triage.

- Run `rsyslogd -N1` after edits to validate configuration syntax.

- Consider preceding `mmjsonparse` with mmutf8fix if invalid encodings are suspected.

## Statistics Counters

When using find-json mode, `mmjsonparse` maintains scan-related counters (available via rsyslog’s statistics subsystem):

Counter | Description
--- | ---
`scan.attempted` | Messages inspected by the find-json scanner.
`scan.found` | Messages where a JSON object was located (may still fail later on trailing rules).
`scan.failed` | Messages where a located JSON object could not be parsed (malformed).
`scan.truncated` | Scans aborted due to the `max_scan_bytes` limit.

### Exposure and collection

- impstats (recommended): Configure `impstats` to emit periodic statistics as messages.

- Prometheus via imhttp: If imhttp is loaded with metricsPath, it exposes rsyslog statistics (including `mmjsonparse` counters) in Prometheus text format at the configured HTTP path. See imhttp’s Prometheus Metrics section and secure endpoints as needed (for example, `metricsBasicAuthFile`).

### Performance guidance

- Success rate: `scan.found / scan.attempted` — ability to locate JSON.

- Parse quality: `scan.failed / scan.found` — data quality problems.

- Scan efficiency: `scan.truncated / scan.attempted` — raise `max_scan_bytes` if frequent.

## Processing Flow (informative)

Note

The diagrams below are informational and do not alter behavior.

            flowchart TD
      A["msg in"] --> B["mmjsonparse<br>mode=find-json|cookie"]
      B -->| OK | C["fields under $!"]
      C --> D["structured path"]
      B -->| FAIL | E["non-conforming"]
      E --> F{"route"}
      F -->| file+stop | G["omfile<br>nonconforming.log"]
      F -->| ruleset+stop | H["inspectNonConforming()"]

### Metrics exposure (optional)

            flowchart LR
      M["mmjsonparse counters"] --> S["stats registry"]
      S --> I["imhttp<br>/metrics"]
      I --> P["Prometheus scrape"]

## Examples

### Enable default normalization (legacy cookie mode)

    module(load="mmjsonparse")
    action(type="mmjsonparse" mode="cookie")

### Permit parsing messages without cookie (do not require @cee:)

    action(type="mmjsonparse" cookie="")

### Find-JSON mode for embedded JSON content

    # Basic find-json mode
    action(type="mmjsonparse" mode="find-json")

    # With limits and strict trailing control
    action(type="mmjsonparse"
           mode="find-json"
           max_scan_bytes="32768"
           allow_trailing="off")

### Mixed mode processing

    if $msg startswith "@cee:" then {
        action(type="mmjsonparse" mode="cookie")
    } else if $msg contains "{" then {
        action(type="mmjsonparse" mode="find-json")
    }

## See also

- mmutf8fix — fix invalid UTF-8 before parsing

- mmnormalize — normalization after extraction

- omelasticsearch — indexing structured logs

- imhttp — expose rsyslog statistics in Prometheus text format via HTTP

Support: rsyslog Assistant | GitHub Discussions | GitHub Issues: rsyslog source project

Contributing: Source & docs: rsyslog source project

© 2008–2025 Rainer Gerhards and others. Licensed under the Apache License 2.0.