# Example: JSON Parse and Transformation

Demonstrates a complete mini log pipeline that parses JSON logs, converts dotted keys into nested objects with `unflatten`, and writes a clean structured subtree.

## Overview

This example illustrates how the log pipeline concept applies in a real configuration. It transforms raw QRadar-style JSON messages with dotted keys (e.g., `src.ip`) into fully structured nested JSON. The workflow shows the connection between input, ruleset logic, and output actions, using `mmjsonparse` and `mmjsontransform` modules.

            flowchart LR
      I["Input<br>(raw JSON)"]:::input --> P["mmjsonparse<br>parse JSON"]:::ruleset --> T["mmjsontransform<br>unflatten keys"]:::ruleset --> O["omfile<br>structured output"]:::action

      classDef input fill:#d5e8d4,stroke:#82b366;
      classDef ruleset fill:#dae8fc,stroke:#6c8ebf;
      classDef action fill:#ffe6cc,stroke:#d79b00;

## Configuration

The configuration below defines a small self-contained pipeline that reads JSON from the raw payload, builds a nested subtree, and writes it to a file.

JSON → transform → file (commented)

    # Template: output only the structured subtree we build
    template(name="outfmt" type="subtree" subtree="$!qradar_structured")

    # Load parser and transformer modules
    module(load="mmjsonparse")
    module(load="mmjsontransform")

    # Parse JSON directly from the raw message
    action(
      type="mmjsonparse"
      container="$!qradar"
      mode="find-json"
      useRawMsg="on"
    )

    # Proceed only if parsing succeeded
    if $parsesuccess == "OK" then {
      # Convert dotted keys into nested JSON
      action(
        type="mmjsontransform"
        input="$!qradar"
        output="$!qradar_structured"
        mode="unflatten"
      )

      # Write the structured subtree to a file
      action(
        type="omfile"
        file="/var/log/qradar-structured.json"
        template="outfmt"
      )
    }

## How it fits into the log pipeline

- Input stage: receives messages (could be via `imfile` or network listener).

- Ruleset stage: parses and transforms JSON as shown.

- Action stage: writes the processed subtree using `omfile`.

This mirrors the core architecture introduced in The Log Pipeline, showing a practical transformation flow.

## Testing the configuration

1. Inject a sample JSON line containing dotted keys into rsyslog: `logger '{"src.ip":"10.0.0.5","dst.ip":"10.0.0.6"}'`

2. Verify output: `tail -n1 /var/log/qradar-structured.json` should show nested JSON: `"src": {"ip": "10.0.0.5"}, "dst": {"ip": "10.0.0.6"}`

3. Validate syntax: `rsyslogd -N1`

## Practical notes

- Use `useRawMsg="on"` when uncertain if a syslog header is present.

- Guard transformations with `$parsesuccess` to avoid malformed results.

- Subtree templates keep the output clean and deterministic.

- The `unflatten` mode handles dotted keys automatically, preserving hierarchy.

## Conclusion

This example demonstrates how rsyslog’s modular log pipeline can perform structured transformations inline. The same pattern can extend to larger architectures — for example, parsing JSON at the edge and forwarding structured logs via RELP or Kafka.

## See also

- Pipeline Stages

- Pipeline Design Patterns

- Pipeline Troubleshooting

- JSON Structured Content Extraction Module (mmjsonparse)

- JSON Dotted Key Transformer (mmjsontransform)

- omfile: File Output Module

Support: rsyslog Assistant | GitHub Discussions | GitHub Issues: rsyslog source project

Contributing: Source & docs: rsyslog source project

© 2008–2025 Rainer Gerhards and others. Licensed under the Apache License 2.0.