# Liblognorm 2.1.0 documentation

## TurboVM Bytecode Engine

TurboVM is an optional high-performance normalization engine for liblognorm. It compiles rulebases into bytecode at startup and executes them through a linear virtual machine with SIMD-accelerated parsing primitives. When enabled, it provides significant throughput improvements over the default recursive parser, especially on high-volume log streams.

## Overview

The default liblognorm normalization engine (the “recursive walker”) traverses the parse DAG node-by-node for each log message, allocating json-c objects as fields are extracted. TurboVM replaces this with:

- Bytecode compilation: rulebases are compiled into a compact instruction sequence at load time. Each rule becomes a linear program.

- Arena allocation: all per-message memory comes from a single pre-allocated arena (~16 KB), fitting in L1 cache. Zero malloc/free per message.

- SIMD parsing: character scanning, delimiter search, whitespace skipping, and IP address parsing use SSE4.2 or NEON intrinsics when available.

- Typed field output: extracted fields carry their native type (string, integer, double, boolean) instead of converting everything to JSON strings.

- Nested JSON: dotted field names (e.g. `source.ip`) produce properly nested JSON objects (`{"source":{"ip":"..."}}`), enabling direct ECS (Elastic Common Schema) output.

## Building with TurboVM

TurboVM is an optional build feature, disabled by default:

    ./configure --enable-turbo

The build system automatically detects the CPU architecture and enables the appropriate SIMD instruction set:

- x86-64: SSE4.2 (Intel Nehalem+, AMD Bulldozer+)

- ARM64: NEON (all ARMv8-A processors, including Apple M1/M2)

- Other: scalar fallback (functional but without SIMD acceleration)

No additional dependencies are required.

## Using with lognormalizer

The `lognormalizer` command-line tool supports turbo mode via the `-oturbo` option:

    $ lognormalizer -r rules.rb -e json -oturbo < messages.log

In turbo mode:

- Normalization uses the TurboVM bytecode engine

- Output is compact JSON with nested objects for dotted field names

- Numeric fields are emitted as native JSON numbers (not strings)

- The `getline()` system call is used for input (more efficient than `fgets()` for large-scale processing)

If a rulebase cannot be compiled to bytecode (e.g. it uses unsupported parser types), lognormalizer falls back to standard normalization automatically.

## Library API

To enable TurboVM in your application, set the `LN_CTXOPT_TURBO` option on the normalization context before loading rules:

    #include <liblognorm.h>

    ln_ctx ctx = ln_initCtx();
    ln_setCtxOpts(ctx, LN_CTXOPT_TURBO);
    ln_loadSamples(ctx, "/path/to/rules.rb");

After loading, verify that compilation succeeded:

    if (ln_turbo_is_available(ctx)) {
        /* TurboVM ready — ln_normalize() will use the fast path */
    }

For direct string output (bypassing json-c entirely):

    char *json_str = NULL;
    size_t json_len = 0;
    int r = ln_normalize_to_str(ctx, msg, msg_len, &json_str, &json_len);
    if (r == 0 && json_str) {
        /* json_str contains the normalized JSON string */
        free(json_str);
    }

The standard `ln_normalize()` function also benefits from TurboVM when it is enabled — the bytecode engine is used internally, with automatic fallback to the recursive walker if needed.

## High-performance API (lognorm-turbo.h)

Consumers that want to avoid json-c construction entirely (for example rsyslog’s `mmnormalize` worker hot path) can use the curated public header `lognorm-turbo.h`. It is installed alongside `liblognorm.h` and `lognorm-features.h` and gated on `LOGNORM_TURBO_SUPPORTED`:

    #include <liblognorm/lognorm-features.h>
    #if defined(LOGNORM_TURBO_SUPPORTED)
    #include <liblognorm/lognorm-turbo.h>
    #endif

The header exposes only opaque types and a function-level contract; the internal `turbo*.h` headers and the fast-result/snapshot struct layouts are not installed and may change between releases without affecting the ABI. The contract covers:

- `ln_turbo_normalize_raw()` — normalize into an opaque, context-owned result (valid until the next normalize call on that context);

- `ln_turbo_snapshot_result()` / `ln_fast_result_snapshot_get()` / `ln_fast_result_snapshot_free()` — retain a result beyond the next call;

- typed accessors `ln_fast_result_field_count()`, `ln_fast_result_get_field()`, `ln_fast_result_get_field_typed()` (preserves `LN_FTYPE_*` value type and the `LN_FFIELD_NESTED` flag), `ln_fast_result_get_string()` / `_get_int()`, the tag accessors and `ln_fast_result_get_rule_id()`.

## Supported Parsers

TurboVM supports 32 of the 33 parser types defined in liblognorm v2. The following parsers are compiled to bytecode:

- Text: `word`, `alpha`, `string`, `rest`, `char-to`, `char-separated`, `string-to`, `op-quoted-string`, `quoted-string`, `literal`

- Numeric: `number`, `float`, `hexnumber`

- Network: `ipv4`, `ipv6`, `mac48`

- Date/Time: `date-rfc3164`, `date-rfc5424`, `date-iso`, `time-24hr`, `time-12hr`, `duration`, `kernel-timestamp`

- Structured: `json`, `cee-syslog`, `cef`, `v2-iptables`, `checkpoint-lea`, `name-value-list`

- Special: `whitespace` (as skip), `cisco-interface-spec`

The following parser type falls back to the legacy recursive engine:

- repeat: requires recursive sub-rule invocation, which is outside the scope of the single-pass VM instruction set.

The fallback is automatic and transparent — rulebases using `repeat` will still work correctly via the standard engine.

## Performance Notes

Throughput improvements depend on the rulebase complexity and message format. Typical observations:

- Simple rulebases (5-10 rules): 2-3x throughput improvement

- Complex rulebases (50+ rules with alternatives): 5-10x improvement

- The `ln_normalize_to_str()` path avoids json-c entirely and provides the highest throughput for applications that consume JSON as strings

TurboVM adds no overhead when disabled (`--disable-turbo` or default). When enabled but compilation fails for a specific rule, only that rule falls back to the recursive walker — other rules still use bytecode.