TurboVM Bytecode Engine
=======================

TurboVM is an optional high-performance normalization engine for liblognorm.
It compiles rulebases into bytecode at startup and executes them through a
linear virtual machine with SIMD-accelerated parsing primitives. When enabled,
it provides significant throughput improvements over the default recursive
parser, especially on high-volume log streams.

Overview
--------

The default liblognorm normalization engine (the "recursive walker") traverses
the parse DAG node-by-node for each log message, allocating json-c objects as
fields are extracted. TurboVM replaces this with:

- **Bytecode compilation**: rulebases are compiled into a compact instruction
  sequence at load time. Each rule becomes a linear program.
- **Arena allocation**: all per-message memory comes from a single pre-allocated
  arena (~16 KB), fitting in L1 cache. Zero malloc/free per message.
- **SIMD parsing**: character scanning, delimiter search, whitespace skipping,
  and IP address parsing use SSE4.2 or NEON intrinsics when available.
- **Typed field output**: extracted fields carry their native type (string,
  integer, double, boolean) instead of converting everything to JSON strings.
- **Nested JSON**: dotted field names (e.g. ``source.ip``) produce properly
  nested JSON objects (``{"source":{"ip":"..."}}``), enabling direct ECS
  (Elastic Common Schema) output.

Building with TurboVM
---------------------

TurboVM is an optional build feature, disabled by default::

    ./configure --enable-turbo

The build system automatically detects the CPU architecture and enables the
appropriate SIMD instruction set:

- **x86-64**: SSE4.2 (Intel Nehalem+, AMD Bulldozer+)
- **ARM64**: NEON (all ARMv8-A processors, including Apple M1/M2)
- **Other**: scalar fallback (functional but without SIMD acceleration)

No additional dependencies are required.

Using with lognormalizer
------------------------

The ``lognormalizer`` command-line tool supports turbo mode via the
``-oturbo`` option::

    $ lognormalizer -r rules.rb -e json -oturbo < messages.log

In turbo mode:

- Normalization uses the TurboVM bytecode engine
- Output is compact JSON with nested objects for dotted field names
- Numeric fields are emitted as native JSON numbers (not strings)
- The ``getline()`` system call is used for input (more efficient than
  ``fgets()`` for large-scale processing)

If a rulebase cannot be compiled to bytecode (e.g. it uses unsupported
parser types), lognormalizer falls back to standard normalization
automatically.

Library API
-----------

To enable TurboVM in your application, set the ``LN_CTXOPT_TURBO`` option
on the normalization context before loading rules::

    #include <liblognorm.h>

    ln_ctx ctx = ln_initCtx();
    ln_setCtxOpts(ctx, LN_CTXOPT_TURBO);
    ln_loadSamples(ctx, "/path/to/rules.rb");

After loading, verify that compilation succeeded::

    if (ln_turbo_is_available(ctx)) {
        /* TurboVM ready — ln_normalize() will use the fast path */
    }

For direct string output (bypassing json-c entirely)::

    char *json_str = NULL;
    size_t json_len = 0;
    int r = ln_normalize_to_str(ctx, msg, msg_len, &json_str, &json_len);
    if (r == 0 && json_str) {
        /* json_str contains the normalized JSON string */
        free(json_str);
    }

The standard ``ln_normalize()`` function also benefits from TurboVM
when it is enabled — the bytecode engine is used internally, with
automatic fallback to the recursive walker if needed.

High-performance API (lognorm-turbo.h)
--------------------------------------

Consumers that want to avoid json-c construction entirely (for example
rsyslog's ``mmnormalize`` worker hot path) can use the curated public
header ``lognorm-turbo.h``. It is installed alongside ``liblognorm.h``
and ``lognorm-features.h`` and gated on ``LOGNORM_TURBO_SUPPORTED``::

    #include <liblognorm/lognorm-features.h>
    #if defined(LOGNORM_TURBO_SUPPORTED)
    #include <liblognorm/lognorm-turbo.h>
    #endif

The header exposes only opaque types and a function-level contract; the
internal ``turbo*.h`` headers and the fast-result/snapshot struct layouts
are **not** installed and may change between releases without affecting the
ABI. The contract covers:

- ``ln_turbo_normalize_raw()`` — normalize into an opaque, context-owned
  result (valid until the next normalize call on that context);
- ``ln_turbo_snapshot_result()`` / ``ln_fast_result_snapshot_get()`` /
  ``ln_fast_result_snapshot_free()`` — retain a result beyond the next call;
- typed accessors ``ln_fast_result_field_count()``,
  ``ln_fast_result_get_field()``, ``ln_fast_result_get_field_typed()``
  (preserves ``LN_FTYPE_*`` value type and the ``LN_FFIELD_NESTED`` flag),
  ``ln_fast_result_get_string()`` / ``_get_int()``, the tag accessors and
  ``ln_fast_result_get_rule_id()``.

Supported Parsers
-----------------

TurboVM supports 32 of the 33 parser types defined in liblognorm v2.
The following parsers are compiled to bytecode:

- **Text**: ``word``, ``alpha``, ``string``, ``rest``, ``char-to``,
  ``char-separated``, ``string-to``, ``op-quoted-string``,
  ``quoted-string``, ``literal``
- **Numeric**: ``number``, ``float``, ``hexnumber``
- **Network**: ``ipv4``, ``ipv6``, ``mac48``
- **Date/Time**: ``date-rfc3164``, ``date-rfc5424``, ``date-iso``,
  ``time-24hr``, ``time-12hr``, ``duration``, ``kernel-timestamp``
- **Structured**: ``json``, ``cee-syslog``, ``cef``, ``v2-iptables``,
  ``checkpoint-lea``, ``name-value-list``
- **Special**: ``whitespace`` (as skip), ``cisco-interface-spec``

The following parser type falls back to the legacy recursive engine:

- **repeat**: requires recursive sub-rule invocation, which is outside
  the scope of the single-pass VM instruction set.

The fallback is automatic and transparent — rulebases using ``repeat``
will still work correctly via the standard engine.

Performance Notes
-----------------

Throughput improvements depend on the rulebase complexity and message
format. Typical observations:

- Simple rulebases (5-10 rules): 2-3x throughput improvement
- Complex rulebases (50+ rules with alternatives): 5-10x improvement
- The ``ln_normalize_to_str()`` path avoids json-c entirely and provides
  the highest throughput for applications that consume JSON as strings

TurboVM adds no overhead when disabled (``--disable-turbo`` or default).
When enabled but compilation fails for a specific rule, only that rule
falls back to the recursive walker — other rules still use bytecode.
