Skip to content

# Streaming I/O

Glaze supports streaming serialization and deserialization for processing large data with bounded memory usage. This enables reading/writing files of arbitrary size without loading everything into memory.

## Output Streaming (`basic_ostream_buffer`)

Write directly to files or output streams with incremental flushing:

    #include "glaze/core/ostream_buffer.hpp"

    // Write to file with concrete type (enables devirtualization)
    std::ofstream file("output.json");
    glz::basic_ostream_buffer<std::ofstream> buffer(file);
    auto ec = glz::write_json(obj, buffer);
    if (ec || !file.good()) {
        // Handle error
    }

    // Write to any std::ostream (polymorphic)
    glz::ostream_buffer<> buffer2(any_ostream);  // Alias for basic_ostream_buffer<std::ostream>
    glz::write_json(obj, buffer2);

Template parameters:

    // basic_ostream_buffer<Stream, Capacity>
    // - Stream: Output stream type (must satisfy byte_output_stream concept)
    // - Capacity: Initial buffer size in bytes (default 64KB)

    glz::basic_ostream_buffer<std::ofstream> buf1(file);           // Concrete type
    glz::basic_ostream_buffer<std::ofstream, 4096> buf2(file);     // 4KB buffer
    glz::ostream_buffer<> buf3(any_ostream);                       // Polymorphic, 64KB
    glz::ostream_buffer<4096> buf4(any_ostream);                   // Polymorphic, 4KB

## Input Streaming (`basic_istream_buffer`)

Read directly from files or input streams with automatic refilling:

    #include "glaze/core/istream_buffer.hpp"

    // Read from file with concrete type (enables devirtualization)
    std::ifstream file("input.json");
    glz::basic_istream_buffer<std::ifstream> buffer(file);
    my_struct obj;
    auto ec = glz::read_json(obj, buffer);
    if (ec) {
        // Handle error
    }

    // Read from any std::istream (polymorphic)
    glz::istream_buffer<> buffer2(any_istream);  // Alias for basic_istream_buffer<std::istream>
    glz::read_json(obj, buffer2);

Template parameters:

    // basic_istream_buffer<Stream, Capacity>
    // - Stream: Input stream type (must satisfy byte_input_stream concept)
    // - Capacity: Buffer size in bytes (default 64KB)

    glz::basic_istream_buffer<std::ifstream> buf1(file);           // Concrete type
    glz::basic_istream_buffer<std::ifstream, 4096> buf2(file);     // 4KB buffer
    glz::istream_buffer<> buf3(any_istream);                       // Polymorphic, 64KB
    glz::istream_buffer<4096> buf4(any_istream);                   // Polymorphic, 4KB

## Streaming JSON/NDJSON (`json_stream_reader`)

Process streams of JSON objects (NDJSON or multiple JSON values) one at a time:

    #include "glaze/json/json_stream.hpp"

    struct Event {
        int id;
        std::string type;
    };

    // Manual iteration
    std::ifstream file("events.ndjson");
    glz::json_stream_reader<Event> reader(file);
    Event event;
    while (!reader.read_next(event)) {
        process(event);
    }

    // Range-based for loop
    for (auto&& event : glz::json_stream_reader<Event>(file)) {
        process(event);
    }

    // Convenience function to read all values
    std::vector<Event> events;
    auto ec = glz::read_json_stream(events, file);

Key features:

- Processes one complete value at a time (bounded memory)
- Supports NDJSON (newline-delimited) and consecutive JSON values
- Iterator interface for range-based for loops
- Automatic whitespace/newline skipping between values

## Stream Concepts

Only byte-oriented streams are supported. Wide character streams are rejected at compile time:

    // OK - byte streams
    static_assert(glz::byte_output_stream<std::ostream>);
    static_assert(glz::byte_output_stream<std::ofstream>);
    static_assert(glz::byte_input_stream<std::istream>);
    static_assert(glz::byte_input_stream<std::ifstream>);

    // Compile error - wide streams not supported (JSON is UTF-8)
    // glz::basic_ostream_buffer<std::wostream> bad(wstream);  // Error!
    // glz::basic_istream_buffer<std::wistream> bad(wstream);  // Error!

## Format Support

Format | Output Streaming | Input Streaming
--- | --- | ---
JSON | `ostream_buffer` | `istream_buffer`
BEVE | `ostream_buffer` | `istream_buffer`
NDJSON | `ostream_buffer` | `json_stream_reader`

## See Also

- Writing - Writing to buffers and error handling
- Reading - Reading from buffers