Hide navigation sidebar
Hide table of contents sidebar
Skip to content
Back to top
View this page
Toggle table of contents sidebar

# Benchmarks

Note

Benchmarks are hard.

Repeatedly calling the same function in a tight loop will lead to the instruction cache staying hot and branches being highly predictable. That’s not representative of real world access patterns. It’s also hard to write a nonbiased benchmark. I wrote msgspec, naturally whatever benchmark I publish it’s going to perform well in.

Even so, people like to see benchmarks. I’ve tried to be as nonbiased as I can be, and the results hopefully indicate a few tradeoffs you make when you choose different serialization formats. I encourage you to write your own benchmarks before making these decisions.

In all cases benchmarks were run on my local development machine (a ~2020 x86 Linux laptop) using CPython 3.11.

## JSON Serialization & Validation

This benchmark covers the common case when working with `msgspec` or other validation libraries. It measures two things:

- Decoding some JSON input, validating it against a schema, and converting it into user-friendly python objects.

- Encoding these same python objects back into JSON.

The data we’re working with has the following schema (defined here using `msgspec.Struct` types):

    import enum
    import datetime
    import msgspec

    class Permissions(enum.Enum):
        READ = "READ"
        WRITE = "WRITE"
        READ_WRITE = "READ_WRITE"

    class File(msgspec.Struct, kw_only=True, tag="file"):
        name: str
        created_by: str
        created_at: datetime.datetime
        updated_by: str | None = None
        updated_at: datetime.datetime | None = None
        nbytes: int
        permissions: Permissions

    class Directory(msgspec.Struct, kw_only=True, tag="directory"):
        name: str
        created_by: str
        created_at: datetime.datetime
        updated_by: str | None = None
        updated_at: datetime.datetime | None = None
        contents: list[File | Directory]

The libraries we’re comparing are the following:

- msgspec (0.18.5)

- mashumaro (3.11)

- pydantic (both 1.10.13 and 2.5.2)

- cattrs (23.2.3)

Each benchmark creates a message containing one or more `File`/`Directory` instances, then then serializes, deserializes, and validates it in a loop.

The full benchmark source can be found here.

In this benchmark `msgspec` is ~6x faster than `mashumaro`, ~10x faster than `cattrs`, and ~12x faster than `pydantic` V2, and ~85x faster than `pydantic` V1.

This plot shows the performance benefit of performing type validation during message decoding (as done by `msgspec`) rather than as a secondary step with a third-party library like cattrs or pydantic V1. Validating after decoding is slower for two reasons:

- It requires traversing over the entire output structure a second time (which can be slow due to pointer chasing)

- It may require converting some python objects to their desired output types (e.g. converting a decoded `dict` to a pydantic model), resulting in allocating many temporary python objects.

In contrast, libraries like `msgspec` that validate during decoding have none of these issues. Only a single pass over the decoded data is taken, and the specified output types are created correctly the first time, avoiding the need for additional unnecessary allocations.

This benefit also shows up in the memory usage for the same benchmark:

Here we compare the peak increase in memory usage (RSS) after loading the schemas and data. `msgspec`’s small library size, schema representation, and in-memory state means it uses a fraction of the memory of other tools.

## JSON Serialization

`msgspec` includes its own high performance JSON library, which may be used by itself as a replacement for the standard library’s `json.dumps`/`json.loads` functions. Here we compare msgspec’s JSON implementation against several other popular Python JSON libraries.

- msgspec (0.18.5)

- orjson (3.9.10)

- ujson (5.9.0)

- rapidjson (1.13)

- simdjson (5.0.2)

- json (standard library)

The full benchmark source can be found here.

In this case `msgspec structs` (which measures `msgspec` with `msgspec.Struct` schemas pre-defined) is the fastest. When used without schemas, `msgspec` is on-par with `orjson` (the next fastest JSON library).

This shows that `msgspec` is able to decode JSON faster when a schema is provided. Due to a more efficient in memory representation, JSON decoding AND schema validation with `msgspec` than just JSON decoding alone.

## MessagePack Serialization

Likewise, `msgspec` includes its own high performance MessagePack library, which may be used by itself without requiring usage of any of msgspec’s validation machinery. Here we compare msgspec’s MessagePack implementation against several other popular Python MessagePack libraries.

- msgspec (0.18.5)

- msgpack (1.0.7)

- ormsgpack (1.4.1)

As with the JSON benchmark above, `msgspec` with a schema provided (`msgspec structs`) is faster than `msgspec` with no schema. In both cases though `msgspec` is measurably faster than other Python MessagePack libraries like `msgpack` or `ormsgpack`.

## JSON Serialization - Large Data

Here we benchmark loading a large JSON file (~77 MiB) containing information on all the `noarch` packages in conda-forge. We compare the following libraries:

- msgspec (0.18.5)

- orjson (3.9.10)

- ujson (5.9.0)

- rapidjson (1.13)

- simdjson (5.0.2)

- json (standard library)

For each library, we measure both the peak increase in memory usage (RSS) and the time to JSON decode the file.

The full benchmark source can be found here.

Results (smaller is better):

| memory (MiB) | vs. | time (ms) | vs.
--- | --- | --- | --- | ---
msgspec structs | 67.6 | 1.0x | 176.8 | 1.0x
msgspec | 218.3 | 3.2x | 630.5 | 3.6x
json | 295.0 | 4.4x | 868.6 | 4.9x
ujson | 349.1 | 5.2x | 1087.0 | 6.1x
rapidjson | 375.0 | 5.6x | 1004.0 | 5.7x
orjson | 406.3 | 6.0x | 691.7 | 3.9x
simdjson | 603.2 | 8.9x | 1053.0 | 6.0x

- `msgspec` decoding into Struct types uses the least amount of memory, and is also the fastest to decode. This makes sense; `Struct` types are cheaper to allocate and more memory efficient than `dict` types, and for large messages these differences can really add up.

- `msgspec` decoding without a schema is the second best option for both memory usage and speed. When decoding without a schema, `msgspec` makes the assumption that the underlying message probably still has some structure; short dict keys are temporarily cached to be reused later on, rather than reallocated every time. This means that instead of allocating 10,000 copies of the string `"name"`, only a single copy is allocated and reused. For large messages this can lead to significant memory savings. `json` and `orjson` also use similar optimizations, but not as effectively.

- `orjson` and `simdjson` use 6-9x more memory than `msgspec` in this benchmark. In addition to the reasons above, both of these decoders require copying the original message into a temporary buffer. In this case, the extra copy adds an extra 77 MiB of overhead!

## Structs

Here we benchmark common `msgspec.Struct` operations, comparing their performance against other similar libraries. The cases compared are:

- Standard Python classes

- dataclasses

- msgspec (0.18.5)

- attrs (23.1.0)

- pydantic (2.5.2)

For each library, the following operations are benchmarked:

- Time to define a new class. Many libraries that abstract away class boilerplate add overhead when defining classes, slowing import times for libraries that make use of these classes.

- Time to create an instance of that class.

- Time to compare two instances for equality (`==`/`!=`).

- Time to compare two instances for order (`<`/`>`/`<=`/`>=`)

The full benchmark source can be found here.

Results (smaller is better):

| import (μs) | create (μs) | equality (μs) | order (μs)
--- | --- | --- | --- | ---
msgspec | 12.51 | 0.09 | 0.02 | 0.03
standard classes | 7.88 | 0.35 | 0.08 | 0.16
attrs | 483.10 | 0.37 | 0.14 | 1.87
dataclasses | 506.09 | 0.36 | 0.14 | 0.16
pydantic | 673.47 | 1.54 | 0.60 | N/A

- Standard Python classes are the fastest to import (any library can only add overhead here). Still, `msgspec` isn’t that much slower, especially compared to other options.

- Structs are optimized to be cheap to create, and that shows for the creation benchmark. They’re roughly 4x faster than standard classes/`attrs`/`dataclasses`, and 17x faster than `pydantic`.

- For equality comparison, msgspec Structs are roughly 4x to 30x faster than the alternatives.

- For order comparison, msgspec Structs are roughly 5x to 60x faster than the alternatives.

## Garbage Collection

`msgspec.Struct` instances implement several optimizations for reducing garbage collection (GC) pressure and decreasing memory usage. Here we benchmark structs (with and without gc=False) against standard Python classes (with and without __slots__).

For each option we create a large dictionary containing many simple instances of the benchmarked type, then measure:

- The amount of time it takes to do a full garbage collection (gc) pass

- The total amount of memory used by this data structure

The full benchmark source can be found here.

Results (smaller is better):

| GC time (ms) | Memory Used (MiB)
--- | --- | ---
standard class | 80.46 | 211.66
standard class with __slots__ | 80.06 | 120.11
msgspec struct | 13.96 | 120.11
msgspec struct with gc=False | 1.07 | 104.85

- Standard Python classes are the most memory hungry (since all data is stored in an instance dict). They also result in the largest GC pause, as the GC has to traverse the entire outer dict, each class instance, and each instance dict. All that pointer chasing has a cost.

- Standard classes with `__slots__` are less memory hungry, but still results in an equivalent GC pauses.

- `msgspec.Struct` instances have the same memory layout as a class with `__slots__` (and thus have the same memory usage), but due to deferred GC tracking a full GC pass completes in a fraction of the time.

- `msgspec.Struct` instances with `gc=False` have the lowest memory usage (lack of GC reduces memory by 16 bytes per instance). They also have the lowest GC pause (75x faster than standard classes!) since the entire composing dict can be skipped during GC traversal.

## Library Size

Here we compare the on-disk size of `msgspec` and `pydantic`, its closest equivalent.

The full benchmark source can be found here.

Results (smaller is better)

| version | size (MiB) | vs. msgspec
--- | --- | --- | ---
msgspec | 0.18.4 | 0.46 | 1.00x
pydantic | 2.5.2 | 6.71 | 14.66x

For applications where dependency size matters, `msgspec` is roughly 15x smaller on disk.