# Developer Documentation

This documentation is intended for people who wants to contribute or work with the internals of the project, it is not for end-users.

## Storage

Since version 0.9, Libervia uses SQLAlchemy with its Object–Relational Mapping as a backend to store persistent data, and Alembic is used to handle schema and data migrations.

SQLite is currently the only supported database, but it is planned to add support for other ones (notably PostgreSQL), probably during the development of 0.9 version.

The mapping is done in `libervia.backend.memory.sqla_mapping` and working with database is done through high level methods found in `libervia.backend.memory.sqla`.

Before the move to SQLAlchemy, there was a strict separation between database implementation and the rest of the code. With 0.9, objects mapping to database can be used and manipulated directly outside of `libervia.backend.memory.sqla` to take profit of SQLAlchemy possibilities.

Database state is detected when the backend starts, and the database will be created or migrated automatically if necessary.

To create a new migration script, `Alembic` may be used directly. To do so, be sure to have an up-to-date database (and a backup in case of troubles), then activate the virtual environment where Libervia is installed (Alembic needs to access ORM mapping), go to `libervia/backend/memory/migration` directory, and enter the following command:

    alembic revision --autogenerate -m "some revision message"

This will create a base migration file in `versions` directory. Adapt it to your needs, try to create both `upgrade` and `downgrade` method whenever possible, and be sure to test it in both directions (`alembic upgrade head` and `alembic downgrade <previous_revision>`). Please check Alembic documentation for more details.

## Pubsub Cache

There is an internal cache for pubsub nodes and items, which is done in `plugin_pubsub_cache`. The `PubsubNode` and `PubsubItem` class are the one mapping the database.

The cache is operated transparently to end-user, when a pubsub request is done, it uses a trigger to check if the requested node is or must be cached, and if possible returns result directly from database, otherwise it lets the normal workflow continue and query the pubsub service.

To save resources, not all nodes are fully cached. When a node is checked, a series of analysers are checked, and the first one matching is used to determine if the node must be synchronised or not.

Analysers can be registered by any plugins using `register_analyser` method:

PubsubCache.register_analyser(analyser: dict) → None

  Register a new pubsub node analyser

  Parameters:

    analyser – An analyser is a dictionary which may have the following keys

  (keys with a `*` are mandatory, at least one of `node` or `namespace` keys must be used):

  > name (str)*

  >   a unique name for this analyser. This name will be stored in database to retrieve the analyser when necessary (notably to get the parsing method), thus it is recommended to use a stable name such as the source plugin name instead of a name which may change with standard evolution, such as the feature namespace.

  > type (str)*

  >   indicates what kind of items we are dealing with. Type must be a human readable word, as it may be used in searches. Good types examples are blog or event.

  > node (str)

  >   prefix of a node name which may be used to identify its type. Example: urnxmpp:microblog0 (a node starting with this name will be identified as blog node).

  > namespace (str)

  >   root namespace of items. When analysing a node, the first item will be retrieved. The analyser will be chosen its given namespace match the namespace of the first child element of `<item>` element.

  > to_sync (bool)

  >   if True, the node must be synchronised in cache. The default False value means that the pubsub service will always be requested.

  > parser (callable)

  >   method (which may be sync, a coroutine or a method returning a “Deferred”) to call to parse the `domish.Element` of the item. The result must be dictionary which can be serialised to JSON.

  >   The method must have the following signature:

  >   parser(client: SatXMPPEntity, item_elt: domish.Element, service: jid.JID | None, node: str | None) → dict
  > match_cb (callable)

  >   method (which may be sync, a coroutine or a method returning a “Deferred”) called when the analyser matches. The method is called with the curreny analyse which is can modify in-place.

  >   The method must have the following signature:

  >   libervia.backend.plugins.plugin_pubsub_cache.match_cb(client: SatXMPPEntity, analyse: dict) → None

  Raises:

    exceptions.Conflict – a analyser with this name already exists

If no analyser is found, `to_sync` is false, or an error happens during the caching, the node won’t be synchronised and the pubsub service will always be requested.

Specifying an optional parser will store parsed data in addition to the raw XML of the items. This is more space consuming, but may be desired for the following reasons:

- the parsing is resource consuming (network call or some CPU intensive operations are done)

- it is desirable to do queries on parsed data. Indeed the parsed data are stored in a JSON field and its keys may be queried individually.

The Raw XML is kept as the cache operates transparently, and a plugin may need raw data, or an user may be doing a low-level pubsub request.

## Libervia Apps

Libervia has a mechanism to embed applications and configure them to integrate with the ecosystem. This is notably used to embed docker applications. Description of the application description file is available at Libervia App Configuration (YAML).