# Debugging with Record/Replay

The `ovs-replay` library provides a set of internal functions for recording certain events for later replay. This library is integrated into the `stream` and some other modules to record all incoming data across all streams (ssl, tcp, unixctl) of applications based on Open vSwitch libraries and play these streams later for debugging or performance testing purposes.

Support for this feature is currently integrated into the `ovsdb-server` and `ovsdb-client` applications. As a result, this allows to record lifecycle of the `ovsdb-server` process in large OVN deployments. Later, by using only the recorded data, the user can replay transactions and connections that occurred in a large deployment on their local PC. At the same time it is possible to tweak various log levels, run a process under debugger or tracer, measure performance with `perf`, and so on.

> Note

> The current version of record/replay engine does not work correctly with internal time-based events that leats to communications with other processes. For this reason it can not be used with clustered databases (RAFT implementation is heavily time dependent). In addition, recording automatically disables inactivity probes on JSONRPC connections and updates for the Manager status in a _Server database.

High-level feature overview was presented on Open vSwitch and OVN 2020 Fall Conference: Debugging OVSDB with stream record/replay

## Recording ovsdb-server events

To start recording events for the `ovsdb-server` process, there is a special command line argument `--record`. Before starting the database server, make sure that you have a copy of a database file, so you can use it for replay later. Here are the general steps to take:

1. Create a directory where the replay files will be stored:

       $ mkdir replay-dir
       $ REPLAY_DIR=$(pwd)/replay-dir

2. Copy the current database file for later use:

       $ cp my_database $REPLAY_DIR/

3. Run `ovsdb-server` with recording enabled:

       $ ovsdb-server --record=$REPLAY_DIR <other arguments> my_database

4. Work with the database as usual.

5. Stop the `ovsdb-server` process at the end (it is important to send an `exit` command so that during replay the process will exit in the end too):

       $ ovs-appctl -t ovsdb-server exit

After that `$REPLAY_DIR` should contain replay files with recorded data.

## Replay of recorded session

During replay, the `ovsdb-server` will receive all the same connections, transactions and commands as it had at the time of recording, but it will not create any actual network/socket connections and will not communicate with any other process. Everything will be read from the replay files.

Since there is no need to wait for IPC, all events will be received one by one without any delays, so the application will process them as quickly as possible. This can be used as a performance test where the user can measure how quickly the `ovsdb-server` can handle some workload recorded in a real deployment.

The command line argument to start a replay session is `--replay`. The steps will look like this:

1. Restore the database file from a previous copy:

       $ cp $REPLAY_DIR/my_database my_database

2. Start `ovsdb-server` with the same set of arguments as in the recording stage, except for `--record`:

       $ ovsdb-server --replay=$REPLAY_DIR <other arguments> my_database

3. The process should exit in the end when the `exit` command is replayed.

On step 2 it is possible to add extra logging arguments to debug some recorded issue, or run the process under debugger. It’s also possible to replay with a different version of `ovsdb-server` binary as long as this does not affect the data that goes in and out of the process, e.g. pure performance optimizations.

### Limitations

The record/replay engine has the following limitations:

1. Record/Replay of clustered databases is not supported.

2. Inactivity probes on JSONRPC connections are suppressed.

3. Manager status updates suppressed in `ovsdb-server`.

To remove above limitations, it is necessary to implement correct handling of internally generated time-based events. (possibly by recording of time and subsequent time warping).