..
      Licensed under the Apache License, Version 2.0 (the "License"); you may
      not use this file except in compliance with the License. You may obtain
      a copy of the License at

          http://www.apache.org/licenses/LICENSE-2.0

      Unless required by applicable law or agreed to in writing, software
      distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
      WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
      License for the specific language governing permissions and limitations
      under the License.

      Convention for heading levels in OVN documentation:

      =======  Heading 0 (reserved for the title in a document)
      -------  Heading 1
      ~~~~~~~  Heading 2
      +++++++  Heading 3
      '''''''  Heading 4

      Avoid deeper levels because they do not render well.

============
OVN Upgrades
============

Since OVN is a distributed system, special consideration must be given to
the process used to upgrade OVN across a deployment.  This document discusses
the two recommended `Upgrade procedures`_, `Rolling upgrade`_ and `Fail-safe
upgrade`_.

Which one to choose depends on whether you are running a version of OVN that is
within range of upstream support for upgrades to the version of OVN you want to
upgrade to.

Release Notes
-------------

You should always check the OVS and OVN release notes (NEWS file) for any
release specific notes on upgrades.

Open vSwitch
------------

OVN compiles with a particular version of Open vSwitch.  This is a build-time
dependency.

In runtime, OVN should be able to work with any reasonably fresh version of
Open vSwitch (not necessarily the version that it was compiled against.)

OVN may make use of new runtime features of Open vSwitch that are only
available in a particular release. OVN is expected to test for an Open vSwitch
feature presence before using it, and gracefully handle scenarios where Open
vSwitch doesn't support a particular optional feature, yet.

Upgrade procedures
------------------

Rolling upgrade
~~~~~~~~~~~~~~~

1. `Upgrade ovn-controller`_

2. `Upgrade OVN Databases and ovn-northd`_

3. `Upgrade OVN Integration`_

In order to successfully perform a rolling upgrade, the ovn-controller process
needs to understand the structure of the database for the version you are
upgrading from and to simultaneously.

To avoid buildup of complexity and technical debt we limit the span of versions
supported for a rolling upgrade on :ref:`long-term-support` (LTS), and it
should always be possible to upgrade from the previous LTS version to the next.

The following rolling upgrade paths are supported:

1. LTS to the very next LTS release, or to any non-LTS in between the two.
2. Any non-LTS to the very next LTS release.

The first LTS version of OVN was 22.03.  Subsequent LTS versions have
been released every two years since then.  For more information on the
lifetime of LTS versions, please see
https://www.ovn.org/en/releases/#long-term-support .  If you want to
upgrade between other versions, you can use the `Fail-safe upgrade`_
procedure.

Fail-safe upgrade
~~~~~~~~~~~~~~~~~

1. Upgrade to the most recent point release or package version available for
   the major version of OVN you are upgrading from.

2. Enable the version pinning feature in the ovn-controller by setting the
   ``external_ids:ovn-match-northd-version`` flag to 'true' as documented in
   the `ovn-controller man page`_.

3. If the version of OVN you are upgrading from does not have the `version
   pinning check in the incremental processing engine`_, you must stop
   ovn-northd and manually change the northd_internal_version to ensure the
   controllers go into fail-safe mode before processing changes induced by the
   upgrade.

    $ sudo /usr/share/ovn/scripts/ovn-ctl stop_northd --ovn-manage-ovsdb=no
    $ sudo ovn-sbctl set sb-global . options:northd_internal_version="foo"

4. `Upgrade OVN Databases and ovn-northd`_

5. `Upgrade ovn-controller`_

6. `Upgrade OVN Integration`_

When upgrading between a span of versions that is not supported, you may be at
risk for the new ovn-controller process not understanding the structure of the
old database, which may lead to data plane downtime for running instances.

To avoid this there is a fail safe approach, which involves making the
ovn-controller process refrain from making changes to the local flow state when
a version mismatch between the ovn-controller and ovn-northd is detected.

Steps
-----

This section documents individual steps in an upgrade procedure in no particular
order.  For information on ordering of the steps, please refer to the `Upgrade
procedures`_ section.

Upgrade ovn-controller
~~~~~~~~~~~~~~~~~~~~~~

You should start by upgrading ovn-controller on each host it's running on.
First, you upgrade the OVS and OVN packages.  Then, restart the
ovn-controller service.  You can restart with ovn-ctl::

    $ sudo /usr/share/ovn/scripts/ovn-ctl restart_controller

or with systemd::

    $ sudo systemctl restart ovn-controller

Upgrade OVN Databases and ovn-northd
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The OVN databases and ovn-northd should be upgraded next.  Since ovn-controller
has already been upgraded, it will be ready to operate on any new functionality
specified by the database or logical flows created by ovn-northd.

Upgrading the OVN packages installs everything needed for an upgrade.  The only
step required after upgrading the packages is to restart ovn-northd, which
automatically restarts the databases and upgrades the database schema, as well.

You may perform this restart using the ovn-ctl script::

    $ sudo /usr/share/ovn/scripts/ovn-ctl restart_northd

or if you're using a Linux distribution with systemd::

    $ sudo systemctl restart ovn-northd

In case your deployment utilizes OVN Interconnection (OVN IC) functionality,
it is also needed to restart ovn-ic daemons and separately restart ovn-ic
databases.

You may perform this restart using the ovn-ctl script::

    $ sudo /usr/share/ovn/scripts/ovn-ctl restart_ic
    $ sudo /usr/share/ovn/scripts/ovn-ctl restart_ic_ovsdb

or if you're using a Linux distribution with systemd::

    $ sudo systemctl restart ovn-ic
    $ sudo systemctl restart ovn-ic-db

Schema Change
+++++++++++++

During database upgrading, if there is schema change, the DB file will be
converted to the new schema automatically, if the schema change is backward
compatible.  OVN tries the best to keep the DB schemas backward compatible.

However, there can be situations that an incompatible change is reasonable.
An example of such case is to add constraints in the table to ensure
correctness.  If there were already data that violates the new constraints
got added somehow, it will result in DB upgrade failures.  In this case,
user should manually correct data using ovn-nbctl (for north-bound DB) or
ovn-sbctl (for south-bound DB), and then upgrade again following previous
steps.  Below is a list of known incompatible schema changes and how to
fix when error encountered.

Post-split releases (20.03+)
''''''''''''''''''''''''''''

#. Release 22.12: index [transit_switch, availability_zone, route_table,
   ip_prefix, nexthop] added for OVN Interconnection Southbound DB table
   Route.  If there are duplicated records in this table, users are
   advised to upgrade ovn-ic daemons in all availability zones first and
   after that convert the schema (restart ovn-ic database daemon).

Pre-split releases (OVN 2.x)
'''''''''''''''''''''''''''''

The following notes apply only when upgrading from versions where OVN
was part of the Open vSwitch tree.  They can be ignored when upgrading
between standalone OVN releases (20.03+).

#. Release 2.11: index [type, ip] added for Encap table of south-bound
   DB to prevent duplicated IPs being used for same tunnel type.  If
   there are duplicated data added already (e.g. due to improper chassis
   management), a convenient way to fix is to find the chassis that is
   using the IP with command::

    $ ovn-sbctl show

   Then delete the chassis with command::

    $ ovn-sbctl chassis-del <chassis>

#. Release 2.11 and earlier: The availability_zone name is added to the
   external_ids in NB_Global. Therefore, when upgrading to latest
   versions, update the name column in NB_Global for each availability
   zone before starting the ovn-ic daemons. The Northbound database
   upgrade does not handle populating the name column part of schema
   upgrade. Failing to do this could result in data plane impact, such
   as remote AZs being unable to update port bindings during gateway
   chassis failover, new gateway chassis CRUD, etc. Run below command
   to set the name column in NB_Global if upgrading from very old 2.*
   versions to latest/newer versions:

    $ ovn-nbctl set NB_Global . name=<availability zone name>

    Then restart ovn-ic daemons in all availability zones.

   Missing above step will result in CPU of ovn-ic active instance spike
   to 100%. This occurs due to null name column as ovn-ic in each
   availability zone repeatedly fails in committing transaction to
   Southbound in remote availability zones due to duplicate chassis.
   This will also result in manual clean up of remote availability zones
   gateway chassis of type interconnection in local availability zone
   causing data plane impact. Fix these issues by below command on each
   availability zone part of upgrade from very old 2.* versions after
   stopping ovn-ic on all availability zones:

    $ ovn-nbctl set NB_Global . name=<availability zone name>
    $ ovn-sbctl chassis-del <remote-ic-gateway-chassis-uuid>

Upgrade OVN Integration
~~~~~~~~~~~~~~~~~~~~~~~

Lastly, you may also want to upgrade integration with OVN that you may be
using.  For example, this could be the OpenStack Neutron driver or
ovn-kubernetes.

OVN's northbound database schema is a backwards compatible interface, so
you should be able to safely complete an OVN upgrade before upgrading
any integration in use.

.. LINKS
.. _ovn-controller man page:
   https://www.ovn.org/support/dist-docs/ovn-controller.8.html
.. _version pinning check in the incremental processing engine:
   https://github.com/ovn-org/ovn/commit/c2eeb2c98ea8
