# Connecting VMs Using Tunnels (Userspace)

This document describes how to use Open vSwitch to allow VMs on two different hosts to communicate over VXLAN tunnels. Unlike Connecting VMs Using Tunnels, this configuration works entirely in userspace.

Note

This guide covers the steps required to configure VXLAN tunneling. The same approach can be used for any of the other tunneling protocols supported by Open vSwitch.

    +--------------+
    |     vm0      | 192.168.1.1/24
    +--------------+
       (vm_port0)
           |
           |
           |
    +--------------+
    |    br-int    |                                    192.168.1.2/24
    +--------------+                                   +--------------+
    |    vxlan0    |                                   |    vxlan0    |
    +--------------+                                   +--------------+
           |                                                  |
           |                                                  |
           |                                                  |
     172.168.1.1/24                                           |
    +--------------+                                          |
    |    br-phy    |                                   172.168.1.2/24
    +--------------+                                  +---------------+
    |  dpdk0/eth1  |----------------------------------|      eth1     |
    +--------------+                                  +---------------+
    Host A with OVS.                                     Remote host.

## Setup

This guide assumes the environment is configured as described below.

### Two Physical Hosts

The environment assumes the use of two hosts, named host1 and host2. We only detail the configuration of host1 but a similar configuration can be used for host2. Both hosts should be configured with Open vSwitch (with or without DPDK), QEMU/KVM and suitable VM images. Open vSwitch should be running before proceeding.

## Configuration Steps

Perform the following configuration on host1:

1. Create a `br-int` bridge:

       $ ovs-vsctl --may-exist add-br br-int \
         -- set Bridge br-int datapath_type=netdev \
         -- br-set-external-id br-int bridge-id br-int \
         -- set bridge br-int fail-mode=standalone

2. Add a port to this bridge. If using tap ports, first boot a VM and then add the port to the bridge:

       $ ovs-vsctl add-port br-int tap0

   If using DPDK vhost-user ports, add the port and then boot the VM accordingly, using `vm_port0` as the interface name:

       $ ovs-vsctl add-port br-int vm_port0 \
         -- set Interface vm_port0 type=dpdkvhostuserclient \
            options:vhost-server-path=/tmp/vm_port0

3. Configure the IP address of the VM interface in the VM itself:

       $ ip addr add 192.168.1.1/24 dev eth0
       $ ip link set eth0 up

4. On host1, add a port for the VXLAN tunnel:

       $ ovs-vsctl add-port br-int vxlan0 \
         -- set interface vxlan0 type=vxlan options:remote_ip=172.168.1.2

   Note

   `172.168.1.2` is the remote tunnel end point address. On the remote host this will be `172.168.1.1`

5. Create a `br-phy` bridge:

       $ ovs-vsctl --may-exist add-br br-phy \
           -- set Bridge br-phy datapath_type=netdev \
           -- br-set-external-id br-phy bridge-id br-phy \
           -- set bridge br-phy fail-mode=standalone \
                other_config:hwaddr=<mac address of eth1 interface>

   Note

   This additional bridge is required when running Open vSwitch in userspace rather than kernel-based Open vSwitch. The purpose of this bridge is to allow use of the kernel network stack for routing and ARP resolution. The datapath needs to look-up the routing table and ARP table to prepare the tunnel header and transmit data to the output port.

   Note

   `eth1` is used rather than `eth0`. This is to ensure network connectivity is retained.

6. Attach `eth1`/`dpdk0` to the `br-phy` bridge.

   If the physical port `eth1` is operating as a kernel network interface, run:

       $ ovs-vsctl --timeout 10 add-port br-phy eth1
       $ ip addr add 172.168.1.1/24 dev br-phy
       $ ip link set br-phy up
       $ ip addr flush dev eth1 2>/dev/null
       $ ip link set eth1 up
       $ iptables -F

   If instead the interface is a DPDK interface and bound to the `igb_uio` or `vfio` driver, run:

       $ ovs-vsctl --timeout 10 add-port br-phy dpdk0 \
         -- set Interface dpdk0 type=dpdk options:dpdk-devargs=0000:06:00.0
       $ ip addr add 172.168.1.1/24 dev br-phy
       $ ip link set br-phy up
       $ iptables -F

   The commands are different as DPDK interfaces are not managed by the kernel, thus, the port details are not visible to any `ip` commands.

   Important

   Attempting to use the kernel network commands for a DPDK interface will result in a loss of connectivity through `eth1`. Refer to Basic Configuration for more details.

Once complete, check the cached routes using ovs-appctl command:

    $ ovs-appctl ovs/route/show

If the tunnel route is missing, adding it now:

    $ ovs-appctl ovs/route/add 172.168.1.1/24 br-phy

Repeat these steps if necessary for host2, but using the below commands for the VM interface IP address:

    $ ip addr add 192.168.1.2/24 dev eth0
    $ ip link set eth0 up

And the below command for the the host2 VXLAN tunnel:

    $ ovs-vsctl add-port br-int vxlan0 \
      -- set interface vxlan0 type=vxlan options:remote_ip=172.168.1.1

## Testing

With this setup, ping to VXLAN target device (`192.168.1.2`) should work. Traffic will be VXLAN encapsulated and sent over the `eth1`/`dpdk0` interface.

## Tunneling-related Commands

### Tunnel routing table

To add route:

    $ ovs-appctl ovs/route/add <IP address>/<prefix length> <output-bridge-name> <gw>

To see all routes configured:

    $ ovs-appctl ovs/route/show

To delete route:

    $ ovs-appctl ovs/route/del <IP address>/<prefix length>

To look up and display the route for a destination:

    $ ovs-appctl ovs/route/lookup <IP address>

### ARP

To see arp cache content:

    $ ovs-appctl tnl/arp/show

To flush arp cache:

    $ ovs-appctl tnl/arp/flush

To set a specific arp entry:

    $ ovs-appctl tnl/arp/set <bridge> <IP address> <MAC address>

### Ports

To check tunnel ports listening in ovs-vswitchd:

    $ ovs-appctl tnl/ports/show

To set range for VxLan UDP source port:

    $ ovs-appctl tnl/egress_port_range <num1> <num2>

To show current range:

    $ ovs-appctl tnl/egress_port_range

### Datapath

To check datapath ports:

    $ ovs-appctl dpif/show

To check datapath flows:

    $ ovs-appctl dpif/dump-flows