Scenario - Using Open vSwitch w/ ASAP ² (Direct Mode)¶

Overview¶

With appropriate hardware, operators can choose to utilize ASAP ²-accelerated Open vSwitch instead of unaccelerated Open vSwitch for the Neutron virtual network infrastructure. ASAP ² technology offloads packet processing onto hardware built into the NIC rather than using the CPU of the host. It requires careful consideration and planning before implementing. This document outlines how to set it up in your environment.

Recommended reading¶

This guide is a variation of the standard Open vSwitch and SR-IOV deployment guides available at:

• https://docs.openstack.org/openstack-ansible-os_neutron/latest/app-openvswitch.html
• https://docs.openstack.org/openstack-ansible-os_neutron/latest/configure-network-services.html#sr-iov-support-optional

The following resources may also be helpful:

• https://docs.openstack.org/neutron/latest/admin/config-sriov.html
• https://docs.openstack.org/neutron/latest/admin/config-ovs-offload.html
• https://www.mellanox.com/related-docs/prod_software/ASAP2_Hardware_Offloading_for_vSwitches_User_Manual_v4.4.pdf

Prerequisites¶

To enable SR-IOV and PCI passthrough capabilities on a Linux platform, ensure that VT-d/VT-x are enabled for Intel processors and AMD-V/AMD-Vi are enabled for AMD processors. Such features are typically enabled in the BIOS.

On an Intel platform, the following kernel parameters are required and can be added to the GRUB configuration:

GRUB_CMDLINE_LINUX="... iommu=pt intel_iommu=on"

On an AMD platform, use these parameters instead:

GRUB_CMDLINE_LINUX="... iommu=pt amd_iommu=on"

Update GRUB and reboot the host(s).

SR-IOV provides virtual functions (VFs) that can be presented to instances as network interfaces and are used in lieu of tuntap interfaces. Configuration of VFs is outside the scope of this guide. The following links may be helpful:

• https://community.mellanox.com/s/article/getting-started-with-mellanox-asap-2
• https://community.mellanox.com/s/article/howto-configure-sr-iov-for-connectx-4-connectx-5-with-kvm–ethernet-x

Deployment¶

Configure your networking according the Open vSwitch implementation docs:

• Scenario - Using Open vSwitch

An example provider network configuration has been provided below:

- network:
    container_bridge: "br-provider"
    container_type: "veth"
    type: "vlan"
    range: "700:709"
    net_name: "physnet1"
    network_interface: "ens4f0"
    group_binds:
      - neutron_openvswitch_agent

Add a nova_pci_passthrough_whitelist entry to user_variables.yml, where devname is the name of the interface connected to the provider bridge and physical_network is the name of the provider network.

nova_pci_passthrough_whitelist: '{"devname":"ens4f0","physical_network":"physnet1"}'

To enable the openvswitch firewall driver rather than the default iptables_hybrid firewall driver, add the following overrides to user_variables.yml:

neutron_ml2_conf_ini_overrides:
  securitygroup:
    firewall_driver: openvswitch
neutron_openvswitch_agent_ini_overrides:
  securitygroup:
    firewall_driver: openvswitch

Once the OpenStack cluster is configured, start the OpenStack deployment as listed in the OpenStack-Ansible Install guide by running all playbooks in sequence on the deployment host.

Post-Deployment¶

Once the deployment is complete, create the VFs that will be used for SR-IOV. In this example, the physical function (PF) is ens4f0. It will simultaneously be connected to the Neutron provider bridge br-provider.

1. On each compute node, determine the maximum number of VFs a PF can support:

# cat /sys/class/net/ens4f0/device/sriov_totalvfs

2. On each compute node, create the VFs:

# echo '8' > /sys/class/net/ens4f0/device/sriov_numvfs

# for vf in `ls -ld /sys/class/net/ens4f0/device/virt* | cut -f 11 -d ' ' | cut -b 4-`
  do
    echo $vf > /sys/bus/pci/drivers/mlx5_core/unbind
  done

# PCI_ADDR=`grep PCI_SLOT_NAME /sys/class/net/ens4f0/device/uevent | sed 's:.*PCI_SLOT_NAME=::'`
# devlink dev eswitch set pci/$PCI_ADDR mode switchdev

# ethtool -K ens4f0 hw-tc-offload on

# for vf in `ls -ld /sys/class/net/ens4f0/device/virt* | cut -f 11 -d ' ' | cut -b 4-`
  do
    echo $vf > /sys/bus/pci/drivers/mlx5_core/bind
  done

# ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
# ovs-vsctl set Open_vSwitch . other_config:max-idle=30000

# systemctl restart openvswitch-switch

# systemctl restart neutron-openvswitch-agent

# systemctl restart nova-compute

Verify operation¶

To verify operation of hardware-offloaded Open vSwitch, you must create a virtual machine instance using an image with the proper network drivers.

The following images are known to contain working drivers:

• Fedora 24
• Ubuntu 18.04 LTS (Bionic)
• Centos 7 (1901)

Before creating an instance, a Neutron port must be created that has the following characteristics:

--vnic-type direct --binding-profile '{"capabilities": ["switchdev"]}'

To ensure flows are offloaded, disable port security with the --disable-port-security argument.

An example of the full command can be seen here:

# openstack port create \
  --network <network> \
  --vnic-type direct --binding-profile '{"capabilities": ["switchdev"]}' \
  --disable-port-security \
  <name>

The port can then be attached to the instance at boot. Once booted, the port will be updated to reflect the PCI address of the corresponding virtual function:

root@aio1-utility-container-8c0b0916:~# openstack port show -c binding_profile testport2
+-----------------+------------------------------------------------------------------------------------------------------------------+
| Field           | Value                                                                                                            |
+-----------------+------------------------------------------------------------------------------------------------------------------+
| binding_profile | capabilities='[u'switchdev']', pci_slot='0000:21:00.6', pci_vendor_info='15b3:1016', physical_network='physnet1' |
+-----------------+------------------------------------------------------------------------------------------------------------------+

root@compute1:~# tcpdump -nnn -i eth1 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes

root@infra2:~# ping 192.168.88.151 -c5
PING 192.168.88.151 (192.168.88.151) 56(84) bytes of data.
64 bytes from 192.168.88.151: icmp_seq=1 ttl=64 time=48.3 ms
64 bytes from 192.168.88.151: icmp_seq=2 ttl=64 time=1.52 ms
64 bytes from 192.168.88.151: icmp_seq=3 ttl=64 time=0.586 ms
64 bytes from 192.168.88.151: icmp_seq=4 ttl=64 time=0.688 ms
64 bytes from 192.168.88.151: icmp_seq=5 ttl=64 time=0.775 ms

--- 192.168.88.151 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4045ms
rtt min/avg/max/mdev = 0.586/10.381/48.335/18.979 ms

root@compute1:~# tcpdump -nnn -i eth1 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
19:51:09.684957 IP 192.168.88.254 > 192.168.88.151: ICMP echo request, id 11168, seq 1, length 64
19:51:09.685448 IP 192.168.88.151 > 192.168.88.254: ICMP echo reply, id 11168, seq 1, length 64