Open vSwitch L2 Agent¶

Rationale¶

At the time the first design for the OVS agent came up, trunking in OpenStack was merely a pipe dream. Since then, lots has happened in the OpenStack platform, and many many deployments have gone into production since early 2012.

In order to address the vlan-aware-vms use case on top of Open vSwitch, the following aspects must be taken into account:

• Design complexity: starting afresh is always an option, but a complete rearchitecture is only desirable under some circumstances. After all, customers want solutions…yesterday. It is noteworthy that the OVS agent design is already relatively complex, as it accommodates a number of deployment options, especially in relation to security rules and/or acceleration.
• Upgrade complexity: being able to retrofit the existing design means that an existing deployment does not need to go through a forklift upgrade in order to expose new functionality; alternatively, the desire of avoiding a migration requires a more complex solution that is able to support multiple modes of operations;
• Design reusability: ideally, a proposed design can easily apply to the various technology backends that the Neutron L2 agent supports: Open vSwitch and Linux Bridge.
• Performance penalty: no solution is appealing enough if it is unable to satisfy the stringent requirement of high packet throughput, at least in the long term.
• Feature compatibility: VLAN transparency is for better or for worse intertwined with vlan awareness. The former is about making the platform not interfere with the tag associated to the packets sent by the VM, and let the underlay figure out where the packet needs to be sent out; the latter is about making the platform use the vlan tag associated to packet to determine where the packet needs to go. Ideally, a design choice to satisfy the awareness use case will not have a negative impact for solving the transparency use case. Having said that, the two features are still meant to be mutually exclusive in their application, and plugging subports into networks whose vlan-transparency flag is set to True might have unexpected results. In fact, it would be impossible from the platform’s point of view discerning which tagged packets are meant to be treated ‘transparently’ and which ones are meant to be used for demultiplexing (in order to reach the right destination). The outcome might only be predictable if two layers of vlan tags are stacked up together, making guest support even more crucial for the combined use case.

It is clear by now that an acceptable solution must be assessed with these issues in mind. The potential solutions worth enumerating are:

• VLAN interfaces: in layman’s terms, these interfaces allow to demux the traffic before it hits the integration bridge where the traffic will get isolated and sent off to the right destination. This solution is proven to work for both iptables-based and native ovs security rules (credit to Rawlin Peters). This solution has the following design implications:
  • Design complexity: this requires relative small changes to the existing OVS design, and it can work with both iptables and native ovs security rules.
  • Upgrade complexity: in order to employ this solution no major upgrade is necessary and thus no potential dataplane disruption is involved.
  • Design reusability: VLAN interfaces can easily be employed for both Open vSwitch and Linux Bridge.
  • Performance penalty: using VLAN interfaces means that the kernel must be involved. For Open vSwitch, being able to use a fast path like DPDK would be an unresolved issue (Kernel NIC interfaces are not on the roadmap for distros and OVS, and most likely will never be). Even in the absence of an extra bridge, i.e. when using native ovs firewall, and with the advent of userspace connection tracking that would allow the stateful firewall driver to work with DPDK, the performance gap between a pure userspace DPDK capable solution and a kernel based solution will be substantial, at least under certain traffic conditions.
  • Feature compatibility: in order to keep the design simple once VLAN interfaces are adopted, and yet enable VLAN transparency, Open vSwitch needs to support QinQ, which is currently lacking as of 2.5 and with no ongoing plan for integration.
• Going full openflow: in layman’s terms, this means programming the dataplane using OpenFlow in order to provide tenant isolation, and packet processing. This solution has the following design implications:
  • Design complexity: this requires a big rearchitecture of the current Neutron L2 agent solution.
  • Upgrade complexity: existing deployments will be unable to work correctly unless one of the actions take place: a) the agent can handle both the ‘old’ and ‘new’ way of wiring the data path; b) a dataplane migration is forced during a release upgrade and thus it may cause (potentially unrecoverable) dataplane disruption.
  • Design reusability: a solution for Linux Bridge will still be required to avoid widening the gap between Open vSwitch (e.g. OVS has DVR but LB does not).
  • Performance penalty: using Open Flow will allow to leverage the user space and fast processing given by DPDK, but at a considerable engineering cost nonetheless. Security rules will have to be provided by a learn based firewall to fully exploit the capabilities of DPDK, at least until user space connection tracking becomes available in OVS.
  • Feature compatibility: with the adoption of Open Flow, tenant isolation will no longer be provided by means of local vlan provisioning, thus making the requirement of QinQ support no longer strictly necessary for Open vSwitch.
• Per trunk port OVS bridge: in layman’s terms, this is similar to the first option, in that an extra layer of mux/demux is introduced between the VM and the integration bridge (br-int) but instead of using vlan interfaces, a combination of a new per port OVS bridge and patch ports to wire this new bridge with br-int will be used. This solution has the following design implications:
  • Design complexity: the complexity of this solution can be considered in between the above mentioned options in that some work is already available since Mitaka and the data path wiring logic can be partially reused.
  • Upgrade complexity: if two separate code paths are assumed to be maintained in the OVS agent to handle regular ports and ports participating a trunk with no ability to convert from one to the other (and vice versa), no migration is required. This is done at a cost of some loss of flexibility and maintenance complexity.
  • Design reusability: a solution to support vlan trunking for the Linux Bridge mech driver will still be required to avoid widening the gap with Open vSwitch (e.g. OVS has DVR but LB does not).
  • Performance penalty: from a performance standpoint, the adoption of a trunk bridge relieves the agent from employing kernel interfaces, thus unlocking the full potential of fast packet processing. That said, this is only doable in combination with a native ovs firewall. At the time of writing the only DPDK enabled firewall driver is the learn based one available in the networking-ovs-dpdk repo;
  • Feature compatibility: the existing local provisioning logic will not be affected by the introduction of a trunk bridge, therefore use cases where VMs are connected to a vlan transparent network via a regular port will still require QinQ support from OVS.

To summarize:

• VLAN interfaces (A) are compelling because will lead to a relatively contained engineering cost at the expense of performance. The Open vSwitch community will need to be involved in order to deliver vlan transparency. Irrespective of whether this strategy is chosen for Open vSwitch or not, this is still the only viable approach for Linux Bridge and thus pursued to address Linux Bridge support for VLAN trunking. To some extent, this option can also be considered a fallback strategy for OVS deployments that are unable to adopt DPDK.
• Open Flow (B) is compelling because it will allow Neutron to unlock the full potential of Open vSwitch, at the expense of development and operations effort. The development is confined within the boundaries of the Neutron community in order to address vlan awareness and transparency (as two distinct use cases, ie. to be adopted separately). Stateful firewall (based on ovs conntrack) limits the adoption for DPDK at the time of writing, but a learn-based firewall can be a suitable alternative. Obviously this solution is not compliant with iptables firewall.
• Trunk Bridges (C) tries to bring the best of option A and B together as far as OVS development and performance are concerned, but it comes at the expense of maintenance complexity and loss of flexibility. A Linux Bridge solution would still be required and, QinQ support will still be needed to address vlan transparency.

All things considered, as far as OVS is concerned, option (C) is the most promising in the medium term. Management of trunks and ports within trunks will have to be managed differently and, to start with, it is sensible to restrict the ability to update ports (i.e. convert) once they are bound to a particular bridge (integration vs trunk). Security rules via iptables rules is obviously not supported, and never will be.

Option (A) for OVS could be pursued in conjunction with Linux Bridge support, if the effort is seen particularly low hanging fruit. However, a working solution based on this option positions the OVS agent as a sub-optminal platform for performance sensitive applications in comparison to other accelerated or SDN-controller based solutions. Since further data plane performance improvement is hindered by the extra use of kernel resources, this option is not at all appealing in the long term.

Embracing option (B) in the long run may be complicated by the adoption of option (C). The development and maintenance complexity involved in Option (C) and (B) respectively poses the existential question as to whether investing in the agent-based architecture is an effective strategy, especially if the end result would look a lot like other maturing alternatives.

Implementation VLAN Interfaces (Option A)¶

This implementation doesn’t require any modification of the vif-drivers since Nova will plug the vif of the VM the same way as it does for traditional ports.

+----------------------------+
|             VM             |
|   eth0            eth0.100 |
+-----+-----------------+----+
      |
      |
  +---+---+       +-----+-----+
  | tap1  |-------|  tap1.100 |
  +---+---+       +-----+-----+
      |                 |
      |                 |
  +---+---+         +---+---+
  | qbr1  |         | qbr2  |
  +---+---+         +---+---+
      |                 |
      |                 |
+-----+-----------------+----+
|    port 1          port 2  |
|   (tag 3)         (tag 5)  |
|           br-int           |
+----------------------------+

+----------------------------+
|             VM             |
|   eth0            eth0.100 |
+-----+-----------------+----+
      |
      |
  +---+---+       +-----+-----+
  | tap1  |-------|  tap1.100 |
  +---+---+       +-----+-----+
      |                 |
      |                 |
      |                 |
+-----+-----------------+----+
|    port 1          port 2  |
|   (tag 3)         (tag 5)  |
|           br-int           |
+----------------------------+

Implementation Trunk Bridge (Option C)¶

This implementation is based on this etherpad. Credits to Bence Romsics. The option use_veth_interconnection=true won’t be supported, it will probably be deprecated soon, see [1]. The IDs used for bridge and port names are truncated.

+--------------------------------+
|             VM                 |
|   eth0               eth0.100  |
+-----+--------------------+-----+
      |
      |
+-----+--------------------------+
|    tap1                        |
|          tbr-trunk-id          |
|                                |
| tpt-parent-id   spt-subport-id |
|                   (tag 100)    |
+-----+-----------------+--------+
      |                 |
      |                 |
      |                 |
+-----+-----------------+---------+
| tpi-parent-id    spi-subport-id |
|  (tag 3)           (tag 5)      |
|                                 |
|           br-int                |
+---------------------------------+

tpt-parent-id: trunk bridge side of the patch port that implements a trunk. tpi-parent-id: int bridge side of the patch port that implements a trunk. spt-subport-id: trunk bridge side of the patch port that implements a subport. spi-subport-id: int bridge side of the patch port that implements a subport.

[1] https://bugs.launchpad.net/neutron/+bug/1587296

ovs-vsctl add-port tbr-trunk-id tpt-parent-id -- set Interface tpt-parent-id type=patch options:peer=tpi-parent-id
ovs-vsctl add-port br-int tpi-parent-id tag=3 -- set Interface tpi-parent-id type=patch options:peer=tpt-parent-id

ovs-vsctl add-port tbr-trunk-id spt-subport-id tag=100 -- set Interface spt-subport-id type=patch options:peer=spi-subport-id
ovs-vsctl add-port br-int spi-subport-id tag=5 -- set Interface spi-subport-id type=patch options:peer=spt-subport-id

Agent resync¶

During resync the agent should check that all the trunk and subports are still valid. It will delete the stale trunk and subports using the procedure specified in the previous paragraphs according to the implementation.