Chapter 8: SR-IOV Setup

Home Prev	cOS Core Getting Started Guide for KVM	Next

Overview

Single Root I/O Virtualization (SR-IOV) is a specification that can allow direct access to an external PCI Ethernet interface by cOS Core running under KVM. It is only available on Intel based hardware.

The direct access provided by SR-IOV can give dramatically higher traffic throughput capability for a virtual firewall since it circumvents the overhead involved with normal virtual interfaces. A disadvantage of using SR-IOV is the static nature of configurations that use it.

	Important: SR-IOV consumes an entire core
	When SR-IOV is enabled, cOS Core will consume virtually all the resources of the processor core on which it runs. This is true even if cOS Core has no traffic load. The reason for this is that SR-IOV uses continuous interface polling to check for new traffic.

SR-IOV Interfaces for cOS Core

By default, cOS Core provides three virtual Ethernet ports with the logical cOS Core names I1, I2 and I3. The setup procedure described in this section adds hardware PCI Ethernet ports as additional interfaces with logical cOS Core names I4, I5 and so on.

Once the setup is complete, only traffic routed through these additional ports will benefit from the throughput increases provided by SR-IOV.

Prerequisites for SR-IOV

In order to make use of SR-IOV with cOS Core under KVM, the following is required:

Support for IOMMU with IOMMU enabled in the BIOS.
Hardware support for SR-IOV with Intel™ VT-d or AMD-Vi.
Support for SR-IOV enabled in the BIOS.
Available slots supporting PCI Express v2.0 (5.0GT/s) x8 Lanes with ARI and ACS.
An Intel Ethernet Converged Network Adapter x520/X540 with Intel 82599 chipsets (10 Gb) or an Intel i350 adapter (1 Gb).

Set up of the hardware platform for virtualization is not discussed further here. For details on this subject refer to the Intel document entitled: Using Intel Ethernet and the PCI-SIG Single Root I/O Virtualization (SR-IOV) and Sharing Specification on Red Hat Enterprise Linux .

Adding SR-IOV Interfaces

The following are the steps for SR-IOV interface setup with cOS Core:

If it is running, stop the cOS Core virtual machine.

In virt-manager, select Add Hardware.

Select PCI Host Device and the correct virtual function. For the first PCI device, the final digit in the numeric designation on the left should be even (below it is 1D:10:0). Press Finish to add the device.

The same is repeated if a second PCI device is added but the final digit of the selected virtual function should be odd.

The new adapters are now listed in virt-manager.

Start the cOS Core virtual machine.

Start cOS Core again and issue the following console CLI command:
```
Device:/> pciscan -cfgupdate
```
cOS Core will scan the available interfaces and include the added PCI interfaces into the configuration. Some example output is shown below.

Finally, save the configuration changes using the following commands:
```
Device:/> activate
			
Device:/> commit
```

	Note: Do not pre-assign the SR-IOV MAC address
	For any usage of SR-IOV interfaces with cOS Core, the MAC address should not be preassigned by the hypervisor so that it is fixed. This will prevent cOS Core from controlling the MAC address which can be needed in certain circumstances.

Achieving Maximum Throughput

Once the SR-IOV interfaces exist as logical interfaces in cOS Core they can used for both receiving and sending in traffic as well as being part of rule sets and other cOS Core objects.

On order to reach much higher throughput speeds, traffic must both enter and leave the firewall via SR-IOV interfaces. Having the traffic enter or leave on a normal interface will create a bottleneck, reducing throughput back to non-SR-IOV speeds.

Features Not Supported by SR-IOV Interfaces

The following cOS Core features are not supported by SR-IOV interfaces:

Proxy ARP/ND using XPUBLISH is not supported.
Multicast is not supported.