Remote Direct Memory Access for Virtual Machines in vSphere 6.5

InfiniBand (IB) is a new generation network protocol which supports RDMA natively. This is a new network technology, hence it requires NICs and switches that support this technology.

RoCE is a network protocol which allows performing RDMA over Ethernet network. Its lower network headers are Ethernet headers and its upper network headers (including the data) are InfiniBand headers. This allows using RDMA over standard Ethernet infrastructure.

With vSphere 6.5, VMware introduced RDMA over Converged Ethernet (RoCE). RDMA over Converged Ethernet (RoCE) allows remote direct memory access (RDMA) over an Ethernet network. RoCE dramatically accelerates communication between two network endpoints leveraging DMA over converged Ethernet infrastructure.

RoCE is supported in two modes RoCE v1 and RoCE v2.

The RoCE v1 protocol is an Ethernet link layer protocol with Ethertype 0x8915, it means that the frame length limits of the Ethernet protocol apply – 1500 bytes for a regular Ethernet frame and 9000 bytes for a jumbo frame.

The RoCEv2 protocol exists on top of either the UDP/IPv4 or the UDP/IPv6 protocol.[2] The UDP destination port number 4791 has been reserved for RoCE v2. Since RoCEv2 packets are routable the RoCE v2 protocol is sometimes called Routable RoCE or RRoCE.

Following diagram shows the packet format for RoCEv1 and v2 protocols.

Using RDMA in vSphere:

vSphere 6.5 and later releases support remote direct memory access (RDMA) communication between virtual machines that have para-virtualized RDMA (PVRDMA) network adapters. The virtual machines must be connected to the same vSphere Distributed Switch.

A PVRDMA network adapter for virtual machine provides remote direct memory access in a virtual environment. Virtual machine uses PVRDMA network adapter to communicate with other virtual machines that has the PVRDMA devices. The transfer of memory is offloaded to the RDMA-capable Host Channel Adapters (HCA) as shown in images below in order

The PVRDMA device automatically selects the method of communication between the virtual machines. For virtual machines that run on the same ESXi host with or without a physical RDMA device, the data transfer is a memcpy between the two virtual machines. The physical RDMA hardware is not used in this case.

For virtual machines that reside on different ESXi hosts and that have a physical RDMA connection, the physical RDMA devices must be uplinks on the distributed switch. In this case, the communication between the virtual machines occurs using PVRDMA adapter which uses the underlying physical RDMA devices.

For two virtual machines that run on different ESXi hosts, when at least one of the hosts does not have a physical RDMA device, the communication falls back to a TCP-based channel and the performance is reduced.

PVRDMA Requirements in vSphere 6.5:

vSphere:

• ESXi host 6.5 or later.
• vCenter Server or vCenter Server Appliance 6.5 or later.
• vSphere Distributed Switch.

Host Channel Adapter (HCA): 

• Virtual machines that reside on different ESXi hosts require HCA to use RDMA.
• HCA must be assigned as an uplink for the vSphere Distributed Switch.
• PVRDMA does not support NIC teaming.
• The HCA must be the only uplink on the vSphere Distributed Switch.
• For virtual machines on the same ESXi hosts or virtual machines using the TCP-based fallback, the HCA is not required (This point will be discussed later in this article).

Virtual machine:

• Virtual hardware verison 13 or later.

Guest OS:

• Linux (64-bit)

Assign a PVRDMA Adapter to a Virtual Machine:

To enable a virtual machine to exchange data using RDMA, you must associate the virtual machine with a PVRDMA network adapter.

Procedure:

• Locate the virtual machine in the vSphere Web Client inventory and Power off the virtual machine.
• On the Configure tab of the virtual machine, expand Settings and select VM Hardware.
• Click Edit and select the Virtual Hardware tab in the dialog box displaying the settings.
• From the New device drop-down menu, select Network and click Add.
• Expand the New Network section and connect the virtual machine to a distributed port group.
• From the Adapter type drop-down menu, select PVRDMA.
• Expand the Memory section, select Reserve all guest memory (All locked), and click OK .
• Power on the virtual machine.

Hope this helps understand the vRDMA in VMware vSphere 6.5.