Persistent Memory (PMEM) aka Non-Volatile Memory (NVM) is a type of non-volatile DRAM (NVDIMM) that has the speed of DRAM and it retains the contents through power cycles. Persistent memory (PMEM) is a solid-state high-performance byte-addressable memory device that resides on the memory bus.
PMem provides new storage tier that can be used by enterprise applications such as HPC that require reduced latency and high performance.
vSphere support for Persistent Memory (PMEM) NVDIMM
With the release of vSphere 6.7, NVDIMMs are supported.
When NVDIMM modules are installed in supported hardware and with vSphere 6.7 or later, a PMEM datastore is automatically created on the host.
PMEM is treated as a local datastore. So features like vSphere HA do not work with PMEM as yet.
PMem name spaces
PMEM name spaces are configured before ESXi starts. Name spaces are similar to the disks on the system. ESXi reads and combines multiple name spaces into one logical volume.
PMEM is formatted automatically by default, if not done previously. If it has already been formatted, ESXi attempts to mount the PMEM. If the data in PMEM is corrupted it may cause ESXi to fail. To avoid this, ESXi checks for errors in the metadata during PMEM mount time.
NVDIMM and traditional storage
When traditional storage devices are used, software exists between applications and storage devices which causes delay in processing.
When PMEM is used, the workloads use the storage directly. PMEM being on the memory bus allows it to have DRAM-like access to data, so performance is far better with PMem than traditional storage.
PMEM also being cacheable, allows CPU to access the data as if it was in RAM. This significantly reduces the storage latency and delivers average latency of less than 0.5 microseconds.
PMEM is byte addressable and persistent, which means that it can be used as storage. VMs can be configured to use PMEM on a standalone host, or in a cluster.
Once the VM is powered on initially, PMEM is reserved for it regardless of whether it is in powered on state or in powered off state later. PMEM allocated to VM is reserved until the VM is migrated or removed from the ESXi host.
Persistent memory can be consumed by virtual machines in two different modes.
- Virtual Persistent Memory (vPMEM)
- Virtual Persistent Memory Disk (vPMEMDisk)
Virtual Persistent Memory
Using vPMEM, the memory is exposed to a guest OS as a virtual NVDIMM. This enables the guest OS to use PMEM in byte addressable random mode. VM must have hardware version 14 at least and also the guest OS that supports NVM technology. e.g. Windows Server 2016.
Virtual Persistent Memory Disk
vPMEMDisk is supported by all hardware versions. Using vPMEMDisk, memory can be accessed by the guest OS as a virtual SCSI device. Legacy guest OS can take advantage of virtual persistent memory disk feature. However, the virtual disk is stored in a PMEM datastore in this mode.
When VM is created with PMEM, memory is reserved for it at the time of hard disk creation.
The consumption of PMEM includes both powered on and powered off VMs. If a VM is configured to use PMEM and DRS is not configured on the cluster, then manual placement needs to be done on a host that has sufficient persistent memory.
VM Migrations
Since PMEM is treated as local datastore, to migrate a VM, storage vMotion is required. A VM with vPMEM can only be migrated to an ESXi host that has PMEM resource. A VM with vPMEMDisk can be migrated to an ESXi host without a PMEM resource as well.
Error handling and NVDimm management
Host failures can result in a loss of availability of data. In worst case scenario, all data may be lost and manual steps must be taken to reformat the PMEM.