VMware vSphere 7.x Memory Reclamation-Part 4: Ballooning

After TPS in previous post, we will explore Ballooning reclamation technique in this post. Basically I am just following the order in which these techniques are initialized. Ballooning is a process where the hypervisor reclaims memory back from the virtual machine. Also if ballooning is started on an ESXi host, it indicates the beginning of memory contention on the ESXi host.

As we discussed in previous post, TPS run in ESXi by default (intra-VM) for small memory pages. It is true even when ESXi host more memory free on the host beyond High memory state. Hence, TPS is not considered as indication of memory contention.

Ballooning gets initiated when the ESXi host has started running out of physical memory and is in the Soft ESXi memory state and demand of the virtual machine is too high for the ESXi host to handle.

Ballooning is the reclamation techniques that enables an ESXi host to reclaim the memory that is consumed by virtual machines.

Why take memory back that is already consumed by a VM?

Refer Scenario 1 in the Part1: Basics in this series to understand the need for reclaiming consumed memory from VM.

The reason for using ballooning is that the ESXi host does not have the visibility inside VM and is not aware of idle memory pages, active memory pages, and Free list memory pages.

Ballooning technique helps to overcome this limitation and allows an ESXi host to reclaim memory from virtual machine. Balloon driver (VMMEMCTL) is loaded into the guest operating system when we install VMware tools.

How Ballooning works?

In Figure (A), four guest physical pages are mapped in the host physical memory. Two of the pages are used by the guest application and the other two pages (marked by stars) are in the guest operating system free list. Note that since the hypervisor cannot identify the two pages in the guest free list, it cannot reclaim the host physical pages that are backing them. Assuming the hypervisor needs to reclaim two pages from the virtual machine, it will set the target balloon size to two pages.

After obtaining the target balloon size, the balloon driver allocates two guest physical pages inside the virtual machine and pins them, as shown in Figure (B). Here, “pinning” is achieved through the guest operating system interface, which ensures that the pinned pages cannot be paged out to disk under any circumstances.

Once the memory is allocated, the balloon driver notifies the hypervisor the page numbers of the pinned guest physical memory so that the hypervisor can reclaim the host physical pages that are backing them. In Figure (B), RED and GREEN are representing these pages.

The hypervisor can safely reclaim this host physical memory pages because neither the balloon driver nor the guest operating system relies on the contents of these pages.

Above explanation takes only free list pages into consideration. In order to understand this more let’s discuss this same process further with another example scenario here.

If you look at above scenario, ESXi host is overcommitted and there are 4 VMs on it each with 4 GB allocation, in total 16 GB allocation. VM1, VM2 and VM3 are already running with 3 GB memory consumption each, so approx. 9 GB is already consumed on given ESXi host. Now we are trying to Power On VM4 which has 2 GB reservation. However, server has only 1 GB memory free (Hypothetically just for understanding) which is not sufficient to Power On VM4. Also note that I have not considered TPS memory savings here. Assuming all this,

So how does ESXi responds to this sort of situation?

• When ESXi host is under memory contention, ESXi host sets the target for balloon driver to free the memory across all VMs. In above example it will be roughly 1 GB.
• As per the target, balloon driver inside virtual machine, will fake itself as another application and demand memory from Operating system of virtual machine.
• Considering the request from an application (Fake), VM operating system will start allocating memory pages to balloon driver from Free list first, then LRU and if required from MRU as well in case there is situation to satisfy reservation demand as above.
• As soon as balloon driver receives memory pages from operating system of VM, it starts inflating from its size just like what happens with actual balloon when we pump air into it.
• Memory pages that are consumed by balloon driver, are marked and pinned (Red and Green pages in first image with pin icon) so that they are not swapped out.
• Balloon driver communicates with the hypervisor through a private channel and informs hypervisor about pinned pages.
• Hypervisor then reclaims these pages by setting up lower target, this causes balloon driver to deflate back to initial state, just like in actual balloon, if we take air out of it, it comes back to initial state.
• Below image describes this process graphically.

ESXi host will try to reclaim memory from virtual machines as per the target received. How much memory is reclaimed from each VM is calculated with the help of Memory Taxing (mem.idletax).

If a virtual machine is not actively using all of its currently allocated memory, ESXi charges more for idle memory than for memory that is in use. This is done to help prevent virtual machines from hoarding idle memory.

You can modify the idle memory tax rate with the Mem.IdleTax option. Use this option, together with the Mem.SamplePeriod advanced attribute, to control how the system determines target memory allocations for virtual machines.

If necessary, you can limit the amount of memory vmmemctl reclaims by setting the sched.mem.maxmemctl parameter for a specific virtual machine. This option specifies the maximum amount of memory that can be reclaimed from a virtual machine in megabytes (MB). 

Wrapping up:

As mentioned earlier, Ballooning indicates beginning of memory contention on an ESXi host. If only ballooning is happening on the host, it can be treated as warning. However, along with ballooning on the host, if guest level swapping is also happening simultaneously, it needs to treated as critical situation, This indicates the VM performance may be degraded.

Inside VM:

The guest OS determines whether it needs to page out guest physical memory to satisfy the balloon driver’s allocation requests. If the VM has plenty of free or idle guest physical memory, inflating the balloon does not cause guest OS-level paging and does not affect guest performance. However, if the virtual machine is already under memory pressure, the guest OS decides which guest physical pages are to be paged to satisfy the balloon driver’s allocation requests.

You must configure the guest operating system with sufficient swap space. Ballooning technique increases or decreases memory pressure on the guest OS, causing the guest to use its own native memory management algorithms. When memory is low, the guest OS determines which pages to reclaim and, if necessary, swaps them to its own swap file such as to pagefile.sys in Windows.

That is all for this post. See you in the next one on Memory Compression.