After going through details about content libraries in previous three posts, here we are with yet another post on syncing contents of content library. Read earlier posts about content libraries here.
- Part-1: Content library concepts and creating local library
- Part-2: Subscribed library and Deployment models
- Part-3: Content library Permissions
We have already discussed about deployment models for content libraries in part-2 of this series, but that was purely from vCenter Server perspective. Here in this post we are going to discuss about synchronizing content library data from one datastore to another since libraries are stored across the datastore’s (Local site or remote site). Though it is possible to keep published library and subscribed library on same datastore, that would be a much unlikely scenario you will come across.
Content Library Architecture
Before we talk about how content sync is performed, let us have quick overview of content library architecture. Content Library has three main components that run on a vCenter server.
- Content Library Service
- Transfer Service and
- Database
The diagram below shows these three components and interaction between these components.
Content Library Service provides the control plane to manage storage and handle content operations (not actual transfer of data) such as upload, download, deploy and synchronize through UI. The Content library service coordinates with UI, Transfer service, database, and VPXD daemon for content operations.
Transfer service is an internal service in vSphere 7 integrated with content library service and not available for changes like privilege modification, start/stop service. Read these articles for info. Transfer service in 7 and Privileges transfer service.
Transfer service provides the data plane where actual data transfers is performed between datastore’s. These datastore are attached to ESXi hosts as VMFS datastore or NFS file share. The Transfer Service coordinates with content library service, Download/upload Client, and vpxd daemon while syncing content.
Database is used to store metadata associated with content such as type of content, date of creation, vendor, etc.
Library synchronization
Once the content libraries (Local and subscribed) are created, content of local library can be synced with subscribed library. Synchronization process basically clones a published local library by downloading all the content to a subscribed library at first sync. For later sync operations, only the content that is new to subscribed library which includes new content as well as modified content will be synced as versioning is supported in content libraries.
However, how the sync is performed, that depends on which replication option you selected while creating the subscribed library.
Synchronization of a subscribed library with option download all the contents of the published library immediately, synchronizes item metadata as well as the content of item. Published library items that are new to the subscribed library are fully downloaded to the datastore of the subscribed library.
Synchronization of a subscribed library with option download library content only when needed, synchronizes only the metadata of the items from the published library, not the contents of the items. This saves storage space on target datastore. This is also referred as on demand download. If content of library item is needed, manually synchronize that item. Once used, content of item can be deleted to free up the space on subscribed library datastore.
Below are defaults for sync frequency and optimization operations in vSphere 7. Apart from default sync, we can also initiate manual sync if required. In fact, we did it earlier, when we created subscribed library in part-2.
How data is transferred
The Transfer Service transfers data from published library to subscribed library in two ways, namely streaming mode or direct copy mode. Below diagram shows how the two modes operate in a data transfer between datastore’s.
- If the source and destination ESXi hosts have direct connectivity between them, the Transfer Service through vCenter server instructs the source host to directly copy the content to target host. This is also referred as Network file copy or NFC.
- On the other hand, if the source and destination ESXi hosts are managed by different vCenter server, in this case streaming mode is used which uses standard HTTPS (443) for transfer through vCenter server.
- In streaming mode the data flows through the Transfer Service running on vCenter Server. So in comparison to NFC, Streaming mode will have less performance due to added hops.
- When multiple vCenter Server instances are configured in an Enhanced Linked Mode environment, the contents of the published library can be synchronized to a subscribed library by directly copying the files from the source datastore to the destination datastore. Read more about this here.
- When Enhanced Linked Mode is not available, in this case the contents of a published library must be streamed through the Transfer Service component. There are three scenarios that are possible, based on the storage configuration:
- Both published and subscribed libraries reside on a datastore. Check image 1 below.
- Both published and subscribed libraries reside on a NFS share. Check image 2 below.
- The published library resides on NFS share, whereas the subscribed library resides on a datastore. Check image 3 below.
Image 1
Image 2
Image 3
There are multiple possible designs in terms of datastore placement of content libraries in which a subscribed content library can sync contents (data transfer) from published local library. Below are the examples of the library placements for content libraries along with their sync path. Read about them here. It’s an article for previous versions of vSphere but still applicable.
With single vCenter server deployment model
With multi vCenter Server Deployment Model
Versioning
Well, I am not discussing template versioning here which is a cool new feature in vSphere 7.
We are going to discuss about content versioning here which basically means modified contents in library and its versions. Similar concept we use in incremental backups for modified data.
When an item is changed in a local content library, the item number is incremented to mark the modification. The version number for the content library as a container entity is also incremented. This process is called as simple versioning. The Content Library Service determines whether synchronization should occur by using simple versioning. Simple versioning does not store multiple versions of content or offer rollback functionality, it is simply a numerical value that is assigned to the content library and its content.
How versioning works?
- The version number of published content library and subscribed content library is checked first.
- If version number matches of both published and subscribed libraries, no synchronization is required.
- If not matched, the version number of each item in the library is checked.
- If discrepancies are found, only items with discrepancies are synchronized.
Thats it for this post. And also for this series on Content libraries in vSphere 7. I hope it was insightful.
For new features like template versioning and using templates with checkin and checkout feature, do check VMware blog