You may see some incomprehensible messages in the log of Spectrum Scale like "[E} AFM: File system fs1 fileset afm encountered an error synchronizing with the remote cluster. Cannot synchronize with the remote cluster until AFM recovery is executed.", and IBM support team will say "that's no problem, please wait until the AFM Recovery procedure is to be completed." That's true but I hope this blog will be helpful for those who are considering explaining the reason to your clients.
This is the result of activities in Spectrum Scale CoC, and thanks to members for their valuable feedback and discussions.
1.Active File Management(AFM)
This section describes an overview of AFM, so please skip it and go to section 2 when you are familiar with AFM.
Active File Management (AFM) enables data sharing across clusters. AFM requires that you create associations between IBM Spectrum Scale clusters, or between an IBM Spectrum Scale cluster and an NFS data source.
An AFM fileset can be enabled on a GPFS independent fileset. IBM Spectrum Scale cluster that contains AFM filesets is called an AFM cache cluster. An AFM cache cluster has a relationship with another remote site called the AFM home, where either the AFM cache or the AFM home can be the data source or destination.
Each AFM fileset in an AFM cache cluster is served by one of the nodes that is designated as the gateway in the cluster. The gateway node that is mapped to a fileset is called the primary gateway of the fileset. The primary gateway acts as the owner of the fileset and communicates with the AFM home.
2. AFM queue
A primary gateway has an AFM queue whose maximum size is configured by "afmHardMemThreshod" configuration parameter, and the AFM queue is created for each fileset and stores user writing request information called "message" that does not include actual user data. A message is processed in multi-threads as long as no dependencies exist among the messages. The number of threads for each fileset is configured by "afmNumFlushThreads" configuration parameter.
3. AFM Recovery
When a primary gateway loses messages in an AFM queue for a fileset and gets out of sync, the primary gateway compares data between the AFM cache and the AFM home, then rebuilds incremental messages. This series of processes is called "AFM Recovery". The subsections describe in more detail the AFM Recovery processes.
3-1. When will AFM Recovery happen?
AFM Recovery will happen when a primary gateway loses messages by following events.
-
-
An administrator runs mmafmctl flushPending command.
-
An AFM queue reaches the limit of the memory used by messages defined by afmHardMemThreshod, then some messages are dropped.
-
A GPFS daemon on a primary gateway is suddenly shutting down for some reason.
-
A primary gateway tries to transfer messages in an AFM queue to another gateway when an administrator runs mmshutdown command but the transfer process is failed.
3-2. Normal queue and priority queue
A user can write data to an AFM fileset in the midst of AFM Recovery. The primary gateway for the fileset must complete AFM Recovery processes before processing user writing requests to ensure data consistency between the AFM cache and the AFM home. A primary gateway has two types of AFM queue, a normal queue and a priority queue, for a fileset. A priority queue stores messages rebuilt by AFM Recovery operations. A normal queue stores messages by user writing requests, and a process of the normal queue is suspended until the priority queue of the fileset gets empty.
3-3. AFM Recovery process
This subsection describes the entire flow of the AFM Recovery.
-
-
A primary gateway loses messages.
-
The primary gateway creates a snapshot of a fileset on AFM cache cluster.
-
The primary gateway scans files on the AFM home, and generates a file list in /var/mmfs directory.
-
The primary gateway finds unprocessed data by comparing the snapshot and the file list.
-
The primary gateway rebuilds messages and puts them into the priority queue.
-
The primary gateway removes the snapshot and the file list.
-
The primary gateway processes messages in the priority queue.
-
The primary gateway processes messages in the normal queue.
4. AFM Resync
When a primary gateway gets into the situation which needs to synchronize all files of an AFM fileset, then rebuilds messages for all of the files and queues them to the priority queue of the fileset. This series of processes is called "AFM Resync". The subsections describe the AFM Resync processes in more detail.
4-1. When will AFM Resync happen?
AFM Resync will happen when two or more inconsistencies that can not be recovered are detected or unexpected error occur, for example a product defect happens.
4-2. Normal queue and Priority queue
The primary gateway uses a priority queue for AFM Resync similar to AFM Recovery, please see the section 3-2.
4-3. AFM Resync process
A primary gateway works almost the same way of AFM Recovery, please see the section 3-3. The only difference is that the primary gateway rebuilds messages for all of the files without comparing.
5. ATTENTION! /var/mmfs/afm shortage
A file list is generated when AFM Recovery and Resync, so please ensure that there should be sufficient disk space, 250B per inode, in /var/mmfs/afm. If the AFM fileset has 1,000,000 files or directories, /var/mmfs/afm needs 250MB disk space.
#Highlights-home#Highlights