Earlier in my career, I did work on the NetApp filers which at that point in time were also known as NAS filers. Typically used for storing data in forms of files only. At later point in time, they came up with support for the blocks protocols allowing the storage to be used in the form of blocks. The protection of data on larger filers can be complex and take a long time. As data continues to grow, every day we lose time or need more time to backup the same.  To help with this, technologies that utilize the "incremental forever" process of backing up data have been able to keep up with the growth of data. For filers, performing the incremental forever backup is difficult, however NetApp, a number of years ago, came out with a piece of technology for its ONTAP OS in its filers to help with this. It is  called SnapDiff. This added one more option for backing up filers and enabled product such as Storage Protect to be able to do incremental forever backups and be able to meet backup windows for customers using Snapshot differencing.

Overview  of Snapdiff:

"Snapdiff” refers to NetApp differencing engine/api, “snapshot differential backup”refers to TSM function. Internal differencing engine that quickly identifies file and directory differences between two Snapshot copies (processing is on the filer). The differences are loaded into the databases which avoids scanning entire file system. 

How SnapDiff Works ?

The first full backup job creates a baseline snapshot. Data is scanned by traversing the file system, a snapshot is created, and then data is backed up from the snapshot.

Subsequent backups are incremental. SnapDiff identifies the data that changed by comparing the previous and current snapshots, without traversing the file system. Data is then backed up from the snapshot.

At the end of each job, snapshots created by the previous job are deleted from the filer, and only one snapshot is retained on the filer.

The scan operation that traverses the file system is used only in these cases:

• The snapshot from the previous job is not available
• The snapshot creation fails in the current job
• The SnapDiff operation fails

Protecting a filer is a mission critical for almost any infrastructure lab. IBM Storage Protect has multiple ways to back up a NetApp filer including NDMP, Mirroring to tape or using “snapshot difference incremental backups --> Snapdiff”. It provides an efficient file-level based “incremental forever” backup method to back up files on a NetApp filer to an IBM Storage Protect server.

Snapdiff History with Storage Protect:

Use of Snapdiff was introduced in TSM 6.1 and functions were added to improve the support. Like mentioned below:

TSM 6.4 (2012)

• Read-only snapshot support for snapmirror
• Multistore (vFiler) support

TSM 7.1 (2013)

• Persistent change log
• HTTPS support

TSM 7.1.2 (2014 – 2015)

• Support for c-mode

ZAPI and Version 3 APIS

The backup vendors would use Snapdiff V1 apis(RPC based) referred to as ZAPI to perform the backups. Communication to the filer via HTTP or HTTPS connection. It would support the data volumes only. With the improvements in the RestApi technology, NetApp replaced ZAPI with Version3 apis. Mostly REST based,  and a portion of the differencing API uses RPC. To use Snapdiff with newer Ontap versions (9.8 and above), or to use the version 3 apis backup vendors are required to use the license from Netapp. This license used for generating the access tokens.  Which returns an RPC handle which is used by RPC api’s to manage the snapdiff session over a secure RPC session. Snapdiff V3 api supports both "Data and Flex" volumes.

Support with 8.1.23 Storage Protect Client:

Upgrading to IBM Storage Protect Client - 8.1.23, will enable the backup administrators to backup "Data" as well as "Flex" volumes on the Netapp filers with the support for Snapdiff. It will continue to  support all supported version of Ontap. By default, Ontap Version 9.8 and above use the version 3 APIs and for older versions ZAPI is the only choice. Existing volumes backed up by older clients with the version 1 API will work seamlessly with the version 3 API.  The logic of using snapdiff with Storage Protect remains more or less the same. The end user will not notice in any change in the usage. Based on availability of license file on the Storage protect client, version 1 or version 3 apis will be used. 

Advantage of using the Version 3 apis:

Use of version 3 REST API’s validate the NetApp filer SSL certificate to avoid “man in the middle” attacks.The client obtains and stores the filer certificate on disk or can optionally use a user provided certificate which is validated by REST api’s avoiding any breach.

 Thanks!

References:

https://www.ibm.com/support/pages/using-ibm-spectrum-protect-protect-netapp%C2%AE-filers-snapdiff%E2%84%A2-changing

https://kb.netapp.com/on-prem/ontap/DP/SnapDiff/SnapDiff-KBs/What_is_SnapDiff