Primary Storage

 View Only

IBM Spectrum Scale Safeguarded Copy Next milestone for Cyber Resilient Storage

By Shashank Shingornikar posted 9 days ago

  

IBM Spectrum Scale Safeguarded Copy, Next milestone for Cyber Resilient Storage.

Spectrum Scale recently announced the release of 5.1.5. You can read about the summary of all the changes under Resources. One of the notable features available with this release, the Safeguarded Copy is a mechanism to protect fileset and file system data from accidental or deliberate compromise.

In this blog I’ll be going over a threat scenario and also the mitigation mechanism to help people see the usage and creatively think about other scenarios that will help minimize damage in an event of cyber threat.

What are Safeguarded Copies on Spectrum Scale?

The safeguarded copy (SGC) is a mechanism to protect data in IBM Spectrum Scale file systems. It is based on the immutable snapshot feature. The mechanism secures data from deliberate or accidental compromise. These immutable snapshots share the storage space with the production file system, and can be accessed online along with other regular snapshots.

The following facts help to explicitly define the Safeguarded copies:

  • Safeguarded copies (SGCs) are immutable copies of a file system or fileset, and they help minimize the impact of cyber attacks, disasters and failures.
  • SGCs are snapshots with a retention time, and which cannot be altered, and also cannot be deleted until the retention time is expired.
  • The administrator can use the GUI to schedule periodic snapshot creation, and also snapshot deletion after the retention time has expired.
  • The SGC cannot be modified or deleted by a single bad actor, and the SGC’s retention time cannot be changed by a single bad actor.

 These snapshots act as backup copies for recovering data if the primary data is corrupted or destroyed. If the content of files is damaged, you can run the mmrestorefs command to restore the production file system or its independent filesets from these immutable snapshots.

 Data protection can be ensured by using the following three workflows:

  • Safeguarded environment setup
  • Restoring contents from Safeguarded backups
  • Snapshots deletion ahead of expiration period

Let us now look at each of these aspects.

Safeguarded environment setup

The first step in setting up the Safeguarded environment is to enable sudo administration environment. Nils Haustein has an excellent blog post that takes a deep dive in explaining the  sudo environment and how it is relevant with Spectrum Scale. The blog also provides step by step instructions on how to enable sudo wrapper for Spectrum Scale. The link to the blog post is provided under Resources section.

The second step is to configure the schedule for periodic snapshot creation with retention time for snapshots and also deletion of snapshots post elapsed expiration time.

Log into Spectrum Scale GUI and from top left corner click on 4 parallel lines  to open Spectrum Scale side menu. From the side menu, click on Files ® Snapshots. When multiple snapshots are present a (+) sign is displayed in front of the file system name. Clicking the (+) sign revels all snapshots. (see figure 1)

Figure 1: File system with multiple snapshots

Under the Type column we see many entries, but we're interested in these two, Scheduled & Manual. As the name indicates the Scheduled one's are created by schedule configured by the user and the manual one's is created using Cyber Resiliency workflow involving QRadar as threat detection mechanism.

For now, let’s look at how the periodic scheduling of Safeguarded Copy snapshots are configured.

By creating a snapshot rule, you can automate the snapshot creation, retention , and deletion schedules. That is, in a snapshot rule you can specify a frequency in which the snapshots must be created and the number of snapshots that must be retained for a period. The retention policy helps to avoid unwanted storage of snapshots that result in waste of storage resources.

Creating schedule for periodic snapshots

From the snapshots screen, click on Snapshot Rules and then click on Create Rule.

Update the parameters on the form to define the schedule. The sample snapshot rule creation is shown here.

 

As per the sample shown here the snapshot retention will have following characteristics.

·       The rule 2-hourly-Safeguarded-Snapshots

will be executed multiple times of day, starting with 6AM in the morning and thereafter every 2 hours till 10PM.

 

The retention parameters indicate,

 

·       2 most recent the snapshots will be per day.

 

·       Latest snapshot created every day will be kept for 6 days

 

·       Latest snapshot from last 2 weeks will be kept available. (Week end = Satureday )

 

·      Latest snapshot from last 3 months will be kept available.

Figure 2: Create snapshot rule

For more information refer Spectrum Scale 5.1.5 documentation under Resources section

Once the rule is defined, associate the rule with a filesystem. The system will now automatically schedule snapshots creation based on frequency and retention, and will also take into account, whether a snapshot can be removed if it has passed the expiration time.

Integrating IBM QRadar and IBM Spectrum Scale Safeguarded Copy

To explain the need of this integration, let’s take into consideration following scenario.

A user is modifying confidential data outside business hours. This trivial use case brings up lot of questions the business is answerable to, they are;

  1. Business impact.
  2. How are user actions tracked (business users & administrators).
  3. Data recovery.
  4. Accountability, further scrutiny for business compliance.

We can consider following possible Mitigation aspects to safeguard the data against such incidents.

  • Enable filesystem audit.
  • Send audit events to IBM QRadar.
  • Define rules, actions in IBM QRadar to identify anomaly and act.
  • Mechanism to recover data.

This scenario is explained using the figure below.

Figure 3: Cyber resiliency solution overview.

The cyber resiliency solution that will cover the scenario is described below with all the relevant configuration options.

Enable Spectrum Scale Audit logging

IBM Spectrum Scale File Audit Logging that, when enabled, logs all of the file access to the file system with the required audit information.

To enable Spectrum Scale auditing log on to Spectrum Scale Management GUI and from the side menu, select Files and then select File Systems. 

A list of all the file systems on the cluster are listed here. Click and Select the filesystem you wish to enable audit for and then click Actions menu and choose Enable File Audit Logging option.

Figure 4: Enable file system auditing from Spectrum Scale GUI

To identify and detect potential malicious data access, and for compliance auditing purposes, such file audit logs can be integrated with the SIEM solution. This capability is provided by IBM QRadar, to which the IBM Spectrum Scale file access logs can be securely directed.

To enable the file audit logs take following actions.

  • Install RSyslog.
  • Create configuration file for RSyslog in /etc/rsyslog.d folder. Table 1 below shows same configuration file crated in lab environment.
  • Start / Restart the RSyslog daemon.
/etc/rsyslog.d/qradar.conf

module(load="imfile" PollingInterval="5")

input(

type="imfile"      File="/ibm/gpfs01/.audit_log/SpectrumScale_150_13060915503498309490_1668006526_FSYS_gpfs01_audit/auditLogFile*"

      Tag="FAL"

      Severity="error"

      Facility="local4"

)

local4.* action(type="omfwd" target="9.11.221.149" port="514" protocol="tcp")

Table 1: A sample custom rsyslog configuration file.

The target IP address belongs to IBM QRadar listening on incoming events on port 514.

Note: The default RedHat Linux installation sets selinux to enforcing to ensure maximum security. This policy has effect the way, rsyslog imfile module loading events from Spectrum Scale audit log. To mitigate this, check the var log message for “ausearch -c 'in:imfile' --raw | audit2allow -M my-inimfile” messages. Executing this command generates a policy. Upon applying the policy, the module is correctly loaded to relay events from Spectrum Scale cluster to IBM QRadar

At the time of this writing IBM QRadar needs manual parsing of events received from IBM QRadar. The parsing procedure normalizes and categorizes incoming events making it simpler to apply rules. To perform parsing use following instructions.

Create required QRadar Identification entries (QID)

This section describes how to create your own QIDs in QRadar. You can do this using GUI too, but when creating a large number of QIDs the creation is quicker using command line interface.

  • Log on to IBM QRadar
  • Get a list of all the QIDs currently defined execute following command

/opt/qradar/bin/qidmap_cli.sh -l > /tmp/Category_type_CLI.txt


Sample entry from /tmp/Category_type_CLI.txt is shown here.

ID Name Parent
8028 File Created   System

Table 2: Sample qidmap entry defined in QRadar

  • Using the QID definitions from /tmp/Category_type_CLI.txt file, you can decide under which low level you want to categorize your QID for Spectrum Scale. In his case the Create_File entry is being created under 8028 low level category connected to System category. Both the sample command and output generated is shown below.

    command : opt/qradar/bin/qidmap_cli.sh -c --qname Create_File --qdescription "Attempt to create file" --severity 1 --lowlevelcategoryid 8028

    Created entry:
           qid: 2000027
           name: Create_File
           description: Attempt to create file
           severity: 1
           low level category id: 8028
           ratethreshold: 0
          catpipename: Golf
           rateshortwindow: 0
           ratelongwindow: 0
           reverseip: false
    Table 3: Sample qid creation along with output

  • Repeat the process to create qidmap for RENAME, OPEN, UNLINK, CLOSE, XATTRCHANGE, CREATE, ACLCHANGE, RMDIR, DESTROY audit events.

Creating Log source types

The log source types help you categorize and also map the events appropriately based on the classification shown earlier to distinguish the events.

  • Log into IBM QRadar, and in Log Activity tab, use Add filter button and select Source or Destination IP address in parameter field. Choose the value Equals in Operator and finally enter the IP address of Spectrum Scale node where RSyslog is configured.

This will help filter all the audit log events relayed by IBM Spectrum Scale. Adjust the view by selecting past events. See sample figure below.

Figure 5: Displaying events from past time.

  • Select a few events, right click the selection and choose option View in DSM Editor.
  • Click on Change button in the left pane and in the Select Log Source Type screen, click on Create New to create a new log source. Name the log source as IBM Spectrum Scale.
  • The lower right pane titled Log Activity Preview shows the selected events with parsing status NOT parsed and NOT mapped.
  • Select the Event ID property in the left pane and enter following regular expression. The sample regular expression is shown in table 3
Entity Value
Expression type event”:\s”([^”]+)
Format string Default value or $1
Table 3: Sample regular expression

  • Repeat the process for Event Category property

The parsing status for the events shown in Log Activity Preview change to Parsed but NOT Mapped.

  • Repeat the process for the properties you want to extract from the event payload. It is also possible to define any custom property and populate it using regular expressions.
For this setup create following properties with respective regular expressions.

Property Regular expression Capture group
Filename "path":\s"(.*)\/(.*\.csv)" 2
File path "path":\s"(.*?)" 1
clientGroupID \s"clientGroupId":\s"(.*?)" 1
clientUserID \s"clientUserId":\s"(.*?)" 1

Table 4: Custom event properties

  • Once all the properties are defined, use Event Mapping tab from left pane and assign the QIDs we created earlier for each event. Only when the mapping is completed, the parsing status for the events shown in Log Activity Preview changes to Parsed and Mapped.
  • Save all the changes made so far by clicking the Save button and click Close to close DSM editor.
  • Back in Log Activity Window, click on Admin menu item to define a new log source listed under Data sources
  • Click on +New Log Source button to start the log creation wizard. Enter following values.
Question Response

How many log sources will you be adding?

Single

Select Log Source Type

IBM Spectrum Scale
Select Protocol Type Syslog
Configure Log Source Parameters
  • Name
IBM Spectrum Scale
  • Description
IBM Spectrum Scale Log Source
  • Enabled
Default value
  • Groups
Other
  • Extension
IBMSpectrumScaleCustom_EXT
  • Language
Default value
  • Target Event Collector
Default value
  • Credibility
Default value
  • Coalescing Events
Default value
  • Store Event Payload
Default value
  • Log Source Identifier
2nd field from audit event (see spectrum scale sample audit event)
  • Incoming Event Payload Encoding
Default value

Table 5: Responses for new Log Source definition

Sample Spectrum Scale audit event
<163>Nov 15 00:31:07 cid-vm-23 FAL {"LWE_JSON": "0.0.5", "path": "/ibm/gpfs01/sales/car-sales-figures-YOY-2021-22.csv", "clusterName": "cldr-cid-gpfs.cldrisv.tuc.stglabs.ibm.com", "nodeName": "cid-vm-23", "nfsClientIp": "", "fsName": "gpfs01", "event": "OPEN", "inode": "27650", "linkCount": "1", "openFlags": "0", "poolName": "system", "fileSize": "823", "ownerUserId": "1055", "ownerGroupId": "1055", "atime": "2022-11-14_10:00:32.723-0500", "ctime": "2022-11-15_00:30:28.572-0500", "mtime": "2022-11-11_03:07:51.437-0500", "eventTime": "2022-11-15_00:31:02.048-0500", "clientUserId": "0", "clientGroupId": "0", "accessMode": null, "processId": "6963", "bytesRead": "0", "bytesWritten": "0", "minReadOffset": "9223372036854775807", "maxReadOffset": "0", "minWriteOffset": "9223372036854775807", "maxWriteOffset": "0", "permissions": "200100664", "acls": null, "xattrs": null, "subEvent": "NONE"}

  • Once the Log Source definition is complete, from Admin screen click on Deploy so QRadar is able to map future audit events to newly defined Spectrum Scale Log Source. See figure 6.

Figure 6: Normalized Spectrum Scale Audit Events

Definition of Reference sets

Reference sets are used to store set of value used as lookup table. Reference sets are defined from Admin screen. For this solution 2 reference sets were defined. Authorized-Sales-Users reference was created with numeric UID values corresponding to operating system operating system UID’s. The other reference set, Confidential-Files was created with list of filenames to be monitored from file system.

Definition of Custom action

Custom actions are used to execute specific action on the target device. They provide an automation mechanism for task execution based on parameters provided. The programs / scripts are uploaded from Admin screen Custom Action section. Currently three interpreters bash, perl and python are supported. For this solution, a simple Python script accepting parameters like Spectrum Scale Management GUI address, GUI login id, password and filesystem was written. The script made REST API calls using given parameters, to Spectrum Scale Management GUI to create ad-hoc snapshots. For more information about Spectrum Scale Management API see Resources section.

Threat detection using IBM QRadar

This section shows configuration steps to setup Custom Rules in IBM QRadar to determine threat by reading normalized event data. Click on Rules and then click on Rules option from the menu. Start the New Event Rule creation wizard.

Figure 7: Custom rule configured for threat detection based on audit event properties

With the cyber resiliency workflow in place now, every time a confidential file stored on Spectrum Scale filesystem is accessed and/or modified by a user outside of business hours, a filesystem snapshot will be created to protect the original data and also an offence is generated in IBM QRadar against the user performing the action.

Restoring data from Safeguarded Copy Snapshot

If fileset or file system data is damaged or corrupted, it can be restored from one of the snapshots with the mmrestorefs command. If only a few files have been affected then these can be restored by copying them directly from the snapshot to the corresponding file in the file system. The snapshot appears as part of the file system or fileset namespace. For more information, see Resources section for link to mmrestorefs command.

Resources

  • Spectrum Scale 5.1.5 Release information

www.ibm.com/docs/en/spectrum-scale/5.1.5?topic=summary-changes

  • Configure sudo wrapper for Spectrum Scale

community.ibm.com/community/user/storage/blogs/nils-haustein1/2020/12/17/spectrum-scale-sudo-wrappers

  • Spectrum Scale Safeguarded Copy

www.ibm.com/docs/en/spectrum-scale/5.1.5?topic=administering-protecting-file-data-spectrum-scale-safeguarded-copy

  • Managing Snapshots using Spectrum Scale GUI

www.ibm.com/docs/en/spectrum-scale/5.1.5?topic=systems-managing-snapshots-using-spectrum-scale-gui

    • Spectrum Scale Management API

    www.ibm.com/docs/en/spectrum-scale/5.1.5?topic=overview-spectrum-scale-management-api

    Thank You!!


    #Featured-area-2
    #Featured-area-2-home
    1 comment
    61 views

    Permalink

    Comments

    Nice one , well done. 

    Is this feature available on all scale license? DM, DE, ECE? or only DM