Accelerating IO500 with IBM Storage Scale Data Acceleration Tier (DAT)

One of the most satisfying performance improvements comes from putting previously idle resources to work. 

The Saguaro cluster, one of the IBM Storage Scale systems used for our IO500 submissions, consists of 10 AMD client nodes connected through InfiniBand to a single IBM Storage Scale System 6000 building block. Each client contains eight 3.2 TB NVMe SSDs. One drive is used for the operating system, leaving seven available for other purposes—a total of 70 NVMe devices across the cluster. 

For our SC24 IO500 submission, those 70 devices remained idle during benchmark execution. The system still delivered strong performance for a single storage building block but leaving that much high-performance NVMe capacity unused during a storage benchmark was a clear opportunity. 

In 2025, IBM Storage Scale introduced decentralized data acceleration tier (also known as decentralized DAT). The key benefits of this feature include: 

The decentralized DAT feature is particularly beneficial for AI and HPC workloads that exhibit predictable, locality-sensitive data access patterns. By placing data closer to the compute resources that consume it, DAT improves application performance, reduces data movement, and increases infrastructure efficiency.  With the introduction of Decentralized DAT, we asked whether our knowledge of the IO500 data-access pattern could be used to place replicas on the same nodes that would later read the data.  The answer was yes. By aligning replica placement with the benchmark’s access pattern, we enabled the IO500 workload to read from client-local NVMe and exploit the aggregate bandwidth of all 70 devices. This locality-aware design produced substantial performance gains by putting storage already installed in the client nodes to productive use, without adding new hardware. 

Results

The most dramatic improvements were observed in the ior-easy workloads that were moved onto the DAT performance pool built from client-local NVMe devices. 

In the table, the following two submissions are compared: 
1. https://io500.org/submissions/view/722 (Saguaro IBM Storage Scale System 6000 SC24 submission) 

2. https://io500.org/submissions/view/784 (Saguaro IBM Storage Scale System 6000 SC25 submission) 

1. The aggregate bandwidth of the client NVMe drives exceeded the bandwidth available from the IBM Storage Scale System 6000 central storage alone. 
2. DAT allowed data to be placed close to the nodes that would later read it, thus reducing network traffic and improving locality. 
   
   

System Architecture 

The following figure describes the Saguaro configuration that was used for the SC25 submission:

• A single IBM Storage Scale System 6000 building block provided the primary storage infrastructure.
• Seven NVMe drives on each of the 10 client nodes were combined into a decentralized DAT performance pool.
• Metadata-intensive benchmarks and the ior-hard workloads continued to use the IBM Storage Scale System 6000. The optimization focused only on the ior-easy files. 
  
  

Understanding the IO500 Access Pattern 

The IO500 benchmark deliberately avoids simple client-cache effects.

During the ior-easy write phase, each MPI task creates its own file. In the subsequent read phase, IO500 uses the IOR -C (reorderTasks) option to ensure that clients do not read the files they originally created. Instead, each task reads a file written by another task.

This processing is intended to prevent benchmark results from being inflated by local caching.

However, it also creates a predictable relationship between the node that writes a file and the node that will later read it.

That predictability creates an opportunity for locality-aware placement. 

Using DAT to Exploit Locality

Instead of placing each file on the node that created it, we used DAT placement policies to place each file on the node that would later consume it. 

The workflow was: 

This optimization reduced network traffic while allowing the benchmark to exploit the aggregate bandwidth of the client-local NVMe devices. 

DAT Placement Policy 

The placement policy established a deterministic relationship between: 

Conceptually, the policy mapped ranges of files to specific DAT placement groups: 

Files 0-24 -> Placement Group 10, which maps to Storage Scale's "Failure Group ID 1,0,10"
Files 25-49 -> Placement Group 1, which maps to Storage Scale's "Failure Group ID 1,0,1" 

Files 50-74 -> Placement Group 2, which maps to Storage Scale's "Failure Group ID 1,0,2" 
[…] 
Files 225-249 -> Placement Group 9, which maps to Storage Scale's "Failure Group ID 1,0,3"

Each placement group corresponded to storage located on a particular client node.

The full policy used for the benchmark is shown below:

cat ior_policy.pol 
 RULE 'ior_file_easy.0-24' SET POOL 'system' REPLICATE(0:1) WHERE (name >= 'ior_file_easy.00000000' AND name <= 'ior_file_easy.00000024') AND (setWADFGP(1,0,10)) 

RULE 'ior_file_easy.25-49' SET POOL 'system' REPLICATE(0:1) WHERE (name >= 'ior_file_easy.00000025' AND name <= 'ior_file_easy.00000049') AND (setWADFGP(1,0,1)) 

RULE 'ior_file_easy.50-74' SET POOL 'system' REPLICATE(0:1) WHERE (name >= 'ior_file_easy.00000050' AND name <= 'ior_file_easy.00000074') AND (setWADFGP(1,0,2)) 

RULE 'ior_file_easy.75-99' SET POOL 'system' REPLICATE(0:1) WHERE (name >= 'ior_file_easy.00000075' AND name <= 'ior_file_easy.00000099') AND (setWADFGP(1,0,3)) 

RULE 'ior_file_easy.100-124' SET POOL 'system' REPLICATE(0:1) WHERE (name >= 'ior_file_easy.00000100' AND name <= 'ior_file_easy.00000124') AND (setWADFGP(1,0,4)) 

RULE 'ior_file_easy.125-149' SET POOL 'system' REPLICATE(0:1) WHERE (name >= 'ior_file_easy.00000125' AND name <= 'ior_file_easy.00000149') AND (setWADFGP(1,0,5)) 

RULE 'ior_file_easy.150-174' SET POOL 'system' REPLICATE(0:1) WHERE (name >= 'ior_file_easy.00000150' AND name <= 'ior_file_easy.00000174') AND (setWADFGP(1,0,6)) 

RULE 'ior_file_easy.175-199' SET POOL 'system' REPLICATE(0:1) WHERE (name >= 'ior_file_easy.00000175' AND name <= 'ior_file_easy.00000199') AND (setWADFGP(1,0,7)) 

RULE 'ior_file_easy.200-224' SET POOL 'system' REPLICATE(0:1) WHERE (name >= 'ior_file_easy.00000200' AND name <= 'ior_file_easy.00000224') AND (setWADFGP(1,0,8)) 

RULE 'ior_file_easy.225-249' SET POOL 'system' REPLICATE(0:1) WHERE (name >= 'ior_file_easy.00000225' AND name <= 'ior_file_easy.00000249') AND (setWADFGP(1,0,9)) 

RULE 'default' SET POOL 'system' 

Mechanics of Building a DAT File System

The previous policy, and the overall IO500 optimization described here, depended on first creating a decentralized DAT file system. In a decentralized DAT deployment, the DAT performance pool is built from NVMe devices that physically reside in the client nodes rather than in centralized storage enclosures. IBM Storage Scale combines these distributed devices into a shared file system while maintaining a single global namespace, as described in the decentralized DAT deployment instructions.

For the Saguaro configuration, the process was straightforward:

1. An IBM Storage Scale System 6000 building block hosted the IBM Storage Scale RAID pool (also called GPFS Native RAID, or GNR) pool, where data was protected by using erasure coding. The GNR pool worked in conjunction with the client-local DAT performance pool, which maintained performance replicas on the client NVMe devices. Together, the two pools provided both high performance and data resilience. 
2. For this configuration, the fs1 file system was created by using the standard deployment procedure for an IBM Storage Scale System 6000 performance model (equipped with 48 NVMe drives).
3. Seven of the local NVMe drives on each of the 10 client nodes were designated as DAT performance-pool devices, for a total of 70 drives.  An NSD stanza file for the performance pool NSDs was created, to define each of these drives. Shown below, the file begins with a pool stanza that defines the performance pool characteristics as supporting affinitized allocations (allowWriteAffinity=yes), followed by NSD stanzas to list each of the disks: 
   

After this policy is configured, applications access files exactly as they would in any other IBM Storage Scale file system. The difference is that placement policies can influence where the data is physically stored. In our case, policy rules directed the IO500 ior-easy files to specific placement groups (called failure groups) that mapped to the NVMe resources of the nodes that would later consume the data.

One of the key advantages of decentralized DAT is that it allows locally attached NVMe resources to participate in a clustered IBM Storage Scale file system without requiring applications to directly manage data placement. The shared namespace remains unchanged, while IBM Storage Scale handles the mapping between files, placement groups, and the underlying client-local storage.

For more information about decentralized DAT deployment, consult the complete setup procedure, best practices, and file-system creation steps in the IBM Storage Scale documentation for decentralized DAT deployments: decentralized DAT deployment instructions. 

Beyond IO500 

The significance of this result extends beyond IO500. 

Many IBM Storage Scale deployments already contain substantial storage capacity within compute or client nodes. Those devices often sit idle or are reserved for specialized applications. 

Decentralized DAT allows those resources to participate in the IBM Storage Scale architecture while preserving a shared namespace and normal application access patterns. 

Applications continue to access files normally. Placement policies determine where the data is physically located. 

Many AI and HPC workflows exhibit predictable data-access patterns that can benefit from locality-aware placement. 

Examples include: 

Summary