This is part of a multi-part series on IBM Sterling Global Mailbox. 

Introduction 

IBM Sterling Global Mailbox helps companies address demands for high availability operations and redundancy with a robust and reliable data storage solution available across geographically distributed locations.  It is an add-on to Sterling B2B Integrator and Sterling File Gateway. 

Global Mailbox leverages IBM and open source technology to replicate data to many data centers (DCs) helping to minimize downtime during planned and unplanned outages. 

This article explains, at a high level, how files are replicated with Global Mailbox. 

What is a payload? 

When a file is uploaded to Global Mailbox, a “Message” is created.  This Message object consists of: 

Metadata 

This is data that describes the message such as who created it, when it was created, what mailbox it is in and how many times it can be downloaded.  Metadata is stored in Cassandra and replicated by Cassandra. 

Payload 

This is the file data itself.   

Files less than 10kb are called Inline Payloads.  For performance reasons, these are stored and replicated by Cassandra. 

Files larger than 10kb are stored in disk in 100mb segments.  These segments are replicated with the Aspera FASP protocol.  

What is asynchronous/delayed replication? 

With asynchronous replication, the replication happens some point after the file/message is created. 

For example, if you use FTP put to add a file to a mailbox, when the put completes, the message is only guaranteed to be in the local data center.  It will appear in the other data centers later. 

Advantages 

• Faster than async replication because the system doesn’t have to wait for replication 

Disadvantages 

• A failure of the receiving data center before replication completes could result in a “lost” file 

What is a synchronous/immediate replication? 

With synchronous replication, the replication happens as part of the file/message creation. 

For example, if you use FTP put to add a file to a mailbox, when the put completes, the message is guaranteed to be in a majority (n/2+1) of the datacenters. 

Advantages 

• Guaranteed to have the file/message in more than one DC 

Disadvantages 

• Operations can take longer due to the added cost of replication.  The extra time is dependent on many factors such as network conditions and workload. 

• Requires at least 3 DCs to be viable
  • Majority of 2 DCs is 2 – cannot survive the outage of a full DC 
  • Majority of 3 DCs is 2 – can survive the outage of a full DC 

How does async replication work? 

The following diagram provides the overall flow of async replication of payloads. 

Summary: 

1. A protocol client starts uploading a file 
2. Global Mailbox checks file size.
   1. If less than 10kb, the payload is stored in Cassandra 
   2. If greater than 10kb, the payload is stored on the local shared disk in 100mb segments 
3. Bytes finish uploading 
4. If there is an event rule/routing rule for this message, put an event on the MQ queue to trigger processing 
5. The file upload completes 
6. Replication happens later 

In the other DCs (happens outside of the upload process): 

1. The Replicator is continuously polling for a list of payload segments to replicate 
2. For payload segments in Cassandra 
   1. When the payload is available in the Cassandra table in the local data center, the message is marked as “available” in this data center.
3. For payload segments on disk 
   1. Replicator uses the Aspera FASP protocol to pull the segment to the local data center 
   2. If all segments for the payload/file are replicated to this data center, the message is marked as “available” in this data center.   

Note: If a message is not marked as available, it is not visible in mailbox listings and not available for download. 

How does sync replication work? 

The following diagram provides the overall flow of sync replication of payloads. 

Summary (differences in bold): 

1. A protocol client starts uploading a file 
2. Global Mailbox checks file size
   1. If less than 10kb, the payload is stored in Cassandra 
   2. If greater than 10kb, the payload is stored on the local shared disk in 100mb segments 
3. Bytes finish uploading 
4. Global Mailbox waits for replication to a majority of data centers 
5. If there is an event rule/routing rule for this message, put on event on the MQ queue to trigger processing 
6. The file upload completes 

In the other DCs: 

1. Replicator is continuously polling for a list of segments to replicate 
2. For payload segments in Cassandra 
   1. When the payload is available in the Cassandra table in the local data center, the message is marked as “available” in this data center. 
3. For payload segments on disk 
   1. Replicator uses the Aspera FASP protocol to pull the segment to the local data center 
   2. If all segments for the payload/file are replicated to this data center, the message is marked as “available” in this data center. 
4. If this payload is fully replicated to a majority of data centers, notify the receiving data center to complete the upload 

Note: If a message is not marked as available, it is not visible in mailbox listings and not available for download. 

Summary 

By using IBM and opensource technology, Global Mailbox is able to minimize disruption to your business by replicating mailbox data quickly and efficiently to all of your data centers. 

Asynchronous replication is the default and ideal in most situations. It balances the needs of data consistency and system availability. 

Synchronous replication is an option for enterprises that require a higher level of consistency while still having availability.