Sterling Managed File Transfer

Sterling Managed File Transfer

Come for answers, stay for best practices. All we're missing is you.

 View Only

'IBM Spectrum Scale or GPFS' Integration with IBM B2B Sterling Integrator

By Tanvi Kakodkar posted Thu February 13, 2020 02:00 AM

  
Originally updated on January 10, 2018 by BipinChandra

Authored and Edited By Bipin Chandra & Swarnamba S Kakarla

 

Introduction:

IBM Spectrum Scale or GPFS™, is a high performance shared-disk file management solution that provides fast, reliable access to data from multiple servers. Applications can readily access files using standard file system interfaces, and the same file can be accessed concurrently from multiple servers and protocols. IBM Spectrum Scale is designed to provide high availability through advanced clustering technologies, dynamic file system management, and data replication. IBM Spectrum Scale can continue to provide data access even when the cluster experiences storage or server malfunctions. IBM Spectrum Scale scalability and performance are designed for data intensive applications such as cloud storage engineering design, digital media, data mining, relational databases, financial analytics, seismic data processing, scientific research and scalable file serving. It is is a cluster file system that provides concurrent access to a single file system or set of file systems from multiple nodes. The nodes can be SAN attached, network attached, a mixture of SAN attached and network attached, or in a shared nothing cluster configuration. This enables high performance access to this common set of data to support a scale-out solution or to provide a high availability platform. 

Enterprises around the globe have deployed IBM Spectrum Scale to:


Benefits:
GPFS provides high performance by allowing data to be accessed over multiple computers at once. Most existing file systems are designed for a single server environment, and adding more file servers does not improve performance. GPFS provides higher input/output performance by striping blocks of data from individual files over multiple disks, and reading and writing these blocks in parallel. Other features provided by GPFS include high availability, support for heterogeneous clusters, disaster recovery, security,

Comparisions with Hadoop and its File System: 
1. HDFS does not expect reliable disks, so instead stores copies of the blocks on different nodes. The failure of a node containing a single copy of a block is a minor issue, dealt with by re-replicating another copy of the set of valid blocks, to bring the replication count back up to the desired number. In contrast, while GPFS supports recovery from a lost node, it is a more serious event, one that may include a higher risk of data being (temporarily) lost.
2. GPFS supports full Posix filesystem semantics where as HDFS do not support full Posix compliance.
3. GPFS distributes its directory indices and other metadata across the filesystem. Hadoop, in contrast, keeps this on the Primary and Secondary Namenodes, large servers which must store all index information in-RAM.
4. GPFS breaks files up into small blocks. Hadoop HDFS likes blocks of 64 MB or more, as this reduces the storage requirements of the Namenode. Small blocks or many small files fill up a filesystem's indices fast, so limit the filesystem's size.

 

IBM GPFS Client Service:

IBM GPFS Client Service from Sterling Integrator creates seamless connectivity to the Spectrum Scale storage environment. It is a scalable, reliable, and portable storage solution.

SI users can easily put and delete files and can perform many more operations. This article only focuses on putting the file into GPFS system and deleting it.

 

IBM GPFS-SI Integration Architecture Diagram:

 

Setting Up Your Project

Applications can readily access files using standard file system interfaces, and the same file can be accessed concurrently from multiple servers and protocols.

> You can simply use SI built in SFTP adapters to put/get/delete files from or to GPFS system using IBM B2Bi.

> For this client service we have used standalone service deployed in B2Bi independent of any other B2Bi adapters and services..

> You will need the JSch - Java Secure Channel API, please download it here. You will also need to integrate the .JAR files from the archive into your project.

> To download and setup IBM GPFS or Spectrum Scale contact IBM.

 

Installing IBM GPFS Client Service jar

Session creation with GPFS System:

            int sshPort = Integer.parseInt(sshPortStr);

            JSch jsch = new JSch();

            Session session = jsch.getSession(user, host, sshPort);

            java.util.Properties config = new java.util.Properties();

            config.put(STRICTHOSTKEYCHECKING, strictHostKeyChecking);

            session.setConfig(config);

            session.setPassword(password);

            session.connect();

 

Creating Channel:

            // Remember that this is just for testing and we need a quick

            // access, you can add an identity and known_hosts file to prevent

            // Man In the Middle attacks

            Channel channel = session.openChannel("sftp");

            channel.connect();

            ChannelSftp sftp = (ChannelSftp) channel;

 

 

PUT and DELETE the files from GPFS system:

Any protocol supported by GPFS can be used. We used SCP and SFTP. Read GPFS documentation for more details.

Please see the below implementation as how it is achieved for a quick reference.

 

Complete Java implementation:

This is how we create a java service for IBM GPFS Client. Customer can use this as a default implementation and can build their requirement on top of it.

 

/*
 *
 * IBM and Sterling Commerce Confidential 
 *
 * OCO Source Materials 
 *
 * IBM B2B Sterling Integrator 
 *
 * (c) Copyright Sterling Commerce, an IBM Company 2001, 2011
 *
 * The source code for this program is not published or otherwise
 * divested of its trade secrets, irrespective of what has been
 * deposited with the U.S. Copyright Office. 
 *
 */
package com.sterlingcommerce.woodstock.services.integration.IBMGPFSClientService;

import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InterruptedIOException;
import java.io.OutputStream;
import java.rmi.RemoteException;

import com.jcraft.jsch.Channel;
import com.jcraft.jsch.ChannelExec;
import com.jcraft.jsch.ChannelSftp;
import com.jcraft.jsch.JSch;
import com.jcraft.jsch.JSchException;
import com.jcraft.jsch.Session;
import com.jcraft.jsch.SftpException;
import com.sterlingcommerce.woodstock.services.IService;
import com.sterlingcommerce.woodstock.services.ServiceStoppedException;
import com.sterlingcommerce.woodstock.util.frame.log.LogService;
import com.sterlingcommerce.woodstock.util.frame.log.Logger;
import com.sterlingcommerce.woodstock.workflow.WorkFlowContext;
import com.sterlingcommerce.woodstock.workflow.WorkFlowException;


/**
 * 
 * @author Bipin Chandra
 */
public class IBMGPFSClientService implements IService {
    protected static Logger log;
    static {
        log = (Logger) LogService.getLogger("integrationservices");
    }
    private static boolean isOK = true;
    private static final String STRICTHOSTKEYCHECKING = "StrictHostKeyChecking";

    @Override
    public WorkFlowContext processData(WorkFlowContext wfc)
            throws RemoteException, WorkFlowException, ServiceStoppedException,
            InterruptedIOException {
        isOK = true;
        try {
            wfc.harnessRegister();
            String action = wfc.getParm("operation");// PUT,  DELETE
            String user = wfc.getParm("username");
            String host = wfc.getParm("gpfs_system_ip_address");// GPFS system ip
            String password = wfc.getParm("password");
            String local_path = wfc.getParm("local_path");// Path to read file from for uploading
            // String localFile = "C:\\localfile.txt";
            String remote_path = wfc.getParm("remote_path");
            // String remoteFile = "/root/localfile.txt";
            String sshPortStr = wfc.getParm("ssh_port");
            String strictHostKeyChecking = wfc.getParm("StrictHostKeyChecking");
            String messageId = wfc.getParm("messageId");// Message ID for mailbox

            log.log("IBMGPFSClientService.processData(): action:" + action);
            log.log("IBMGPFSClientService.processData(): user:" + user);
            log.log("IBMGPFSClientService.processData(): host:" + host);
            log.log("IBMGPFSClientService.processData(): password:" + password);
            log.log("IBMGPFSClientService.processData(): local_path:" + local_path);
            log.log("IBMGPFSClientService.processData(): remote_path:" + remote_path);
            log.log("IBMGPFSClientService.processData(): sshPort:" + sshPortStr);
            log.log("IBMGPFSClientService.processData(): strictHostKeyChecking:" + strictHostKeyChecking);
            log.log("IBMGPFSClientService.processData(): messageId:" + messageId);

            int sshPort = Integer.parseInt(sshPortStr);
            JSch jsch = new JSch();
            Session session = jsch.getSession(user, host, sshPort);
            java.util.Properties config = new java.util.Properties();
            config.put(STRICTHOSTKEYCHECKING, strictHostKeyChecking);
            session.setConfig(config);
            session.setPassword(password);
            session.connect();

            if ("put".equalsIgnoreCase(action)) {
                putFileToGPFS(session, remote_path, local_path);
            } else if ("delete".equalsIgnoreCase(action)) {
                log.log("Usage: IBMGPFSclient delete <s3_key>");
                delete(session, remote_path);
            } else {
                log.log("Usage: Incorrect Operation - It should be put/delete");
            }
            log.log("Done!");
        } catch (Exception e) {
            isOK = false;
            log.log(e.getMessage());
            log.logException(e);
            wfc.setBasicStatus(1);
            throw new WorkFlowException(e.toString());
        } finally {
            wfc.unregisterThread();
            if (isOK) {
                wfc.setBasicStatusSuccess();
            } else {
                wfc.setBasicStatusError();
            }
        }
        return wfc;
    }

    private void putFileToGPFS(Session session, String remote_path,
            String local_path) throws IOException {
        FileInputStream fis = null;
        boolean ptimestamp = true;
        try {
            // exec 'scp -t rfile' remotely
            String command = "scp " + (ptimestamp ? "-p" : "") + " -t " + remote_path;
            Channel channel = session.openChannel("exec");
            ((ChannelExec) channel).setCommand(command);

            // get I/O streams for remote scp
            OutputStream out = channel.getOutputStream();
            InputStream in = channel.getInputStream();

            channel.connect();

            if (checkAck(in) != 0) {
                return;
            }

            File _lfile = new File(local_path);

            if (ptimestamp) {
                command = "T " + (_lfile.lastModified() / 1000) + " 0";
                // The access time should be sent here,
                // but it is not accessible with JavaAPI ;-<
                command += (" " + (_lfile.lastModified() / 1000) + " 0\n");
                out.write(command.getBytes());
                out.flush();
                if (checkAck(in) != 0) {
                    return;
                }
            }

            // send "C0644 filesize filename", where filename should not include
            // '/'
            long filesize = _lfile.length();
            command = "C0644 " + filesize + " ";
            if (local_path.lastIndexOf('/') > 0) {
                command += local_path
                        .substring(local_path.lastIndexOf('/') + 1);
            } else {
                command += local_path;
            }
            command += "\n";
            out.write(command.getBytes());
            out.flush();
            if (checkAck(in) != 0) {
                return;
            }

            // send a content of lfile
            fis = new FileInputStream(local_path);
            byte[] buf = new byte[1024];
            while (true) {
                int len = fis.read(buf, 0, buf.length);
                if (len <= 0)
                    break;
                out.write(buf, 0, len); // out.flush();
            }
            fis.close();
            fis = null;
            // send '\0'
            buf[0] = 0;
            out.write(buf, 0, 1);
            out.flush();
            if (checkAck(in) != 0) {
                return;
            }
            out.close();

            channel.disconnect();
            session.disconnect();
            return;
        } catch (Exception e) {
            System.out.println(e);
            try {
                if (fis != null)
                    fis.close();
            } catch (Exception ee) {
            }
        }
    }

    private int checkAck(InputStream in) throws IOException {
        int b = in.read();
        // b may be 0 for success,
        // 1 for error,
        // 2 for fatal error,
        // -1
        if (b == 0)
            return b;
        if (b == -1)
            return b;

        if (b == 1 || b == 2) {
            StringBuffer sb = new StringBuffer();
            int c;
            do {
                c = in.read();
                sb.append((char) c);
            } while (c != '\n');
            if (b == 1) { // error
                System.out.print(sb.toString());
            }
            if (b == 2) { // fatal error
                System.out.print(sb.toString());
            }
        }
        return b;
    }

    private void delete(Session session, String remotePath) {
        try {
            // Remember that this is just for testing and we need a quick
            // access, you can add an identity and known_hosts file to prevent
            // Man In the Middle attacks
            Channel channel = session.openChannel("sftp");
            channel.connect();
            ChannelSftp sftp = (ChannelSftp) channel;
            // use the get method , if you are using android remember to remove
            // "file://" and use only the relative path
            sftp.rm(remotePath);
            Boolean success = true;
            if (success) {
                // The file has been DELETED succesfully
                System.out.println("ScpTo.remove(): SUCCESS");
            }
            channel.disconnect();
            session.disconnect();
        } catch (JSchException e) {
            System.out.println(e.getMessage().toString());
            e.printStackTrace();
        } catch (SftpException e) {
            System.out.println(e.getMessage().toString());
            e.printStackTrace();
        }
    }
}

 

 

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

 

 

Business Processes:

Following is the sample business process consisting of AWS S3 Client Service to perform GET, PUT and DELETE operations on a file, and create a directory on AWS S3  system..

<process name="IBMGPFSClientProcess">
  <sequence>
        <operation name="Extract File">
          <participant name='TestGPFSClientForB2Bi'/>
          <output message='xout'>
            <assign to='.' from='PrimaryDocument' />
          </output>
         <input message="xin">
           <assign to="." from="*"/>
         </input>
        </operation>
  </sequence>
</process>

 

Service Implementation IntegrationServices.xml.in

<?xml version="1.0" encoding="UTF-8"?>
<services>
 <service parentdefname="IBMGPFSClientForB2Bi" name="TestGPFSClientForB2Bi"
         displayname="TestGPFSClientForB2Bi" description="IBM GPFS Service for Sterling B2B Integrator"
         targetenv="all" activestatus="1" systemservice="0" parentdefid="-1"/>
</services>

 

 

Service Definitions – servicedefs/TestServices.xml

<?xml version="1.0" encoding="UTF-8"?>
<SERVICES>

 <SERVICE name="IBMGPFSClientForB2Bi" description="IBM GPFS integration with B2Bi" label="IBMGPFSClientForB2Bi" implementationType="CLASS" JNDIName="com.sterlingcommerce.woodstock.services.integration.IBMGPFSClientService" type="BASIC" adapterType="N/A" adapterClass="N/A" version="10.0" SystemService="NO">
  <VARS type="instance">
   <GROUP title="fs.wfd.group1.title" instructions="Instructions">
    <VARDEF varname="operation" type="String" htmlType="select" label="Action" options="actiontype" />
    <VARDEF varname="gpfs_system_ip_address" type="String" htmlType="text" label="GPFS System IP Address" maxsize="512" />
    <VARDEF varname="username" type="String" htmlType="text" label="GPFS System Username" maxsize="512" />
    <VARDEF varname="password" type="String" htmlType="password" label="GPFS System Password" maxsize="512" />
    <VARDEF varname="local_path" type="String" htmlType="text" label="Local Path" maxsize="512" />
    <VARDEF varname="remote_path" type="String" htmlType="text" label="Remote Path" maxsize="512" />
    <VARDEF varname="ssh_port" type="String" htmlType="text" label="SSH PORT" maxsize="512" />
    <VARDEF varname="StrictHostKeyChecking" type="String" htmlType="select" label="Strict Host Key Checking" options="StrictHostKeyCheckingtype" />
    <VARDEF varname="messageId" type="String" htmlType="text" label="Message Id (optional if Local Path provided)" maxsize="512" />
    <!--VARDEF varname="s3_fileName" type="String" htmlType="text" label="File Name" /-->
   </GROUP>
  </VARS>
 </SERVICE>

 <OPTION name="actiontype">
  <ELE value="put" displayname="PUT" />
  <ELE value="delete" displayname="DELETE" />
 </OPTION>

 <OPTION name="StrictHostKeyCheckingtype">
  <ELE value="yes" displayname="YES" />
  <ELE value="no" displayname="NO" />
 </OPTION>
</SERVICES>

 

Service Configuration in B2B UI

Installation of all dependant client jar – list shows all the awss3 client jars need to be in dynamicclasspath.cfg

Go to ./install/bin

./install3rdParty.sh jsch 0_1_54 -j /ais_local/share/bchandra/jsch-0.1.54.jar
 

 

Installing AWS S3 Client Service jar

Download the attached AWS S3 Client jar to test the sample example. Click this link to download -  integrationservices_1010000.jar

Now, go to ./install/bin

./InstallService.sh /home/bchandra/gpfs/clientservice/integrationservices_1010000.jar

 

Execute CRUD operation

Execute the above BPs to perform desired action. Any logging info goes to integrationservices.log under <INSTALL_DIR>/logs folder.

 

References:

http://www-03.ibm.com/systems/storage/spectrum/scale/

https://en.wikipedia.org/wiki/IBM_General_Parallel_File_System

http://www.jcraft.com/jsch/

 

 


#IBMSterlingB2BIntegratorandIBMSterlingFileGatewayDevelopers
#DataExchange
0 comments
11 views

Permalink