Managed File Transfer

 View Only

IBM Bluemix Cloud Object Storage Integration with IBM B2B Sterling Integrator

By Tanvi Kakodkar posted Mon February 24, 2020 01:49 AM

  
Originally updated on January 10, 2018 by BipinChandra

Written & Edited By Naveen Suryawanshi & Bipin Chandra


Introduction:

In today's world, multiple businesses need to communicate with each other to meet their needs. As the transactional communication between different applications grows, there is a proportional growth in the data that needs to be stored by each of them

There could be certain limitations/use cases with each application/enterprise which could force them to find an alternate plan for their data storage. For e.g.

a. Low scale enterprises not having enough bandwidth to accommodate for storing data over network.

b. Data associated with the actual transactional data which could be repetitive in nature and not accessed frequently.

c. Paying a lump sum amount each year for storage but actual data stored is lesser.

d. Big data analysis and many more.

 

IBM hosts a solution to address such problems via providing web services API Called IBM Cloud Object Storage (COS).

IBM Cloud Object Storage is easy to use object storage, with a simple web service interface to store and retrieve any amount  of data from anywhere on the web With Cloud Object Storage,you pay only for the storage you
actually use.

There is no minimum fee and no setup cost In Simple Words
“Flexible storage for today’s dynamic unstructured data needs”

This enables the companies to focus more on the scalability and performance of their application than the cost and other factors involved in storing data.

 

This step by step documentation provides a seamless integration of IBM Cloud Object Storage with IBM Sterling B2B Integrator.

Sterling Integrator is a business process-centric transaction engine which enables enterprises to server all of their B2B integration needs through a single, flexible B2B gateway. We make use of business processes and services to integrate with IBM COS which can be enhanced to perform all the COS supported operations, like – put files, get files, delete files, create a container or a directory and many more.

Assessing current technology can help companies understand basic functional needs of various IT initiatives, discover new capabilities they can already deliver. In many cases, existing B2B Gateway can extend itself to enable the success of other non-Ecommerce IT initiatives.

 

Benefits:

  • Secure, durable, highly-scalable cloud based storage solution.

  • Client has choice of Regional and  Cross Regional  Deployment.

  • Always on Avalaiblity with Built in security which protect against digital and physical breeches.

  • Store and retrieve any amount of data.

  • Various range of storage classes designed for different use cases.

  • Easy to use as it comes with an interactive web based UI console.

 

IBM Cloud Object Storage Client Service:

IBM COS Client Service from Sterling Integrator creates seamless connectivity to the cloud storage environment hosted by IBM Cloud . It is a scalable, reliable, and portable storage solution.

SI users can easily get, put and delete files and can perform many more operations.

 

IBM Cloud Object Storage -SI Integration Architecture Diagram:

 

 

 

 

 

We will explain how to use IBM storage with the Java API provided by S3 and OpenStack Swift.

The example shows you how to create a Container, list it’s content, create a folder into a Container, upload a file, give the file a public access and finally how to delete all this items.

 

Setting Up Your Project

> You will need to integrate the .JAR files from the archive into your project.The jar file will

> Create a user in IBM Bluemix  (https://console.bluemix.net/registration/)  if you don’t have one. Here you will get a “auth_url” , “userId” and "password" in encrypted fromat. You will need this credentials to connect to IBM Blumix cloud storage

 

Authenticate with IBm Blumix Cloud Object Storage.

Authenticate your requests against IBM COS by using OSClient

Programmatically set credentials – in this example I will use this method.

We will need to set following t properties in integrationservices.properties file under <INSTALL_DIR>/properties folder.

 

cos.userId=<Your_Blumemix_UserID or Username>

cos.password=<Your_API_key or password>

cos.auth_url=<Authentication Service_URL>

cos.project=<Tenant name or Project>

cos.domainName=<Region or Domain name>

 

Before authentication neeed to indentify domain and project suing Indentifire

               Identifier domainIdent = Identifier.byName("YourDomainName");
               Identifier projectIdent = Identifier.byName("YouProjectName");

 


Create IBM Object Storage Client

To be able to communicate withIBm COS, you have to use an implementation of OSClient. You will use the instance to address requests to the server

Use following in your code

OSClient osClient = OSFactory.builderV3().endpoint("AuthenticationServiceURL") .credentials("YourBluemixUserId", "YourBluemixPassword").scopeToProject(projectIdent, domainIdent) .authenticate();

 


Create Container

Container must have unique names within the whole Cloud Object Storage realm.

String containerName= "javatutorial-net-example-Container";
ActionResponse ac=osClient.objectStorage().containers().create("containerName");

 


List Container

You can get the list off all Container like this.

List<? extends SwiftContainer> containers = osClient.objectStorage().containers().list();

            for(int i=0;i<containers.size();i++){
              System.out.println(" - " +containers.get(i));
            }

 

 

Create Folder in IBM Cloud Storage Container

Use this code to create an empty folder in your bucket

public void createFolderIntoContainer(OSClient osClient, String container_name,
            String folderName) {

        osClient.objectStorage().containers().createPath(container_name, folderName);
    }

 

Upload file in IBM Cloud Storage Container
If you want to upload a file use below one

InputStream docAsStream = new FileInputStream( new File("C:\\Users\\user\\Desktop\\testvideo.mp4")) ));
 putFileToCOS(docAsStream, file_Name, osClient, container_name);

//Put file into Container

public void putFileToCOS(InputStream source, String file_Name, OSClient osClient, String container_name)

{

String success = osClient.objectStorage().objects().put(container_name, file_Name, Payloads.create(source));
        if(success!=null){
            System.out.println("File Uploaded succesfully");}
    }

 


Deleting Files

To Delete files use below

 

osClient.objectStorage().objects().delete(container_name,file_Name);


Complete Java implementation:

This is how we create a java service for IBM Cloud Object storage Client. Customer can use this as a default implementation and can build their requirement on top of it.

/*
 *
 * IBM and Sterling Commerce Confidential
 *
 * OCO Source Materials
 *
 * IBM B2B Sterling Integrator
 *
 * (c) Copyright Sterling Commerce, an IBM Company 2001, 2011
 *
 * The source code for this program is not published or otherwise
 * divested of its trade secrets, irrespective of what has been
 * deposited with the U.S. Copyright Office.
 *
 */
package com.sterlingcommerce.fg.services;


import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.sql.SQLException;
import java.util.List;

import org.openstack4j.api.OSClient;
import org.openstack4j.model.common.ActionResponse;
import org.openstack4j.model.common.DLPayload;
import org.openstack4j.model.common.Identifier;
import org.openstack4j.model.common.Payloads;
import org.openstack4j.model.storage.object.SwiftContainer;
import org.openstack4j.model.storage.object.SwiftObject;
import org.openstack4j.openstack.OSFactory;

import com.sterlingcommerce.woodstock.mailbox.MailboxException;
import com.sterlingcommerce.woodstock.mailbox.MailboxFactory;
import com.sterlingcommerce.woodstock.mailbox.repository.IRepository;
import com.sterlingcommerce.woodstock.mailbox.repository.Message;
import com.sterlingcommerce.woodstock.services.IService;
import com.sterlingcommerce.woodstock.util.frame.Manager;
import com.sterlingcommerce.woodstock.util.frame.log.LogService;

import com.sterlingcommerce.woodstock.util.frame.log.Logger;
import com.sterlingcommerce.woodstock.workflow.Document;
import com.sterlingcommerce.woodstock.workflow.DocumentNotFoundException;
import com.sterlingcommerce.woodstock.workflow.WorkFlowContext;
import com.sterlingcommerce.woodstock.workflow.WorkFlowException;

import jarjar.orgapachecommonsio.IOUtils;

 

/**
 * The Reroute Service is started as part of a business process.
 *
 * It takes no input.
 *
 * @author Bipin Chandra
 */
public class IBMCloudObjectStorageService implements IService {

    private static final String CLASS_NAME = "IBMCloudObjectStorageService";

    private Document primaryDoc = null;

    private InputStream docAsStream = null;

    private static boolean isOK = true;

    protected static Logger log;
    static {
        log = LogService.getLogger("integrationservices");
    }
    /**
     * This method is entered when a Service is invoked during a business
     * process.
     * <P>
     *
     * @return Updated WorkFlowContext after Service processes
     */
    public WorkFlowContext processData(final WorkFlowContext wfc)
            throws WorkFlowException {
        final String METHOD = ".processData() - ";
        if (log.isDebug())
            log.log(CLASS_NAME + METHOD + "Starting ...");

        isOK = true;

        //Hard
        /*String auth_url= "https://identity.open.softlayer.com"+ "/v3";
        String project= "object_storage_92180e99_fd78_438b_860f_d8f9dee3b164";
        String userId="5732f0bf39254486a3e8a67991ac4ea7";
        String password= "z-_SYiu0}T7.(2SZ";
        String domainName="1365701";*/

        //OR USE PROPERTIES FILE
        String userId = Manager.getProperties("integrationservices").getProperty("cos.userId");
        String password = Manager.getProperties("integrationservices").getProperty("cos.password");
        String auth_url = Manager.getProperties("integrationservices").getProperty("cos.auth_url");
        String project = Manager.getProperties("integrationservices").getProperty("cos.project");
        String domainName = Manager.getProperties("integrationservices").getProperty("cos.domainName");

        try {
            wfc.harnessRegister();
            String action = wfc.getParm("operation");//PUT, GET, DELETE, MKDIR
            String local_path = wfc.getParm("local_path");//Path to read file from for uploading
            String messageId = wfc.getParm("messageId");//Message ID for mailbox
            String container_name = (String)wfc.getWFDContent("container_name");

            String file_Name = wfc.getParm("file_Name");  //Name with which resource resides in Cloud object Storage
            String folderName = wfc.getParm("foldername");//Folder name OR structure

            Identifier domainIdent = Identifier.byName(domainName);
            Identifier projectIdent = Identifier.byName(project);

            OSClient osClient = OSFactory.builderV3().endpoint(auth_url)
                    .credentials(userId, password)
                    .scopeToProject(projectIdent, domainIdent)
                    .authenticate();

            log.log("ObjectStorage Client is created!! Connection established!!");

            // create container - name must be unique for all S3 users
            if(container_name!=null){
                log.log("container_name"+container_name);
                System.out.println("container_name"+container_name);
                try{
                    ActionResponse ac=osClient.objectStorage().containers().create(container_name);
                }catch(Exception e){
                    e.printStackTrace();
                    System.out.println("Container trace"+e.getMessage());
                }

            }
            // list containers
            List<? extends SwiftContainer> containers = osClient.objectStorage().containers().list();

            for(int i=0;i<containers.size();i++){
                log.log("-"+containers.get(i));
            }

            //Operations on files
            if ("put".equalsIgnoreCase(action))
            {
                if ( wfc.getPrimaryDocument() != null ){// put file into COS
                    docAsStream = wfc.getPrimaryDocument().getBodyInputStream();
                    putFileToCOS(docAsStream, file_Name, osClient, container_name);
                }
                else if (local_path != null && !"".equalsIgnoreCase(local_path)) {// put file from FileSystem to COS
                    docAsStream = new FileInputStream( new File( local_path ));
                    putFileToCOS(docAsStream, file_Name, osClient, container_name);
                } else if (messageId != null && !"".equalsIgnoreCase(messageId) && !"null".equalsIgnoreCase(messageId)) {// put file from mailbox to COS
                    putFileFromMailboxToCOS(messageId, file_Name, osClient, container_name);
                }

            }
            else if ("get".equalsIgnoreCase(action)) {// read files from COS
                log.log("Usage: s3client read <s3_key>");
                readFileFromCOS(file_Name, osClient, container_name);
                wfc.putPrimaryDocument(primaryDoc);

            } else if ("delete".equalsIgnoreCase(action)) {// Delete files from COS
                log.log("Usage: s3client delete <s3_key>");
                deleteFileFromCOS(file_Name, osClient, container_name);

            } else if ("mkdir".equalsIgnoreCase(action)) {// create folder into Container of COS
                log.log("Usage: s3client mkdir <s3_key>");
                createFolderIntoContainer(osClient, container_name,folderName);

            } else {
                log.log("Usage: s3client put/read/delete/mkdir" + " [<local_path> <s3_key>]");
            }

            log.log("Done!");
        } catch (Exception e) {
            isOK = false;
            e.printStackTrace();
            System.out.println(e);
            log.log(e.getMessage());
            log.logException(e);
            wfc.setBasicStatus(1);
            throw new WorkFlowException(e.toString());
        } finally {
            wfc.unregisterThread();
            if (isOK) {
                wfc.setBasicStatusSuccess();
            } else {
                wfc.setBasicStatusError();
            }
        }
        return wfc;

    }
    public void createFolderIntoContainer(OSClient osClient, String container_name,
            String folderName) {

        osClient.objectStorage().containers().createPath(container_name, "Myfolder");
    }
    public void deleteFileFromCOS(String file_Name, OSClient osClient, String container_name) {
        System.out.println("===========File Delete============");
        osClient.objectStorage().objects().delete(container_name,file_Name);
        System.out.println("file deleted succesfully");


    }
    public void readFileFromCOS(String file_Name, OSClient osClient, String container_name) throws IOException {

        SwiftObject file = osClient.objectStorage().objects().get(container_name, file_Name);
        DLPayload downloadLPayload=file.download();
        InputStream in=downloadLPayload.getInputStream();

        OutputStream out;

        //String filename = file_Name.substring(file_Name.lastIndexOf('/') + 1, file_Name.length());
        //out = new BufferedOutputStream(new FileOutputStream(new File("C:\\NAVEEN\\newfile.txt1")));
        out = new BufferedOutputStream(new FileOutputStream(new File(file_Name)));

        byte[] b = new byte[1024];
        int numBytes = 0;
        while ((numBytes = in.read(b)) > 0) {
            out.write(b, 0, numBytes);
        }
        in.close();
        out.close();
        primaryDoc = new Document();
        primaryDoc.setBody(b);
    }

    public void putFileFromMailboxToCOS(String messageId, String file_Name, OSClient osClient, String container_name) throws IOException {
        IRepository repos = MailboxFactory.getRepository();
        String filename = null;
        Message message = null;
        Document batchDoc = null;
        try {
            log.log("CloudObjectStorageClient.putFileFromMailboxToS3():messageId:" + messageId);
            message = repos.getMessage(Long.parseLong(messageId));// source here is messageid
            log.log("CloudObjectStorageClient.putFileFromMailboxToS3(): message:" + message.toString());
            String docId = message.getDocumentId();
            log.log("CloudObjectStorageClient.putFileFromMailboxToS3(): docId:" + docId);
            filename = message.getMessageName();
            log.log("CloudObjectStorageClient.putFileFromMailboxToS3():filename:" + filename);
            if (docId != null) {
                batchDoc = new Document(docId, true);
            }
            log.log("CloudObjectStorageClient.putFileFromMailboxToS3(): batchDoc:" + batchDoc);
        } catch (NumberFormatException nfe) {
            log.logException(nfe);
        } catch (MailboxException me) {
            log.logException(me);
        } catch (DocumentNotFoundException dnfe) {
            log.logException(dnfe);
        } catch (SQLException sqle) {
            log.logException(sqle);
        }

        InputStream in = batchDoc.getInputStream();
        final File aFile = File.createTempFile("tmp", "txt");
        aFile.deleteOnExit();
        try{
            FileOutputStream out = new FileOutputStream(aFile);
            IOUtils.copy(in, out);
        } catch (IOException ioe) {
            log.logException(ioe);
        }
        System.out.println("===========File Upload============");
        String success = osClient.objectStorage().objects().put(container_name, file_Name, Payloads.create(aFile));
        if(success!=null){
            System.out.println("File Uploaded succesfully");}

    }
    public void putFileToCOS(InputStream source, String file_Name, OSClient osClient, String container_name) {
        System.out.println("===========File Upload============");
        String success = osClient.objectStorage().objects().put(container_name, file_Name, Payloads.create(source));
        if(success!=null){
            System.out.println("File Uploaded succesfully");}
    }
}

 

 

 

Business Processes:

Following is the sample business process consisting of AWS S3 Client Service to perform GET, PUT and DELETE operations on a file, and create a directory on AWS S3  system..

 

Service Implementation – serviceinstances.xml

<?xml version="1.0" encoding="UTF-8"?>
<services>
 <service parentdefname="IBMCosClientForB2Bi" name="TestClientForB2Bi"
                  displayname="TestClientForB2Bi"

                  description="IBM cloud Storage Service for Sterling B2B Integrator"
                   targetenv="all" activestatus="1"

                   systemservice="0" parentdefid="-1"/>
</services>

 


Service Definitions – servicedefs/TestServices.xml

<?l version="1.0" encoding="UTF-8"?>
<SERVICES>

 <SERVICE name="AWSS3ClientForB2Bi" description="AWS Simple Storage Solution with B2Bi" label="AWSS3ClientForB2Bi" implementationType="CLASS" JNDIName="com.sterlingcommerce.woodstock.services.integration.AWSS3ClientService" type="BASIC" adapterType="N/A" adapterClass="N/A" version="10.0" SystemService="NO">
  <VARS type="instance">
   <GROUP title="fs.wfd.group1.title" instructions="Instructions">
    <VARDEF varname="local_path" type="String" htmlType="text" label="Local Path" maxsize="512" />
    <VARDEF varname="messageId" type="String" htmlType="text" label="Message Id" maxsize="512" />
    <VARDEF varname="conatiner_name" type="String" htmlType="text" label="IBM Cloud Object Storage Container Name" maxsize="512" />
    <VARDEF varname="file_name" type="String" htmlType="text" label="File Name" maxsize="512" />
    <VARDEF varname="foldername" type="String" htmlType="text" label="Folder Name" maxsize="512" />
   </GROUP>
  </VARS>
 </SERVICE>

 <OPTION name="actiontype">
  <ELE value="put" displayname="PUT" />
  <ELE value="delete" displayname="DELETE" />
  <ELE value="get" displayname="GET" />
  <ELE value="mkdir" displayname="MKDIR" />
 </OPTION>
</SERVICES>

 


Service Configuration in B2B UI

Installation of all dependant client jar – list shows all the IBm Cloud Storage client jars need to be in dynamicclasspath.cfg

Go to ./install/bin

./install3rdParty.sh CloudJar 1_0 -j /ais_local/share/naveens/CloudJar/jar/commons-codec-1.6.jar
./install3rdParty.sh CloudJar 1_0 -j /ais_local/share/naveens/CloudJar/jar/commons-logging-1.1.3.jar
./install3rdParty.sh CloudJar 1_0 -j /ais_local/share/naveens/CloudJar/jar/guava-17.0.jar
./install3rdParty.sh CloudJar 1_0 -j /ais_local/share/naveens/CloudJar/jar/httpclient-4.3.6.jar
./install3rdParty.sh CloudJar 1_0 -j /ais_local/share/naveens/CloudJar/jar/httpcore-4.3.3.jar
./install3rdParty.sh CloudJar 1_0 -j /ais_local/share/naveens/CloudJar/jar/jackson-annotations-2.4.0.jar
./install3rdParty.sh CloudJar 1_0 -j /ais_local/share/naveens/CloudJar/jar/jackson-core-2.4.1.1.jar
./install3rdParty.sh CloudJar 1_0 -j /ais_local/share/naveens/CloudJar/jar/jackson-databind-2.4.1.3.jar
./install3rdParty.sh CloudJar 1_0 -j /ais_local/share/naveens/CloudJar/jar/openstack4j-core-2.0.2.jar
./install3rdParty.sh CloudJar 1_0 -j /ais_local/share/naveens/CloudJar/jar/openstack4j-httpclient-2.0.2.jar

 

 

Installing IBMCloud Storage Client Service jar

Download the attached  Client jar to test the sample example. Click this link to download - integrationservices_1010000.jar

Now go to ./install/bin

./InstallService.sh /home/naveens/IBMCos/client/integrationservices_1010000.jar

 

Execute CRUD operation.

Execute the above BPs to perform desired action. Any logging info goes to integrationservices.log under <INSTALL_DIR>/logs folder.

 

References:

https://developer.ibm.com/recipes/tutorials/connecting-to-ibm-object-storage-for-bluemix-with-java/

 


#IBMSterlingB2BIntegratorandIBMSterlingFileGatewayDevelopers
#DataExchange
0 comments
11 views

Permalink