webMethods

 View Only
Expand all | Collapse all

CloudStreams S3 Connector - Prefix Issue

  • 1.  CloudStreams S3 Connector - Prefix Issue

    Posted Mon August 07, 2023 03:51 PM

    SoftwareAG Designer – version 10.15

    I’m trying to list files in a particular folder in my S3 bucket using CloudStreams S3 connector but its not working if there is “/” in prefix parameter (key name).
    Here if I use only “POC” in prefix, or no prefix at all, it works fine and listing files. But as soon as I get into subfolder and add a “/” in prefix, I get this error.
    I tried with a “\” (backslash) and “%2F” in place of slash, that way I get a 200(OK) response, but no file list in body. So I’m sure it has to do something with “/”, but no idea what.
    Any Suggestion?

    Error: The request signature we calculated does not match the signature you provided. Check your key and signing method.


    #CloudStreams
    #webMethods
    #Integration-Server-and-ESB


  • 2.  RE: CloudStreams S3 Connector - Prefix Issue

    Posted Fri August 11, 2023 01:01 PM

    Do the objects have to be created with a leading slash? In a traditional file system, the leading slash represents the root directory but in S3 it has no special meaning. If you could avoid creating the object with a leading slash, that would likely be your best bet. As you can see from the examples in this page, object keys don’t typically start with a leading slash: Organizing objects using prefixes - Amazon Simple Storage Service. In fact, when I tried to create an object using CloudStreams with a leading slash in the key, the leading slash was removed (either by the connector or S3) when I looked up the object in S3.

    Hope this helps,
    Percio


    #Integration-Server-and-ESB
    #webMethods
    #CloudStreams


  • 3.  RE: CloudStreams S3 Connector - Prefix Issue

    Posted Mon August 21, 2023 07:33 AM

    You are right in S3 slash isn’t necessary. But here I am making same use of slash as in other file systems. There will be parent folder, further could be client wise folders, business segments, etc.


    #Integration-Server-and-ESB
    #CloudStreams
    #webMethods


  • 4.  RE: CloudStreams S3 Connector - Prefix Issue

    Posted Wed August 23, 2023 07:01 AM

    You can use operation getBucket to get all the objects in the bucket. For those folders in the bucket, the content will have the variable key and size. Folder object usually ending with “/” and of size=0. Just iterate through the content and get what you want.

    Eg:
    key= ‘innerfolder/’
    key= ‘innerfolder/dummy.txt’

    I hope this answers your question.


    #Integration-Server-and-ESB
    #webMethods
    #CloudStreams