API Connect

 View Only
  • 1.  Analytics offloading to different targets based on organization

    Posted 20 days ago

    I found discussion around offloading to multiple targets and seems to be possible in v10, but wondering if anyone implemented offloading to different targets based on organization (some kind of filtering before offloading)? We would like to separate offloading based on organization.



  • 2.  RE: Analytics offloading to different targets based on organization

    Posted 20 days ago

    It's possible but if you have multiple organisations (more than 2) we would recommend offloading to your own logstash server where you can then do whatever you want.

    How many orgs do you have, how many offload targets?



    ------------------------------
    Chris Dudley
    ------------------------------



  • 3.  RE: Analytics offloading to different targets based on organization

    Posted 20 days ago

    Thanks Chris for the reply, logstash may not be an option at this point. We have one org now and offloading to Splunk, trying onboard another org, just two organizations and both are running on different gateways. We would like to send these analytics logs to Splunk with different source type, so that we can avoid accessing data across organizations. I think you mentioned on another thread, you can have multiple offloading targets, but not sure how do we send logs based on the org to different targets.  




  • 4.  RE: Analytics offloading to different targets based on organization

    Posted 20 days ago

    If you want to send data to the same offload target but change some attributes then that's doable.

    Have a look at the docs here: Customizing data with ingestion filters

    Ibm remove preview
    Customizing data with ingestion filters
    Customize you API event records as they are received from the gateway.
    View this on Ibm >

    Shout if need further help :)



    ------------------------------
    Chris Dudley
    ------------------------------



  • 5.  RE: Analytics offloading to different targets based on organization

    Posted 20 days ago

    Hi Chris, documentation is very helpful, I think we can filter/drop out events based on the catalog name (see YAML below), but the problem is, though same log target, Splunk requires to send over different TCP port in order to recognize the source type, so as far as analytics offloading concerned, different log targets because of different ports. Is there a way to setup multiple offload targets with filters?

      external:
        offload:
          filter: |
            if  [catalog_name] =~ /org2/ {
          drop { }
        }




  • 6.  RE: Analytics offloading to different targets based on organization

    Posted 18 days ago

    Hiya

    There is only a single offload pipeline, so any filters apply to everything in that pipeline and all data will be sent to all output targets of that pipeline (by default). To change that behaviour you need to do something more sophisticated (and untested by us) such as cloning the event payload so that different events can be sent to different offload targets.

    e.g. something kind of like this. This is untested code for a scenario where you want to send all events to Noname with request and response bodies but want to offload them somewhere else as well without the body fields.

    filter { 
       clone { 
          clones => ["not_noname"]
          remove_field => [ "request_body", "response_body"]
       }
    }
       
    output {
      if [type] == "not_noname" {
         <first output plugin config> 
      } else { 
         <noname output plugin config>
      }
    }

    If you use filters to add a tag when it matches whatever your criteria is, you should be able to then switch based on that in the output section.

    I should make clear I have not tested this - this it just what should logically work in logstash, I've not tried setting this configuration via our operator.

    I would recommend raising an RFE for formal dual offload pipeline support as that would make doing this much easier.



    ------------------------------
    Chris Dudley
    ------------------------------



  • 7.  RE: Analytics offloading to different targets based on organization

    Posted 18 days ago

    Thanks a lot Chris, that is brilliant idea cloning in filter and sending to multiple targets, totally understood this logic not validated, but you have given enough pointers to test it out. Right now this is nice to have feature for us, not mandatory yet. If this doesn't work, probably will go thru RFE route if needed. 

    Thanks again for your help.




  • 8.  RE: Analytics offloading to different targets based on organization

    Posted 12 days ago

    There's already AHA idea for it: https://ideas.ibm.com/ideas/APICONN-I-826

    If this is what you need, please vote on it.



    ------------------------------
    Szymon Stupkiewicz
    ------------------------------