API Connect

I found discussion around offloading to multiple targets and seems to be possible in v10, but wondering if anyone implemented offloading to different targets based on organization (some kind of filtering before offloading)? We would like to separate offloading based on organization.

It's possible but if you have multiple organisations (more than 2) we would recommend offloading to your own logstash server where you can then do whatever you want.

How many orgs do you have, how many offload targets?

I found discussion around offloading to multiple targets and seems to be possible in v10, but wondering if anyone implemented offloading to different targets based on organization (some kind of filtering before offloading)? We would like to separate offloading based on organization.

Thanks Chris for the reply, logstash may not be an option at this point. We have one org now and offloading to Splunk, trying onboard another org, just two organizations and both are running on different gateways. We would like to send these analytics logs to Splunk with different source type, so that we can avoid accessing data across organizations. I think you mentioned on another thread, you can have multiple offloading targets, but not sure how do we send logs based on the org to different targets.  

It's possible but if you have multiple organisations (more than 2) we would recommend offloading to your own logstash server where you can then do whatever you want.

How many orgs do you have, how many offload targets?

I found discussion around offloading to multiple targets and seems to be possible in v10, but wondering if anyone implemented offloading to different targets based on organization (some kind of filtering before offloading)? We would like to separate offloading based on organization.

If you want to send data to the same offload target but change some attributes then that's doable.

Have a look at the docs here: Customizing data with ingestion filters

Shout if need further help :)

Thanks Chris for the reply, logstash may not be an option at this point. We have one org now and offloading to Splunk, trying onboard another org, just two organizations and both are running on different gateways. We would like to send these analytics logs to Splunk with different source type, so that we can avoid accessing data across organizations. I think you mentioned on another thread, you can have multiple offloading targets, but not sure how do we send logs based on the org to different targets.  

It's possible but if you have multiple organisations (more than 2) we would recommend offloading to your own logstash server where you can then do whatever you want.

How many orgs do you have, how many offload targets?

I found discussion around offloading to multiple targets and seems to be possible in v10, but wondering if anyone implemented offloading to different targets based on organization (some kind of filtering before offloading)? We would like to separate offloading based on organization.

Hi Chris, documentation is very helpful, I think we can filter/drop out events based on the catalog name (see YAML below), but the problem is, though same log target, Splunk requires to send over different TCP port in order to recognize the source type, so as far as analytics offloading concerned, different log targets because of different ports. Is there a way to setup multiple offload targets with filters?

  external:
    offload:
      filter: |
        if  [catalog_name] =~ /org2/ {
      drop { }
    }

If you want to send data to the same offload target but change some attributes then that's doable.

Have a look at the docs here: Customizing data with ingestion filters

Shout if need further help :)

Thanks Chris for the reply, logstash may not be an option at this point. We have one org now and offloading to Splunk, trying onboard another org, just two organizations and both are running on different gateways. We would like to send these analytics logs to Splunk with different source type, so that we can avoid accessing data across organizations. I think you mentioned on another thread, you can have multiple offloading targets, but not sure how do we send logs based on the org to different targets.  

It's possible but if you have multiple organisations (more than 2) we would recommend offloading to your own logstash server where you can then do whatever you want.

How many orgs do you have, how many offload targets?

I found discussion around offloading to multiple targets and seems to be possible in v10, but wondering if anyone implemented offloading to different targets based on organization (some kind of filtering before offloading)? We would like to separate offloading based on organization.

Hiya

There is only a single offload pipeline, so any filters apply to everything in that pipeline and all data will be sent to all output targets of that pipeline (by default). To change that behaviour you need to do something more sophisticated (and untested by us) such as cloning the event payload so that different events can be sent to different offload targets.

e.g. something kind of like this. This is untested code for a scenario where you want to send all events to Noname with request and response bodies but want to offload them somewhere else as well without the body fields.

filter {    clone {       clones => ["not_noname"]      remove_field => [ "request_body", "response_body"]   }}   output {  if [type] == "not_noname" {     <first output plugin config>   } else {      <noname output plugin config>  }}

If you use filters to add a tag when it matches whatever your criteria is, you should be able to then switch based on that in the output section.

I should make clear I have not tested this - this it just what should logically work in logstash, I've not tried setting this configuration via our operator.

I would recommend raising an RFE for formal dual offload pipeline support as that would make doing this much easier.

Hi Chris, documentation is very helpful, I think we can filter/drop out events based on the catalog name (see YAML below), but the problem is, though same log target, Splunk requires to send over different TCP port in order to recognize the source type, so as far as analytics offloading concerned, different log targets because of different ports. Is there a way to setup multiple offload targets with filters?

  external:
    offload:
      filter: |
        if  [catalog_name] =~ /org2/ {
      drop { }
    }

If you want to send data to the same offload target but change some attributes then that's doable.

Have a look at the docs here: Customizing data with ingestion filters

Shout if need further help :)

Thanks Chris for the reply, logstash may not be an option at this point. We have one org now and offloading to Splunk, trying onboard another org, just two organizations and both are running on different gateways. We would like to send these analytics logs to Splunk with different source type, so that we can avoid accessing data across organizations. I think you mentioned on another thread, you can have multiple offloading targets, but not sure how do we send logs based on the org to different targets.  

It's possible but if you have multiple organisations (more than 2) we would recommend offloading to your own logstash server where you can then do whatever you want.

How many orgs do you have, how many offload targets?

I found discussion around offloading to multiple targets and seems to be possible in v10, but wondering if anyone implemented offloading to different targets based on organization (some kind of filtering before offloading)? We would like to separate offloading based on organization.

Thanks a lot Chris, that is brilliant idea cloning in filter and sending to multiple targets, totally understood this logic not validated, but you have given enough pointers to test it out. Right now this is nice to have feature for us, not mandatory yet. If this doesn't work, probably will go thru RFE route if needed. 

Thanks again for your help.

Hiya

There is only a single offload pipeline, so any filters apply to everything in that pipeline and all data will be sent to all output targets of that pipeline (by default). To change that behaviour you need to do something more sophisticated (and untested by us) such as cloning the event payload so that different events can be sent to different offload targets.

e.g. something kind of like this. This is untested code for a scenario where you want to send all events to Noname with request and response bodies but want to offload them somewhere else as well without the body fields.

filter {    clone {       clones => ["not_noname"]      remove_field => [ "request_body", "response_body"]   }}   output {  if [type] == "not_noname" {     <first output plugin config>   } else {      <noname output plugin config>  }}

If you use filters to add a tag when it matches whatever your criteria is, you should be able to then switch based on that in the output section.

I should make clear I have not tested this - this it just what should logically work in logstash, I've not tried setting this configuration via our operator.

I would recommend raising an RFE for formal dual offload pipeline support as that would make doing this much easier.

Hi Chris, documentation is very helpful, I think we can filter/drop out events based on the catalog name (see YAML below), but the problem is, though same log target, Splunk requires to send over different TCP port in order to recognize the source type, so as far as analytics offloading concerned, different log targets because of different ports. Is there a way to setup multiple offload targets with filters?

  external:
    offload:
      filter: |
        if  [catalog_name] =~ /org2/ {
      drop { }
    }

If you want to send data to the same offload target but change some attributes then that's doable.

Have a look at the docs here: Customizing data with ingestion filters

Shout if need further help :)

Thanks Chris for the reply, logstash may not be an option at this point. We have one org now and offloading to Splunk, trying onboard another org, just two organizations and both are running on different gateways. We would like to send these analytics logs to Splunk with different source type, so that we can avoid accessing data across organizations. I think you mentioned on another thread, you can have multiple offloading targets, but not sure how do we send logs based on the org to different targets.  

It's possible but if you have multiple organisations (more than 2) we would recommend offloading to your own logstash server where you can then do whatever you want.

How many orgs do you have, how many offload targets?

I found discussion around offloading to multiple targets and seems to be possible in v10, but wondering if anyone implemented offloading to different targets based on organization (some kind of filtering before offloading)? We would like to separate offloading based on organization.