API Connect

 View Only
Expand all | Collapse all

Getting API Connect v10 on Openshift logs into Splunk

  • 1.  Getting API Connect v10 on Openshift logs into Splunk

    Posted Thu September 02, 2021 09:15 AM
    Hello!

    We have API Connect v10 setup in Openshift and are working on migrating our v5 environment to v10.  In our v5 environment we have log targets on the APIC Datapower gateways to send those logs to splunk.  An example of the log that we are getting are latency logs and errors from the mpgw.  These help the developers troubleshoot issues with their services.  I'd like to understand how we setup something similar in Openshift?  Do we need to apply this via the gateway extension?

    ------------------------------
    Jennifer Stipe
    ------------------------------


  • 2.  RE: Getting API Connect v10 on Openshift logs into Splunk

    IBM TechXchange Speaker
    Posted Fri September 03, 2021 11:55 AM
    Hi Jennifer, 
    Someone on the team recommended this resource:  https://www.ibm.com/docs/en/api-connect/10.0.x?topic=pyt-planning-offload-data-third-party-system and mentioned it was usually done similarly (analytics offload) as the stacks are similar. 

    If you decide to go directly to Splunk, you would use the gateway extension.

    ------------------------------
    Stephanie Wilkerson
    IBM
    ------------------------------



  • 3.  RE: Getting API Connect v10 on Openshift logs into Splunk

    Posted Mon September 06, 2021 05:35 AM
    Hi Jennifer,
    Analytics offload as recommended by Stephanie is a good place to start. You will need to use the HTTP output plugin for Splunk. There is an example at the end of this topic: https://www.ibm.com/docs/en/api-connect/10.0.x?topic=myad-sample-filters-modifying-analytics-data

    I hope that helps.

    Mark S Taylor
    IBM

    ------------------------------
    Mark Taylor
    ------------------------------



  • 4.  RE: Getting API Connect v10 on Openshift logs into Splunk

    Posted Wed September 29, 2021 10:16 AM
    Edited by Jennifer Stipe Wed September 29, 2021 10:18 AM
    Hi Mark,

    For our on-premises Splunk we have a syslog server assigned to us (udp://server:port) and don't use http.  This works fine when we deploy a log target to Datapower but I was hoping for v10 in Openshift we could do something a little more interesting than that.

    I have tried the following:

    Added the following snippet to our gatewayCluster:

      syslogConfig:
        enabledtrue
        remoteHost: {syslog host}
        remotePort: {syslog port}
        secretName''

    This applied successfully but did not result in any logs in splunk.

    I then tried these two articles for setting up cluster logging in Openshift:

    https://docs.openshift.com/container-platform/4.6/logging/cluster-logging-deploying.html
    https://docs.openshift.com/container-platform/4.6/logging/cluster-logging-external.html

    I used this section of the second link: Forwarding logs using the syslog protocol

    This worked, but too well.  It forwarded logs for every application in the cluster even though I specified only the namespace where apic is installed using this in the yaml:

    spec:
      inputs:
        - application:
            namespaces:
              - ibm-cp-pn
          nameapic-logs

    Also the logs did not seem to be from the gateway pod, so I am not sure this method will work for API Connect v10.

    Here is an example of the logs, they all look similar to this:

    Sep 29 09:57:06 10.156.7.208 Sep 29 13:57:06 dhcp-10-156-7-208 fluentd: docker:{"container_id"=>"111d9a018f047c1e53fa8c47f18dc0ab0110310f2842d392772df9cb0e2b5183"} kubernetes:{"container_name"=>"icp-mongodb", "namespace_name"=>"ibm-common-services", "pod_name"=>"icp-mongodb-2", "container_image"=>"quay.io/opencloudio/ibm-mongodb@sha256:2930a1ae42c7abf3849f5c7cdd45a64d94eafe406048152ab90bd618abc53bf0", "container_image_id"=>"quay.io/opencloudio/ibm-mongodb@sha256:2930a1ae42c7abf3849f5c7cdd45a64d94eafe406048152ab90bd618abc53bf0", "pod_id"=>"9d0f4167-38cd-486c-88ab-ff84ff51d7ae", "host"=>"dhcp-10-156-7-208", "master_url"=>"https://kubernetes.default.svc", "namespace_id"=>"2fbbe393-3af0-4e53-a21b-1e5e279fae83", "namespace_labels"=>{"managed-by-common-service-webhook"=>"true"}, :flat_labels=>["app=icp-mongodb", "controller-revision-hash=icp-mongodb-845974575b", "release=mongodb", "app_kubernetes_io/instance=common-mongodb", "statefulset_kubernetes_io/pod-name=icp-mongodb-2"]} message:2021-09-29T13:57:05.319+0000 I ACCESS [conn1230247]

    I saved the yaml but deleted the clusterLogForwarder object so I wouldn't get yelled at for dumping 2 million log entries into splunk in just 1 hour.  :)

    I just want to reiterate that we want the actual Datapower gateway logs, the above links seem to be for APIC analytics which is fine but it's not getting the actual Datapower gateway logs right?




    ------------------------------
    Jennifer Stipe
    ------------------------------



  • 5.  RE: Getting API Connect v10 on Openshift logs into Splunk

    Posted Wed September 29, 2021 11:27 AM
    Hello Jennifer,

    Apologies: It sounded like you were interested in sending the DataPower activity log to Splunk, in which case you would use the analytics offload process. However, I now see you are trying to use DataPower's syslog logging mechanism. Unfortunately, I can't help you too much further with that, but I will highlight your query and see if others might be able to chime in.

    Let's see if we can avoid you being yelled at!

    Mark

    ------------------------------
    Mark Taylor
    ------------------------------



  • 6.  RE: Getting API Connect v10 on Openshift logs into Splunk

    Posted Wed September 29, 2021 11:56 AM
    Hi Mark - 

    I started thinking about this more after looking in these files on the pod:

    /opt/ibm/datapower/drouter/config/auto-startup.cfg
    /opt/ibm/datapower/drouter/datapower-operator-init.cfg

    I attended a TechCon conference about Datapower in containers and referred back to the slides from that presentation.  I see that they mentioned having to update config maps to change Datapower configurations in Openshift, so I looked for config maps that had that datapower-operator-init.cfg content.  Sure enough I found this one:  np-apic-gw-default-domain-config 

    I went to one of our existing Datapower appliances with the functioning syslog and copied the logging target command from it.  Then I pasted that into the yaml for the np-apic-gw-default-domain-config config map.  I was a little puzzled that the pod didn't restart automatically but I went ahead and deleted it.

    When it started back up I starting seeing the gateway logs in splunk!

    They are formatted a little strangely, but it appears to be working otherwise. 


    Is this a proper way to do log targets for syslog in v10?


    ------------------------------
    Jennifer Stipe
    ------------------------------



  • 7.  RE: Getting API Connect v10 on Openshift logs into Splunk

    Posted Wed September 29, 2021 03:00 PM
    I fixed the weird splunk formatting, I had dashes in the local-ident server name, when I removed them the logs look normal again and actually display the hostname like it should.

    ------------------------------
    Jennifer Stipe
    ------------------------------



  • 8.  RE: Getting API Connect v10 on Openshift logs into Splunk

    Posted Wed September 29, 2021 12:01 PM
    Hello Jennifer,

    The syslogConfig spec in the GatewayCluster creates a syslog-tcp type logging target in the Gateway pods, so this is likely why you are not seeing any logs in Splunk (which is expecting UDP).

    DataPower itself does support a syslog type logging target, which is UDP. You could in theory create a (UDP) syslog type logging target using a ConfigMap, and include that in the additionalDomainConfig spec in the GatewayCluster CR, to inject this logging target into the pods.

    https://www.ibm.com/docs/en/datapower-gateways/10.0.x?topic=targets-types-log

    https://www.ibm.com/docs/en/api-connect/10.0.x?topic=subsystem-customizing-datapower-deployment

    Hope this helps.

    ------------------------------
    Aidan Harbison
    ------------------------------



  • 9.  RE: Getting API Connect v10 on Openshift logs into Splunk

    Posted Wed September 29, 2021 09:15 PM
    Thanks Aidan,

    If I understand correctly, to add the customizations this way I would first create a new config map, let's call it np-apic-gw-apiconnect-domain-custom.  I created that config map with the following definition:

    kindConfigMap
    apiVersionv1
    data:
      auto-startup.cfg: |
        logging target "Splunk_Logging_Target"
          type syslog
          priority normal
          soap-version soap11
          format text
          timestamp syslog
          no fixed-format 
          local-ident "npapicv10"
          size 500
          archive-mode rotate
          upload-method ftp
          rotate 3
          no ansi-color 
          remote-address "{syslog server}" "{port}"
          facility user
          rate-limit 1000
          connect-timeout 60
          idle-timeout 15
          active-timeout 0
          no feedback-detection 
          no event-detection 
          suppression-period 10
          event-filter 0x080e0018d
          event-filter 0x80000001
          event-filter 0x80e0018d
          event-filter 0x80e0027c
          ssl-client-type proxy
          retry-interval 1
          retry-attempts 1
          long-retry-interval 20
          precision second
          event "all" "error"
          event "latency" "info"
          event "capacity" "info"
          event "gatewayscript-user" "notice"
          event "DebugLog" "info"
          event "cert-monitor" "warn"
        exit
    Then in the gatewayCluster CR I added this under the spec section:

      additionalDomainConfig:
        - dpApp:
            config:
              - np-apic-gw-default-domain-custom
          namedefault

    The pod restarted automatically and the Splunk logging target seems to be included, although it is not in auto-startup.cfg.  It's in the datapower-operator-init.cfg.  I'm not sure I did this correctly.

    ------------------------------
    Jennifer Stipe
    ------------------------------



  • 10.  RE: Getting API Connect v10 on Openshift logs into Splunk

    Posted Thu September 30, 2021 10:15 AM

    Hello Jennifer,

    Yes, that's expected. During DataPower pod initialization, the Operator processes all of the configuration mounted via ConfigMaps & Secrets and places the cfg in managed files (ex. datapower-operator-init.cfg) to ensure that at startup all config is in the correct place, executed in the right order.

    The name of your file in the ConfigMap does not need to be auto-startup.cfg, rather it could be something more specific to the contents, such as splunk-logging-target.cfg. Then if you ever needed to find this in the aggregated config (datapower-operator-init.cfg) you should be able to search for this name (splunk-logging-target.cfg) as we annotate the cfg sections with comments of the file from which the cfg came from.

    Happy you got it working, and I hope the above details help. Let me know if you have any other questions!



    ------------------------------
    Aidan Harbison
    ------------------------------



  • 11.  RE: Getting API Connect v10 on Openshift logs into Splunk

    Posted Fri October 01, 2021 09:49 AM
    Hi Aidan, 

    Thanks again for the help.  I wanted to know if you have an idea on how I can accomplish this.  I'd like to change the local-ident field to reference the dynamic pod name instead of a static value, so that each pod will be uniquely identified in the splunk logs.

    Example of how I have it now (static):
    local-ident "npapicv10"


    I found this article about environment variables in Openshift (this is an old version, I can't seem to find the same for our current version 4.6):

    https://docs.openshift.com/container-platform/3.11/dev_guide/environment_variables.html

    and tried this command with our gateway pod:

    oc set env pod/np-apic-gw-0 --list

    However it doesn't seem to have any variable for the actual pod name.  I looked at the existing config maps hoping this might have been done already in one of them but am not seeing it.  Is it even possible?

    I was thinking I could create a deployment policy to handle this also if it's not available as an openshift environment variable, but I'm not sure how Openshift Datapower gateways would handle a deployment policy.


    ------------------------------
    Jennifer Stipe
    ------------------------------



  • 12.  RE: Getting API Connect v10 on Openshift logs into Splunk

    Posted Fri October 01, 2021 12:15 PM
    Edited by Aidan Harbison Fri October 01, 2021 12:28 PM
    Hi Jennifer,

    There is not currently an out-of-the-box way to generate the logging target configs with a unique local-ident per pod; however, this is an enhancement we are looking into for future delivery.

    As a stop-gap alternative in the meantime, you could could use a custom init script (extraExe & initCmds) that performed a text replacement of the cfg before the main DataPower process is started. For example, if in your ConfigMap which holds the syslog target cfg, you used a string such as REPLACE_LOCAL_IDENT for the value of local-ident, then you could perform a text substitution during initialization of the pod where this value is substituted with the value of an environment variable holding the pod name. The HOSTNAME environment variable will automatically contain the pod's name, so this could be a viable option.

    Here's an example ConfigMap for the script:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: local-ident-sh
    data:
      local-ident.sh: |
        #!/bin/bash

        echo "Replacing REPLACE_LOCAL_IDENT with $HOSTNAME in config"
        find /opt/ibm/datapower/drouter/config -type f -exec sed "s|REPLACE_LOCAL_IDENT|$HOSTNAME|g" -i {} \;​

    Here's a snippet for the DataPowerService spec additions to reference this ConfigMap and invoke it during pod initialization:

    spec:
      extraExe:
      - local-ident-sh
      initCmds:
      - /usr/local/extra/local-ident.sh​

    As a proof of concept I tested this with some dummy cfg, where REPLACE_LOCAL_IDENT existed in a comment block, in a DataPowerService named minimal (thus the pod name in my test is minimal-0). Here are some outputs showing the init log and hostname present in the "final" config:

    $ oc logs minimal-0 | grep REPLACE_LOCAL_IDENT
    Replacing REPLACE_LOCAL_IDENT with minimal-0 in config​

    $ oc exec -it minimal-0 -- grep -R 'minimal' /opt/ibm/datapower/drouter/config/
    /opt/ibm/datapower/drouter/config/datapower-operator-init.cfg:# minimal-0
    /opt/ibm/datapower/drouter/config/foo/datapower-operator-foo-init.cfg:# minimal-0

    Note that the string REPLACE_LOCAL_IDENT existed in both the default domain and a domain named foo, hence the two outputs above.

    I hope this helps to demonstrate how you can customize the syslog logging target local-ident during pod initialization with the pod's $HOSTNAME.

    ------------------------------
    Aidan Harbison
    ------------------------------



  • 13.  RE: Getting API Connect v10 on Openshift logs into Splunk

    Posted Mon October 11, 2021 09:50 AM
    Hi Aidan, thank you.

    I am trying to apply your example, however when I update the yaml in the GatewayCluster with the extraExe and initCmds section it just disappears, so I assume I am trying to apply it to the incorrect object.

    When I try to edit the yaml for the DatapowerService instance, it is changing so often that I cannot edit it successfully in the Openshift gui, it keeps telling me i need to reload it before applying my change.  I tried the cli (oc apply -f) and get the same error:

    Name: "np-apic-gw", Namespace: "ibm-cp-pn"
    for: "datapowerservice-np-apic-gw.yaml": Operation cannot be fulfilled on datapowerservices.datapower.ibm.com "np-apic-gw": the object has been modified; please apply your changes to the latest version and try again

    Do I need to take the stateful set down to 0 before applying this change?

    ------------------------------
    Jennifer Stipe
    ------------------------------



  • 14.  RE: Getting API Connect v10 on Openshift logs into Splunk

    Posted Mon October 11, 2021 10:04 AM
    Edited by Aidan Harbison Mon October 11, 2021 10:04 AM
    Hi Jennifer,

    Sorry, I should have clarified the GatewayCluster steps. In the GatewayCluster CR, the two properties would be:

    - additionalInitCmds
    - additionalExtraExe

    These will pass through to the DataPowerService as initCmds and extraExe respectively. The syntax should be the same, only the names of the fields are different in the GatewayCluster CR.

    Reference: https://www.ibm.com/docs/en/api-connect/10.0.x?topic=subsystem-customizing-datapower-deployment

    You should not need to scale down to 0 to make this change.

    ------------------------------
    Aidan Harbison
    ------------------------------



  • 15.  RE: Getting API Connect v10 on Openshift logs into Splunk

    Posted Thu October 14, 2021 03:04 PM
    Hi Aidan!  Thanks so much, I think I have everything setup correctly but when the pod restarts it is not getting the local ident field updated.  If I login to the terminal and run the script manually from /usr/local/extra it works.

    I did have to update the find command, the one from your example kept giving me a "find: missing argument to -exec" error.  At first I thought it was a copy/paste issue but then I tried typing it directly and still have that error.  I wrote the command like this and it is working if I run it manually:

    [drouter@np-apic-gw-1 /]$ find /opt/ibm/datapower/drouter/config -type f -exec sed 's|REPLACE_LOCAL_IDENT|$HOSTNAME|g' -i {} \;​
    find: missing argument to `-exec'

    [drouter@np-apic-gw-1 /]$ find /opt/ibm/datapower/drouter/config -type f -exec sed -i s/REPLACE_LOCAL_IDENT/$HOSTNAME/g {} \;
    [drouter@np-apic-gw-1 /]$ cat /opt/ibm/datapower/drouter/config/datapower-operator-init.cfg | grep local
    local-address "0.0.0.0" "5554"
    local-address 0.0.0.0 5550
    local-address 0.0.0.0
    local-ident "np-apic-gw-1"

    When I run the logs command you mentioned I do not see any evidence of the echo statement, so I'm guessing the script is not running.  It definitely is in /etc/local/extra:

    [drouter@np-apic-gw-1 /]$ cat /usr/local/extra/local-ident.sh
    #!/bin/bash

    echo "Replacing REPLACE_LOCAL_IDENT with $HOSTNAME in config"

    find /opt/ibm/datapower/drouter/config -type f -exec sed -i s/REPLACE_LOCAL_IDENT/$HOSTNAME/g {} \;

    Here is my np-apic-gw-default-domain-local-ident.yaml: (note that I put a | in after the filename, but it gets changed to a >)

    kind: ConfigMap
    apiVersion: v1
    metadata:
      name: np-apic-gw-default-domain-local-ident
      namespace: ibm-cp-pn
      selfLink: >-
        /api/v1/namespaces/ibm-cp-pn/configmaps/np-apic-gw-default-domain-local-ident
      uid: c5626dd4-20b0-4c61-84ad-c8a2bd15afcc
      resourceVersion: '404906941'
      creationTimestamp: '2021-10-11T13:18:14Z'
      managedFields:
        - manager: Mozilla
          operation: Update
          apiVersion: v1
          time: '2021-10-14T18:59:08Z'
          fieldsType: FieldsV1
          fieldsV1:
            'f:data':
              .: {}
              'f:local-ident.sh': {}
    data:
      local-ident.sh: >
        #!/bin/bash
    
        echo "Replacing REPLACE_LOCAL_IDENT with $HOSTNAME in config"
    
        find /opt/ibm/datapower/drouter/config -type f -exec sed -i
        s/REPLACE_LOCAL_IDENT/$HOSTNAME/g {} \;
    ​


    Here is the gatewayCluster yaml section:

      additionalExtraExe:
        - np-apic-gw-default-domain-local-ident
      profilen1xc4.m8additional
      webGUIManagementPort9090
      additionalDomainConfig:
        - dpApp:
            config:
              - np-apic-gw-default-domain-custom
          namedefault
      additionalInitCmds:
        - /usr/local/extra/local-ident.sh​

    Could you please point me to which logs should I look into to see why the script is not firing?

    ------------------------------
    Jennifer Stipe
    ------------------------------



  • 16.  RE: Getting API Connect v10 on Openshift logs into Splunk

    Posted Fri December 03, 2021 10:19 AM
    Edited by Jennifer Stipe Tue December 07, 2021 08:39 PM
    Hi Aidan,

    We recently upgraded to APIC 10.0.3 and the Datapower gateway is now IDG.10.0.3.0sr.  When I tried to apply this same log target, it isn't working as it did before.  I noticed on the datapower gateway that the log target is in an "external" state and when I click the object it says it needs to be applied.  I attached some screenshots:



    I tried saving the config but that did not change the status nor did it make the logging functional.  Is this method no longer viable in 10.0.3 or are there more steps I need to take to get this working in the new version?

    [edit]  I also noticed that in the filesystem on the appliance, the datapower-operator-init.cfg file has the syslog target in it, but the auto-startup.cfg does not.  I created a syslog target manually and it works fine, so it seems to be an issue with creating the syslog target via the configMap.  I also enabled the web-mgmt and created a user in the configMap and both of those worked. I'm going to keep experimenting and will update.

    ------------------------------
    Jennifer Stipe
    ------------------------------



  • 17.  RE: Getting API Connect v10 on Openshift logs into Splunk

    Posted Tue December 07, 2021 09:22 PM
    I got this working again, what I did was create a log target manually that worked on the appliance (in the openshift pod).  Then I compared what I had in my configMap with the contents of the working syslog target.  I updated my configMap to make them look the same, then I killed the gateway pod so it would restart.  The syslog target was recreated (still had the external status) but it's now working.  So there must have been something incompatible in my syslog target definition that worked in 10.0.1 but not 10.0.3. 

    The main differences I noted (maybe it will help others) were:

    - my old syslog target had "syslog" for the timestamp, the manually created one had "zulu" 
    - the rate limit on my old syslog target was 1000, the manually created one had 100
    - the old syslog target had these event filters:
    event "mgmt" "error"
    event "latency" "info"
    event "gatewayscript-user" "notice"
    event "cert-monitor" "warn"
    However in the manually created one I just specified event "all" "info".

    With those changes, it's now working after being created by the configMap.  YAY!  :)

    ------------------------------
    Jennifer Stipe
    ------------------------------



  • 18.  RE: Getting API Connect v10 on Openshift logs into Splunk

    Posted Wed December 08, 2021 12:22 PM
    Hi Jennifer,

    Sorry for the delay in response. I'm glad you were able to get it working. The configuration being marked as external is expected, as the source of the configuration is the ConfigMap(s). Thus any changes (which are to persist) to the configuration should be made at the source - in the ConfigMap, or any CI/CD that feeds the ConfigMap.

    ------------------------------
    Aidan Harbison
    ------------------------------



  • 19.  RE: Getting API Connect v10 on Openshift logs into Splunk

    Posted Tue October 05, 2021 10:14 AM
    Edited by Jessica Long Tue October 05, 2021 10:47 AM
    I also got problem while integrate in my site then this ibm help me : https://www.ibm.com/docs/en/datapower-gateways/10.0.x?topic=targets-types-reviews

    ------------------------------
    Methew Eskier
    ------------------------------