Hi Aidan! Thanks so much, I think I have everything setup correctly but when the pod restarts it is not getting the local ident field updated. If I login to the terminal and run the script manually from /usr/local/extra it works.
I did have to update the find command, the one from your example kept giving me a "find: missing argument to -exec" error. At first I thought it was a copy/paste issue but then I tried typing it directly and still have that error. I wrote the command like this and it is working if I run it manually:
[drouter@np-apic-gw-1 /]$ find /opt/ibm/datapower/drouter/config -type f -exec sed 's|REPLACE_LOCAL_IDENT|$HOSTNAME|g' -i {} \;
find: missing argument to `-exec'
[drouter@np-apic-gw-1 /]$ find /opt/ibm/datapower/drouter/config -type f -exec sed -i s/REPLACE_LOCAL_IDENT/$HOSTNAME/g {} \;
[drouter@np-apic-gw-1 /]$ cat /opt/ibm/datapower/drouter/config/datapower-operator-init.cfg | grep local
local-address "0.0.0.0" "5554"
local-address 0.0.0.0 5550
local-address 0.0.0.0
local-ident "np-apic-gw-1"
When I run the logs command you mentioned I do not see any evidence of the echo statement, so I'm guessing the script is not running. It definitely is in /etc/local/extra:
[drouter@np-apic-gw-1 /]$ cat /usr/local/extra/local-ident.sh
#!/bin/bash
echo "Replacing REPLACE_LOCAL_IDENT with $HOSTNAME in config"
find /opt/ibm/datapower/drouter/config -type f -exec sed -i s/REPLACE_LOCAL_IDENT/$HOSTNAME/g {} \;
Here is my
np-apic-gw-default-domain-local-ident.yaml: (note that I put a | in after the filename, but it gets changed to a >)
kind: ConfigMap
apiVersion: v1
metadata:
name: np-apic-gw-default-domain-local-ident
namespace: ibm-cp-pn
selfLink: >-
/api/v1/namespaces/ibm-cp-pn/configmaps/np-apic-gw-default-domain-local-ident
uid: c5626dd4-20b0-4c61-84ad-c8a2bd15afcc
resourceVersion: '404906941'
creationTimestamp: '2021-10-11T13:18:14Z'
managedFields:
- manager: Mozilla
operation: Update
apiVersion: v1
time: '2021-10-14T18:59:08Z'
fieldsType: FieldsV1
fieldsV1:
'f:data':
.: {}
'f:local-ident.sh': {}
data:
local-ident.sh: >
#!/bin/bash
echo "Replacing REPLACE_LOCAL_IDENT with $HOSTNAME in config"
find /opt/ibm/datapower/drouter/config -type f -exec sed -i
s/REPLACE_LOCAL_IDENT/$HOSTNAME/g {} \;
Here is the gatewayCluster yaml section:
additionalExtraExe:
- np-apic-gw-default-domain-local-ident
profile: n1xc4.m8additional
webGUIManagementPort: 9090
additionalDomainConfig:
- dpApp:
config:
- np-apic-gw-default-domain-custom
name: default
additionalInitCmds:
- /usr/local/extra/local-ident.sh
Could you please point me to which logs should I look into to see why the script is not firing?
------------------------------
Jennifer Stipe
------------------------------
Original Message:
Sent: Mon October 11, 2021 10:03 AM
From: Aidan Harbison
Subject: Getting API Connect v10 on Openshift logs into Splunk
Hi Jennifer,
Sorry, I should have clarified the GatewayCluster steps. In the GatewayCluster CR, the two properties would be:
- additionalInitCmds
- additionalExtraExe
These will pass through to the DataPowerService as initCmds and extraExe respectively. The syntax should be the same, only the names of the fields are different in the GatewayCluster CR.
Reference: https://www.ibm.com/docs/en/api-connect/10.0.x?topic=subsystem-customizing-datapower-deployment
You should not need to scale down to 0 to make this change.
------------------------------
Aidan Harbison
Original Message:
Sent: Mon October 11, 2021 09:49 AM
From: Jennifer Stipe
Subject: Getting API Connect v10 on Openshift logs into Splunk
Hi Aidan, thank you.
I am trying to apply your example, however when I update the yaml in the GatewayCluster with the extraExe and initCmds section it just disappears, so I assume I am trying to apply it to the incorrect object.
When I try to edit the yaml for the DatapowerService instance, it is changing so often that I cannot edit it successfully in the Openshift gui, it keeps telling me i need to reload it before applying my change. I tried the cli (oc apply -f) and get the same error:
Name: "np-apic-gw", Namespace: "ibm-cp-pn"
for: "datapowerservice-np-apic-gw.yaml": Operation cannot be fulfilled on datapowerservices.datapower.ibm.com "np-apic-gw": the object has been modified; please apply your changes to the latest version and try again
Do I need to take the stateful set down to 0 before applying this change?
------------------------------
Jennifer Stipe
Original Message:
Sent: Fri October 01, 2021 12:15 PM
From: Aidan Harbison
Subject: Getting API Connect v10 on Openshift logs into Splunk
Hi Jennifer,
There is not currently an out-of-the-box way to generate the logging target configs with a unique local-ident per pod; however, this is an enhancement we are looking into for future delivery.
As a stop-gap alternative in the meantime, you could could use a custom init script (extraExe & initCmds) that performed a text replacement of the cfg before the main DataPower process is started. For example, if in your ConfigMap which holds the syslog target cfg, you used a string such as REPLACE_LOCAL_IDENT
for the value of local-ident
, then you could perform a text substitution during initialization of the pod where this value is substituted with the value of an environment variable holding the pod name. The HOSTNAME
environment variable will automatically contain the pod's name, so this could be a viable option.
Here's an example ConfigMap for the script:
apiVersion: v1
kind: ConfigMap
metadata:
name: local-ident-sh
data:
local-ident.sh: |
#!/bin/bash
echo "Replacing REPLACE_LOCAL_IDENT with $HOSTNAME in config"
find /opt/ibm/datapower/drouter/config -type f -exec sed "s|REPLACE_LOCAL_IDENT|$HOSTNAME|g" -i {} \;
Here's a snippet for the DataPowerService spec additions to reference this ConfigMap and invoke it during pod initialization:
spec:
extraExe:
- local-ident-sh
initCmds:
- /usr/local/extra/local-ident.sh
As a proof of concept I tested this with some dummy cfg, where REPLACE_LOCAL_IDENT
existed in a comment block, in a DataPowerService named minimal
(thus the pod name in my test is minimal-0
). Here are some outputs showing the init log and hostname present in the "final" config:
$ oc logs minimal-0 | grep REPLACE_LOCAL_IDENT
Replacing REPLACE_LOCAL_IDENT with minimal-0 in config
$ oc exec -it minimal-0 -- grep -R 'minimal' /opt/ibm/datapower/drouter/config/
/opt/ibm/datapower/drouter/config/datapower-operator-init.cfg:# minimal-0
/opt/ibm/datapower/drouter/config/foo/datapower-operator-foo-init.cfg:# minimal-0
Note that the string REPLACE_LOCAL_IDENT
existed in both the default
domain and a domain named foo
, hence the two outputs above.
I hope this helps to demonstrate how you can customize the syslog logging target local-ident
during pod initialization with the pod's $HOSTNAME
.
------------------------------
Aidan Harbison
Original Message:
Sent: Fri October 01, 2021 09:49 AM
From: Jennifer Stipe
Subject: Getting API Connect v10 on Openshift logs into Splunk
Hi Aidan,
Thanks again for the help. I wanted to know if you have an idea on how I can accomplish this. I'd like to change the local-ident field to reference the dynamic pod name instead of a static value, so that each pod will be uniquely identified in the splunk logs.
Example of how I have it now (static):
local-ident "npapicv10"
I found this article about environment variables in Openshift (this is an old version, I can't seem to find the same for our current version 4.6):
https://docs.openshift.com/container-platform/3.11/dev_guide/environment_variables.html
and tried this command with our gateway pod:
oc set env pod/np-apic-gw-0 --list
However it doesn't seem to have any variable for the actual pod name. I looked at the existing config maps hoping this might have been done already in one of them but am not seeing it. Is it even possible?
I was thinking I could create a deployment policy to handle this also if it's not available as an openshift environment variable, but I'm not sure how Openshift Datapower gateways would handle a deployment policy.
------------------------------
Jennifer Stipe
Original Message:
Sent: Thu September 30, 2021 10:15 AM
From: Aidan Harbison
Subject: Getting API Connect v10 on Openshift logs into Splunk
Hello Jennifer,
Yes, that's expected. During DataPower pod initialization, the Operator processes all of the configuration mounted via ConfigMaps & Secrets and places the cfg in managed files (ex. datapower-operator-init.cfg) to ensure that at startup all config is in the correct place, executed in the right order.
The name of your file in the ConfigMap does not need to be auto-startup.cfg, rather it could be something more specific to the contents, such as splunk-logging-target.cfg. Then if you ever needed to find this in the aggregated config (datapower-operator-init.cfg) you should be able to search for this name (splunk-logging-target.cfg) as we annotate the cfg sections with comments of the file from which the cfg came from.
Happy you got it working, and I hope the above details help. Let me know if you have any other questions!
------------------------------
Aidan Harbison
Original Message:
Sent: Wed September 29, 2021 09:14 PM
From: Jennifer Stipe
Subject: Getting API Connect v10 on Openshift logs into Splunk
Thanks Aidan,
If I understand correctly, to add the customizations this way I would first create a new config map, let's call it np-apic-gw-apiconnect-domain-custom. I created that config map with the following definition:
kind: ConfigMap
apiVersion: v1
data:
auto-startup.cfg: |
logging target "Splunk_Logging_Target"
type syslog
priority normal
soap-version soap11
format text
timestamp syslog
no fixed-format
local-ident "npapicv10"
size 500
archive-mode rotate
upload-method ftp
rotate 3
no ansi-color
remote-address "{syslog server}" "{port}"
facility user
rate-limit 1000
connect-timeout 60
idle-timeout 15
active-timeout 0
no feedback-detection
no event-detection
suppression-period 10
event-filter 0x080e0018d
event-filter 0x80000001
event-filter 0x80e0018d
event-filter 0x80e0027c
ssl-client-type proxy
retry-interval 1
retry-attempts 1
long-retry-interval 20
precision second
event "all" "error"
event "latency" "info"
event "capacity" "info"
event "gatewayscript-user" "notice"
event "DebugLog" "info"
event "cert-monitor" "warn"
exit
Then in the gatewayCluster CR I added this under the spec section:
additionalDomainConfig:
- dpApp:
config:
- np-apic-gw-default-domain-custom
name: default
The pod restarted automatically and the Splunk logging target seems to be included, although it is not in auto-startup.cfg. It's in the datapower-operator-init.cfg. I'm not sure I did this correctly.
------------------------------
Jennifer Stipe
Original Message:
Sent: Wed September 29, 2021 12:01 PM
From: Aidan Harbison
Subject: Getting API Connect v10 on Openshift logs into Splunk
Hello Jennifer,
The syslogConfig spec in the GatewayCluster creates a syslog-tcp type logging target in the Gateway pods, so this is likely why you are not seeing any logs in Splunk (which is expecting UDP).
DataPower itself does support a syslog type logging target, which is UDP. You could in theory create a (UDP) syslog type logging target using a ConfigMap, and include that in the additionalDomainConfig spec in the GatewayCluster CR, to inject this logging target into the pods.
https://www.ibm.com/docs/en/datapower-gateways/10.0.x?topic=targets-types-log
https://www.ibm.com/docs/en/api-connect/10.0.x?topic=subsystem-customizing-datapower-deployment
Hope this helps.
------------------------------
Aidan Harbison
Original Message:
Sent: Wed September 29, 2021 10:15 AM
From: Jennifer Stipe
Subject: Getting API Connect v10 on Openshift logs into Splunk
Hi Mark,
For our on-premises Splunk we have a syslog server assigned to us (udp://server:port) and don't use http. This works fine when we deploy a log target to Datapower but I was hoping for v10 in Openshift we could do something a little more interesting than that.
I have tried the following:
Added the following snippet to our gatewayCluster:
syslogConfig:
enabled: true
remoteHost: {syslog host}
remotePort: {syslog port}
secretName: ''
This applied successfully but did not result in any logs in splunk.
I then tried these two articles for setting up cluster logging in Openshift:
https://docs.openshift.com/container-platform/4.6/logging/cluster-logging-deploying.html
https://docs.openshift.com/container-platform/4.6/logging/cluster-logging-external.html
I used this section of the second link: Forwarding logs using the syslog protocol
This worked, but too well. It forwarded logs for every application in the cluster even though I specified only the namespace where apic is installed using this in the yaml:
spec:
inputs:
- application:
namespaces:
- ibm-cp-pn
name: apic-logs
Also the logs did not seem to be from the gateway pod, so I am not sure this method will work for API Connect v10.
Here is an example of the logs, they all look similar to this:
Sep 29 09:57:06 10.156.7.208 Sep 29 13:57:06 dhcp-10-156-7-208 fluentd: docker:{"container_id"=>"111d9a018f047c1e53fa8c47f18dc0ab0110310f2842d392772df9cb0e2b5183"} kubernetes:{"container_name"=>"icp-mongodb", "namespace_name"=>"ibm-common-services", "pod_name"=>"icp-mongodb-2", "container_image"=>"quay.io/opencloudio/ibm-mongodb@sha256:2930a1ae42c7abf3849f5c7cdd45a64d94eafe406048152ab90bd618abc53bf0", "container_image_id"=>"quay.io/opencloudio/ibm-mongodb@sha256:2930a1ae42c7abf3849f5c7cdd45a64d94eafe406048152ab90bd618abc53bf0", "pod_id"=>"9d0f4167-38cd-486c-88ab-ff84ff51d7ae", "host"=>"dhcp-10-156-7-208", "master_url"=>"https://kubernetes.default.svc", "namespace_id"=>"2fbbe393-3af0-4e53-a21b-1e5e279fae83", "namespace_labels"=>{"managed-by-common-service-webhook"=>"true"}, :flat_labels=>["app=icp-mongodb", "controller-revision-hash=icp-mongodb-845974575b", "release=mongodb", "app_kubernetes_io/instance=common-mongodb", "statefulset_kubernetes_io/pod-name=icp-mongodb-2"]} message:2021-09-29T13:57:05.319+0000 I ACCESS [conn1230247]
I saved the yaml but deleted the clusterLogForwarder object so I wouldn't get yelled at for dumping 2 million log entries into splunk in just 1 hour. :)
I just want to reiterate that we want the actual Datapower gateway logs, the above links seem to be for APIC analytics which is fine but it's not getting the actual Datapower gateway logs right?
------------------------------
Jennifer Stipe
Original Message:
Sent: Mon September 06, 2021 05:35 AM
From: Mark Taylor
Subject: Getting API Connect v10 on Openshift logs into Splunk
Hi Jennifer,
Analytics offload as recommended by Stephanie is a good place to start. You will need to use the HTTP output plugin for Splunk. There is an example at the end of this topic: https://www.ibm.com/docs/en/api-connect/10.0.x?topic=myad-sample-filters-modifying-analytics-data
I hope that helps.
Mark S Taylor
IBM
------------------------------
Mark Taylor
Original Message:
Sent: Thu September 02, 2021 09:14 AM
From: Jennifer Stipe
Subject: Getting API Connect v10 on Openshift logs into Splunk
Hello!
We have API Connect v10 setup in Openshift and are working on migrating our v5 environment to v10. In our v5 environment we have log targets on the APIC Datapower gateways to send those logs to splunk. An example of the log that we are getting are latency logs and errors from the mpgw. These help the developers troubleshoot issues with their services. I'd like to understand how we setup something similar in Openshift? Do we need to apply this via the gateway extension?
------------------------------
Jennifer Stipe
------------------------------