Increase the time out setting for Cloud Pak for Data API call

Introduction

Cloud Pak for Data has default time out settings for the APIs. These settings work well under the normal situations.  While, there could be API call time out issues caused by the slow performance of network, data sources or other environment issues. These time out issues should be resolved from the root cause perspective. While,  sometimes it may take long time to resolve the performance issues. That's why the workarounds for addressing the time out issues maybe required from the short term solution perspective.

This article will showcase the workarounds for addressing the API call time out issues. The example user scenario is as follows. 

Customer would like to run metadata discovery for the data in their Hive database. Because of Cloud Pak for Data and the Hive database are located in different network subsets, there is low network performance which causes frequent time out issue when making data connection to the Hive database. By increasing the API call time out setting, it can help to resolve this problem as a workaround.

Step-by-step guide

1.Identify the zen-extension to be edited

1)Go into the ibm-ngix pod

oc rsh ibm-nginx-xxx

2)Go to the nginx folder

cd /nginx_data/extensions/locations

3)Identify the zen-extension to be edited using grep

Grep the zen-extension using the keyword of the API call.

For example, we can infer the “project” as the keyword from below time out error message.

2024/10/22 17:23:54 [error] 4621#4621: *2067731 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.56.2.9, server: localhost, request: "POST /projects/api/connections-v2?test=true&project_id=bae7b0e2-637d-44c4-971a-ab8dfdfd4689 HTTP/1.1", upstream: "https://10.52.20.12:443/projects/api/connections-v2?test=true&project_id=bae7b0e2-637d-44c4-971a-ab8dfdfd4689", host: "cpd-app00236429-wkc-cpd-instance.apps.arop02.xxx.net", referrer: "https://cpd-app00236429-wkc-cpd-instance.apps.arop02.xxx.net/connections/new/e59b1c36-6f30-4879-9f74-7e81dde4cca6?project_id=bae7b0e2-637d-44c4-971a-ab8dfdfd4689&context=icp4data&tearsheet_mode=true"

Use the keyword “project” for searching the zen-extension.

grep project ./*
./cpd_wkc-base-zen-frontdoor-extension_ie_159.conf:location /transactional/v2/projects {
./cpd_wkc-base-zen-frontdoor-extension_ie_159.conf:  rewrite (.*) /projects$1  break;

From the result, we can tell ws-base-zen-frontdoor-extension is the zen-extension we should edit.

2.Put the relevant custom resource into maintenance mode

As the ws-base-zen-frontdoor-extension zen-extension belongs to CCS, we need to put CCS into maintenance mode for preventing it from being reverted back during the reconciliation.

oc patch -n ${PROJECT_CPD_INST_OPERANDS} ccs ccs-cr --type merge --patch '{"spec": {"ignoreForMaintenance": true}}'

3. Add the section to the location part

1) Have a backup of the zen-extension ws-base-zen-frontdoor-extension

oc get zenextension wkc-base-zen-frontdoor-extension -n ${PROJECT_CPD_INST_OPERANDS} -o yaml > ws-base-zen-frontdoor-extension.yaml

2)Edit the zen-extension ws-base-zen-frontdoor-extension

oc edit zenextension wkc-base-zen-frontdoor-extension -n ${PROJECT_CPD_INST_OPERANDS}

Update and increase the time out settings to the location corresponding to the API call accordingly.

proxy_read_timeout 600s;
proxy_send_timeout 600s;

The edition looks like below.

Save the changes and wait the zenextension become the ‘Completed’ status.

oc get zenextension wkc-base-zen-frontdoor-extension -n ${PROJECT_CPD_INST_OPERANDS}

NAME                              STATUS      AGE
wkc-base-zen-frontdoor-extension   Completed   19d

4.Restart nginx pods for picking up the changes

oc get pods -n ${PROJECT_CPD_INST_OPERANDS}| grep ibm-nginx

#Restart all the ibm-nginx pods one by one
oc delete pod ibm-nginx-57dc9bd685-94gd7 -n ${PROJECT_CPD_INST_OPERANDS}

Make sure there's at least one ibm-nginx pod up and running when you delete other ibm-nginx pods 

5.Validate the changes applied

Follow similar approaches in Step 1 and check whether the time out setting in the location of the cpd_ws-base-zen-frontdoor-extension_ie_159.conf is with the changes made in previous steps.