Introduction: IBM API Connect provides REST APIs to download Analytics data present in multiple pages. This blog helps user to download analytics data by using API Connect toolkit commands and scroll APIs in a shell script.
Architecture: IBM API Connect provides REST APIs to get the cursor of the API Analytics data first page and scroll through the pages of 500 records and save them in JSON format.
Steps to write shell script:
1. Get bearer token using https://<hostname>/api/token. This will be used in the shell script to execute APIC REST APIs.
curl -k -s -X POST -d '{"username": "<API Manager username>", "password": "<API Manager password>", "realm": "provider/default-idp-2", "client_id": "ClientId-API Manager", "client_secret": "CLIENT Secret – API Manager ", "grant_type": "password"}' -H 'Content-Type: application/json' -H 'Accept: application/json' https://<API Manager hostname>/api/token
2. Construct parameter to call first scroll
body="{\"size\":\"500\",\"scroll\":\"10m\"}"
"$url/analytics/$analytics_service/orgs/$porg/events/scroll?start=$start1&end=$end" --header "Authorization: Bearer $bearertoken" --header "accept: application/json" --header "accept-language: en" --header "content-type: application/json" --data $body
Save output into a file.
3. Construct parameter to call next scroll
body="{\"size\":\"500\",\"scroll\":\"10m\",$scroll1}"
Loop through multiple scrolls till you get empty Events[] list in the json output Events Array.
"$url/analytics/$analytics_service/orgs/$porg/events/scroll?start=$start1&end=$end" --header "Authorization: Bearer $bearertoken" --header "accept: application/json" --header "accept-language: en" --header "content-type: application/json" --data $body
4. If empty events list is received for the scroll, then stop the loop and exit the code.
Shell Script example:
#!/bin/bash
# Update following for parameters.
url=<API Mgr URL>
#Example : url="https://apimanager.com"
#analytics-service=<REPLACE_ANALYTICS-SERVICE>
analytics_service="analytics-service"
#start=<REPLACE_THIS_VALUE_With_Start_DateTime> # format 2025-06-14T00:01:01.935Z
#Example start="2025-06-14T00:01:01.935Z"
#Or read it from shell command as first argument – start date time
# Take start date time parameter from command line input of the shell script
start1="$1"
#end=<REPLACE_THIS_VALUE_With_End_DateTime> # format 2024-06-14T23:59:59.935Z
#Example end="2024-06-14T23:59:59.935Z"
#Read it from shell command as second argument end date time.
end="$2"
porg=<provider org name>
#Example porg="abc"
catalog="<CatalogName>"
# Example catalog="private"
# Get the bearer token using curl command given in the Step 1 of the document
#bearertoken=<provide bearertoken from step1>
bearertoken="ey<bearer token from step1 .........> "
accept_language="en"
#Body parameter for the first call of scroll API
body="{\"size\":\"500\",\"scroll\":\"10m\"}"
events="event"
echo $events
scroll_num=0
# Create a timestamp variable to uniquely create a new file for each scroll output
timestamp=$(date +%s)
mkdir $timestamp
# loop through until you get empty events list array
while [[ "$events" != *"[]"* ]]
do
if [ $scroll_num -eq "0" ]
then
scroll_num=`expr $scroll_num + 1`
# call the scroll API
curl -kv --request POST --url "$url/analytics/$analytics_service/orgs/$porg/events/scroll?start=$start1&end=$end" --header "Authorization: Bearer $bearertoken" --header "accept: application/json" --header "accept-language: en" --header "content-type: application/json" --data $body > $timestamp/${porg}_${catalog}_${start_date}_${end_date}_${scroll_num}.json
echo $scroll_num
else
#scroll=`cat output.txt | grep scroll_id`
# Get the scroll id from the output of the scroll API call
scroll=`cat $timestamp/${porg}_${catalog}_${start_date}_${end_date}_${scroll_num}.json | grep scroll_id`
scroll1=`echo $scroll | sed 's/,//g'`
scroll2=`echo $scroll1 | sed 's/' '//g'`
echo $scroll1
echo $scroll2
# call the scroll API first time onward with additional parameter with having scroll id provided in the output of the 1st scroll call.
body="{\"size\":\"500\",\"scroll\":\"10m\",$scroll1}"
scroll_num=`expr $scroll_num + 1`
# call the scroll API to get next set of scroll events (next page analytics events)
curl -kv --request POST --url "$url/analytics/$analytics_service/orgs/$porg/events/scroll?start=$start1&end=$end" --header "Authorization: Bearer $bearertoken" --header "accept: application/json" --header "accept-language: en" --header "content-type: application/json" --data "$body" > $timestamp/${porg}_${catalog}_${start_date}_${end_date}_${scroll_num}.json
events=`cat $timestamp/${porg}_${catalog}_${start_date}_${end_date}_${scroll_num}.json | grep event`
echo $events
fi
done
References:
1. API to get bearer token from API Connect
API Explorer
https://apic-api.apiconnect.ibmcloud.com/v10/#/documentation/authentication/auth_bearertoken
2. API to scroll through results in batches
API Explorer
https://apic-api.apiconnect.ibmcloud.com/v10/#/IBMAPIConnectAnalyticsAPI_200/operation/%2F{analytics-service}%2Forgs%2F{org}%2Fevents%2Fscroll/post