Hi Vishal!
Im experiencing similar issues with Universal Cloud REST API integration that used to be working properly and have been interrupted in the last 2 months.
Wondered if you can help me with that issue:
I have used the Community Developed XML workflow and provided the needed parameters for the integration:
XML Workflow:
<?xml version="1.0" encoding="UTF-8" ?>
<Workflow name="Ariel" version="1.0" xmlns="http://qradar.ibm.com/UniversalCloudRESTAPI/Workflow/V1">
<Parameters>
<Parameter name="host" label="Host" required="true" />
<Parameter name="path" label="Path" required="true" />
<Parameter name="username" label="Username" required="true" />
<Parameter name="password" label="Password" required="true" />
</Parameters>
<Actions>
<!--
/////////////////////
// Post the Search //
/////////////////////
-->
<!-- Initialize the Bookmark -->
<Initialize path="/bookmark" value="51" />
<CallEndpoint url="https://${/host}/${/path}/logs.index" method="GET" savePath="/logs_index">
<SSLConfiguration allowUntrustedServerCertificate="true" />
<BasicAuthentication username="${/username}" password="${/password}" />
<RequestHeader name="Accept" value="*/*" />
</CallEndpoint>
<!-- Handle Errors -->
<If condition="/logs_index/status_code != 200">
<Abort reason="${/logs_index/status_message}: ${/logs_index/body/message}" />
</If>
<!-- Extract the Search -->
<Set path="/index" value="${/logs_index/body}" />
<Split value="${/logs_index/body}" delimiter="\n" savePath="/log_files" />
<ForEach item="/log_file" items="/log_files">
<RegexCapture pattern="[0-9]+_([0-9]+)\.log" value="${/log_file}" savePath="/current_file" />
<If condition="/current_file > /bookmark" >
<!-- Update Bookmark -->
<CallEndpoint url="https://${/host}/${/path}/${/log_file}" method="GET" savePath="/log_content">
<SSLConfiguration allowUntrustedServerCertificate="true" />
<BasicAuthentication username="${/username}" password="${/password}" />
<RequestHeader name="Accept" value="*/*" />
</CallEndpoint>
<If condition="/log_content/status_code != 200">
<Abort reason="${/log_content/status_message}: ${/log_content/body/message}" />
</If>
<Set path="/log_events" value="${/log_content/body}" />
<Split value="${/log_events}" delimiter="\n" savePath="/events" />
<Set path="/recordcount" value="${count(/events)}" />
<If condition="/recordcount > 1" >
<ForEach item="/current_event" items="/events">
<RegexCapture pattern="(LEEF):1.0\|Incapsula" value="${/current_event}" savePath="/event_header" />
<If condition="/event_header = 'LEEF'" >
<PostEvent path="/current_event" source="${/username}" />
</If>
</ForEach>
</If>
<RegexCapture pattern="[0-9]+_([0-9]+)\.log" value="${/log_file}" savePath="/bookmark" />
</If>
<!-- Handle Errors -->
</ForEach>
</Actions>
<Tests>
<DNSResolutionTest host="${/host}" />
<TCPConnectionTest host="${/host}" />
<HTTPConnectionThroughProxyTest url="https://${/host}" />
</Tests>
</Workflow>
XML Workflow Parametrs:
<?xml version="1.0" encoding="UTF-8" ?>
<WorkflowParameterValues xmlns="http://qradar.ibm.com/UniversalCloudRESTAPI/Work
flowParameterValues/V1">
<Value name="host" value="logs1.incapsula.com" />
<Value name="path" value="1111_2222222" />
<Value name="username" value="123456" />
<Value name="password" value="000aa0a0-1111-22b2-cc33-dd44d55555d5" />
</WorkflowParameterValues>
First, we used Postman in order to validate that there are logs present under the designated folder in the cloud.
After we saw a lot of logs, we moved further to the QRadar Test.
First the "Start Test" was not fetching logs so we tried modifying certain values in the script like the bookmark
value, which led us to successfully receiving the logs, only on the Test mentioned but not on the Log Activity Tab.
When we perform a "Start Test" in the Log Source Management Interface, we were able seeing that logs are present and indeed being fetched as needed (with the desired format (LEEF)), but according to their epoch timestamp, the logs are from 2 days ago (2 days gap) and not winding up in the Log Activity Tab (Therefore, the logs are probably not arriving to the associated Collector).
Also, i saw a README.md file in the Community Developed that suggests that Log Compression must be off if you
already produce logs from before - you can adjust the value of the bookmark to bigger than 1.
According to the component venodr (Imperva Incapsula) seems like a python script is needed (on the associated collector) that will download the logs.
But as far as i know, there is a functionality in the XML Workflow that is able fetching logs actively from the Incapsula Cloud (Correct me if im wrong).
I have already tried consulting the QRadar Support, apparently there is no support for the Community Developed Scripts.
Also, seems like the identifier of the log source being extracted from the XML Workflow in this line:
<PostEvent path="/current_event" source="${/username}" />
I would highly appreciate your help since this issue persist for too long.
Best Regards, and Thank you in advance.
Nehoray Kanizo.
------------------------------
Nehoray Kanizo
------------------------------
Original Message:
Sent: Mon January 27, 2025 01:28 AM
From: Vishal Tangadkar
Subject: Universal Cloud REST API - troubleshooting help
Hi Raymond,
The workflow test seems to be working fine, so the workflow itself must be good. Events are displaying correctly, but it appears they aren't being tagged properly to the log source.
Could you confirm the value set for ${/host} in the workflow parameters? Ensure the same value is used in the log source identifier for consistency.
Best regards,
------------------------------
Vishal Tangadkar
IBM INDIA PVT LTD
Original Message:
Sent: Fri January 24, 2025 11:52 AM
From: Raymond Tam
Subject: Universal Cloud REST API - troubleshooting help
I am working on a custom Universal Cloud REST API log source. I created the workflow.xml and workflow parameter values.xml. I tested the Log Source from the GUI without error, and I see records returned. I also tried the /opt/qradar/bin/test-workflow.sh command with no errors and I see JSON records in the output. The Log Source status is "OK" but it never downloaded any logs. I did the payload search and confirmed the logs are not in Qradar
Can anyone help review the Workflow xml if I am missing anything?
<?xml version="1.0" encoding="UTF-8"?>
<Workflow name="Sailpoint" version="1.0" xmlns="http://qradar.ibm.com/UniversalCloudRESTAPI/Workflow/V1">
<Parameters>
<Parameter name="host" label="Host" required="true" />
<Parameter name="client_id" label="Client ID" required="true" />
<Parameter name="client_secret" label="Client Secret" required="true" secret="true" />
</Parameters>
<Actions>
<!--
//////////////////////////
// Request Access Token //
//////////////////////////
-->
<CallEndpoint url="https://${/host}/oauth/token" method="POST" savePath="/get_access_token">
<UrlEncodedFormRequestBody>
<Parameter name="grant_type" value="client_credentials" />
<Parameter name="client_id" value="${/client_id}" />
<Parameter name="client_secret" value="${/client_secret}" />
</UrlEncodedFormRequestBody>
</CallEndpoint>
<!-- Handle Errors -->
<If condition="/get_access_token/status_code != 200">
<Abort reason="${/get_access_token/body}" />
</If>
<!-- Extract the Access Token -->
<Set path="/access_token" value="${/get_access_token/body/access_token}" />
<!--
///////////////////////////////////////
// Initialize bookmark default value //
///////////////////////////////////////
-->
<Initialize path="/bookmark" value="2025-01-23T00:00:00Z" />
<!--
////////////////////
// Request Events //
////////////////////
-->
<CallEndpoint url="https://${/host}/v3/search" method="POST" savePath="/search">
<QueryParameter name="limit" value="5000" />
<RequestHeader name="Content-Type" value="application/json" />
<RequestHeader name="Accept" value="application/json" />
<RequestHeader name="Authorization" value="Bearer ${/access_token}" />
<RequestBody type="application/json" encoding="UTF-8">
<![CDATA[
{
"query": {
"query": "created:[${/bookmark} TO now]"
},
"indices": [
"events"
],
"includeNested": true,
"sort": [
"-created"
]
}
]]>
</RequestBody>
</CallEndpoint>
<!-- Handle Errors -->
<If condition="/search/status_code != 200">
<Abort reason="${/search/body/error_description}" />
</If>
<!-- Post the Events -->
<PostEvents path="/search/body" source="${/host}" />
<!-- Update the Bookmark -->
<If condition="/search/body != null">
<Set path="/bookmark" value="${/search/body[0]/created}" />
</If>
</Actions>
<Tests>
<DNSResolutionTest host="${/host}" />
<TCPConnectionTest host="${/host}" />
<HTTPConnectionThroughProxyTest url="https://${/host}" />
</Tests>
</Workflow>
------------------------------
Raymond Tam
------------------------------