IBM QRadar

February 2024 Update

QRadar App Development Tools

There are two tools provided to assist with app development - the App Framework SDK and the App Editor. Let's look at these tools in more detail.

Firstly a note on versions. These examples use:

1. SDK v2.0.2 (published on the App Exchange Jan 18th 2021)
2. App Editor v3.0.0 (published on the App Exchange April 1st 2021 )

The above versions of the SDK and App Editor are intended to work with a QRadar instance running v2 of the application framework which supports a container image based on Red Hat UBI and Python 3. That is, a QRadar instance at one of these versions or later:

• 7.3.3 Fix Pack 6 or later
• 7.4.1 Fix Pack 2 or later
• 7.4.2 GA or later
• 7.4.3 GA or later

For the examples in this blog I'm using a 7.4.3 GA system (with the exception of the Future Enhancements section).

The App Editor is installed on your QRadar instance as an app. It's an app for creating and editing other apps. You can use it to upload app zip files or clone from sample apps on GitHub. It then provides an embedded editor as a tab within the QRadar UI. It's great for when you are just getting started with QRadar app development as you can use it to get an app up and running quickly.

The SDK installs on your local development machine with support for packaging and deploying apps to a remote QRadar instance. The intention is for the SDK's qapp commands to be used alongside your favourite IDE (e.g Eclipse, VS Code, IntelliJ). The embedded editor that comes as part of the App Editor has useful features but is slightly limited when compared with fully-fledged IDEs so the natural progression is to explore app samples with the App Editor then move on to using the SDK and an IDE when you're familiar with the basics.

Let's demonstrate the use of these tools by applying a commonly required change - adding a log statement. For this I'll use a sample app that demonstrates the use of a REST method for populating a new dashboard widget. This sample is available on GitHub.

Applying a change using the App Editor

On my QRadar system, I've installed the App Editor via Extension Management. It's launched using Develop Applications on the Admin tab:

I've configured a proxy to allow the QRadar console to access github.com to populate the list of samples. Next we choose the REST method sample and click Install:

This sample's app manifest contains rest_methods and dashboard_items entries:

"rest_methods": [
{
"name": "getArielDBList",
"url": "/getArielDBList",
"method": "GET",
"argument_names": []
}
],
"dashboard_items": [
{
"text": "Ariel DB names",
"description": "Dashboard item that populates a dropdown using REST method",
"rest_method": "getArielDBList",
"required_capabilities": ["ADMIN"]
}
]

The rest method is assigned to URL /getArielDBList. It's implemented as Python function get_ariel_databases() in views.py and it returns data to populate the dashboard item. We can open views.py in the Development tab for the app:

get_ariel_databases() already uses the qpylib.log function in several places. Let's add this new log statement at the start of get_ariel_databases():

 qpylib.log('Calling REST method to fetch ariel DB names.',level='info')

Apply and save this change using the embedded editor. Next we add the dashboard item in the UI:

And here is the dashboard item populated with data:

Now each time /getArielDBList is called we see the new message written to the app.log along with the messages that came with the sample originally:

tail -f /store/docker/volumes/qapp-1061/log/app.log

2021-08-27 13:03:12,091 [Thread-7] [INFO] [APP_ID:1061] [NOT:0000006000] Calling REST method to fetch ariel DB names.
2021-08-27 13:03:12,107 [Thread-7] [INFO] [APP_ID:1061] [NOT:0000006000] Ariel DB name: events
2021-08-27 13:03:12,107 [Thread-7] [INFO] [APP_ID:1061] [NOT:0000006000] Ariel DB name: flows
2021-08-27 13:03:12,108 [Thread-7] [INFO] [APP_ID:1061] [NOT:0000006000] Ariel DB name: simarc
2021-08-27 13:03:12,108 [Thread-7] [INFO] [APP_ID:1061] [NOT:0000006000] Ariel DB name: statistics

In this case we changed the Python code in views.py which was picked up automatically by Flask. That happens because Flask is running in debug mode (this is enabled by default for any apps created via the App Editor). If we want to make a change to the UI components or services configured in the app's manifest.json we need to do an app upgrade so that other QRadar system services detect the change. The app upgrade can be done using Actions -> Deploy App in the embedded editor.

To see this and more App Editor functionality in action, there is a helpful demo video available here.

Applying a change using the SDK

The workflow for adding a new log statement to the app using the SDK is very similar.

To follow this example these development tools need to be installed on the local machine:

• Git
• Docker Desktop v3.2.0 or later (this version resolved a problem with handling the base image .xz file)
• Python 3.6.8 or later
• QRadar SDK v2.0.0 or later

First, git clone the sample apps repo and cd into the RESTMethod directory. Next create a qenv.ini file so that the local Docker container created by the SDK can reach our remote QRadar instance. This is the format of a qenv.ini file:

[qradar]
QRADAR_CONSOLE_FQDN=< your fully qualified hostname>
QRADAR_CONSOLE_IP=<your QRadar IP>

[app]
SEC_ADMIN_TOKEN=< your SEC token>
QRADAR_APP_UUID=

Now build the app Docker image to run in a local container:

> qapp build
Found base image internal.docker.instance/gaf/qradar-app-base:2.0.4
Preparing image build directory /Users/mmaclynn/qradarappsdk/docker/build
Using /Users/mmaclynn/qradarappsdk/docker/build/Dockerfile
Creating Supervisor program entry for Flask
Building image [restmethod]
Using user ID 501 and group ID 20
DOCKER BUILD LOG: START
Step 1/14 : FROM internal.docker.instance/gaf/qradar-app-base:2.0.4
Step 2/14 : LABEL com.ibm.si.app.origin=SDK
Step 3/14 : ARG APP_USER_ID
Step 4/14 : ARG APP_GROUP_ID
Step 5/14 : ARG APP_USER_NAME=appuser
Step 6/14 : ARG APP_GROUP_NAME=appuser
Step 7/14 : ENV APP_ROOT /opt/app-root
Step 8/14 : ENV APP_USER_ID $APP_USER_ID
Step 9/14 : ENV APP_GROUP_ID $APP_GROUP_ID
Step 10/14 : ENV PATH $APP_ROOT/bin:$PATH
Step 11/14 : COPY / $APP_ROOT
Step 12/14 : RUN groupadd -o -g $APP_GROUP_ID $APP_GROUP_NAME && useradd -l -u $APP_USER_ID -g $APP_GROUP_ID $APP_USER_NAME && mkdir -p /etc/supervisord.d && if [ -f $APP_ROOT/init/supervisord.conf ]; then mv $APP_ROOT/init/supervisord.conf /etc; fi && rm -rf $APP_ROOT/init/* && if [ -d $APP_ROOT/bin ]; then chmod -R 755 $APP_ROOT/bin; fi && if [ -d $APP_ROOT/container/build ]; then chmod -R 755 $APP_ROOT/container/build; fi && if [ -d $APP_ROOT/container/run ]; then chmod -R 755 $APP_ROOT/container/run; fi && if [ -d $APP_ROOT/container/clean ]; then chmod -R 755 $APP_ROOT/container/clean; fi && if [ -d $APP_ROOT/container/service ]; then chmod -R 755 $APP_ROOT/container/service; fi && if [ -d $APP_ROOT/startup.d ]; then chmod -R 755 $APP_ROOT/startup.d; fi && if [ -d $APP_ROOT/container/conf/supervisord.d ]; then mv $APP_ROOT/container/conf/supervisord.d/*.conf /etc/supervisord.d; fi && if [ -d /etc/supervisord.d ]; then chmod -R 755 /etc/supervisord.d ; fi && echo -e "appuser ALL=(ALL) NOPASSWD:ALL\n" >> /etc/sudoers && visudo -cf /etc/sudoers
Step 13/14 : USER $APP_USER_NAME
Step 14/14 : ENTRYPOINT ["sh", "/opt/app-root/bin/start.sh"]
Successfully built 68ec99be1af7
Successfully tagged restmethod:latest
DOCKER BUILD LOG: END
Image [restmethod] build completed successfully

Run the app container specifying the -d to run Flask in development mode so that our changes to views.py will be automatically detected:

> qapp run -d
Starting container [qradar-restmethod] using image [restmethod]
Mounting /Users/mmaclynn/workspace_github_sample_apps/qradar-sample-apps/RESTMethod/manifest.json to /opt/app-root/manifest.json
Mounting /Users/mmaclynn/workspace_github_sample_apps/qradar-sample-apps/RESTMethod/app to /opt/app-root/app
Mounting /Users/mmaclynn/workspace_github_sample_apps/qradar-sample-apps/RESTMethod/store to /opt/app-root/store
Setting FLASK_ENV=development
Reading environment variables from /Users/mmaclynn/workspace_github_sample_apps/qradar-sample-apps/RESTMethod/qenv.ini
Setting QRADAR_CONSOLE_FQDN=<my QRadar hostname>
Setting QRADAR_CONSOLE_IP=<my QRadar IP>
Setting QRADAR_REST_PROXY=socks5h://host.docker.internal:1080
Setting SEC_ADMIN_TOKEN=<my SEC token>
Ignoring empty variable QRADAR_APP_UUID
Container [qradar-restmethod] started with ID 1429583e2a
Host port 65211 is mapped to container port 5000/tcp
Flask is running in development mode
Access Flask endpoints at http://localhost:65211
Use docker ps to monitor container status
Log files are located in /Users/mmaclynn/workspace_github_sample_apps/qradar-sample-apps/RESTMethod/store/log

Curl to URL http://localhost:65211/getArielDBList. The response is JSON containing HTML that's formatted to populate a dashboard widget - a <select> and a list of <option> elements:

curl http://localhost:65211/getArielDBList | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 983 100 983 0 0 1394 0 --:--:-- --:--:-- --:--:-- 1392
{
"id": "ArielDBs",
"title": "Ariel DB names",
"HTML": "<!--\nCopyright 2020 IBM Corporation\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n-->\n<!DOCTYPE html>\n<html>\n <body>\n <select id=\"arielDatabases\">\n \n <option value=\"events\">events</option>\n \n <option value=\"flows\">flows</option>\n \n <option value=\"simarc\">simarc</option>\n \n <option value=\"statistics\">statistics</option>\n \n </select>\n </body>\n</html>"
}

At this stage we're ready to add the new log message. Edit views.py to add this line to get_ariel_databases():

 qpylib.log('Calling REST method to fetch ariel DB names.',level='info')

Now if we curl to the app endpoint again, the new log message "Calling REST method to fetch ariel DB names." is written to app.log locally:

> tail -f store/log/app.log
2021-09-14 15:32:55,062 [Thread-4] [INFO] [APP_ID:0] [NOT:0000006000] Calling REST method to fetch ariel DB names.
2021-09-14 15:32:55,720 [Thread-4] [INFO] [APP_ID:0] [NOT:0000006000] Ariel DB name: events
2021-09-14 15:32:55,722 [Thread-4] [INFO] [APP_ID:0] [NOT:0000006000] Ariel DB name: flows
2021-09-14 15:32:55,724 [Thread-4] [INFO] [APP_ID:0] [NOT:0000006000] Ariel DB name: simarc
2021-09-14 15:32:55,726 [Thread-4] [INFO] [APP_ID:0] [NOT:0000006000] Ariel DB name: statistics

At this point we're happy enough with the code change that we want to install the app on our remote QRadar. To do this, first package the app as a zip:

> qapp package -p ../rest.zip
Adding file: README.md
Adding file: manifest.json
Adding directory: app
Adding directory: app/templates
Adding file: app/__init__.py
Adding file: app/views.py
Adding file: app/templates/ariel.html
Created package ../rest.zip
WARNING: image "qradar-app-base:2.0.0" in manifest differs from SDK image "qradar-app-base:2.0.4"

And then deploy to QRadar:

> qapp deploy -u admin -p ../rest.zip -q <QRadar console IP>
Please enter password for user admin:
Application fresh install detected
Uploading ../rest.zip 3628 bytes
Installing application 1102
Application 1102: CREATING
Application 1102: CREATING
Application 1102: CREATING
Application 1102: CREATING
Application 1102: CREATING
Application 1102: CREATING
Application 1102: CREATING
Application 1102: CREATING
Application 1102: CREATING
Application 1102: CREATING
Application 1102: CREATING
Application 1102: COMPLETED
Final application state: RUNNING 1.0.3

From this point to test further changes we run qapp package then qapp deploy and refresh the QRadar UI in the browser.

It's worth mentioning here that the SDK also supports connecting to your QRadar instance via a proxy. There's a prompt to configure a SOCKS proxy that is shown the first time the qapp deploy command is executed for a particular QRadar IP and the settings are then saved for future executions of the command:

Do you use a SOCKS proxy to connect to the server? [y/N]:



If you answer yes to this you can then input the SOCKS version, server details and port. Here is an example of the first execution of qapp deploy to a given IP that illustrates this (I choose the default proxy setting for each):

> qapp deploy -u admin -p ../rest.zip -q <QRadar console IP>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
No CA certificate bundle found for <QRadar console IP>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
To enable verification of server certificates, the CA certificate bundle must be downloaded from the server
Do you wish to proceed with the CA certificate bundle download? [Y/n]: Y
Please answer the following questions detailing how to connect to the server
Do you use a SOCKS proxy to connect to the server? [y/N]: y
Enter SOCKS protocol version (4, 5) [5]:
Enter SOCKS proxy server [localhost]:
Enter SOCKS proxy port [1080]:
Enter user ID for connecting to the server [root]:
Enter <QRadar console IP> password for user root:
Initialising transfer of CA certificate bundle from server, please wait...
100%|################################################################################################################################| 226k/226k [00:01<00:00, 117kb/s]
Transfer complete
CA certificate bundle for <QRadar console IP> saved to /Users/mmaclynn/.qradar_app_sdk/<QRadar console IP>/ca-bundle.crt
Please enter password for user admin:

< more output follows as per previous qapp deploy example ... >

Conclusion

Both the App Editor and the SDK support you with incrementally improving your app and deploying your changes to a QRadar instance for testing. In both cases if you are only making small changes to your Python code or HTML you can see them applied quickly with File -> Save in the Editor and executing a qapp run with the SDK.

For more substantial changes, like adding new object types to the manifest.json, in the Editor we use Actions -> Deploy App and with the SDK we use qapp package and qapp deploy. There's an extra step to package with the SDK there but it runs a in a few seconds. It would be straightforward to write a shell script to combine these into one step. There is also a planned feature for the the SDK where an app can be running in a local Docker container, but is registered with a remote QRadar instance, so that we can log into the QRadar UI and see the content being served by the local Docker container. The Future Enhancements section below has more information on this.

Choosing which tool to use comes down to whether you prefer working at the command line or within the QRadar UI and whether company policy permits you to use Python 3 and Docker on your local machine. After becoming familiar with QRadar apps, most developers will likely prefer a fully-featured IDE of their own choosing rather than the embedded editor. Both tools will support you with your QRadar app development as you progress from learning the basics to writing more complex apps.

Future Enhancements

Following on from the earlier mention of planned features for the SDK I'll now preview the upcoming support for development apps. This allows me to run an app container locally, register it as development app on my QRadar instance, and have QRadar request app content from the local container instead of a container running on the server. This is great for testing app changes in the context of the full QRadar UI. Please note that it is not recommended to use development apps in a production QRadar environment.

For this preview I'm using a development build both of QRadar and of the SDK and the GUIActions sample app. This sample sets up several toolbar buttons and right click menu items that, when clicked, produce an alert box showing the row(s) that are selected. I've modified the sample slightly:

list_function() in views.py now hard codes a message to be displayed in the alert box:

@viewsbp.route('/listFunction')
def list_function():
qpylib.log("ListFunction", "debug") 
rows = {"ids":["Message one"]}
return rows

custom_script.js has been changed so that there's some text and square brackets delimiting the message that's passed through:

alert("Message is: [" + Object.values(rows)[0]  + "]")

Finally, I've updated manifest.json so that the button label for the Log Activity screen is now "Generate Alert"

{
"id": "EventListToolbarButton",
"text": "Generate Alert",
"description": "Pass IDs for events",
"icon": "static/images/bookmarks_small.png",
"rest_method": "listFunction",
"javascript": "my_toolbar_button_action(result)",
"groups": [
"customEventListToolbar"
]
},

In order to enable the required API endpoints for a development app on the server side we need to copy the sdk_developer_tools.jar to /opt/qradar/jars on the QRadar console and restart Tomcat with systemctl restart tomcat. This can take a few minutes to complete and progress can be monitored in /opt/tomcat/logs/catalina.log. The sdk_developer_tools.jar is provided with the SDK.

Next, to register our development app so that it appears on the QRadar UI we need to complete these steps in the app workspace on the local machine:

• Preregister the app with the new qapp preregister action:

> qapp preregister -q <QRadar Console IP> -u admin
Please enter password for user admin:
App in workspace [registration_app_two] successfully preregistered on server <QRadar Console IP>

• Run the app container locally:

> qapp run -d -q <QRADAR CONSOLE IP>
No image found for workspace [registration_app_two]
Found base image internal.docker.instance/gaf/qradar-app-base:2.1.3
Preparing image build directory /Users/mmaclynn/qradarappsdk/docker/build
Using /Users/mmaclynn/qradarappsdk/docker/build/Dockerfile
Creating Supervisor program entry for Flask
Building image [registration_app_two]
Using user ID 501 and group ID 20
DOCKER BUILD LOG: START
Step 1/14 : FROM internal.docker.instance/gaf/qradar-app-base:2.1.3
Step 2/14 : LABEL com.ibm.si.app.origin=SDK
Step 3/14 : ARG APP_USER_ID
Step 4/14 : ARG APP_GROUP_ID
Step 5/14 : ARG APP_USER_NAME=appuser
Step 6/14 : ARG APP_GROUP_NAME=appuser
Step 7/14 : ENV APP_ROOT /opt/app-root
Step 8/14 : ENV APP_USER_ID $APP_USER_ID
Step 9/14 : ENV APP_GROUP_ID $APP_GROUP_ID
Step 10/14 : ENV PATH $APP_ROOT/bin:$PATH
Step 11/14 : COPY / $APP_ROOT
Step 12/14 : RUN groupadd -o -g $APP_GROUP_ID $APP_GROUP_NAME && useradd -l -u $APP_USER_ID -g $APP_GROUP_ID $APP_USER_NAME && mkdir -p /etc/supervisord.d && if [ -f $APP_ROOT/init/supervisord.conf ]; then mv $APP_ROOT/init/supervisord.conf /etc; fi && rm -rf $APP_ROOT/init/* && if [ -d $APP_ROOT/bin ]; then chmod -R 755 $APP_ROOT/bin; fi && if [ -d $APP_ROOT/container/build ]; then chmod -R 755 $APP_ROOT/container/build; fi && if [ -d $APP_ROOT/container/run ]; then chmod -R 755 $APP_ROOT/container/run; fi && if [ -d $APP_ROOT/container/clean ]; then chmod -R 755 $APP_ROOT/container/clean; fi && if [ -d $APP_ROOT/container/service ]; then chmod -R 755 $APP_ROOT/container/service; fi && if [ -d $APP_ROOT/startup.d ]; then chmod -R 755 $APP_ROOT/startup.d; fi && if [ -d $APP_ROOT/container/conf/supervisord.d ]; then mv $APP_ROOT/container/conf/supervisord.d/*.conf /etc/supervisord.d; fi && if [ -d /etc/supervisord.d ]; then chmod -R 755 /etc/supervisord.d ; fi && echo -e "appuser ALL=(ALL) NOPASSWD:ALL\n" >> /etc/sudoers && visudo -cf /etc/sudoers
/etc/sudoers: parsed OK
Step 13/14 : USER $APP_USER_NAME
Step 14/14 : ENTRYPOINT ["sh", "/opt/app-root/bin/start.sh"]
Successfully built 8a8f3928c775
Successfully tagged registration_app_two:latest
DOCKER BUILD LOG: END
Image [registration_app_two] build completed successfully
Starting container [qradar-registration_app_two] using image [registration_app_two]
Configuring container environment
Setting QRADAR_APP_ID using development app ID
Setting FLASK_ENV
Setting REQUESTS_CA_BUNDLE
No qenv.ini file found in workspace
Mounting /Users/mmaclynn/Documents/75_work/qradar-8489_app_editor_blog/registration_app_two/manifest.json to /opt/app-root/manifest.json
Mounting /Users/mmaclynn/Documents/75_work/qradar-8489_app_editor_blog/registration_app_two/app to /opt/app-root/app
Mounting /Users/mmaclynn/Documents/75_work/qradar-8489_app_editor_blog/registration_app_two/store to /opt/app-root/store
Mounting /Users/mmaclynn/.qradar_app_sdk/<QRADAR CONSOLE IP>/ca-bundle.crt to /etc/qradar_pki/ca-bundle.crt
Setting memory limit 100MB
Container [qradar-registration_app_two] started with ID 708828bcdc
Host port 59818 is mapped to container port 5000/tcp
Flask is running in development mode
Access Flask endpoints at http://localhost:59818
Use docker ps to monitor container status
Log files are located in /Users/mmaclynn/Documents/75_work/qradar-8489_app_editor_blog/registration_app_two/store/log

• Configure network access from the QRadar server to your local machine

In the previous step when the app container was started, a port was assigned to the app's Flask service - port 59818 in this example:

Host port 59818 is mapped to container port 5000/tcp

In order for the App Framework on the QRadar server to reach the endpoints on container port 5000, it must be able to access the assigned port on your local machine. In my lab environment I configure this using SSH port forwarding.

• Register the app with the new qapp register action:

> qapp register -q <QRadar Console IP> -u admin
Please enter password for user admin:
App in workspace [registration_app_two] successfully registered on server <QRadar Console IP> to serve requests from 127.0.0.1 on these ports: 59818

More details on each of these steps will be included in the SDK documentation.

Having registered the development app and logged into the UI we now see the app-supplied toolbar button. Clicking it shows the message set in the code:

As Flask is in development mode, I can make a change to the python code, have Flask detect it automatically and see the change displayed on the UI. As a simple example, I'll update the code in list_function() to change the text of the message:

rows = {"ids":["Message two"]}

The app store/log/startup.log shows a Flask message indicating that the change has been detected:

 * Detected change in '/opt/app-root/app/views.py', reloading

Now, if we click the button in the UI we see the updated text:

This gives us a nice responsive feedback loop for testing changes.

If we want to update manifest.json to change the config any of the app's UI components or services, rather than run qapp deploy we run the qapp register SDK action (again). As development apps are treated differently by the app framework, there is no app upgrade performed with qapp register nor is there an upload of the app zip file. This means a qapp register completes considerably faster than a qapp deploy:

> qapp register
Please enter password for user admin:
Registration of app in workspace [registration_app_two] successfully updated on server <my QRadar IP> to serve requests from 127.0.0.1 on these ports: 59818

These SDK improvements greatly enhance app developer productivity. They are currently planned for the upcoming QRadar and SDK releases.