IBM QRadar SOAR

IBM QRadar SOAR

Join this online user group to communicate across Security product users and IBM experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.

 View Only
Expand all | Collapse all

app-fn_scheduler database configuration

  • 1.  app-fn_scheduler database configuration

    Posted 08/23/21 05:55 PM
    Edited by Tyler Bennett 08/23/21 06:10 PM
    Can I get some guidance on deploying the app-fn_scheduler? I've read the documentation but it's not really straightforward on the setup of the sqllite/psql database connection.

    We are using appHosts.

    I tried the following option in the configuration and touched the file on disk(on the dev apphost)
    datastore_dir = /tmp/app-fn_scheduler.sqllite

    fn-scheduler: selftest: failure selftest output: {'state': 'failure', 'reason': '(sqlite3.OperationalError) unable to open database file\n(Background on this error at: http://sqlalche.me/e/14/e3q8)'} Elapsed time: 0.025000 seconds

    touch /tmp/app-fn_scheduler.sqllite
    chmod 777 /tmp/app-fn_scheduler.sqllite

    -----------------------------
    Tyler Bennett
    ------------------------------


  • 2.  RE: app-fn_scheduler database configuration

    Posted 08/24/21 10:33 AM
    Hi Tyler,

    I do agree that it's a bit of confusing. The app.config setting `datastore_dir` refers to a directory where the app will create the `scheduler.sqlite` file. So, make sure your entry is a `folder`:

    mkdir /tmp/app-fn_scheduler​


    ------------------------------
    Mark Scherfling
    ------------------------------



  • 3.  RE: app-fn_scheduler database configuration

    Posted 08/24/21 12:04 PM
    Okay, that makes sense.

    I configured the following folder for testing in dev.
    mkdir /tmp/app-fn_scheduler-sqllite

    Then tries the following two configurations and received the same error messages. 
    datastore_dir = /tmp/app-fn_scheduler-sqllite
    datastore_dir = /tmp/app-fn_scheduler-sqllite/

    Then I made the folder world writeable and get the same error messages.
    chmod 777 /tmp/app-fn_scheduler-sqllite

    datastore_dir = /tmp/app-fn_scheduler-sqllite
    datastore_dir = /tmp/app-fn_scheduler-sqllite/


    Error for all four attempts is as follows

    fn-scheduler: selftest: failure selftest output: {'state''failure''reason''(sqlite3.OperationalError) unable to open database file\n(Background on this error at: http://sqlalche.me/e/14/e3q8)'} Elapsed time: 0.025000 seconds


    ------------------------------
    Tyler Bennett
    ------------------------------



  • 4.  RE: app-fn_scheduler database configuration

    Posted 08/24/21 12:50 PM

    I followed similar steps:

    $ mkdir /tmp/scheduler.sqlite

    My app.config file:

    [fn_scheduler]
    timezone=America/New_York
    thread_max=20
    datastore_dir=/tmp/scheduler.sqlite
    Then, the sqlite file is created:
    ls /tmp/scheduler.sqlite/scheduler.sqlite


    ------------------------------
    Mark Scherfling
    ------------------------------



  • 5.  RE: app-fn_scheduler database configuration

    Posted 08/24/21 01:15 PM
    I followed those steps exactly.

    Logged onto the dev apphost and ran
    $ mkdir /tmp/scheduler.sqlite


    I guess I raise a ticket with IBM then?



    ------------------------------
    Tyler Bennett
    ------------------------------



  • 6.  RE: app-fn_scheduler database configuration

    Posted 08/24/21 04:51 PM
    Hi Tyler,

    I believe the issue is permissions within the container. A better location within the container would be datastore_dir=/var/rescircuits. This location is intended for app file access.

    Keep in mind that containers are ephemeral. They can be restarted and the state of the container is reset, losing your schedules. This is where we now support external a postgre databases. Rather than using datastore_dir=, use db_url=postgresql+pypostgresql://username:password@host:port/database referencing an externally accessible database.

    Regards,
    Mark

    ------------------------------
    Mark Scherfling
    ------------------------------



  • 7.  RE: app-fn_scheduler database configuration

    Posted 08/24/21 05:17 PM
    So to be clear, it is not recommended to use datastore_dir when configuring this app on an apphost? Do you have any details for how to create the postgres system? I know next to nothing about postgres or container networking and the docs are extremely sparse.

    My assumption is we can create a postgres database on the apphost, then configure the containers to connect to the apphost database. Can you layout the steps to create this database, and allow container access to it?

    ------------------------------
    Tyler Bennett
    ------------------------------



  • 8.  RE: app-fn_scheduler database configuration

    Posted 08/26/21 09:05 AM
    Edited by John Quirke 08/26/21 10:27 AM
    Hi Tyler

    On apphost the /tmp folder reference is the location on the container where you would want to configure your sqllite database.
    Typically if you were using an integration server environment (non container) we would be referencing a folder on the integration server.
    If used on apphost and the container were to restart the database would be overwritten.
    When we are using apphost we therefore use  postgres connection to an external database.
    If you would like to test with postgres , I can message you privately and share some quick docker environment setup commands to test.
    I have it running on a linux environment.

    Regards
    John

    ------------------------------
    John Quirke
    ------------------------------



  • 9.  RE: app-fn_scheduler database configuration

    Posted 10/24/23 05:04 PM

    would you mind sharing with me the docker environment setup commands for postgres?



    ------------------------------
    Ryan Terry
    ------------------------------



  • 10.  RE: app-fn_scheduler database configuration

    Posted 10/25/23 01:33 AM

    Hey Ryan,

    Check out the video and the documentation I created in the link, documentation in the link in the description of the video, where I explain how to get this all setup! Hope that helps!

    https://www.youtube.com/watch?v=k04zz4Jps8w&t

    Thanks!



    ------------------------------
    Nick Mumaw, GPEN, GPYC
    Cyber Security Specialist - SOAR
    IBM - Security
    ------------------------------



  • 11.  RE: app-fn_scheduler database configuration

    Posted 05/05/25 06:37 AM

    Hi all ,

    In my scenario , I don't care if the schedules got deleted once a restart is performed.

    Still Im not able to get an app host version to work with sqlite , I have tried creating the file from the GUI and referencing it in the app with no luck.

    If anyone have that working, I would appreciate the help, thanks.



    ------------------------------
    Mohamad islam Hamadieh
    I post SOAR content and tips on linkedIn , follow me :)
    https://linkedin.com/in/mohamadislam
    ------------------------------



  • 12.  RE: app-fn_scheduler database configuration

    Posted 05/05/25 07:48 AM

    Hi Mohamad,

    Can yo share what your app.config file looks like? It should contain the setting `datastore_dir` to reference the directory which the sqlite file will reside, such as /tmp.



    ------------------------------
    Mark Scherfling
    ------------------------------



  • 13.  RE: app-fn_scheduler database configuration

    Posted 05/08/25 03:44 AM

    Hi Mark , 

    I have tried both creating in /tmp (from apphost ssh) and /etc/rescircuits/ (from app config GUI)

    I tried inputting both directories and files in  datastore_dir =   with no luck.

    also I have done chmod 777 on both the files and directories

    some examples I used : 

    datastore_dir =/tmp/test.sqlite(where test.sqlite is a directory)

    datastore_dir =/tmp/file.sqlite(where file.sqlite is a file)

    datastore_dir = /etc/rescircuits/test.sqlite

    datastore_dir = /etc/rescircuits/test

    datastore_dir = /etc/rescircuits/test/test.sqlite

    it always returns the same error : 

    return self.loaded_dbapi.connect(*cargs, **cparams) sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file (Background on this error at: https://sqlalche.me/e/20/e3q8)

    also tried DEBUG log level its returning the same error



    ------------------------------
    Mohamad islam Hamadieh
    I post SOAR content and tips on linkedIn , follow me :)
    https://linkedin.com/in/mohamadislam
    ------------------------------



  • 14.  RE: app-fn_scheduler database configuration

    Posted 05/16/25 01:29 AM

    hi @Mark Scherfling , please let me know  if you know of a fix for this issue , thanks.



    ------------------------------
    Mohamad islam Hamadieh
    I post SOAR content and tips on linkedIn , follow me :)
    https://linkedin.com/in/mohamadislam
    ------------------------------



  • 15.  RE: app-fn_scheduler database configuration

    Posted 05/19/25 04:34 AM

    Hi Mohamad,

    You cannot use SQLite with App Host because containers are ephemeral and the database will not persist after a container restart.

    https://ibmresilient.github.io/resilient-community-apps/fn_scheduler/README.html#migrating-to-v1-0-2

    ******

    # db url if using a postgreSQL DB. Use this with AppHost
    # db_url=postgresql://username:password***@host:port/database (or) db_url=postgresql+pypostgresql://username:password***@host:port/database
    
    

    Use this setting rather than the SQLite datastore_dir setting to persist the scheduler DB in PostgreSQL. This is necessary in an App Host environment to retain your schedules outside the app container.

    ********

    In short, stand up Postgres elsewhere and point the scheduler application to it so the data persists regardless of whether the container is restarted. You cannot use Postgres that comes with SOAR.



    ------------------------------
    BEN WILLIAMS
    ------------------------------



  • 16.  RE: app-fn_scheduler database configuration

    Posted 11/07/25 12:28 PM

    Hi Ben,

    little late to the party, I am trying to achieve the same in my lab without making the app pointing at another server that has the Postgres with no luck sadly.

    Any workaround using the dir rather than the url would be much appreciated :)



    ------------------------------
    abd el rahman Hamed
    ------------------------------



  • 17.  RE: app-fn_scheduler database configuration

    Posted 11/12/25 12:36 PM

    Hi abd el rahman Hamed,

    There is no supported way to use SQLite for the reasons I gave I'm afraid.



    ------------------------------
    BEN WILLIAMS
    ------------------------------



  • 18.  RE: app-fn_scheduler database configuration

    Posted 06/04/26 06:05 AM

    Hi,

    Do you have solution for problem with create:

     datastore_dir

    I have tried some solution but I have still problem:

    The last test failed

    Error: unable to open database file



    ------------------------------
    Radomír Franěk
    ------------------------------



  • 19.  RE: app-fn_scheduler database configuration

    Posted 06/04/26 09:00 AM

    Hi Radomír,

    Are you using a SOAR App Host? If you are, then the message is still the same, you must use db_url.

    Remember that containers are ephemeral meaning that if the container or in this case the pods is restarted the SQLite database will be removed losing the data. Using db_url is the only solution for an App Host. If the pod is restarted it will simply reconnect to the remote Postgres server on startup with no data loss.



    ------------------------------
    BEN WILLIAMS
    ------------------------------



  • 20.  RE: app-fn_scheduler database configuration

    Posted 06/05/26 07:57 AM

    H Ben,

    thanks for your answer.

    Is  diferrent solution for call from my playbook to other system (Call REST API)? I cant use db, now



    ------------------------------
    Radomír Franěk
    ------------------------------



  • 21.  RE: app-fn_scheduler database configuration

    Posted 06/05/26 09:28 AM

    Hi Radomír,

    I don't understand. The REST API application doesn't use a database. Can you explain the problem?



    ------------------------------
    BEN WILLIAMS
    ------------------------------



  • 22.  RE: app-fn_scheduler database configuration

    Posted 06/05/26 09:38 AM

    Srry my usecase

    Playbook:

    function REST API Call create Incident (SOAR) in HelpDesk system  -> Script postprocess write to Notes (SOAR) detail about created INC in Helpdesk.

    My idea was Scheduler call run to playbook for get information from  Helpdesk and write to Notes (SOAR)

    I need call to every hour: when changed detail (priority,desc and status)  in Inc (Helpdesk) then write information to Notes (SOAR)



    ------------------------------
    Radomír Franěk
    ------------------------------



  • 23.  RE: app-fn_scheduler database configuration

    Posted 06/05/26 12:34 PM

    Hi Radomír,

    It sounds like you still need the scheduler application and still the requirement stands that you need to use a remote Postgres server if the application is deployed to a SOAR App Host.

    It is this database where the schedules you define are stored. If it was possible to write to a database inside the pod/container then the database would be deleted and recreated on each application restart, losing your schedules. The pod can be restarted often which is why the remote database is a requirement.



    ------------------------------
    BEN WILLIAMS
    ------------------------------



  • 24.  RE: app-fn_scheduler database configuration

    Posted 06/08/26 07:31 AM

    It's nuts that scheduling playbook runs is not a bundled capability in core SOAR. There have been postings like Tyler's for years with challenges getting the Scheduler app to work while SOAR's competition provide scheduling out of the box for their automations. 

    I have created a feature request to replace the Scheduling App with a bundled SOAR capability. Titled "Add scheduler for playbooks native to SOAR, not as an app". Please vote for it on IBM Ideas: https://ideas.ibm.com/ideas/RIRP-I-1630.



    ------------------------------
    ken ching
    ------------------------------