IBM Security QRadar SOAR

 View Only
Expand all | Collapse all

Running multiple resilient circuits in multiple virtualenv for MSSPs

  • 1.  Running multiple resilient circuits in multiple virtualenv for MSSPs

    Posted Mon March 09, 2020 05:01 AM
    Hello,

    I'm using resilient for MSSPs with more than 28 child organization, and i created a virtual environment for each child.

    I installed resilient circuits for each virtual environment and run circuits by created a run.py .

    ~/my_virtual_env/lib/python2.7/site-packages/resilient_circuits/run.py


    #!/usr/bin/env python

    from resilient_circuits import app
    import os

    os.environ["APP_CONFIG_FILE"] = "childorg.app.config"
    os.environ["APP_LOCK_FILE"] = "childorg.lock.file"
    os.environ["APP_LOG_DIR"] = "childorg.app.log"

    app.run()

    My request, is how can keep the resilient circuits running in each virtual environment ?

    I asked in an other threat about the ability to use this config :


    [Unit]
    Description=Resilient-Circuits Service
    After=resilient.service
    Requires=resilient.service

    [Service]
    Type=simple

    ExecStart=~/my-virtual_env/bin/python  ~/my_virtual_env/lib/python2.7/site-packages/resilient_circuits/
    Restart=always
    TimeoutSec=10
    Environment=APP_CONFIG_FILE=~/my_virtual_env/lib/python2.7/site-packages/resilient_circuits/childorg.app.config
    Environment=APP_LOCK_FILE=/home/integration/.resilient/childorg.lock.file

    [Install]
    WantedBy=multi-user.target

    Can you help me resolving this or there is another way to do this ? thanks in advance.

    ------------------------------
    Ayman Sabri Cyber Security Analyst II
    ------------------------------


  • 2.  RE: Running multiple resilient circuits in multiple virtualenv for MSSPs

    Posted Mon March 09, 2020 05:15 AM
    Hi Ayman,

    Assuming that you are not using any keyring to store your secrets and that requires manual interaction, using a systemd unit is a good approach. However, I think you have a few issues with your unit definition:
    • You cannot refer to the home path of the user in ExecStart. Although this is a full path, Systemd forces you to start the path with '/'. In your case I think it would be something like '/home/integration/.../python'
    • You should not run the unit as root. For this, you can add "User=integration"
    • If you are planning to use this unit like this for multiple Circuits instances running in parallel, then it won't work. They all will try to use the same lock file and therefore they will refuse to start.

    Regarding the last point, having 28 different Systemd units for all your child organizations is inconvenient but there are nice systemd tricks you can use to use the same unit and I can give you an basic example of how to do this. Can you share the link to the other post you mentioned? I would post the answer there so more people can make use of it if they need it.

    ------------------------------
    Regards,
    Carlos Ortigoza
    ------------------------------



  • 3.  RE: Running multiple resilient circuits in multiple virtualenv for MSSPs

    Posted Mon March 09, 2020 05:23 AM
    Hi Carlos,

    In the other post, Williams asked me to create a new post for my request:
    https://community.ibm.com/community/user/security/communities/community-home/digestviewer/viewthread?GroupId=2845&MessageKey=251b90f8-1dcd-408a-90d6-856b18bce452&CommunityKey=d2f71e8c-108e-4652-b59c-29d61af7163e&tab=digestviewer&ReturnUrl=%2fcommunity%2fuser%2fsecurity%2fcommunities%2fcommunity-home%2fdigestviewer%3fcommunitykey%3dd2f71e8c-108e-4652-b59c-29d61af7163e%26tab%3ddigestviewer

    looking forward to see the solution.

    PS: should i create in integration user for each virtual env to run resilient circuits ?

    Thank you soo much



    ------------------------------
    Ayman Sabri Cyber Security Analyst II
    ------------------------------



  • 4.  RE: Running multiple resilient circuits in multiple virtualenv for MSSPs

    Posted Tue March 10, 2020 04:30 AM
    Hi Carlos,

    Any updates on how to resolve this :( ?

    Thank you

    ------------------------------
    Ayman Sabri Cyber Security Analyst II
    ------------------------------



  • 5.  RE: Running multiple resilient circuits in multiple virtualenv for MSSPs
    Best Answer

    Posted Tue March 10, 2020 07:52 AM
    Hi Ayman,

    Ok, I'll reply here and you can link this post to the other one afterwards. Bear in mind that I'm assuming a few things:
    - You want to use different config files
    - You want to run all the instances of Circuits as the same user
    - Each Circuits instance will use a different log file and in the same directory (actually, this must be this way or you will corrupt the log file)

    Based of those conditions, the following systemd unit file should do the job:

    [Unit]
    Description=Start Circuits component in its own Python Virtual Environment
    After=network-online.target

    [Service]
    Environment="APP_LOCK_FILE=~/.resilient/%I"
    User=integration
    ExecStart=/path/to/python /path/to/resilient_circuits/run.py --logfile /path/to/your/logs/directory/%I.log --config-file /path/to/your/configuration directory/%I/app.config
    ExecStop=/bin/kill -p $MAINPID

    [Install]
    WantedBy=multi-user.target

    Notice that you don't actually need different Python virtual environments unless you want to have different components and/or Python dependencies installed in each of them. If that's the case, you can simply used the path to the Python binary in the virtual environment instead of the default one. More importantly, this unit allows you to run N instances, which you can do in this way:

    systemctl start circuits@my_first_org

    Basically, whatever you type after the '@' will we pass to the unit in the %I variable. Using this trick you can tweak your unit definition to make it as reusable as needed.

    I hope this helps. If you have any problem, just let me know :)

    ------------------------------
    Regards,
    Carlos Ortigoza
    ------------------------------



  • 6.  RE: Running multiple resilient circuits in multiple virtualenv for MSSPs

    Posted Tue March 10, 2020 10:08 AM
    Hi carlos,

    Thanks for your feedback, but i still have some questions please:

    • I'm actualy using virtual envirenment, you sad that "you don't actually need different Python virtual environments", how can i install resilient circuits for each child org, or all of them they use the same resilient circuits with differente conf file ?

      For installing resilient circuits in multiple virtual environment , should i create an integration user for each virtual env ?
    • I have multiple virtual environments, which mean multiple path to python binaries, this is confusing. (~/my-virtual_env1/bin/python ,  ~/my-virtual_env2/bin/python ....)

    Can you please help me resolving this using the inputs i have.

    Thank you soo much for your time.

    ------------------------------
    Ayman Sabri Cyber Security Analyst II
    ------------------------------



  • 7.  RE: Running multiple resilient circuits in multiple virtualenv for MSSPs

    Posted Tue March 10, 2020 04:29 PM
    Hi Ayman,

    Sorry but I didn't fully understand all of your questions. Answering to the first one: you can use different configuration files even if you use the same instance of Circuit (and no Python virtual environments at all). Actually, in the systemd unit that I created for you, you can see that I used the "--config-file" option in the "ExecStart" entry. You can use this to point to different configuration files.

    Hope that clarifies.

    ------------------------------
    Regards,
    Carlos Ortigoza
    ------------------------------



  • 8.  RE: Running multiple resilient circuits in multiple virtualenv for MSSPs

    Posted Wed March 11, 2020 04:42 AM
    Hi Carlos,

    I got it i think:

    I need to generate a config file for each child org : resilient-circuits config -c /path/to/your/configuration directory/childorg1/app.config and then use the @ will passs the variable for %I in systemd.

    still have one small question, what is the variables i should use in the run.py ?

    Thanks alot.






    ------------------------------
    Ayman Sabri Cyber Security Analyst II
    ------------------------------



  • 9.  RE: Running multiple resilient circuits in multiple virtualenv for MSSPs

    Posted Wed March 11, 2020 10:59 AM
    Hi Carlos, 

    I start a new integration using the recommendations you provided, i installed resilient circuits.
    I created child orgs folder in /home/integration/ path ( /home/integration/childorg1 , /home/integration/childorg2 ...) and the config org in the /home/integration/.resilient/ path.

    my systemd configuration should look like this right ?

    [Unit]
    Description=Start Circuits component in its own Python Virtual Environment
    After=network-online.target

    [Service]
    Environment="APP_LOCK_FILE=~/.resilient/%I"
    User=integration
    ExecStart=/bin/python /path/to/resilient_circuits/run.py --logfile /home/integration/logs/%I.log --config-file /home/integration/%I/app.config
    ExecStop=/bin/kill -p $MAINPID

    [Install]
    WantedBy=multi-user.target


    still have one small question, what is the variables i should use in the run.py ?


    Thank you  @Carlos Ortigoza

    ​​

    ------------------------------
    Ayman Sabri Cyber Security Analyst II
    ------------------------------



  • 10.  RE: Running multiple resilient circuits in multiple virtualenv for MSSPs

    Posted Thu March 12, 2020 03:42 AM
    Hi Ayman,

    From your first post I see you were changing this:
    os.environ["APP_CONFIG_FILE"] = "childorg.app.config"
    os.environ["APP_LOCK_FILE"] = "childorg.lock.file"
    os.environ["APP_LOG_DIR"] = "childorg.app.log"

    If that's the case, then you can change this as well either from the command line with parameters like --config-file, --log-dir, --log-file, etc. or using those very same variables when you start Circuits and rather than using this "run.py" script in the ExecStart entry, you can simply use "/path/to/resilient-circuits run" and you would get the same result. I'm not really sure anymore to understand what you want to reuse among different instances of Circuits and what not, but you can play with these settings in the unit file I sent to you. Actually it already specifies the lockfile, log file and config file.

    ------------------------------
    Regards,
    Carlos Ortigoza
    ------------------------------



  • 11.  RE: Running multiple resilient circuits in multiple virtualenv for MSSPs

    Posted Thu March 12, 2020 10:18 AM
    Edited by Ayman Sabri Thu March 12, 2020 10:29 AM
    I sorry but i tried without success. i'll try to explain what is the config i did step by step:

    • I created a folder for each child org in /home/integration/
    • I created a config file in each child org using the command resilient-circuits config -c /home/integration/childorg/app.config
    • I specified my host and port in the config file and select the log directoy : logdir=/home/integration/childorg
    • I created a unit : sudo vi /etc/systemd/system/resilient_circuits.service
      [Unit]

      Description=Start Circuits component in its own Python Virtual Environment
      After=network-online.target

      [Service]
      Environment="APP_LOCK_FILE=/home/integration/%I/resilient_circuits.lock"
      User=integration
      ExecStart=/usr/local/bin/resilient-circuits run --logfile /home/integration/%I/app.log --config-file /home/integration/%I/app.config
      ExecStop=/bin/kill -p $MAINPID

      [Install]
      WantedBy=multi-user.target
    • I executed the following commands to enable and start the service:
      sudo chmod 664 /etc/systemd/system/resilient_circuits.service
      sudo systemctl daemon-reload
      sudo systemctl start resilient_circuits@childorg
    • I got the following error :
      Failed to start resilient_circuits@childorg.service: Unit not found.


    I realy need help to solve this, i hope my explanation is clear , and i'm sorry about my bad english :(

    Thank you.



    ------------------------------
    Ayman Sabri Cyber Security Analyst II
    ------------------------------



  • 12.  RE: Running multiple resilient circuits in multiple virtualenv for MSSPs

    Posted Thu March 12, 2020 10:27 AM
    Hi Ayman,

    Thanks, now it's much clearer. Two things:
    • If you want to use instances (use the %I), the unit's name must end with "@": resilient_circuits@.service
    • Although it will work in the way you described it, as you specified in each config file the "log dir", you can even remove the "--logfile" flag from the unit file.
    Hope it works now :)


    ------------------------------
    Regards,
    Carlos Ortigoza
    ------------------------------



  • 13.  RE: Running multiple resilient circuits in multiple virtualenv for MSSPs

    Posted Thu March 12, 2020 10:36 AM

    Hi Carlos,

    If you want to use instances (use the %I), the unit's name must end with "@": resilient_circuits@.service

    About this point i need the specify the name of the child org which i want to run righ ? ( i mean the variable i should pass the %I), where i need to specify it ine this command sudo systemctl start resilient_circuits@.service ?

    Or i need to create a unit for each child org ?


    Thanks alot for the quick reply





    ------------------------------
    Ayman Sabri Cyber Security Analyst II
    ------------------------------



  • 14.  RE: Running multiple resilient circuits in multiple virtualenv for MSSPs

    Posted Thu March 12, 2020 10:45 AM
    Uh or the unit name , you mean this one sudo vi /etc/systemd/system/resilient_circuits@.service, should look like this right ?

    ------------------------------
    Ayman Sabri Cyber Security Analyst II
    ------------------------------



  • 15.  RE: Running multiple resilient circuits in multiple virtualenv for MSSPs

    Posted Thu March 12, 2020 10:49 AM
    Yes, Well Done. Thank you Soo much @Carlos Ortigoza  ​

    ------------------------------
    Ayman Sabri Cyber Security Analyst II
    ------------------------------