IBM QRadar SOAR

Hey! 

We are trying to switch from Python 2 to Python 3 on our Linux Integration server. Unfortunately we broke something and the resilient_circuits service wont start up now. What is the recommended way to do this upgrade? 

Thanks, 
Adina

------------------------------
Adina Bodkins
------------------------------

Hey Adina,

Can you clarify, "we broke something" and why specifically RC fails to start?

------------------------------
Jared Fagel
Cyber Security Analyst Intern
Public Utility
------------------------------

Original Message:
Sent: Thu November 21, 2019 03:16 PM
From: Adina Bodkins
Subject: Switching from Python 2 > Python 3 on Linux Integration Server

Hey! 

We are trying to switch from Python 2 to Python 3 on our Linux Integration server. Unfortunately we broke something and the resilient_circuits service wont start up now. What is the recommended way to do this upgrade? 

Thanks, 
Adina

------------------------------
Adina Bodkins
------------------------------

Hi @Adina Bodkins,

Just a guess :) 

You have to check and modify which python you are using with the resilient-circuits command header.

To check python version deployed :

[circuitslab@rhel75 .resilient]$ whereis python
python: /usr/bin/python /usr/bin/python2.7 /usr/bin/python3.6 /usr/bin/python3.6m /usr/bin/python3.6-config /usr/bin/python3.6m-config /usr/bin/python3.6m-x86_64-config /usr/bin/python2.7-config /usr/lib/python2.7 /usr/lib/python3.6 /usr/lib64/python2.7 /usr/lib64/python3.6 /etc/python /usr/local/lib/python3.6 /usr/include/python2.7 /usr/include/python3.6m /usr/share/man/man1/python.1.gz


Then Check : /usr/bin/resilient-circuits (python header depending your python 3 version, here for my own, python 3.6)

#!/usr/bin/python36

# -*- coding: utf-8 -*-
import re
import sys

from resilient_circuits.bin.resilient_circuits_cmd import main​
...
...


Then check on command line the start :

[circuitslab@rhel75 .resilient]$ resilient-circuits run
Please enter password for encrypted keyring:
2019-11-22 09:40:27,182 INFO [app] Configuration file: app.config
2019-11-22 09:40:27,184 INFO [app] Resilient server: resilient.zoldax.local
2019-11-22 09:40:27,184 INFO [app] Resilient user: api_user@zoldax.local
2019-11-22 09:40:27,184 INFO [app] Resilient org: XXXXXXX
2019-11-22 09:40:27,186 INFO [app] Logging Level: DEBUG
2019-11-22 09:40:27,187 DEBUG [actions_component] create idle timer

Hope this helps,
Regards,
Zoldax

------------------------------
Pascal Weber
-
Abakus Sécurité
PARIS
------------------------------

Original Message:
Sent: Thu November 21, 2019 03:16 PM
From: Adina Bodkins
Subject: Switching from Python 2 > Python 3 on Linux Integration Server

Hey! 

We are trying to switch from Python 2 to Python 3 on our Linux Integration server. Unfortunately we broke something and the resilient_circuits service wont start up now. What is the recommended way to do this upgrade? 

Thanks, 
Adina

------------------------------
Adina Bodkins
------------------------------

Hi @Jared Fagel  @Pascal Weber

Thank you for your responses. We have Python2 as part of the OS and then a user installed version of Python3. After doing some more digging it appears that we have to update the location of Python to point to the user installed version probably via a symlink so that /usr/bin/python points to the user installed version of Python3.

When we tried to modify the Python header in /usr/bin/resilient-circuits to the user installed location of Python3 we couldn't start the resilient-circuits service.

We have to do a bit more research into how to switch the server from pointing to Python2. I believe we need to do something along the lines of this: 
sudo update-alternatives --install /usr/bin/python python /user/installed/version/of/python3 3

If anyone has any experience with those and knows that that is the correct way that would be great as we're hesitant to break something. 

Thanks!
Adina

------------------------------
Adina Bodkins
------------------------------

Original Message:
Sent: Fri November 22, 2019 03:42 AM
From: Pascal Weber
Subject: Switching from Python 2 > Python 3 on Linux Integration Server

Hi @Adina Bodkins,

Just a guess :) 

You have to check and modify which python you are using with the resilient-circuits command header.

To check python version deployed :

[circuitslab@rhel75 .resilient]$ whereis python
python: /usr/bin/python /usr/bin/python2.7 /usr/bin/python3.6 /usr/bin/python3.6m /usr/bin/python3.6-config /usr/bin/python3.6m-config /usr/bin/python3.6m-x86_64-config /usr/bin/python2.7-config /usr/lib/python2.7 /usr/lib/python3.6 /usr/lib64/python2.7 /usr/lib64/python3.6 /etc/python /usr/local/lib/python3.6 /usr/include/python2.7 /usr/include/python3.6m /usr/share/man/man1/python.1.gz


Then Check : /usr/bin/resilient-circuits (python header depending your python 3 version, here for my own, python 3.6)

#!/usr/bin/python36

# -*- coding: utf-8 -*-
import re
import sys

from resilient_circuits.bin.resilient_circuits_cmd import main​
...
...


Then check on command line the start :

[circuitslab@rhel75 .resilient]$ resilient-circuits run
Please enter password for encrypted keyring:
2019-11-22 09:40:27,182 INFO [app] Configuration file: app.config
2019-11-22 09:40:27,184 INFO [app] Resilient server: resilient.zoldax.local
2019-11-22 09:40:27,184 INFO [app] Resilient user: api_user@zoldax.local
2019-11-22 09:40:27,184 INFO [app] Resilient org: XXXXXXX
2019-11-22 09:40:27,186 INFO [app] Logging Level: DEBUG
2019-11-22 09:40:27,187 DEBUG [actions_component] create idle timer

Hope this helps,
Regards,
Zoldax

------------------------------
Pascal Weber
-
Abakus Sécurité
PARIS

Original Message:
Sent: Thu November 21, 2019 03:16 PM
From: Adina Bodkins
Subject: Switching from Python 2 > Python 3 on Linux Integration Server

Hey! 

We are trying to switch from Python 2 to Python 3 on our Linux Integration server. Unfortunately we broke something and the resilient_circuits service wont start up now. What is the recommended way to do this upgrade? 

Thanks, 
Adina

------------------------------
Adina Bodkins
------------------------------

Perhaps not a direct solution to your specific problem, but I'd consider using some sort of Python virtual environment - https://virtualenv.pypa.io/en/latest/. This is great as it can be used to isolate you Python version to a specific one, even if your system version of python needs to be something different.

Of course, if this machine is solely used for resilient-circuits it should not matter if you use the system Python.

------------------------------
Marty James
------------------------------

Original Message:
Sent: Mon November 25, 2019 11:16 AM
From: Adina Bodkins
Subject: Switching from Python 2 > Python 3 on Linux Integration Server

Hi @Jared Fagel  @Pascal Weber

Thank you for your responses. We have Python2 as part of the OS and then a user installed version of Python3. After doing some more digging it appears that we have to update the location of Python to point to the user installed version probably via a symlink so that /usr/bin/python points to the user installed version of Python3.

When we tried to modify the Python header in /usr/bin/resilient-circuits to the user installed location of Python3 we couldn't start the resilient-circuits service.

We have to do a bit more research into how to switch the server from pointing to Python2. I believe we need to do something along the lines of this: 
sudo update-alternatives --install /usr/bin/python python /user/installed/version/of/python3 3

If anyone has any experience with those and knows that that is the correct way that would be great as we're hesitant to break something. 

Thanks!
Adina

------------------------------
Adina Bodkins

Original Message:
Sent: Fri November 22, 2019 03:42 AM
From: Pascal Weber
Subject: Switching from Python 2 > Python 3 on Linux Integration Server

Hi @Adina Bodkins,

Just a guess :) 

You have to check and modify which python you are using with the resilient-circuits command header.

To check python version deployed :

[circuitslab@rhel75 .resilient]$ whereis python
python: /usr/bin/python /usr/bin/python2.7 /usr/bin/python3.6 /usr/bin/python3.6m /usr/bin/python3.6-config /usr/bin/python3.6m-config /usr/bin/python3.6m-x86_64-config /usr/bin/python2.7-config /usr/lib/python2.7 /usr/lib/python3.6 /usr/lib64/python2.7 /usr/lib64/python3.6 /etc/python /usr/local/lib/python3.6 /usr/include/python2.7 /usr/include/python3.6m /usr/share/man/man1/python.1.gz


Then Check : /usr/bin/resilient-circuits (python header depending your python 3 version, here for my own, python 3.6)

#!/usr/bin/python36

# -*- coding: utf-8 -*-
import re
import sys

from resilient_circuits.bin.resilient_circuits_cmd import main​
...
...


Then check on command line the start :

[circuitslab@rhel75 .resilient]$ resilient-circuits run
Please enter password for encrypted keyring:
2019-11-22 09:40:27,182 INFO [app] Configuration file: app.config
2019-11-22 09:40:27,184 INFO [app] Resilient server: resilient.zoldax.local
2019-11-22 09:40:27,184 INFO [app] Resilient user: api_user@zoldax.local
2019-11-22 09:40:27,184 INFO [app] Resilient org: XXXXXXX
2019-11-22 09:40:27,186 INFO [app] Logging Level: DEBUG
2019-11-22 09:40:27,187 DEBUG [actions_component] create idle timer

Hope this helps,
Regards,
Zoldax

------------------------------
Pascal Weber
-
Abakus Sécurité
PARIS

Original Message:
Sent: Thu November 21, 2019 03:16 PM
From: Adina Bodkins
Subject: Switching from Python 2 > Python 3 on Linux Integration Server

Hey! 

We are trying to switch from Python 2 to Python 3 on our Linux Integration server. Unfortunately we broke something and the resilient_circuits service wont start up now. What is the recommended way to do this upgrade? 

Thanks, 
Adina

------------------------------
Adina Bodkins
------------------------------

Hey @Adina Bodkins,

We have not made this change in our environment yet, but plan to. Have you made any progress on this as of yet?

One interesting thing I just noticed, is that the shebang lines in resilient-circuits source code all use "#!/usr/bin/env python" -- meaning it will always use the environment's first available "python" executable (ie, not python3). See this: type -a python

This hack solution may work... if you add a new directory (ex: /home/integrations/python_fix) that contains just an executable symlink called "python" that points to python3 and then add "/home/integrations/python_fix" to the start of ~/.bash_profile PATH variable, I believe this would work.

So in there, you may have: PATH=/home/integrations/python_fix:$PATH

Let me know if this works for you, or if you make progress in better manor.

------------------------------
Jared Fagel
Cyber Security Analyst Intern
Public Utility
------------------------------

Original Message:
Sent: Mon November 25, 2019 11:16 AM
From: Adina Bodkins
Subject: Switching from Python 2 > Python 3 on Linux Integration Server

Hi @Jared Fagel  @Pascal Weber

Thank you for your responses. We have Python2 as part of the OS and then a user installed version of Python3. After doing some more digging it appears that we have to update the location of Python to point to the user installed version probably via a symlink so that /usr/bin/python points to the user installed version of Python3.

When we tried to modify the Python header in /usr/bin/resilient-circuits to the user installed location of Python3 we couldn't start the resilient-circuits service.

We have to do a bit more research into how to switch the server from pointing to Python2. I believe we need to do something along the lines of this: 
sudo update-alternatives --install /usr/bin/python python /user/installed/version/of/python3 3

If anyone has any experience with those and knows that that is the correct way that would be great as we're hesitant to break something. 

Thanks!
Adina

------------------------------
Adina Bodkins

Original Message:
Sent: Fri November 22, 2019 03:42 AM
From: Pascal Weber
Subject: Switching from Python 2 > Python 3 on Linux Integration Server

Hi @Adina Bodkins,

Just a guess :) 

You have to check and modify which python you are using with the resilient-circuits command header.

To check python version deployed :

[circuitslab@rhel75 .resilient]$ whereis python
python: /usr/bin/python /usr/bin/python2.7 /usr/bin/python3.6 /usr/bin/python3.6m /usr/bin/python3.6-config /usr/bin/python3.6m-config /usr/bin/python3.6m-x86_64-config /usr/bin/python2.7-config /usr/lib/python2.7 /usr/lib/python3.6 /usr/lib64/python2.7 /usr/lib64/python3.6 /etc/python /usr/local/lib/python3.6 /usr/include/python2.7 /usr/include/python3.6m /usr/share/man/man1/python.1.gz


Then Check : /usr/bin/resilient-circuits (python header depending your python 3 version, here for my own, python 3.6)

#!/usr/bin/python36

# -*- coding: utf-8 -*-
import re
import sys

from resilient_circuits.bin.resilient_circuits_cmd import main​
...
...


Then check on command line the start :

[circuitslab@rhel75 .resilient]$ resilient-circuits run
Please enter password for encrypted keyring:
2019-11-22 09:40:27,182 INFO [app] Configuration file: app.config
2019-11-22 09:40:27,184 INFO [app] Resilient server: resilient.zoldax.local
2019-11-22 09:40:27,184 INFO [app] Resilient user: api_user@zoldax.local
2019-11-22 09:40:27,184 INFO [app] Resilient org: XXXXXXX
2019-11-22 09:40:27,186 INFO [app] Logging Level: DEBUG
2019-11-22 09:40:27,187 DEBUG [actions_component] create idle timer

Hope this helps,
Regards,
Zoldax

------------------------------
Pascal Weber
-
Abakus Sécurité
PARIS

Original Message:
Sent: Thu November 21, 2019 03:16 PM
From: Adina Bodkins
Subject: Switching from Python 2 > Python 3 on Linux Integration Server

Hey! 

We are trying to switch from Python 2 to Python 3 on our Linux Integration server. Unfortunately we broke something and the resilient_circuits service wont start up now. What is the recommended way to do this upgrade? 

Thanks, 
Adina

------------------------------
Adina Bodkins
------------------------------

Hi @Marty James - I am going to look into that virtualenv and see if that could help thanks for the suggestion. 

@Jared Fagel - we put the migration on hold for about a week or so because we doing heavy development with Resilient currently and don't want to break anything playing around with the Python stuff (we were hoping it was an easy 1​​,2,3 switch over but now it seems not to). Will look into your suggestion and will let you know what worked for us when we do the migration.

------------------------------
Adina Bodkins
------------------------------

Original Message:
Sent: Tue November 26, 2019 11:14 AM
From: Jared Fagel
Subject: Switching from Python 2 > Python 3 on Linux Integration Server

Hey @Adina Bodkins,

We have not made this change in our environment yet, but plan to. Have you made any progress on this as of yet?

One interesting thing I just noticed, is that the shebang lines in resilient-circuits source code all use "#!/usr/bin/env python" -- meaning it will always use the environment's first available "python" executable (ie, not python3). See this: type -a python

This hack solution may work... if you add a new directory (ex: /home/integrations/python_fix) that contains just an executable symlink called "python" that points to python3 and then add "/home/integrations/python_fix" to the start of ~/.bash_profile PATH variable, I believe this would work.

So in there, you may have: PATH=/home/integrations/python_fix:$PATH

Let me know if this works for you, or if you make progress in better manor.

------------------------------
Jared Fagel
Cyber Security Analyst Intern
Public Utility

Original Message:
Sent: Mon November 25, 2019 11:16 AM
From: Adina Bodkins
Subject: Switching from Python 2 > Python 3 on Linux Integration Server

Hi @Jared Fagel  @Pascal Weber

Thank you for your responses. We have Python2 as part of the OS and then a user installed version of Python3. After doing some more digging it appears that we have to update the location of Python to point to the user installed version probably via a symlink so that /usr/bin/python points to the user installed version of Python3.

When we tried to modify the Python header in /usr/bin/resilient-circuits to the user installed location of Python3 we couldn't start the resilient-circuits service.

We have to do a bit more research into how to switch the server from pointing to Python2. I believe we need to do something along the lines of this: 
sudo update-alternatives --install /usr/bin/python python /user/installed/version/of/python3 3

If anyone has any experience with those and knows that that is the correct way that would be great as we're hesitant to break something. 

Thanks!
Adina

------------------------------
Adina Bodkins

Original Message:
Sent: Fri November 22, 2019 03:42 AM
From: Pascal Weber
Subject: Switching from Python 2 > Python 3 on Linux Integration Server

Hi @Adina Bodkins,

Just a guess :) 

You have to check and modify which python you are using with the resilient-circuits command header.

To check python version deployed :

[circuitslab@rhel75 .resilient]$ whereis python
python: /usr/bin/python /usr/bin/python2.7 /usr/bin/python3.6 /usr/bin/python3.6m /usr/bin/python3.6-config /usr/bin/python3.6m-config /usr/bin/python3.6m-x86_64-config /usr/bin/python2.7-config /usr/lib/python2.7 /usr/lib/python3.6 /usr/lib64/python2.7 /usr/lib64/python3.6 /etc/python /usr/local/lib/python3.6 /usr/include/python2.7 /usr/include/python3.6m /usr/share/man/man1/python.1.gz


Then Check : /usr/bin/resilient-circuits (python header depending your python 3 version, here for my own, python 3.6)

#!/usr/bin/python36

# -*- coding: utf-8 -*-
import re
import sys

from resilient_circuits.bin.resilient_circuits_cmd import main​
...
...


Then check on command line the start :

[circuitslab@rhel75 .resilient]$ resilient-circuits run
Please enter password for encrypted keyring:
2019-11-22 09:40:27,182 INFO [app] Configuration file: app.config
2019-11-22 09:40:27,184 INFO [app] Resilient server: resilient.zoldax.local
2019-11-22 09:40:27,184 INFO [app] Resilient user: api_user@zoldax.local
2019-11-22 09:40:27,184 INFO [app] Resilient org: XXXXXXX
2019-11-22 09:40:27,186 INFO [app] Logging Level: DEBUG
2019-11-22 09:40:27,187 DEBUG [actions_component] create idle timer

Hope this helps,
Regards,
Zoldax

------------------------------
Pascal Weber
-
Abakus Sécurité
PARIS

Original Message:
Sent: Thu November 21, 2019 03:16 PM
From: Adina Bodkins
Subject: Switching from Python 2 > Python 3 on Linux Integration Server

Hey! 

We are trying to switch from Python 2 to Python 3 on our Linux Integration server. Unfortunately we broke something and the resilient_circuits service wont start up now. What is the recommended way to do this upgrade? 

Thanks, 
Adina

------------------------------
Adina Bodkins
------------------------------