Hello,

We are having an issue with our Circuits instance. In particular, we have one integration that because of the nature of the actions it performs, sometimes it takes a long time to return. Now we are noticing that Circuits is only processing a few messages in parallel from this queue but not any other message from other queues.

We also notice that still Circuits is aware that there are other messages being received in other queues are messages like this are logged:

2019-06-04 12:56:33,049 INFO [actions_component] Event: <my_function[] (id=50, workflow=my_workflow, user=user@example.com) 2019-06-04 12:56:31.925000> Channel: functions.my_function

We are already thinking about forking this long task and return so Circuits can keep processing other messages but, before doing so, we would like to know what is the approach that Circuits follows to process the pending messages in all of its queues. In particular, shouldn't it process messages from other queues regardless it's processing messages from a different queue? Or is there a limit on the number of actions that can be process in parallel? (as we see several messages from this specific queue being processed simultaneously)

Thanks in advance for your reply.

Regards,

------------------------------
Carlos Ortigoza
------------------------------

Hi Carlos, 

The answer is a bit mixed. Functions, which are based on the ResilientComponent class, are multi-threaded. So long running processes will not effect other functions as well as other invocations of that same function. However, depending on how they're written, non-function based integrations can block resilient-circuits and other integrations as that execution is not multi-threaded. 

The best advise for these integrations is to consider creating a thread of execution, as you indicated, for long running and blocking processes.

Hope this helps.

------------------------------
Mark Scherfling
------------------------------

Hi Mark,

Regarding this problem, we have an environment with functions based on ResilientComponent class and other component (pollers) that are based on ResilientComponents but are not function that because of the long time they need it seems that are blocking this other processes.

Example:
class PollerTest(ResilientComponent):
def __init__(self, opts):
super(PollerTest, self).__init__(opts)
self.options = opts.get("fn_poller", {})
self.channel = CHANNEL
self.delay = int(self.options["polling_sec_poller"]) if self.options.get("polling_sec_poller") else 86400 # defaults every day
Timer(self.delay, PollEvent(), persist=False).register(self)

Is there any advice on how to solve this and have this two kind of integrations working together?
We are thinking to use virtual environments to separate it. What do you think?

Thanks in advance!

------------------------------
JOSE JAVIER GOMEZ MENDEZ
------------------------------

Original Message:
Sent: Tue June 04, 2019 04:30 PM
From: Mark Scherfling
Subject: Circuits queues management

Hi Carlos, 

The answer is a bit mixed. Functions, which are based on the ResilientComponent class, are multi-threaded. So long running processes will not effect other functions as well as other invocations of that same function. However, depending on how they're written, non-function based integrations can block resilient-circuits and other integrations as that execution is not multi-threaded. 

The best advise for these integrations is to consider creating a thread of execution, as you indicated, for long running and blocking processes.

Hope this helps.

------------------------------
Mark Scherfling
------------------------------