Hi,

I see in top that drs-agent is consuming 99-100 % cpu all the time. This makes me think that it is limited to one cpu core only. Is this so? I have 100 databases in the Console now and it has started to slow down and there are occasional freezes as well. I have a Db2 11.5 community edition on the same server with DMC that it uses as its repository.  Server has 8 cores, but I seldom see any other process going over 10 % but the drs-agent. Is there any way to improve the performance? Could you also elaborate what drs-agent is actually doing?

------------------------------
Petri Helin
------------------------------

#Db2

Thanks Petri for contacting us! drs-agent is the backend agent used for the notification center in Data Management Console. Using cpu to 100% seems abnormal behavior. I will involve developer owner to step in to take a further look. Is that OK for you to open a DMC case to facilitate further investigation?

------------------------------
Su Li Hou
------------------------------

I have now opened a case for this. Looking at top in more detail it seems that it is at least not totally bound to one cpu only.

top - 07:45:00 up 22 days, 31 min, 1 user, load average: 1.12, 1.18, 1.29
Tasks: 542 total, 1 running, 541 sleeping, 0 stopped, 0 zombie
%Cpu0 : 15.4/0.0 15[||||||||||||||| ]
%Cpu1 : 0.0/0.3 0[ ]
%Cpu2 : 0.0/0.0 0[ ]
%Cpu3 : 0.7/1.7 2[||| ]
%Cpu4 : 84.4/0.0 84[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
%Cpu5 : 0.0/0.0 0[ ]
%Cpu6 : 0.0/0.0 0[ ]
%Cpu7 : 0.0/0.0 0[ ]
KiB Mem : 98836112 total, 13482132 free, 6497868 used, 78856112 buff/cache
KiB Swap: 4194300 total, 4194300 free, 0 used. 84066432 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4481 root 20 0 1427.6m 91.6m 21.2m S 100.0 0.1 18442:23 drs-agent
2880 root 20 0 757.2m 23.8m 9.7m S 2.0 0.0 763:23.31 BESClient
19869 db2inst4 20 0 13.4g 7.2g 7.2g S 0.7 7.7 3757:10 db2sysc

------------------------------
Petri Helin
------------------------------

Original Message:
Sent: Wed May 27, 2020 10:19 AM
From: Su Li Hou
Subject: About drs-agent and its cpu consumption

Thanks Petri for contacting us! drs-agent is the backend agent used for the notification center in Data Management Console. Using cpu to 100% seems abnormal behavior. I will involve developer owner to step in to take a further look. Is that OK for you to open a DMC case to facilitate further investigation?

------------------------------
Su Li Hou
------------------------------

Thanks Petri for your time! May you post the case# here so our support engineer and dev owner can follow up accordingly? @Cintia Ogura​​

------------------------------
Su Li Hou
------------------------------

Original Message:
Sent: Thu May 28, 2020 01:48 AM
From: Petri Helin
Subject: About drs-agent and its cpu consumption

I have now opened a case for this. Looking at top in more detail it seems that it is at least not totally bound to one cpu only.

top - 07:45:00 up 22 days, 31 min, 1 user, load average: 1.12, 1.18, 1.29
Tasks: 542 total, 1 running, 541 sleeping, 0 stopped, 0 zombie
%Cpu0 : 15.4/0.0 15[||||||||||||||| ]
%Cpu1 : 0.0/0.3 0[ ]
%Cpu2 : 0.0/0.0 0[ ]
%Cpu3 : 0.7/1.7 2[||| ]
%Cpu4 : 84.4/0.0 84[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
%Cpu5 : 0.0/0.0 0[ ]
%Cpu6 : 0.0/0.0 0[ ]
%Cpu7 : 0.0/0.0 0[ ]
KiB Mem : 98836112 total, 13482132 free, 6497868 used, 78856112 buff/cache
KiB Swap: 4194300 total, 4194300 free, 0 used. 84066432 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4481 root 20 0 1427.6m 91.6m 21.2m S 100.0 0.1 18442:23 drs-agent
2880 root 20 0 757.2m 23.8m 9.7m S 2.0 0.0 763:23.31 BESClient
19869 db2inst4 20 0 13.4g 7.2g 7.2g S 0.7 7.7 3757:10 db2sysc

------------------------------
Petri Helin
------------------------------


Original Message:
Sent: Wed May 27, 2020 10:19 AM
From: Su Li Hou
Subject: About drs-agent and its cpu consumption

Thanks Petri for contacting us! drs-agent is the backend agent used for the notification center in Data Management Console. Using cpu to 100% seems abnormal behavior. I will involve developer owner to step in to take a further look. Is that OK for you to open a DMC case to facilitate further investigation?

------------------------------
Su Li Hou

Sure, here it is: TS003755506

------------------------------
Petri Helin
------------------------------

Original Message:
Sent: Thu May 28, 2020 02:37 AM
From: Su Li Hou
Subject: About drs-agent and its cpu consumption

Thanks Petri for your time! May you post the case# here so our support engineer and dev owner can follow up accordingly? @Cintia Ogura​​

------------------------------
Su Li Hou
------------------------------


Original Message:
Sent: Thu May 28, 2020 01:48 AM
From: Petri Helin
Subject: About drs-agent and its cpu consumption

I have now opened a case for this. Looking at top in more detail it seems that it is at least not totally bound to one cpu only.

top - 07:45:00 up 22 days, 31 min, 1 user, load average: 1.12, 1.18, 1.29
Tasks: 542 total, 1 running, 541 sleeping, 0 stopped, 0 zombie
%Cpu0 : 15.4/0.0 15[||||||||||||||| ]
%Cpu1 : 0.0/0.3 0[ ]
%Cpu2 : 0.0/0.0 0[ ]
%Cpu3 : 0.7/1.7 2[||| ]
%Cpu4 : 84.4/0.0 84[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
%Cpu5 : 0.0/0.0 0[ ]
%Cpu6 : 0.0/0.0 0[ ]
%Cpu7 : 0.0/0.0 0[ ]
KiB Mem : 98836112 total, 13482132 free, 6497868 used, 78856112 buff/cache
KiB Swap: 4194300 total, 4194300 free, 0 used. 84066432 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4481 root 20 0 1427.6m 91.6m 21.2m S 100.0 0.1 18442:23 drs-agent
2880 root 20 0 757.2m 23.8m 9.7m S 2.0 0.0 763:23.31 BESClient
19869 db2inst4 20 0 13.4g 7.2g 7.2g S 0.7 7.7 3757:10 db2sysc

------------------------------
Petri Helin


Original Message:
Sent: Wed May 27, 2020 10:19 AM
From: Su Li Hou
Subject: About drs-agent and its cpu consumption

Thanks Petri for contacting us! drs-agent is the backend agent used for the notification center in Data Management Console. Using cpu to 100% seems abnormal behavior. I will involve developer owner to step in to take a further look. Is that OK for you to open a DMC case to facilitate further investigation?

------------------------------
Su Li Hou

Hi Petri. Thanks for reaching out. drs-agent is a backend process that scans the DB2 instances to generate configuration based alerts. It is a process launched by JVM, so JVM should automatically do the load balance on different cores. I will look into it in detail for sure.

As for the drs-agent consuming 80-90 % of the time, it could be because of two reasons:

It could be that scan interval to monitor the DB2 instances is small, especially given the 100s of connections.

Another reason could be that the alerts are getting accumulated given 100s of connections and it takes time to go through all the accumulated alerts (which we have addressed in the next version of DMC).

We will perform a thorough verification on our side by replicating 100s of DB connections, finding the exact root cause and get back to you.

Meanwhile, could you please try out these two things?

Go the folder where DMC is installed.

Navigate to this path    ---->    <ibm-data-management-console-path>/addons/drs/drs-agent

Open the .env file using  vi .env 

Please change these two variables to the values that I have mentioned and see if the performance improves:
SCAN_INTERVAL=30
QUEUE_LIMIT=50

------------------------------
Abhinandan Kelgere Ramesh
------------------------------

Hi Petri. Thanks for reaching out. drs-agent is a process that scans the DB2 instances to generate a Configuration based alerts. It is a process launched by JVM, so JVM should automatically do the load balance on different cores. I will look into it in detail for sure.

As for the drs-agent consuming 80-90 % of the time, it could be because of two reasons:

It could be that scan interval to monitor the DB2 instances is small, especially given the 100s of connections.
Another reason could be that the alerts are getting accumulated given 100s of connections and it takes time to go through all the accumulated alerts (which we have addressed in the next version of DMC).

We will perform a thorough verification on our side by replicating 100s of DB connections, finding the exact root cause and get back to you.

Meanwhile, could you please try out these two things?

Go the folder where DMC is installed.

Navigate to this path  ----> <ibm-data-management-console-path>/addons/drs/drs-agent

Open the .env file using --> vi .env

Please change these two variables to the values that I have mentioned and see if the performance improves:
SCAN_INTERVAL=30
QUEUE_LIMIT=50

------------------------------
Abhinandan Kelgere Ramesh
------------------------------

Hi Abhinandan,

I added these to the .env since the variables were not there before at all:

SCAN_INTERVAL=30
QUEUE_LIMIT=50

After that I recycled the service and gave it 30 minutes to settle, as I could see that the repository Db2 was working on something quite a lot, before I started to browse through the databases and their monitors. Everything feels quite snappy and I am now unable to get drs-agent to consume even more than 1 %.

------------------------------
Petri Helin
------------------------------

Original Message:
Sent: Wed May 27, 2020 05:35 PM
From: Abhinandan Kelgere Ramesh
Subject: About drs-agent and its cpu consumption

Hi Petri. Thanks for reaching out. drs-agent is a process that scans the DB2 instances to generate a Configuration based alerts. It is a process launched by JVM, so JVM should automatically do the load balance on different cores. I will look into it in detail for sure.

As for the drs-agent consuming 80-90 % of the time, it could be because of two reasons:

It could be that scan interval to monitor the DB2 instances is small, especially given the 100s of connections.
Another reason could be that the alerts are getting accumulated given 100s of connections and it takes time to go through all the accumulated alerts (which we have addressed in the next version of DMC).

We will perform a thorough verification on our side by replicating 100s of DB connections, finding the exact root cause and get back to you.

Meanwhile, could you please try out these two things?

Go the folder where DMC is installed.

Navigate to this path  ----> <ibm-data-management-console-path>/addons/drs/drs-agent

Open the .env file using --> vi .env

Please change these two variables to the values that I have mentioned and see if the performance improves:
SCAN_INTERVAL=30
QUEUE_LIMIT=50

------------------------------
Abhinandan Kelgere Ramesh
------------------------------

Hello,
I have created APAR IT33123 - drs-agent is consuming 99-100 % cpu all the time to track this issue:
https://www.ibm.com/support/pages/node/3152541
Thanks, Cintia.

------------------------------
Cintia Ogura
------------------------------

Original Message:
Sent: Thu May 28, 2020 11:31 AM
From: Petri Helin
Subject: About drs-agent and its cpu consumption

Hi Abhinandan,

I added these to the .env since the variables were not there before at all:

SCAN_INTERVAL=30
QUEUE_LIMIT=50

After that I recycled the service and gave it 30 minutes to settle, as I could see that the repository Db2 was working on something quite a lot, before I started to browse through the databases and their monitors. Everything feels quite snappy and I am now unable to get drs-agent to consume even more than 1 %.

------------------------------
Petri Helin
------------------------------


Original Message:
Sent: Wed May 27, 2020 05:35 PM
From: Abhinandan Kelgere Ramesh
Subject: About drs-agent and its cpu consumption

Hi Petri. Thanks for reaching out. drs-agent is a process that scans the DB2 instances to generate a Configuration based alerts. It is a process launched by JVM, so JVM should automatically do the load balance on different cores. I will look into it in detail for sure.

As for the drs-agent consuming 80-90 % of the time, it could be because of two reasons:

It could be that scan interval to monitor the DB2 instances is small, especially given the 100s of connections.
Another reason could be that the alerts are getting accumulated given 100s of connections and it takes time to go through all the accumulated alerts (which we have addressed in the next version of DMC).

We will perform a thorough verification on our side by replicating 100s of DB connections, finding the exact root cause and get back to you.

Meanwhile, could you please try out these two things?

Go the folder where DMC is installed.

Navigate to this path  ----> <ibm-data-management-console-path>/addons/drs/drs-agent

Open the .env file using --> vi .env

Please change these two variables to the values that I have mentioned and see if the performance improves:
SCAN_INTERVAL=30
QUEUE_LIMIT=50

------------------------------
Abhinandan Kelgere Ramesh