A colleague reinstalled the component "Thycotic MemoryMq Site Connector Service (x64)" on the PAM server and since then we have a problem with the Distributed Engine for all sites.

Synchronization of users with AD does not work, access via RDP Launcher using RDP proxy does not work, users cannot log in through accounts from AD.

If I go to Administration > Distributed Engine > Logs, here are the last logs from before reinstalling the "Thycotic MemoryMq Site Connector Service (x64)" on the PAM server.

If I look in Administration > Distributed Engine > Sites and Engines, all engines have last connections within the last few minutes.

If I view the log for a specific engine, the last record is again from before the reinstallation of "Thycotic MemoryMq Site Connector Service (x64)" on the PAM server.

If I view Administration > Directory Services > open an entry for a specific domain > go to Groups and click on a group from the Synchronized Groups list, the Error Connect Failed banner appears. After clicking on Diagnostic, I see error logs

Message: Global Catch: [object Object]
URL: #/admin/directory-services/domain/6/groups
Stack: Connect Failed.

Log from Distributed engine:

2023-10-24 17:17:25,777 [CID:] [C:] [TID:PriorityScheduler Elastic Thread @ Lowest] INFO Thycotic.MessageQueue.Common.Wrappers.Factories.PermanentConsumerWrapperFactory - All basic consumers have started successfully. - (null)

2023-10-24 17:17:25,808 [CID:] [C:] [TID:PriorityScheduler Elastic Thread @ Lowest] INFO Thycotic.MessageQueue.Common.Wrappers.Factories.PermanentConsumerWrapperFactory - All blocking consumers have started successfully. - (null)

2023-10-24 17:17:35,691 [CID:] [C:] [TID:PriorityScheduler Elastic Thread @ Lowest] WARN Thycotic.MessageQueue.Common.Wrappers.Factories.PermanentConsumerWrapperFactory - Unable to start consumer [Thycotic.DE.Feature.SS.PasswordChanging.Areas.SecretPrivilegeChangePasswordConsumer]. Will retry... - (null)

If you need additional logs for the answer, let me know which ones and from where.

Does anyone have any advice on where to look and what to check?

What is the correct procedure if I reinstall "Thycotic MemoryMq Site Connector Service (x64)" on the PAM server and "Thycotic Distributed Engine Service (x64)" on the Distributed Engine server?

Hello,

I am the colleague who reinstalled the MemoryMQ service, because all the DEs were failing to communicate with it - there were dozens of ESTABLISHED connections from each DE on 443 or 8672 ports according to netstat, but they reported in their SSDE.log:

Could not connect to net.tcp://PAMDNSAddress:8672/. The connection attempt lasted for a time span of 00:00:01.0312644. TCP error code 10061: No connection could be made because the target machine actively refused it PAMIPAddress:8672

This started since we changed the PAMIPAddress for PAMDNSAddress from the first node of the PAM cluster to IP Address of the load balancer. The reinstallation of MemoryMQ was attempt to fix the issue, since it seemed the TCP connections were successful and just the MemoryMQ was refusing to communicate.

But since the issue started since the switch to load balancer, we later switched back to IP address of the first node and at first it seemed to help, because logs were being updated on the PAM server. But the next day users who were able to log on with their AD accounts in the morning were later unable to do so and we also found the synchronization of AD groups didn't work. Also RDP proxying was broken.

After some time looking for information how to make the DE service reinitialize its connection to MemoryMQ, we decided to try to reinstall DE service, which I did so after deactivating, removing, deleting and adding it again in PAM server - after the activation it says what Martin Hansgut posted and then continues to spam the SSDE.log with the warnings about not being able to start all the various consumers. And the communication still doesn't work.

Any information how to fix the issue or troubleshoot the communication with DE is appreciated.

Regards

Hello,

Check the sync secret you defined in the site connector. If the credential is incorrect, LDAP sync will not work. Even if the heartbeat value shows as successful, the credential might have expired, please test it.

Also, ensure that the connector you are using is selected for your site under Settings > Internal Site.

Finally, I recommend using RabbitMQ as the site connector.