MQ

 View Only
Expand all | Collapse all

MQ Service Compromise -> documented process for auditing?

  • 1.  MQ Service Compromise -> documented process for auditing?

    Posted Tue September 17, 2024 11:36 AM

    Hello,

    Does anybody know , whether there is a formal or at least an informal process posted for restoring MQ service (on Multiplatforms: Windows, AIX, Linux) after it has been compromised?

    Such document including downtime and MQ service unavailability considerations would be valuable against any organization's cybersecurity and audit divisions claims..   

    P.S.

    Do you have in mind whether MQ service has been actually compromised in real life? 

    I am asking that, because I haven't had such experience with MQ for  a long time (at least 15 years..)

    Furthermore, after searching in the web, i have found that IBM has addressed via Fix the denial-service-attack-cve-2023-26285  posted at URL:

    https://www.ibm.com/support/pages/security-bulletin-ibm-mq-vulnerable-denial-service-attack-cve-2023-26285 

     

    Thanks in advance,

    Chers, Nick.

    Any advise on the following will be much appreciated.

    Cheers, Nick.



    ------------------------------
    Nick Dakoronias
    ------------------------------


  • 2.  RE: MQ Service Compromise -> documented process for auditing?

    Posted Thu September 19, 2024 08:48 AM

    Hi Nick,

    First of all, I think we need to define what a 'compromised MQ service' is.
    Does it mean an unauthorised change to the configuration, change in MQ rights on certain objects, modification of binaries, etc...? 
    If the server hosting MQ has been compromised in the classic sense of the term (intrusion by an outside party and potential modification of the configuration and/or binaries), the process is simple: delete the compromised server and deploy a new one.

    In this case, of course, you need to have :
    - MQ binaries
    - Complete MQ configuration in MQSC format (dmpmqcfg)
    - Any additions to the qm.ini file
    - Certificate stores
    - Scripts launched by the triggering process
    Please note that the QMID of the Queue Manager will change (important if MQ cluster) and the DQM channels will have to be resynchronised.

    If it is only a question of modifying certain objects, re-applying a configuration export may help, but it will not remove objects or permissions added by the 'compromision'.

    In terms of preventive solutions, you need to secure access to the :
    - CHLAUTH on SVRCONN channels
    - MCAUSER (possibly dynamic) on SVRCONN and RECEIVER channels
    - Symmetrical TLS encryption on SVRCONN and DQM channels
    Enabling configuration events provides an audit trail, so you can see who changed what, and when, with a before and after image.



    ------------------------------
    Luc-Michel Demey
    DEMEY CONSULTING
    lmd@demey-consulting.fr
    #IBMChampion
    ------------------------------



  • 3.  RE: MQ Service Compromise -> documented process for auditing?

    Posted Thu September 19, 2024 11:57 AM

    Hi Luc-Michel,

    At First many thanks for your time and response.

    I fully agree with the listed items (i.e. binaries, triggers, SSL store, qm.ini, qm config, ..) you have pinpointed  in case of MQ server (hosting QMs)  is compromised.

    But my question actually refers to the case where specifically the MQ Windows (admin/startup) service which is configured as service account in Active Directory domain (where  MQ host is joined) is being compromised by an external attacker who will  gain access to the password of this account.

    The above hypothetical case has been set by cybersecurity & audit divisions in a bank organization, asking for official (vendor) password recovery process documentation.

    Furthermore this request is also related to another recent (5 days old) post subjected  "Does MQ Windows Service Support or is it compatible with group Managed Service Account (gMSA)?" considering that their plan is to move all Windows (domain) service accounts to gMSA leveraging its auto password rotation feature..

    Relying on my experience related to both Microsoft AD & MQ technologies, my perception is that  not only MQ Service password can be cracked but also all Microsoft Windows Services credentials can be hacked by an external attacker who will gain access to the  AD SAM & NTDS.DIT databases (in PDCs), where all these are stored. In that case it is obvious that the whole AD might be at risk of compromise and possible collapse, so MQ will be down in the priority list of business continuity. 

    The same applies to gMSA credentials risk for hacking, given that they are all stored in PDCs (AD Domain Controllers) KDC (Key Distribution Service) object. So my point is that everything can be hacked. There is no absolute safety when a hacker decides to perform a dark penetration.

    Btw, there publicly available articles & posts (known to me) that provide full guidance & methods to compromise both service accounts & gMSA credentials..

    At last, the auto pwd rotation feature provided by gMSA, is not compliant with the fact that Windows MQ service account credentials need to be provided during MQ setup process (i.e. new MQ instance). MQ admins cannot retrieve the MQ service password from gMSA store, since only Enterprise Admins and MQ Admins can access that store.

    I would like to have your comments (if any) or further commendations.

    Cheers, Nick. 

          

       

    To be honest       

      



    ------------------------------
    Nick Dakoronias
    ------------------------------



  • 4.  RE: MQ Service Compromise -> documented process for auditing?

    Posted Fri September 20, 2024 09:33 AM

    With Windows, if the server belongs to an AD domain, the MQ service must run with an AD account, which in some cases causes difficulties.
    If the password for the service account has to be changed regularly, this is possible, but all the MQ servers involved have to be updated at the same time, which is complex.
    That's why it's a good idea to have several service accounts, at least one per environment.
    After that, if security constraints are very tight for an organisation, it's likely that Windows will be used only for workstations, and that the MQ servers will be running Linux or another Unix operating system. 
    In this case, there would be no need for a service account, AD access, changing passwords, etc.

    Several of my customers using MQ on Windows have had 'disasters' linked to a change in AD accounts. 
    Examples include:
    - switching from local accounts to global accounts, thus changing the SID, thus losing all MQ authorisations.  
    - when installing a new MQ server, the IT team could not find the service account password and decided to change it. All the production servers were using this service account ...



    ------------------------------
    Luc-Michel Demey
    DEMEY CONSULTING
    lmd@demey-consulting.fr
    #IBMChampion
    ------------------------------



  • 5.  RE: MQ Service Compromise -> documented process for auditing?

    Posted Mon September 23, 2024 10:11 AM
    Edited by NICK DAKORONIAS Mon September 23, 2024 10:18 AM

    Hi Luc-Michel,

    I fully agree with your concerns.

    Fro the specific customer, there are  MQ servers deployed on RHEL (Linux) and AIX (UNIX) including TEST/QA & PROD environments, which are -fortunately- not impacted by this gMSA stuff which is dependent on Microsoft AD / Domain accounts technology. 

    After discussing with several application owners from different business units (i.e. cards, swift, etc) their major concern is about regular pwd changes, as you mentioned, in their own service accounts (triggered by their APIs) .  They are wondering whether their application will be still able to connect against target MQ resources every time their corresponding service account password changes... This scenario needs to be tested for each app in QA/UAT, before goes to the production.

    Cheers, Nick.



    ------------------------------
    Nick Dakoronias
    ------------------------------



  • 6.  RE: MQ Service Compromise -> documented process for auditing?

    Posted Fri September 20, 2024 03:02 PM

    I don't understand this comment by you:

    But my question actually refers to the case where specifically the MQ Windows (admin/startup) service which is configured as service account in Active Directory domain (where  MQ host is joined) is being compromised by an external attacker who will  gain access to the password of this account.

    Years ago, when I was learning about MS AD, if I remember correctly, I was told that the passwords to AD users were stored encrypted using a 1-way encryption mechanism. i.e. If you forgot your password, there was no way of looking it up.

    So, are you saying that an AD Admin with proper credentials/authorization can look up the password of an account? Hence, because of this, a hacker, if they can gain access, can also look up passwords for any account in MS AD? Because that doesn't seem right.

    later

    Roger



    ------------------------------
    Roger Lacroix
    CTO
    Capitalware Inc.
    London Canada
    https://capitalware.com
    ------------------------------



  • 7.  RE: MQ Service Compromise -> documented process for auditing?

    Posted Mon September 23, 2024 10:46 AM

    It is true that MS AD creates one-way cryptographic hashes of the passwords that are used in authentication exchanges.  While those technically cannot be reversed mathematically, it is possible once those hashes are obtained to test large numbers of common (and not so common) passwords against those hashes offline using GPU farms to determine the original password.   The most common tool used for that is probably Hashcat - a google search will return more information.  There are a variety of techniques used to collect the hashes. 

    Bad passwords are common, and knowing the password rules can actually result in a smaller set of passwords to use in the hash attack.  Ask yourself how often you just add a special character or number to the end of your password when you get the message that a special character or number is required?  How often is that a '$', '#' or '1'?



    ------------------------------
    Vincent Greene
    IT Consultant
    Technology Expert labs
    IBM
    Vincent.Greene@ibm.com


    The postings on this site are my own and don't necessarily represent IBM's positions, strategies or opinions.
    ------------------------------



  • 8.  RE: MQ Service Compromise -> documented process for auditing?

    Posted Mon September 23, 2024 04:47 PM

    Hi Vincent,

    True, but you are glossing over an important point, that the hacker requires BOTH the encrypted password and the hash. 

    Most online companies now require a more complex password compared to 10 or 15 years ago.  Most login web sites state something like:

    • It must be at least 8 characters in length.
    • It must contain a lowercase character (a-z).
    • It must contain an uppercase character (A-Z).
    • It must contain a numeric digit (0-9).
    • It must contain a special character. i.e. !\"#$%&'()*+,-./:;><=?@[\\]^_`{|}\

    Also, if you follow Komando's password length recommendations (which I posted on my blog ) of a minimum of 11 characters (min. 13 is better IMO), then hackers will not waste their time/resources attempting to crack your password.

    later 

    Roger



    ------------------------------
    Roger Lacroix
    CTO
    Capitalware Inc.
    London Canada
    https://capitalware.com
    ------------------------------



  • 9.  RE: MQ Service Compromise -> documented process for auditing?

    Posted Mon September 23, 2024 06:10 PM

    No, there are just multiple types of hashes and multiple uses for those hashes.  I was thinking mostly of NTLMv2 hashes captured using Responder, which presumes a hacker is on the internal network and able to intercept.  If they get one of those, they can use software to "brute force" it.  I don't need access to the encrypted passwords from the AD controller, although its not at all uncommon for pentesters to access that as well when testing internal controls.

    Password length and complexity rules are a nice start, but 'P@55w0rd!' meets all of the rules and will get cracked in less than a minute.  If you require 13 characters, some significant portion of the population is going to just use  'P@55w0rd!1234' instead.   I know we would all love to say that IT professionals would never do such a thing, but we also all know at least one that not only would, but does regularly.

    I think Nick's point earlier today is the important one.  We need to presume it is possible for the password to get exposed and address the issues .  Its a good question.  It does nobody any good to dismiss it.



    ------------------------------
    Vincent Greene
    IT Consultant
    Technology Expert labs
    IBM
    Vincent.Greene@ibm.com


    The postings on this site are my own and don't necessarily represent IBM's positions, strategies or opinions.
    ------------------------------



  • 10.  RE: MQ Service Compromise -> documented process for auditing?

    Posted Mon September 23, 2024 07:02 PM
    Edited by Roger Lacroix Tue September 24, 2024 05:20 PM

    Hi Vincent,

     I was thinking mostly of NTLMv2 hashes captured using Responder, which presumes a hacker is on the internal network and able to intercept.

    True and it has happened because of phishing scams.

    > If you require 13 characters, some significant portion of the population is going to just use  'P@55w0rd!1234' instead.

    Some people will throw their hands in the air but if you look at Komanda's chart, mixed case with numbers & special characters will require 2 million years (brute force attack) to crack, so it may look bad but really is still quite secure.

    Update (09/24/24):  NordPass (a VPN company) has published a list of the top 200 most used passwords here. Nobody should ever use a password from that list or a longer password that is made up of a password from the list.  As they point out on their website, those passwords, when encrypted, can be cracked in seconds.  So, to be safe, do not use any of those passwords.

    > I think Nick's point earlier today is the important one.  We need to presume it is possible for the password to get exposed and address the issues .  Its a good question.  It does nobody any good to dismiss it.

    Oh no, I wasn't dismissing it. I was questioning the availability of plain text passwords of an AD account. The security issue is really not an MQ issue because it applies to the credentials of any software running as a Windows service. 

    A 2 prong attack is the only recourse: (1) as Luc-Michel said, separate AD accounts for each environment or Windows server and (2) substantial mitigation around AD administrative accounts.

    later

    Roger



    ------------------------------
    Roger Lacroix
    CTO
    Capitalware Inc.
    London Canada
    https://capitalware.com
    ------------------------------



  • 11.  RE: MQ Service Compromise -> documented process for auditing?

    Posted Mon September 23, 2024 11:35 AM

    Have you paid the proper attention to my whole particular response in order to get  the whole idea?  (asked gently) .

    As I have mentioned in that response, the specific question that you didn't understand (you highlighted it in blue) has been asked -exactly as is- by customer's cybersecurity & audit divisions. To be honest, I don't know how familiar are these guys with Microsoft AD technology, but their question makes sense.. Let me  explain what i mean by that, by leveraging  my expertise as AD Certified member of Microsoft Support Services for a long time,  along with my education & skills on Cryptography / Key Management.         

    This question is not actually translated in the way you did, meaning that an AD enterprise/domain admin with proper access can look up the  pwd store, so a hacker can do the same... No way by all means. 

    The question and my comments after, refer purely to unexpected external attacks, somehow relevant to the well known planned "black penetration scenarios".

    It is certain that passwords are stored in AD SAM & NTDS.DIT dbases as hash digests. Microsoft names  this hash algorithm  OWF (One way function). The AD account pwd is stored using LM (LAN Manager) hash in older Windows OS versions   and NT (New Tecnhology) hash (DES, AES-256) in recent Windows  OS versions. The authentication is being performed upon kerberos authentication mechanism against KDC (Key distribution Center) hosted on Domain Controller (DC).  So it is really difficult for a local AD Admin to break the hash and perform internal intrusion, unless he has has cracking skills..

    But in this case we refer to external attackers, which are well skilled hackers and have developed methods of retrieving gMSA pwds.

    In brief what an external attacker usually tries to achieve, is  to get access to any AD machine (client, server, ..) with PowerShell installed, so that he can get the full list of service accounts with just a simple PowerShell command. If he achieves that he can then look into specific attributes in order to get gMSAs and by looking into SPNs he can find where these gMSA are installed. After that he can target the systems found in the SPNs in order to do the ...rest of the job..

    FYI, there are Powershell modules  -available in worldwide powershell hacking programming  community - developed just to retrieve gMSA secrets.. I don't want to further elaborate on this, for obvious reasons.  So, at last the specific question makes sense in a way that an external experienced / skilled attacker can  decide to compromise the gMSA secrets.

    But for the benefit of MQ community users, I think we should avoid to further elaborate on this and instead focus on how the frequent pwd change on gMSA can affect the stability of Windows MQ service (if they decide to use it), especially on  Production MQ (Microsoft) failover setups hosting 24 x 7 flows. In other words, how compliant is the Windows MQ service against the gMSA, since as far as i know there is no such evidence in IBM documentation.  If there is something relevant, pls keep me posted. 

     Rgds, Nick.

       



    ------------------------------
    Nick Dakoronias
    ------------------------------



  • 12.  RE: MQ Service Compromise -> documented process for auditing?

    Posted Mon September 23, 2024 06:34 PM

    Hello Nick,

    Have you paid the proper attention to my whole particular response in order to get  the whole idea?  (asked gently) .

    As I have mentioned in that response, the specific question that you didn't understand ...

    Hummm. You do realize that there are a variety of people participating in this forum: IBMers, IBM Champions & regular MQ users who all do it out of their free time to help individual users like yourself. Posting a toxic response is not how to ingratiate yourself with the community. Given you are a Kyndryl consultant and this is a public forum, I'm surprised at your response. Also, explaining security to someone who creates and sells MQ security products (encryption/authentication) is the definition of mansplaining.

    Your original post was very generic and wide-ranging, asking for help regarding MQ being compromised on various distributed platforms (Windows, AIX, Linux) and I bet most people took it as a newbie looking for someone else to do their homework. Only after Luc-Michel took a shot at answering it did you get to the real/heart of your concern.

    If you take Luc-Michel's suggestion of having separate MQ UserIds for each environment or Windows server and the Enterprise MSAD-SysAdmin will work with the MQAdmin regarding MQ account passwords, then only about 5% of the issue is related to MQ and 95% is related to MS AD or Windows servers/clients in general.  Hence, as pointed out by numerous websites, the best solution involves strong mitigation practices. Here are several websites with excellent mitigation strategies:

    later

    Roger



    ------------------------------
    Roger Lacroix
    CTO
    Capitalware Inc.
    London Canada
    https://capitalware.com
    ------------------------------



  • 13.  RE: MQ Service Compromise -> documented process for auditing?

    Posted Tue September 24, 2024 10:53 AM
    Edited by NICK DAKORONIAS Tue September 24, 2024 10:53 AM

    Hummm. You do realize that there are a variety of people participating in this forum: IBMers, IBM Champions & regular MQ users who all do it out of their free time to help individual users like yourself. Posting a toxic response is not how to ingratiate yourself with the community. Given you are a Kyndryl consultant and this is a public forum, I'm surprised at your response. Also, explaining security to someone who creates and sells MQ security products (encryption/authentication) is the definition of mansplaining.

    Your response is offensive and full of insults against me.  You accused me that i am a regular MQ user trying to help individuals on my free time, making a post for newbies.  I haven't seen such errogant & offending response like yours, throughout my 35 years in IT. Try to behave yourself with people you don't know.

    You don't know my background and what i have been dealing with, all these years.  I really don't want to further engage and elaborate on this.

    Just FYI, you really misunderstood my perception. I have tried gently to explain the AD security part with intention to contribute on the thread and not to promote myself or something like that. This is not compliant with my ethical standards and the way i was raised up. Believe me, I have no idea about the background of any community user who read my post, so I have tried to explain some topics for users who might don't have such huge expertise on this topic. My response to you is also read by other community users, who might want to elaborate. So, why did you react like that?  

    My intention from the beginning was to find a guidance or documentation evidence on MQ service compromise, as per customer's cybersecurity/audit divisions request. But during the elaboration of responses given by  Luc-Michel & Vincent (thank you both), I have also thought to ask about how compliant Windows MQ Service against AD gMSA auto pwd rotation, considering technical impacts such as unique SID per service account (mentioned by Luc-Michel) or whether Windows MQ service uses its pwd to re-authenticate while it is up & running, because  if it is the case, then gMSA auto pwd rotation could not be the case.  



    ------------------------------
    Nick Dakoronias
    ------------------------------



  • 14.  RE: MQ Service Compromise -> documented process for auditing?

    Posted Tue September 24, 2024 05:35 PM

    Hello Nick,

    I'm French-Canadian & my French blood tends to make me speak bluntly, like responding to an insult with an insult (as you pointed out).

    And yes, this part of the thread should end here.

    Later
    Roger



    ------------------------------
    Roger Lacroix
    CTO
    Capitalware Inc.
    London Canada
    https://capitalware.com
    ------------------------------



  • 15.  RE: MQ Service Compromise -> documented process for auditing?

    Posted Wed September 25, 2024 05:51 AM

    Hi Roger,

    In my last reply to you, I have tried hard to make clear that my intention was to elaborate and contribute on the thread rather than to insult you. It is not my style to offend people. I have never done it and i will never do it. It is against my ethical standards. So, i assume that you just misunderstood my purpose.

    Considering the above, I will take your last response as an unintended misperception of my approach and i agree to leave this (toxic) part behind us. At the end, all of us working in IT business (generally speaking), i assume we are willing to collaborate with each other, aiming to support and contribute on this business taking into consideration each individual's added value i.e. (skills, experience, etc).

    Now, let's come to the "core stuff".   In my last response, i was wondering about how compliant Windows MQ Service could be against AD gMSA auto pwd rotation feature, considering technical impacts such as unique SID per service account (mentioned by Luc-Michel) or whether Windows MQ service uses its pwd to re-authenticate while it is up & running.

    Any comment on this by anybody that can share his thoughts will be much appreciated, considering that customer's (major local bank organization) cybersecurity/audit dpts, asks for evidences/justifications, so that Windows MQ service NOT to be "onboarded" on gMSA account.

    Rgds, Nick.

     

     

     



    ------------------------------
    Nick Dakoronias
    ------------------------------



  • 16.  RE: MQ Service Compromise -> documented process for auditing?

    Posted Thu September 26, 2024 02:06 AM

    Hi Nick,

    Have you set up a gmsa test bed environment for MQ? How does it work when you change the password? How does it work when you restart the service?



    ------------------------------
    Francois Brandelik
    ------------------------------



  • 17.  RE: MQ Service Compromise -> documented process for auditing?

    Posted Thu September 26, 2024 04:13 AM

    Hi Francois,

    Thanks for your response.

    This is exactly what we have mutually agreed along with Bank's IT infra (Wintel support team). We have decided to initiate tests first at DEVELOPMENT environment and then go for TEST/QA environments, each one with its own Windows MQ service account and hosted at different AD Domains (internal & external DMZ) not trusted each other. 

    It will take some time to "onboard" MQ service at gMSA for each environment (domain) and evaluate the results.
    I will keep you posted when i have some feedback.

    Rgds, Nick. 

     



    ------------------------------
    Nick Dakoronias
    ------------------------------