IBM Security QRadar

 View Only
  • 1.  Docker client certificate expired

    Posted Tue May 11, 2021 04:09 AM
    Edited by Milen Rangelov Tue May 11, 2021 04:18 AM
    Hello,

    Several days ago, suddenly my app deploys started to fail and additionally all my deployed apps stopped working (empty UI content in the QRadar console).

    In the docker logs (journalctl -u docker.service) I noticed errors like those:

    May 11 03:56:05 localhost dockerd[14791]: http: TLS handshake error from 10.126.6.203:44148: tls: failed to verify client's certificate: x509: certificate has expired or is not yet valid

    May 11 03:56:08 localhost dockerd[14791]: http: TLS handshake error from 10.126.6.203:44174: tls: failed to verify client's certificate: x509: certificate has expired or is not yet valid

    May 11 03:56:15 localhost dockerd[14791]: http: TLS handshake error from 10.126.6.203:44274: tls: failed to verify client's certificate: x509: certificate has expired or is not yet valid

    May 11 03:56:25 localhost dockerd[14791]: http: TLS handshake error from 10.126.6.203:44398: tls: failed to verify client's certificate: x509: certificate has expired or is not yet valid


    I decided to check the QRadar-related certificates on my system, by running the following command:

    for i in $(find /etc/conman/tls /etc/traefik/tls /etc/docker/tls /etc/vault-qrd/tls /etc/httpd/conf/certs /etc/pki/ca-trust/source/anchors -type f \( -name "*.cert" -o -name "*.pem" -o -name "*.crt" \));do echo $i; openssl verify -CAfile /etc/pki/tls/cert.pem $i; done

    The output is as follows:

    /etc/conman/tls/conman.cert

    /etc/conman/tls/conman.cert: OK

    /etc/conman/tls/conman_ca.crt

    /etc/conman/tls/conman_ca.crt: OK

    /etc/traefik/tls/traefik.cert

    /etc/traefik/tls/traefik.cert: OK

    /etc/traefik/tls/docker/traefik-client-docker.cert

    /etc/traefik/tls/docker/traefik-client-docker.cert: CN = localhost

    error 10 at 0 depth lookup:certificate has expired

    OK

    /etc/traefik/tls/docker/si-docker_ca.crt

    /etc/traefik/tls/docker/si-docker_ca.crt: OK

    /etc/traefik/tls/traefik_ca.crt

    /etc/traefik/tls/traefik_ca.crt: OK

    /etc/docker/tls/si-docker.cert

    /etc/docker/tls/si-docker.cert: OK

    /etc/docker/tls/registry/docker-client-registry.cert

    /etc/docker/tls/registry/docker-client-registry.cert: CN = localhost

    error 10 at 0 depth lookup:certificate has expired

    OK

    /etc/docker/tls/registry/si-registry_ca.crt

    /etc/docker/tls/registry/si-registry_ca.crt: OK

    /etc/docker/tls/si-docker_ca.crt

    /etc/docker/tls/si-docker_ca.crt: OK

    /etc/vault-qrd/tls/vault-qrd.cert

    /etc/vault-qrd/tls/vault-qrd.cert: CN = localhost

    error 18 at 0 depth lookup:self signed certificate

    OK

    /etc/httpd/conf/certs/cert.cert

    /etc/httpd/conf/certs/cert.cert: OK

    /etc/pki/ca-trust/source/anchors/vault-qrd_ca.pem

    /etc/pki/ca-trust/source/anchors/vault-qrd_ca.pem: OK

    /etc/pki/ca-trust/source/anchors/vault-qrd_ca_int.pem

    /etc/pki/ca-trust/source/anchors/vault-qrd_ca_int.pem: OK

    /etc/pki/ca-trust/source/anchors/conman_ca.crt

    /etc/pki/ca-trust/source/anchors/conman_ca.crt: OK

    /etc/pki/ca-trust/source/anchors/QRadarSAML_ca.crt

    /etc/pki/ca-trust/source/anchors/QRadarSAML_ca.crt: OK

    /etc/pki/ca-trust/source/anchors/si-docker_ca.crt

    /etc/pki/ca-trust/source/anchors/si-docker_ca.crt: OK

    /etc/pki/ca-trust/source/anchors/si-registry_ca.crt

    /etc/pki/ca-trust/source/anchors/si-registry_ca.crt: OK

    /etc/pki/ca-trust/source/anchors/traefik_ca.crt

    /etc/pki/ca-trust/source/anchors/traefik_ca.crt: OK


    So it looks like two of the certificates - docker-client-registry.cert and traefik-client-docker.cert have expired. Additionaly vault-qrd certificate is self-signed (but this apparently is not an issue - the vault-qrd service starts and runs fine).

    I decided to confirm the certificates in question have indeed expired several days ago:

    openssl x509 -text -in /etc/docker/tls/registry/docker-client-registry.cert |grep -i "not"

                Not Before: Feb  3 15:04:04 2021 GMT

                Not After : May  4 15:04:04 2021 GMT


    My question is - is there an easy way to renew the certificates in question?

    Best Regards,
    Milen Rangelov

    ------------------------------
    Milen Rangelov
    ------------------------------


  • 2.  RE: Docker client certificate expired

    Posted Tue May 11, 2021 04:39 AM
    The issue is resolved. I ran this (as suggested):

    rm -rf /opt/qradar/ca/certs/*; /opt/qradar/ca/bin/reset-qradar-ca.sh all --reset

    This recreated all the certificates. Everything works fine now. Thanks a lot!

    ------------------------------
    Milen Rangelov
    ------------------------------



  • 3.  RE: Docker client certificate expired

    Posted Tue December 27, 2022 06:19 AM
    Thank you a lot, Milen! Your suggestion and discussion was super helpful! We had quite similar issue, so your commands werehelpful for our case.

    ------------------------------
    Thomas Jaeger
    ------------------------------



  • 4.  RE: Docker client certificate expired

    IBM Champion
    Posted Wed May 12, 2021 10:45 AM
    Hello Milen,

    if you're gonna restart the vault-qrd service on console/apphost, this will renew the outlined certificates as well. For some reasons, sometimes it's necessary you'll need to restart the responsible services for the app framework, the issue you've mentioned: vault-qrd, traefik, conman and docker. finally sometimes you'll need to restart the affected apps.

    Usually afterwards the app framework is fine again.

    Regards,
    Ralph

    ------------------------------
    Ralph Belfiore
    SIEM Expert
    pro4bizz GmbH
    ------------------------------



  • 5.  RE: Docker client certificate expired

    Posted Mon March 14, 2022 12:13 PM
    Hello, my name is Luca, and i write you because i need your help, is 1 mounth that i try to install a new Qradar's backup on my system but i obtain only the failure; This backup file is created on another qradar's system.
    the qradar's version is 7.3.3 Pack1, and reading the logs file i found the following expired certificates:

    /opt/qradar/…/external-scanner_qradar_ibmcloud_com.crt
    /opt/qradar/…/QRadarSAML.crt
    /opt/qradar/…/tomcat-client-conman.cert
    /opt/qradar/…/traefik-client-docker.cert
    /opt/qradar/…/docker-client-registry.cert
    /opt/qradar/…/tomcat-client-traefik.cert
    /etc/pki/tls/…/localhost.crt
    /etc/…/traefik-client-docker.cert
    /etc/…/docker-client-registry.cert
    /etc/…/tomcat-client-traefik.cert
    /etc/traefik/tls/docker/traefik-client-docker.cert
    /etc/docker/tls/registry/docker-client-registry.cert
    /etc/docker/certs.d/xxxxxxxx.localdeployment:5000/docker-client-registry.cert
    /etc/docker/certs.d/xxxxxxxx:5000/docker-client-registry.cert
    /etc/docker/certs.d/console.localdeployment:5000/docker-client-registry.cert
    /etc/tomcat/tls/traefik/tomcat-client-traefik.cert
    /etc/tomcat/tls/conman/tomcat-client-conman.cert

    What can i do to renew this certificate?

    Best regards
    Luca

    ------------------------------
    Luca Giffi
    ------------------------------



  • 6.  RE: Docker client certificate expired

    IBM Champion
    Posted Tue March 15, 2022 12:02 PM
    Hello Luca,

    thank you for your inquiry. If I understand your description correctly, the backup config restore fails.
    I assume that the new qradar target system has the same version as the version with which the qradar backup was created.
    If this is ensured, it should help with version 7.3.3 FPx to restart the following service:
    - vault-qrd

    This should lead to these internal certificates being renewed and from this moment also being automatically updated in the background by this service in the future...

    Many greetings,
    ralph