AIOps

 View Only
  • 1.  OpenShift pods in the openshift-marketplace namespace with status of ImagePullBackoff

    Posted Thu December 01, 2022 10:04 AM
    vSphere 6.7U3
    Openshift 4.6.16, 4.7.0, 4.7.60, or 4.11.9 (all have the exact same behavior)
    Installer node is running CentOS 7

    I ultimately want to install Watson AIOps on OpenShift running on vSphere, but after installing OpenShift, the pods in the openshift-marketplace namespace remain in a state of ImagePullBackoff. Using 'oc get pods -A | grep ImagePullBackOff' gets me a list of those pods. I can get details with 'oc describe -n openshift-marketplace pds/redhat-marketplace-jcx9n' to see the following error:

    Events:
      Type 	Reason      	Age                 	From           	Message
      ---- 	------      	----                	----           	-------
      Normal   Scheduled   	7m47s               	default-scheduler  Successfully assigned openshift-marketplace/community-operators-6qx58 to ovv-rq4kn-master-0 by ovv-rq4kn-master-1
      Normal   AddedInterface  7m43s               	multus         	Add eth0 [10.129.0.48/23] from openshift-sdn
      Normal   Pulling     	6m21s (x4 over 7m43s)   kubelet        	Pulling image "registry.redhat.io/redhat/community-operator-index:v4.11"
      Warning  Failed      	6m21s (x4 over 7m43s)   kubelet        	Failed to pull image "registry.redhat.io/redhat/community-operator-index:v4.11": rpc error: code = Unknown desc = unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/articles/3399531
      Warning  Failed      	6m21s (x4 over 7m43s)   kubelet        	Error: ErrImagePull
      Warning  Failed      	5m55s (x6 over 7m42s)   kubelet        	Error: ImagePullBackOff
      Normal   BackOff     	2m30s (x21 over 7m42s)  kubelet        	Back-off pulling image "registry.redhat.io/redhat/community-operator-index:v4.11"
    

    Does anyone have a pointer to fix this? I downloaded my pull secret from Red Hat, and the install completes successfully, so it sure seems like it's correct. However, Google is failing me when trying to find a way for me to test or validate my pull secret. I can use docker to login to the repository with my email and password successfully, but I can't find a way to actually test my pull secret file to verify that it's correct.

    Also, some of the other pods in the same ImagePullBackOff state only have these events:

    Events:
      Type 	Reason   Age                  	From 	Message
      ---- 	------   ----                 	---- 	-------
      Normal   Pulling  50m (x48 over 4h30m) 	kubelet  Pulling image "registry.redhat.io/redhat/community-operator-index:latest"
      Normal   BackOff  5m7s (x1169 over 4h30m)  kubelet  Back-off pulling image "registry.redhat.io/redhat/community-operator-index:latest"
      Warning  Failed   7s (x1192 over 4h30m)	kubelet  Error: ImagePullBackOff
    ​


    While my cluster is on a NAT network, it does have access to the Internet.

    Anyone have any ideas? 

    Thanks,
    Frank



    ------------------------------
    Frank Tate
    Gulfsoft Consulting
    https://www.gulfsoft.com
    AIOps Experts. Contact us for implementation help.
    ------------------------------


  • 2.  RE: OpenShift pods in the openshift-marketplace namespace with status of ImagePullBackoff

    Posted Fri December 02, 2022 06:20 PM
    Hi Frank. That's an odd one. If it's happening on all of your builds then there must be something wrong with the key or how it's being injected during the install. You can check your current pull key with:

    oc get secret pull-secret -n openshift-config --template='{{index .data ".dockerconfigjson" | base64decode}}'

    You can set the new pull secret with this:

    oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=/path/to/downloaded/pull-secret 


    If I recall correctly, install-config.yaml isn't checked exhaustively during install so maybe double check the syntax being used. In this example, I'm using a UPI install with the pull key and a SSH key being set in additional files and read into install-config.yaml with the values enclosed in single quotes. 
    cat <<EOF > install-config.yaml
    apiVersion: v1
    baseDomain: systemsmanaged.co.uk
    compute:
    - hyperthreading: Enabled
    name: worker
    replicas: 0
    controlPlane:
    hyperthreading: Enabled
    name: master
    replicas: 3
    metadata:
    name: lab4
    networking:
    clusterNetworks:
    - cidr: 10.254.0.0/16
    hostPrefix: 24
    networkType: OpenShiftSDN
    serviceNetwork:
    - 172.30.0.0/16
    platform:
    none: {}
    pullSecret: '$(< ~/.openshift/pull-secret)'
    sshKey: '$(< ~/.ssh/helper_rsa.pub)'
    EOF



    ------------------------------
    Mark Delaney
    SYSTEMSMANAGED Ltd
    ------------------------------



  • 3.  RE: OpenShift pods in the openshift-marketplace namespace with status of ImagePullBackoff

    Posted Sat December 03, 2022 10:21 AM
    Thanks, Mark.
    I posed this same question on the openshift subreddit (https://www.reddit.com/r/openshift/comments/zap8zs/openshiftmarketplace_pods_in_imagepullbackoff/) and I'm making some progress. The command you've provided to change the pull secret is very helpful.

    Basically, so far I've found that there is a problem with the pull secret file I pulled down from console.redhat.com/openshift , and I can see the userids and passwords by b64-decoding the strings in that file. Also (from the reddit thread) I was given a skopeo command I can use to unit test my pull secret. I'm currently trying different values in the pull secret file to try to find something that works.

    Thanks again!

    ------------------------------
    Frank Tate
    Gulfsoft Consulting
    https://www.gulfsoft.com
    AIOps Experts. Contact us for implementation help.
    ------------------------------



  • 4.  RE: OpenShift pods in the openshift-marketplace namespace with status of ImagePullBackoff

    Posted Sat December 03, 2022 05:56 PM
    Edited by Frank Tate Sat December 03, 2022 05:59 PM
    OK, I got it to work by base64-encoding my Red Hat Developer credentials into the pull secret. The format of that file from redhat is:

    {"auths":
    {"cloud.openshift.com":{"auth":"tZGV2K2ZyYW5rdGF0ZWd1bGZzb2Z0Y29tMWFvOHBqeTg4ZHU1cnozZHpkMWZiZmx6bH8=","email":"your@email.com"},"quay.io":
    {"auth":"tZGV2K2ZyYW5rdGF0ZWd1bGZzb2Z0Y29tMWFvOHBqeTg4ZHU1cnozZHpkMWZiZmx6bH8=","email":"your@email.com"},
    "registry.connect.redhat.com":
    {"auth":"XxE3OTYwN3x1aGMtMWFvOFBqeTg4ZFU1clozRFpkMWZCRmxaTFdOOmV5SmhiR2NpT2lKU1V6VXhNaUo5LmV5SnpkV0lpT2lKaE1qUTFPVEl3WkdNek1HSTBNV1kxWVdSbE4yUXhabVpqWWpVMU1qTmlZU0o5LnY5T185eXJqa0I4WWY3Z2Z2LWZxRkJ1NFRyR0VNcGllVXQ5aVcwQktrTDM5RnBpcHhtX1gyTVlvTW9laklvRTIyMzNvTHN3WHdpMlZ5S1BQYWFLMTZZcFVRYnRlR0ctaVBE","email":"your@email.com"},
    "registry.redhat.io":
    {"auth":"XxE3OTYwN3x1aGMtMWFvOFBqeTg4ZFU1clozRFpkMWZCRmxaTFdOOmV5SmhiR2NpT2lKU1V6VXhNaUo5LmV5SnpkV0lpT2lKaE1qUTFPVEl3WkdNek1HSTBNV1kxWVdSbE4yUXhabVpqWWpVMU1qTmlZU0o5LnY5T185eXJqa0I4WWY3Z2Z2LWZxRkJ1NFRyR0VNcGllVXQ5aVcwQktrTDM5RnBpcHhtX1gyTVlvTW9laklvRTIyMzNvTHN3WHdpMlZ5S1BQYWFLMTZZcFVRYnRlR0ctaVBE","email":"your@email.com"}}}​

    The "auth" values are base64 encoded versions of the string "yourUserName:yourPassword" (just like an HTTP BasicAuthentication header. (The ones above won't work since I made sure they're not valid.) So I encoded my Red Hat ID and pass into the redhat registry entries, and all is good now. Thanks for the replies @Mark Delaney and @Veronica Boychuk.​​

    ------------------------------
    Frank Tate
    Gulfsoft Consulting
    https://www.gulfsoft.com
    AIOps Experts. Contact us for implementation help.
    ------------------------------



  • 5.  RE: OpenShift pods in the openshift-marketplace namespace with status of ImagePullBackoff

    Posted Fri December 02, 2022 10:51 PM

    Hi Frank,

    (Before you start to consider installing  Cloud Pak for Watson AIOps - please only select a Supported Openshift version as outlined in our docs, appropriate for the version of AIOPs you are installing: ibm.com/docs/en/cloud-paks/cloud-pak-watson-aiops/3.5.1?topic=requirements-supported-openshift-versions )

    The first thing I'd suggest checking is the access article linked in the error message: https://access.redhat.com/RegistryAuthentication, as registry.redhat.io requires a Red Hat login to access. Seems like you're missing those credentials in your global pull secret, or they're not quite right.

    If they aren't there, try adding 2 entries to the Secret "pull-secret" in your openshift-config Namespace, for the Registry Server addresses "registry.connect.redhat.com" and "registry.redhat.io". Use your Red Hat username and password. Email is likely not a required field.

    If you're using the OCP web console, just right click the Secret and hit Edit Secret. If you're using the CLI, try https://docs.openshift.com/container-platform/4.10/openshift_images/managing_images/using-image-pull-secrets.html#images-update-global-pull-secret_using-image-pull-secrets

    There is a possibility you will need to add a quay.io entry as well, but I'm not sure. 

     

    Please try that out and let me know how it goes.

    ~~~

    Veronica Boychuk

    Cloud Pak for Watson AIOps

    Install Developer



    ------------------------------
    Veronica Boychuk
    ------------------------------



  • 6.  RE: OpenShift pods in the openshift-marketplace namespace with status of ImagePullBackoff

    Posted Sat December 03, 2022 10:28 AM
    Thanks for the reply, Veronica. The editor messed up the link you provided such that you get a 404 error if you click on it. The correct link is https://www.ibm.com/docs/en/cloud-paks/cloud-pak-watson-aiops/3.5.1?topic=requirements-supported-openshift-versions . I initially found other docs that stated that OpenShift 4.6 and 4.7 were the only ones supported, so that's why I tried those versions first. Thank you for the corrected information.

    What I've found so far is that the pull secret isn't quite right, as you suggested. I'm getting some good help in this reddit thread: https://www.reddit.com/r/openshift/comments/zap8zs/openshiftmarketplace_pods_in_imagepullbackoff/ . I can now use 'skopeo inspect --debug --authfile filename image' to unit test my pull secret with different values.

    Frank

    ------------------------------
    Frank Tate
    Gulfsoft Consulting
    https://www.gulfsoft.com
    AIOps Experts. Contact us for implementation help.
    ------------------------------