Cloud Pak for Data

Cloud Pak for Data

Come for answers. Stay for best practices. All we’re missing is you.

 View Only

  • 1.  Failed to Docker Login

    Posted Thu May 14, 2020 12:37 AM
    Hello,

    Being sure when the issue began though,
    These days, I got to fail to login to docker now.

    # docker login $(oc get routes docker-registry -n default -o template={{.spec.host}}) -u $(oc whoami) -p $(oc whoami -t)
    Error response from daemon: Get https://docker-registry-default.apps.xxxx.com/v1/users/: dial tcp xx.xx.xx.xxx(LB's IP):443: connect: connection refused

    1. Do you know how to investigate this cause?
    2. Originally, where is docker? Is that /var/lib/docker(or containers) created on each nodes? 

    Kind regards,
    Chris

    ------------------------------
    Chris
    ------------------------------

    #CloudPakforDataGroup


  • 2.  RE: Failed to Docker Login

    Posted Thu May 14, 2020 04:38 AM
    Hi,

    It's really an OpenShift question, but let's try this ( I assume that you are on OCP 3.x based on the command):

    1) oc registry login

    2) check the pods in the default namespace - this is where docker is running.
    Maybe your docker pod is down, or potentially you have an issue with some of the router pods.
    Have a look at the docker-registry pod's logs.

    3) Check if:
    kubectl get po --no-headers --all-namespaces -o wide| grep -Ev '([[:digit:]])/\1.*R' | grep -v 'Completed'
    returns anything.

    Thanks

    ------------------------------
    TOMASZ HANUSIAK
    ------------------------------

    Original Message


  • 3.  RE: Failed to Docker Login

    Posted Thu May 14, 2020 10:36 AM
    1. Can you try log into master node and try connecting to docker registry using the command:
    docker login -u $(oc whoami) -p $(oc whoami -t) docker-registry.default.svc:5000

    2. If this succeeded, your registry is fine,  Then problem is on the external routing.

    Check, if all the the router pods are up.

    oc get routes -n default

    ------------------------------
    Lakshmana Ekambaram
    IBM
    Lenexa KS
    913-599-8756
    ------------------------------

    Original Message


  • 4.  RE: Failed to Docker Login

    Posted Thu May 14, 2020 08:57 PM
    Hello,

    Thank you for your replys.
    I logined to docker successfully using docker login -u $(oc whoami) -p $(oc whoami -t) docker-registry.default.svc:5000.
    Is that this right way of logging in?

    When I inputed this command, I found that one of dockergc pods restarted about every 10s randomly.
    Do you know whether it is one of the designated system of openshift?
    # oc get po |grep docker
    docker-registry-1-g57w9 1/1 Running 1 22h
    dockergc-4kbmb 1/1 Running 136 113d
    dockergc-4vk9m 1/1 Running 127 113d
    dockergc-dzjx7 1/1 Running 129 113d
    dockergc-knjzn 1/1 Running 130 113d
    dockergc-mxlj8 1/1 Running 142 113d
    dockergc-n9wbb 1/1 Running 131 113d
    dockergc-qcjhr 0/1 Pending 0 7s
    *One pod is pending, but after 10 seconds it will be Running and another one is pending.

    Regrads.
    Chris

    ------------------------------
    Chris
    ------------------------------

    Original Message


  • 5.  RE: Failed to Docker Login

    Posted Fri May 15, 2020 04:45 AM

    Hi,

    That depends where you are logging in from.
    If its inside of the OCP cluster, then using the service is OK, if you are outside the route is the way to go.
    Both should work for you from withing the cluster.

    The pods thats are starting are docker garbage collectors, they should not run so often.
    Can you check disk space on your machines ? Maybe docker drive is filling up ?

    Can you open a support case ? This may need deeper investigation.

    Thanks



    ------------------------------
    TOMASZ HANUSIAK
    ------------------------------

    Original Message


  • 6.  RE: Failed to Docker Login

    Posted Mon May 18, 2020 08:46 PM
    Hi,

    Thank you  for your reply.

    After uninstalling Watson Discovery,
    the disk usage of Master01 increased steeply.

    [master01]#du -ha --max-depth=1 /var/lib/containers
    4.0K /var/lib/containers/atomic
    43G /var/lib/containers/docker
    du: cannot access '/var/lib/containers/storage/overlay/841558e3d1794587fa21ae682afd180edaad13a73554c6af22608d39b31648c2': No such file or directory
    8.8G /var/lib/containers/storage
    128K /var/lib/containers/cache
    52G /var/lib/containers
    ​
    [master01]# df -Th
    Filesystem Type Size Used Avail Use% Mounted on
    devtmpfs devtmpfs 16G 0 16G 0% /dev
    tmpfs tmpfs 16G 0 16G 0% /dev/shm
    tmpfs tmpfs 16G 3.1M 16G 1% /run
    tmpfs tmpfs 16G 0 16G 0% /sys/fs/cgroup
    /dev/sda1 xfs 200G 185G 16G 93% /
    xx.xx.xx.xx:/data nfs 1007G 71G 886G 8% /data
    shm tmpfs 64M 0 64M 0% /run/containers/storage/overlay-containers/4f64af9e233c9f6bbb599e2db64f5ddd894a8683b4085765a346e3d6bda45680/userdata/shm
    shm tmpfs 64M 0 64M 0% /run/containers/storage/overlay-containers/8b3d12d2c77b5bfbea5ef5f07eda46e5f6dd431148cbf2f450ba262bb8ad5d3d/userdata/shm
    shm tmpfs 64M 0 64M 0% /run/containers/storage/overlay-containers/c0ee5eb0b96a3a524629f183d6d85670fbdfad74f860fcd5cc0bd42394ea0995/userdata/shm
    overlay overlay 200G 185G 16G 93% /var/lib/containers/storage/overlay/af207d28fedd7c04b17565ebde37ecdb016022dec47a397e3b4dac351ca88001/merged
    overlay overlay 200G 185G 16G 93% /var/lib/containers/storage/overlay/7154910e9f29408b94398e93289dad4bc38d43fe37a93e21a1b2063f607317ee/merged
    overlay overlay 200G 185G 16G 93% /var/lib/containers/storage/overlay/0d22e31a89fdce44521e09d74d720954a157514786cdf85ae23971f15bc1d04b/merged
    overlay overlay 200G 185G 16G 93% /var/lib/containers/storage/overlay/6aeddab794d48db472df1f0b9793ca4f08fbc2488f0c13e32cae811adf4569e7/merged
    overlay overlay 200G 185G 16G 93% /var/lib/containers/storage/overlay/89e42665cd1a9fd7cf173d0ab8db9f8a7148c44e59cb0e729bc373fcabf510bf/merged
    overlay overlay 200G 185G 16G 93% /var/lib/containers/storage/overlay/95ac328c27a2485f30a51b285ffd3ffe2afb48ce23be6ba52b0e48960f437e03/merged
    tmpfs tmpfs 3.2G 0 3.2G 0% /run/user/0
    overlay overlay 200G 185G 16G 93% /var/lib/containers/storage/overlay/ba75e21092a249682b9d6d475f16a617cff6dfb0c1b9e251e8116bf25541a1c4/merged
    tmpfs tmpfs 16G 16K 16G 1% /var/lib/origin/openshift.local.volumes/pods/d0ddc5a0-9969-11ea-a445-42010a920004/volumes/kubernetes.io~secret/sync-token-z6cm9
    shm tmpfs 64M 0 64M 0% /run/containers/storage/overlay-containers/16d0605ad4da9efc34cfb9556a5ceedc6839c4a4a0cd826366b8b53db4869388/userdata/shm
    overlay overlay 200G 185G 16G 93% /var/lib/containers/storage/overlay/21b69f74f9c4d5f50128b480ec9cdaba8cfdc24044adcf9919e5b25c800f4f3f/merged
    overlay overlay 200G 185G 16G 93% /var/lib/containers/storage/overlay/78cca396e34f004e72831daf4f4793b666eb20aeadd80e6a77037ce12e275f4a/merged
    tmpfs tmpfs 16G 16K 16G 1% /var/lib/origin/openshift.local.volumes/pods/4ced397a-9969-11ea-a445-42010a920004/volumes/kubernetes.io~secret/sdn-token-2d9tz
    shm tmpfs 64M 0 64M 0% /run/containers/storage/overlay-containers/5f44e69b1e852d62afc5f1a4c9b0e887867b6779b9eb0ca0e0185159c79bdcb2/userdata/shm
    overlay overlay 200G 185G 16G 93% /var/lib/containers/storage/overlay/437acabb81ebf1051687595f56963340f98cf7e675ecfd98d01697205dcf575a/merged
    overlay overlay 200G 185G 16G 93% /var/lib/containers/storage/overlay/4915b430ac77e97a3434bfbf0667495f98e962dcbcbfa414d10f9962b3be9ccf/merged
    overlay overlay 200G 185G 16G 93% /var/lib/containers/storage/overlay/3a00669465846ae8c9407c7482e5e819067ce01dcbb5752bf0c1668011df9060/merged


    Could you guess any cause of this?

    Kind regards,
    Chris

    ------------------------------
    Chris
    ------------------------------

    Original Message