Cloud Pak for Data

 View Only
  • 1.  openshift node STOPPED

    Posted Mon February 01, 2021 01:52 PM
    I periodically receive warning messages like "Alert SW_NEEDS_ATTENTION (446) from appliance - ICP4D service is not ready" and "Alert SW_NEEDS_ATTENTION (439) from appliance - Openshift node is not ready".

    ap hw comes up clean (all OK status), but ap sw will sometimes show one of the Openshift nodes as STOPPED.  Then a few minutes later it is fixed.  Then eventually it is STOPPED again.

    Thoughts?

    [root@e1n1 ~]# ap sw
    +-----------------------+-----------------------------------------------+---------+-----------------+
    | Name | Location | Status | Version |
    +-----------------------+-----------------------------------------------+---------+-----------------+
    | Appliance Application | nps | OK | 11.0.7.0 |
    | Application Node | nps.spa1.spu1/enclosure3.node1 | OK | |
    | Application Node | nps.spa1.spu2/enclosure3.node2 | OK | |
    | Application Node | nps.spa1.spu3/enclosure3.node3 | OK | |
    | Application Node | nps.spa1.spu4/enclosure3.node4 | OK | |
    | ICP4D Service | icp4d | OK | 3.0.1 |
    | Openshift Node | openshift/kvm.e1n1-1-control/enclosure1.node1 | OK | v1.11.0+d4cacc0 |
    | Openshift Node | openshift/kvm.e1n2-1-control/enclosure1.node2 | OK | v1.11.0+d4cacc0 |
    | Openshift Node | openshift/kvm.e1n3-1-control/enclosure1.node3 | OK | v1.11.0+d4cacc0 |
    | Openshift Node | openshift/kvm.e1n4-1-worker/enclosure1.node4 | OK | v1.11.0+d4cacc0 |
    | Openshift Node | openshift/kvm.e2n1-1-worker/enclosure2.node1 | OK | v1.11.0+d4cacc0 |
    | Openshift Node | openshift/kvm.e2n2-1-worker/enclosure2.node2 | STOPPED | v1.11.0+d4cacc0 |
    | Openshift Service | openshift | OK | v3.11.188 |
    | Portworx Cluster | portworx.cluster1 | OK | |
    | Web Console | webconsole | OK | 1.0.7.0 |
    +-----------------------+-----------------------------------------------+---------+-----------------+

    Generated: 2021-02-01 13:05:29


    [root@e1n1 ~]# ap sw
    +-----------------------+-----------------------------------------------+--------+-----------------+
    | Name | Location | Status | Version |
    +-----------------------+-----------------------------------------------+--------+-----------------+
    | Appliance Application | nps | OK | 11.0.7.0 |
    | Application Node | nps.spa1.spu1/enclosure3.node1 | OK | |
    | Application Node | nps.spa1.spu2/enclosure3.node2 | OK | |
    | Application Node | nps.spa1.spu3/enclosure3.node3 | OK | |
    | Application Node | nps.spa1.spu4/enclosure3.node4 | OK | |
    | ICP4D Service | icp4d | OK | 3.0.1 |
    | Openshift Node | openshift/kvm.e1n1-1-control/enclosure1.node1 | OK | v1.11.0+d4cacc0 |
    | Openshift Node | openshift/kvm.e1n2-1-control/enclosure1.node2 | OK | v1.11.0+d4cacc0 |
    | Openshift Node | openshift/kvm.e1n3-1-control/enclosure1.node3 | OK | v1.11.0+d4cacc0 |
    | Openshift Node | openshift/kvm.e1n4-1-worker/enclosure1.node4 | OK | v1.11.0+d4cacc0 |
    | Openshift Node | openshift/kvm.e2n1-1-worker/enclosure2.node1 | OK | v1.11.0+d4cacc0 |
    | Openshift Node | openshift/kvm.e2n2-1-worker/enclosure2.node2 | OK | v1.11.0+d4cacc0 |
    | Openshift Service | openshift | OK | v3.11.188 |
    | Portworx Cluster | portworx.cluster1 | OK | |
    | Web Console | webconsole | OK | 1.0.7.0 |
    +-----------------------+-----------------------------------------------+--------+-----------------+

    Generated: 2021-02-01 13:14:30


    [root@e1n1 ~]# ap sw
    +-----------------------+-----------------------------------------------+---------+-----------------+
    | Name | Location | Status | Version |
    +-----------------------+-----------------------------------------------+---------+-----------------+
    | Appliance Application | nps | OK | 11.0.7.0 |
    | Application Node | nps.spa1.spu1/enclosure3.node1 | OK | |
    | Application Node | nps.spa1.spu2/enclosure3.node2 | OK | |
    | Application Node | nps.spa1.spu3/enclosure3.node3 | OK | |
    | Application Node | nps.spa1.spu4/enclosure3.node4 | OK | |
    | ICP4D Service | icp4d | OK | 3.0.1 |
    | Openshift Node | openshift/kvm.e1n1-1-control/enclosure1.node1 | OK | v1.11.0+d4cacc0 |
    | Openshift Node | openshift/kvm.e1n2-1-control/enclosure1.node2 | OK | v1.11.0+d4cacc0 |
    | Openshift Node | openshift/kvm.e1n3-1-control/enclosure1.node3 | OK | v1.11.0+d4cacc0 |
    | Openshift Node | openshift/kvm.e1n4-1-worker/enclosure1.node4 | OK | v1.11.0+d4cacc0 |
    | Openshift Node | openshift/kvm.e2n1-1-worker/enclosure2.node1 | OK | v1.11.0+d4cacc0 |
    | Openshift Node | openshift/kvm.e2n2-1-worker/enclosure2.node2 | STOPPED | v1.11.0+d4cacc0 |
    | Openshift Service | openshift | OK | v3.11.188 |
    | Portworx Cluster | portworx.cluster1 | OK | |
    | Web Console | webconsole | OK | 1.0.7.0 |
    +-----------------------+-----------------------------------------------+---------+-----------------+

    Generated: 2021-02-01 13:43:21


    [root@e1n1 ~]# ap sw
    +-----------------------+-----------------------------------------------+--------+-----------------+
    | Name | Location | Status | Version |
    +-----------------------+-----------------------------------------------+--------+-----------------+
    | Appliance Application | nps | OK | 11.0.7.0 |
    | Application Node | nps.spa1.spu1/enclosure3.node1 | OK | |
    | Application Node | nps.spa1.spu2/enclosure3.node2 | OK | |
    | Application Node | nps.spa1.spu3/enclosure3.node3 | OK | |
    | Application Node | nps.spa1.spu4/enclosure3.node4 | OK | |
    | ICP4D Service | icp4d | OK | 3.0.1 |
    | Openshift Node | openshift/kvm.e1n1-1-control/enclosure1.node1 | OK | v1.11.0+d4cacc0 |
    | Openshift Node | openshift/kvm.e1n2-1-control/enclosure1.node2 | OK | v1.11.0+d4cacc0 |
    | Openshift Node | openshift/kvm.e1n3-1-control/enclosure1.node3 | OK | v1.11.0+d4cacc0 |
    | Openshift Node | openshift/kvm.e1n4-1-worker/enclosure1.node4 | OK | v1.11.0+d4cacc0 |
    | Openshift Node | openshift/kvm.e2n1-1-worker/enclosure2.node1 | OK | v1.11.0+d4cacc0 |
    | Openshift Node | openshift/kvm.e2n2-1-worker/enclosure2.node2 | OK | v1.11.0+d4cacc0 |
    | Openshift Service | openshift | OK | v3.11.188 |
    | Portworx Cluster | portworx.cluster1 | OK | |
    | Web Console | webconsole | OK | 1.0.7.0 |
    +-----------------------+-----------------------------------------------+--------+-----------------+

    Generated: 2021-02-01 13:47:15


    ------------------------------
    Chris Rodgers
    ------------------------------

    #CloudPakforDataGroup


  • 2.  RE: openshift node STOPPED

    Posted Tue February 02, 2021 08:19 PM
    This was related to a known issue in 1.0.7.x.  I opened a ticket and tech support did quite a few things to get this to stop as well as a few other known issues.

    ------------------------------
    Chris Rodgers
    ------------------------------



  • 3.  RE: openshift node STOPPED

    Posted Wed February 03, 2021 04:27 AM
    On the e1n1-control node, try the following commands and share the output.

    1. Check the openshift node status"

    oc get nodes
    oc describe node  e2n2-1-worker
    oc get pods | egrep -iv '1/1|2/2|3/3|4/4|completed'

    2. check the cluster resource utilization
    oc describe pods | egrep 'hostname|cpu   |memory   '

    This will provide openshift cluster status and pods. To understand any underlying issues.





    ------------------------------
    Lakshmana Ekambaram
    Client Experience
    IBM
    Lenexa KS
    ------------------------------



  • 4.  RE: openshift node STOPPED

    Posted Wed February 03, 2021 04:41 AM

    Resolved here I believe:

    https://community.ibm.com/community/user/cloudpakfordata/communities/community-home/digestviewer/viewthread?GroupId=3043&MessageKey=adea476f-8559-462a-bee6-666cf4bbb19a&CommunityKey=c0c16ff2-10ef-4b50-ae4c-57d769937235&tab=digestviewer&ReturnUrl=%2fcommunity%2fuser%2fcloudpakfordata%2fcommunities%2fcommunity-home%2fdigestviewer%3fcommunitykey%3dc0c16ff2-10ef-4b50-ae4c-57d769937235%26tab%3ddigestviewer




    ------------------------------
    TOMASZ HANUSIAK
    ------------------------------



  • 5.  RE: openshift node STOPPED

    Posted Fri February 05, 2021 06:28 PM
    Hi Chris,

    I am facing the same problem.
    Is it possible to describe the action here?
    If you have difficulty, I would like to ask support for your opened case.
    I will contact you in another way for case information.

    ------------------------------
    KAZUHISA MISONO
    ------------------------------