File and Object Storage

 View Only
Expand all | Collapse all

IBM Spectrum Scale CSI driver v1.1.0 with Rancher v2.5.9

  • 1.  IBM Spectrum Scale CSI driver v1.1.0 with Rancher v2.5.9

    Posted Mon January 10, 2022 04:13 PM
    Hi,

    I have created following setup;
    1) 2 node Spectrum scale Cluster 5.1.0.2 on RHEL 7
    2) Create RKE (Racher) v2.5.9 Kubermetes v1.17.x setup on spectrum scale cluster.
    3) Deployed spectrum scale CSI driver v1.1.0.

    After creating setup I am seeing that attacher and provisioner PODs are not not getting started. 



    Could you please help me to understand why these pods are not got started.
    Secondly what are the steps to start or create both pods manually.


    ------------------------------
    Jickriya Jamadar
    ------------------------------


  • 2.  RE: IBM Spectrum Scale CSI driver v1.1.0 with Rancher v2.5.9

    Posted Tue January 11, 2022 02:45 AM
    Hi Jickriya,

    IBM Spectrum Scale CSI driver v1.1.0 is no longer supported, and would have several security vulnerabilities. Is there any specific reason to use this very old version of CSI driver ? Can you please try installing the latest IBM Spectrum Scale CSI driver v2.4.0, check the documentation at: https://www.ibm.com/docs/en/spectrum-scale-csi?topic=spectrum-scale-container-storage-interface-driver-240

    Regards,
    Madhu.

    ------------------------------
    Madhu Punjabi
    ------------------------------



  • 3.  RE: IBM Spectrum Scale CSI driver v1.1.0 with Rancher v2.5.9

    Posted Tue January 11, 2022 11:51 PM
    Thank you Madhu, this will help me to create the environment.

    ------------------------------
    Jickriya Jamadar
    ------------------------------



  • 4.  RE: IBM Spectrum Scale CSI driver v1.1.0 with Rancher v2.5.9

    Posted Tue January 11, 2022 03:26 AM
    Hi Jickriya,

    please first of all note that RKE is not officially supported with Spectrum Scale CSI. To get a supported environment, you might use Rancher as management interface for a plain Kubernetes installation.
    Based on my own experiences experimenting with Rancher 2.5.9 / RKE I'd recommend for experimental environments to
    1. Use Spectrum Scale CSI version 2.2.0 or later with Kubernetes 1.19 or 1.20
    2. Disable Pod Security Policy support (this is deprecated in Kubernetes anyway)
    You can find a summary of my early RKE experiments on slide 22 of the following presentation: https://www.spectrumscaleug.org/wp-content/uploads/2020/03/SSSD20DE-Spectrum-Scale-use-cases-and-field-lessons-learned-with-Kubernetes-and-OpenShift.pdf

    Best regards,

    ------------------------------
    Harald Seipp
    Senior Technical Staff Member
    ------------------------------



  • 5.  RE: IBM Spectrum Scale CSI driver v1.1.0 with Rancher v2.5.9

    Posted Tue January 11, 2022 11:54 PM
    Thank you Harald, I will try to install CSI driver v2.2.0.

    ------------------------------
    Jickriya Jamadar
    ------------------------------



  • 6.  RE: IBM Spectrum Scale CSI driver v1.1.0 with Rancher v2.5.9

    Posted Mon January 17, 2022 02:06 AM
    Hi Harald,

    As per your suggestion I have created setup with 2.2.0 IBM driver with Rancher 2.5.9 and Kubernetes cluster 1.19.xx,
    Also I have disabled POD security.
    However still Attacher and Provisioner PODs are not getting created.

    Do let me know if I am missing something or we need to check any logs.
    Please do let me know.

    ------------------------------
    Jickriya Jamadar
    ------------------------------



  • 7.  RE: IBM Spectrum Scale CSI driver v1.1.0 with Rancher v2.5.9

    Posted Mon January 17, 2022 06:24 AM
    Hi Jickriya,

    you might try:

    Best regards,

    ------------------------------
    Harald Seipp
    Senior Technical Staff Member
    ------------------------------



  • 8.  RE: IBM Spectrum Scale CSI driver v1.1.0 with Rancher v2.5.9

    Posted Mon January 17, 2022 09:33 AM
    To check the Custom Resource, use
    kubectl describe csiscaleoperator -n ibm-spectrum-scale-csi-driver​


    ------------------------------
    Harald Seipp
    Senior Technical Staff Member
    ------------------------------



  • 9.  RE: IBM Spectrum Scale CSI driver v1.1.0 with Rancher v2.5.9

    Posted Fri January 21, 2022 12:30 AM
    Hi Harald,

    Following are the outputs command suggested,
    -----------------------------------------
    [root@xxx]# kubectl describe pod ibm-spectrum-scale-csi-operator-6c8b6c5d74-5bmhl -n ibm-spectrum-scale-csi-driver
    Name: ibm-spectrum-scale-csi-operator-6c8b6c5d74-5bmhl
    Namespace: ibm-spectrum-scale-csi-driver
    Priority: 0
    Node: xxxxxxx/xx.xx.xx.xx
    Start Time: Fri, 21 Jan 2022 10:38:21 +0530
    Labels: app.kubernetes.io/instance=ibm-spectrum-scale-csi-operator
    app.kubernetes.io/managed-by=ibm-spectrum-scale-csi-operator
    app.kubernetes.io/name=ibm-spectrum-scale-csi-operator
    name=ibm-spectrum-scale-csi-operator
    pod-template-hash=6c8b6c5d74
    product=ibm-spectrum-scale-csi
    release=ibm-spectrum-scale-csi-operator
    Annotations: cni.projectcalico.org/podIP: 10.42.1.5/32
    cni.projectcalico.org/podIPs: 10.42.1.5/32
    productID: ibm-spectrum-scale-csi-operator
    productName: IBM Spectrum Scale CSI Operator
    productVersion: 2.2.0
    Status: Running
    IP: 10.42.1.5
    IPs:
    IP: 10.42.1.5
    Controlled By: ReplicaSet/ibm-spectrum-scale-csi-operator-6c8b6c5d74
    Containers:
    operator:
    Container ID: docker://771b9e42d8409e1e63b1b08315124320aa85d428a3fa52b4c695540003db75b4
    Image: quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-operator:v2.2.0
    Image ID: docker-pullable://quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-operator@sha256:859a1f18b653a81fd06f0ba053bfe22b9443bd2dbc824b72fe7322dc1d4627da
    Port: <none>
    Host Port: <none>
    Args:
    --metrics-addr=0.0.0.0:8383
    --enable-leader-election
    --leader-election-id=ibm-spectrum-scale-csi-operator
    State: Running
    Started: Fri, 21 Jan 2022 10:38:22 +0530
    Ready: True
    Restart Count: 0
    Limits:
    cpu: 500m
    memory: 500Mi
    Requests:
    cpu: 50m
    memory: 50Mi
    Liveness: exec [./health_check.sh] delay=10s timeout=1s period=30s #success=1 #failure=3
    Readiness: exec [./health_check.sh] delay=3s timeout=1s period=1s #success=1 #failure=3
    Environment:
    MAX_CONCURRENT_RECONCILES_CSISCALEOPERATOR_CSI_IBM_COM: 1
    MAX_CONCURRENT_RECONCILES_SECRET_: 1
    ANSIBLE_DEBUG_LOGS: False
    WATCH_NAMESPACE: ibm-spectrum-scale-csi-driver (v1:metadata.namespace)
    CSI_DRIVER_IMAGE: quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-driver:v2.2.0
    Mounts:
    /tmp/ansible-operator/runner from runner (rw)
    /var/run/secrets/kubernetes.io/serviceaccount from ibm-spectrum-scale-csi-operator-token-j58qb (ro)
    Conditions:
    Type Status
    Initialized True
    Ready True
    ContainersReady True
    PodScheduled True
    Volumes:
    runner:
    Type: EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit: <unset>
    ibm-spectrum-scale-csi-operator-token-j58qb:
    Type: Secret (a volume populated by a Secret)
    SecretName: ibm-spectrum-scale-csi-operator-token-j58qb
    Optional: false
    QoS Class: Burstable
    Node-Selectors: <none>
    Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
    node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
    Events:
    Type Reason Age From Message
    ---- ------ ---- ---- -------
    Normal Scheduled 69s default-scheduler Successfully assigned ibm-spectrum-scale-csi-driver/ibm-spectrum-scale-csi-operator-6c8b6c5d74-5bmhl <Server_name>
    Normal Pulled 52s kubelet Container image "quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-operator:v2.2.0" already present on machine
    Normal Created 52s kubelet Created container operator
    Normal Started 52s kubelet Started container operator
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    Second Command output

    ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    [root@ ~]# kubectl logs ibm-spectrum-scale-csi-operator-6c8b6c5d74-5bmhl -n ibm-spectrum-scale-csi-driver {"level":"info","ts":1642741702.32836,"logger":"cmd","msg":"Version","Go Version":"go1.15.11","GOOS":"linux","GOARCH":"amd64","ansible-operator":"v1.5.0+git","commit":"b03e8c0cc056d76cbc97519740c4bcbc9f28c9c7"}
    {"level":"info","ts":1642741702.3286374,"logger":"cmd","msg":"Watching single namespace.","Namespace":"ibm-spectrum-scale-csi-driver"}
    {"level":"info","ts":1642741703.0364814,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"0.0.0.0:8383"}
    {"level":"info","ts":1642741703.037385,"logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_CSISCALEOPERATOR_CSI_IBM_COM","default":2}
    {"level":"info","ts":1642741703.0374873,"logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_SECRET_","default":2}
    {"level":"info","ts":1642741703.0375376,"logger":"ansible-controller","msg":"Watching resource","Options.Group":"csi.ibm.com","Options.Version":"v1","Options.Kind":"CSIScaleOperator"}
    {"level":"info","ts":1642741703.0377247,"logger":"controller-runtime.injectors-warning","msg":"Injectors are deprecated, and will be removed in v0.10.x"}
    {"level":"info","ts":1642741703.0377743,"logger":"controller-runtime.injectors-warning","msg":"Injectors are deprecated, and will be removed in v0.10.x"}
    {"level":"info","ts":1642741703.0378253,"logger":"ansible-controller","msg":"Watching resource","Options.Group":"","Options.Version":"v1","Options.Kind":"Secret"}
    {"level":"info","ts":1642741703.0378456,"logger":"controller-runtime.injectors-warning","msg":"Injectors are deprecated, and will be removed in v0.10.x"}
    {"level":"info","ts":1642741703.037857,"logger":"controller-runtime.injectors-warning","msg":"Injectors are deprecated, and will be removed in v0.10.x"}
    {"level":"info","ts":1642741703.0384924,"logger":"proxy","msg":"Starting to serve","Address":"127.0.0.1:8888"}
    {"level":"info","ts":1642741703.0387266,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
    I0121 05:08:23.038697 6 leaderelection.go:243] attempting to acquire leader lease ibm-spectrum-scale-csi-driver/ibm-spectrum-scale-csi-operator...
    I0121 05:08:40.450635 6 leaderelection.go:253] successfully acquired lease ibm-spectrum-scale-csi-driver/ibm-spectrum-scale-csi-operator
    {"level":"info","ts":1642741720.4509962,"logger":"controller-runtime.manager.controller.secret-controller","msg":"Starting EventSource","source":"kind source: /v1, Kind=Secret"}
    {"level":"info","ts":1642741720.4510772,"logger":"controller-runtime.manager.controller.csiscaleoperator-controller","msg":"Starting EventSource","source":"kind source: csi.ibm.com/v1, Kind=CSIScaleOperator"}
    {"level":"info","ts":1642741720.5515027,"logger":"controller-runtime.manager.controller.secret-controller","msg":"Starting Controller"}
    {"level":"info","ts":1642741720.5515845,"logger":"controller-runtime.manager.controller.secret-controller","msg":"Starting workers","worker count":1}
    {"level":"info","ts":1642741720.5520465,"logger":"controller-runtime.manager.controller.csiscaleoperator-controller","msg":"Starting Controller"}
    {"level":"info","ts":1642741720.55209,"logger":"controller-runtime.manager.controller.csiscaleoperator-controller","msg":"Starting workers","worker count":1}
    {"level":"info","ts":1642741725.1027527,"logger":"logging_event_handler","msg":"[playbook task]","name":"ibm-spectrum-scale-csi","namespace":"ibm-spectrum-scale-csi-driver","gvk":"csi.ibm.com/v1, Kind=CSIScaleOperator","event_type":"playbook_on_task_start","job":"7504504064263669287","EventData.Name":"Gathering Facts"}

    --------------------------- Ansible Task StdOut -------------------------------

    TASK [Gathering Facts] *********************************************************

    -------------------------------------------------------------------------------

    ----- Ansible Task Status Event StdOut (/v1, Kind=Secret, default-token-rtksr/ibm-spectrum-scale-csi-driver) -----


    PLAY RECAP *********************************************************************


    ----------
    {"level":"info","ts":1642741725.9967945,"logger":"runner","msg":"Ansible-runner exited successfully","job":"4751997750760398084","name":"default-token-rtksr","namespace":"ibm-spectrum-scale-csi-driver"}
    {"level":"info","ts":1642741730.4020476,"logger":"logging_event_handler","msg":"[playbook task]","name":"ibm-spectrum-scale-csi","namespace":"ibm-spectrum-scale-csi-driver","gvk":"csi.ibm.com/v1, Kind=CSIScaleOperator","event_type":"playbook_on_task_start","job":"7504504064263669287","EventData.Name":"csiscaleoperator : Get security context constraint information"}

    --------------------------- Ansible Task StdOut -------------------------------

    TASK [csiscaleoperator : Get security context constraint information] **********
    task path: /opt/ansible/roles/csiscaleoperator/tasks/main.yml:40

    -------------------------------------------------------------------------------
    {"level":"info","ts":1642741731.5884957,"logger":"runner","msg":"Ansible-runner exited successfully","job":"1976235410884491574","name":"ibm-spectrum-scale-csi-operator-token-j58qb","namespace":"ibm-spectrum-scale-csi-driver"}

    ----- Ansible Task Status Event StdOut (/v1, Kind=Secret, ibm-spectrum-scale-csi-operator-token-j58qb/ibm-spectrum-scale-csi-driver) -----


    PLAY RECAP *********************************************************************


    ----------
    {"level":"info","ts":1642741731.7391317,"logger":"logging_event_handler","msg":"[playbook task]","name":"ibm-spectrum-scale-csi","namespace":"ibm-spectrum-scale-csi-driver","gvk":"csi.ibm.com/v1, Kind=CSIScaleOperator","event_type":"playbook_on_task_start","job":"7504504064263669287","EventData.Name":"csiscaleoperator : Ensure SCC are present"}

    --------------------------- Ansible Task StdOut -------------------------------

    TASK [csiscaleoperator : Ensure SCC are present] *******************************
    task path: /opt/ansible/roles/csiscaleoperator/tasks/main.yml:71

    -------------------------------------------------------------------------------
    {"level":"info","ts":1642741731.801771,"logger":"logging_event_handler","msg":"[playbook task]","name":"ibm-spectrum-scale-csi","namespace":"ibm-spectrum-scale-csi-driver","gvk":"csi.ibm.com/v1, Kind=CSIScaleOperator","event_type":"playbook_on_task_start","job":"7504504064263669287","EventData.Name":"csiscaleoperator : Check for old csiaccess"}

    --------------------------- Ansible Task StdOut -------------------------------

    TASK [csiscaleoperator : Check for old csiaccess] ******************************
    task path: /opt/ansible/roles/csiscaleoperator/tasks/main.yml:78

    -------------------------------------------------------------------------------
    {"level":"info","ts":1642741731.8269148,"logger":"logging_event_handler","msg":"[playbook task]","name":"ibm-spectrum-scale-csi","namespace":"ibm-spectrum-scale-csi-driver","gvk":"csi.ibm.com/v1, Kind=CSIScaleOperator","event_type":"playbook_on_task_start","job":"7504504064263669287","EventData.Name":"csiscaleoperator : Ensure old SCC is removed"}

    --------------------------- Ansible Task StdOut -------------------------------

    TASK [csiscaleoperator : Ensure old SCC is removed] ****************************
    task path: /opt/ansible/roles/csiscaleoperator/tasks/main.yml:84

    -------------------------------------------------------------------------------
    {"level":"info","ts":1642741731.9923446,"logger":"logging_event_handler","msg":"[playbook task]","name":"ibm-spectrum-scale-csi","namespace":"ibm-spectrum-scale-csi-driver","gvk":"csi.ibm.com/v1, Kind=CSIScaleOperator","event_type":"playbook_on_task_start","job":"7504504064263669287","EventData.Name":"csiscaleoperator : Ensure the clusters are valid"}

    --------------------------- Ansible Task StdOut -------------------------------

    TASK [csiscaleoperator : Ensure the clusters are valid] ************************
    task path: /opt/ansible/roles/csiscaleoperator/tasks/main.yml:92

    -------------------------------------------------------------------------------
    {"level":"info","ts":1642741732.0784554,"logger":"logging_event_handler","msg":"[playbook task]","name":"ibm-spectrum-scale-csi","namespace":"ibm-spectrum-scale-csi-driver","gvk":"csi.ibm.com/v1, Kind=CSIScaleOperator","event_type":"playbook_on_task_start","job":"7504504064263669287","EventData.Name":"csiscaleoperator : Ensure secret guisecret defined in ibm-spectrum-scale-csi-driver"}

    --------------------------- Ansible Task StdOut -------------------------------

    TASK [csiscaleoperator : Ensure secret guisecret defined in ibm-spectrum-scale-csi-driver] ***
    task path: /opt/ansible/roles/csiscaleoperator/tasks/cluster_check.yml:2

    -------------------------------------------------------------------------------
    {"level":"info","ts":1642741734.6205087,"logger":"proxy","msg":"Cache miss: /v1, Kind=Secret, ibm-spectrum-scale-csi-driver/guisecret"}
    {"level":"info","ts":1642741735.137228,"logger":"runner","msg":"Ansible-runner exited successfully","job":"7504504064263669287","name":"ibm-spectrum-scale-csi","namespace":"ibm-spectrum-scale-csi-driver"}

    ----- Ansible Task Status Event StdOut (csi.ibm.com/v1, Kind=CSIScaleOperator, ibm-spectrum-scale-csi/ibm-spectrum-scale-csi-driver) -----


    PLAY RECAP *********************************************************************
    localhost : ok=4 changed=0 unreachable=0 failed=0 skipped=8 rescued=0 ignored=0


    ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    Third command output

    ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    [root@ ~]# kubectl describe csiscaleoperator -n ibm-spectrum-scale-csi-driver​
    No resources found in ibm-spectrum-scale-csi-driver​ namespace.

    ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------





    ------------------------------
    Jickriya Jamadar
    ------------------------------



  • 10.  RE: IBM Spectrum Scale CSI driver v1.1.0 with Rancher v2.5.9

    Posted Fri January 21, 2022 01:56 AM
    "csiscaleoperators.csi.ibm.com_cr.yaml" file output;

    --------------------
    ---
    apiVersion: csi.ibm.com/v1
    kind: "CSIScaleOperator"
    metadata:
    name: "ibm-spectrum-scale-csi"
    namespace: "ibm-spectrum-scale-csi-driver"
    labels:
    app.kubernetes.io/name: ibm-spectrum-scale-csi-operator
    app.kubernetes.io/instance: ibm-spectrum-scale-csi-operator
    app.kubernetes.io/managed-by: ibm-spectrum-scale-csi-operator
    release: ibm-spectrum-scale-csi-operator
    status: {}
    spec:

    # A passthrough option that distributes an imagePullSecrets array to the containers
    # generated by the csi scale operator. Please refer to official k8s documentation for
    # your environment for more details. https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
    # ==================================================================================
    # imagePullSecrets:
    # - APullSecret
    # - AnotherOptional

    # Below specifies the details of a SpectrumScale cluster configuration used by the
    # plugin. It can have multiple values.
    # ==================================================================================
    clusters:
    - id: "xxxxxxxxxxxxxxxx"
    secrets: "guisecret"
    secureSslMode: false
    primary:
    primaryFs: "gpfs_test"
    # primaryFset: "< Fileset in Primary Filesystem >" # Optional - default:spectrum-scale-csi-volume-store
    # inodeLimit: "< inode limit for Primary Fileset >" # Optional
    # remoteCluster: "< Remote ClusterID >" # Optional - This is only required if primaryFs is remote cluster's filesystem and this ID should have separate entry in Clusters map too.
    # cacert: "< Name of CA cert configmap for GUI >" # Optional
    restApi:
    - guiHost: "test_01.test.test01.com"
    #
    # In the case we have multiple clusters, specify their configuration below.
    # ==================================================================================
    # - id: "< Cluster ID >"
    # secrets: "< Secret for Cluster >"
    # secureSslMode: false
    # restApi:
    # - guiHost: "< Cluster GUI IP/Hostname >"
    # cacert: "< Name of CA cert configmap for GUI >" # Optional

    # Attacher image name, in case we do not want to use default image.
    # ==================================================================================
    # attacher: "us.gcr.io/k8s-artifacts-prod/sig-storage/csi-attacher:v3.1.0"

    # Provisioner image name, in case we do not want to use default image.
    # ==================================================================================
    # provisioner: "us.gcr.io/k8s-artifacts-prod/sig-storage/csi-provisioner:v2.1.0"

    # Driver Registrar image name, in case we do not want to use default image.
    # ===============================================================================