Hi Harald,
Following are the outputs command suggested,
-----------------------------------------
[root@xxx]# kubectl describe pod ibm-spectrum-scale-csi-operator-6c8b6c5d74-5bmhl -n ibm-spectrum-scale-csi-driver
Name: ibm-spectrum-scale-csi-operator-6c8b6c5d74-5bmhl
Namespace: ibm-spectrum-scale-csi-driver
Priority: 0
Node: xxxxxxx/xx.xx.xx.xx
Start Time: Fri, 21 Jan 2022 10:38:21 +0530
Labels: app.kubernetes.io/instance=ibm-spectrum-scale-csi-operator
app.kubernetes.io/managed-by=ibm-spectrum-scale-csi-operator
app.kubernetes.io/name=ibm-spectrum-scale-csi-operator
name=ibm-spectrum-scale-csi-operator
pod-template-hash=6c8b6c5d74
product=ibm-spectrum-scale-csi
release=ibm-spectrum-scale-csi-operator
Annotations: cni.projectcalico.org/podIP: 10.42.1.5/32
cni.projectcalico.org/podIPs: 10.42.1.5/32
productID: ibm-spectrum-scale-csi-operator
productName: IBM Spectrum Scale CSI Operator
productVersion: 2.2.0
Status: Running
IP: 10.42.1.5
IPs:
IP: 10.42.1.5
Controlled By: ReplicaSet/ibm-spectrum-scale-csi-operator-6c8b6c5d74
Containers:
operator:
Container ID: docker://771b9e42d8409e1e63b1b08315124320aa85d428a3fa52b4c695540003db75b4
Image: quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-operator:v2.2.0
Image ID: docker-pullable://quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-operator@sha256:859a1f18b653a81fd06f0ba053bfe22b9443bd2dbc824b72fe7322dc1d4627da
Port: <none>
Host Port: <none>
Args:
--metrics-addr=0.0.0.0:8383
--enable-leader-election
--leader-election-id=ibm-spectrum-scale-csi-operator
State: Running
Started: Fri, 21 Jan 2022 10:38:22 +0530
Ready: True
Restart Count: 0
Limits:
cpu: 500m
memory: 500Mi
Requests:
cpu: 50m
memory: 50Mi
Liveness: exec [./health_check.sh] delay=10s timeout=1s period=30s #success=1 #failure=3
Readiness: exec [./health_check.sh] delay=3s timeout=1s period=1s #success=1 #failure=3
Environment:
MAX_CONCURRENT_RECONCILES_CSISCALEOPERATOR_CSI_IBM_COM: 1
MAX_CONCURRENT_RECONCILES_SECRET_: 1
ANSIBLE_DEBUG_LOGS: False
WATCH_NAMESPACE: ibm-spectrum-scale-csi-driver (v1:metadata.namespace)
CSI_DRIVER_IMAGE: quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-driver:v2.2.0
Mounts:
/tmp/ansible-operator/runner from runner (rw)
/var/run/secrets/kubernetes.io/serviceaccount from ibm-spectrum-scale-csi-operator-token-j58qb (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
runner:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
ibm-spectrum-scale-csi-operator-token-j58qb:
Type: Secret (a volume populated by a Secret)
SecretName: ibm-spectrum-scale-csi-operator-token-j58qb
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 69s default-scheduler Successfully assigned ibm-spectrum-scale-csi-driver/ibm-spectrum-scale-csi-operator-6c8b6c5d74-5bmhl <Server_name>
Normal Pulled 52s kubelet Container image "quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-operator:v2.2.0" already present on machine
Normal Created 52s kubelet Created container operator
Normal Started 52s kubelet Started container operator
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Second Command output
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[root@ ~]# kubectl logs ibm-spectrum-scale-csi-operator-6c8b6c5d74-5bmhl -n ibm-spectrum-scale-csi-driver {"level":"info","ts":1642741702.32836,"logger":"cmd","msg":"Version","Go Version":"go1.15.11","GOOS":"linux","GOARCH":"amd64","ansible-operator":"v1.5.0+git","commit":"b03e8c0cc056d76cbc97519740c4bcbc9f28c9c7"}
{"level":"info","ts":1642741702.3286374,"logger":"cmd","msg":"Watching single namespace.","Namespace":"ibm-spectrum-scale-csi-driver"}
{"level":"info","ts":1642741703.0364814,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"0.0.0.0:8383"}
{"level":"info","ts":1642741703.037385,"logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_CSISCALEOPERATOR_CSI_IBM_COM","default":2}
{"level":"info","ts":1642741703.0374873,"logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_SECRET_","default":2}
{"level":"info","ts":1642741703.0375376,"logger":"ansible-controller","msg":"Watching resource","Options.Group":"csi.ibm.com","Options.Version":"v1","Options.Kind":"CSIScaleOperator"}
{"level":"info","ts":1642741703.0377247,"logger":"controller-runtime.injectors-warning","msg":"Injectors are deprecated, and will be removed in v0.10.x"}
{"level":"info","ts":1642741703.0377743,"logger":"controller-runtime.injectors-warning","msg":"Injectors are deprecated, and will be removed in v0.10.x"}
{"level":"info","ts":1642741703.0378253,"logger":"ansible-controller","msg":"Watching resource","Options.Group":"","Options.Version":"v1","Options.Kind":"Secret"}
{"level":"info","ts":1642741703.0378456,"logger":"controller-runtime.injectors-warning","msg":"Injectors are deprecated, and will be removed in v0.10.x"}
{"level":"info","ts":1642741703.037857,"logger":"controller-runtime.injectors-warning","msg":"Injectors are deprecated, and will be removed in v0.10.x"}
{"level":"info","ts":1642741703.0384924,"logger":"proxy","msg":"Starting to serve","Address":"127.0.0.1:8888"}
{"level":"info","ts":1642741703.0387266,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
I0121 05:08:23.038697 6 leaderelection.go:243] attempting to acquire leader lease ibm-spectrum-scale-csi-driver/ibm-spectrum-scale-csi-operator...
I0121 05:08:40.450635 6 leaderelection.go:253] successfully acquired lease ibm-spectrum-scale-csi-driver/ibm-spectrum-scale-csi-operator
{"level":"info","ts":1642741720.4509962,"logger":"controller-runtime.manager.controller.secret-controller","msg":"Starting EventSource","source":"kind source: /v1, Kind=Secret"}
{"level":"info","ts":1642741720.4510772,"logger":"controller-runtime.manager.controller.csiscaleoperator-controller","msg":"Starting EventSource","source":"kind source: csi.ibm.com/v1, Kind=CSIScaleOperator"}
{"level":"info","ts":1642741720.5515027,"logger":"controller-runtime.manager.controller.secret-controller","msg":"Starting Controller"}
{"level":"info","ts":1642741720.5515845,"logger":"controller-runtime.manager.controller.secret-controller","msg":"Starting workers","worker count":1}
{"level":"info","ts":1642741720.5520465,"logger":"controller-runtime.manager.controller.csiscaleoperator-controller","msg":"Starting Controller"}
{"level":"info","ts":1642741720.55209,"logger":"controller-runtime.manager.controller.csiscaleoperator-controller","msg":"Starting workers","worker count":1}
{"level":"info","ts":1642741725.1027527,"logger":"logging_event_handler","msg":"[playbook task]","name":"ibm-spectrum-scale-csi","namespace":"ibm-spectrum-scale-csi-driver","gvk":"csi.ibm.com/v1, Kind=CSIScaleOperator","event_type":"playbook_on_task_start","job":"7504504064263669287","EventData.Name":"Gathering Facts"}
--------------------------- Ansible Task StdOut -------------------------------
TASK [Gathering Facts] *********************************************************
-------------------------------------------------------------------------------
----- Ansible Task Status Event StdOut (/v1, Kind=Secret, default-token-rtksr/ibm-spectrum-scale-csi-driver) -----
PLAY RECAP *********************************************************************
----------
{"level":"info","ts":1642741725.9967945,"logger":"runner","msg":"Ansible-runner exited successfully","job":"4751997750760398084","name":"default-token-rtksr","namespace":"ibm-spectrum-scale-csi-driver"}
{"level":"info","ts":1642741730.4020476,"logger":"logging_event_handler","msg":"[playbook task]","name":"ibm-spectrum-scale-csi","namespace":"ibm-spectrum-scale-csi-driver","gvk":"csi.ibm.com/v1, Kind=CSIScaleOperator","event_type":"playbook_on_task_start","job":"7504504064263669287","EventData.Name":"csiscaleoperator : Get security context constraint information"}
--------------------------- Ansible Task StdOut -------------------------------
TASK [csiscaleoperator : Get security context constraint information] **********
task path: /opt/ansible/roles/csiscaleoperator/tasks/main.yml:40
-------------------------------------------------------------------------------
{"level":"info","ts":1642741731.5884957,"logger":"runner","msg":"Ansible-runner exited successfully","job":"1976235410884491574","name":"ibm-spectrum-scale-csi-operator-token-j58qb","namespace":"ibm-spectrum-scale-csi-driver"}
----- Ansible Task Status Event StdOut (/v1, Kind=Secret, ibm-spectrum-scale-csi-operator-token-j58qb/ibm-spectrum-scale-csi-driver) -----
PLAY RECAP *********************************************************************
----------
{"level":"info","ts":1642741731.7391317,"logger":"logging_event_handler","msg":"[playbook task]","name":"ibm-spectrum-scale-csi","namespace":"ibm-spectrum-scale-csi-driver","gvk":"csi.ibm.com/v1, Kind=CSIScaleOperator","event_type":"playbook_on_task_start","job":"7504504064263669287","EventData.Name":"csiscaleoperator : Ensure SCC are present"}
--------------------------- Ansible Task StdOut -------------------------------
TASK [csiscaleoperator : Ensure SCC are present] *******************************
task path: /opt/ansible/roles/csiscaleoperator/tasks/main.yml:71
-------------------------------------------------------------------------------
{"level":"info","ts":1642741731.801771,"logger":"logging_event_handler","msg":"[playbook task]","name":"ibm-spectrum-scale-csi","namespace":"ibm-spectrum-scale-csi-driver","gvk":"csi.ibm.com/v1, Kind=CSIScaleOperator","event_type":"playbook_on_task_start","job":"7504504064263669287","EventData.Name":"csiscaleoperator : Check for old csiaccess"}
--------------------------- Ansible Task StdOut -------------------------------
TASK [csiscaleoperator : Check for old csiaccess] ******************************
task path: /opt/ansible/roles/csiscaleoperator/tasks/main.yml:78
-------------------------------------------------------------------------------
{"level":"info","ts":1642741731.8269148,"logger":"logging_event_handler","msg":"[playbook task]","name":"ibm-spectrum-scale-csi","namespace":"ibm-spectrum-scale-csi-driver","gvk":"csi.ibm.com/v1, Kind=CSIScaleOperator","event_type":"playbook_on_task_start","job":"7504504064263669287","EventData.Name":"csiscaleoperator : Ensure old SCC is removed"}
--------------------------- Ansible Task StdOut -------------------------------
TASK [csiscaleoperator : Ensure old SCC is removed] ****************************
task path: /opt/ansible/roles/csiscaleoperator/tasks/main.yml:84
-------------------------------------------------------------------------------
{"level":"info","ts":1642741731.9923446,"logger":"logging_event_handler","msg":"[playbook task]","name":"ibm-spectrum-scale-csi","namespace":"ibm-spectrum-scale-csi-driver","gvk":"csi.ibm.com/v1, Kind=CSIScaleOperator","event_type":"playbook_on_task_start","job":"7504504064263669287","EventData.Name":"csiscaleoperator : Ensure the clusters are valid"}
--------------------------- Ansible Task StdOut -------------------------------
TASK [csiscaleoperator : Ensure the clusters are valid] ************************
task path: /opt/ansible/roles/csiscaleoperator/tasks/main.yml:92
-------------------------------------------------------------------------------
{"level":"info","ts":1642741732.0784554,"logger":"logging_event_handler","msg":"[playbook task]","name":"ibm-spectrum-scale-csi","namespace":"ibm-spectrum-scale-csi-driver","gvk":"csi.ibm.com/v1, Kind=CSIScaleOperator","event_type":"playbook_on_task_start","job":"7504504064263669287","EventData.Name":"csiscaleoperator : Ensure secret guisecret defined in ibm-spectrum-scale-csi-driver"}
--------------------------- Ansible Task StdOut -------------------------------
TASK [csiscaleoperator : Ensure secret guisecret defined in ibm-spectrum-scale-csi-driver] ***
task path: /opt/ansible/roles/csiscaleoperator/tasks/cluster_check.yml:2
-------------------------------------------------------------------------------
{"level":"info","ts":1642741734.6205087,"logger":"proxy","msg":"Cache miss: /v1, Kind=Secret, ibm-spectrum-scale-csi-driver/guisecret"}
{"level":"info","ts":1642741735.137228,"logger":"runner","msg":"Ansible-runner exited successfully","job":"7504504064263669287","name":"ibm-spectrum-scale-csi","namespace":"ibm-spectrum-scale-csi-driver"}
----- Ansible Task Status Event StdOut (csi.ibm.com/v1, Kind=CSIScaleOperator, ibm-spectrum-scale-csi/ibm-spectrum-scale-csi-driver) -----
PLAY RECAP *********************************************************************
localhost : ok=4 changed=0 unreachable=0 failed=0 skipped=8 rescued=0 ignored=0
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Third command output
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[root@ ~]# kubectl describe csiscaleoperator -n ibm-spectrum-scale-csi-driver
No resources found in ibm-spectrum-scale-csi-driver namespace.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------
Jickriya Jamadar
------------------------------
Original Message:
Sent: Mon January 17, 2022 09:32 AM
From: Harald Seipp
Subject: IBM Spectrum Scale CSI driver v1.1.0 with Rancher v2.5.9
To check the Custom Resource, use
kubectl describe csiscaleoperator -n ibm-spectrum-scale-csi-driver
------------------------------
Harald Seipp
Senior Technical Staff Member
Original Message:
Sent: Mon January 17, 2022 06:24 AM
From: Harald Seipp
Subject: IBM Spectrum Scale CSI driver v1.1.0 with Rancher v2.5.9
Hi Jickriya,
you might try:
Best regards,
------------------------------
Harald Seipp
Senior Technical Staff Member
Original Message:
Sent: Mon January 17, 2022 02:05 AM
From: Jickriya Jamadar
Subject: IBM Spectrum Scale CSI driver v1.1.0 with Rancher v2.5.9
Hi Harald,
As per your suggestion I have created setup with 2.2.0 IBM driver with Rancher 2.5.9 and Kubernetes cluster 1.19.xx,
Also I have disabled POD security.
However still Attacher and Provisioner PODs are not getting created.
Do let me know if I am missing something or we need to check any logs.
Please do let me know.
------------------------------
Jickriya Jamadar
Original Message:
Sent: Tue January 11, 2022 03:25 AM
From: Harald Seipp
Subject: IBM Spectrum Scale CSI driver v1.1.0 with Rancher v2.5.9
Hi Jickriya,
please first of all note that RKE is not officially supported with Spectrum Scale CSI. To get a supported environment, you might use Rancher as management interface for a plain Kubernetes installation.
Based on my own experiences experimenting with Rancher 2.5.9 / RKE I'd recommend for experimental environments to
- Use Spectrum Scale CSI version 2.2.0 or later with Kubernetes 1.19 or 1.20
- Disable Pod Security Policy support (this is deprecated in Kubernetes anyway)
You can find a summary of my early RKE experiments on slide 22 of the following presentation: https://www.spectrumscaleug.org/wp-content/uploads/2020/03/SSSD20DE-Spectrum-Scale-use-cases-and-field-lessons-learned-with-Kubernetes-and-OpenShift.pdf
Best regards,
------------------------------
Harald Seipp
Senior Technical Staff Member
Original Message:
Sent: Mon January 10, 2022 02:02 AM
From: Jickriya Jamadar
Subject: IBM Spectrum Scale CSI driver v1.1.0 with Rancher v2.5.9
Hi,
I have created following setup;
1) 2 node Spectrum scale Cluster 5.1.0.2 on RHEL 7
2) Create RKE (Racher) v2.5.9 Kubermetes v1.17.x setup on spectrum scale cluster.
3) Deployed spectrum scale CSI driver v1.1.0.
After creating setup I am seeing that attacher and provisioner PODs are not not getting started.
![](https://dw1.s81c.com/IMWUC/MessageImages/9b376b2a48904a2d83151785582311ac.png)
Could you please help me to understand why these pods are not got started.
Secondly what are the steps to start or create both pods manually.
------------------------------
Jickriya Jamadar
------------------------------