Hi Harald,
Following are the outputs command suggested,
-----------------------------------------
[root@xxx]# kubectl describe pod ibm-spectrum-scale-csi-operator-6c8b6c5d74-5bmhl -n ibm-spectrum-scale-csi-driver
Name: ibm-spectrum-scale-csi-operator-6c8b6c5d74-5bmhl
Namespace: ibm-spectrum-scale-csi-driver
Priority: 0
Node: xxxxxxx/xx.xx.xx.xx
Start Time: Fri, 21 Jan 2022 10:38:21 +0530
Labels: app.kubernetes.io/instance=ibm-spectrum-scale-csi-operator
app.kubernetes.io/managed-by=ibm-spectrum-scale-csi-operator
app.kubernetes.io/name=ibm-spectrum-scale-csi-operator
name=ibm-spectrum-scale-csi-operator
pod-template-hash=6c8b6c5d74
product=ibm-spectrum-scale-csi
release=ibm-spectrum-scale-csi-operator
Annotations: cni.projectcalico.org/podIP: 10.42.1.5/32
cni.projectcalico.org/podIPs: 10.42.1.5/32
productID: ibm-spectrum-scale-csi-operator
productName: IBM Spectrum Scale CSI Operator
productVersion: 2.2.0
Status: Running
IP: 10.42.1.5
IPs:
IP: 10.42.1.5
Controlled By: ReplicaSet/ibm-spectrum-scale-csi-operator-6c8b6c5d74
Containers:
operator:
Container ID: docker://771b9e42d8409e1e63b1b08315124320aa85d428a3fa52b4c695540003db75b4
Image: quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-operator:v2.2.0
Image ID: docker-pullable://quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-operator@sha256:859a1f18b653a81fd06f0ba053bfe22b9443bd2dbc824b72fe7322dc1d4627da
Port: <none>
Host Port: <none>
Args:
--metrics-addr=0.0.0.0:8383
--enable-leader-election
--leader-election-id=ibm-spectrum-scale-csi-operator
State: Running
Started: Fri, 21 Jan 2022 10:38:22 +0530
Ready: True
Restart Count: 0
Limits:
cpu: 500m
memory: 500Mi
Requests:
cpu: 50m
memory: 50Mi
Liveness: exec [./health_check.sh] delay=10s timeout=1s period=30s #success=1 #failure=3
Readiness: exec [./health_check.sh] delay=3s timeout=1s period=1s #success=1 #failure=3
Environment:
MAX_CONCURRENT_RECONCILES_CSISCALEOPERATOR_CSI_IBM_COM: 1
MAX_CONCURRENT_RECONCILES_SECRET_: 1
ANSIBLE_DEBUG_LOGS: False
WATCH_NAMESPACE: ibm-spectrum-scale-csi-driver (v1:metadata.namespace)
CSI_DRIVER_IMAGE: quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-driver:v2.2.0
Mounts:
/tmp/ansible-operator/runner from runner (rw)
/var/run/secrets/kubernetes.io/serviceaccount from ibm-spectrum-scale-csi-operator-token-j58qb (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
runner:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
ibm-spectrum-scale-csi-operator-token-j58qb:
Type: Secret (a volume populated by a Secret)
SecretName: ibm-spectrum-scale-csi-operator-token-j58qb
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 69s default-scheduler Successfully assigned ibm-spectrum-scale-csi-driver/ibm-spectrum-scale-csi-operator-6c8b6c5d74-5bmhl <Server_name>
Normal Pulled 52s kubelet Container image "quay.io/ibm-spectrum-scale/ibm-spectrum-scale-csi-operator:v2.2.0" already present on machine
Normal Created 52s kubelet Created container operator
Normal Started 52s kubelet Started container operator
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Second Command output
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[root@ ~]# kubectl logs ibm-spectrum-scale-csi-operator-6c8b6c5d74-5bmhl -n ibm-spectrum-scale-csi-driver {"level":"info","ts":1642741702.32836,"logger":"cmd","msg":"Version","Go Version":"go1.15.11","GOOS":"linux","GOARCH":"amd64","ansible-operator":"v1.5.0+git","commit":"b03e8c0cc056d76cbc97519740c4bcbc9f28c9c7"}
{"level":"info","ts":1642741702.3286374,"logger":"cmd","msg":"Watching single namespace.","Namespace":"ibm-spectrum-scale-csi-driver"}
{"level":"info","ts":1642741703.0364814,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"0.0.0.0:8383"}
{"level":"info","ts":1642741703.037385,"logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_CSISCALEOPERATOR_CSI_IBM_COM","default":2}
{"level":"info","ts":1642741703.0374873,"logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_SECRET_","default":2}
{"level":"info","ts":1642741703.0375376,"logger":"ansible-controller","msg":"Watching resource","Options.Group":"csi.ibm.com","Options.Version":"v1","Options.Kind":"CSIScaleOperator"}
{"level":"info","ts":1642741703.0377247,"logger":"controller-runtime.injectors-warning","msg":"Injectors are deprecated, and will be removed in v0.10.x"}
{"level":"info","ts":1642741703.0377743,"logger":"controller-runtime.injectors-warning","msg":"Injectors are deprecated, and will be removed in v0.10.x"}
{"level":"info","ts":1642741703.0378253,"logger":"ansible-controller","msg":"Watching resource","Options.Group":"","Options.Version":"v1","Options.Kind":"Secret"}
{"level":"info","ts":1642741703.0378456,"logger":"controller-runtime.injectors-warning","msg":"Injectors are deprecated, and will be removed in v0.10.x"}
{"level":"info","ts":1642741703.037857,"logger":"controller-runtime.injectors-warning","msg":"Injectors are deprecated, and will be removed in v0.10.x"}
{"level":"info","ts":1642741703.0384924,"logger":"proxy","msg":"Starting to serve","Address":"127.0.0.1:8888"}
{"level":"info","ts":1642741703.0387266,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
I0121 05:08:23.038697 6 leaderelection.go:243] attempting to acquire leader lease ibm-spectrum-scale-csi-driver/ibm-spectrum-scale-csi-operator...
I0121 05:08:40.450635 6 leaderelection.go:253] successfully acquired lease ibm-spectrum-scale-csi-driver/ibm-spectrum-scale-csi-operator
{"level":"info","ts":1642741720.4509962,"logger":"controller-runtime.manager.controller.secret-controller","msg":"Starting EventSource","source":"kind source: /v1, Kind=Secret"}
{"level":"info","ts":1642741720.4510772,"logger":"controller-runtime.manager.controller.csiscaleoperator-controller","msg":"Starting EventSource","source":"kind source: csi.ibm.com/v1, Kind=CSIScaleOperator"}
{"level":"info","ts":1642741720.5515027,"logger":"controller-runtime.manager.controller.secret-controller","msg":"Starting Controller"}
{"level":"info","ts":1642741720.5515845,"logger":"controller-runtime.manager.controller.secret-controller","msg":"Starting workers","worker count":1}
{"level":"info","ts":1642741720.5520465,"logger":"controller-runtime.manager.controller.csiscaleoperator-controller","msg":"Starting Controller"}
{"level":"info","ts":1642741720.55209,"logger":"controller-runtime.manager.controller.csiscaleoperator-controller","msg":"Starting workers","worker count":1}
{"level":"info","ts":1642741725.1027527,"logger":"logging_event_handler","msg":"[playbook task]","name":"ibm-spectrum-scale-csi","namespace":"ibm-spectrum-scale-csi-driver","gvk":"csi.ibm.com/v1, Kind=CSIScaleOperator","event_type":"playbook_on_task_start","job":"7504504064263669287","EventData.Name":"Gathering Facts"}
--------------------------- Ansible Task StdOut -------------------------------
TASK [Gathering Facts] *********************************************************
-------------------------------------------------------------------------------
----- Ansible Task Status Event StdOut (/v1, Kind=Secret, default-token-rtksr/ibm-spectrum-scale-csi-driver) -----
PLAY RECAP *********************************************************************
----------
{"level":"info","ts":1642741725.9967945,"logger":"runner","msg":"Ansible-runner exited successfully","job":"4751997750760398084","name":"default-token-rtksr","namespace":"ibm-spectrum-scale-csi-driver"}
{"level":"info","ts":1642741730.4020476,"logger":"logging_event_handler","msg":"[playbook task]","name":"ibm-spectrum-scale-csi","namespace":"ibm-spectrum-scale-csi-driver","gvk":"csi.ibm.com/v1, Kind=CSIScaleOperator","event_type":"playbook_on_task_start","job":"7504504064263669287","EventData.Name":"csiscaleoperator : Get security context constraint information"}
--------------------------- Ansible Task StdOut -------------------------------
TASK [csiscaleoperator : Get security context constraint information] **********
task path: /opt/ansible/roles/csiscaleoperator/tasks/main.yml:40
-------------------------------------------------------------------------------
{"level":"info","ts":1642741731.5884957,"logger":"runner","msg":"Ansible-runner exited successfully","job":"1976235410884491574","name":"ibm-spectrum-scale-csi-operator-token-j58qb","namespace":"ibm-spectrum-scale-csi-driver"}
----- Ansible Task Status Event StdOut (/v1, Kind=Secret, ibm-spectrum-scale-csi-operator-token-j58qb/ibm-spectrum-scale-csi-driver) -----
PLAY RECAP *********************************************************************
----------
{"level":"info","ts":1642741731.7391317,"logger":"logging_event_handler","msg":"[playbook task]","name":"ibm-spectrum-scale-csi","namespace":"ibm-spectrum-scale-csi-driver","gvk":"csi.ibm.com/v1, Kind=CSIScaleOperator","event_type":"playbook_on_task_start","job":"7504504064263669287","EventData.Name":"csiscaleoperator : Ensure SCC are present"}
--------------------------- Ansible Task StdOut -------------------------------
TASK [csiscaleoperator : Ensure SCC are present] *******************************
task path: /opt/ansible/roles/csiscaleoperator/tasks/main.yml:71
-------------------------------------------------------------------------------
{"level":"info","ts":1642741731.801771,"logger":"logging_event_handler","msg":"[playbook task]","name":"ibm-spectrum-scale-csi","namespace":"ibm-spectrum-scale-csi-driver","gvk":"csi.ibm.com/v1, Kind=CSIScaleOperator","event_type":"playbook_on_task_start","job":"7504504064263669287","EventData.Name":"csiscaleoperator : Check for old csiaccess"}
--------------------------- Ansible Task StdOut -------------------------------
TASK [csiscaleoperator : Check for old csiaccess] ******************************
task path: /opt/ansible/roles/csiscaleoperator/tasks/main.yml:78
-------------------------------------------------------------------------------
{"level":"info","ts":1642741731.8269148,"logger":"logging_event_handler","msg":"[playbook task]","name":"ibm-spectrum-scale-csi","namespace":"ibm-spectrum-scale-csi-driver","gvk":"csi.ibm.com/v1, Kind=CSIScaleOperator","event_type":"playbook_on_task_start","job":"7504504064263669287","EventData.Name":"csiscaleoperator : Ensure old SCC is removed"}
--------------------------- Ansible Task StdOut -------------------------------
TASK [csiscaleoperator : Ensure old SCC is removed] ****************************
task path: /opt/ansible/roles/csiscaleoperator/tasks/main.yml:84
-------------------------------------------------------------------------------
{"level":"info","ts":1642741731.9923446,"logger":"logging_event_handler","msg":"[playbook task]","name":"ibm-spectrum-scale-csi","namespace":"ibm-spectrum-scale-csi-driver","gvk":"csi.ibm.com/v1, Kind=CSIScaleOperator","event_type":"playbook_on_task_start","job":"7504504064263669287","EventData.Name":"csiscaleoperator : Ensure the clusters are valid"}
--------------------------- Ansible Task StdOut -------------------------------
TASK [csiscaleoperator : Ensure the clusters are valid] ************************
task path: /opt/ansible/roles/csiscaleoperator/tasks/main.yml:92
-------------------------------------------------------------------------------
{"level":"info","ts":1642741732.0784554,"logger":"logging_event_handler","msg":"[playbook task]","name":"ibm-spectrum-scale-csi","namespace":"ibm-spectrum-scale-csi-driver","gvk":"csi.ibm.com/v1, Kind=CSIScaleOperator","event_type":"playbook_on_task_start","job":"7504504064263669287","EventData.Name":"csiscaleoperator : Ensure secret guisecret defined in ibm-spectrum-scale-csi-driver"}
--------------------------- Ansible Task StdOut -------------------------------
TASK [csiscaleoperator : Ensure secret guisecret defined in ibm-spectrum-scale-csi-driver] ***
task path: /opt/ansible/roles/csiscaleoperator/tasks/cluster_check.yml:2
-------------------------------------------------------------------------------
{"level":"info","ts":1642741734.6205087,"logger":"proxy","msg":"Cache miss: /v1, Kind=Secret, ibm-spectrum-scale-csi-driver/guisecret"}
{"level":"info","ts":1642741735.137228,"logger":"runner","msg":"Ansible-runner exited successfully","job":"7504504064263669287","name":"ibm-spectrum-scale-csi","namespace":"ibm-spectrum-scale-csi-driver"}
----- Ansible Task Status Event StdOut (csi.ibm.com/v1, Kind=CSIScaleOperator, ibm-spectrum-scale-csi/ibm-spectrum-scale-csi-driver) -----
PLAY RECAP *********************************************************************
localhost : ok=4 changed=0 unreachable=0 failed=0 skipped=8 rescued=0 ignored=0
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Third command output
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[root@ ~]# kubectl describe csiscaleoperator -n ibm-spectrum-scale-csi-driver
No resources found in ibm-spectrum-scale-csi-driver namespace.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------
Jickriya Jamadar
------------------------------
Original Message:
Sent: Mon January 17, 2022 09:32 AM
From: Harald Seipp
Subject: IBM Spectrum Scale CSI driver v1.1.0 with Rancher v2.5.9
To check the Custom Resource, use
kubectl describe csiscaleoperator -n ibm-spectrum-scale-csi-driver
------------------------------
Harald Seipp
Senior Technical Staff Member
Original Message:
Sent: Mon January 17, 2022 06:24 AM
From: Harald Seipp
Subject: IBM Spectrum Scale CSI driver v1.1.0 with Rancher v2.5.9
Hi Jickriya,
you might try:
Best regards,
------------------------------
Harald Seipp
Senior Technical Staff Member
Original Message:
Sent: Mon January 17, 2022 02:05 AM
From: Jickriya Jamadar
Subject: IBM Spectrum Scale CSI driver v1.1.0 with Rancher v2.5.9
Hi Harald,
As per your suggestion I have created setup with 2.2.0 IBM driver with Rancher 2.5.9 and Kubernetes cluster 1.19.xx,
Also I have disabled POD security.
However still Attacher and Provisioner PODs are not getting created.
Do let me know if I am missing something or we need to check any logs.
Please do let me know.
------------------------------
Jickriya Jamadar
Original Message:
Sent: Tue January 11, 2022 03:25 AM
From: Harald Seipp
Subject: IBM Spectrum Scale CSI driver v1.1.0 with Rancher v2.5.9
Hi Jickriya,
please first of all note that RKE is not officially supported with Spectrum Scale CSI. To get a supported environment, you might use Rancher as management interface for a plain Kubernetes installation.
Based on my own experiences experimenting with Rancher 2.5.9 / RKE I'd recommend for experimental environments to
- Use Spectrum Scale CSI version 2.2.0 or later with Kubernetes 1.19 or 1.20
- Disable Pod Security Policy support (this is deprecated in Kubernetes anyway)
You can find a summary of my early RKE experiments on slide 22 of the following presentation: https://www.spectrumscaleug.org/wp-content/uploads/2020/03/SSSD20DE-Spectrum-Scale-use-cases-and-field-lessons-learned-with-Kubernetes-and-OpenShift.pdf
Best regards,
------------------------------
Harald Seipp
Senior Technical Staff Member
Original Message:
Sent: Mon January 10, 2022 02:02 AM
From: Jickriya Jamadar
Subject: IBM Spectrum Scale CSI driver v1.1.0 with Rancher v2.5.9
Hi,
I have created following setup;
1) 2 node Spectrum scale Cluster 5.1.0.2 on RHEL 7
2) Create RKE (Racher) v2.5.9 Kubermetes v1.17.x setup on spectrum scale cluster.
3) Deployed spectrum scale CSI driver v1.1.0.
After creating setup I am seeing that attacher and provisioner PODs are not not getting started.
Could you please help me to understand why these pods are not got started.
Secondly what are the steps to start or create both pods manually.
------------------------------
Jickriya Jamadar
------------------------------