Originally posted by: Sylvain DELABARRE

Hello,

Just installed K8s on AIX 7.3 TL3 SP3. Everything seems ok, except that my AIX node is "NotReady"...
It seems like NetworkPluginNotReady is a hint, but I don't really know where to look at.

# kubectl get nodes
NAME             STATUS     ROLES                                 AGE   VERSION
10.7.14.20       NotReady   worker                                49m   v1.12.4+aix
maic-pp-master   Ready      etcd,management,master,proxy,worker   77d   v1.12.4+icp-ee

# kubectl describe nodes | head -30
Name:               10.7.14.20
Roles:              worker
Labels:             beta.kubernetes.io/arch=ppc64
                    beta.kubernetes.io/os=aix
                    kubernetes.io/hostname=10.7.14.20
                    node-role.kubernetes.io/worker=true
Annotations:        node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Fri, 28 Jun 2019 16:32:58 +0200
Taints:             ibm.com/aix=true:NoSchedule
                    node.kubernetes.io/not-ready:NoSchedule
Unschedulable:      false
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  OutOfDisk        False   Fri, 28 Jun 2019 17:24:52 +0200   Fri, 28 Jun 2019 17:21:22 +0200   KubeletHasSufficientDisk     kubelet has sufficient disk space available
  MemoryPressure   False   Fri, 28 Jun 2019 17:24:52 +0200   Fri, 28 Jun 2019 17:21:22 +0200   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Fri, 28 Jun 2019 17:24:52 +0200   Fri, 28 Jun 2019 17:21:22 +0200   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Fri, 28 Jun 2019 17:24:52 +0200   Fri, 28 Jun 2019 16:32:58 +0200   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Fri, 28 Jun 2019 17:24:52 +0200   Fri, 28 Jun 2019 17:21:22 +0200   KubeletNotReady              runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:criwpar: network plugin is not ready: cni config uninitialized
Addresses:
  InternalIP:  10.7.14.20
  Hostname:    10.7.14.20
Capacity:
 cpu:                40
 ephemeral-storage:  0
 memory:             32Gi
 pods:               400

Originally posted by: Chris Gibson

Just as a point of interest, initially, my AIX nodes were also showing as "NotReady". 
In the kubernetes log I saw an error message re: resolv.conf. 

Oct 15 10:40:47 aixk8-1 local0:err|error kubelet[7930152]: E1015 10:40:47.784114 7930152 pod_workers.go:186] Error syncing pod b3437fcc-d005-11e8-aee6-fa07a3479f20 ("calico-node-aix-wxpb6_kube-system(b3437fcc-d005-11e8-aee6-fa07a3479f20)"), skipping: failed to "CreatePodSandbox" for "calico-node-aix-wxpb6_kube-system(b3437fcc-d005-11e8-aee6-fa07a3479f20)" with CreatePodSandboxError: "GeneratePodSandboxConfig for pod \"calico-node-aix-wxpb6_kube-system(b3437fcc-d005-11e8-aee6-fa07a3479f20)\" failed: open /etc/resolv.conf: no such file or directory"

Once I created a /etc/resolv.conf file (on each AIX node), the status changed to "Ready". 

I hope this helps.

Originally posted by: vmansur

We worked with Sylvain Delabarre and found that the issue was with ICP (IBM Cloud Private) - the K8s master node failing to deploy calico daemonset on AIX nodes

# kubectl get ds/calico-node-aix -n kube-system
NAME              DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR               AGE
calico-node-aix   0         0         0       0            0           beta.kubernetes.io/os=aix   74s

# kubectl get events -n kube-system|grep Failed
2m58s       Warning   FailedCreate   DaemonSet    Error creating: Internal error occurred: failed calling admission webhook "spectrum-inject.ibm.com": Post https://spectrum-webhook.kube-system.svc:443/?timeout=30s: service "spectrum-webhook" not found
2m58s       Warning   FailedCreate   DaemonSet    Error creating: Internal error occurred: failed calling admission webhook "spectrum-inject.ibm.com": Post https://spectrum-webhook.kube-system.svc:443/?timeout=30s: service "spectrum-webhook" not found
2m58s       Warning   FailedCreate   Job          Error creating: Internal error occurred: failed calling admission webhook "spectrum-inject.ibm.com": Post https://spectrum-webhook.kube-system.svc:443/?timeout=30s: service "spectrum-webhook" not found
2m58s       Warning   FailedCreate   ReplicaSet   Error creating: Internal error occurred: failed calling admission webhook "spectrum-inject.ibm.com": Post https://spectrum-webhook.kube-system.svc:443/?timeout=30s: service "spectrum-webhook" not found

Sylvain is currently validating his ICP installation