Cloud User Group

IBM Cloud User Group

 View Only

Catch Cluster Health Issues Early on IBM Cloud Kubernetes Services

By Tamás SOÓS posted 05/11/26 05:38 AM

  

Keeping production Kubernetes environments healthy is rarely simple. As dependencies across networking, ingress, and platform services grow, small faults can spread quickly. IBM Cloud Kubernetes Service (IKS 1.33+ and ROKS 4.20+) introduces Cluster Health to catch those issues early and direct teams toward practical next steps.

The Challenge: Early Detection Matters

Running a Kubernetes cluster in production means balancing compute, networking, and orchestration all at once. When something breaks, root cause is rarely obvious. Is DNS failing on a subset of nodes? Is a network policy blocking traffic? Is registry access degraded? Without clear signals, troubleshooting quickly becomes a long, high-pressure investigation.

Cluster Health brings those signals forward before incidents escalate. It continuously evaluates cluster conditions and presents common problems in a straightforward, actionable view.

cluster-health-concept-flow


From partial visibility to clear action: detect, aggregate, report health via CLI and API, then follow actionable guidance.

Introducing Cluster Health

At its core, Cluster Health adds diagnostics that you can consume from the IBM Cloud console, the ibmcloud CLI, or IBM Cloud Kubernetes Service APIs:

  • Automatically detects health issues affecting your cluster
  • Categorizes issues by component and severity
  • Provides context about affected resources
  • Suggests solutions for resolving identified problems

Because it is available through the existing ibmcloud CLI, teams can adopt it without changing how they operate.

Getting Started: Basic CLI commands

CLI flow


Cluster Health CLI flow from the entry command through listing issues, detailed diagnostics, and remediation

You access Cluster Health through the ibmcloud ks cluster health command hierarchy:

ibmcloud ks cluster health

From there, you list detected issues and request details for any issue by ID—either from the CLI or through the Kubernetes Service API.

Listing Issues

To see all health issues affecting a specific cluster:

ibmcloud ks cluster health issues --cluster <cluster-name-or-id>

This command returns a concise summary of each issue, including ID, component, severity, and description:

$ ibmcloud ks cluster health issues --cluster d7o9ce42052m6kjvtip0
Retrieving health issues for cluster d7o9ce42052m6kjvtip0
OK
ID         Component   Severity   Description
ERRADRUH   Ingress     Warning    One or more ALB pod is not in running state (ERRADRUH).
NHC004     Network     Error      Some worker nodes in the cluster cannot resolve VPE gateway hostnames.

Note: Replace d7o9ce42052m6kjvtip0 with your actual cluster name or ID. ALB means Application Load Balancer.

Getting Detailed Issue Information

When you need additional context for triage, use the detailed issue command:

ibmcloud ks cluster health issue get --cluster <cluster-name-or-id> --issue <issue-id>

This command returns the core details most teams need to assess impact and choose a response:

  • Issue Description: What the problem is
  • Affected Component: Which cluster subsystem is impacted
  • Severity Level: How critical the issue is
    • Warning: The component might not function properly.
    • Error: The component is malfunctioning.
  • Detection Time: When the issue was first identified
  • Possible Solution: Recommended steps to investigate and fix
  • More Details: Specific affected resources (e.g., which worker nodes are impacted)

Real-World Examples

Investigating Ingress Issues: ERRADRUH Example

Consider a common scenario: your team sees intermittent reachability problems and needs a fast starting point. You run a cluster health check:

$ ibmcloud ks cluster health issues --cluster d7o9ce42052m6kjvtip0
Retrieving health issues for cluster d7o9ce42052m6kjvtip0
OK
ID         Component   Severity   Description
ERRADRUH   Ingress     Warning    One or more ALB pod is not in running state (ERRADRUH).
NHC004     Network     Error      Some worker nodes in the cluster can not resolve VPE gateway hostnames.

Note: Replace d7o9ce42052m6kjvtip0 with your actual cluster name or ID The Application Load Balancer (ALB) pod handles incoming traffic and routes it to the appropriate backend service. The ALB pod must be healthy before the deployment can proceed.

The output surfaces both Network and Ingress findings in one place. Next, you drill into the ingress warning:

ibmcloud ks cluster health issue get --cluster d7o9ce42052m6kjvtip0 --issue ERRADRUH

Note: Replace d7o9ce42052m6kjvtip0 with your actual cluster name or ID

The detailed output reveals:

  • Problem: One or more ALBs have pods that are not running.
  • Affected Component: Ingress
  • Solution Guidance: Follow the remediation guidance in the issue details to validate ALB health checks, ingress configuration, and related networking controls. For step-by-step guide see: Why does the Ingress status show an ERRADRUH error?

Instead of spending hours combing through logs first, you get a focused path forward: validate ALB health checks, ingress configuration, and related network controls.

Investigating Network Issues: NHC004 Example

Now consider a network-focused case. The same issue list flags NHC004, which indicates that some worker nodes cannot resolve VPE gateway hostnames. To investigate, you request detailed diagnostics for that issue:

ibmcloud ks cluster health issue get --cluster d7o9ce42052m6kjvtip0 --issue NHC004

The detailed output helps your team move from detection to action:

  • Problem: Some worker nodes cannot resolve VPE gateway hostnames.
  • Affected Component: Network
  • Likely Impact: Pull failures, control-plane communication delays, or intermittent connectivity to private endpoints.
  • Solution Guidance: Validate DNS resolver reachability from affected nodes, review VPE DNS records, and confirm that network controls allow required DNS traffic. For step-by-step guide see: Why does the Network status show an NHC004 error?

With that context, troubleshooting becomes targeted: start with the affected worker nodes listed in the issue details, then verify DNS and VPE configuration end to end.

Understanding Common Issues

Cluster Health evaluates multiple layers of the stack. Common examples include:

Network Issues

Network problems are among the most common issues detected. Examples include:

  • DNS Resolution Failures - Worker nodes unable to reach DNS resolvers
  • Network Plugin Degradation - Calico or other network plugins in unhealthy states
  • Registry Connectivity - Worker nodes unable to reach container image registries
  • Pod-to-Pod Communication - Issues affecting inter-pod connectivity

Ingress Issues

Cluster Health also surfaces Ingress-related conditions, for example:

  • ALB Health Check Failures - ALB instances unable to respond to health requests
  • Ingress Reachability Warnings - Conditions that can impact external traffic entering workloads

Why Detection Matters

Issues like these can quietly erode application reliability. DNS resolution failures can trigger intermittent pod startup errors, while network plugin degradation can introduce packet loss or latency. Early detection helps teams respond before users see impact.

Output Options

For teams integrating diagnostics into automation pipelines or monitoring workflows, JSON output is available:

ibmcloud ks cluster health issues --cluster <cluster-name-or-id> --output json

Monitor health issues using IBM Cloud Kubernetes Service API

Call the Kubernetes Service APIs directly to list issues for a cluster and to fetch full detail for a given issue ID—the same operations the CLI performs against the aggregation layer. That fits service-to-service integrations, CI pipelines, and environments where installing the CLI is not practical.
 
Both approaches expose the same underlying health data; pick the one that matches your toolchain. For operation names and schemas, see the https://cloud.ibm.com/apidocs/kubernetes/containers-v3#getclusterhealthissue reference.

Best Practices

To get consistent value from Cluster Health, adopt a few operational habits:

  1. Act on Errors: Issues marked with "Error" severity should be investigated and resolved promptly. These indicate real problems affecting your cluster.

  2. Regular Checks: Incorporate ibmcloud ks cluster health issue get into your regular cluster maintenance routine (or schedule the equivalent API calls).

  3. Use Detailed Output: Always fetch full issue details—via ibmcloud ks cluster health issue get or the matching API—before attempting remediation.  

  4. Monitor Affected Resources: When an issue lists specific worker nodes or components, focus your investigation there first.

  5. Automate Monitoring: For production clusters, consider driving health checks from the CLI, from API clients, or both, and integrating results into your monitoring platform.

What's Next

Cluster Health reflects a practical shift from reactive troubleshooting to earlier, evidence-driven intervention. With faster visibility into emerging issues, teams can reduce mean time to resolution and lower the risk of avoidable incidents.

As environments grow and workloads become more business-critical, that clarity matters even more. Cluster Health supports day-to-day Kubernetes operations with faster diagnosis and clearer remediation direction.

Ready to go deeper?

Use the resources below to put cluster health checks into daily practice.

0 comments
28 views

Permalink