Instana

Instana

The community for performance and observability professionals to learn, to share ideas, and to connect with others.

 View Only

  • 1.  Creating kubernetes node alert in SchedulingDisabled

    Posted 09/06/24 02:15 PM

    Kind regards,
    I hope you are all well. I would like to know if there is a way to create an alert in Instana IBM, so that when a node is in Cordon status, it generates a status alert. I have made several configurations but it has not been possible and in custom metrics, the entity type only appears in Java. Could you please help or guide me? Thank you in advance.




    ------------------------------
    Carlos Valencia
    ------------------------------


  • 2.  RE: Creating kubernetes node alert in SchedulingDisabled

    Posted 21 days ago

    Alerting on a Cordoned Kubernetes Node in Instana

    Why "Custom Metrics" won't work here

    Custom metrics in Instana are for application/JVM-emitted metrics (hence only Java appearing as an entity type). A cordoned node is a Kubernetes infrastructure state, not a metric - so a different alert path is needed.

    The correct approach: Smart Alerts → Kubernetes Events + Built-in Infrastructure

    There are two viable methods, depending on your Instana version.

    Method 1 - Smart Alert on a Built-in Kubernetes Metric (Recommended)

    Instana collects a built-in metric called kubernetes.node.unschedulable from the Kubernetes API. A value of 1 means the node is cordoned.

    Steps:

    1. Go to Alerts → Smart Alerts → + New Smart Alert
    2. Set Alert Type → Metric Threshold
    3. Under Scope, choose:
      • Entity Type: Kubernetes Node
      • Filter: your cluster (or leave blank for all nodes)
    4. Under Metric, select:
      • Metric: Node Unschedulable (maps to kubernetes.node.unschedulable)
      • Condition: ≥ 1
      • Aggregation: max
      • Window: 1 minute (shortest available - avoids flapping)
    5. Set Severity → Warning or Critical
    6. Assign an Alert Channel (email, Slack, PagerDuty, etc.)
    7. Save and enable the alert

    ⚠️ If Node Unschedulable does not appear in the metric picker, confirm that your Instana agent DaemonSet has the Kubernetes sensor enabled and that it has RBAC permissions to read nodes.

    Method 2 - Kubernetes Events Alert

    Instana can also ingest Kubernetes events. A cordon emits an event of type Normal with reason NodeNotSchedulable.

    1. Go to Alerts → Smart Alerts → + New Smart Alert
    2. Set Alert Type → Event
    3. Under Event Source, choose Kubernetes
    4. Filter by:
      • Kind: Node
      • Reason: NodeNotSchedulable
    5. Assign severity and alert channel, then save

    Why Only Java Appears in Custom Metrics

    What you tried Why it doesn't work
    Custom Metrics → entity type filter Custom metrics only shows entity types that have emitted a custom metric via SDK/agent API - Kubernetes nodes don't do this
    Manual metric name entry The metric namespace for K8s nodes is internal (kubernetes.*), not user-defined

    Prerequisites Checklist

    Before either method will work, verify:

    •  The Instana agent is deployed as a DaemonSet in the cluster
    •  The agent ConfigMap includes com.instana.plugin.kubernetes: enabled: true
    •  The agent's ClusterRole includes get, list, watch on nodes
    •  Your Instana license covers Infrastructure Monitoring (not just APM)
    •  You are on Instana agent version ≥ 1.2.x (older versions did not expose node.unschedulable)

    If after checking all of the above kubernetes.node.unschedulable still doesn't appear, share a screenshot of your Smart Alert metric picker filtered to Kubernetes Node and I can help narrow it down further.



    ------------------------------
    Miodrag Mikrut
    ------------------------------