Kubernetes Explained: From API Request to Self-Healing Systems

Kubernetes often looks complex at first glance—dozens of components, YAML files, controllers, and abstractions. But underneath it all lies a simple, powerful idea:

Kubernetes is a distributed system that continuously reconciles actual state with desired state.

This blog breaks down how Kubernetes really works behind the scenes—from request flow to pod lifecycle, RBAC decisions, and self-healing behavior.

1. The Core Philosophy: Declarative State

In Kubernetes, you don’t tell the system how to do things. You define what you want.

apiVersion: apps/v1
kind: Deployment
spec:
  replicas: 3

This means:

“Ensure 3 replicas are always running.”

Kubernetes takes responsibility for maintaining this state—even if nodes fail or pods crash.

2. Cluster Architecture: Control Plane vs Worker Nodes

A Kubernetes cluster has two main parts:

Control Plane (The Brain)

• API Server → Entry point for all operations

• Scheduler → Assigns pods to nodes

• Controller Manager → Ensures desired state

Worker Nodes (The Muscle)

• Kubelet → Executes instructions from control plane

• Container Runtime → Runs containers

• Pods → Actual workloads

Mental model:

• Control Plane decides

• Worker Nodes execute

3. The Request Flow: What Happens When You Apply YAML?

When you run:

kubectl apply -f deployment.yaml

The flow looks like:

kubectl → API Server → etcd → Scheduler → Node → Kubelet → Pod

Step-by-step:

1. API Server validates request

2. Stores desired state in etcd

3. Scheduler picks a node

4. Kubelet creates the pod

Important:
Everything flows through the API Server—no direct communication between components.

4. Read vs Write Paths

Write Path (State Changes)

User → API Server → etcd → Controllers → Kubelet

• Persistent

• Triggers reconciliation

Read Path (Fetching State)

Component → API Server → Cache / etcd

• Optimized for speed

• Often served from in-memory cache

5. The Heart of Kubernetes: Reconciliation Loop

Every controller follows a control loop:

1. Observe current state  
2. Compare with desired state  
3. Decide action  
4. Act  
5. Verify (self-heal)

Example:

Desired: 3 pods
Actual: 2 pods

→ Controller creates 1 more pod

This loop runs continuously, making Kubernetes self-healing by design.

6. RBAC: How Authorization Decisions Are Made

When a request hits the API Server:

Step 1: Authentication

Identify the user/service account

Step 2: Authorization (RBAC)

Check:

• Role / ClusterRole

• RoleBinding / ClusterRoleBinding

Step 3: Decision

• If any rule matches → ALLOW

• Else → DENY

Key rule: RBAC is additive. There are no explicit deny rules.

7. Pod Lifecycle: Creation to Deletion

Pod Creation Flow

API Server → Scheduler → Node → Kubelet → Container Runtime → Running Pod

Pod Deletion Flow

When you delete a pod:

1. API Server sets deletionTimestamp

2. Pod enters Terminating state

3. Removed from Service endpoints

4. preStop hook runs (if defined)

5. SIGTERM sent to container

6. Graceful shutdown period

7. SIGKILL (if needed)

8. Pod removed from cluster

Key takeaway:
Deletion is graceful, not instant.

8. DaemonSets: Running One Pod Per Node

DaemonSets ensure:

“Run exactly one pod on every node.”

Common use cases:

• Logging agents

• Monitoring tools

• Networking components

Troubleshooting Unhealthy DaemonSets

Check in this order:

1. Scheduling issues

   • NodeSelector mismatch

   • Missing tolerations

2. Node health

   • Node not Ready

   • Resource pressure

3. Pod issues

   • CrashLoopBackOff

   • ImagePull errors

4. Configuration issues

   • Missing ConfigMaps/Secrets

   • RBAC permissions

9. Why Kubernetes Feels Complex (But Isn’t)

Kubernetes complexity comes from:

• Distributed architecture

• Eventual consistency

• Multiple independent controllers

But the core idea remains simple:

Kubernetes is just a control loop system acting on a desired state stored in etcd.

10. Final Mental Model

You declare → API stores → Controllers watch → Scheduler assigns → Kubelet runs → System keeps fixing drift

Conclusion

Once you understand:

• Declarative state

• Control loops

• API-driven architecture

Kubernetes stops being “magic” and becomes predictable.

Everything—pods, deployments, scaling, failures—is just the system continuously working to match reality with what you declared.

Bonus Tip

If you’re debugging Kubernetes:

Always start with “What is the desired state vs actual state?”

That question alone solves most issues.

Thanks for reading!
If this helped, consider sharing or connecting to discuss Kubernetes, DevOps, and system design.