Kubernetes ETCD Architecture: v2 vs v3 APIs, PKI Snapshots, and Node Eviction Timelines
Demystifying Kubernetes state management: configuring etcdctl API v3 with mTLS certificates, snapshot backups and restores, and how the node controller manages 5s heartbeats, 40s timeouts, and 5m eviction windows.
Watch the video
More on the channelIn a Kubernetes cluster, ETCD is the single source of truth. Every pod spec, namespace, secret, deployment, and node status is stored as a key-value record inside this distributed, consistent datastore.
When control planes degrade or disaster strikes, understanding how ETCD operates under the hood — and how controllers react to failure states — is what separates successful cluster recoveries from catastrophic data loss.
1. The ETCDCTL API v2 vs v3 Paradigm
etcdctl is the command-line utility used to communicate with ETCD. A frequent source of confusion during administration and CKA exams is the difference between API version 2 and API version 3.
# Check or configure API version in your shell
export ETCDCTL_API=3
[!IMPORTANT] If
ETCDCTL_APIis not set,etcdctldefaults to Version 2. In API v2, modern commands likesnapshot saveandendpoint healthwill fail with command not found errors!
Command Differences:
| API Version 2 (Legacy) | API Version 3 (Modern Kubernetes) | Purpose |
|---|---|---|
etcdctl backup | etcdctl snapshot save <file> | Snapshot database state |
etcdctl cluster-health | etcdctl endpoint health | Cluster quorum check |
etcdctl set /key value | etcdctl put /key value | Write key-value pair |
etcdctl get /key | etcdctl get /key | Read key |
2. Authenticating with ETCD mTLS Certificates
ETCD runs with mutual TLS (mTLS). To query it directly with etcdctl, you must provide the Certificate Authority (ca.crt), client certificate (server.crt), and private key (server.key):
# Standard PKI location on kubeadm control-plane nodes
--cacert /etc/kubernetes/pki/etcd/ca.crt
--cert /etc/kubernetes/pki/etcd/server.crt
--key /etc/kubernetes/pki/etcd/server.key
Inspecting Keys Inside the Control Plane
kubectl exec -n kube-system etcd-controlplane -- sh -c \
"ETCDCTL_API=3 etcdctl get / \
--prefix --keys-only --limit=10 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key"
Creating and Restoring a Cluster Snapshot
# Save snapshot
sudo ETCDCTL_API=3 etcdctl snapshot save /var/lib/etcd-snapshot.db \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
# Verify snapshot integrity
sudo ETCDCTL_API=3 etcdctl snapshot status /var/lib/etcd-snapshot.db --write-out=table
3. Controller Manager & The Node Failure Lifecycle
All Kubernetes controllers run packaged inside the kube-controller-manager. One of the most critical controllers is the Node Controller, which watches node health and orchestrates pod rescheduling during infrastructure outages.
The Three Critical Node Timeouts:
- Heartbeat Frequency (
nodeStatusUpdateFrequency: 5s): The Kubelet on each worker node reports its health tokube-apiserverevery 5 seconds. - Node Monitor Grace Period (
node-monitor-grace-period: 40s): If the API server does not hear from a node for 40 seconds, the Node Controller marks the node status asNotReadyorUnknown. - Pod Eviction Timeout (
pod-eviction-timeout: 5m): After a node is markedUnknown, Kubernetes gives the host a 5-minute grace period to recover before forcibly evicting pods and rescheduling them onto surviving cluster nodes.
4. Control Plane Core Summary
kube-apiserver: The front-door for all cluster requests. Authenticates users, validates admission webhooks, updates ETCD, and mediates controller communication.kube-scheduler: Watches for newly created pods with no assigned node and selects the optimal node based on resource requests, taints, and affinity rules.kube-proxy: Runs on each worker node as a virtual network router, synchronizing iptables or IPVS rules to forward service traffic to pod backends.kubelet: The node agent ensuring containers described in PodSpecs are running and healthy.