etcd troubleshooting and backup
In the Certified Kubernetes Administrator (CKA) exam, etcd is the critical component storing the entire cluster state. Mastery of its maintenance is essential. **Troubleshooting** If the API server cannot communicate with etcd, the cluster becomes unresponsive. To troubleshoot: 1. **Check Process/Pod:** If etcd runs as a static pod, ensure the container is running via `crictl ps`. If it is a systemd service, check `systemctl status etcd`. 2. **Verify Health:** Use the command line tool. Ensure `ETCDCTL_API=3` is set. Run `etcdctl endpoint health` and `etcdctl endpoint status --write-out=table`. You must provide TLS flags (`--cacert`, `--cert`, `--key`), typically found in `/etc/kubernetes/pki/etcd`. 3. **Analyze Logs:** Check logs for 'fsync' latency warnings (indicating disk I/O issues) or space quota errors. **Backup** Kubernetes does not back up automatically; you must create snapshots. Command: `ETCDCTL_API=3 etcdctl snapshot save <path-to-backup> [flags]`. Use the necessary TLS certificates for authentication. Verify the snapshot integrity afterwards using `etcdctl snapshot status <path-to-backup>`. **Restore** Restoring is a destructive action for the current state. 1. **Stop Access:** It is often best to stop the static pod or API server to prevent writes. 2. **Restore Command:** Run `etcdctl snapshot restore <path-to-backup> --data-dir <new-directory>`. Do not overwrite the existing data directory; restore to a new path to isolate the restored state. 3. **Update Manifest:** Edit the etcd static pod manifest (usually `/etc/kubernetes/manifests/etcd.yaml`). Change the `hostPath` volume configuration to point to the `<new-directory>` created in the previous step. 4. **Restart:** Kubelet will restart the pod upon manifest modification. Verify the cluster state recovers once the pod is running.
etcd Troubleshooting and Backup Guide
etcd is a consistent and highly-available key-value store used as the backing store for all Kubernetes cluster data. It is the "brain" of the cluster; without it, the API server cannot retrieve or save state, and the cluster essentially freezes. In the CKA exam, this is a critical topic because a corrupted cluster state requires a restore from backup to recover functionality. Mastery of these commands is essential for Disaster Recovery scenarios.
How it Works
etcd typically runs as a static pod on the Control Plane node. It requires strict authentication using TLS certificates. The primary tool for interacting with it is etcdctl.
The workflow generally involves:
1. Backup: Taking a snapshot of the database.
2. Restore: Generating a new data directory from a snapshot file.
3. Reconfiguration: Pointing the etcd static pod to the new restored data directory.
Troubleshooting Common Issues
If etcd is failing, check the following:
Pod Status: Is the etcd-master pod running? (kubectl get pods -n kube-system)
Logs: Check logs for permission denied errors or disk space issues (kubectl logs etcd-master -n kube-system)
Endpoint Health: Verify if the cluster member is healthy (etcdctl endpoint health)
How to Answer Questions on etcd in the Exam
1. Locate Certificates: First, inspect the etcd static pod manifest (usually at /etc/kubernetes/manifests/etcd.yaml). Look for --trusted-ca-file, --cert-file, and --key-file.
2. Authenticate: Construct your command using these certificates. To save time, export them as environment variables:
export ETCDCTL_API=3
export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
export ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt
export ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key
3. Snapshot Save: Run etcdctl snapshot save /path/to/backup.db.
4. Snapshot Restore: Run etcdctl snapshot restore /path/to/backup.db --data-dir /var/lib/etcd-restored. Note that you must restore to a new directory, not the existing one.
5. Update Manifest: Edit /etc/kubernetes/manifests/etcd.yaml. Update the hostPath of the volume to point to your new directory (/var/lib/etcd-restored). The kubelet will detect the file change and restart the pod with the restored data.
Exam Tips: Answering Questions on etcd troubleshooting and backup
API Version: Always ensure you are using API version 3. If you type etcdctl and see a warning about version 2, export ETCDCTL_API=3 immediately.
Use Internal IP: When defining the endpoint, use https://127.0.0.1:2379 if you are on the control plane node to avoid network issues.
Volume Mounts: When restoring, the most common mistake is forgetting to update the volumes: hostPath in the static pod YAML. If you don't update this, etcd will restart using the old, potentially corrupted data.
Verification: After restoring, wait 60 seconds and run kubectl get pods -n kube-system to ensure the etcd pod is back in a Running state.