
A hard-earned rule from incident retrospectives:
LinkedIn Draft — Workflow (2026-03-28) A hard-earned rule from incident retrospectives: GitOps drift: the silent accumulation that makes clusters unmanageable GitOps promises Git as the source of truth. The reality: every manual kubectl during an incident is a lie you told your cluster and forgot to retract. GitOps truth gap over time: Week 1: Git ══════════ Cluster (clean) Week 4: Git ══════╌╌╌╌ Cluster (2 manual patches) Week 12: Git ════╌╌╌╌╌╌╌╌╌╌╌╌╌ (drift accumulates) Cluster (unknown state) Where it breaks: ▸ Manual patches during incidents create cluster state Git doesn't know about — Argo/Flux will overwrite it silently. ▸ Secrets managed outside GitOps (sealed-secrets, Vault agent) drift independently — invisible in sync status. ▸ Multi-cluster setups multiply drift: each cluster diverges at its own pace once human intervention happens. The rule I keep coming back to: → Treat every manual cluster change as a 5-minute loan. Commit it back to Git before the incident closes — or
Continue reading on Dev.to
Opens in a new tab

