Back to articles
A hard-earned rule from incident retrospectives:
How-ToTools

A hard-earned rule from incident retrospectives:

via Dev.toNeeraja Khanapure

LinkedIn Draft — Workflow (2026-03-28) A hard-earned rule from incident retrospectives: GitOps drift: the silent accumulation that makes clusters unmanageable GitOps promises Git as the source of truth. The reality: every manual kubectl during an incident is a lie you told your cluster and forgot to retract. GitOps truth gap over time: Week 1: Git ══════════ Cluster (clean) Week 4: Git ══════╌╌╌╌ Cluster (2 manual patches) Week 12: Git ════╌╌╌╌╌╌╌╌╌╌╌╌╌ (drift accumulates) Cluster (unknown state) Where it breaks: ▸ Manual patches during incidents create cluster state Git doesn't know about — Argo/Flux will overwrite it silently. ▸ Secrets managed outside GitOps (sealed-secrets, Vault agent) drift independently — invisible in sync status. ▸ Multi-cluster setups multiply drift: each cluster diverges at its own pace once human intervention happens. The rule I keep coming back to: → Treat every manual cluster change as a 5-minute loan. Commit it back to Git before the incident closes — or

Continue reading on Dev.to

Opens in a new tab

Read Full Article
7 views

Related Articles