How I Rebuilt CSV Imports to Handle 10,000 Messy Rows Without Breaking

If you've ever built a CSV importer, you know it looks simple — until real users show up with messy data. Ambiguous dates, duplicate rows, partial updates, "Acme Corp" spelled three different ways. The basic parse-validate-insert pipeline crumbles fast. I'm building Relaticle , an open-source CRM, and for V3 I completely rebuilt the import system as a dedicated module with a strict workflow, staged processing, and explicit failure handling. Here's the architecture and the tradeoffs behind it. Why I Rebuilt Imports for V3 The old pattern most teams start with is straightforward: Parse CSV Validate in memory Insert/update directly That works for small clean files. It falls apart the moment someone uploads a CSV where "Acme Corp" appears three different ways, dates mix MM/DD and DD/MM , and half the rows should update existing records instead of creating duplicates. What you actually need: Matching against existing records without false positives Safe handling of relationships (company, c