Fuzzy-match millions of rows in Databricks (2026)

When you fuzzy-match 10 million rows, you aren't "just comparing strings." A naïve dedupe implies roughly n(n−1)/2 ≈ 5×10¹³ potential pairs. At this scale, approaches that feel "quick" on small tables start to break. In Databricks, most teams reach for one of three options: Spark-native candidate generation (LSH/MinHash) Fast to start, but you end up tuning a tradeoff between missed matches and huge candidate sets. Entity-resolution frameworks Powerful, but often heavier than you want for "dedupe this column." Custom Python scoring (UDFs / pandas UDFs) Easy to prototype, but at large scale jobs become dominated by Python overhead, skew, and shuffles. A practical approach is to let Databricks handle what it's best at (data access, ETL, governance) and offload the actual matching step to a service built specifically for high-scale deduplication. In this tutorial, we'll do that using Similarity API — an async "job" style matching service where you: upload a dataset once (CSV or Parquet) s