Building an Incremental Zoho Desk to BigQuery Pipeline: Lessons from the Trenches

What I thought would take a few days ended up taking weeks so here's everything I learned building a production Zoho Desk to BigQuery pipeline from scratch. When my company decided to centralise customer support analytics, the task landed on my plate: pull data from Zoho Desk, land it in BigQuery, transform with dbt, done. The plan looked clean on paper. What followed was a masterclass in why production data engineering is never as simple as the happy path suggests. This is the story of building that pipeline; the architecture decisions, the walls I hit, and the lessons I'll carry into every pipeline I build from here. The Starting Point: A Full Load That Took Forever The first working version of the pipeline was blunt but functional. Every day, pull every ticket ever created from the Zoho Desk API and overwrite the BigQuery table. Simple. Predictable. And for tickets, completely untenable at scale because the table had over a million rows and was growing daily. I needed incremental lo