Building a ‘simple’ async service in Rust (and why it wasn’t simple)

I thought this async Rust service would be simple I wanted to build a small async service in Rust. Accept events, process them, retry on failure. Nothing fancy. It looked like a weekend project. It turned into a lesson in how quickly “simple” systems stop being simple once you care about correctness. The full project is available here: https://github.com/yourname/eventful The naive version The initial design looked something like this: HTTP → queue → worker pool Handler receives an event Push it into a channel Workers pull from the channel and process That works fine — until you actually try to make it correct. As soon as you introduce retries, idempotency, and failure handling, things start to break in ways that aren’t obvious at first. Problem 1: Idempotency isn’t just “don’t insert twice” I wanted ingestion to be idempotent by event_id . At first, that just meant: If the ID exists, return the existing record Otherwise insert it But that leaves a hole. What if the same ID comes in wi