
I Built a Stream Processor That Only Recomputes What Changed
I spent weeks studying how incremental computation works in production trading systems. Not the papers. The actual implementations. How self-adjusting computation engines track dependencies, propagate changes, and avoid redundant work. One thing kept bothering me: the model is incredibly powerful, but it's locked inside single-process libraries. If you want surgical recomputation — where changing one input only touches the nodes that actually depend on it — you have to give up distribution. If you want distribution, you're back to recomputing entire windows on every tick. That gap is where Ripple came from. The experiment that started it I built a prototype. A simple incremental graph: 10,000 leaf nodes (one per stock symbol), each feeding through a map node into a fold that aggregates them all. The question was simple: when one leaf changes, how many nodes actually need to recompute? The answer should be 3. The leaf, its map, and the fold. Not 10,000. Not 40,000. Three. The first impl
Continue reading on Dev.to
Opens in a new tab




