Why You Can't Reproduce AI Agent Failures (And Why That's a Huge Problem)

Why You Can't Reproduce AI Agent Failures (And Why That's a Huge Problem) If you've used Claude Code, Cursor, or any AI coding agent for more than a week, you've probably experienced this: The agent does something wrong. Maybe it deletes a file it shouldn't have. Maybe it rewrites your auth module and breaks everything. Maybe it makes a chain of 15 edits and somewhere in the middle, something went sideways. So you try to figure out what happened. You look at the conversation. You stare at the diffs. You try to piece together the sequence of events. And then you think "let me just re-run it and watch more carefully this time." And it does something completely different. The Nondeterminism Problem This isn't a bug. It's fundamental to how LLMs work. Every time an LLM generates a response, it's sampling from a probability distribution over possible next tokens. Temperature, top_p, and the inherent randomness in the sampling process mean that the same prompt can produce meaningfully differ