Prompt Diffing: Treat Prompts Like Code to Improve Reliability

If you’ve ever “improved” a prompt, got a nicer output once, and then couldn’t reproduce it a week later… you’ve experienced prompt drift. Most teams treat prompts like notes: ad‑hoc edits, no tests, no record of why something changed. Then a tiny tweak (“be concise”) silently breaks an edge case (“keep all numbers”). The fix is boring and powerful: treat prompts like code . Specifically, use prompt diffing : make small changes, compare outputs across a fixed set of inputs, and keep a changelog so you can tell whether you actually improved the system. This post gives you a lightweight workflow you can start today. What “prompt diffing” means Prompt diffing is the same idea as a git diff : You have a baseline prompt (v1). You make a small, intentional change (v2). You run both versions against the same test inputs . You compare the outputs side by side. You keep the version that wins and write down what changed. The goal isn’t perfection. It’s reliability . Step 0: Put prompts in files