How to Detect LLM Drift Before It Breaks Your Users

The most common LLM production incident I see is not prompt injection or model hallucinations. It is silent quality degradation — the model outputs look fine, but they are subtly worse than they used to be. This is LLM drift. Here is how to detect it before it breaks your users. What Drift Looks Like You shipped a classification endpoint in January. It was 94% accurate. In March, you check and it is 89% accurate. You did not change anything. The model provider changed something. This happens. Providers update models, fine-tune weights, change inference infrastructure. The model name is the same. The model behavior is different. The Simple Detection Method Run your prompt with 10 baseline inputs Store the outputs as your "golden" set Re-run weekly with the same inputs Compare new outputs to golden outputs using embedding similarity If similarity drops below 0.85, investigate. The Code from sklearn.metrics.pairwise import cosine_similarity import numpy as np BASELINE_OUTPUTS = [...] # Yo