How I Replaced LLM-Based Code Analysis with Static Analysis (And Got Better Results)

When I started building a security scanner for AI-generated code, I did what everyone does in 2026: I threw an LLM at it. That was a mistake. Here's why I ripped it out and replaced it with static analysis — and why the results are objectively better. The LLM Approach (Week 1) The idea was simple: feed code into an LLM, ask it to identify security vulnerabilities, return a severity score. Modern, elegant, "AI-powered." I built the prototype in a day. It worked... sort of. Input: eval(user_input) Run 1: Severity 8.5 - "Critical command injection vulnerability" Run 2: Severity 6.2 - "Moderate risk, depends on context" Run 3: Severity 9.1 - "Extremely dangerous, immediate fix required" Run 4: Severity 7.0 - "High risk injection vector" Run 5: Severity 8.5 - "Critical vulnerability" Same code. Five runs. Five different answers. The severity scores ranged from 6.2 to 9.1. This is not a security tool. This is a random number generator with opinions. The p-Hacking Problem If you're not famili