Building a Security Scanner with Claude Code Skills - How I Tackled LLM's "p-hacking" Problem

The Problem That Emerged from Previous Articles In my previous article, Claude Code Security: 500+ Zero-Days Found, Security Stocks Crash 9.4% , I covered Anthropic's announcement of Claude Code Security. It's genuinely impressive technology, but it's Enterprise/Team only - individual developers like me can't use it yet. Meanwhile, Snyk's research shows that 36.8% of free Skills have security issues . There's no review process for the Skills marketplace, and Anthropic's own documentation states that "security verification of SKILL.md is not performed." Waiting for the Enterprise version wasn't going to help, so I built my own security scanner using Claude Code Skills . With nothing but a SKILL.md definition, you can build a hybrid scanner combining static pattern matching and LLM semantic analysis. But here's what I didn't expect: building the scanner was the easy part. The real challenge was a fundamental issue with LLM-based tools - the same input can produce different results every