I Built a Tool to Test Whether Multiple LLMs Working Together Can Beat a Single Model

The Question Can you get a better answer by having multiple LLMs collaborate than by just asking one directly? That's the thesis behind Occursus Benchmark — an open-source benchmarking platform that systematically tests multi-model LLM synthesis pipelines against single-model baselines across 4 providers and 22 orchestration strategies. What It Does Occursus Benchmark runs the same task through 22 different orchestration strategies — from a simple single-model call to a 13-call graph-mesh collaboration — and scores every output using dual blind judging (two frontier models score independently on a 0-100 scale, averaged). This tells you whether adding pipeline complexity actually improves quality, or just burns tokens and money. The tool supports 4 LLM providers : Ollama (local/free), OpenAI (GPT-4o), Anthropic (Claude Sonnet 4), and Google Gemini. You toggle models on and off; the tool auto-assigns them to pipeline roles (generator, critic, synthesizer, reviewer). The 22 Pipelines Tier