Local LLM Agent Benchmark: Comparing 6 Models in Real-World Scenarios

Measuring AI agent performance by actual outcome correctness , not just tool call presence Why We Built This Benchmark "To make it accessible for general users, it is crucial to find an LLM with the lowest possible VRAM footprint. Most LLM benchmarks evaluate models on academic metrics like MMLU, HumanEval, or HellaSwag. But for tool-using AI agents , what truly matters isn't "did it call the right tool?" — it's "did it actually produce the correct result?" Our project Androi is a local AI agent that uses 10+ tools including web search, Python execution, file management, email, and calendar. We connected various LLMs to the same agent and ran 5 identical complex real-world scenarios , scoring each based on the correctness of their outputs. Test Environment Server : Ubuntu VM (3.8GB RAM, 20GB SSD) Runtime : Ollama (local inference) Framework : Androi Agent (Node.js + Python tool pipeline) Validation : Outcome-Based Validation (v2) Test Date : 2026-02-28 The 5 Real-World Test Scenarios (