New Benchmark for Open-Source Agents: What is Claw-Eval? How Step 3.5 Flash Secured the #2 Spot

Recently, a new Agent evaluation framework called Claw-Eval has sparked significant discussion within the developer community. In its latest rankings, Step 3.5 Flash emerged as the #2 open-source model, trailing only GLM 5, while sharing the top spot for the Pass@3 metric. What makes this leaderboard unique is that it doesn't test "knowledge breadth" or "abstract reasoning." Instead, it focuses on a more fundamental question: Can the model actually call tools, execute steps, and complete tasks reliably in a real-world environment? Today, we’ll explore the design philosophy behind Claw-Eval and analyze why Step 3.5 Flash performed so exceptionally under this rigorous evaluation system. Claw-Eval: Testing "Doing," Not Just "Knowing" Developed by a joint team from Peking University and the University of Hong Kong, Claw-Eval features tasks that are entirely human-verified. Its positioning is clear: End-to-end testing of an AI Agent’s ability to complete tasks in the real world. Traditional