PA Bench: Evaluating Frontier Models on Multi-Tab Pa Tasks

We’re the team at Vibrant Labs (W24). We’ve been building envs for browser agents and quickly realized that existing benchmarks in this space didn’t capture the primary failure modes we were seeing in production (which scaled up as the number of applications and horizon length increase). We built PA Bench (Personal Assistant Benchmark) to evaluate frontier computer/web use models on their ability to handle multi-step workflows across simulated clones of Gmail and Calendar. *What’s next:* We’re currently scaling the dataset to 3+ tabs and are building more high-fidelity simulations for common enterprise workflows. We’d love to hear feedback on the benchmark and notes about what was/wasn’t surprising about the results. Blog post: https://vibrantlabs.com/blog/pa-bench Comments URL: https://news.ycombinator.com/item?id=47157160 Points: 4 # Comments: 0