Why AI Agents shouldn't rely on screenshots: Building a cross-platform alternative to Anthropic's Computer Use

Anthropic recently released their Computer Use feature for macOS. It is a big step forward for AI agents, allowing models to interact with local software. However, this release also highlights a major technical bottleneck in how we are building GUI agents today. The current approach relies heavily on taking continuous screenshots and using large vision models to figure out where to click. This method is slow, expensive, and currently leaves Windows users out of the loop. When an agent uses screenshots, it essentially treats the operating system like a flat picture. It takes an image, sends it to the cloud, waits for the vision model to calculate pixel coordinates, and then finally moves the mouse. If a UI element shifts by a few pixels or the network is delayed, the action easily fails. Clicking a single button can take several seconds and consume a lot of tokens. We need a more efficient way for agents to interact with software. Human developers use APIs to talk to applications, and A