Building TaskPilot: An AI Agent That Sees Your Screen and Takes Control

I created this blog for detailing about my project in the Gemini Live Agent Challenge hackathon. #GeminiLiveAgentChallenge The Problem: Automation That Breaks the Moment the UI Changes Every developer has been there. You write a Selenium script, it works perfectly, and then the website updates its CSS class names and the whole thing falls apart. You set up an RPA workflow, it runs fine for a week, and then someone moves a button and it starts clicking the wrong thing. Traditional automation is brittle because it's blind. It relies on DOM selectors, API hooks, and hardcoded coordinates. It doesn't actually see the screen. It just pokes at it. But humans don't automate that way. When you ask a colleague to "find the cheapest flight to New York and book it," they open a browser, look at the screen, read what's there, and make decisions based on what they see. They don't need an API. They don't need a DOM inspector. They just need eyes. That's the gap TaskPilot fills. It's an AI agent that