PixelPaws - Ai Companion

PixelPaws — a pixel-art desktop buddy for Work & Productivity that turns plain-English requests into safe, reviewable dev workflows: browser automation, skills, and scheduled coding tasks.

Comment

Inspiration

PixelPaws was inspired by the idea of having a friendly desktop companion that can understand what users are doing and help them complete repetitive tasks. Existing automation tools often feel hidden, fragile, or difficult to recover when something goes wrong. We wanted to make automation visible, understandable, and safer.

What it does

PixelPaws is a Windows AI desktop companion with: Screen-aware chat using manual screenshots. Visible Playwright browser automation. Natural-language automation planning. Reusable saved automations with Play, Edit, and Delete. Automation chat for explaining failures. Live screen observation when a page is visually unclear. Reusable AI skills for automation. Scheduled developer tasks such as generating files, opening VS Code, and running safe commands. Demo mode when no OpenAI API key is configured. A pixel-art cat companion named Tuxi. The system uses semantic actions such as navigation, search, click, type, and scroll instead of relying on arbitrary XPath selectors.

How we built it

We built PixelPaws with Electron, React, Vite, Express, Playwright, SQLite/local JSON storage, and the OpenAI Node SDK. GPT-5.6 helps interpret user requests, create structured automation plans, choose semantic browser actions, summarize results, explain failures, and generate scheduled code tasks. Playwright runs the approved plan in a visible browser with one-tab protection and duplicate-action detection. Codex helped us design components, debug Electron IPC issues, repair automation loops, improve the UI, validate model responses, and prepare the project for submission.

Challenges we ran into

Some of our biggest challenges were: Preventing the browser from opening duplicate tabs. Handling changing page layouts without relying on fragile selectors. Fixing invalid AI action formats. Keeping screenshots from causing the chat modal to flicker. Making screen sharing useful without sending every frame to the AI. Handling CAPTCHA and human-verification pages safely. Supporting Demo mode when no API key is available. Running scheduled Windows tasks reliably. Keeping the chat, automation, scheduler, and settings features separate without breaking existing functionality.

Accomplishments that we're proud of

We are proud that PixelPaws: Runs as a real Windows desktop application. Lets users review automation plans before execution. Shows what the browser is doing instead of hiding it. Stops safely when it detects CAPTCHA or human verification. Provides explanations when automation fails. Supports reusable automation history and reusable AI skills. Includes a secure OpenAI API key settings flow. Works in Demo mode without requiring credentials. Combines screen chat, browser automation, and scheduling in one companion. Maintains a playful pixel-art identity through Tuxi.

What we learned

We learned that reliable AI automation requires more than sending a prompt to a model. It needs structured outputs, validation, state tracking, clear permissions, recovery logic, and good user feedback. We also learned that visual context is useful, but sending screenshots continuously is expensive and can create a confusing user experience. A better approach is to keep live observation local and send a frame only when the user or automation explicitly needs AI assistance.

What's next for PixelPaws - Ai Companion

Next, we plan to add: More reliable page understanding through accessibility-tree and visual signals. Better recovery when pages change. More browser profiles and supported browsers. Optional integrations for Gmail, calendars, and developer tools. A stronger local scheduler with task history and notifications. Voice commands and spoken responses. More reusable automation skills. A permission system for sensitive actions. Better result extraction and citations. Packaging PixelPaws as a simple downloadable Windows application.

Built With

Share this project:

Updates