Inspiration

CopyCat was inspired by a simple idea: people already teach others how to perform digital tasks through screen recordings, demonstrations, and spoken instructions, but computers usually forget these workflows. We wanted to build an assistant that could learn from these demonstrations, remember them as reusable skills, and perform them again when asked.

What it does

CopyCat is a voice-first digital assistant that learns reusable skills from screen recordings, voice, or text instructions. Users can review and approve a generated skill before it is used. Later, they can simply ask CopyCat by voice or text to perform that task.

Our current prototype demonstrates this through real browser-based workflows.

How we built it

The frontend is built with Next.js and the backend with FastAPI. We use AI models to understand demonstrations and natural-language instructions.

For browser execution, we use Playwright MCP with an AI agent to interact with a real browser session. Skills and execution history are stored using Supabase.

For privacy, video frames are processed locally using RapidOCR and OpenCV to detect and redact potentially sensitive information before cloud-based AI analysis.

Challenges we ran into

Our main challenges were making browser automation reliable, handling AI model availability, managing persistent browser sessions, and protecting sensitive information in screen recordings.

We also had to carefully manage the project scope and focus on building one complete end-to-end workflow instead of attempting to implement too many incomplete features.

Accomplishments that we're proud of

We built a complete workflow where users can:

Teach → Review → Save → Ask → Execute

We are proud of combining skill learning, voice interaction, privacy redaction, skill memory, and real browser execution into one working prototype.

What we learned

Through CopyCat, we learned about AI agents, multimodal AI, browser automation, Model Context Protocol (MCP), OCR-based privacy protection, voice interfaces, and building systems that separate learning, memory, planning, and execution.

We also learned that user trust, privacy, and control are just as important as making an AI agent autonomous.

What's next for CopyCat

Next, we want to improve CopyCat's memory and personalization so it can better understand a user's workflows over time.

We also plan to explore compound workflows, additional execution environments, stronger permission systems, improved privacy detection, and support for multiple AI models.

Our long-term vision is to make CopyCat a personal assistant that learns how you work.

CopyCat — Teach once. Run by voice.

Built With

Share this project:

Updates

Submission history