Inspiration
CopyCat was inspired by a simple idea: people already teach others how to perform digital tasks through screen recordings, demonstrations, and spoken instructions, but computers usually forget these workflows. We wanted to build an assistant that could learn from these demonstrations, remember them as reusable skills, and perform them again when asked.
What it does
CopyCat is a voice-first digital assistant that learns reusable skills from screen recordings, voice, or text instructions. Users can review and approve a generated skill before it is used. Later, they can simply ask CopyCat by voice or text to perform that task.
Our current prototype demonstrates this through real browser-based workflows.
How we built it
The frontend is built with Next.js and the backend with FastAPI. We use AI models to understand demonstrations and natural-language instructions.
For browser execution, we use Playwright MCP with an AI agent to interact with a real browser session. Skills and execution history are stored using Supabase.
For privacy, video frames are processed locally using RapidOCR and OpenCV to detect and redact potentially sensitive information before cloud-based AI analysis.
Challenges we ran into
Our main challenges were making browser automation reliable, handling AI model availability, managing persistent browser sessions, and protecting sensitive information in screen recordings.
We also had to carefully manage the project scope and focus on building one complete end-to-end workflow instead of attempting to implement too many incomplete features.
Accomplishments that we're proud of
We built a complete workflow where users can:
Teach → Review → Save → Ask → Execute
We are proud of combining skill learning, voice interaction, privacy redaction, skill memory, and real browser execution into one working prototype.
What we learned
Through CopyCat, we learned about AI agents, multimodal AI, browser automation, Model Context Protocol (MCP), OCR-based privacy protection, voice interfaces, and building systems that separate learning, memory, planning, and execution.
We also learned that user trust, privacy, and control are just as important as making an AI agent autonomous.
What's next for CopyCat
Next, we want to improve CopyCat's memory and personalization so it can better understand a user's workflows over time.
We also plan to explore compound workflows, additional execution environments, stronger permission systems, improved privacy detection, and support for multiple AI models.
Our long-term vision is to make CopyCat a personal assistant that learns how you work.