Perception

Perception
system_architecture
Email
Composio

Inspiration

Traditional CAD is powerful, but getting from an idea to a 3D model is hard, and that's only the first step in a long engineering lifecycle. We asked whether an AI assistant could automate that lifecycle end-to-end, from a spoken task to a finished, 3D-printed part.

What it does

To combat this, we built Perception: your room turned into a 3D design studio on the Meta Quest 3, automating a lifecycle that used to take an entire engineering team. At the heart of it is Percy, our AI voice assistant. Using Composio's MCP, Percy connects to 70+ platforms your team already works in (Notion, Gmail, Slack, Google Drive, Jira), so instead of bouncing between apps to figure out what you're building, you just ask. Percy pulls the context and starts from there.

From there the whole cycle happens in the headset. Pull up your last design or spin up a new one from a single command, then tailor it with your hands until it's exactly what you want. When it's right, export and push it straight to your 3D printer, and have Percy update your team while you wait.

How we built it

Codex for planning, development and testing.
Meta Quest 3 for the mixed reality environment and hand tracking
OpenAI for Percy's reasoning, turning speech and reference images into modeling instructions
ElevenLabs API for voice capture and Percy's custom voice model
Composio MCP to connect Percy to 70+ apps (Notion, Gmail, Google Drive, Photos)
CadQuery for CAD generation and TRELLIS for sculpting and meshing, both hosted on Hugging Face
Ultimaker Cura to slice and print

Challenges we ran into

VR is still new, so the resources surrounding it are scarce and most of what we hit had no answer to look up. The biggest issue we ran into was latency: CAD geometry is heavy and the generated models are heavier, slow enough to break the feel of a conversation. We fixed it by off loading generation and hosting to Hugging Face, so the headset never has to load heavy models and only renders the result, improving latency to < 30s.

Accomplishments that we're proud of

We got the full loop working: you can speak an idea in the headset and walk away with a printed part, without ever opening a CAD program. Getting voice, AI reasoning, CAD generation, mixed reality, and a slicer to behave as one responsive system in a weekend is the part we're proudest of.

What we learned

NEVER GIVE UP !!!
Understanding a request is the easy half. Knowing which tool to fire, in what order, and when to stop and ask is the actual work.
A 2D interface doesn't port. Selecting, scaling, and comparing all needed rebuilding from scratch once the model was sitting on the table in front of us.
Context is what makes Percy useful. Which model you mean, what you already changed, what you're going for next matters more than parsing the sentence.

What's next for Perception

In the future, we would like to implement a multiplayer sandbox, where multiple engineers jump into the same room and work on a model together, editing and seeing changes land in real time.