Foundry
Inspiration
Hardware development today is slow and fragmented. Builders constantly switch between tutorials, guess connections, and debug blindly without real-time feedback. We wanted to eliminate that friction by turning the workspace itself into an intelligent system—something closer to a real-world “Jarvis” that understands what you’re building and guides you as you build it.
What it does
Foundry transforms any physical table into a real-time intelligent engineering workspace.
- Detects components placed on the surface
- Builds a live model of the system being assembled
- Projects step-by-step guidance directly onto the workspace
- Highlights errors, missing connections, and next actions instantly
- Adapts in real time as components move or change
It also supports:
- Hardware learning and education
- Real-time debugging assistance
- Interactive system explanation for beginners and engineers
How we built it
Foundry is a real-time perception–reasoning–rendering system.
It runs a continuous loop:
- Perception Layer
- Computer vision detects components and workspace state
- Hand tracking and spatial mapping maintain live context
- Reasoning Layer (AI)
- GPT-4o (via AIP) interprets intent and generates the next build step
- Outputs structured JSON diagrams instead of images
- Rendering Layer
- Deterministic SVG-based diagram engine visualizes steps
- Real-time projection overlays guidance onto the physical workspace
- Feedback Loop
- System updates instantly as components move or are added
- Re-plans steps dynamically based on current state
Tech Stack
- AI Reasoning: GPT-4o (via Palantir AIP)
- Computer Vision: OpenCV + custom detection pipeline
- Tracking: Hand + object tracking system (real-time state mapping)
- Backend: Node.js / Python orchestration layer
- Diagram Engine: Custom SVG renderer (deterministic diagram system)
- Frontend: Real-time projection UI + interactive dashboard
- Data Model: JSON-based structured circuit/assembly graphs
System Overview
| Layer | Function | Key Technologies |
|---|---|---|
| Perception Layer | Detects components + tracks workspace state in real time | OpenCV, hand/object tracking |
| Reasoning Layer | Interprets intent and generates next-step structured diagrams | GPT-4o via Palantir AIP |
| State Model | Maintains live structured representation of the build | JSON graph-based schema |
| Rendering Layer | Converts structured steps into deterministic visuals | Custom SVG renderer |
| Projection Layer | Overlays guidance directly onto physical workspace | Real-time display system |
Challenges we ran into
- Achieving reliable real-time component detection at small scale
- Reducing system complexity while maintaining accuracy
- Avoiding fragile wire-level detection by abstracting into component graphs
- Camera connectivity and latency optimization across multiple tools and setups
We ultimately shifted from raw visual interpretation to a structured component-state model, which significantly improved stability.
Accomplishments that we're proud of
- Built a fully interactive physical-to-digital build system
- Achieved real-time recognition → reasoning → projection loop
- Created a system that feels intuitive without requiring instructions
- Demonstrated live adaptive guidance during physical assembly
What we learned
- Simplicity and structure outperform overly complex vision pipelines
- Representing systems as graphs is more reliable than pixel-level reasoning
- Real-time constraints force better system design decisions
- AI is most powerful when it operates on structured data, not raw outputs
What's next for Foundry
We plan to expand Foundry into a full engineering and education platform:
- More complex hardware and circuit support
- Classroom and lab integration
- Expanded component libraries
- Deeper simulation + prediction of system behavior
- Collaboration mode for multi-user builds
Log in or sign up for Devpost to join the conversation.