ProtoMind

Turn a single prompt into a complete, buildable hardware system.

Comment

About ProtoMind

Inspiration

Hardware prototyping today is fragmented and slow. Even a simple idea requires jumping between CAD tools, writing firmware, sourcing parts, and manually testing if it actually works.

We kept thinking: why does building hardware feel like coordinating multiple jobs instead of expressing one idea?

Most AI tools today generate text or code, but they stop before the real world. We wanted to push beyond that and build something that could bridge the gap between imagination and physical systems.

What It Does

ProtoMind is an autonomous prototyping system that takes a single natural language prompt and generates everything needed to build a hardware system:

  • 3D-printable CAD (OpenSCAD -> STL)
  • Arduino firmware
  • A structured parts list with pricing
  • A physics simulation to validate the design

Instead of just generating outputs, ProtoMind verifies them, and if something fails, it fixes itself.

How We Built It

At the core of ProtoMind is a multi-agent pipeline orchestrated using LangGraph.

The system works as follows:

  1. Prompt Decomposition
    The user’s input is structured into a build specification

  2. Parallel Generation

    • CAD Agent -> generates parametric geometry
    • Firmware Agent -> writes Arduino code
    • Sourcing Agent -> finds real components + prices

  3. Simulation Verification
    The full system is tested using PyBullet physics simulation

  4. Self-Correction Loop
    If the system fails (e.g. unstable), constraints are extracted and fed back into the pipeline: fail -> extract constraint -> regenerate -> re-test

All agents communicate through a shared state object, allowing the system to remain modular while still tightly integrated.

Tech Stack

  • LangGraph - orchestration and stateful agent pipeline
  • Gemini API - reasoning and generation across agents
  • OpenSCAD - programmatic CAD generation
  • PyBullet - physics-based validation
  • ReactPy / Python frontend - interactive UI

Setup Instructions

ProtoMind runs locally with a simple Python app. We’ve provided a Gemini API key, so you can run the full pipeline.

Steps

git clone https://github.com/Nikhil-Mohan-01/ProtoMind
cd protomind
pip install -r requirements.txt
python frontend.py

Then open:

http://localhost:8000/

Running the App

  • After running the command, the app will start locally
  • Open the URL shown in your terminal (if prompted)

In the sidebar:

  • Paste the provided Gemini API key:
    AIzaSyB-yAHC658Muk2v9hgjhmDX3f3p2hsGt8k
  • Make sure "Use demo data" is OFF

Then:

  • Enter a prompt (or use an example)
  • Click Run Pipeline

What to Expect

  • The pipeline runs step-by-step in the UI
  • Total runtime: ~1–2 minutes

You’ll see:

  • CAD generation
  • Firmware generation
  • Parts sourcing
  • Physics simulation + possible auto-retry

Challenges We Faced

1. Coordinating Multiple Agents
Ensuring that CAD, firmware, and sourcing stayed consistent required a shared state design and careful structuring of outputs.

2. Reliable Structured Outputs
LLMs don’t always return clean JSON or code. We had to build defensive parsing and fallback handling.

3. Simulation Feedback Loop
The hardest problem was enabling the system to detect failure, extract meaningful constraints, and regenerate improved designs automatically.

What We Learned

  • Multi-agent systems are far more powerful when combined with verification loops, not just generation
  • Bridging software and hardware requires thinking in terms of constraints and physical validation, not just outputs
  • Good system design (state, modularity, orchestration) matters just as much as model choice

Why It Matters

ProtoMind shows that AI can go beyond generating text and start interacting with the physical world in meaningful ways.

By compressing hours of work into minutes, it makes hardware prototyping:

  • Faster
  • More accessible
  • Less dependent on specialized expertise

What’s Next

With more time, we would:

  • Improve simulation fidelity and real-world accuracy
  • Add stricter validation for part compatibility and tolerances
  • Support more complex systems like robotics and multi-component assemblies
  • Introduce human-in-the-loop controls for safety-critical builds

Final Thought

ProtoMind isn’t just generating designs, it’s reasoning about them, testing them, and improving them.

From idea -> to object -> automatically.

Built With

Share this project:

Updates