HCP Schematic-to-Fetch Robot Arm

Turn a PDF circuit schematic into a parts list, then use an LLM-powered robotic arm with camera vision and AprilTag localization to find, pick, and stage components for assembly.

Comment

💡 Inspiration

Electronics prototyping often starts digitally with a schematic, but quickly becomes physical: finding the right resistors, capacitors, ICs, diodes, motors, and other components scattered across a workspace.

We wanted to bridge that gap by giving an LLM the ability to understand an electrical schematic and connect that understanding to a physical robot.

That led us to HCP Schematic-to-Fetch 🤖: a system that turns a circuit schematic into an actionable component request and allows a DJI RoboMaster EP to physically retrieve the corresponding component groups.

Instead of an AI simply telling you what components you need, we wanted to explore what happens when the AI can help bring those components to you.

⚡ What it does

A user uploads a PDF or image of a circuit schematic through our web interface.

The schematic is processed using Baseten-powered vision models, which extract the components into a structured Bill of Materials (BOM). A separate AI review can also analyze the visible circuit for potential issues such as shorts, excessive current, polarity/rating problems, and possible component damage.

The extracted components are then classified into one of seven physical component groups:

  • 🔵 Resistors
  • 🟣 Capacitors
  • 🟢 Integrated Circuits
  • 🟠 Motors
  • 🟡 Inductors
  • 🔴 Custom PCBs
  • ⚪ Diodes / LEDs

Each group corresponds to a physical container identified using a tag36h11 AprilTag.

Before anything moves, the user can inspect, edit, and approve the BOM through the interface. AI-generated commands only create proposals — they cannot directly authorize robot movement.

Once approved, our deterministic controller sends commands over HTTP to a Raspberry Pi, which acts as the execution node for a DJI RoboMaster EP.

The RoboMaster uses its onboard camera 📷 to detect the requested AprilTag. It first performs a coarse approach, then uses the tag's pixel center and apparent size to make bounded visual corrections until the target is aligned.

The RoboMaster's arm and gripper then:

🔍 Detect → 🚗 Approach → 👁️ Visually Align → 🦾 Grip → ↩️ Retrace → 📦 Deliver → 🏠 Return

The tagged component cup is delivered to a fixed collection point, giving the user the parts required to begin building their circuit.

🔗 End-to-end pipeline

📄 Schematic → 🧠 Baseten Vision → 📋 BOM → 🏷️ Component Classification → ✏️ Human Approval → 📡 HTTP Controller → 🍓 Raspberry Pi → 👁️ AprilTag Vision → 🤖 DJI RoboMaster EP → 📦 Component Delivery

🛠️ How we built it

We designed HCP Schematic-to-Fetch as a modular AI + robotics pipeline connecting document intelligence, LLM reasoning, computer vision, human approval, and physical execution.

🧠 1. Schematic Intelligence with Baseten

Our ingestion pipeline accepts PDF, JPEG, PNG, BMP, TIFF, and legacy KiCad/EAGLE .sch files.

For PDFs and images, schematic pages are sent to a Baseten vision model to extract the original component BOM. A separate vision-model request performs an advisory schematic review, looking for visible problems such as incorrect polarity, possible shorts, excessive current, or component-rating issues.

Legacy .sch files can instead be parsed locally while preserving component information such as type, value, and reference designator.

🏷️ 2. Structured BOM + Component Classification

Once the components are extracted, an LLM classifies every component into one of our seven registered physical storage groups.

Each group maps to a specific AprilTag ID, creating a connection between an abstract schematic component and a physical location the robot can identify.

Importantly, the model cannot modify component quantities or invent tag IDs. Quantities are aggregated locally, and invalid or missing classifications are rejected.

💻 3. Human-in-the-Loop Web Interface

Our browser UI gives the user a complete view of the proposed request before the robot moves.

Users can inspect:

requested components, grouped BOM data, AprilTag mappings, AI schematic suggestions, robot/node health, detections, execution progress, and errors.

The BOM can be manually edited before execution.

Most importantly, uploading a schematic or asking the LLM for an action never moves the robot. Every physical run passes through an explicit human confirmation gate.

🗣️ 4. Natural-Language + Voice Commands

We also built an optional command interface that allows requests such as:

“Fetch resistors and capacitors.”

Typed commands can be interpreted by a Baseten-backed command model into a validated action proposal.

We also implemented a push-to-talk browser prototype 🎙️ using browser speech recognition. The resulting transcript is interpreted exactly like typed input.

Voice and LLM commands still cannot bypass the approval system — they only prepare an action for the user to review.

🍓 5. Raspberry Pi Robot Node

After approval, the Mac-side controller sends a fixed sequence of HTTP requests to a server running on a Raspberry Pi.

The Pi communicates with the RoboMaster EP through USB/RNDIS using DJI's plaintext SDK protocol.

This separation keeps high-level AI reasoning away from low-level robot control: the LLM proposes what should happen, while deterministic software controls how the robot actually moves.

👁️ 6. AprilTag Computer Vision

The RoboMaster's camera streams H.264 video to the Raspberry Pi, where frames are decoded and searched for tag36h11 AprilTags.

Rather than relying on full metric 3D localization, our approach uses each detected tag's:

ID, pixel center, corners, and apparent size.

The robot performs a coarse movement toward the expected component area and then enters a bounded image-based feedback loop.

Horizontal tag error controls lateral corrections, while apparent tag size controls forward/backward corrections. Multiple aligned frames are required before the system is allowed to continue to the grasp.

🤖 7. RoboMaster Retrieval

Once visually aligned, the DJI RoboMaster EP arm and gripper execute a scripted grasp.

The robot then retraces its successful approach path, drives toward a predefined collection point, releases the component cup, clears its arm, and returns to its observation position.

Each component type therefore completes the sequence:

Detect → Approach → Align → Grip → Retrace → Deliver → Drop → Return → Observe

🧩 Challenges we ran into

One of our biggest challenges was connecting information from completely different domains.

A schematic represents a resistor as a symbolic electrical component. Our AI pipeline needs to understand that representation, classify it correctly, associate it with a physical component group, map that group to an AprilTag, and ultimately turn that information into a sequence of safe robot actions.

👁️ Closing the gap between vision and movement

Simply detecting an AprilTag wasn't enough.

The RoboMaster needed to use what it saw to actually correct its position relative to the target. We developed a bounded visual approach controller that continuously compares the tag's image position and apparent size against a taught target configuration.

This gave us a lightweight way of performing terminal alignment without requiring a complete metric camera calibration pipeline.

🧠 Separating AI reasoning from robot control

We also had to determine how much authority the LLM should actually have.

We intentionally designed the system so that AI models can extract information, classify components, review schematics, and interpret commands — but they cannot directly execute motion.

Instead, a human approves the proposal and a deterministic controller executes a predefined sequence.

🤖 Integrating real robotics

Moving from simulation to a physical DJI RoboMaster EP introduced an entirely different set of problems: camera streaming, arm units, chassis movement, networking, gripper timing, robot state, alignment, and communication between the Mac, Raspberry Pi, and RoboMaster.

Getting AI, computer vision, networking, and robotics to behave as one coherent system required debugging across nearly every layer of the stack.

🏆 Accomplishments that we're proud of

We're especially proud that HCP Schematic-to-Fetch goes beyond building another AI chatbot.

We built a system where an AI model can interpret a real engineering artifact, transform it into structured information, connect that information to a physical workspace, and initiate a human-approved robotic workflow.

We successfully combined:

🧠 Multimodal AI 📄 Document ingestion 📋 Structured BOM generation 🔍 AI schematic review 🏷️ LLM component classification 🗣️ Natural-language commands 🎙️ Voice interaction 👁️ AprilTag computer vision 🍓 Raspberry Pi edge computing 📡 HTTP-based distributed control 🤖 DJI RoboMaster EP robotics 🦾 Physical manipulation

We're also proud of the project's human-in-the-loop architecture. AI is responsible for tasks where reasoning is valuable, while deterministic software and explicit approval remain responsible for physical execution.

Most importantly, we created the foundation of a workflow where:

a circuit schematic can determine what a robot retrieves from the physical world. 🌎

📚 What we learned

This project taught us that connecting AI to hardware requires much more than simply giving an LLM access to a robot.

We learned how important it is to separate a system into perception, reasoning, validation, approval, and execution layers.

LLMs and multimodal models are extremely useful for interpreting schematics and human intent, but deterministic software remains much better suited for tasks such as robot state management, motion sequencing, safety limits, and feedback control.

We also gained hands-on experience with:

Baseten, multimodal LLMs, structured outputs, PDF/image ingestion, AprilTags, OpenCV-style computer vision concepts, Raspberry Pi, HTTP APIs, H.264 video streams, robot feedback loops, DJI RoboMaster EP control, browser speech recognition, and hardware-software integration.

Most importantly, we learned how to build an AI system that crosses the boundary between digital reasoning 🧠 and physical action 🤖 while keeping a human in control.

🚀 What's next for HCP Schematic-to-Fetch

Our long-term goal is to move from:

Schematic-to-Fetch → Schematic-to-Build 🔧

Today, our system focuses on identifying the required component groups and retrieving their tagged containers.

The next step is improving schematic understanding so the system can reason more deeply about component values, electrical relationships, wiring, and circuit topology.

On the robotics side, we want to move beyond tagged containers toward direct component recognition, allowing computer vision models to distinguish individual resistors, capacitors, ICs, and other parts.

We also want to add richer physical feedback so the robot can verify successful grasps, deliveries, collection-point occupancy, and its position, rather than relying primarily on commanded motion.

Eventually, we envision the system not just retrieving components but helping determine where they belong and how they should be assembled.

The bigger idea behind HCP is creating a bridge between AI reasoning and physical engineering environments — where an AI system can understand an engineering artifact, perceive the hardware around it, interact with robotic tools, and collaborate with humans to turn a digital design into something physical. 🤖🌎

Built With

Share this project:

Updates

Submission history