RackHand

RackHand is a physical AI coworker for hardware engineers, powered by Strands Agents on Bedrock. It enables a standard parts rack to understand goals, verify inventory, and catch shortages early.

Comment

Most inventory systems know what someone last recorded. RackHand can go check.

Inspiration

Hunting for parts steals our engineering focus. We’re a small hardware team. Every day, we stop engineering to search, count, and update inventory by hand. Then one missed update becomes a surprise: the software says we have the part—but the bin is empty. And our work stops. So we asked:

What if the rack could manage itself?

That became RackHand—a physical AI coworker for hardware engineers. Our goal was not to build another inventory dashboard for engineers to manage. It was to make parts management stop being the engineer’s job.

What RackHand does

The engineer gives RackHand an assembly goal, not a sequence of robot commands:

“RackHand, prep the parts for the control module.”

RackHand understands the request, reads the assembly plan, and identifies the required parts:

  • Two mounting kits
  • One motor driver
  • Four aluminum spacers

The engineer approves the plan once, and RackHand starts the multi-bin workflow. Our demo then shows four different behaviors. Together, they demonstrate that RackHand can complete routine work, repair trusted discrepancies, preserve uncertainty, and protect upcoming work.

1. Normal fulfillment — do the work RackHand retrieves the bin, the engineer takes the required parts, and RackHand verifies the remaining weight, updates the database, and returns the bin. No manual counting or data entry required.

2. Inventory mismatch — repair trusted state If the software says four parts remain but RackHand verifies three, it automatically corrects the trusted state. When evidence is clear, reconciliation happens quietly in the background.

3. Uncertainty — know when not to guess If the camera spots a foreign object in the bin, RackHand knows the scale weight cannot be trusted. It refuses to guess, leaves the inventory unchanged, and asks the engineer for a visual check.

This is a central design principle:

Automate when the evidence is clear. Ask for human judgment when it is not. With one request, RackHand handles the normal case, a wrong software record, and uncertain physical evidence—without searching or manual inventory entry.

4. Proactive audit — protect upcoming work While idle, RackHand reads our upcoming assembly spreadsheet. It notices a job needing 11 sensors. Software says 12 are available, but the bin hasn't been checked in a week. RackHand autonomously audits it, finds only 10, and warns the team.

The progression is intentional:

Do the work → repair bad state → know when not to act → think ahead.

Why this needs an agent

A fixed automation can be programmed to retrieve a known bin. But the engineer does not ask:

“Move bin B4-01.”

The engineer asks:

“Prep the parts for the control module.”

RackHand must connect that professional goal to an assembly plan, determine which parts are required, coordinate work across multiple bins, and respond differently depending on what it finds. If inventory is correct, it continues. If inventory is confidently wrong, it reconciles the record and continues. If the evidence is uncertain, it preserves the record and asks for help. If upcoming work depends on stale, low-buffer inventory, it prioritizes a check. The sequence is not completely known in advance. It depends on the assembly context and on the results returned by RackHand’s tools. That judgment is the agentic part.

How we built it

RackHand is built around one architectural principle:

AI for judgment. Deterministic control for physics.

At the center is the RackHand Agent, built with the Strands Agents SDK and powered by Amazon Bedrock. The agent runs inside our Next.js application server and calls Amazon Bedrock directly. It starts from an engineer’s request or an upcoming assembly plan, invokes specialized capabilities and controlled tools, receives their structured results, and decides what should happen next.

How we use Strands Agents

RackHand uses two focused agents as specialist tools:

  • The Parts Planning Agent resolves an assembly goal into required parts and quantities.
  • The Inventory Auditor Agent interprets inventory checks and audit history.

These specialists are exposed to the main RackHand Agent through Strands’ Agent.asTool() pattern. RackHand invokes them when their expertise is useful; they are not a mandatory sequence for every request.

We also use:

  • Strands Graph to coordinate guarded retrieval, verification, return, and audit workflows
  • Structured outputs to validate planning and audit results
  • Human-in-the-loop controls to protect restricted actions and pause when attention is required
  • Lifecycle hooks to expose tool activity and results without displaying private model reasoning

Each capability produces an observable product behavior. Specialist agents resolve and interpret the work. Graphs maintain workflow state. Structured evidence determines whether inventory can change. Human review appears only when the system cannot continue safely.

Agentic and deterministic responsibilities

Strands Agents handles Deterministic services handle
Understanding the engineering goal Validating requests and permissions
Resolving required parts Sequencing retrieval and return
Choosing tools and specialists Sampling and validating scale readings
Interpreting structured outcomes Calculating quantities from weight
Prioritizing inventory checks Applying evidence-acceptance rules
Deciding when a human is needed Committing inventory changes

The language model does not generate motor coordinates, invent sensor readings, or perform inventory arithmetic. Controlled workflows validate physical evidence and enforce safe inventory changes. Their structured outcomes return to the RackHand Agent, which decides whether to advance the overall job, invoke another capability, or report an exception.

The physical system

We kept our existing standard parts rack and designed and built the gantry system around it from scratch. RackHand combines:

  • A three-axis gantry for retrieving and returning bins
  • A Raspberry Pi camera for visual verification and anomaly detection
  • A digital scale for quantitative evidence
  • A browser interface for requests, approvals, activity, and results
  • PostgreSQL and Supabase for inventory, workflow state, and evidence
  • Strands Agents with Amazon Bedrock for orchestration and decision-making

At the inspection station, the scale and camera provide complementary evidence. The scale provides quantitative evidence using the container tare and known unit weight. The camera provides visual confirmation and detects conditions—such as an unexpected object—that could invalidate the scale-based count. Each demo slot maps to a known SKU. RackHand does not pretend the camera performs perfect open-world part recognition. The known bin location provides the expected identity, while the camera and scale determine whether the observed contents can be trusted. A structured result returns to RackHand. The agent then decides whether to continue, reconcile the record, invoke another capability, or ask for human attention.

This creates a closed physical agent loop:

Understand the job → choose tools → act → observe reality → decide again.

For repeatable development and failure testing, the software also supports simulated workflows without requiring every test to move the physical gantry.

Challenges we faced

Deciding where AI should stop We had to resist making everything “AI.” Direct language-model control of motors or raw quantity calculations would make the system unreliable.

Turning physical observations into trustworthy evidence Trusting noisy physical signals: Scales fluctuate and visual evidence can be ambiguous. We built a verification layer that returns structured evidence rather than raw sensor streams. Clear evidence supports automatic correction; conflicting evidence triggers human review.

Accomplishments that we’re proud of

We are proud that one engineering request can become assembly understanding, multi-bin planning, controlled execution, verification, inventory reconciliation, continuation, and escalation when needed. RackHand does not trust the database blindly. It repairs stale state when the evidence is clear and refuses to guess when the evidence is unreliable. The proactive audit best represents our larger vision: connect upcoming work to physical inventory confidence, investigate the uncertainty that matters, and surface a shortage while the team still has time to respond.

What we learned

The biggest lesson is that useful physical AI is not about giving a language model more control. It is about giving the agent the right decisions and deterministic systems the right actions. Most importantly, autonomy should reduce attention, not demand it.

The best outcome is often the interruption that never happens.

Potential impact

In a controlled comparison of a parts-preparation task, RackHand required 60% less active human time than the manual workflow. Across 20 controlled inventory trials, all 20 final records perfectly matched manually verified reference counts. In our upcoming sensor-array scenario, RackHand’s proactive audit gave the team 25 hours of shortage warning before assembly was scheduled to begin.

Hackathon build disclosure

RackHand was developed during the Agents for Humans Hackathon submission period. We reused our existing standard parts rack as the foundation. During the hackathon, we designed and built the gantry system from scratch and developed the RackHand agent architecture, Strands orchestration, specialist-agent workflows, browser interface, inventory workflows, idle-triggered upcoming-plan audit, camera-and-scale integration, and submitted demo scenarios.

RackHand demonstrates how an existing passive parts rack can be retrofitted into an agent-driven physical system without replacing the rack itself.

What’s next

Our next step is to test RackHand across more parts, assemblies, and repeated engineering tasks. We want to measure long-term inventory accuracy, intervention frequency, and how often early warnings give teams enough time to resolve a shortage.

Over time, RackHand can build a physical memory across repeated assemblies: which inventory loses confidence fastest, which parts repeatedly become blockers, where actual consumption differs from the plan, and which checks are most valuable before work begins. From there, one rack can expand to multiple racks and workcells.

Our long-term thesis is simple:

Most inventory systems know what someone last recorded. RackHand can go check.

We started with engineers managing a rack. We turned the rack into a coworker.

You build hardware. RackHand handles the parts.

Built With

Share this project:

Updates

Submission history