FactoryMind

Inspiration

FactoryMind was inspired by the idea that autonomous agents should not just talk. They should observe a live system, make decisions, take bounded action, and adapt when the environment changes. Warehouses and factories are a natural place to show that: work is visual, time-sensitive, constraint-heavy, and full of routing decisions.

What it does

FactoryMind is an autonomous factory digital twin. Factories continuously produce colored packages, a Nemotron-powered leader observes the floor, assigns worker robots to matching pickup/dropoff tasks, and workers move packages through the warehouse while obeying safety and policy constraints.

The demo supports live controls like start/stop, speed changes, order spikes, worker failure, builder mode, and PNG layout upload. When the environment changes, the system reroutes workers and keeps dispatching work.

How we built it

We built a Python simulation backend with a Three.js browser frontend. The backend owns factory state, package spawning, worker movement, pathfinding, policy validation, and metrics. The frontend visualizes the digital twin in 3D and provides controls for operations, layout editing, and scenario testing.

NVIDIA integration is handled through an OpenAI-compatible inference layer. The leader is configured for Nemotron Super 49B, while workers are configured for Nemotron Nano 9B. The system can run against local NIM endpoints on a GX10 or NVIDIA cloud endpoints. We also added a NemoClaw/OpenClaw-style policy YAML and policy logs to show allow/deny decisions for agent actions.

Challenges we ran into

The hardest part was keeping the demo reliable while still being meaningfully agentic. Running large models locally introduces startup time, endpoint timeouts, and port/model configuration issues. We also had to balance LLM reasoning with deterministic safety: robots should not hallucinate movement, so pathfinding and policy checks stay code-controlled.

Another challenge was making autonomy visible. A working backend is not enough for judging; the UI has to show that the leader is observing, assigning, rerouting, and enforcing constraints.

Accomplishments that we're proud of

We built a live autonomous system instead of a static chatbot. The factory keeps running, workers take action, the leader assigns tasks, policy logs record decisions, and the 3D interface makes the behavior understandable.

We are especially proud of the digital twin workflow: uploading a PNG layout, converting it into a simulated warehouse, adjusting worker count/order volume, and comparing scenarios. That makes the project feel like a real operations tool, not only a hackathon animation.

What we learned

We learned that agentic systems need more than a model call. They need state, tools, constraints, observability, and graceful fallback. Nemotron is strongest when used for high-level reasoning and planning, while deterministic tools handle safety-critical execution like routing and collision avoidance.

We also learned that deployment details matter: NIM endpoints, model ports, timeout settings, and policy runtime availability can make or break a live demo.

What's next for FactoryMind

Next, we would make worker agents use Nemotron Nano for lightweight assignment rationales, add persistent memory so the leader learns which strategies work best on each layout, and strengthen real NemoClaw integration with enforced policy logs.

Longer term, FactoryMind could become a real warehouse planning assistant: upload a floorplan, simulate throughput, test worker counts, identify bottlenecks, and recommend layout changes before making expensive physical changes.