Inspiration Collaboration was the starting point for Robotic Mind.
I wanted to explore what collaboration could look like when humans and robots share the same workspace. A human should be able to express a goal, robots should contribute their capabilities, and what one member learns should become useful knowledge for the entire team.
That led me to build Robotic Mind: a shared cognitive workspace for humans and autonomous machines.
The central idea is simple: a robot’s discovery should become a team asset, rather than disappearing into an individual log. What it does Robotic Mind brings a human and four simulated drones into a persistent space called Warehouse Alpha. Robots join through an invite mechanism, publish their capabilities, and appear alongside human members.
A human can give the team a mission: “Find all red objects. Split up and avoid searching the same places twice.”
NVIDIA Nemotron, accessed through Nebius Token Factory, interprets the request and assigns search regions. Each drone then follows its own waypoint sweep and observes the warehouse through its own camera.
As the robots work, their knowledge develops across three layers:
- Active Mind: The robot’s current objective, observation summary, action, and next step.
- Deep Mind: Its persistent observations, discoveries, and task history.
- Group Mind: Knowledge contributed to the shared workspace, including discoveries, cleared regions, and mission results. Confirmed discoveries become available to other robots. Spatial deduplication combines repeated sightings of the same object while preserving who discovered or confirmed it.
The dashboard makes this collaboration visible through live camera feeds, telemetry, mission progress, and an interactive neural graph. Users can select a discovery and trace its contributors and supporting memories.
The verified end-to-end demo found three unique red objects, completed all four search sectors, and achieved 100% of the planned search coverage. How I built it I built the backend with Python, FastAPI, Pydantic, SQLAlchemy, and SQLite, using WebSockets to deliver live updates to a React and TypeScript dashboard.
The robots run in Webots, with independent Python controllers and RGB cameras. OpenCV detects red objects from camera images and estimates their world positions.
Nemotron handles three mission-level responsibilities:
- Understanding the human’s command.
- Assigning search regions to capable robots.
- Summarizing the final results from Group Mind. Movement remains deterministic, keeping language-model orchestration separate from low-level control.
To make the architecture reusable, I created a RobotClient SDK, capability manifests, and a RobotAdapter abstraction. The platform coordinates through capabilities rather than assuming every member is a drone. Challenges I faced One major challenge was turning independent observations into reliable shared knowledge. Different sightings can refer to the same object, so I needed a merging approach that preserved both uniqueness and provenance.
Camera geometry also required debugging. An early orientation error produced incorrect world-position estimates and inflated discovery counts. Inspecting actual camera frames helped me correct the transformation and obtain consistent results.
Another challenge was making collaboration understandable on screen. A conventional dashboard showed status, but did not clearly communicate how knowledge connected. The neural Group Mind visualization made those relationships explorable.
Finally, I needed to distinguish genuine model execution from development mocks. Strict mode requires authenticated Nebius inference and reports failures explicitly. The complete mission was verified with real Nemotron calls. What I learned I learned that collaboration requires more than assigning different tasks to different robots. It needs a shared representation of what was observed, who contributed it, how confident the system is, and what knowledge should persist.
I also learned how useful clear boundaries are: mission coordination, deterministic control, perception, memory, and visualization each have different responsibilities.
Most importantly, this project helped me explore collaboration as an architectural foundation—humans express intent, robots contribute evidence, and the workspace preserves what the team learns together.
Built With
- fastapi
- nebius-token-factory
- numpy
- nvidia-nemotron
- openai-python-sdk
- opencv
- playwright
- pydantic
- pytest
- python
- react
- ruff
- sqlalchemy
- sqlite
- tailwind-css
- typescript
- vite
- webots
- websockets
Log in or sign up for Devpost to join the conversation.