-
-
Aisle Pilot interface
Inspiration
My background is in multi-agent robotic systems, where typical use cases include warehouse and fleet management. In these systems, individual robots are already equipped with sensors such as cameras and LiDAR, as well as software packages for planning, perception, and control. The complexity shifts to managing these robots effectively so that they can complete tasks continuously and in a coordinated manner.
However, in multi-agent systems, robots can sometimes become one another’s adversaries. They may compete for the same task, receive redundant task assignments, block one another, or encounter an unexpected problem that was not modeled in the original algorithm. Because of this, robot fleets often require constant supervision.
This was the motivation behind AislePilot.
What it does
AislePilot is a warehouse fleet-management interface that shows the position, battery level, operating status, and assigned task of every robot. It also displays the warehouse layout, task locations, and active robot movements.
I connected the interface to ChatGPT’s browser agent through WebMPC. This allows ChatGPT to inspect the current warehouse state, monitor the robots, identify failures, and determine which tasks have been affected.
For example, if a robot fails while transporting an item, ChatGPT can:
- Detect that the robot is no longer operational.
- Identify its incomplete task.
- inspect the location, battery level, status, and workload of the remaining robots.
- Select an appropriate replacement robot.
- Propose the reassignment to the human operator.
- Execute the change only after receiving human approval.
This keeps a human in control while reducing the need to watch every robot continuously.
How I built it
I built AislePilot as an interactive web-based warehouse simulation. The interface maintains a live state for each robot, including:
- Robot ID
- Current position
- Battery level
- Operating status
- Assigned task
- Task destination
- Task progress
I exposed this information to the ChatGPT browser agent through WebMPC tools. Instead of relying only on screenshots or trying to interpret the visual interface, ChatGPT can request structured information about the warehouse and its robots.
I also exposed actions that allow the agent to inspect a robot, identify unassigned or interrupted tasks, and propose a task reassignment. Reassignment is treated as a critical action, so AislePilot presents the proposed change to the user for approval before updating the fleet state.
To test the workflow, I introduced robot failures during warehouse operation. ChatGPT had to recognize the failure, inspect the available robots, choose a suitable replacement, explain its decision, and wait for human confirmation.
Challenges I ran into
One major challenge was keeping the visual interface and the state available to ChatGPT consistent. If a robot’s position, battery, or task changes, both the human operator and the browser agent need to receive the same updated information.
Another challenge was giving ChatGPT enough context to make a useful reassignment. Choosing another robot is not simply a matter of finding an idle one. The agent must consider whether the robot is available, how far it is from the task, whether it has sufficient battery, and whether assigning another task would interfere with its current workload.
I also had to decide which actions ChatGPT could perform independently and which actions required human approval. Monitoring and diagnosing failures can happen automatically, but changing a robot’s assignment can affect the entire warehouse. I therefore added a human-in-the-loop approval step before a reassignment is executed.
Accomplishments that I’m proud of
I am proud that AislePilot demonstrates an agent doing more than answering questions. ChatGPT observes a changing warehouse through the browser, recognizes when a robot fails, reasons about the effect on the fleet, and proposes a concrete recovery action.
I am also proud of the human-in-the-loop workflow. The system allows ChatGPT to handle continuous monitoring and analysis while leaving consequential decisions, such as reassigning a task, under human control.
What I learned
I learned that an agent’s reasoning is only as useful as the state and actions made available to it. Giving ChatGPT structured access to robot data made its decisions more reliable than asking it to infer everything from the visual interface alone.
I also learned that agent-facing tools need to describe their actions and consequences clearly. The agent needs to understand not only that it can reassign a task, but also which robot will receive it, what will happen to the robot’s existing workload, and whether human approval is required.
Most importantly, I learned that browser agents can provide a practical supervision layer for multi-robot systems. They do not need to replace the robots’ existing planning, perception, or control software. Instead, they can monitor the overall fleet, respond to unexpected events, and involve a human when an important decision must be made.
What’s next for AislePilot
My next step is to connect AislePilot to a high-fidelity robot simulator and eventually to physical robots. I also want to support additional events such as low battery, blocked aisles, delayed tasks, conflicting assignments, and communication failures.
I plan to improve the reassignment logic by considering distance, battery level, workload, task priority, and estimated completion time. I also want to measure failure-detection time, reassignment time, task completion rate, robot utilization, and the number of interventions required from the human operator.
Ultimately, I want AislePilot to become an interface through which humans and ChatGPT can supervise large robot fleets together.
Log in or sign up for Devpost to join the conversation.