Home Page
Operations Dashboard
Warehouse Operations Map
Product Recognition (Gemini Vision)
Incident Center
Arize Trace
Arize Trace
Arize project dashboard
Multi-Agent Architecture Diagram
Google Cloud Architecture Diagram

OpsPilot AI – AI-Powered Warehouse Operations Nerve Center

Inspiration

Warehouse operations still rely heavily on manual validation, disconnected systems, spreadsheet-driven workflows, and delayed incident investigations.

When a shipment arrives, workers often need to manually verify labels, package types, storage locations, and inventory records across multiple systems. If a product is stored in the wrong warehouse zone or inventory maps are not updated correctly, the issue may not be discovered until much later, resulting in operational delays, inventory confusion, compliance risks, and costly investigations.

We wanted to explore a different future: what if warehouse operations could be monitored, validated, investigated, and orchestrated autonomously by AI agents?

That idea became OpsPilot AI.

OpsPilot AI is an AI-powered warehouse operations nerve center that combines Gemini Vision, Google Cloud services, multi-agent orchestration, MCP-based tool interoperability, and Arize observability to continuously validate warehouse operations and respond to operational anomalies in real time.

What It Does

OpsPilot AI coordinates nine specialized AI agents that work together across the warehouse workflow.

The Warehouse Status Check Agent monitors inbound shipments using BigQuery data. The Orchestrator Agent coordinates downstream workflows and determines which agents should execute based on shipment timing and workflow state.

The Map Agent maintains warehouse spatial intelligence by refreshing rack and inventory state. The Product Recognition Agent uses Gemini Vision to analyze worker-uploaded shipment photos and extract labels, package information, dimensions, warehouse zone evidence, and confidence scores.

The Item Master RAG Agent retrieves operational reference knowledge, including expected package dimensions, shipment metadata, warehouse zones, and contact information. The Validation Agent compares detected operational evidence against the expected warehouse state.

When mismatches are detected, the Misload Detection Agent evaluates wrong-zone risk and abnormal inventory placement. If escalation is required, the Incident Agent creates an operational investigation ticket, and the Contact and Notification Agent routes the incident to the appropriate supervisor.

The result is an AI system that not only detects problems but also explains, investigates, and responds to them.

How We Built It

OpsPilot AI is built on Google Cloud and uses a modern AI operations architecture.

Gemini Vision powers image understanding and product validation. BigQuery stores operational warehouse data including shipment schedules, inventory maps, rack metadata, and reference product information. Google Cloud Storage stores reference images used during product validation.

Cloud Scheduler triggers daily orchestration workflows and shipment monitoring jobs. Cloud Run hosts the backend services and operational APIs. Vertex AI Agent Builder serves as the agent interface layer, while an MCP-compatible server exposes warehouse intelligence tools for agent interaction.

The frontend was built using React and Vite, while FastAPI powers the backend orchestration layer.

To provide production-grade observability, we integrated Arize AX and OpenTelemetry tracing. Every AI workflow, retrieval operation, validation decision, and incident escalation can be inspected through Arize, allowing operators and developers to understand exactly how the system arrived at a decision.

Challenges We Faced

One of the biggest challenges was designing a workflow that felt like a real operational system instead of a simple chatbot.

Rather than creating a conversational assistant, we focused on building an autonomous operational workflow where multiple agents collaborate to solve warehouse problems. Designing agent responsibilities, orchestration logic, and agent handoffs required multiple iterations before the workflow became reliable and explainable.

Another challenge was observability. Once multiple agents, retrieval workflows, and validation steps were introduced, debugging became increasingly difficult. Integrating OpenTelemetry and Arize helped us trace every step of the workflow and understand how information flowed between agents.

We also spent significant effort designing warehouse spatial intelligence workflows that could compare operational reality against expected warehouse state. This became one of the most valuable parts of the platform because it allowed the system to reason about physical warehouse operations rather than simply processing text.

What We Learned

This project taught us that the future of enterprise AI extends far beyond chat interfaces.

The most valuable AI systems are often those that operate behind the scenes, continuously monitoring business processes, validating operational data, coordinating specialized agents, and taking action when anomalies occur.

We also learned the importance of observability in agentic systems. As workflows become more autonomous, understanding why an AI system made a decision becomes just as important as the decision itself.

Building OpsPilot AI reinforced the idea that explainability, traceability, and operational accountability will be critical requirements for production AI systems.

Accomplishments We’re Proud Of

We are particularly proud of building a complete operational workflow that combines computer vision, retrieval-augmented intelligence, multi-agent orchestration, warehouse spatial reasoning, incident management, and AI observability into a single platform.

We successfully integrated Gemini Vision, Vertex AI Agent Builder, BigQuery, Cloud Storage, Cloud Scheduler, Cloud Run, MCP-based tool interoperability, and Arize AX into a cohesive warehouse operations solution.

Most importantly, we created a system that can explain every operational decision through end-to-end traces and agent workflows.

What's Next

Our vision for OpsPilot AI extends beyond warehouse operations.

Future versions will support live camera feeds, predictive inventory congestion detection, autonomous root-cause analysis, IoT sensor integration, digital warehouse twins, and multimodal operational copilots.

We believe AI systems will increasingly evolve from passive assistants into active operational infrastructure capable of monitoring, validating, investigating, and orchestrating real-world business processes autonomously.

OpsPilot AI is our step toward that future.