A.U.R.O.R.A. – Autonomous Utility Rover for Object Retrieval and Allocation

1. Problem Statement

Modern Third-Party Logistics (3PL) warehouses face significant challenges in product retrieval, order fulfilment and internal material transportation. Traditional warehouse operations rely heavily on manual product identification and picking, followed by separate transportation processes. Conventional Autonomous Mobile Robots (AMRs) primarily transport products that have already been loaded onto them, leaving the actual retrieval operation dependent on human intervention.

The lack of integration between SKU-level product identification, rack-coordinate mapping, robotic manipulation and autonomous transportation creates an operational gap.

AURORA addresses this problem by proposing an integrated Physical AI system capable of understanding warehouse orders, identifying products through computer vision, navigating to predefined storage coordinates, physically retrieving products through a telescopic vacuum-picking mechanism and delivering them to designated destinations.

2. Our Approach – From Perception to Physical Action

AURORA is designed as an intelligent autonomous mobile manipulation platform that combines artificial intelligence, edge inference, embedded computing and robotic mechanisms.

The primary objective is to establish a complete perception–reasoning–planning–execution pipeline.

Instead of treating product recognition, warehouse navigation and object retrieval as independent operations, AURORA coordinates them through a unified mission architecture.

The proposed architecture consists of:

NVIDIA Nemotron for high-level reasoning and warehouse mission planning.
Hailo AI acceleration for edge-based computer vision inference.
Raspberry Pi 5 for central processing and system orchestration.
STM32 for deterministic embedded motor control.
A telescopic robotic arm with a single vacuum suction picking mechanism.
Coordinate-based navigation and onboard product storage.

3. AI Architecture and Model Selection

A. NVIDIA Nemotron 3.5 Lightning – Mission Intelligence

We selected NVIDIA Nemotron 3.5 Lightning as the proposed reasoning engine, accessed through Nebius Token Factory.

Model identifier: nvidia/Nemotron-3_5-Lightning

The model has 30 billion total parameters with 3 billion active parameters and is designed for efficient reasoning and agentic workflows. https://build.nvidia.com/nvidia/nemotron-3.5-lightning-30b-a3b/modelcard

Its responsibility in AURORA is to transform structured warehouse information into machine-readable retrieval plans.

The model receives:

Customer order information.
Product SKU and OCR results.
Product-category identification.
Rack and shelf identifiers.
Pickup coordinates.
Rover position.
Destination coordinates.
Mechanical operating constraints.

It generates a structured JSON mission plan that is subsequently validated before execution.

Nemotron is not used as a direct motor controller. Its role is high-level reasoning, task decomposition and mission coordination.

B. YOLOv8n + Hailo – Edge Perception

We selected YOLOv8n as a lightweight baseline for the proposed object-detection pipeline.

The model is intended to identify product categories such as speakers, glass products and air coolers.

Hailo is used as the target edge-inference accelerator for the compiled vision model.

The perception pipeline includes:

Camera image acquisition.
Object detection and bounding-box extraction.
Product-category classification.
OCR-based label extraction.
SKU verification against the warehouse inventory database.

This division allows visual recognition to operate locally rather than requiring every camera frame to be uploaded to a cloud language model.

C. Raspberry Pi 5 – Central Orchestration

The Raspberry Pi 5 acts as the main onboard computing platform.

It coordinates the perception output, Nemotron mission-planning interface, warehouse coordinate database, navigation controller and embedded communication layer.

D. STM32 – Real-Time Control

The STM32 handles deterministic low-level control, including:

Wheel motor control.
PWM generation.
Encoder feedback.
Rover movement.
Telescopic arm actuation.
Vacuum pump and valve control.
Position and limit-switch monitoring.

This distributed architecture separates probabilistic AI reasoning from time-critical physical actuation.

4. AI Inference Pipeline and Sample Outputs

We designed the perception and reasoning pipeline as two distinct AI stages.

Stage 1 – Edge Perception

Consider an incoming request to retrieve a SONIX speaker with product identifier SPK-200.

The camera captures the shelf image. The edge perception pipeline identifies the product and extracts its label information.

Illustrative perception output:

{
  "detection_id": "DET-001",
  "class": "speaker_box",
  "confidence": 0.962,
  "bounding_box": [412, 126, 628, 341],
  "ocr_text": "SONIX SPEAKER SPK-200",
  "sku_match": true,
  "status": "PRODUCT_IDENTIFIED"
}

The detection confidence is supplied by the example perception model. SKU verification is performed against the inventory record.

Stage 2 – Nemotron Mission Reasoning

The verified product information and warehouse coordinates are passed to Nemotron through the mission-planning interface.

Illustrative mission-planning output:

{
  "mission_id": "AUR-SIM-001",
  "task": "PRODUCT_RETRIEVAL",

  "product": {
    "sku": "SPK-200",
    "category": "speaker_box",
    "payload_g": 200,
    "identity_verified": true
  },

  "pickup": {
    "rack": "R02",
    "shelf": "S03",
    "coordinates": [2.40, 1.10]
  },

  "manipulation": {
    "arm": "TELESCOPIC",
    "suction_count": 1,
    "suction_location": "LEFT_ARM_END",
    "support_pad": "OUTSIDE_RACK",
    "actions": [
      "EXTEND_ARM",
      "EXTEND_SUCTION_HEAD",
      "ACTIVATE_VACUUM",
      "EXTRACT_PRODUCT",
      "SUPPORT_PRODUCT",
      "ROTATE_TO_LEFT",
      "TRANSFER_TO_FRONT_STORAGE",
      "RELEASE_PRODUCT",
      "RETRACT_ARM"
    ]
  },

  "delivery": {
    "destination": "D01",
    "coordinates": [8.00, 3.50]
  },

  "execution": {
    "validation_required": true,
    "status": "PENDING_VALIDATION"
  }
}

The output represents the expected structured response for the example mission, not a captured Nemotron API response.

The validation layer checks SKU consistency, coordinate validity, arm travel limits and operating constraints before passing approved commands to the STM32.

5. Robotic Manipulation and Retrieval Mechanism

A major engineering contribution of AURORA is its telescopic vacuum-based product retrieval mechanism.

The rover approaches the storage rack and positions itself at the predefined pickup coordinates. Its telescopic arm extends towards the selected product.

The suction mechanism, positioned at the LEFT end of the arm, independently extends into the rack while the external support pad remains outside.

The vacuum mechanism attaches to the product and extracts it from the rack. The external support pad assists in supporting the product during extraction.

After retrieval, the suction assembly changes orientation to face left and positions the product towards the front section of the rover.

The product is released into the onboard storage section, and the arm retracts completely.

AURORA uses exactly ONE suction-picking mechanism. There is no additional suction mechanism on the rover.

6. Quantitative AI Evaluation

To establish a reproducible evaluation framework, we constructed an illustrative synthetic product-recognition dataset containing 300 labelled samples across three warehouse product categories.

Each class contains 100 samples.

The following confusion matrix represents the generated example evaluation.

Product Recognition Confusion Matrix

Actual / Predicted	Speaker	Glass	Air cooler
Speaker	94	3	3
Glass	2	95	3
Air cooler	2	4	94

Out of 300 samples, 283 were correctly classified.

The evaluation metrics are calculated using:

[ Precision=\frac{TP}{TP+FP} ]

[ Recall=\frac{TP}{TP+FN} ]

[ F1=2\frac{Precision\times Recall}{Precision+Recall} ]

[ Accuracy=\frac{\text{Correct Predictions}}{\text{Total Samples}} ]

Illustrative classification results

Class	Precision	Recall	F1-score
Speaker	95.92%	94.00%	94.95%
Glass	93.14%	95.00%	94.06%
Air cooler	94.00%	94.00%	94.00%
Macro average	94.35%	94.33%	94.34%

Overall example metrics

Total evaluation samples: 300
Correct classifications: 283
Incorrect classifications: 17
Classification accuracy: 94.33%
Macro precision: 94.35%
Macro recall: 94.33%
Macro F1-score: 94.34%

These figures represent a synthetic classification benchmark generated for demonstrating the proposed evaluation methodology. They are not measured YOLOv8n or Hailo inference results.

7. Mission-Planning Evaluation

We also defined an illustrative acceptance-test framework for evaluating the structured mission-planning output.

A synthetic suite of 50 mission scenarios can be evaluated using predefined expected outputs and validation rules.

Evaluation criterion	Example passes	Pass rate
Valid JSON structure	49/50	98%
Correct SKU-to-rack mapping	48/50	96%
Coordinate preservation	50/50	100%
Correct picking-stage sequence	47/50	94%
Invalid mission rejection	19/20	95%

These are illustrative test outcomes, not live Nemotron evaluation results.

They establish measurable acceptance criteria for the mission planner, particularly around structured output, inventory consistency, coordinate preservation and safe execution.

8. Coordinate-Based Autonomous Navigation

AURORA operates using a predefined warehouse coordinate map.

Each product is associated with an inventory identifier and a corresponding rack location. Once a mission is approved, the rover receives the pickup coordinates.

The distance between the current rover position and the target location is represented as:

[ d=\sqrt{(x_t-x_r)^2+(y_t-y_r)^2} ]

Where:

(x_r,y_r): Current rover coordinates.
(x_t,y_t): Target warehouse coordinates.
(d): Euclidean distance to the target.

The navigation controller uses position feedback to approach the target and verifies the arrival condition before enabling the picking sequence.

The same approach is used to reach the predefined delivery location after successful product placement.

9. Development and Simulation

We developed a Three.js-based 3D warehouse simulation to represent AURORA's mechanical architecture and operational workflow.

The simulation provides a visualization environment for the warehouse racks, product locations, rover movement, telescopic manipulation, vacuum picking and product transfer.

The workflow is designed to demonstrate:

Product identification.
Coordinate-based navigation.
Accurate rover positioning.
Telescopic extension and retraction.
Independent suction-head movement.
Product extraction.
Support-pad interaction.
Product transfer into onboard storage.
Autonomous delivery.

The simulation supports mechanical-sequence verification and provides a foundation for future digital-twin development.

10. Engineering Challenges and Solutions

Perception-to-Action Integration: Product recognition alone is insufficient to perform warehouse retrieval. We designed a structured interface connecting perception outputs to inventory records and mission planning.

Constrained Shelf Access: Requiring the full rover to enter storage racks would introduce mechanical and navigation constraints. The telescopic arm enables access while keeping the rover outside the rack.

AI Reliability: Generative AI outputs cannot be trusted as unrestricted hardware commands. We introduced a deterministic validation stage between mission planning and embedded execution.

Product Stability: Product extraction requires controlled movement and support. The external platform is included to assist with product stabilization during transfer.

Distributed Computation: Edge perception, cloud reasoning and real-time motor control have different latency and reliability requirements. We separated these responsibilities across Hailo, Raspberry Pi 5, Nebius-hosted Nemotron and STM32.

11. Impact and Future Scope

AURORA aims to reduce manual warehouse picking effort by integrating retrieval and internal transportation into a single autonomous robotic workflow.

Future engineering development will focus on:

Physical prototype integration.
Closed-loop telescopic positioning.
Vacuum-pressure monitoring.
Product handling validation.
Accurate localization and collision avoidance.
Real-time mission execution monitoring.
Multi-order scheduling.
Warehouse fleet coordination.
Simulation-to-reality validation.

Our long-term vision is to establish AURORA as a scalable Physical AI platform capable of understanding warehouse tasks, reasoning over inventory information and executing validated robotic actions.

AURORA – Perceive. Reason. Retrieve. Navigate. Deliver.