Knight-Wagon

Robot
NVIDIA and robot
Touchscreen Display Monitor
Attaching Cameras

Inspiration

We found that UCF's campus is huge, and getting food isn't always easy. For example, a student with a disability, someone stuck in a lab or a long study session, or anyone without a car on a hot Florida day. Food delivery apps either don't reach inside campus or cost a lot for a short trip. We wanted on-campus delivery that's accessible and cheap: order from your phone, and a robot brings it to wherever you are.

What it does

Knight-Wagon is a self-driving delivery robot with a full ordering app around it. With this, a person can order from a phone, browse 18 UCF campus shops and their menus in the iOS app, build a cart, and check out with the current location becoming the drop-off point. The whole point is that the robot comes to the customer. The order goes to the robot, which drives itself along sidewalks using cameras, depth, lidar, and GPS. It stops for obstacles and people, and treats crosswalks carefully. Live tracking is also available, following the robot on a map in real time, with order status from "placed" through "on the way", "arrived," and "delivered". In contact between the person and robot, a secure hand-off takes place. Each order gets a PIN. Enter it on the robot's touchscreen to release your food. Five wrong tries lock it for 10 minutes. There are apps for every role: customers order and track, shop staff accept orders and send the robot, and operators see the robot's position and battery, and can emergency-stop or resume it from their phone. Safety is always a priority. Every motion command passes a motion gate that blocks driving without fresh localization and a clear path ahead. Speed is hard-capped. Physical buttons on the robot latch a stop that no remote command can override.

How we built it

We built the robot on a Hiwonder JetAuto, a four-wheel mecanum base with an STM32 motor board (buttons, buzzer, OLED, IMU), driven by an NVIDIA Jetson Orin Nano 8 GB running ROS 2 Humble. For sensing, it carries an Orbbec Astra Pro Plus RGB-depth camera in front, an RPLidar A1 for 360° range, three ELP AR0234 global-shutter fisheye cameras (left, right, rear) on a powered USB 3 hub, an iFlytek six-microphone ring array with a USB speaker, a 7-inch touchscreen kiosk for the PIN hand-off, and GPS from the hotspot phone riding on the robot after the USB GPS puck proved deaf. Everything runs on the robot itself: RTAB-Map visual odometry fused with wheel and IMU data by an EKF, NVIDIA's X-Mobility model as the local driving policy (ONNX on the GPU), and a motion gate with lidar clearance checks that has the final say on every wheel command. A single GPU vision server runs YOLO26n-seg plus a fine-tuned person detector (TensorRT), and three models from Hugging Face: Depth Anything V2 Metric-Outdoor-Small for depth from the color image in daylight, NVIDIA's SegFormer-B0 (ADE20K) as the night-time ground model, and Facebook's Mask2Former Swin-Large (Mapillary Vistas) with our own adapter to tell sidewalk from road, crosswalk, and grass. A Vosk speech model listens for "help" and "stop that," saves video from every camera, and announces a simulated 911 call, and a FastAPI backend (Postgres, Redis, Firebase, Cloudflare tunnel) and an iOS app send orders to the robot over a WebSocket bridge. We also evaluated VILA 1.5, Qwen-Drive, SmolVLM, and SmolVLA from Hugging Face but did not deploy them, and used NVIDIA Isaac Sim for early simulation and reinforcement-learning experiments.

Challenges we ran into

One of the biggest challenges and the greatest thing we realized is that GPS is hard on a small robot. Our USB GPS receiver heard no satellites, so we switched to the iPhone's GPS and compass. That meant designing for about 5 m of error: arrival requires three consistent position readings, not just one. There were real-time limits. On the robot, visual odometry ran about 300 ms behind, while our motion gate requires a pose under 200 ms old. We had to tune the robot to meet that rather than loosen the safety rule. Bringing the simulation into reality was a tough job. We started with reinforcement learning, then found our simulator's obstacle model was measurably wrong. We threw out the learned pipeline and rebuilt navigation as deterministic, testable obstacle avoidance. Networking in the field with QuickTunnel's address changes every restart. The robot booting offline with its clock at 1970 broke HTTPS. Campus Wi-Fi's captive portal silently swallowed connections. We built automatic reconnection, clock-setting from GPS time, and plain-language connection errors. Integration was complicated. We merged robot, backend, and app work from several branches. One of our bugs was a .gitignore rule that hid the app's whole data layer on case-insensitive file systems.

Accomplishments that we're proud of

As a team, we are incredibly proud of a full order-to-robot pipeline. We were able to build an ordering app that becomes a route the robot drives, with live tracking and a PIN-secured hand-off. Autonomous delivery through the full safety chain in simulation was able to handle people crossing its path, barrier detours, and a hidden destination it had to search for. On the real robot, the motor path and safety checks pass, the motion gate is deployed, and localization meets its timing target.

What we learned

Throughout this process, we learned that measuring the real robot before trusting the simulator is key: wheel radius, sensor latency, and GPS error are all very realistic things. We also had to adjust to the fact that hardware projects are mostly integration with clear interfaces between robot, backend, and app, letting a team build them in parallel. Planning ahead and designing for failure was one of the best things we did in this process. Then dropped connections, restarted servers, and bad GPS fixes all happen, so every part needs a safe default.

What's next for Knight-Wagon

Our goal is to definitely set up smooth real-world tracking in the app, as well as making it accessible to campuses all around the nation. There are things we want to build on and improve, like creating a temperature-adjustable, lockable compartment that opens only with the right PIN. Subsequently, side and rear fisheye cameras feeding obstacle checks and person detection running entirely on the robot. Integrating businesses and the concept of campus dining is also a bullet point on what is next for Knight-Wagon. Real menus, live prices, and payment with student meal plans.