Tiny Robot LM

Pillar Category

Best overall, Best technical

Inspiration

Harini and a few of her classmates started building an autonomous food delivery robot as a summer project. The robot takes a picture every five seconds, queries a cloud-LLM with the frame for a command, and the executes that command. The problem with using a cloud-LLM is the extremely high latency and costs. Our aim is to build an autonomous movement system with fully-offline inference, and is small enough to run on our robot's computer.

What it does

Our TinyRobotLM works together with a pre-trained CNN to turn natural language instructions into real robotic movement. The video stream from the robot's camera is fed into the CNN, while TinyRobotLM parses movement instructions, along with the CNN's detections to send real movement commands to the robot.

Aside from the resource-intensiveness, using LLMs for image recognitions tasks is just redundant. Tiny language models in simpler tasks like ours removes the worry for computation and GPU resources and training time. Having full control over the system lets us customize the language model to our own needs and makes it easier for us to apply to our use case.

Our approach cuts down on latency to the point that it's barely noticeable.

How we built it

We created our tiny language model from scratch. It has 4 layers, 4 attention heads, 3 million parameters and fits in 11 MB. We trained it for 30 epochs with batch sizes of 128 to achieve 93.8% accuracy. It turns an instruction like "move forward until you see a bottle" into short action codes (FWD UNTIL bottle) that the robot's own code carries out.

We used a large language model to brainstorm our training dataset for this tiny language model. This is a form of distillation where a large AI model teaches a smaller AI model. We used our own script to split, fill, filter, balance and add noise to LLM generated dataset, and had 40000 examples to train our tiny language model with.

We reused Jetson Nano's open-source code for motor-bridging and controlling the movement of the vehicle.

Challenges we ran into

In the beginning, we aimed to build a small language model, but we soon realized that even a small language model would require millions of training data and large compute and memory. It was way more than we can handle and we had to settle on a tiny language model instead.

The older version of Jetson Nano that we're using definitely influenced many our decisions because we had to ensure it doesn't break when receiving commands from our model.

Accomplishments that we're proud of

We learn about concepts and theories in class that we never get to apply in real life. Building this model on our own from scratch was a great hands-on learning experience, one that we didn't think we would be able to achieve in the beginning; not to mention our 24-hour time limit. Creating an AI system that works so well in such a short amount of time is an achievement we're incredibly proud of.

What we learned

It's always quality over quantity when it comes to training data. We wrote the labels using our own script, instead of relying on an LLM and manually reviewed them to make sure they were all labeled correctly.

What's next for TinyRobotLM

Our goal is a delivery robot that has its own fully-offline environment-aware navigation system, that takes instructions such as: "Deliver this parcel from ECS 416 to DESN 112." The next step would be to refine the CNN to detect and recognize walls, doors and other obstacles.

Built With

• css
• html
• javascript
• jetson-nano
• python
• transformers
• yolo