TARS | Devpost

Inspiration

Interstellar has been our favorite movie for years. And its not just for the science and the amazing cinematography its also because of TARS, the robot that accompanies Cooper throughout his entire journey through the universe. TARS was sarcastic, smart and very dependable. We have always wanted to build something like TARS and starkhacks was our excuse to try.

What it does

TARS is an AI rover built for planetary exploration. It can drive through terrain and monitors atmospheric conditions in real time like ammonia and Carbon Monoxide levels, temperature, and humidity. You can talk to it naturally, and it responds through a local LLM running. And just like the movie version, it has adjustable humor settings. You can dial it up and it becomes your sarcastic copilot and dial it down if you actually want to get work done.

How we built it

The hardware stack is a Raspberry Pi as the main brain controlling the display screen, speaker and speech and an ESP32 managing motor control and movement. We used MQ-135 sensors to collect CO and ammonia levels and an Osoyoo DHT-11 for temperature and humidity that are converted into digital signals using an MCP-3008 ADC and fed into the LLM context so TARS always knows what it's breathing. For our LLM we used llama3 from the ollama family.

Challenges we ran into

We faced quite a few compatibility issues. Our main obstacle was that we started out using the Rubik Pi as our microcontroller but we faced many issues with the speaker on it and we found out that the Rubik Pi struggles with audio over HDMI. This made us switch to the Raspberry Pi. Working across the Raspberry Pi and ESP32 was quite complex and the microcontrollers weren't always powerful enough to handle what we wanted on device, which forced us to rethink what ran where. Getting all the pieces to talk to each other reliably took more time than building any single feature.

Accomplishments that we're proud of

Getting a complete interaction loop using a local LLM working end to end by having mic to Whisper to LLM to speech with real sensor data injected into the context in real time. The humour system works well too. Ask it for a joke.

What we learned

That hardware is extremely humbling. Software bugs have stack traces but hardware proved to be much harder to debug. We learned a lot about the physical side of embedded systems like flashing and booting microcontrollers, wiring sensors correctly and other things like why the ESP32 would randomly reset under motor load due to voltage drops, and wiring sensors to the right GPIO pins. We got deep into Linux kernel modules, optimal speech transcription techniques, ALSA audio routing, and learned a lot about Linux distros like Ubuntu.

What's next for TARS

We want to have autonomous navigation with obstacle avoidance and a proper computer vision pipeline once the camera issues are fully resolved and a real time web dashboard showing live sensor telemetry. We want to encase a hardware enclosure that actually looks like the TARS from the movie. Next up would be to expand the sensors and have UV index and radiation sensors to measure surface exposure, barometric pressure for altitude and weather pattern detection, soil moisture and pH sensors for terrain analysis, and wind speed and direction sensors for atmospheric modeling. Finally, of course we would love to make TARS even more personable with many more adjustable personality settings. The goal is to have a rover you could actually deploy somewhere hostile but you can still talk to as if you were human.