TravelLong - A traveling knowledge buddy

Inspiration

Our project, TravelLong, was inspired by the character BingBong from the videogame Peak. In the game, BingBong is a nonplayable character (NPC) that helps you make decisions by playing random voice lines, sort of like a magic 8 ball. But what if BingBong could actually really help you? What if his voice lines weren't random?

What it does

TravelLong is a low power, high performance communication tool used to interact with local LLM's in the most remote places.

How we built it

TravelLong was separated into two core components, Bingbong and The Luggage.

The Luggage

"The Luggage" is our name for the core processing unit of Bingbong. This unit sits somewhere with a power source (e.g. a generator, solar panel, battery, etc) and is in charge of running many of the critical services such as.

• A local LLM for on the fly response generation
• The Voice to Text model to feed into the local LLM
• The Text to Voice model to feed back into Bingbong for audible responses

The computer powering this is a Jetson Nano, a small but powerful computing device designed for LLM/AI operations due to its extensive array of neural processing units (NPU).

Bingbong

"Bingbong" is the physical device that someone would carry on a long trip or hike with them. BingBong was originally intended to be equipped with a long range radio designed to transmit questions but due to constraints was replaced with a local WiFi micro-controller. In order to support the "low power" aspect of this product, Bingbong is equipped with a small button that enables the micro-controller when pressed in order to both save power and ensure privacy. Bingbong is also equipped with a smaller speaker that allows it to speak back, audibly providing responses to your questions.

Challenges we ran into

There were two critical challenges we ran into throughout our project that caused us to shift gears. First, our initial exploration of using LoRa modules, designed for long range communication reaching hundreds of kilometers, proved fruitful at first but getting them to communicate using custom firmware provided to be a problem. Custom firmware was critical in getting the speed necessary for sending voice data as the default firmware flashed onto these devices often is based on users manually entering bytes to send across the network, and thus was sub optimal for our goals. The second major issue when switching away from our high performance LoRa modules was getting those aforementioned speeds on other LoRa chips that we did have success writing to but ultimately were not able to get the speed out of them.

Accomplishments that we're proud of

We were able to make a fun friend, Bingbong, and learn lots of information about LoRa and sending information in compressed and encoded formats. Despite not making a lot of physical progress, we made a lot of learning progress!

What we learned

While we may not have been able to get it working in the end, we learned a great deal about the development of custom firmware for STM LoRa modules. Furthermore, we spent a lot of time on writing custom software in order to test the throughput of LoRa by generating a sin wave and sending it over the network, receiving it on another device and plotting it to view either the resolution of the wave or if there were any missing chunks.

We also learned a lot about Audio Codecs and the various ways that you can encode, decode, and compress audio data. This was a necessary learning path as we tried to understand all of the different ways we could optimize the amount of data we were sending over the network.

What's next for TravelLong

TravelLong has a lot of work in the future that could turn it into a viable product

• Further development into the use of high performance LoRa modules for the ultimate high range, high throughput communication system
• Further development into the physical cases (Bingbong, The Luggage) that hold all of our electronics. Staying in theme with the game Peak, the actual device that holds the local LLM was drawn up to be luggage but ultimately time did not allow for that to come to creation.
• Further refinement to LLM prompting to optimize for bandwidth across the network

Built With

• esp32
• jetson
• linux
• lora
• openai