Bando

Inspiration

We took inspiration from both the rise in the use of conversational chatbots in many consumer facing sectors that promote a more "natural" style of conversation and the multitude of AI "companion" projects and products that have been circulating as of late that try to provide more personable AI interactions with their users.

What it does

Bando is a voice assistant designed to provide financial information and advice using the user's financial providing a more casual and personable way to interact with personal finance contrasting with current banking apps which simply give you account ledgers and balances. Bando can also provide context aware financial advice to a user based on their personal finances and even provide suggestions for shopping and dining that fit within a user's desired budget. We see Bando as a financial companion rather then just a standard chat bot and hope it can remove some of the stress that comes with personal finance.

How we built it

Using a Raspberry Pi 5 as our hardware basis we build a software stack on the device meant to transcribe voice inputs from the user, parse the context of those inputs, and along with the user financial data utilize an LLM to provide a response to the user.

The foundation of the device is the Open Voice OS (OVOS), specifically it's raspberry pi focused fork RaspOVOS built on top of Raspbian. RaspOVOS enabled the processing of voice inputs and examining the context of those inputs to establish what relevant functions should be called in response and integrate other necessary software packages making use of the fact that underneath the OVOS functionality there's still a fully functional Raspbian installation.

The project originally intended to use the Nessie API for mock financial data but due to its lack of functionality at the beginning of this hackathon we decided to get creative. We made use of a containerized instance of the software package Firefly 3, a personal finance manager, to act effectively as a simulated bank account with its ability to store account balance information and transaction ledgers with robust categorization options. This in conjunction with it's easy to use API and python integrations made it a perfect choice for this project, additionally with tools like SimpleFIN a hypothetical production ready version of Bando could import real world financial data quite easily.

With our source of financial information sorted we turned to configuring OVOS to fit our needs, specifically OVOS makes use of a system called "skills" which define how it responds to certain contexts of voice inputs. In this case we created a finance skill running under OVOS with a custom example file of a variety of ways an end user could possible ask a financial question, referred to as "utterances" by OVOS, to create a strong contextual foundation for skill calling. With this established tooling functions were written into the skill for the easy parsing of a full financial data set into relevant pieces for cloud processing.

When we originally began work we were told that the Pi 5 we had been given was running 16GB of memory, it was not, it had 2 Gb. Due to this we were much more limited in our edge processing options and decided to make use of cloud services for our AI workflows. In particular the LLM is being accessed through the Google Gemini API and the natural voice synthesis is performed via the ElevenLabs API.

When OVOS determines a financial question has been asked and has fully parsed the question that question and the user financial data are passed over to the Gemini API which uses the provided tooling functions to modify the data into a more suitable state and then produces a context aware response to the user question constrained to around three sentences for compatibility for the upcoming step. Once the Gemini output is received it's immediately sent in a call to the ElevenLabs API to use natural voice synthesis to audibly respond to the user with the answer without sounding robotic or cold.

Challenges we ran into

The first challenge we faced was the speed of flashing the required software onto the Pi. The storage provided to us was quite slow and placed a considerable limit of the write speeds of the Pi Imager software causing a very extended delay simply awaiting the flashing to finish. This is omitting the several times the flashing failed due to the Pi Imager software's extreme sensitivity to external interference requiring the entire process to repeat.

With image loaded onto the Pi we came to another realization: we couldn't configure the Pi because we lacked a proper display and keyboard to connect to the Pi physically and when trying to utilize local link via Ethernet on one of our computers it would not work. The solution to this problem ended up taking the Pi from the rice campus over to the University of Houston campus where one half of the team had a more suitable networking setup and could properly ssh into the Pi and configure it setting up RPI Connect and and a fallback wireless network connected to a mobile hotspot in case the Ethernet network did not properly function upon return to Rice the next day.

The Ethernet network did not properly function upon return to Rice the next day. Unfortunately we found Rice has extremely strict firewall policies and would absolutely not allow the pi to connect via Ethernet or wifi. We even went as far as retrieving a wifi router from UH only to find that too was blocked off the network by the Rice firewall policy. Thankfully we were able to make use of the previously mentioned mobile hotspot backup which worked despite the slow speeds due to low quality cell signal in the Rice student center.

When trying to install Docker so we could containerize Firefly it quickly became apparent that Docker's installation documentation could be described as subpar. The documentation listed several advanced methods for install as if they were the one true and only way of installing the program which became a challenge to follow and didn't seem to properly function. This was resolved when looking into community contributions there was an official single run script which bundled all the advanced actions into one action to ease installation which was not mentioned on the software docs.

As we got into writing the software we created a sample data sheet to import into Firelfy but we forgot to check what the accounts actually totaled to in our data. We had uploaded accounts that were overdraft by approximately $2.3k, while this isn't an impossible situation it goes without saying this isn't ideal so we created new data and verified its integrity before importing it.

Have I mentioned we haven't even begun getting into writing the software yet?

When we began writing the code to be written on the device starting with the cloud services pipeline as we were trying to test our API access we noticed significant performance slowdowns and found we only had about 400 MB of ram remaining. Firefly only used about 190 MB at max so it had to be an issue within RaspOVOS itself that was causing issues. As it turns out RaspOVOS contains several skills meant for certain common tasks and when RaspOVOS was parsing an input all of these skills would be loaded into memory essentially becoming a memory bomb. We had to disable several of these tasks to ensure they were not weighing down our memory usage and were able to get a consistent 1 GB of free memory but don't worry this won't be the only time we see these preinstalled skills throw a wrench in things.

At this point we moved on to developing the skill we required for the financial questions and this is where things really got fun.

When we began testing with audio input for the OVOS speech to text protocol it was not accurately picking up the voice input at all and incorrectly transcribing what was being said. As it turns out some combination of bad microphone calibration resulted in the input audio being distorted heavily as far as the software was concerned and the microphone had to be readjusted so that audio came in clearly.

Previously we had developed the cloud software pipeline and tested it by directly querying it with the user question in text while it pilled from Firefly for financial data and this process was fully functional. When trying to invoke the same process through the OVOS voice input those it wasn't functioning correctly. We'd come to find out the input was being "intercepted" by yet more rouge preinstalled skills and it became the world's most frustrating game of whack-a-mole to try and disable and uninstall these unnecessary skills. We got to the point all offending skills were taken care of and the input was able to proceed without issue.

Is what I would say if it was working yet at this point. Remember that comment about freeing up memory and having 1 GB available now? Well turns out that wasn't enough, the skill process was automatically terminating itself due to memory overruns. We configured soft memory in the form of a SD card page file for the additional data to be pushed to in order to correct the issue.

An additional issue was found with a deprecated piece of code that was never commented out or deleted outright causing issues. Thankfully this one was as simple as deleting the offending query.

Finally, there was a typo in the Gemini model selection we had to fix.

Accomplishments that we're proud of

• Optimizing traditionally high memory impact software for use on memory constrained hardware.
• Troubleshooting network issues overcoming access issues that would have otherwise made the project impossible.
• Effectively working as a team to compliment each other's areas of higher skill to contribute to a quicker project turnaround.
• Scheduling to make the most of the time constraints we have to operate in.
• Diving headfirst into unfamiliar software and learning as we go.
• Not ending up in a psych ward.
• Going with our gut and immense egos sticking with the hardware project.
  ## What we learned How to manage a complete pipeline of audio -> processing -> retrieve information -> audio Developing effective local to cloud data exchange pipelines. Optimizing audio capture for high clarity and ease of post-processing. Advanced configuration of Raspberry Pis running Rasbian based operating systems. Collaborative software packages for networked remote access to Raspberry Pis. Handling and integration of external APIs into programs. Memory optimization. Financial data handling practices. Advanced use cases for Google Gemini and ElevenLabs. Usage of Notability for documentation. ## What's next for Bando If Bando has a life beyond this career fair we'll likely focus on building in data anonymization and better skill handling to offer more responsive user interactions. We'd also like to bring it into a security standard that could justify the interaction between Bando and real world financial data. Finally we'd like to move to bespoke hardware to reduce Bando's footprint and give it a compact aesthetically pleasing housing.

Built With

• elevenlabs
• firefly
• gemini
• github
• google-cloud
• ovos
• powershell
• python
• raspberrypi
• raspovos
• ssh
• winscp