Amelia

• 

  Dashboard

• 

  Test Message Notifications

• 

  AI Assistant Console Output

Inspiration

Preventative maintenance is an important part of making sure that a car runs smoothly for its expected longevity. When looking at it from a cost perspective though not only does doing proper maintenance decrease the number of claims from easily avoidable accidents. It can cost up to 40% less than reactive maintenance. We wanted to take that concept and help people make sure that they are always on top of the many obscure systems in a vehicle that most people aren't aware of. For example replacing windshield wipers is not a normal part of maintenance, but should be done every year.

What it does

The app has two main features. Our primary feature is taking a mock of data already collected by State Farm safe driving sensors in the vehicle such as cornering, speed, distance, time and putting that data through a model at the end of each drive to check if a user is getting close to needing maintenance. In the case where there is a required event it then sends a text message to the user with the specific alert that pertains to their case and a number. Users can then call the number if they have further questions about why they need that specific maintenance and how it benefits them as a user to receive it. Our secondary feature is a mobile app that contains an approximate guess until the number of miles or time until a specific piece needs to be replaced. These numbers are based off of typical service numbers. Users can then update when they did maintenance. This goes to the backend as additional information for the model.

How we built it

We built the main alerting flow through a series of scripts that include alerting services such as Twillio and answering services through Google Voice. The conversation service is built using a combination of ChatGPT for the response and eleven labs for the voice. ChatGPT is fed specialized prompts and specific information about car maintenance as well as the analytical data from the user for case-specific responses. The app is built using Expo Go, a React native framework. Our backend uses Python to process the audio and send requests.

Challenges we ran into

The main challenge we ran into was being able to pipe in audio from a real phone call into the whisper API. This is due to Google Voice not directly supporting answering machine-styled requests. We had to build a program that maintained a seamless corridor of streamed data between the different services. Another issue we ran into was that when we mocked the data it was difficult to train a good model with it due to how the data was generated.

Accomplishments that we're proud of

We are most proud of our ability to switch projects 5-6 hours into the hackathon because we felt that our original idea was lacking for the challenge statement. We went back to the planning board and executed a new idea that we felt better accomplished the goal of the statement. We are also proud that we used technologies such as ChatGPT, ElevenLabs, and Google Cloud in a responsive and scalable way in a complex end-to-end workflow.

What we learned

We learned a lot about virtual cables and how they are processed in Windows. This was what allowed us to actually connect these technologies. We also learned a lot about classification models in particular. This was necessary for the backend to know when an alert needed to be sent.

What's next for Amelia

The primary thing we need to improve for future iterations of Amelia is the alerting model. With better data, we will know with greater certainty when something is wrong with the vehicle. We also want to open up the responsive phone call for people to be able to ask questions about issues that they notice such as specific sounds or other issues that can't be detected by a sensor.

Built With

• chatgpt
• elevenlabs
• google-voice
• python
• react-native