Drive-I

Inspiration

As amateur drivers, we often face confusion regarding yielding right-of-way and using lane change signals. This inspired us to leverage Apple’s mobile intelligence to create an intelligent co-pilot and active passenger system. Our goal is to develop a fine-tuned language model (LLM) that focuses on traffic signals, driver safety, and vehicle specifications.

What it does

Our system interacts with users through zero-shot speech synthesis, guiding both drivers and passengers for a quality travel experience. It can anticipate human errors and detect vehicle defects, ultimately helping to prevent major accidents..

How we built it

We utilised the Whisper Turbo model for its rapid speech-to-text conversion capabilities and it is integrated as uAgents using Fetch.ai starter kit. The core of our system is powered by LLAMA (META's LLM), which processes the driver’s spoken questions and generates text-based answers. These answers are then converted back into speech using XTTS.

Challenges we ran into

1.) Model Installation: The PYLAMA model, trained with 30 billion parameters, posed installation challenges due to its high computational costs. As a solution, we opted for the 7 billion parameter version, which is more manageable for low-cost GPU models. 2.) Audio Capture Limitations: Initially, capturing 3-second audio clips for TTS was problematic. We adjusted our approach to capture audio within a duration of 6-10 seconds to improve functionality.

Accomplishments that we're proud of

Speech Recognition Efficiency: We achieved speech recognition output within 5 seconds, considering our computational resources. LLAMA Response Time: The output from the PYLAMA model is generated within 2 seconds. TTS Output Speed: Text-to-speech (TTS) output is produced within 5 seconds. With this setup, given an audio prompt, users can expect an audio response within 12 seconds. Our implementation of zero-shot speech synthesis using the powerful LLAMA model positions us competitively against systems like ChatGPT-4.0, which interacts with pre-trained voice models. Despite resource limitations, our system achieves comparable performance.

What we learned

Through this project, we gained extensive insights into LLMs (specifically LLAMA) and their time efficiency, which is crucial for our system's performance.

What's next for DriveI

We plan to scale up the system by incorporating a dedicated driver model and vehicle model that continuously monitor the driver's activities using multi-modal capabilities. This approach will ensure the system remains fully integrated with the vehicle, enhancing its ability to assist with safety and decision-making in real time.