Mercury: The Multilingual Virtual Assistant

Spectrograms that standardize audio data according to frequencies
hacking

💡 Inspiration

While numerous people enjoy the privilege of being multilingual, there are others who can only speak their mother tongue. Often, the inability to speak the most common languages prevents these people from truly adopting technology. For instance, the last few years have witnessed the increasing popularity of virtual assistants. From setting alarms to creating reminders, virtual assistants perform mundane tasks that enrich human lives; however, most virtual assistants lack the ability to recognize multiple languages. To promote the use of virtual assistants in unilingual people, we created Mercury, a multilingual virtual assistant.

💻 What it does

Analyzes spoken input through language processing
Performs daily tasks like setting alarms and scheduling events
Facilitates interactions between people and technology

⚙️ How we built it

Used a dataset of audio files to train a MobileNetV2 ML model through transfer learning
Created state of the arc spectrogram synthesis application
Modified an open-source virtual assistant to include language detection capabilities
Built command line voice interface enabling users to query Mercury with any question in numerous different languages

🧠 Challenges we ran into

Converting audio files to spectrograms
Picking the ideal model for our purposes
Tuning the model using different parameters
Collecting user input from the front end

🏅 Accomplishments that we're proud of

Implementing a working and functioning prototype of our idea
Designing and developing a minimalist and clean user interface
Create a program that accurately recognizes language and responds accordingly
Potential to accelerate and revolutionize the dissemination of information to billions of users