Inspiration

Each year, millions of immigrants and nonnative english speakers enter the job market. In Los Angeles County alone, the English Second Language employee population grows by 5.75% per year. Given that English is not their primary language, many of them face cultural struggles and bias when interacting with potential employers.

We created Fluent after recognizing a major resource gap between members of immigrant families in the US and employer expectations of job applicants, both implicit and explicit. To fix this resource deficit, the creators of Fluent wanted a solution that starts at the entry level: at the job application stage. With cultural context in hand, Fluent users can better navigate the intricacies and cultural norms of American job interviews

What It Does

Driven by the millions of job applicants who desire a better mastery of the English language and knowledge of American societal norms, Fluent is a vocabulary teacher, interview prepper, and a cultural guide all in one handy application:

Word WalkThru: learn industry vocabulary and train your speech skills in seconds with Fluent’s Word WalkThru, a guided voice coach that analyzes your pronunciation and clarity!

Interview Tutor: put your interview skills to the test with our list of sample questions with Fluent’s Interview Tutor, a question-and-answer coach that checks you for time, grammar, and annunciation!

Culture Guide: learn how to culturally connect with your employer with Fluent’s Culture Guide, a compilation of important cultural norms that are essential when working in the US!

Fluent’s target demographic may not have access to smartphones, so we took the extra step to making all of our services available for any devices with SMS and calling capabilities using Twilio. Text 310-919-0244 to get started!

How We Built It

Fluent is a mobile app and Twilio SMS and calling bot.

Throughout the app, we tried to make the experience as seamless as possible for a nonnative speaker. Google Cloud's Translate API greatly helped with this and allowed us to show learning opportunities in a user's native language.

A lot of resources were put into Word WalkThru's pronunciation evaluating. The foundation of Fluent’s spoken word analysis is a weighted-sum formula that assigns user responses with a confidence score between 0 and 1. We analyze words by identifying and comparing spoken to standard English phoneme sounds, a form of “word substructure”. Unfortunately, phoneme recognition technology is nowhere near perfect, so we could use a direct delta comparison This weighted sum relies on three component subscores:

  • Spoken Word Analysis Subscore which looks for similar patterns in consonants and vowels
  • Levenshtein Distance Subscore between expected and actual phonemes
  • Google Cloud Platform Speech-To-Text Confidence Score, We were able to expand on Google' Speech-To-Text API to give proper feedback on pronunciation.

Using phonemes, we were able to dynamically generate mouth animations from a collection of gifs we created in after effects.

To bring Fluent’s breakthrough language assist features to our key target demographics, we harnessed the power of popular phone communication API Twilio to handle texting and phone call versions of Fluent. Our implementation of Twilio allows for a seamless user experience, from a simple text menu to access Fluent’s three core features to an instantaneous phone call recording of user responses to a concise feedback text instructing users on how to improve their pronunciation and clarity.

Challenges

Interpretation of the user’s speech was mainly hampered in quality by the lack of phoneme recording technology. Fluent entirely relies upon the recognition of phonemes within spoken words, and without the recording quality, certain words come off as unrecognizable and hard to match without an algorithm to handle the interpretation load.

Accomplishments We’re Proud Of

We’re proud of Fluent’s seamless Twilio experience and how it brings our project’s benefits to anyone without a smartphone, especially immigrant or ESL citizens aiming to land that one dream job by working on interview skills not traditionally afforded to them.

Additionally, we’re proud of Fluent’s usage of GCP in our weighted-sum algorithm that identifies spoken words and provides the users with specific ways to improve their speech. This level of detail would not be possible without Google Cloud Platform’s powerful Speech-To-Text API.

What We Learned

When designing an educational application, user experience is absolutely necessary because the user needs to enjoy the learning process to continue reaping the benefits of the application. Ensuring that Twilio was a seamless experience was a major lesson learned as our target demographic may not have access to smartphone technology because we need to deliver that same level of quality experience a user would have normally received on the iOS application.

Another key lesson learned is the balance between informing users of cultural norms without offending anyone due to the differences between one culture and another.

What’s Next

Integrating this project into job application portals company-side would be a major breakthrough for Fluent. We believe that creating a web-based plug-in that can be easily added into a company’s website would help bring additional support to the growing number of ESL (English Second Language) applicants looking for work in the United States.

+ 8 more
Share this project:

Updates