Social X-ray

Inspiration

We were inspired to create Social X-ray to help our peers become stronger conversationalists. In a competitive field like computer science, making a strong impression on the interviewer can be the difference between getting the job and needing to job-hunt for another six months. We feel that our app can help autistic people, as well; Social X-ray can prompt the user with social cues and explanations of sarcasm so that misunderstandings are less likely to occur.

What it does

Social X-ray provides real-time conversation coaching through AI-powered social skills analytics. It grades your conversations on a scale of A-F, with A being that the user is a good conversational partner, and F being that the user needs more work to become a good conversational partner. Every 7 seconds, the user's speech is analyzed for several key aspects: interrupting the other speaker, tone, pace, and so forth, and a report is given to the user based on these factors.

How we built it

We built this project using React, integrating Gemini, Cloudfare, and ElevenLabs capabilities for seamless execution. Cloudfare allowed us to piece together our front and back end; the results of the application scanning the sentences were stored inside Cloudfare and later retrieved. Gemini allowed us to find filler words and perform analysis on the sentences, allowing for a deeper understanding on the perception of a sentence. With ElevenLabs, we converted text to speech to AI speech, while Gemini was voice to text and gave feedback, while Cloudfare connected Gemini and text to speech.

Challenges we ran into

Challenges we ran into were our Gemini quota being used up very quickly: after forty-five seconds of talking, we would use up our daily API quota. We also ran into the problem of half the conversation getting lost if the audio clip was above 7 seconds.

Accomplishments that we're proud of

We are proud of our team work, our ability to quickly learn how to integrate APIs, and our brainstorming capabilities. We came up with several good ideas before settling on Social X-ray, and we are proud of the work we put into the project.

What we learned

We learned how to effectively implement APIs into our systems to connect back and front end, and how to utilize ElevenLabs and Cloudfare within a short timeframe.

What's next for social x-ray

To extend our project, we want Social X-Ray to take into account the cultural nuances of the environment the speaker is in. For example, fast talking can be a sign of nervousness. In Minnesota, however, speaking quickly is common. As such, taking into account culture in comparison to emotional states can give more tailored feedback.