Emoɉion

Emojion in action!

Inspiration

With the explosive growth in AI technology, many doors have been opened, yet not fully explored. One prominent example is accessibility, where applications have room to be built and issues room to be solved.

Over 350 million people worldwide suffer from an assortment of disorders which make it difficult/impossible to "read" others' emotions. The intricacies of facial expression and vocal intonation escape them, often leading to them failing to read the room, both on calls and in person.

What it does

Emoɉion is a browser-based extension where the user will upload an audio file, receiving a transcript and emoji in return. The emoji indicates the emotion of the one speaking, with a smiley face for positive emotions, a sad face for negative emotions, etc.

How we built it

We created the extension with HTML, CSS, JavaScript, and Chrome’s extension API. We programmed the backend to send requests to AssemblyAI’s transcription and sentiment analysis APIs and returned the data to the frontend. Finally, we combined setTimeout() with recursion to display each sentence and word at the same speed that they were said.

Challenges we ran into

Our first 5 to 6 hours of Ignition Hacks were spent on coming up with our idea—and even after we did, we ran into many challenges when building the application.

AssemblyAI greatly lacked JavaScript documentation (mostly only had python)
Certain chrome-extension API permissions were only allowed for packaged apps, not extensions (ex. fileSystem, audioCapture)
Many NodeJS functions did not work in the browser (ex. require())
Could not use import statements “outside a module”

We also attempted to integrate live captioning with AssemblyAI’s WebSocket API… and ran into even more challenges.

Captured audio was in the form of a Blob…
Could not convert the audio data to a buffer (Buffer.from() did not work in the browser)
Finally, after successfully capturing audio in the browser and converting it… the AssemblyAI WebSocket API denied the requests (Not Authorized) — it was a paid feature all along

Accomplishments that we're proud of

We’re proud of our commitment to the project, our resilience when running into challenges, and our final product as a whole. We’re especially proud that we were able to create a functional application that can be used to help so many disadvantaged people around the world.

What we learned

We learned that great ideas are very difficult to come by. We spent the first 5 hours pursuing various ideas, realizing they were all fruitless. Finally, after another hour of deliberation, we came up with one that we agreed to be great: Emoɉion.

We also learned that time-constrained projects are best done by dividing and conquering. We divided the work according to each of our strengths, and conquered by delivering all of our sub-tasks on time. This, along with other time-saving techniques—rewatching workshops at 2x speed and sacrificing leisure time—allowed us to create a finished product within 42 hours.

What's next for Emoɉion

In the near future, we plan on integrating the WebSocket API into our application to support live captioning and developing a mobile app to improve our reach and accessibility.

Built With

assemblyai
chromeextension
css
html
javascript

Submitted to

Ignition Hacks 2023

Created by

I worked on planning, writing the script, recording audio, designing the extension, and the backend (APIs and file manipulation).

Matthew Yang
I worked on the scripting, writing, some preliminary design as well as general brainstorming

Allen Yao
I worked on the front-end of the extension, designed the logos, and created the code to process the data from the API.

Derek Gou
I worked on design, planning the idea, running tests, frontend, music production, video production, food production, script writing, recording audio and moving furniture.

Taust weis