Inspiration
In the modern age, social media is the main way for social engagement, surpassing face-to-face interaction, and could provide a comparable degree of emotional support to users. However, Aphasia patients are unable to tap into these benefits and be more extensively connected to the rest of the world as current social media relies heavily on words for communication and expressions.
What it does
From research, we understand that Aphasia patients could understand the meanings behind simple drawings of human faces. As such, we decided to use emoji to serve as a bridge to allow Aphasia patients to understand the meanings behind the words on social media. There are 2 parts to our solution:
Emoji translator - Bringing existing contents on social media to Aphasia patients The semantic features of natural language sentences on social media are extracted, and the words are replaced by the most semantically similar emoji.
Emoji Board - Providing an easy way for Aphasia patients to express themselves An emoji-based, simple and structured interface to allow Aphasia patients to express their ideas easily. The semantic features of each chosen emojis are then extracted, to combine with the syntactic structure specified by the user, as to generate a corresponding sentence in natural language - allowing normal users to understand Aphasia patients.
How we built it
Emoji Translator
| What is used | Why is it used |
|---|---|
| Part-of-speech based algorithm | For keyword extraction |
| spacy, word2vec | Turns emojis and words turned into vector embeddings |
| sklearn | To represent a space of all emoji-embeddings and find nearest neighbor (i.e. nearest emoji) for a given keyword |
| Heroku | To host our text-to-emoji API |
| Google cloud | To host our word2vec models efficiently |
EmojiBoard
Challenges we ran into
- Lack of "accurate" pretrained emoji-embeddings
- Difficulty in hosting machine learning models with high performance
- Stringent requirements for accessing official Twitter API
- Difficulty to generate linguistically accurate and sophisticated sentences from keywords
Accomplishments that we're proud of
- A web-hosted working prototype, indicating scalability down the line and readiness for deployment.
- Overcoming all the challenges along the way, obtaining lots of first-hand learning outcomes through experimenting on different solutions.
- Broad technical achievements at different domains from algorithms to web-scraping to frameworks and web hosting services.
What we learned
About Aphasia
This is a group that never caught much attention. As we were doing research, we were rather surprised that there is little to none solutions developed to help mitigate their difficulties.
Multi-disciplinary collaboration
This project a true manifestation that all professions are equally important. Without Su Kee providing professional psychological insights about Aphasia, our UI would have been unsuitable for Aphasia patients at all; and without the tech guys, this project will not be alive in such a short period.
The latest tech
This project has essentially updated our knowledge on frameworks and platforms that could be used for fast prototyping and deployment.
What's next for Aphasia Social
User's feedback
Just like any other machine learning application, real-world usage is the most important factor for improving performance. Also, we would want to tune our emoji options in Emoji Board according to Aphasia patients' degree of understanding of each of them.
Develop a public API
In order for our technology to truly change the life of Aphasia patients, it needs to be implemented widely across different platforms and social media. As such, having a public API is the first step to realize this.
Built With
- google-cloud
- gpt2
- heroku
- javascript
- python
- sklearn
- spacy
Log in or sign up for Devpost to join the conversation.