Heptapod

Bridge communication gaps with EMOJIs 😊 Inspired by learning about faulty asylum case decisions due to inaccurate translation.

Comment

🎉 my college's final assignment - dedicated to Migration and Border tutorial.

"HEPTAPOD" is a multi-modal translation software, involving both text-to-text and text-to-emoji auto translation.

💡 Inspiration

Through the Migration and Borders tutorial, I learned the pressing issues in asylum seeking process:

  • lack of qualified human translators
  • inaccurate AI-based translation leading to faulty asylum case decisions (often due to the skewed language samples)

This inspired me to develop a solution that integrates AI-powered translation tools with AI-generated visual representation. My approach aims to empower individuals/organizations helping the asylum filing process by presenting visualization of texts to asylum seekers and cross-checking if the translation matches the original meaning.

✍🏼 Learning

Many takeaways from this project!

Prompt Optimization

I take an extensive amount of time to tune prompts to generate emojis from English text. Through many trial and error, I found the following best practices

  • Clearly specify the output (e.g., when the input is two sentences, I mention "two lists of emojis", rather than generic "lists of emojis")
  • Incorporate examples (one-shot or few-shot learning)

Emojis, not AI-generated Images

Initially, I was planning to use AI-generated images instead of emojis. However, the professor's feedback made me realize the difficulty in segmenting the original text into shorter and meaningful units that could be translated into individual images. The punctuation marks can help but are not helpful in long sentences.

On the contrary, text-to-emoji translations do not require meaningful sentence breaks since the combination of emojis can deliver meanings given that emojis cover a wide range of words and scenarios.

💪🏼 Building Project

The project began with extensive research into existing language translation and communication tools. I then identified the limitations and potential areas for improvement.

By using both Google Translate API and Gemnini API, I developed HEPTAPOD, a text-to-text-to-emoji translation tool. The project involved iterative development, testing, and feedback to optimize performance and usability.

🔥 Challenges

Here are some challenges I've faced in building HEPTAPOD

  • How to overcome the limitation of text-to-image translation...? -> Use emoji instead!
  • How to optimize the prompts to generate meanigful Emoji sequences...? -> iterative trial and error while asking Gemini AI itself as well!

🤜 Next Steps

If I have more time to develop, here are some features that could be meaningful to implement on top of the current version.

  • Use the accumulated glossary to tune language model
  • Highlight corresponding sentences in both the emoji and original input text to enhance comprehension and usability
  • Integrating dictionaries into the English text, as Papago translation does, to aid users in understanding nuanced meanings and expressions
Korean dictionary appears when I highlighted Korean words on Papago Papago - dictionary

🙌 Throughout the ideation and implementation process, my initial motivation to bridge communication gaps remained steadfast. Borrowing the name from Ted Chiang’s “Story of Your Life,” the website Heptapod mirrors that language can transcend different dimensions of the world. Just like Heptapod’s arrival on Earth initiates the novel form of communication, I hope the website’s arrival in this world streamlines the asylum seekers’ communication with the institutions. 🙌

Share this project:

Updates