dialect. - Language Learning Extension

Learn languages passively as you browse the internet.

Comment

Inspiration

We've always been fascinated with learning new languages. So much so that in high school, we opted to drop our mother tongue, Hindi, as a subject and opt for French and Spanish instead.

We wanted to learn new languages, however all existing tools required too much active effort on our part, and with our busy schedules as college students, learning a new language was a daunting task. So we decided to build something that could help us learn a new language passively, which gave birth to this idea.

What it does

dialect. is a chrome extension that helps people passively learn a new language while normally surfing the web.

It randomly translates words/phrases/sentences on any webpage that the user is browsing and uses the power of context to better facilitate learning. The user can choose the difficulty as well as the frequency of the translation, and can always hover over the translated word to see the original text in case they don't understand. The extension also helps bilingual readers surf the web.

The users periodically take quizzes to test how they're learning, and the extension provides numerous resources for the users to check their progress, such as favourite words, mastered words, and a knowledge graph that creates clusters to help users understand what they've learnt.

How we built it

Once we had a concrete idea in mind, we looked at existing alternatives to see if something such existed. Convinced of our differentiation, as well as the products’ viability, we first explored the Chrome on-device APIs and played around with some of them to see their capabilities. Once we realized that our functionality could be executed using the APIs, we prototyped a UI and started separately working on the frontend and the backend.

We used React.js for the frontend (with Figma used for the designs), and Express.js for the backend. The backend work was mostly just dealing with databases and authentication. The frontend dealt with the UI/UX, as well as integrating the Chrome APIs. Once we had our core functionality ready, we worked on the knowledge graph—which we see a lot of potential in—and came up with a rudimentary implementation for now. The knowledge graph had some challenges, as we later discuss.

Challenges

Since the extension requires continuous translations, we struggled a lot to optimize our task to be able to work through Gemini Nano. In the end, we realized that treating our translation output as a chunk of Promises can help efficiently translate words/sentences on a webpage while the user is scrolling.

We also faced challenges creating the knowledge graph which required heavy computations and structured output. There were a few tasks that needed us to resort to a server-side AI by using the Gemini 1.5-Flash API on the backend (such as knowledge graph creation, and some part of quizzes); all other tasks were performed using client side APIs. These are generated on the backend mainly due to the limitations of Gemini Nano's prompt API, and it's ability to make complex inferences on large inputs.

Accomplishments

The main that that we’re proud of is that despite whatever challenges and constraints we faced, we managed to produce a fully-working chrome extension that does what we intended it to do, along with a robust, performant knowledge graph that is able to cluser words together without the use of expensive embedding methods.

Learnings

Nobody in our team was particularly proficient in building extensions before this hackathon, having only built 1 before. More than that, we'd never worked with cutting edge technologies such as on-device AI, and it was really fascinating to see how a browser was able to serve a whole LLM.

We only just learnt a lot about linguistics, and how different language semantics matter during translations.

Future

The extension is currently an MVP, and there is a lot of scope for further improvement. We'd like to add Google OAuth, a leaderboard system, and figure out a way to transfer the knowledge graph to client-side AI too. Most of these weren't done by us due to time constraints, but will be done once the hack is over.

Along with that, we feel the extension requires a lot of polishing and beta testing before being released as a usable product.

Our end goal is to further iterate and build the extension properly, and eventually release it on the Chrome Web-Store. We see a lot of value in this and hope that others will too. We also haven’t monetized the extension in any way but may add features in the future that could use a tier system. Currently, we are using MongoDB Atlas' free plan as our database, however, given the fact that all words seen are being saved for the knowledge graph, we may need to upgrade or switch architectures to be able to scale.

The main challenge with releasing the extension right now is the friction of downloading the language packs and all the files needed to get the on-device APIs to work. Hopefully, as we iterate and build further, the APIs also become more consumer-friendly and the setup of the extension isn’t a friction point.

Built With

  • chrome-prompt-api
  • chrome-translation-api
  • express.js
  • gemini-1.5-flash
  • gemini-nano
  • react
Share this project:

Updates