OnPause

Our Story

As 4 team members who grew up in non-English speaking communities, we recognize the significance and efficacy of learning new languages simply by watching TV shows and the news. Having seen our parents, grandparents, and community members time and time again reach for the remote, pause the TV, and ask to translate a word that they did not understand, we seek to create an automated tool to empower our immediate communities. While TV screen time has begun to decline, younger generations now lean towards using their computers and phones, choosing to view Youtube, online video streams, and podcasts in its place. Instead of getting up to pause with a remote, people now race to hit the pause button upon hearing a word that sounds unfamiliar and attempt to drag the slider back to the precise instant that they encountered a challenging word. Once they find this word, users start a lengthy process of translating and defining it back in their native language.

Through OnPause, we seek to streamline this process of language learning by giving our users the ability to immediately learn these challenging words the very instant a user pauses their media.

How Does OnPause Work?

OnPause is a language learning tool that better allows its users to learn a language through Youtube, streamed online content, and podcasts. We take transcripts from the media our users consume and preprocess it with Rev's proprietary text transcription services and, upon pausing of the content being watched, we immediately display confusing words and their definitions in the user's native tongue. This bypasses the extra step of needing to search Google Translate or holler at a friend and empowers students to learn faster and more efficiently with minimal extra effort.

How Did We Build OnPause?

All our audio/video processing, as well as our front end, was created with Python. Special thanks to Rev.com for their essential transcription services which allowed us to create smart closed captions that can handle whatever time stamp a user chooses to stop at. By tweaking open-source Python media players, we were able to record the precise time within a video that a user pauses at, and, in turn, compare this time to our Rev transcript to pull nearest words. Through the magic of bilingual dictionaries and translation APIs, we are able to ascertain the Spanish equivalent and Spanish definitions of potentially challenging English words. Then, we push this information on a new window so that the user has immediate access to the necessary context to add confusing words to their own vocabularies.

Challenges

Python dependencies and file system setup was a significant hurdle when we started. File collaboration also was a bit difficult given the variety of operating systems we were running. However, once we solidified our hackathon idea and found the necessary libraries, we were all motivated and empowered to forge ahead.

Notes to Ourselves

Be cognizant of Mac OS / Windows compatibility issues
Communicate frequently to create code that is cross-compatible without heavy editing

Accomplishments!

As rookie hackathon members, we're proud of the work we've done from the use of APIs to our front end displays. When we started, we were unsure if we would be able to submit a working prototype and we're extremely proud that we were able to incorporate some many of our core features into our submission.

Lessons Learned

Time management and prioritization. Not all the features that we wanted to incorporate were needed to display the full capabilities of on OnPause and, in the short time scale of a hackathon, it's important to prioritize core functionality. As we developed our numerous features, we came to this particular conclusion as the morning light drew nigh.

What's Next for OnPause?

We'd love to continue working on our front end as well as expand the number of languages that we support and be more accessible to the rest of the world. We also saw that our Spanish dictionary API service did not have as many Spanish words in its repository and that's something we'd like to explore a fix for as well. Recognizing the growing popularity of podcasts and mobile media consumption, we'd like to develop mobile-specific applications some of which could include storing user's challenging words to a notepad that can be accessed at a future time. We're proud of where we're at right now but there's also plenty of work to be done!

Built With

pyglet
python
rev.com
tkinter

Submitted to

HackMIT 2019

Created by

I worked on the front end of our application and created a way to push the transcribed text and its translated words / their definitions to a pop up window once the user presses pause. I also worked on a script to translate words from English into Spanish.

Diego-Raygoza
I worked on tweaking open source Python media players so that we could accurately record the time stamps whenever a user paused. I also worked on a script to hit a Spanish dictionary API to return Spanish definitions of Spanish words.

pileofscraps Lau
I worked on back-end logic, particularly on taking the data from Rev and returning useful information based on the status of the video player. I also helped with code integration between all 4 members to ensure error handling and cross script functionality.

Sergio Perez
I worked on writing a script to interact with the Rev.com transcription service and further processed the JSON file received to remove filler words and commonly used articles. I also assisted Sergio with code integration and fixed compatibility problems within our 4 scripts.

isaaknh Hernandez