It happens all the time - to students, professors, and working professionals alike. Writer's block, that moment where you lose all writing momentum and sucumb to a seemingly empty mind.
Writing is undoubtely one of the most powerful things we can do. It is the only way we are able to share our thoughts, ideas, and opinions with each other across a medium.
This is why we decided to make Integra, a neural-network powered writing assistant, which utilizes the power of NLP to give you selective prompts in order to further your essays.
What it does
Integra provides Machine-Learning generated prompts to encourage organic thinking when writing. This helps the writer to think critically of their writing topics and spur idea growth.
How we built it
Integra is composed of both the Front-end "document" web interface and the Back-end Machine learning prompt generator. We used Firebase for our pipeline; the user logs into the Front-end via Firebase authentication and then a button click sends the document data (e.g., title and body of the paragraph) to the central Firebase Realtime Database. We also have a python server running in the background which listens for updates in the user stream via the Pyrebase library. The entire pipeline works in realtime, meaning that no refreshes are needed to view changes from the backend to the frontend.
Once the Back-end receives an update from the database, it proceeds with the ML prompt generation. We use multiple toolkits for this process - Spacy, Gensim and the Google Cloud Language API. We first parse the entire body paragraphs into Google Cloud for topic extraction on the entire text body. This is done to provide a focus for the entire article, thus establishing a central topic.
Continuing, we then use Gensim and a custom trained Word2Vec ML model and to identify the "topic sentence". Once the topic sentence has been found, we used Spacy to extrapolate nouns and verbs to compose prompts to submit back to the front-end.
Challenges we ran into
Deciding which library to use, had many issues with training our fasttext dataset in the beginning. Choosing the perfect combination of different ML algorithms was a challenge that took us most of the day.
In addition, we also ran into problems with the integration; we initially tried Flask but the need to refresh the page everytime without a tunnel proved that live updating with realtime database is more effective.
Accomplishments that we're proud of
Getting the machine learning working. It was very tough but we eventually remembered the tech talk from yesterday on Word2Vec and that provided us with a tool for similarity analysis.
What we learned
How to use Natural Language Processing - none of us worked on a NLP app prior to this endeavor
What's next for Integra
Dynamic question generation using Markov algorithms and Recurrent Neural Networks, UI/UX Improvements.