discussAI

Home Page
Video Conferencing #1: Face Share
Video Conferencing #2: Screen Share
Searchbox (queries for information)
Searchbox Result (retrieves information from Microsoft Azure Storage with the image link)
Microsoft Azure OCR (analyzing text images and storing keywords/info)
Django API Framework
Postman
Heroku
Microsoft Azure Storage (storing edited and non-edited images)

What inspired us:

The pandemic has changed the university norm to being primarily all online courses, increasing our usage and dependency on textbooks and course notes. Since we are all computer science students, we have many math courses with several definitions and theorems to memorize. When listening to a professor’s lecture, we often forget certain theorems that are being referred to. With discussAI, we are easily able to query the postgresql database with a command and receive an image from the textbook explaining what the definition/theorem is. Thus, we decided to use our knowledge with machine learning libraries to filter out these pieces of information.

We believe that our program’s concept can be applied to other fields, outside of education. For instance, business meetings or training sessions can utilize these tools to effectively summarize long manuals and to search for keywords.

What we learned:

We had a lot of fun building this application since we were new to using Microsoft Azure applications. We learned how to integrate machine learning libraries such as OCR and sklearn for processing our information, and we deepened our knowledge in frontend (Angular.js) and backend(Django and Postgres).

How we built it:

We built our web application’s frontend using Angular.js to build our components and Agora.io to allow video conferencing. On our backend, we used Django and Postgresql for handling API requests from our frontend. We also used several Python libraries to convert the pdf file to png images, utilize Azure OCR to analyze these text images, apply the sklearn library to analyze the individual text, and finally crop the images to return specific snippets of definitions/theorems.

Challenges we faced:

The most challenging part was deciding our ML algorithm to derive specific image snippets from lengthy textbooks. Some other challenges we faced varies from importing images from Azure Storage to positioning CSS components. Nevertheless, the learning experience was amazing with the help of mentors, and we hope to participate again in the future!

Built With

Submitted to

Hack the 6ix 2020

Created by

I worked on implementing the Agora to the front-end of the website, as well as helping create ML Models such as the DocumentPages and identifying the correct images to display per request.

Felix He
I developed the front end application with Angular 2+ (in particular using the agora module). I developed the APIs for the project.

Rolf Li
I developed the backend API, converting between files types, image snapshots, uploading PDFs to the endpoints to Azure, and splitting them into images to use. I setup Azure Storages and deployed the API to Heroku.

Richard Chen
I worked on Microsoft Azure OCR and sklearn (data processing and handling errors). I also helped with some styling tasks.

Dennis Bae