InkQuery

💭 Inspiration

InkQuery came about out of the need to better understand textual documents without taking your eyes off the page to go look up a question or pull out a dictionary, taking you away from the experience. We originally came up with the idea of command-driven annotations as a way to help note-takers stay focused on annotating while learning, without needing to type or look up extra information. LLMs have drastically changed the way we live and work, but if not harnessed smoothly, the tool can end up being more of a distraction than a benefit, especially for a note taker who prefers to keep all of their annotations in one place. InkQuery solves this.

🖥 What it does

InkQuery revolutionizes the way we approach documents.

Any webpage (extensible to any document, including powerpoints and handwritten notes), can be imported into the app, allowing for users to annotate the article and receive AI-powered context and understanding. By simply circling any piece of text and writing a keyword (listed below), the app adds summary, context, or visualizations to the article in real time.

For a website, we pull from the actual article's html and embed all of our annotations directly into the DOM!

Here are a few of the available annotations:

Context: Sometimes, articles and documents can reference events or terminology unknown to the reader. By circling the bewildering piece of text in question and writing the word “context”, instantly view a deep dive into the subject backed by trustworthy publications from top outlets.
Summarize: Tired of long, winding paragraphs in the New York Times? Shorten any block of text to a more readable, easy-to-understand piece of content and get through the article in half the time.
Visualization: More of a visual person? Turn any article into a picture book by circling a piece of text and writing the word “visualization”. A corresponding image picturing the text in question will be loaded onto the screen for maximum viewing pleasure.
Question: Any other questions? Simply circle a piece of text and write down a question, and you’ll receive an answer backed by real publications and articles.

We also enable users to specify their age group, determining the reading level of the output of our annotations.

🛠 How we built it

InkQuery is a fusion of innovative design and technology. The main components include:

Flutter SDK to create a beautiful, cross-platform application perfect for on-the-go note taking
WebSockets with REST API calls to integrate real-time communication between the frontend and backend server (Node.js), a constant stream to maintain updated HTML as annotations are added

The supporting cast includes:

ML Kit Digital Ink Detection for detecting circles and commands
ML Kit Text Recognition to detect the text within the bounded area of the circles to pass as context to - GPT prompts
OpenAI’s completions API (GPT 3.5)
Web Scraping with Cheerio to locate relevant articles that could help the user when calling the ‘context’ annotation
Bing Image Search API for finding relevant images for the visualize command
Phantom.js to create and modify the HTML as annotations are added
Tunnel for sharing development endpoints

🛑 Challenges we ran into

Challenges make up the majority of a project. Without challenges, though, the finished product wouldn’t be nearly as gratifying.

“Creating a canvas overlay on top of a webview to allow for users to write on websites was challenging—I needed to keep track of individual strokes and points written using a stylus while ensuring the user could still interact with the webpage.” - Samuel

“Determining an ideal context (web-scraper) and image finder for our different annotations proved difficult, and handling real-time HTML updates on the client side required some wrinkles to be worked out. Had an amazing time!” - Tyler

“Downloading webpages as raw HTML was surprisingly difficult. We needed to wait for any client side JS to run, inline styles, and handle unique elements differently. Also, Iended up needing a fuzzy search mechanism to take text and find the corresponding HTML tags to edit- all super fun work!” - Charan

🏆 Accomplishments that we're proud of

We were able to achieve what we sought out for and more! Many of the annotations we ended up with were products of finding cool interactions with our technology as we continued working with it. We also got to use new technologies through the process to handle the different AI/ML tasks as well as help us during production (shoutout to Tunnel!).

🧠What we learned

We had a blast working together and ultimately learned a bunch. On the technical side, we got to learn a bunch about web scraping, canvas input on Flutter, OCR, and more. We also learned a lot working together as a team like planning our work in parallel, the importance of getting an idea we were all happy with, and testing an app with a bunch of parts all at once together.

Overall, this project was a super fun cross application of networking concepts, ML, and core engineering. We learned a bunch!

✏️ What's next for InkQuery

At the moment, we support any public article on the internet to be uploaded and viewed in the app. Our ultimate goal is to extend this to anything—from handwritten notes to meeting minutes to powerpoints, users should be able to import anything and everything into the app to gain a new host of easy-to-use tools that make understanding documents a breeze.

Long term, we want to take this entire idea public and become the ultimate annotation/note-taking app. We’re the AI-powered, supercharged Goodnotes/Notability of the future, and we can see ourselves growing and scaling in the future as we release on the App Store and revolutionize note-taking.

spongebob