Insight AI

• 

  Landing Page

Inspiration

Have you ever been stuck when reading because you don't understand a word or a formula? You might be a 6 year old reading an english book or a college student reading a 500 page physics paper. You need to read through that while taking notes, solving questions that you might have, Copy and pasting to ChatGPT just to get your answer explained vaguely. With our app, you can do all of that with a simple gesture, and it can automatically detects if you are stuck on one thing using eye tracking.

What it does

InsightAI is a visionary AI assistant focused on education. By performing eye-tracking to find where and what the user is reading, our product can offer real-time insights. Moreover, we integrated in gesture control so the AI can complete tasks for the user. Gesture triggered tasks include taking a screenshot, flipping to the next page, flipping to the previous page, asking a question, and writing notes. Moreover, our AI can analyze the emotions of the user and offer responses accordingly with the goal of instilling confidence in our users. Overall, InsightAI offers a seamless reading experience!

How we built it

InsightAI was built on using ReactJS on the frontend and Python on the backend. We used an eye tracking called Eyegazer, to get initial estimates of where the eyes are focused on the screen. Then, we apply a Bayesian prior to bias it towards selecting words that appear after the previously seen word. We then combine it with a hand gesture detection using by applying Transfer Learning to find the 21 hand landmarks, then we check what pose we are doing based on the position of landmarks. Then we implemented a RAG pipeline system connected to an LLM that uses HUME AI for sentiment analysis and have 2 prompts that take emotion, eye tracking, and hand gesture into consideration.

Challenges we ran into

• Connecting backend and frontend :(
• Using 1 camera for 2 processes (eye-tracking and gesture-tracking)
• Creating dynamic bounding boxes for each word
• Sleep deprivation
• peer dependency issues -> ALOT of time spent debugging

Accomplishments that we're proud of

• Drawing correspondence between the eye movement and word
• Eye-tracking and gesture-tracking

What we learned

What's next for Insight AI

-In the future, we hope to also accommodate for images along with our pdfs -We also hope to have AR integration

Built With

• javascript
• python
• react
• tailwindcss
• typescript