Neuro-Assist

EEE data acqusation and Visulization using spike recorder in real time for brain states:1.Concetration and 2.Relaxation
EEG data acqusation and Visulization in real time for brain states:1.Decision Making and 2.Relaxation
EEG data acqusation and Visulization in real time for brain states:1.Decision Making and 2.Relaxation
eye-gazing data acqusation in real time
Result Generated by the Eye-gazing module showing the focus of intrest using heatmaps
Predicting set of questionries that might have arised in the mind of the user in the

Inspiration

The idea for Neuro-Assist emerged from observing the immense pressure faced by professionals in high-stakes fields such as surgery, space exploration, driving, and research. These individuals must make split-second decisions where errors or delays can have significant consequences. We were inspired by the potential of combining AI, neurotechnology, and multimodal data analysis to proactively assist such professionals. The vision of creating a system capable of understanding and predicting a user's needs, without explicit input, motivated us to develop Neuro-Assist.

What it does

Neuro-Assist is an intelligent biomonitoring system designed to assist users in critical environments by:

Predicting questions or challenges arising in the user's mind.
Analyzing real-time data such as gaze patterns, speech, and brain activity to assess the user's state.
Providing immediate, tailored support using specialized large language models (LLMs) to improve decision-making and task performance.

How we built it

The project utilizes a multimodal approach, combining data from:

Eye-Gazing Module: Built using MediaPipe to track head pose and pupil movement, predicting the user's focus on a screen or environment.
Voice Analysis Module: Extracts keywords and emotions from speech using natural language processing (NLP) techniques.
EEG Analysis Module: Employs single-channel EEG data to predict brain states (concentration, relaxation) and analyze event-related potentials (ERPs) for detecting error patterns and emotional states.

These components are integrated to collect real-time data, which is processed and fed into generative AI models to anticipate user needs.

Challenges we ran into

Data Calibration: Ensuring accurate eye-gaze and EEG data calibration for different users was time-intensive and required customization.
Complexity of Multimodal Integration: Combining gaze, speech, and EEG data in real-time while maintaining system performance posed significant technical challenges.
Limited Data for EEG Analysis: Collecting and annotating high-quality EEG datasets for specific brain states and emotions was a resource-intensive task.
Real-World Generalization: Adapting the system to perform effectively beyond controlled environments, such as predicting gaze focus in open settings, was challenging.

Accomplishments that we're proud of

Successfully developed and integrated the three primary modules (eye-gazing, voice analysis, EEG analysis) to work in real-time.
Achieved initial predictions for two brain states (concentration and relaxation) using single-channel EEG data.
Built a functional prototype capable of assisting users based on personalized data.
Developed a foundation for expanding Neuro-Assist into diverse professional applications.

What we learned

The importance of multimodal data fusion to provide a comprehensive understanding of the user's state.
Techniques for fine-tuning models for specific physiological signals like EEG and gaze tracking.
The critical role of user-friendly calibration processes to enhance system usability.
Insights into the potential of neurotechnology to bridge human-machine interactions effectively.

What's next for Neuro-Assist

Enhanced Eye-Gazing Module: Develop smart glasses with external and internal cameras to predict focus points in open environments.
Advanced EEG Module: Expand brain state detection to include decision-making, stress, and error detection, along with improved ERP analysis.
Generalized Calibration: Automate calibration processes for eye-gazing and EEG to improve system scalability across diverse users.
Application-Specific Models: Train LLMs tailored to specific professional domains, such as surgery, space exploration, and emergency response.
Field Testing: Conduct real-world testing in high-pressure environments to refine and validate the system's effectiveness.