Percepta

Inspiration

Millions of adults in the United States experience visual hallucinations, and yet there are little to no devices that offer hallucination identification or reality-checks outside of psychiatric service dogs. In a digital world where the line between reality and illusion is already blurred, we opted to create a solution that provides reliable hallucination confirmation through an interactice AI model along with an audible signal.

What it does

Percepta is an interactive computer-vision program that activates upon user input, in the form of a small button for our prototype, and will audibly signal to the user whether there is a person in the camera-view. Our device has 3 main features:

Multimodal sensing that can detect common visual hallucinations (e.g.: geometric shapes, people, faces, etc.) through the utilization of Ultralytics YoloAI.
Hardware in the form of a small button and speaker system allowing for user interactivity and for clear communication. Using OpenAI TTS to clearly verbalize binary responses determining presence of people.
Confidence in person detection accounting for movement and environmental changes.

Additional Features

Providing non-binary responses in the event of poor detection and/or camera obstruction
User-initiated confirmation to provide user agency and prevent biomedical dependency
Alternative nonverbal cue options including: light sensors or vibrations for more discreet forms of communication
Avoidance of generative responses to provide objective observations and promote patient reassurance
Record of time both within device and in private databases to aid in pattern recognition and clinical observation
Neutral regrounding prompt to establish camera, and by extension, user orientation

How we built it

Our solution incorporates various versatile tools to create a reliable and user-friendly device:

Frontend: Our UI utilizes a button connected to a raspberry pi for a small and portable design, allowing users to carry our device with them wherever they go.
Backend: We used a combination between CV2 and Ultralytics YoloAI for human-detecting computer-vision that serves the main role in hallucination identification.
AI Processing: We used the YOLO8 for real-time video analysis and object identification allowing for immediate hallucination confirmation.
Real-time Processing: Through OpenCV we facilitate real-time performance so Percepta can work immediately regardless of busy environments.

Challenges we ran into

AI Model Accuracy: Training the computer vision to differentiate between humans and human-like objects including but not limited to graphics and dolls.
Interactive Accessibility: Implementing a way for individuals to easily activate the device and receive a clear communication in return.
Clear Communication: Ensuring that the machine learning model can clearly communicate what is truly present even in the presense of multiple objects.

Accomplishments that we're proud of

Developed a device that integrates an AI detection model in less than 24 hours
Implemented user-friendly hardware for accessible detection
Built hardware that is adaptable to various camera models
Utilization of a pre-trained multimodal AI model that detects common hallucinations (ie: shapes, people, etc.)

What we learned

Integration of AI models into hardware
AI model optimization for edge cases like humanoid objects
Utilization of multiple AI/ML models into a single product
Implementation of clear communication that accounts for other physical disabilities
AI modal runtime based on standby power for energy conservation

What's next for Percepta

Enhancements and certifications we plan on attaining:

1. Pre-detection Features

Eye pattern analysis
Automatic operation based on altered saccades
Implementing non-invasive electroencephalography for pre-detection based on brain activity