Accessible Gaming for Moebius Syndrome

Inspiration

Our inspiration for this project came from the desire to make gaming more inclusive and accessible for individuals with Moebius Syndrome. Moebius Syndrome is a rare neurological condition that affects facial muscles and eye movement, making it difficult for individuals to scan screens effectively. We wanted to leverage technology to create a solution that would allow these individuals to enjoy gaming without the physical strain of moving their heads constantly.

What it does

Our solution provides a circular focus window that is centered on the screen, highlighting areas of interest and changes in real-time. Additionally, it includes a secondary view centered on the mouse cursor. The focus window uses AI and computer vision to detect changes between frames and displays arrows pointing towards these changes. It also shows the areas of interest around the edges of the focus window, making it easier for individuals with Moebius Syndrome to see changes without moving their heads.

How we built it

We built this solution using C++ and the OpenCV library for computer vision. The key components include: Screen Capture: Capturing the entire screen in real-time using the Windows API. Change Detection: Using frame differencing to detect changes between consecutive frames. Focus Window: Creating a circular focus window centered on the screen and applying a circular mask. Mouse View: Adding a secondary circular view centered on the mouse cursor. Arrows and Areas of Interest: Drawing arrows pointing towards areas of interest and displaying these areas around the edges of the focus window.

Challenges we ran into

One of the main challenges we faced was efficiently capturing and processing the entire screen in real-time. Ensuring that the focus window and mouse view were updated smoothly without significant lag was crucial. Additionally, accurately detecting changes and highlighting areas of interest required fine-tuning the parameters for frame differencing and contour detection.

Accomplishments that we're proud of

We are proud of successfully creating a real-time solution that enhances the gaming experience for individuals with Moebius Syndrome. The focus window and mouse view work seamlessly to provide a clear and accessible view of the screen. Our solution effectively reduces the physical strain on users by minimizing the need for head movement.

What we learned

Throughout this project, we learned a great deal about computer vision and real-time image processing. We gained experience in using the OpenCV library for various tasks, including screen capture, frame differencing, and contour detection. Additionally, we learned about the unique challenges faced by individuals with Moebius Syndrome and how technology can be leveraged to create more inclusive experiences.

What's next for Accessible Gaming for Moebius Syndrome

Moving forward, we plan to enhance our solution by incorporating more advanced AI techniques for better change detection and context awareness. We aim to add customizable settings to allow users to adjust the focus window and mouse view according to their preferences. Additionally, we plan to conduct user testing with individuals with Moebius Syndrome to gather feedback and further refine our solution. Our ultimate goal is to make this technology widely available and easily accessible to all gamers with physical disabilities.

Built With

languages:-c++-frameworks/libraries:-opencv
tdm-gcc
windows-api-platforms:-windows-development-environment:-visual-studio-code

Created by

I researched the tech stacks and libraries we might use to flesh out this idea. There were two main approaches that came to mind.

Tech stack for Approach 1:
Emulator: Dolphin-Emu or EmuHawk
Language: Python, C++.
PyDolphinMemoryEngine
DolphinMemoryEngine
Use Electron App to configure the settings of the mod.
OpenGL + GLFW: You can use these libraries to create an in-window magnifier that highlights specific regions of the emulator’s display and zooms in on them.
Pygame (Alternative): If you want a Python-only solution, Pygame can be used to build a magnifier that dynamically zooms and pans on certain sections of the emulator window.

Tech Stack for Approach 2:
Use OpenGL/GLFW for creating realtime overlays on top of emulator window. Multiplatform, which is an advantage. GLFW provides a cross-platform windowing system that can run on MacOS, Windows, and Linux, making it a flexible choice for rendering custom UI elements like indicators, focus aids, and character highlights.
Computer Vision (Optional): If you want to track in-game characters and objects for highlighting or zooming without direct memory reading, you can use computer vision techniques. Libraries like OpenCV could help you track movement and detect patterns visually in real-time, especially for creating focal points based on on-screen action.
Create a custom magnifying overlay: Using OpenGL or Python libraries like PyAutoGUI to display a zoomed-in portion of the game screen based on the emulator’s memory data or visual input.
Leverage OpenCV: For dynamic magnification based on in-game actions. For example, when a player's character is near danger, the system could automatically zoom into the danger zone to assist with focus.

Advanced but Applicable to Both:
SDL2 (Simple DirectMedia Layer): SDL2 is a cross-platform development library that provides low-level access to input devices (keyboard, mouse, and controllers). You can use SDL2 to handle haptic feedback on supported game controllers or trigger visual/audio cues when specific actions occur in the game.
HIDAPI: For more advanced controller integration, HIDAPI is a platform-agnostic library that allows you to interact with USB and Bluetooth devices (like game controllers). It can be used to send haptic feedback to controllers based on game events.
Vibration via SDL2: SDL2 has built-in support for force feedback (haptics) on many controllers. It’s cross-platform and could provide a simple interface for triggering vibrations when in-game events (like being hit or proximity warnings) happen.

isotropo Stroubos
Ioannis M
Private user

Updates

Ioannis M started this project — Oct 15, 2024 09:56 PM EDT

Leave feedback in the comments!

Log in or sign up for Devpost to join the conversation.