Accessibility4all

Inspiration

We know from experience what you've experienced a family member of a contributor is autistic. And we've seen firsthand how difficult it is to locate accessible features that will provide consistent results. Most tools only fix one problem for one group and then they stop. We sought to create open-source software that could be used as an all-in-one tool for improving access to technology across a wide range of technologies. The people we develop for do not fit into a single category. Some have visual impairments due to being colorblind, some have visual impairments due to having limited vision, some have physical disabilities related to their ability to move (e.g. motor control), and some are neurodivergent (including autistic users, users with Attention Deficit Hyperactivity Disorder (ADHD), and others that struggle with executive function or understand implied meaning through writing). We wanted to design a single program that would allow us to support all of these groups, with a clearly defined pathway for the community to identify other areas where we may need additional functionality.

What it does

Accessibility4all is a modular hub of accessibility features that you can toggle on and off: -Voice control lets you operate a website by speaking. It converts speech to text, screenshots the page, and asks a Groq vision model where to click. -Page reader helps low-vision users by tracking where the mouse is and summarizing the main content of the screen with a Groq vision model, instead of reading every unimportant word aloud. -Tone and social cues helps autistic users and anyone who struggles with implied meaning. Select text, press a hotkey, and a panel breaks down tone, sarcasm, and indirect requests. -Color blind filter applies system-wide color correction (Deuteranopia, Protanopia, Tritanopia, Grayscale, Invert) across every app, not just one browser tab. -Cursor size scales the mouse pointer system-wide for anyone who loses track of a small cursor. -Focus mode blocks distracting sites system-wide for a timed work session, which helps users with ADHD or anyone fighting distraction.

How we built it

We built the project structure to keep each feature separate from the others. This makes it simple to add features later. We also created a template and a development guide so that anyone joining the project can understand how to write their code

The AI features all use the Groq vision pipeline. For voice control the system takes a screenshot of the page. Asks the model to find the right click coordinates. We switched to this method after our first plan—reading element names directly from the page—turned out to be unreliable. The page reader works similarly: it takes screenshots. Then summarizes the content using Python. Tone and social cues rely on designed prompts to stay within safe and effective boundaries.

The system-level features go deeper interacting directly with the operating system. The color blind filter uses the Windows registry and the Win+Ctrl+C shortcut to enable the systems built-in color filter. If that doesn’t work it falls back to a script for macOS. The cursor size feature writes to the registry. Uses the SetSystemCursor function to actually change the size of the pointer, on screen. Focus mode works by redirecting blocked domains to 127.0.0.1 in the hosts file. We wrap these entries with marker comments so we can easily add or remove them later without breaking anything.

Challenges we ran into

The hardest problems all came down to the gap between what an API returns and what actually happens on the screen or in the operating system.

For voice control reading element names from a page didn’t work well. The names were often different from what we expected. That pushed us toward using screenshots and coordinates instead.

When it came to tone and social cues large language models had a tendency to confidently make up subtext that wasn’t really there. That’s dangerous for someone who depends on the tool for help. So we limited the model to be more cautious. It now uses hedging language quotes the words behind each cue and gives an empty result when the text is straightforward and literal. That way "nothing here" became an acceptable response.

With the color filter directly writing to the registry changed what Windows would use later but it didn’t repaint the screen. This created a desync. One tester ended up stuck on a grey screen because every toggle checked a stored value that no matched reality. The fix was simple: stop writing the "flag ourselves and instead trigger the real Windows shortcut. Then we wait, up to 1.5 seconds to confirm the toggle actually took effect.

For cursor size updating CursorBaseSize and calling SPI_SETCURSORS. The same thing Settings does. Didn’t change the pointer size on screen. Windows only resizes the cursor if the images are reloaded. So we had to find a way to actually reload those images.

In focus mode editing the hosts file requires administrator privileges.. Asking for a password every time you toggle the feature? That adds friction. Much friction makes an accessibility tool hard to use and users end up not using it at all.

Accomplishments that we're proud of

We delivered six functioning accessibility features in a modular program. These features cover motor and cognitive needs instead of focusing on one single group. Some of them go beyond what typical tools provide. For example the page reader gives a summary of a screen of reading every word and the tone reader is intentionally made to say nothing instead of giving misleading information. The important part is that we tackled system‑level features honestly. We made them work by controlling the operating system so they function in every application than, by creating a fake answer that only looks correct.

What we learned

Our largest takeaway has been to confirm your findings in real-world applications as opposed to relying upon information provided by calls. In relation to the color-blind feature, we realized that while the program stores the color blind option, this option does not necessarily match up with the current display; therefore, we decided to make the Operating System (OS) the sole provider of data, and we verified that every change was reflected on the system. As part of the cursor-size feature, we discovered that simply writing the value into the Registry was insufficient alone, since the cursor only changed when the program re-loaded the images. For the focus-mode feature, we found out that we needed to manually review the HOSTS file immediately after toggling to ensure that no previous toggles had resulted in our settings showing an active session with no content blocked. With respect to developing AI models, we learned that providing a useful model and a reliable model are two different things, and that restricting a model to indicate when it cannot provide any relevant response is far more important than forcing the model to appear confident. To address the administration prompt issue with focus mode, we created a one-time scheduled task on Windows to configure the focus mode, which resulted in setup requiring only a single administration prompt, and each subsequent toggle thereafter did not require any further prompts.

What's next for Accessibility4all

The entire project is made to grow. The next step is to open it to contributors and to add features using the template and guide that we have built. We want to fill the platform gaps that we know exist. Some system level features are along on Windows than, on macOS. We also want to tag the speech engine and to make voice control run more reliably. We plan to add features for needs that we have not covered yet. Because everything is open source the long‑term goal is a hub that the community will keep. In that hub any accessibility need that someone has can become a feature that others can use.

Built With

• applescript
• computer-vision
• groq
• html
• javascript
• macos
• osascript
• pyautogui
• pynput
• python
• pywebview
• speech-to-text
• tkinter
• win32-api
• windows
• windows-registry