ModifAI

Inspiration

A friend of Anagha’s, a brilliant automotive engineer, was sharing his experiences of struggling to read heat graphs due to the color gradient. As someone who is colorblind, he was often unable to interpret data solely because of its presentation. This led our team to think about all the ways the web discriminated against those of different abilities. ModifAI is built to change that.

What it does

Modif.AI is revolutionizing web accessibility by using real-time AI-driven code generation to modify websites dynamically, ensuring that users get an experience tailored to their individual needs. Instead of relying on static accessibility settings or waiting for developers to build accessibility features into websites, Modif.AI allows users to instantly adjust websites on their own terms—just by making a request.

At its core, Modif.AI learns and adapts over time. By analyzing the changes users request—such as increasing font sizes, enabling high-contrast themes, or restructuring page elements—it refines its ability to predict and automate future accessibility needs. This means that as more people use the platform, Modif.AI becomes smarter, automatically applying accessibility optimizations based on past interactions. If a user frequently changes a particular website layout or color scheme, the system remembers and applies similar modifications the next time they visit a comparable site, eliminating the need for constant adjustments.

Key Features of Modif.AI -Instant Accessibility Adjustments: Users can request larger text, high-contrast themes, text-to-speech functionality, or hands-free navigation in real-time, and Modif.AI applies the modifications instantly. -Dynamic Code Generation: Instead of relying on preset accessibility tools, Modif.AI analyzes website structures and generates custom code on the fly, ensuring that even poorly designed or inaccessible websites can be transformed. -Hands-Free Navigation: Users with motor impairments or mobility challenges can navigate websites using voice commands, allowing them to click buttons, open links, and interact with elements without needing a mouse or keyboard. -Personalized Learning & Automation: Modif.AI remembers user preferences and applies similar accessibility modifications to related websites, ensuring a consistent and adaptive experience. -Continuous Improvement: As more users interact with the platform, it learns which modifications are most effective, improving accessibility solutions for all users over time.

By combining real-time AI-powered code generation, predictive learning, and adaptive automation, Modif.AI is breaking down barriers and making the internet more accessible for everyone—one modification at a time.

How we built it

We built Modif.AI using a combination of agentic AI, real-time code generation, and automated accessibility tools to create a seamless browsing experience for users with diverse needs. To handle dynamic website modifications, we integrated Mistral’s Codestral model for rapid code generation and Scrapybara to extract and analyze webpage structures. Scrapybara also allows the extension to remotely start an agent instance, enabling users to interact with otherwise inaccessible website elements by automating actions like clicking links or buttons via voice commands—a crucial feature for users with motor impairments.

Our agentic logic is powered by Dain, allowing users to request accessibility changes naturally, while Eleven Labs' text-to-speech API provides high-quality auditory support.

To enhance personalization, we implemented a vector database on InterSystems, which enables Modif.AI to store and retrieve user preferences efficiently. This database plays a key role in recognizing similarities between websites by storing structural embeddings of visited pages and comparing them using tensor-based similarity analysis. When a user visits a new website, the system checks for similarity to previously modified sites and automatically applies the same accessibility preferences. This ensures that users don’t need to reconfigure their settings for every page, creating a smarter, adaptive browsing experience.

Additionally, we leveraged Codium’s Windsurf Browser to aid in development and testing, allowing us to iterate quickly and validate our AI-generated modifications across different web environments. Windsurf’s flexibility helped us fine-tune real-time code execution and ensure that our AI-driven accessibility adjustments worked seamlessly across various sites.

Challenges we ran into

The biggest challenge was to navigate Chrome’s security policies when deploying the extension. The security policies make it difficult to edit code in real time. We eventually used Electron that allowed us to bypass these security constraints.

Another major challenge was to evaluate the cosine similarity between different websites to determine whether it is similar to a previously visited website. Standard metrics were unclear due to variations in the website’s HTML. This was solved using a tensor-based approach, deploying deep-learning feature extraction to determine the website similarity.

Accomplishments that we're proud of

We are incredibly proud of Modif.AI’s speed and effectiveness in modifying websites in real time. With Mistral’s near-instant inference, we can generate and apply code changes almost immediately, providing a seamless, frustration-free user experience.

A key accomplishment is our innovative use of AI-driven code generation, allowing Modif.AI to dynamically adapt websites instead of relying on static accessibility settings. This makes it far more flexible across different online environments.

We also successfully integrated AI agent tools like Scrapybara and Dain, enabling automated interactions such as clicking links and buttons for users with motor impairments. Additionally, Eleven Labs’ text-to-speech enhances accessibility for those with visual impairments, dyslexia, or reading difficulties.

By combining real-time AI modifications, automated navigation, and intelligent accessibility enhancements, Modif.AI allows for a new era of web accessibility.

What we learned

We had to learn several skills along the way, such as vector databases, building AI agents, and working with Electron. Aside from learning how to build a project from scratch, our biggest learning was how to design with empathy and consider the consequences of what we build for the world.

What's next for ModifAI

We want to prepare for public launch, with initial beta testing. Additionally, we want to expand the range of features we have such as introducing voice-controlled browsing, auto filling forms, and integrating sign language for audio content. We also want to bring ModifAI to application and the phone so it is not just limited as a Chrome extension.