AltBridge

Inspiration

I noticed a frustrating trend on social media (such as X/Twitter) where users often write unrelated context, memes, or completely generic placeholders inside image alt attributes. This completely defeats the true purpose of web accessibility, leaving visually impaired users disconnected from the actual rich content of the web. I wanted to build a bridge between missing descriptions and visual reality.

What it does

AltBridge is a local-AI-assisted web accessibility extension. Without modifying the page's original HTML or DOM, it classifies images on a page (Missing alt, Empty alt, Possibly weak alt, or Alt present) and generates descriptive, context-aware captions on-demand using a locally hosted Vision-Language Model.

How we built it

• Frontend: Built with TypeScript, React, and Vite as a modern Chrome Extension.
• Local Proxy: Engineered a custom Node.js/Express backend proxy (ollama-proxy) from scratch that interfaces with local Ollama instances, with an optional LAN mode so the proxy can run on a separate machine with stronger hardware.
• Strict Architecture: Standardized binary image data streaming using a custom-built, dependency-free buffer utility for multipart/form-data parsing, ensuring zero heavy external packages on the server-side.
• Prompt Engineering with GPT-5.6: Worked with GPT-5.6 to run a threat model review of the LAN server mode, and to design strict prompt guidelines that eliminated conversational bloat from lightweight vision models, keeping captions concise and on-topic.

Challenges I ran into

• Full Internationalization: Transitioning the codebase from a Japanese-only MVP to a fully internationalized system required a large refactor of regular expressions, UI layouts, and core storage structures, so English and Japanese users get a consistent, native-feeling experience.
• Handling Model Quirks: I benchmarked several local vision models (including llava:7b, llava-phi3, and gemma3 at multiple sizes) and found real differences in stability and hallucination rates. Getting raw stream boundaries and JSON responses to behave consistently across models required building robust proxy error and confidence-handling logic to keep the UI stable regardless of which model answered.
• Trustworthy Confidence Scores: A single fixed confidence value isn't useful, so I built a heuristic layer that lowers confidence when a response is suspiciously short, repetitive, or contains uncertainty language, and raises it when the response includes concrete visual detail — giving users an honest signal about how much to trust each caption.

Accomplishments that I'm proud of

• True Accessibility Mindset: I built a zero-DOM-pollution architecture. The extension respects existing content and only provides supplemental overlays, avoiding any invasive document modifications.
• LAN-Distributed Inference: The client extension can offload model processing to a dedicated server elsewhere on the local network instead of requiring a powerful local machine, so the thin client stays light while inference runs on whatever hardware is available.

What I learned

I realized the massive value of decoupling the extension interface from the inference backend. By utilizing a local proxy layer, I learned that a true local-first setup doesn't mean squeezing massive models onto a thin client laptop—it's about building light, agnostic extensions that can route seamlessly to powerful hardware residing anywhere on your local network (LAN).

What's next for AltBridge

• Additional Locales: Expand the dictionary structures beyond the initial English and Japanese pre-sets.
• Provider-Native Confidence: Explore surfacing model-native confidence signals (where a backend exposes them) alongside the current heuristic estimate.

Built With

• chrome
• express.js
• javascript
• node.js
• ollama
• react
• typescript
• vite