• 

Ailly - Hackathon Submission

Inspiration

We see many people with vision problems and accessibility needs who struggle to use internet and computers. Many websites are not accessible, and people with low vision, blindness, or dyslexia face difficulties everyday. Also, privacy is big concern - most accessibility tools send user data to cloud servers. We want to build something that help everyone access web content privately, using their own computer's GPU power, no internet needed. This idea inspired us to create Ailly.

What it does

Ailly is Universal On-Device Vision and Accessibility Layer. It works like smart assistant for web browsing but runs 100% on your device. Main features include:

• Real-time OCR - Extract text from any content on screen with multi-language support
• Object Detection - Identify and highlight important objects on webpage
  
• Depth Analysis - Separate foreground and background for better focus
• Pose Detection - Track human movement in real-time with skeleton overlay
• Super Resolution - Make images sharper and clearer using AI
• Web Overlay Engine - Add accessibility features on top of any website

Everything runs in browser using WebGPU for fast performance. Works as Chrome extension or web app. Best part - all AI processing happens on your device, so your data stays private and secure. Can work completely offline!

How we built it

We built Ailly using modern web technologies:

• WebGPU - Main runtime for GPU-accelerated machine learning inference and rendering
• TypeScript + Vite - For clean, type-safe code and fast development
• TensorFlow.js - On-device neural networks
• MediaPipe Tasks - For robust object detection and vision tasks
• ONNX Runtime Web - For specialized models like PaddleOCR
• Skeleton Crew Runtime - Plugin-based architecture for modularity

We create plugin system where different features (Capture, Vision, Reasoning, UI) work as independent modules. Used WGSL shaders for high-performance image processing. Build smart fallback system - if WebGPU not available, automatically switch to WebAssembly (WASM) so app works on older devices too.

For UI, we design premium landing page with glassmorphism effects, smooth animations, and beautiful gradient colors. Make it fully responsive and accessibility-first.

Challenges we ran into

Browser compatibility - WebGPU is very new technology, not all browsers support it yet. We solved by implementing robust WASM fallbacks so app works everywhere.

Model size and performance - ML models are big and slow. We spend lot of time optimizing, using quantized INT8 models, and creating shader-based upscaling that much faster than neural networks.

Runtime crashes - Had big problem with fast-glob dependency causing browser crashes because it try to access Node.js modules. Fixed by lazy-loading and creating browser-specific entry points.

Privacy architecture - Making sure NO data leaves device was challenging. We implement strict opt-in system for any cloud features (like Gemini reasoning) with clear UI indicators.

Plugin system integration - Refactoring whole app to use Skeleton Crew Runtime was complex. Many TypeScript errors, build issues, and need to redesign how components communicate.

Offline functionality - Ensuring app works in "Airplane Mode" required careful caching strategy using IndexedDB for models and service workers for assets.

Accomplishments that we're proud of

✅ True privacy-first design - All core vision processing runs on-device. No internet required!

✅ Desktop-class performance in browser - Achieved 30-50ms latency for full vision pipeline using WebGPU

✅ Beautiful, accessible UI - Premium design with smooth animations and WCAG compliance

✅ Robust architecture - Plugin-based system that is modular and maintainable

✅ Universal accessibility - Works for people with low vision, blindness, dyslexia, motor impairments

✅ Smart fallbacks - App gracefully degrades on older devices, nobody left behind

✅ Comprehensive testing - Unit tests, E2E tests with Playwright, verified on multiple browsers

✅ Complete PWA - Installable app that works offline, caches up to 32MB of models

What we learned

• WebGPU is future - We learn how powerful WebGPU compute shaders are for ML inference. Game changer for browser-based AI!

• Privacy and performance can coexist - You don't need cloud to have smart features. On-device AI is mature enough now.

• Accessibility is hard but important - Building true accessibility requires thinking about many different needs. Screen readers, keyboard navigation, color contrast all matter.

• Plugin architecture pays off - Starting with modular design make debugging and testing much easier.

• Fallback strategies are critical - Never assume cutting-edge features available. Always have backup plan.

• TypeScript saves time - Type safety caught many bugs early in development.

• Performance optimization is iterative - Used many profiling tools, Intersection Observer for lazy loading, hardware-accelerated animations.

What's next for Ailly

Short term:

• Add more language support for OCR (currently focused on English and Chinese)
• Implement depth estimation and pose detection (currently planned but not fully integrated)
• Improve accessibility bridge for better screen reader compatibility
• Package for Chrome Web Store for easy distribution

Medium term:

• Add neural super-resolution (Real-ESRGAN) for high-end GPUs
• Build browser-native audio description and sonification features
• Create JavaScript SDK so other developers can use Ailly features in their apps
• Add adaptive quality profiles based on battery level and device capability

Long term:

• Support for more browsers (Firefox, Safari when they get WebGPU)
• White-label PWA for education and healthcare institutions
• B2B SDK licensing for enterprises
• Mobile app version using same on-device AI approach
• Community-driven model repository for specialized use cases

Our vision is make web truly accessible for everyone while respecting their privacy! 🌟

---

Built with ❤️ for a more accessible web

Built With

• antigravity