Hephy (Hephaestus)

Inspiration

Modern digital learning platforms assume that users can click, scroll, and interact effortlessly. For students with motor disabilities, this creates an invisible “interaction tax” — where accessing knowledge becomes physically exhausting rather than intellectually engaging.

We set out to rethink what accessibility could look like if interaction itself was reimagined. Instead of adapting users to interfaces, we asked: What if interfaces adapted to users?

Hephaestus (Hephy) was born as an AI-powered “Learning Sidekick” — a system that removes friction from learning and transforms passive content into an active, accessible experience.

The name Hephaestus comes from the Greek god of craftsmanship, who, despite physical limitations, forged powerful tools for the gods. In the same spirit, our platform is a modern learning forge — empowering users to learn, create, and excel without being limited by traditional interfaces.

What it does

Hephy is a hands-free Chrome extension that turns any webpage into an interactive learning environment using computer vision and AI.

Using a quick nod gesture, users summon a radial “Orbital Menu” at their gaze location. By simply looking at an icon and opening their mouth, they can activate:

Summary – Converts dense content into concise, Socratic-style explanations
Flashcards – Extracts key concepts for active recall
Quizzes – Generates instant comprehension checks
Video – Scripts visual simulations of complex ideas
Podcast – Transforms content into conversational audio
Lesson Plan (Mastery Path) – Builds a structured, step-by-step study workflow

This creates a seamless, gesture-driven experience that reduces both physical and cognitive load.

How we built it

Hephy enables real-time, hands-free interaction directly in the browser through a combination of computer vision and AI.

Computer Vision Layer
Real-time facial tracking enables gesture recognition (mouth-open trigger) and gaze-based interaction.
Chrome Extension Interface
A lightweight injected UI renders a dynamic radial “Orbital Menu” and extracts context from the current webpage.
AI Layer (Claude-powered)
Claude powers content transformation (summaries, flashcards, quizzes) and generates multimodal outputs (video scripts, podcast scripts, structured lesson plans).

Challenges we ran into

Designing a gesture trigger that feels intentional but avoids false positives
Mapping gaze to screen position accurately in a browser environment
Extracting the right content context without requiring manual selection
Balancing latency vs. quality for real-time AI responses
Creating a UI that is powerful yet cognitively lightweight

Accomplishments that we're proud of

Building a fully hands-free learning interface for arbitrary webpages
Designing an intuitive gaze-driven radial menu system
Integrating computer vision + AI workflows into one seamless experience
Delivering multi-modal learning (visual, auditory, active recall) in one tool
Creating a product with strong real-world accessibility impact

What we learned

Accessibility isn’t an add-on — it fundamentally reshapes system design
Gesture-based interfaces require extremely thoughtful UX tuning
AI is most powerful when paired with context-aware interaction
Privacy-first design improves both trust and usability
The key challenge isn’t generating content — it’s delivering it at the right moment, in the right form