Dora

Inspiration

Over 2 billion people worldwide live with some form of disability. I learned this truth about a year and a half ago, when I was tasked with making our application WCAG AA compliant. I had to dig into the intricacies of web accessibility and realized how complex it is.

I thought this was a great opportunity to not only learn about some new cool Amazon technology, but to make something actually useful.

What it does

Dora is a hands-free web navigation assistant that lets users control any website using natural spoken commands. You talk, Dora takes action.

Say "Open my inbox and read my first email" — Dora navigates there and reads it to you Say "Fill in the form with my name and address" — Dora finds the fields and types for you Say "Add the first product to my cart" — Dora clicks through the page and confirms when it's done Say "Scroll" or "Go back" — Dora responds instantly

Every action you ask of Dora is spoken back to you in natural language, so you always know exactly what's happening — no screen reader required.

How I built it

Dora is built on a three-layer pipeline powered by Amazon Nova: Layer 1 — Speech I/O The browser's Web Speech API handles real-time mic transcription and TTS playback. The architecture is designed around Amazon Nova Sonic (bidirectional streaming speech-to-speech via Bedrock) for production — with a live waveform visualiser powered by the Web Audio API AnalyserNode. Layer 2 — Intent Parsing (Amazon Nova Lite) Each transcript is sent alongside the current page URL to Amazon Nova Lite (amazon.nova-lite-v1:0) via AWS Bedrock. The system prompt is context-aware — it infers the page type from the URL (Amazon product page, Google results, generic form) to improve confidence and reduce unnecessary clarification prompts. The model returns a structured JSON intent: { action, target, value, url, confidence, clarificationNeeded }. Layer 3 — Execution (Amazon Nova Act + Playwright) Actions are routed to one of two runners based on complexity. Amazon Nova Act handles interactions that require genuine AI reasoning — finding a search box across wildly different site structures, typing, submitting forms. Playwright over CDP handles deterministic actions like clicks, scrolls, and navigation where LLM reasoning isn't needed and raw reliability matters more.

Challenges we ran into

There were a few challenges worth highlighting...

1. Timeframe: I didn't sign up for the hackathon until Saturday March 14th at about 11pm. But once I had the idea I could not sleep! With more time, I could really polish this and make it a hardened tool.
2. Latency in the voice pipeline: Chaining three Nova services in sequence introduced latency. I solved this by parallelising the Nova Sonic transcription and Nova 2 Lite intent parsing where possible, and by having Nova Sonic speak an acknowledgement ("On it!") immediately while Nova Act works in the background. This actually makes the application very snappy, and the app responds very quickly once the action is complete!
3. Ambiguous commands: Natural language is messy. Teaching Dora to recognize when it doesn't understand — and ask a good clarifying question rather than taking a wrong action — was one of the more nuanced engineering challenges. I spent a solid few hours here, trying to get Dora to instead of doing the WRONG action, to clarify to the user that she doesn't understand and requires additional information.
4. Confirmation for irreversible actions: Form submissions, purchases, and deletions require an extra confirmation step. Building a lightweight confirmation flow into the voice conversation took careful prompt design. ## Accomplishments that we're proud of
5. A fully closed-loop voice experience — from spoken command to spoken confirmation
6. Graceful handling of ambiguous commands and errors, keeping the user informed at every step
7. An interface that is genuinely usable by someone with severe motor impairment — not just a demo that looks good on video ## What we learned
8. Building Dora was a soft reminder and an appreciation for just how much of the web is designed for able-bodied users, not for users with disabilities.
9. Building Dora also showed me the powerful combination of new AWS technologies, Nova Sonic + Nova Act. Voice input and UI automation together are REALLY cool and fun to work with, but also it sort of unlocks a whole new category of application that really wasn't possible before! ## What's next for Dora I have a few general feature ideas I really want to work on in the next coming weeks:
10. Personalized profiles: Store the user's name, address, and preferences so Dora can fill forms without being told each time. Imagine that, the user could just tell Dora to fill out the form for you and it's done!!!
11. Custom wake word: Activate Dora hands-free without needing to press any button at all, detecting the users voice only will be key here
12. Mobile support: Extend Dora to mobile browsers for on-the-go accessibility
13. Community command library: Let users share and download voice command sets for popular websites. This is wishful thinking, but I think I can pull it off

Built With

• act
• amazon-web-services
• bedrock
• next.js
• nova
• react
• sonic
• tailwind
• typescript