Ora

Ora turns visual websites into conversations navigate, explore, and take action through natural voice commands.

Comment

Inspiration

I wanted to explore a simple question: What if you could talk to a website and watch it respond?

Browsing usually means translating what you want into clicks, menus, filters, and forms. A chatbot can answer questions, but the visitor often still has to find the relevant content and perform the next action manually.

Ora connects conversation to the website’s visual surface and supported actions, letting visitors explore an experience by expressing what they want.

What it does

Ora is a conversational interaction layer that lets people navigate, explore, and take action through natural voice commands.

A spoken request can move the view, focus on a space, change the atmosphere, start a guided tour, or prepare a structured application request.

The current implementation demonstrates this through Aurelia Sanctuary, a concept architectural property environment. Visitors can:

  • Ask to see the living room at sunset.
  • Navigate to the master bedroom or infinity pool.
  • Request a narrated tour.
  • Ask questions about the property.
  • Provide stay details to prepare a reservation request and estimated total.
  • Open a prefilled request in WhatsApp.

Aurelia demonstrates Ora’s broader interaction model. A product adapter interface provides a boundary for connecting other applications, although additional domains have not yet been implemented.

Ora controls actions explicitly supported by its host application; it is not an unrestricted assistant for arbitrary websites.

How I built it

I built Ora with React, TypeScript, and Vite. The repository includes the Devpost Learn Skill Pack and planning documents covering scope, product requirements, technical specification, and implementation tasks.

The system connects several parts:

  • Voice input: Microphone audio streams to AssemblyAI for transcription.
  • Intent interpretation: A deterministic router handles common commands. Other questions use the configured conversational provider: local Ollama or external OpenRouter.
  • Typed actions: Interpreted requests become structured navigation, atmosphere, tour, or request-preparation actions.
  • Product adapter: A TypeScript interface connects those actions to Aurelia’s application behavior.
  • Visual experience: A 300-frame canvas sequence provides visual navigation, with day, sunset, and night environments and additional detail views.
  • Audio: WebAudio manages the ambient soundscape and volume ducking. Browser speech synthesis provides spoken responses.
  • Request preparation: Application code extracts guest details, validates calendar dates, and calculates estimated totals.

The current default Ollama model is Qwen qwen2.5:1.5b, with model selection configurable.

Aurelia’s $1,850 nightly amount is a configured demonstration rate. The application calculates an estimate from that rate; it does not verify real property pricing or availability.

The WhatsApp action opens a prepared message. It does not automatically send the message, collect payment, or confirm a booking.

Challenges I faced

Coordinating conversation and application state

Understanding a request was only the beginning. Ora also needed to connect it to the correct view, atmosphere, spoken response, and application state.

Separating interpretation from execution made those actions explicit and easier to validate.

Managing listening, speech, and interruption

Background sound, microphone input, spoken responses, and tour narration share the same experience.

During review, I found that pausing microphone forwarding while Ora spoke prevented reliable interruption. The implementation was changed to keep continuous listening active during playback, cancel speech and tours when interrupted, and discard older asynchronous responses when a newer request supersedes them.

Live microphone behavior still depends on browser settings, audio hardware, and the surrounding environment.

Handling incomplete and corrected requests

Visitors do not always provide every detail in one sentence. Reservation preparation needed to support separate name and date replies, corrections, and cancellation.

Review also exposed date parsing errors and JavaScript’s normalization of impossible calendar dates. Focused fixes and regression tests were added.

Keeping the demonstration honest

Ora needed to distinguish between preparing a request and completing an external transaction.

Pricing arithmetic stays in application code. The pass shows an estimated total, and final availability and confirmation remain outside this proof of concept.

What I learned

Conversation becomes more useful when understanding leads to visible, meaningful action.

Responsiveness depends on the entire interaction loop: transcription, interpretation, visual changes, speech, and state management. A quick command router alone does not establish end-to-end voice performance.

I also learned that passing automated tests does not prove every live interaction. Microphone behavior, model responses, and external handoffs require separate checks in the browser.

Defining a clear scope helped keep the implementation focused while leaving room for future integrations.

Accomplishments I am proud of

  • Connected voice commands to visual navigation and atmosphere changes.
  • Built a guided exploration experience with narration and coordinated sound.
  • Implemented continuous-listening and interruption handling.
  • Connected conversation to structured reservation-request preparation.
  • Added validation, correction handling, and cancellation isolation.
  • Established a product adapter interface for future integrations.
  • Verified 272 passing automated tests across 63 suites and a successful production build.

Current boundaries

Aurelia is a concept reference environment, not a verified live property listing.

Ora currently has one product adapter. Live voice requires AssemblyAI credentials and internet access. Ollama inference runs locally when selected; OpenRouter sends conversational requests to an external provider.

The reservation flow prepares an unconfirmed request using a demonstration rate. It does not check live availability, take payment, or guarantee WhatsApp delivery.

What’s next for Ora

I want to test the experience with real users, strengthen voice reliability across devices, and improve clarification and correction flows.

A second product adapter for a showroom, configurator, or interactive catalog would help evaluate how well the architecture transfers beyond Aurelia.

The goal is to make supported website actions accessible through conversation, with clear feedback about what the system has actually done.

How did you use the Devpost Learn Skill Pack?”:

I used the Devpost Learn Skill Pack workflow to structure Ora’s scope, product requirements, technical specification, and implementation checklist. The repository includes the Skill Pack and the corresponding documents in the devpost folder. These documents define the interaction model, Aurelia reference environment, supported actions, architecture, and proof-of-concept boundaries. I also reconciled them with the final implementation during submission review.

Ora’s current setup runs locally so its server can connect directly to an installed Ollama model. The default is Qwen qwen2.5:1.5b, and the model is configurable. OpenRouter is also supported as an alternative.

Local Ollama inference avoids LLM API token charges, but Ora is not entirely offline or entirely local: microphone audio streams to AssemblyAI, which requires credentials and internet access and may incur service charges.

A cloud deployment would need server-side API hosting and a reachable inference provider. The repository provides local setup instructions, and the demonstration video shows the implemented experience.

Built With

Share this project:

Updates

Submission history