Duyu

Duyu is an ontology-based accessibility app that turns what you tell it into a personal ontology on your device, so it understands you better every time you use it. Built to adapt to any disability.

Comment

Inspiration

I once saw a blind man standing on a street corner, trying to find a place. He asked someone for directions. The person pointed at somewhere and walked away believing they had helped.

I could not stop thinking about it. We have AI that can describe images, translate languages, and hold conversations. Yet the people who could benefit most are often expected to adapt to tools that were never designed around how they access the world.

Most assistants also meet you as if it were the first time. They answer the immediate question, but they do not understand how your disability, language, location, routines, health context, and trusted relationships fit together.

I wanted to build something different. I wanted an accessible companion that learns only what you choose to share, remembers it with your permission, and uses that context carefully.

That became Duyu. Duyu ("DOO-you" (two syllables, stress first)) means "sense" in Turkish. It reflects the idea of helping people make sense of the world in the way that works best for them.

What it does

Duyu begins by asking how you want to interact. You can choose voice first, writing and text first, or voice and text equally.

You can also select one or more access needs. These include blind or low vision, deaf or hard of hearing, mobility or dexterity, cognitive or learning, or something else. You never need to share a diagnosis.

You then talk or type naturally. Instead of reducing you to a form or a flat profile, Duyu proposes useful facts as a private personal ontology. This is a connected graph of people, preferences, locations, routines, medications, symptoms, goals, constraints, and relationships.

Nothing is silently added. Duyu shows what it understood and asks you to save or discard it.

Over time, this graph gives Duyu context across different sessions. If you previously said that you take insulin and later describe feeling shaky, Duyu can examine those facts together. It does not assume that something happened simply because information is missing. It never diagnoses or changes medication. It presents a possible connection as a gentle question and encourages you to speak with a qualified healthcare professional.

Duyu can also find nearby pharmacies when you provide a specific area.

Its document tools can recognize text from a photographed or uploaded image, explain it in plain language, and add useful cultural context. Duyu can also prepare an editable message for someone you trust.

Duyu never sends a message automatically. Approving a draft only unlocks a separate sharing action that remains under your control.

Personal memory is stored in the browser on your device. When a live AI feature is used, the relevant input and limited context are sent through Duyu's server-side LLM API. Direct structured operations disable response storage, and tracing is disabled for the conversational agent.

How we built it

Duyu is an accessibility-first progressive web app built with React, TypeScript, Vite, Zustand, Zod, and Vercel serverless functions.

It runs in a mobile browser and can be installed as a PWA without going through an app store.

Voice input and output use the phone or browser's built-in speech recognition and speech synthesis through the Web Speech API. Duyu does not require a separately billed speech service. This means there is no additional per-minute or subscription cost for speech. Some browsers may process recognition online, and normal internet or mobile data charges may still apply.

The interface uses semantic HTML, screen reader support, live status announcements, persistent captions, visible focus indicators, large controls, keyboard access, and reduced-motion support.

Accessibility was not added after the app was built. It shaped the interaction model from the first screen.

At the center of Duyu is an idea connected to my doctoral work. A person should not be represented as a flat profile. Their context should be represented as a living knowledge graph.

The ontology has two layers.

The first is a static semantic backbone based on a verified projection of ETSI SAREF4EHAW and SAREF Core. It covers concepts related to impairments, activities, people, devices, and services.

The second is a private personal layer. It contains the user's confirmed facts, attributes, relationships, confidence, source, and session history.

Duyu uses the maintained and versioned SAREF4EHAW foundation. Concepts that are not covered directly are declared transparently as Duyu concepts.

Duyu uses different GPT-5.6 configurations for different kinds of work.

GPT-5.6 Terra handles fast structured tasks such as ontology extraction, document adaptation, and trusted-contact drafting.

The OpenAI Agents SDK powers conversation, ontology retrieval, semantic grounding, memory proposals, and nearby-place searches.

A higher reasoning configuration supports sensitive cross-session health pattern analysis.

Strict schemas, validation rules, confidence thresholds, approved ontology identifiers, and human confirmation keep model output within the application's boundaries.

Tesseract.js performs document recognition in the browser. OpenStreetMap provides nearby-place results.

Codex was my development partner throughout the project. I used it to develop the idea, create technical documentation, choose the name and hero persona, define the Build Week scope, shape the long-term vision, select the technology stack, and decide which GPT-5.6 configuration belonged in each layer.

Codex also helped build the interface, user experience, backend, ontology creation logic, grounding rules, retrieval system, inference logic, tests, safety boundaries, Git workflows, and voiceover scripts.

Challenges we ran into

Building an ontology that grows from natural conversation is not only an extraction problem.

A useful fact needs a clear identity, type, attributes, relationships, confidence, source, and supporting evidence. Duyu also needs to understand when a new statement adds detail to an existing fact instead of creating a duplicate.

Grounding created another challenge. A language model must not be allowed to invent ontology identifiers that only look correct. Duyu gives the model a closed list of permitted concepts and validates every proposed grounding on the server.

Unknown or incompatible concepts are rejected. Uncertain vocabulary remains a local candidate for later human review.

Retrieval also required careful design. Searching only labels was not enough. Duyu considers attributes, synonyms, broader concepts, relationship names, and connected facts. Confidence thresholds and fixed limits keep the retrieved context focused and understandable.

One important lesson was that missing information does not mean something is false. If there is no meal fact, Duyu cannot conclude that the person skipped lunch. Missing information remains unknown unless the person explicitly says otherwise.

The deeper challenge was responsibility.

The moment an application touches health, disability, or trusted relationships, it has a duty not to overstep. Duyu needed to notice useful patterns without pretending to diagnose, remember without making the user feel watched, and offer help without taking control.

Mobile voice access introduced practical challenges too. Browser support differs, microphone access requires a secure connection, and some languages depend on voices installed on the device. Every voice interaction therefore needed a visible and keyboard-accessible alternative.

Accomplishments that we're proud of

I am proud that Duyu represents a person as connected context rather than a list of disconnected settings.

A saved fact can be connected to other facts, grounded in a verified semantic concept, traced to a session, enriched later, and retrieved through its relationships.

I am also proud of the cross-session reasoning design. Duyu can connect something shared earlier with something said today while treating missing information as unknown and keeping every health-related observation conditional and non-diagnostic.

The semantic foundation is real and reproducible. Duyu uses a pinned and validated release of ETSI SAREF4EHAW while keeping Duyu-specific concepts clearly separate.

Most importantly, the user remains in control.

Proposed memories require confirmation. Private memory can be reviewed and cleared. Drafted messages are never sent automatically. Voice is optional. A diagnosis is never required.

Those boundaries are not restrictions added after development. They are part of what makes Duyu trustworthy.

What we learned

The hardest part of building for accessibility was not the code. It was humility.

It is easy to build something technically impressive. It is much harder to build something that helps without taking over, remembers without becoming intrusive, and adapts without reducing a person to a diagnosis.

I learned to stop asking only what the technology could do. I started asking what would give the person more independence and control.

I also learned that accessibility cannot be represented by a single disabled mode.

Blind and low-vision users, deaf and hard-of-hearing users, people with mobility or dexterity needs, and people with cognitive or learning needs may require very different interactions.

Even two people with the same disability may prefer different combinations of speech, text, captions, structure, and detail.

The shared ontology and interaction architecture can support these different experiences, but selecting an access need is only the beginning. Meaningful accessibility still requires direct testing and collaboration with the people who will use the product.

The man on the corner inspired the project, but he was never the only person Duyu was meant to support.

What's next for Duyu

The next step is to bring the document reader, health pattern layer, and trusted-contact flow directly into the main workspace as complete experiences.

I also want to work directly with blind, low-vision, deaf, hard-of-hearing, mobility-disabled, neurodivergent, and cognitively disabled users.

I want to support more languages, regional contexts, and origin countries. We also want to expand into travel, paperwork, appointments, navigation, and the small daily tasks that often assume everyone can access information in the same way.

A major direction is integrating established domain ontologies alongside Duyu's personal layer. Imagine connecting a validated medical ontology so that when a user mentions insulin, Duyu already understands the full context of diabetes, what usually matters, what to watch for, and how different facts relate, without the user having to explain any of it. The same approach extends far beyond health. Any well-structured ontology, whether medical, legal, nutritional, or civic, can be plugged into the same semantic backbone. That is the real power of building on ontologies. Duyu can inherit deep, structured knowledge about a whole domain and ground it safely in what the user has personally chosen to share.

The ontology will gain stronger controls for evidence, correction, retraction, and history. Retrieval will be evaluated with multilingual accessibility scenarios before the system is expanded further.

The long-term vision is not an assistant that acts for disabled people. It is a personal accessibility layer that gives each person more ways to understand, decide, and act for themselves.

Built With

  • accesibilitic-ontology
  • codex
  • gpt-5.6
  • icf
  • openai-api
  • openai-responses-api
  • progressive-web-app
  • react
  • react-flow
  • saref
  • tailwind-css
  • tesseract.js
  • typescript
  • vercel-functions
  • vite
  • vitest
  • web-share-api
  • web-speech-api
  • web-storage-api
  • who
  • zod
  • zustand
Share this project:

Updates