Sunday

Sunday is a voice first AI Agent that helps visually impaired, elderly, and motor-impaired users complete everyday phone tasks independently, beyond what traditional screen readers can easily support.

Comment

Note: Demo is sped up 8* to fit within 1 minute video

The Problem We Could Not Ignore

Smartphones have become essential for everyday life, yet many blind, visually impaired, elderly and motor-impaired users still struggle to use them independently.

The problem wasn't a lack of accessibility tools. It was that they stopped at helping users navigate rather than helping them complete tasks.

What we observed:

  • Existing screen readers require users to remember complex gestures and navigation patterns.
  • Many apps contain unlabeled buttons, custom interfaces and inaccessible WebViews.
  • Completing simple tasks like ordering food or booking transport can require navigating through dozens of interface elements.
  • Users often need to combine multiple accessibility tools just to accomplish one task.
  • We believed users shouldn't have to learn every app seperately. They should simply tell their phone what they want to do.

From Navigation to Action

Instead of building another screen reader, we wanted to build something fundamentally different.

Sunday was designed as a voice-first AI agent that focuses on completing goals rather than reading interfaces.

How Sunday works:

  • User describes a goal naturally.
  • Sunday understands the current screen.
  • Plans the next action.
  • Performs taps, scrolling and typing.
  • Verifies whether each action succeeded.
  • Narrates every important step.
  • Asks follow-up questions whenever information is missing.

Instead of navigating apps manually, users simply have a conversation.

How We Built It

Sunday evolved through multiple iterations.

Phase 1 Understanding the screen

  • Started as an AI-powered conversational screen reader.
  • Converted accessibility information into natural language explanations.
  • Quickly realised understanding the interface wasn't enough.

Phase 2 Taking action

We introduced autonomous interactions.

Sunday now:

  • Understands interfaces
  • Plans actions
  • Controls applications
  • Verifies outcomes
  • Recovers from failures

Making the experience natural

We added:

  • Clear narration explaining every step.
  • Conversational clarification when information is missing.
  • Safety checks before important actions.
  • Continuous verification after every interaction.

Technologies

  • Android Accessibility Framework
  • Accessibility Trees
  • Screenshot understanding when necessary
  • LLM reasoning for planning
  • ElevenLabs for natural voice interaction
  • Exploring Meralion / SEA-LION for multilingual Southeast Asian support

How OpenAI accelerated development

OpenAI became our engineering partner throughout development.

We used it to:

  • Analyse thousands of Android logs.
  • Debug difficult accessibility behaviours.
  • Review agent architecture.
  • Improve prompts and reasoning.
  • Research accessibility best practices.
  • Iterate significantly faster than traditional trial-and-error debugging.

The Hardest Part

Making an agent work once with a curated demo was easy.

Making it work reliably accross many different scenarios and edge cases is extremely difficult.

Challenge 1 WebView text input

Problem:

  • Sunday could identify text fields but couldn't consistently focus them before typing.

Result:

  • Typing silently failed on many websites.

Solution:

  • Used OpenAI to analyse execution logs and redesign how Sunday handled focus and text entry.

Challenge 2 Knowing whether an action actually worked

Initially:

  • Sunday assumed every tap succeeded.

Reality:

  • Mobile interfaces constantly change.
  • Buttons ignore touches.
  • Layouts move.
  • Screens load slowly.

This caused repeated loops and failed tasks.

Solution:

  • Redesigned Sunday into an Observe → Reason → Act → Verify loop.
  • Used OpenAI extensively to identify recurring failure patterns and improve verification logic.

What We're Proud Of

Technical achievements

  • Built a voice-first Android AI agent capable of operating real applications.
  • Created a reliable perception → reasoning → action pipeline.
  • Designed conversational clarification instead of rigid command systems.
  • Developed transparent narration to improve user trust.
  • Continuously improved reliability through real-world testing.

Community impact

  • Piloted Sunday with real users through SAVH.
  • Collected direct feedback from our target users early.
  • Opportunity to expand to organisations such as SG Enable and Gen Singapore.

Recognition

  • Secured funding.
  • Won multiple startup competitions.
  • Represented SUTD at innovation and entrepreneurship events.

What We Learned

Building Sunday taught us lessons far beyond software engineering.

  • Accessibility is about completing tasks, not reading screens.
  • Reliability matters more than raw AI intelligence.
  • Great agents know when to ask questions instead of making assumptions.
  • Verification is just as important as planning.
  • Building with real users from the beginning is the fastest way to discover meaningful problems.
  • AI alone doesn't solve accessibility. Thoughtful user interaction design is way more effective.

What's Next

Our journey is just beginning.

Next, we aim to:

  • Cross session memory
  • Improve reliability across more Android applications.
  • Support more Southeast Asian languages.
  • Optimizing latency between actions
  • Expand trials with visually impaired and elderly communities.
  • Collect quantitative usability metrics.

Our vision remains the same:

To build an accessibility-first AI agent that enables anyone to use their phone independently, confidently and with dignity.

Built With

  • kotlin
  • openai
Share this project:

Updates