Inspiration
Loved by God — driven by the belief that everyone deserves seamless access to technology, regardless of ability.
What it does
Amadeus offers an AI-powered mobile interactive agent that listens to your voice and executes complex, multi-step tasks by directly controlling any app’s UI on your device.
How we built it
We integrated Appium’s UI automation capabilities with a Python-based AI agent (using the OpenAI client for LLM requests). Appium bridges between our agent and the mobile device, allowing the AI to simulate taps, swipes, text entry, and navigation in real time.
Challenges we ran into
-Cross-app UI variability: Each app uses different view hierarchies and element identifiers, so building a reliable, universal selector engine took extensive filtering and pattern detection.
-Latency and responsiveness: Voice recognition, AI planning, and UI execution all incur delays; tuning the prompt and optimizing Appium commands was essential to keep interactions feeling natural.
-Error recovery: Apps sometimes change layouts or load slowly; we had to implement robust fallback logic and voice-guided clarifications to prevent dead-ends.
Accomplishments that we're proud of
Each component (voice I/O, UI automation) is cleanly decoupled, allowing us to swap in new models or add plugins with minimal changes.
What we learned
We learned that moving configuration and test data into separate files (rather than embedding values in code) dramatically reduced maintenance overhead when app UIs changed.
What's next for Amadeus
-On-device autonomy: Move from a host-controlled Appium architecture to a standalone Android service so users can install and run Amadeus directly on their phones without a PC.
-Machine-learned UI patterns: Use ML to automatically recognize UI templates within app categories (e-commerce, banking, social) for faster setup and fewer manual filters.
-iOS support: Explore Swift-based Accessibility and SiriKit integrations to extend Amadeus to the iPhone.
-Healthcare & compliance: Build HIPAA-compliant modules for secure voice-driven EHR navigation and telehealth workflows in clinical settings.
-Plugin ecosystem: Launch a developer SDK so third parties can create specialty plugins (e.g., ERP automation, smart-home control) that tap directly into Amadeus’s voice-to-UI framework.
Log in or sign up for Devpost to join the conversation.