Jeong AI

Inspiration

Jeong AI is a voice companion that lives on their phone. It does three things:

1. Daily conversation — A warm, patient AI that talks with them, remembers their stories, preferences, and routines, and serves as a memory aid
2. Phone automation by voice — "Book me an Uber to the pharmacy", "Call my daughter", "Send Sarah a voice note" — no new apps to learn
3. Invisible safety — Fall detection that sends a push notification first, waits for a response, and only calls family if the elder doesn't respond. Help is there when needed, not hovering when it's not

For family, there's a web dashboard with voice notes from their loved one and gentle daily summaries ("Mom seemed cheerful today") — not activity logs, not location tracking, not cognitive scores. Connection, not surveillance.

## How we built it

• Mobile app — React Native (Expo) with a voice-first interface, large text, high contrast, and minimal screens designed for accessibility
• Backend — Python FastAPI handling voice processing, agent orchestration, and family communication
• AI agent — OpenAI Agents SDK with custom tools for phone control, outbound calling, reminders, and conversational memory
• Voice pipeline — Deepgram for speech-to-text, ElevenLabs for natural text-to-speech, creating fluid voice conversations
• Phone automation — Minitap Mobile-Use SDK allowing the agent to control the phone on the user's behalf
• Emergency calls — Vapi for autonomous outbound calls to family with escalation logic (primary contact → secondary → emergency services)
• Database — PostgreSQL storing conversation summaries (never transcripts), reminders, family messages, and call logs

## Challenges we faced

• Dignity-first design is hard. Every feature decision required asking "would I be embarrassed if this was done to me?" We cut several features that felt useful on paper but failed the dignity test — activity timelines, cognitive scoring, detailed conversation logs for family
• Voice latency — Chaining STT → agent reasoning → TTS creates noticeable delay. We optimized by streaming ElevenLabs responses and keeping agent tool calls minimal during conversation
• Phone automation reliability — Mobile-Use SDK is powerful but unpredictable across different apps and UI states. We built an allowlist of safe apps and confirmation steps before any payment or sensitive action
• Fall detection without hardware — We couldn't add a wearable for a hackathon, so we designed the system around WiFi sensing simulation with a clear path to real sensor integration, keeping the alert → notification → wait → call flow production-ready
• Balancing family peace of mind with elder privacy — The hardest design challenge. We landed on: the elder controls what's shared, family sees summaries not data, and help requests come from the elder's initiative — not automated surveillance triggers

## What we learned

• The best safety is invisible. When people enjoy using something daily (conversation, task help, family connection), you can layer in safety without it feeling like monitoring. The daily value IS the safety mechanism — if they stop chatting, that's the signal
• Elderly users are more capable than we assume. Vihaan's grandfather figured out Surf quickly because it met him where he was — voice, not touch. Designing for accessibility isn't dumbing down; it's removing unnecessary barriers
• AI companions need boundaries. The agent must know when to hand off to a human. No AI should be the sole point of contact in a medical emergency — Vapi outbound calls to real family members are the safety net, not more AI
• Privacy is a feature, not a constraint. Storing summaries instead of transcripts, letting elders control sharing, hiding activity logs from family — these aren't limitations. They're what makes the product trustworthy enough to actually get adopted.

Built With

• python
• react-native
• swift