Inspiration

Part of this came from personal experience. One morning I was rushing out of the house, and I was already partway to my destination when the thought hit me: Did I turn the stove off? Did I lock the front door? I had no way to know. Nobody was home to ask, and checking my phone couldn't tell me either the answer simply wasn't anywhere I could reach. I spent the rest of the drive doing the math on whether to turn around.

That moment stuck with me. What I wanted was an assistant that could check the house for me, message me about the small things, and actually call me if something couldn't wait that where cert fills the gap, the calling part of the application.

We also wanted the kind of assistant we grew up watching in sci-fi: one that notices when something matters, reaches out, and helps us handle it.

Most reminder apps assume you will see the notification. But when life gets busy, another notification is easy to miss. That led us to a question: What if your AI assistant could just call you?

That is Sir Jarvis: A personal assistant built around conversations, live detection of events and placing the faceTime call for critical events like emergency vehicles pulling up to the driveway, an intruder that is sensed through motion sensor and immediatly faceTime you when you are home away status, reminders, and timely follow-ups.

What it does

Sir Jarvis is designed to turn important events into conversations. Users can speak naturally to create reminders, manage tasks, and respond to smart-home alerts. When something needs immediate attention like an intruder in the house, emergency vehicles pulled up to the driveway, Jarvis is designed to reach them through FaceTime Audio unlike the traditional apps that we need to login manually and check the event so with Sir Jarvis you are just one step away from receiving the call in case of emergency event detection, according to their preferences.

Our MVP centers on three experiences:

  • JarvisRemind: Create, acknowledge, snooze, reschedule, and complete reminders through conversation.
  • JarvisHome: Connect home events to relevant low and hight risk alerts and authorized light controls through the voice command.
  • JarvisTalks: Have natural conversations using task context the user has permitted Jarvis to retain.

How we built it

We structured Sir Jarvis around a simple pipeline:

$$ \text{Event} \rightarrow \text{Conversation} \rightarrow \text{Action} $$

The frontend uses Next.js, Tailwind CSS, and shadcn/ui. Supabase provides the foundation for authentication, persistent data, realtime updates, and scheduled workflows. The voice agent combines LiveKit Agents and Gemini Live, while Chert connects the experience to FaceTime Audio and iMessage.

Our first working milestone was a real FaceTime Audio/video conversation with the agent on a phone. From there, we organized the remaining work around shared contracts for reminders, home events, communication preferences, and delivery history.

The broader MVP includes dashboard, reminder, escalation, and home-event workflows. Outbound calling and messaging depend on provider provisioning; simulated paths are identified explicitly in the demo.

Challenges we ran into

The hardest part was connecting a conversation to a dependable action like showing the live feed when some sort of emergency or human intervention/confirmation required events in the house.

Sending a reminder, receiving an acknowledgment, and completing a task are different outcomes:

$$ \text{Delivered} \neq \text{Acknowledged} \neq \text{Completed} $$

Keeping those states separate is essential for reliable follow-ups.

Proactive outreach also required careful decisions about when Jarvis should text, when it should call, and what should happen if the user does not answer. We designed around priorities, quiet hours, grace periods, and call limits.

Coordinating the voice agent, backend, communications, and dashboard under a hackathon deadline was another challenge. Shared data contracts helped keep the work aligned. The back and forth communication when a event is detected that requires immediate faceTime calling and when you do the calling to the home security system.

Accomplishments that we're proud of

  • A real phone conversation with our agent. Our first milestone was a live FaceTime Audio conversation with Jarvis on an actual phone not a simulation.
  • A reminder lifecycle we can trust. Reminders move through delivered, acknowledged, and completed as separate states, and outreach attempts are claimed atomically, so the same reminder can never trigger two calls.
  • An escalation engine with judgment. Jarvis decides between an iMessage and a FaceTime call based on priority, and respects quiet hours, grace periods, and daily call limits before it interrupts anyone.
  • Parallel building without collisions. Seven AI-assisted workstreams database, dashboard, voice, escalation, delivery, and review built against one shared data contract, and the pieces matched field-for-field when they came together.
  • An honest demo. Every path that is simulated is labeled as simulated, so what judges see is exactly what is live.

What we learned

We learned that making an AI talk is only one part of building an assistant. The real work is ensuring it understands the task, has permission to act, saves the result correctly, and reports what actually happened.

We also learned to distinguish our product vision from what we could demonstrate live. A real phone conversation gave us a concrete foundation, while clearly labeled simulations kept the rest of the demo honest.

What's next for Sir Jarvis

Our next steps are to complete and validate the reminder lifecycle, connect reliable outbound outreach, and expand smart-home integrations including the scenarios that inspired the project: a departure check when the owner leaves (stove off, doors locked, emergencies vehicle in the driveway), a message for non-critical events like an unfamiliar car lingering near the driveway or emergency vehicles nearby, and a direct FaceTime call for a break-in or a gas stove left on.

Our bigger vision is an assistant that earns the right to interrupt you because when Jarvis calls, it has something worth saying.

Built With

Share this project:

Updates

Submission history