JARVIS

JARVIS captures meeting context, stores decisions in Supermemory, researches follow-ups in the background, and delivers concise voice-ready answers—so you can focus on work, not remembering it.

Comment

Inspiration

Meetings generate valuable context—decisions, commitments, unanswered questions, and follow-up tasks—but much of it disappears as soon as the call ends. Existing meeting tools can produce transcripts and summaries, yet they rarely help complete the work that follows.

JARVIS was inspired by the idea of a true personal AI butler: an assistant that understands what is happening, remembers it across sessions, and proactively helps the user move from conversation to action.

What it does

JARVIS is a local-first, meeting-aware personal assistant that captures user-authorized screen, system-audio, and microphone context.

It distinguishes the user’s voice from meeting attendees, creates a timestamped transcript, and identifies participants, decisions, action items, owners, deadlines, key moments, and unresolved questions. The complete meeting context is stored in Supermemory Local as durable long-term memory.

When the user asks a question later, JARVIS performs a fresh Supermemory recall before every response while retaining the current conversation. This gives it both long-term memory and natural conversational continuity.

When a meeting produces a question that requires deeper investigation, JARVIS can launch a background research agent. The agent gathers evidence, evaluates its claims, prepares a sourced recommendation, and stores the finished report back in Supermemory.

How we built it

We built JARVIS as a complete real-time pipeline rather than a single model call.

The frontend is a minimal React and Vite workspace that displays meeting capture, Supermemory status, active research agents, meeting briefs, and completed research reports.

A FastAPI backend coordinates the system through HTTP and WebSockets. It streams transcription results, model responses, status updates, and speech audio to the interface.

Supermemory Local acts as JARVIS’s durable context layer. It stores conversations, meeting transcripts, structured summaries, action items, and research reports. Before responding, JARVIS searches this memory so its answers remain grounded in the user’s history.

Groq Whisper handles speech-to-text. Groq GPT-OSS models power conversational responses and meeting analysis, with model fallback when the larger model is rate-limited. NVIDIA NIM or Groq powers the background research workflow.

ElevenLabs provides natural, low-latency speech generation. Kokoro serves as a local fallback, with macOS speech available as an additional recovery option.

The meeting pipeline also captures contextual metadata such as date, time, timezone, locale, meeting platform, capture surface, and duration.

Challenges we ran into

Building a convincing real-time assistant required solving much more than inference.

Voice detection initially interrupted the user too quickly or activated because of keyboard sounds, breathing, and background noise. We added adaptive noise calibration, sustained-speech activation, silence tolerance, short-utterance support, duplicate-transcript protection, mute controls, and push-to-talk.

Text-to-speech was one of the largest latency bottlenecks. We reduced perceived delay by streaming speech as soon as complete punctuation boundaries became available instead of waiting for the full response. We also created a provider fallback chain so voice remains available when a cloud service fails.

Cloud rate limits introduced another reliability challenge. JARVIS now falls back from larger models to faster alternatives while preserving the meeting record and Supermemory context.

Structured model responses were occasionally incomplete. We added schema normalization and transcript-derived fallbacks so an empty or malformed model response cannot leave the dashboard without a usable meeting summary.

Finally, the dashboard originally displayed temporary application state rather than the authoritative memory board. We synchronized briefs directly with Supermemory, added deduplication, and cached unchanged documents to keep the interface current without repeatedly processing the same data.

Accomplishments that we're proud of

We are proud that JARVIS works as a complete product loop:

  • User-authorized screen, system-audio, and microphone capture
  • Separate attribution for the user and meeting attendees
  • Timestamped transcripts with date, timezone, platform, and capture context
  • Automatic meeting briefs containing decisions, actions, owners, questions, and key moments
  • Safe suggested terminal commands that are displayed but never executed automatically
  • Durable Supermemory-backed recall before every response
  • Conversation continuity across multiple requests
  • Background research with evidence review, sources, and recommendations
  • Natural streaming voice through ElevenLabs with local fallbacks
  • A focused dashboard for memory health, active agents, meetings, and completed briefs

Most importantly, JARVIS does not stop at transcription or summarization. It connects what happened in a meeting to the work that needs to happen next.

What we learned

We learned that a useful AI assistant depends on systems design as much as model intelligence.

A powerful model is only one component. Reliable memory, explicit consent, low-latency streaming, structured outputs, provider fallbacks, state synchronization, and a focused interface are what make the experience trustworthy.

We also learned that users do not necessarily need another chat window. They need an assistant that can retain context, identify what matters, surface information at the right moment, and help them follow through.

Supermemory made it possible to treat meetings, conversations, and research as connected parts of the same evolving context rather than isolated interactions.

What's next for JARVIS

The next stage is a complete ChatGPT-like experience built around persistent personal context rather than isolated conversations.

We plan to introduce stronger task detection and automation, richer controls for viewing, editing, and removing memories, improved meeting-platform recognition, proactive reminders, persistent multi-device sessions, and deeper native macOS integration.

We also want to connect JARVIS with Codex CLI so it can move from preparing command plans to performing safe, user-approved actions. Every consequential operation would remain visible, scoped, and explicitly authorized.

The long-term goal is to make JARVIS a trusted personal operating layer: an assistant that understands the user’s work, remembers commitments across sessions, researches what remains unanswered, and safely helps turn intent into completed action.

Built With

Share this project:

Updates