Jarvis — Agentic AI Personal & Voice Assistant

Jarvis is a multimodal (text and voice) personal assistant that connects to a user's Google account and autonomously reads mail, drafts replies, and manages a calendar and to-do list on their behalf — through natural conversation and real-time voice, not forms or menus.

It operates via two complementary interaction loops:

Text Agent (LangGraph ReAct): Given a chat message, the agent reasons about intent, decides which tool(s) to call, invokes them in sequence, observes results, and formulates a grounded response.
Voice Agent (LiveKit Cloud Worker): Powered by LiveKit, Silero VAD, Groq Whisper (STT), Groq Llama 3.3 70B (LLM with native function calling), and Cartesia Sonic 2 (TTS) for low-latency, real-time voice interaction with non-blocking execution.

Hard constraint, by design: Jarvis can read email and create drafts, but it can never send one. This is enforced at two independent layers — the OAuth scope granted (gmail.compose, which has no send capability at the Google API level) and the toolset itself (no "send" tool is ever defined in either the text or voice tools).

Live Deployments:

Web App & REST API: Deployed as a single Vercel project (React SPA frontend + FastAPI serverless backend), split across two serverless functions — see Backend Split below.
Voice Agent: Deployed on LiveKit Cloud as a background agent worker with process prewarming and per-session user credential isolation.

Key Capabilities & Agentic Architecture

Jarvis is agentic in three concrete ways:

Tool Use over Fixed Flows: The agent possesses a toolset (read email, draft email, list/create/update/delete calendar events, list/create/update/delete tasks) and decides dynamically which tools to invoke and with what arguments based on user input.
Multi-Step Reasoning (ReAct Loop & Native Function Calling): For complex requests like "check if I'm free Thursday afternoon and if so schedule a call with Sara at 3pm," the agent inspects schedule availability first, processes the result, and then executes event creation.
Grounded Data Enforcement: System prompts explicitly forbid fabricating IDs, dates, email addresses, or event/task details — every fact originates from real tool calls against the user's actual Google account.
Non-Blocking Real-Time Voice Processing: In the LiveKit voice agent worker (agent.py and tools.py), synchronous Google API client calls are offloaded to background threads using asyncio.to_thread. This prevents audio frame dropouts and stuttering during speech synthesis and turn-taking.

Backend Split: `main.py` + `chat_main.py`

The FastAPI backend is intentionally split into two separate app instances, each deployed as its own Vercel serverless function:

app/main.py — every REST route except chat: auth, emails, events, todos, livekit.
app/chat_main.py — only POST /api/chat.

Why: /api/chat pulls in LangGraph, LangChain, and the Groq client — a heavy dependency chain. Bundling it together with the lighter REST endpoints risked exceeding Vercel's 225MB per-function bundle limit. Splitting it into its own function keeps both bundles under the limit. Functionally, /api/chat behaves identically to before — this is purely a deployment/packaging concern, not a behavior change.

In production, both functions are deployed under the same domain, and Vercel routes /api/chat to the chat function (everything else goes to the main function) via a rewrite in vercel.json. This means the frontend can — and should — call one origin for everything, same as before.

Locally, this split has one consequence: there is no single process serving both /api/todos and /api/chat — you need to run two Uvicorn processes side by side. See Local Development Setup.

Mobile-Friendly UI Features

The frontend interface is built with Vite + React and designed mobile-first:

Off-Canvas Navigation Drawer: On viewports below 768px, the sidebar transforms into an off-canvas drawer triggered by a responsive hamburger button (☰ / ✕) with a blurred background backdrop (rgba(0, 0, 0, 0.6)).
Dynamic Viewport Height (100dvh): Layouts use dynamic viewport units (100dvh) to prevent layout jumps caused by mobile browser address bar auto-hiding (iOS Safari / Android Chrome).
iOS Auto-Zoom Prevention: Form text inputs enforce a minimum 16px font size to disable iOS Safari's default page zoom on field focus.
Touch Target Standards: Interactive controls, inputs, and buttons maintain minimum touch surface dimensions (min-height: 44px, padding: 10px 12px, touch-action: manipulation) to eliminate tap delays and accidental misclicks.
Responsive Text Bubbles & Scrolling: Chat bubbles adapt smoothly up to 85% width on mobile screens with word-break: break-word and momentum scrolling (-webkit-overflow-scrolling: touch).
Integrated Voice Panel: LiveKit VoicePanel component rendered directly above the chat input for one-tap voice session control on mobile and desktop.
Sidebar Draft Emails Panel: DraftMail.jsx lists Gmail drafts the agent has created (subject, recipient, snippet), rendered in its own labeled section alongside Upcoming Events and Tasks — a visible, at-a-glance surface for reviewing what Jarvis has drafted before you send it yourself from Gmail.
Future-Only Events Panel: EventsPanel.jsx filters out any event that has already started, sorting the rest soonest-first, so the sidebar never shows stale/past events even if the API response includes them.

System Architecture

                                  ┌─────────────────────────────────────────┐
                                  │             LiveKit Cloud               │
                                  │          (WebRTC Voice Server)          │
                                  └────────────────────┬────────────────────┘
                                                       │
                                            Audio / WebRTC Session
                                                       │
┌──────────────┐   HTTPS / REST    ┌───────────────────▼─────────────────────┐
│              ├──────────────────►│             Vercel Hosting              │
│   Browser    │                   │ ┌──────────────────┬──────────────────┐ │
│ (Vite/React) │◄──────────────────┤ │  React SPA App   │ FastAPI (/api/*) │ │
└──────────────┘                   │ │                   │  main.py fn      │ │
        ▲                          │ │                   │  chat_main.py fn │ │
        │                          │ └──────────────────┴────────┬─────────┘ │
        │ WebRTC                   └─────────────────────────────┼───────────┘
        │                                                        │
┌───────┴────────────────────────────────────────────────────────▼───────────┐
│                        LiveKit Cloud Agent Worker                          │
│ ┌────────────────────────────────────────────────────────────────────────┐ │
│ │  agent.py — Silero VAD ──► Groq Whisper (STT) ──► Groq Llama 3.3 (LLM) │ │
│ │                │                                           │           │ │
│ │                ▼                                           ▼           │ │
│ │  JarvisVoiceTools (Async Thread Pools)           Cartesia Sonic 2 (TTS)│ │
│ └────────────────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────┬───────────────────────────────────────┘
                                     │
      ┌──────────────────────────────┼──────────────────────────────┐
      ▼                              ▼                              ▼
┌──────────────┐              ┌──────────────┐              ┌──────────────┐
│   Supabase   │              │ Google Gmail │              │ Google Cal / │
│  (Postgres)  │              │     API      │              │  Tasks APIs  │
└──────────────┘              └──────────────┘              └──────────────┘

Project Structure

jarvis-app/
├── agent.py                           LiveKit Cloud Voice Worker entrypoint & AgentSession setup
├── tools.py                           LiveKit Voice Tools class using asyncio.to_thread wrappers
├── api/
│   └── index.py                       Vercel entrypoint importing backend/app/main.py
├── requirements.txt                   Root-level delegate (-r backend/requirements.txt)
├── .python-version                    Pins Python version (3.14 for Vercel, 3.11+ for LiveKit)
├── vercel.json                        Monorepo routing & build configuration for Vercel
│
├── backend/                           FastAPI + LangGraph + Google Services
│   ├── app/
│   │   ├── main.py                    FastAPI app (all routes except /chat), middleware, CORS
│   │   ├── chat_main.py               Separate FastAPI app containing ONLY /api/chat
│   │   │                              (isolates the LangGraph/LangChain/Groq dependency chain
│   │   │                               into its own Vercel function to stay under the 225MB
│   │   │                               per-function bundle limit)
│   │   ├── config.py                  Pydantic-settings configuration for environment variables
│   │   ├── security.py                Fernet symmetric encryption/decryption for OAuth tokens
│   │   ├── database.py                Supabase client initialization (service-role)
│   │   ├── deps.py                    get_authed_user() & get_credentials_for_user()
│   │   ├── models.py                  Pydantic request/response data models
│   │   │
│   │   ├── routers/
│   │   │   ├── auth.py                /auth/login, /auth/callback, /auth/me, /auth/logout
│   │   │   ├── emails.py              GET /emails, GET /emails/drafts, GET /emails/{id}, POST /emails/draft
│   │   │   ├── events.py              Calendar CRUD endpoints (/events)
│   │   │   ├── todos.py               Tasks CRUD endpoints (/todos)
│   │   │   └── chat.py                POST /chat (LangGraph execution & history persistence;
│   │   │                              registered only in chat_main.py, not main.py)
│   │   │
│   │   ├── services/                  Google API operational wrappers
│   │   │   ├── google_oauth.py
│   │   │   ├── google_clients.py
│   │   │   ├── gmail_service.py       list_messages, get_message, list_drafts, draft_email
│   │   │   │                          (drafting only — no send() call exists anywhere in this file,
│   │   │   │                           enforced further by the gmail.compose OAuth scope)
│   │   │   ├── calendar_service.py
│   │   │   └── tasks_service.py
│   │   │
│   │   ├── tools/                     LangGraph text-agent tool factories
│   │   │   ├── gmail_tools.py
│   │   │   ├── calendar_tools.py
│   │   │   └── task_tools.py
│   │   │
│   │   └── agent/                     LangGraph ReAct text agent setup
│   │       ├── graph.py               build_agent(credentials) factory
│   │       └── prompt.py              System prompt definitions & rules
│   │
│   ├── supabase_schema.sql            Database schema definitions
│   └── requirements.txt               Backend Python dependencies
│
└── frontend/                          Vite + React SPA Frontend
    ├── src/
    │   ├── main.jsx                   React entrypoint
    │   ├── App.jsx                    Router, Auth guard, responsive layout wrapper
    │   ├── index.css                  CSS tokens, 100dvh reset, touch styles, drawer animations
    │   ├── api/
    │   │   └── client.js              Axios instances (api + chatApi) with credentials and API endpoints
    │   ├── components/
    │   │   ├── Sidebar.jsx            Mobile responsive navigation drawer + hamburger button
    │   │   ├── EventsPanel.jsx        Sidebar upcoming (future-only) calendar events component
    │   │   ├── DraftMail.jsx          Sidebar Gmail drafts component (subject, recipient, snippet)
    │   │   ├── TasksPanel.jsx         Sidebar tasks component
    │   │   └── VoicePanel.jsx         LiveKit voice session control component
    │   └── pages/
    │       ├── Login.jsx              Google Sign-In landing page
    │       ├── Dashboard.jsx          User dashboard page
    │       └── Chat.jsx               Mobile-friendly conversational text/voice interface
    └── .env.example

Tech Stack

Layer	Technology	Function
Voice Agent Orchestration	LiveKit Agents SDK (`livekit-agents`)	Real-time WebRTC voice pipeline, participant tracking, prewarming
STT (Speech-to-Text)	Groq Whisper (`whisper-large-v3-turbo`)	Low-latency voice transcription
LLM (Voice & Text)	Groq (`llama-3.3-70b-versatile`)	Fast inference for text chat and native function calling for voice
TTS (Text-to-Speech)	Cartesia (`sonic-2`)	Low-latency, natural conversational speech synthesis
VAD (Voice Activity)	Silero VAD	Local turn detection and speech boundary segmentation
Text Agent Orchestration	LangGraph (`create_react_agent`)	ReAct loop for step-by-step reasoning and tool calling
Backend Framework	FastAPI (split across `main.py` + `chat_main.py`)	Async REST API running on serverless Python
Auth & Security	Google OAuth 2.0 + Fernet Encryption	Least-privilege OAuth scopes with server-side token encryption
Database	Supabase (Postgres)	Stores user accounts, encrypted tokens, conversations, and messages
Frontend UI	Vite + React	Mobile-first SPA with CSS tokens and dynamic viewports
Hosting	Vercel + LiveKit Cloud	Vercel for Web App/REST; LiveKit Cloud for real-time Voice Worker

Design System & Color Tokens

Defined centrally in frontend/src/index.css:

Token	Value	Applied To
`--bg`	`#0a0e14`	Primary app background
`--surface`	`#10161f`	Sidebar, cards, headers
`--surface-raised`	`#161e2a`	Elevated UI elements (inputs, chat bubbles)
`--border`	`rgba(255, 255, 255, 0.08)`	Dividers, borders, section bounds
`--text`	`#e7edf5`	High-contrast body text
`--text-muted`	`#7a8699`	Secondary labels, placeholders, timestamps
`--accent`	`#49e0d1`	Arc-reactor cyan — buttons, active states
`--accent-dim`	`rgba(73, 224, 209, 0.15)`	Active navigation backgrounds, user chat bubbles
`--danger`	`#ff6b6b`	Delete buttons, destructive actions

Local Development Setup

1. Prerequisites & Environment Variables

Copy .env.example in both root/backend and frontend, then populate all required keys.

Backend (backend/.env):

# Google OAuth
GOOGLE_CLIENT_ID=your_google_client_id
GOOGLE_CLIENT_SECRET=your_google_client_secret
GOOGLE_REDIRECT_URI=http://localhost:8000/api/auth/callback

# Supabase
SUPABASE_URL=https://your-supabase-project.supabase.co
SUPABASE_SERVICE_ROLE_KEY=your_supabase_service_role_key

# AI Provider Keys
GROQ_API_KEY=gsk_your_groq_api_key
CARTESIA_API_KEY=your_cartesia_api_key

# Security Keys
APP_SECRET_KEY=your_app_secret_key
TOKEN_ENCRYPTION_KEY=your_generated_fernet_key

# LiveKit Cloud Config
LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret

# URLs
FRONTEND_URL=http://localhost:5173

To generate a valid TOKEN_ENCRYPTION_KEY:

python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

Frontend (frontend/.env.local):

VITE_API_URL=http://localhost:8000/api
VITE_CHAT_API_URL=http://localhost:8001/api

VITE_API_URL covers every endpoint except chat. VITE_CHAT_API_URL points at the separate chat_main.py process (see below) — locally these run on two different ports; in production both resolve to the same origin, so this var is left unset on Vercel and chatApi falls back to API_URL automatically.

2. Database Migration

Run backend/supabase_schema.sql inside the Supabase SQL Editor to construct the users, oauth_tokens, conversations, and messages tables.

3. Backend & Voice Agent Setup

# Create virtual environment
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

# Install dependencies
pip install -r backend/requirements.txt

Run both backend servers — two separate terminals, both with the venv activated:

Terminal 1 — REST API (auth, emails, events, todos, livekit):

python -m uvicorn app.main:app --reload --port 8000

Terminal 2 — Chat API (LangGraph agent, isolated dependency chain):

python -m uvicorn app.chat_main:app --reload --port 8001

Windows note: if uvicorn app.main:app --reload fails with Fatal error in launcher: Unable to create process..., the .exe launcher in venv\Scripts has a stale absolute path baked in (usually from moving/renaming the project folder after the venv was created). Using python -m uvicorn ... instead of the bare uvicorn command sidesteps this, since it resolves the interpreter from the active venv on PATH rather than the path embedded in the launcher. To fix it permanently, delete and recreate the venv from the project's current location.

Both servers need to stay running simultaneously for the frontend to work end-to-end — REST calls (/api/todos, /api/events, etc.) go to :8000, and /api/chat goes to :8001.

To run the LiveKit Voice Agent worker locally in development mode:

python agent.py dev

4. Frontend Setup

cd frontend
npm install
npm run dev

Vite only reads .env/.env.local at process startup. If you add or change VITE_CHAT_API_URL after npm run dev is already running, fully stop it (Ctrl+C) and restart — a hot-reload alone won't pick up the new value, and requests will keep silently falling back to VITE_API_URL.

Deployment Guide

Deployment 1: Web App & Backend API (Vercel)

The web app and REST API are deployed to Vercel as a single monorepo project, with the backend split into two serverless functions — one for main.py (all REST routes), one for chat_main.py (/api/chat only) — routed by path under the same domain via vercel.json. Both functions read from the same environment variables configured in the Vercel dashboard.

Connect your repository to Vercel.
Add all environment variables (GOOGLE_CLIENT_ID, SUPABASE_URL, GROQ_API_KEY, etc.) in the Vercel Dashboard under Settings → Environment Variables — these are shared across both functions.
Set VITE_API_URL to /api for same-origin serverless routing. Leave VITE_CHAT_API_URL unset in Vercel — it's a local-dev-only override; when unset, chatApi in client.js falls back to API_URL, so chat requests go to the same /api origin as everything else, and Vercel's path rewrite handles sending /api/chat specifically to the chat_main.py function.
Confirm vercel.json includes a route/rewrite directing /api/chat to the chat_main.py function and all other /api/* paths to main.py — this is what makes the "same origin, two functions" split work in production. Verify this against your actual vercel.json before deploying, since the exact rewrite syntax depends on how each function entrypoint is wired up (e.g. a second functions entry pointing at a chat_main-based entrypoint alongside api/index.py).

Deployment 2: Real-Time Voice Agent (LiveKit Cloud)

The voice worker (agent.py and tools.py) runs on LiveKit Cloud infrastructure.

Step 1: Install LiveKit CLI & Authenticate

lk cloud login

Step 2: Upload Application Secrets to LiveKit Cloud

Push your complete local .env configuration file to LiveKit Cloud so the cloud agent has access to all required Google, Supabase, Groq, and Cartesia keys:

lk agent update-secrets --secrets-file .env

Step 3: Deploy the Agent Worker

Deploy the agent code to LiveKit Cloud:

lk agent deploy

Step 4: Verify LiveKit Cloud Deployment

Check deployment health, active status, and live worker logs:

lk agent status
lk agent logs

API Reference

All backend routes are prefixed with /api. Routes are split across two apps — main.py and chat_main.py — but behave as one API surface behind the same origin.

Method	Endpoint	Description	Served by
`GET`	`/api/auth/login`	Redirects user to Google OAuth consent screen	`main.py`
`GET`	`/api/auth/callback`	Handles OAuth exchange, encrypts tokens, sets httpOnly session cookie	`main.py`
`GET`	`/api/auth/me`	Returns current authenticated user profile	`main.py`
`POST`	`/api/auth/logout`	Clears local session state and cookie	`main.py`
`GET`	`/api/emails`	Fetches recent email headers	`main.py`
`GET`	`/api/emails/drafts`	Lists Gmail drafts (subject, recipient, snippet) — powers `DraftMail.jsx`	`main.py`
`GET`	`/api/emails/{id}`	Fetches detailed message content	`main.py`
`POST`	`/api/emails/draft`	Creates a Gmail draft (never sends)	`main.py`
`GET / POST`	`/api/events`	List or create calendar events	`main.py`
`PATCH / DELETE`	`/api/events/{id}`	Update or delete calendar events	`main.py`
`GET / POST`	`/api/todos`	List or create user tasks	`main.py`
`PATCH / DELETE`	`/api/todos/{id}`	Update or delete user tasks	`main.py`
`POST`	`/api/chat`	Send a text prompt to the LangGraph ReAct agent	`chat_main.py`

Security Guarantees

HttpOnly Session Cookies: User authentication relies on httpOnly, signed session cookies containing internal user IDs. Client-side JavaScript cannot access session tokens.
At-Rest Token Encryption: Google OAuth tokens are encrypted via Fernet (TOKEN_ENCRYPTION_KEY) before insertion into Supabase. Decryption occurs strictly in-memory per request.
Restricted OAuth Scopes: Scopes are limited to gmail.readonly, gmail.compose, calendar, and tasks. gmail.compose structurally lacks the ability to send emails.
No Code-Level Send Capability: Neither the REST API, LangGraph tools (backend/app/tools), nor LiveKit Voice tools (tools.py) define any sending functions. gmail_service.py only ever calls drafts().create(), never messages().send() or drafts().send().
Thread-Safe Session Isolation: LiveKit voice agent instances construct unique per-user Google client bindings during entrypoint execution. One participant cannot access another participant's mailbox or calendar.

Current Limitations

Google OAuth Testing Mode: OAuth registration is currently capped at 100 test users in Google Cloud Console.
Single LLM Provider: Agent execution depends on Groq endpoints; no secondary fallback provider is currently configured.
Task List Scope: Default task management is constrained to the primary @default Google Tasks list.
UI State Synchronization: Modifying events or tasks through chat/voice does not auto-refresh the sidebar panels without a manual UI trigger.
Two-Process Local Dev: Because /api/chat is split into its own app (chat_main.py), local development requires running two Uvicorn processes instead of one — see Local Development Setup.

Roadmap

[ ] Stream text response tokens using Server-Sent Events (SSE).
[ ] Implement an event bus to auto-refresh sidebar panels after voice/text tool executions.
[ ] Add conversation context window pruning and summarization.
[ ] Implement strict Pydantic models for update tool operations.
[ ] Complete Google Cloud OAuth verification to remove user limits.

Jarvis-Text-Voice

Jarvis — Agentic AI Personal & Voice Assistant

Key Capabilities & Agentic Architecture

Backend Split: `main.py` + `chat_main.py`

Mobile-Friendly UI Features

System Architecture

Project Structure

Tech Stack

Design System & Color Tokens

Local Development Setup

1. Prerequisites & Environment Variables

2. Database Migration

3. Backend & Voice Agent Setup

4. Frontend Setup

Deployment Guide

Deployment 1: Web App & Backend API (Vercel)

Deployment 2: Real-Time Voice Agent (LiveKit Cloud)

Step 1: Install LiveKit CLI & Authenticate

Step 2: Upload Application Secrets to LiveKit Cloud

Step 3: Deploy the Agent Worker

Step 4: Verify LiveKit Cloud Deployment

API Reference

Security Guarantees

Current Limitations

Roadmap

Built With

Updates

Jarvis — Agentic AI Personal & Voice Assistant

Key Capabilities & Agentic Architecture

Backend Split: main.py + chat_main.py

Mobile-Friendly UI Features

System Architecture

Project Structure

Tech Stack

Design System & Color Tokens

Local Development Setup

1. Prerequisites & Environment Variables

2. Database Migration

3. Backend & Voice Agent Setup

4. Frontend Setup

Deployment Guide

Deployment 1: Web App & Backend API (Vercel)

Deployment 2: Real-Time Voice Agent (LiveKit Cloud)

Step 1: Install LiveKit CLI & Authenticate

Step 2: Upload Application Secrets to LiveKit Cloud

Step 3: Deploy the Agent Worker

Step 4: Verify LiveKit Cloud Deployment

API Reference

Security Guarantees

Current Limitations

Roadmap

Built With

Updates

Backend Split: `main.py` + `chat_main.py`