Market Call

Inspiration / pivot story

The team started with a different idea: a live FaceTime coach that watches your camera and corrects skill drills in real time (sports form, knot-tying, etc. — graded against the camera feed via Gemini vision). Mid-hackathon, the team pivoted to something with a clearer "why would anyone want this": most people have money in the market and zero attention span for a finance app. The new idea — a fund manager you can call, who calls you — reuses the same underlying FaceTime-agent plumbing (Chert + LiveKit + Gemini) but points it at markets instead of drills. Vestigial files from the drill-coach phase (drills.mjs, face.mjs, tasks.mjs) are still in the repo; the live camera Q&A tool (answer_visual_question) still works generically (ask it about anything you point the camera at), it's just not the headline feature anymore.

What it does

Reachable over a real phone number, via FaceTime and iMessage (not an app you open) — built on Chert, a platform that gives an agent a real phone line.
Introduces itself as "Sid" (configurable display name), your fund manager — never hedges with "as an AI I can't give financial advice." It has real, specific opinions: near an all-time high, it'll suggest thinking about trimming; a solid name down hard, it'll suggest considering adding. It's always explicit that executing any trade is still the caller's call with their own broker.
Shows live charts on screen during the call for any stock, ETF, index, or crypto ticker — not just what's in your portfolio. Renders as a custom hand-drawn video feed (see architecture below), not a screenshot or a static image.
Portfolio awareness — either pulled live from a real (or Plaid-sandbox) brokerage account, or just told to the agent out loud ("I own 10 shares of AAPL") and persisted.
Calendar — upcoming earnings for your holdings, FOMC rate-decision dates, anything you've told it to do on a future date, and recent decisions you've already made, all merged into one on-screen view, past and future together.
Proactive outreach: a background monitor checks the market and calls you on its own when something hits your portfolio — a big drop, an all-time high, a holding's earnings tomorrow, a Fed decision — each thing only fires once.
Action + confirmation loop: say "I want to buy NVDA next week" and it (a) puts it on the calendar, (b) texts you a confirmation via iMessage, speaking as Sid, not a bot. Dated decisions go through schedule_action; undated ones ("just holding everything") go through log_decision.
Point the camera at something and ask about it — a holdover generic vision capability from the original coaching idea, still wired in.

How it's built (architecture)

Call plumbing: Chert gives the agent a real phone line reachable over FaceTime/iMessage. A LiveKit room is the actual meeting place: the agent joins as a participant, and Chert's bridge dials the caller's phone and bridges the FaceTime call's audio/video into that same room. Outbound calls (the agent calling the user) go through Chert's /calls API with a LiveKit participant token; inbound calls route the same way in reverse.

Voice: Google Gemini Live (gemini-3.1-flash-live-preview) handles the realtime conversation — speech in, speech out, function/tool calling — via @livekit/agents-plugin-google. (An earlier dated preview model started failing outright with a "CONTENT_TYPE_AUDIO is not supported" error; swapped to the 3.1 live preview as Google's documented replacement.)

Video / on-screen UI — the interesting part: there's no camera and no video file. Every frame the caller sees is generated by hand: a tiny custom 5x7 bitmap font and plain rectangle/line/dot drawing routines writing directly into a raw RGBA pixel buffer (chart.mjs) — no canvas library, no browser, no images. That buffer is handed to LiveKit's VideoFrame/VideoSource and published as a real video track at 15 fps through Chert's bridge into the FaceTime call. A screen is drawn once per state change and then just copied into every frame (with a few live bits redrawn each frame — a pulsing "LIVE" dot, a voice-level meter) so the 15fps loop never does heavy work on the event loop. Current resolution is 1280x720 (standard landscape 16:9 — portrait was tried and tested worse, see Challenges), rendered at full size then shrunk to 80% on a black backdrop so FaceTime's own edge-cropping eats into margin instead of content.

Market data, with a resilient fallback chain: quotes come from Finnhub (free tier) when a key is set, falling back to Yahoo's public chart endpoint. Chart history comes from Yahoo's chart API first; if that fails (rate-limited, down, etc.) it falls back to Alpha Vantage (a free-tier keyed JSON API, covers intraday too); a third fallback to Stooq exists in code but turned out to be blocked by a JS proof-of-work bot-check that a server-side fetch can't solve, so it's effectively inert (fails closed, harmless). Finnhub's free tier explicitly blocks stock candle/history endpoints (403), confirmed live during testing — that's why Alpha Vantage is the real second line of defense. Everything is cached in memory. Plaid sandbox mutual-fund-style tickers (e.g. CAMYX, MIPTX — 4-5 letters ending in X) are detected by pattern and skip straight past Finnhub (which 403s on them anyway) to a clear spoken reason instead of a dead error.

Portfolio: either synced live from a real or Plaid-sandbox brokerage (npm run link opens a Plaid Link flow once; after that the agent can pull current holdings on command), or built up conversationally — the caller just states a position out loud and it's persisted, and that sticks even across a later Plaid re-sync.

Scheduling / proactive calls: a separate monitor.mjs process polls the market on an interval and places an outbound call through the same placeCall() path when a real trigger fires (dip, all-time high, a holding's big move, upcoming earnings, an FOMC date) — each trigger fires at most once (tracked in local state). There's also a manual npm run call -- --reason <dip|ath|move|earnings|fomc> for forcing a demo call outside of real market conditions.

Outbound texting: outbound.mjs also wraps Chert's /shared-messages API for iMessage, used by schedule_action to text a confirmation whenever the caller schedules a dated decision. Note: on Chert's free trial, a cold outbound text only works as a reply within 24h of the phone's last text to the line — a paid Shared/Dedicated line removes that restriction.

Built with

Chert (FaceTime/iMessage agent platform) · LiveKit Agents (@livekit/agents, @livekit/rtc-node) · Google Gemini Live (realtime voice) · Google Gemini vision (camera Q&A) · Plaid (brokerage sync, sandbox + production) · Yahoo Finance chart API · Finnhub · Alpha Vantage · Node.js, hand-rolled RGBA pixel rendering (no canvas/graphics library) · plain JSON file persistence (no database)

Challenges we ran into

Hand-rendering video from scratch. No canvas or image library — the entire UI (charts, portfolio table, calendar, cards) is drawn pixel-by-pixel into a raw buffer with a custom bitmap font, published as a LiveKit video track.
Display sizing/orientation. A portrait-oriented video buffer got pillarboxed into a small centered box on the caller's actual phone screen instead of filling it — likely because the FaceTime bridge expects/normalizes to a landscape canvas. Reverted to landscape (1280x720, doubled from an initial 640x360 pass for sharper text) and added an intentional 80%-scale-down-on-black-backdrop so the bridge's own edge-cropping can't eat real content.
Data provider throttling, live, mid-demo. Yahoo's public chart API started 429-rate-limiting the team's home network after repeated testing — confirmed with a direct curl test returning 429 for an unauthenticated request to www.google.com-adjacent endpoints wasn't the issue, it was IP-specific to Yahoo. Built a provider fallback chain (Yahoo → Alpha Vantage → Stooq) rather than depending on one unauthenticated source. Also discovered Finnhub's free tier flatly blocks historical candles (403) and that Stooq's endpoint is behind a client-side JS proof-of-work check that a server fetch can't pass — both ruled out live, during testing, by direct curl checks rather than guesswork.
Mutual-fund tickers from the Plaid sandbox (e.g. CAMYX) aren't carried by either free data provider — detected by ticker pattern and given a clear spoken explanation instead of a silent failure.
Chert FaceTime trial minutes. The project's FaceTime trial allotment hit zero mid-testing (credit_exhausted, resets per Chert's billing cycle) — a hard constraint to flag for the live demo, worth checking Chert's billing page for options if it's not reset by demo time.
Persona drift. The default agent persona initially hedged with "as an AI I can't advise you" — rewrote the system prompt to give it a name, a point of view, and explicit instructions to never hide behind being an AI, while still being clear that executing trades is the caller's own call.

Status as of tonight

Core loop (call in, ask about a stock, see a live chart, ask about your portfolio, schedule a decision, get a text confirmation) is implemented and has been exercised live. Outbound/scheduled calling is blocked until Chert's FaceTime trial minutes reset (2026-11-16T00:00:00Z per the API's error response) or the project is upgraded — inbound calls may or may not share that same limit, worth testing directly before assuming either way.