Jerry AI

Inspiration

Phone assistants can set timers. None of them can actually do things — fill out the job application, check your feeds, dig through the web, work your actual computer. We wanted an agent that lives on your own PC, reasons on a local model (no cloud API anywhere in the loop), and is controlled from the one device you always have: your phone. But handing an AI your files, your browser, and your logins means safety can't be a feature — it has to be the architecture.

What it does

Jerry is a local-first AI agent you text like a person, through Telegram. From your phone you can: search the web and get sourced answers, search YouTube and Instagram, read your feeds with proof screenshots, manage files, and even apply to jobs — Jerry fills multi-step applications using your saved profile and resume. The rule that makes it trustworthy: anything irreversible triggers an Approve/Deny card on your phone showing exactly what's about to happen — the exact post text, the exact form values, the exact file being deleted. Deterministic rules decide what's dangerous, never the model's judgment. Every action lands in an append-only audit log.

How we built it

Python end to end. A Telegram bridge (python-telegram-bot) runs everything on one asyncio loop — approval cards resolve asyncio.Futures, so the agent literally suspends mid-task until you tap a button. Reasoning is local Ollama (qwen2.5) driving a hardened tool-calling loop: lenient JSON extraction (small models wrap tool calls in junk), repeat-call guards, and error-as-tool-result recovery so nothing crashes the loop. Browsing is Playwright pinned to a single dedicated thread (Windows + async + greenlets is unforgiving). Job applications use browser-use so an AI actually reads the page instead of guessing CSS selectors — with our own click_final_submit tool injected so the final click can only happen after a human approves it. A JSON job store guarantees we never double-apply.

Challenges we ran into

• Small local models are chaotic. The 7B model wandered off to arXiv mid-task; the 32B model didn't fit in VRAM and timed out on every call. We stopped assuming a perfect model and engineered around an imperfect one.
• Selector-guessing doesn't survive reality. Our first form-filler guessed CSS selectors and face-planted on a real Workday application — that failure drove the browser-use rewrite.
• DuckDuckGo CAPTCHA'd our headless browser — literally "select all squares containing a duck." We refused to bypass it (policy: never solve or evade bot checks) and switched to a legitimate programmatic search library instead. A paused agent is correct behavior, not a failure.
• We learned the credential lesson the hard way: a password pasted into chat ended up in a plaintext audit log. That mistake shaped our domain-scoped sensitive-data design, where the model only ever sees placeholders — never real values.
• Platforms redesign under you. Instagram's hashtag pages silently use a different DOM than the home feed. Our policy: fail loudly with a screenshot sent to your phone, never guess.

Accomplishments that we're proud of

The safety spine. Every action audited, gating decided by code instead of model vibes, and a human checkpoint that cannot be skipped — even a pre-authorized task still requires a fresh approval before a real job application is submitted. And the moments it all clicked: finding real World Cup posts on Instagram, briefing us on RAG with cited sources, and walking a real ATS application to the submit button — all from a phone.

What we learned

Agent reliability is a systems problem, not a prompt problem. Ground truth beats memory: introspect the installed API, read the audit log, look at the failure screenshot. And trust is the actual product — an agent that asks before it acts beats one that's occasionally brilliant and occasionally leaks your password.

What's next for Jerry

A stronger model path for browser-use (bigger local models on better hardware, or an optional cloud fallback), sending files to the bot from your phone (resume in chat → saved and used automatically), voice-note tasks, and extending the gated-action framework to email and calendar — same rule everywhere: Jerry drafts, you approve.

Built With

• asyncio
• browser-use
• duckduckgo
• httpx
• ollama
• playwright
• pytest
• python
• python-telegram-bot
• qwen2.5
• telegram-bot-api
• windows
• yaml
• youtube-data-api