Juli AI

Your AI executive assistant that makes calls, manages your calendar, remembers everything—and gets smarter with every integration

Comment

Building Juli: When Tools Finally Caught Up to the Vision

The Year-Long Wait

I’d been sitting on this idea for over a year. Not just an AI assistant that suggests, but one that acts—makes the call, books the meeting, remembers everything. My notebooks were overflowing with architectures, flows, and big plans. But the tech just wasn’t there yet.

Then Bolt.new dropped. Suddenly, I could prototype at the speed of thought.

Two Weeks of Madness

When I finally started building, I lost all sense of balance. 16-hour days turned into 20-hour marathons—not because I had to, but because I couldn’t stop. Every breakthrough opened a new rabbit hole.

The architecture that emerged? Beautiful chaos. Let Claude orchestrate, but give it real tools. No more fake function calls—real APIs, real actions, real-world results.

MCP Gateway: The Seventh Circle

Let’s talk about the MCP Gateway: the seventh circle of engineering hell.

Day 4: “This’ll be easy. Just proxy some tools.”
Day 5: Tools not registering—why?
Day 6: Gateway silently dropping malformed requests.
Day 7: Ripped out and rewrote the discovery system at 3 AM.
Day 8: Learned about secret, undocumented timeouts.
Day 9: Built a health check system just to know when things broke.

MCP Gateway became my arch-nemesis. Every time I thought I understood it, it changed the rules—rate limits that don’t exist in the docs, registration flows that work locally but break in Docker, failures that look like success. Engineering gaslighting at its finest.

UX Engineering is Just… More Hell

Nobody tells you this: Making an invisible user experience is 10x harder than building the backend.

  • The Credential Problem: Users can’t manage ten API keys. Built a credential injector that dynamically grabs the right keys at runtime. Took three days and a headache.
  • The Waiting Problem: Nobody wants to watch a spinner while Juli makes a call. Built a whole background task system with live updates. You make a request, get on with your life, and Juli pings you when it’s done.
  • The Context Problem: Claude needs history, preferences, state—without dumping everything into the prompt every time. Built a context injection system that pulls only the relevant memories. Still, sometimes a simple task can take a couple minutes. Not ideal, but it's the reality of coordinating multiple live services.
  • The Error Problem: Users don’t care why something failed—they just want options. Spent 48 hours building graceful degradation. If calendar fails, Juli offers alternatives. If phone fails, it drafts an email.

The 3 AM Revelations

  • Night 7: Realized the memory system could store encrypted credentials—no new tables, just special tags.
  • Night 10: Discovered you can invoke tools mid-stream with the right event hooks. Rebuilt the whole response handler.
  • Night 12: Finally cracked the ContextForge “virtual server” idea. Now, instead of connecting to every MCP server, clients hit a unified endpoint. Clean. Unified. Scalable.

Hidden Complexity (A Partial List)

  • Dynamic Tool Discovery: On every startup, Juli discovers available tools. Built a caching system that refreshes every five minutes—because stale tools are worse than none.
  • Tool Pattern Matching: “Call mom” could map to anything. Created a pattern matcher with 200+ regexes mapping intent to provider. Now it just works.
  • Async Everything: Users don’t wait. Orchestrated background tasks, concurrency, retries, failures. Juli can juggle five things per user at once.
  • State Management: Between Redis, Postgres, and in-memory caches, keeping state consistent was like juggling flaming swords. Built a sync layer that keeps everything in line—even across restarts.

What Actually Shipped

After 300+ hours and more pivots than I can count, here’s what made it:

  • Phone Calls That Actually Work: Deep Vapi integration, robust enough for real conversations.
  • Calendar Magic: Not just CRUD—conflict resolution, smart scheduling, the works.
  • Memory That Matters: Vector-indexed, semantic search, actually useful recall.
  • Background Autonomy: Juli keeps working, even while you sleep.
  • Infinite Extensibility: Drop in a new MCP tool and Juli can use it immediately.

The Real Challenge

The hardest part? Making all that complexity invisible.

User says, “call the dentist.” Juli:

  1. Checks your calendar for conflicts
  2. Recalls your dentist’s number from last time
  3. Knows you prefer afternoon appointments
  4. Makes the call
  5. Adds it to your calendar
  6. Updates you when it’s done

That’s six service calls, three AI decisions, and two background processes—boiled down to: “I’ll handle that for you.”

Why It Matters

We’ve built chatbots that talk and agents that pretend to act. Juli Brain 3 actually does things. It’s the difference between an assistant who says “you should call your mom” and one that dials the number for you.

The future isn’t more chat. It’s autonomous action.

Accomplishments That We’re Proud Of

  • Proved out true multi-step AI autonomy: Juli acts, not just suggests.
  • Built robust, extensible integrations with phone, calendar, and memory—no more “demo-only” workflows.
  • Made complexity disappear for the user: seamless, natural experience, no matter how many services are involved.
  • Survived (and mostly tamed) the notorious MCP Gateway and learned to love the battle scars.

What We Learned

  • The tech is finally catching up to agentic dreams—if you’re willing to wrestle the bleeding edge.
  • User experience is 90% of the battle. It’s not enough to be powerful; it has to feel effortless.
  • Background tasks, smart context, and graceful error handling are non-negotiable.
  • Resilience matters. Your code, your architecture, and your mindset all need to survive chaos—especially at 3 AM.

What’s Next for Juli AI

  • Keep building, keep refining. The vision is still bigger than what’s shipped.
  • Targeting a real, official release by the end of summer.
  • Integrations on deck: Uber Eats, email, Slack, and more MCP-powered workflows.
  • More autonomy, more real-world actions, less “maybe someday” and more “it just works.”
  • And, of course, plenty more sleepless nights and coffee.

The Marathon Continues

Two weeks. Seven services. Fifteen MCP tools. One obsessed engineer.

Juli isn’t just another AI project. It’s what happens when you refuse to accept “that’s not possible” and push through every 3 AM debugging session, every gateway timeout, every UX hairball.

Because the best engineering happens when you care too much to quit.

Built with sleepless nights, endless coffee, and the stubborn belief that AI assistants should actually assist.

Built With

  • anthropic-claude-api
  • auth0
  • black
  • bolt.new
  • contextforge-mcp-gateway
  • cryptography
  • docker
  • docker-compose
  • fastapi
  • google-calendar-api
  • jwt
  • model-context-protocol-(mcp)
  • mypy
  • node.js
  • oauth2
  • openai-embeddings
  • openmemory-(mem0)
  • opentelemetry
  • postgresql
  • prometheus
  • pydantic
  • pytest
  • python
  • python-jose
  • qdrant
  • redis
  • sentry
  • server-sent-events-(sse)
  • sqlite
  • structlog
  • typescript
  • vapi
Share this project:

Updates