FERMIX

• 

Inspiration

Fermix started from a simple frustration: most AI assistants can answer questions, but they do not truly belong to the user. They live inside someone else’s product, forget important context, depend on a single model, and rarely have enough access to complete real work.

I initially tried combining existing agent tools into a personal workflow. It worked until changes in third-party services broke the setup and exposed how fragile the entire system was. Around the same time, I was recovering from an injury and spending most of my time at home. I used that period to rethink the project from the ground up.

Instead of building another chat interface, I wanted to create a personal AI that runs on the user’s machine, understands how they work, reaches them through the apps they already use, and can safely take action on their behalf. It should improve over time, work with different model providers, and remain under the user’s control.

That became Fermix.

What it does

Fermix is an open-source personal AI agent that runs locally as an always-available daemon.

Users can interact with it through the command line, messaging platforms, or a real-time voice companion. It can use tools, search the web, work with files, access connected services, run scheduled jobs, coordinate parallel subagents, and perform approved actions on the computer.

Its memory system is designed to learn more than isolated facts. Fermix can gradually understand the user’s preferences, working style, communication patterns, and recurring routines. Users can also directly shape its personality and behavior through its configurable soul.

Fermix supports multiple model providers instead of locking the user into one ecosystem. The latest release supports the GPT-5.6 family, including Sol, Terra, and Luna, along with Grok 4.5. Its live voice mode uses GPT-Realtime-2.1 and can invoke Fermix tools or control the computer during a continuous conversation.

Fermix can also communicate through Telegram, Discord, WhatsApp, Signal, Slack, and other channels. The same agent, memory, tools, permissions, and scheduled work remain available regardless of where the conversation begins.

The core principle is simple: your machine, your models, your keys, and your rules.

How we built it

Fermix is built natively in Elixir because a personal agent is not a single request-response application. It is a long-running, concurrent system that must manage conversations, tool execution, scheduled jobs, external services, model streams, messaging channels, and failures at the same time.

Elixir and the BEAM provide a strong foundation for that architecture. Fermix uses isolated processes and supervision trees so that individual tools, plugins, model connections, or MCP servers can fail without bringing down the entire agent.

The main agent runs as a local daemon. Around it, I built systems for memory, scheduling, model routing, tool discovery, plugin authentication, messaging adapters, sandboxed execution, and parallel subagents.

Tools execute through controlled boundaries rather than receiving unrestricted access to the host machine. File paths are validated against approved roots, secrets are redacted from logs, dangerous Git protocols are blocked, and background computer-control sessions fail closed.

For real-time voice, I built FermixPet, a dedicated macOS companion that can continuously listen, speak, invoke tools, and display the agent’s current state. Its animation follows actual audio playback rather than model generation, so the interface reflects when Fermix is truly speaking.

Computer Use is powered by Compux, a separate Elixir-native helper distributed as signed binaries for supported macOS and Linux systems. Fermix downloads and verifies the helper before enabling it. Computer control remains experimental and disabled by default.

The project is open source and designed around bring-your-own-model access. Users can choose their providers, keys, models, voices, reasoning levels, channels, and permissions during setup.

Challenges we ran into

The hardest part was not generating intelligent responses. It was making an agent dependable enough to remain running on a real computer.

Long-running agents encounter failures that normal chat applications rarely handle. Keychain access can block. A tool can hang and leave child processes running. An MCP server can repeatedly crash. A laptop can sleep during a scheduled delivery. A daemon restart can create a relaunch loop. A host dependency can freeze the setup page.

These problems often appeared only under real operating conditions, not during development.

I had to build strict timeouts, bounded retries, process-tree termination, graceful daemon restarts, MCP server quarantine, wake-from-sleep recovery, and defensive startup behavior. A failed tool or integration should degrade one capability rather than destabilize the entire agent.

Security was another major challenge because Fermix can access files, execute commands, use credentials, and potentially control the computer. Every new capability increases the importance of permission boundaries. I had to balance useful autonomy with explicit user control, sandbox restrictions, secret handling, signed helper verification, and fail-closed behavior.

Voice and computer use introduced additional complexity. A live conversation must remain responsive while tools execute, results stream back, audio plays, and the computer may also be under agent control. The agent must never claim that an action succeeded before receiving a real tool result.

Performance also mattered. Loading credentials sequentially made startup unnecessarily slow, especially when the macOS keychain was degraded. Parallelizing keychain resolution reduced the normal path from roughly 600 milliseconds to 71 milliseconds and reduced a degraded case from around 45 seconds to approximately 6 seconds.

Accomplishments that we're proud of

The main accomplishment is that Fermix has grown from an experimental personal workflow into a functioning local agent platform.

It can maintain memory, work across multiple communication channels, run scheduled tasks, coordinate parallel agents, use external tools, hold real-time voice conversations, and perform approved computer actions through one shared system.

I am particularly proud that the architecture does not depend on a single model provider or hosted agent platform. Fermix supports different model families while keeping the user’s tools, memory, permissions, and interface consistent.

The project has also developed a strong safety foundation. Tool execution is sandboxed, credentials are redacted from logs, computer control is isolated behind a verified helper, unsafe Git protocols are blocked, and failing background systems are contained instead of endlessly restarting.

Fermix has passed its first 100 downloads through Homebrew, gained early users and contributors, and continued shipping through a regular release cadence. Each release has been driven by real failures, performance measurements, and direct use rather than a static feature roadmap.

The latest release brings support for GPT-5.6 Sol, Terra, and Luna, Grok 4.5, upgraded GPT-Realtime-2.1 voice, voice-driven computer use, stronger sandboxing, safer secret handling, faster startup, and significant improvements to scheduled jobs, tools, daemon restarts, and MCP reliability.

What we learned

The biggest lesson was that building a useful agent is primarily a systems-engineering problem.

Model intelligence matters, but it is only one part of the experience. A personal agent also needs reliable state, memory, permissions, scheduling, observability, recovery, and predictable tool execution. A brilliant response is not valuable if the action fails silently or the process crashes overnight.

I also learned that autonomy should not mean unrestricted access. Users need to understand what the agent can do, define its boundaries, and interrupt it immediately. Fermix is designed to act by default inside approved boundaries while returning control as soon as the user intervenes.

Another lesson was that memory should help the agent acquire taste, not simply accumulate data. The goal is not to store every conversation. The goal is to preserve useful preferences, recurring patterns, decisions, and context that make future interactions better.

Voice also changes how an agent must behave. Spoken responses need to be shorter, more direct, and free from filler. The model should lead with the answer, avoid unnecessary closing questions, and never fabricate tool outcomes. These principles are now part of Fermix’s real-time voice prompt design.

Finally, I learned that benchmarks must include operational behavior. Fermix now measures startup paths, tool latency, failure recovery, and other system-level behavior. These internal benchmarks have exposed bottlenecks that would not appear in model evaluations alone.

What's next for FERMIX

The next stage is to make Fermix feel less like software the user opens and more like a dependable personal intelligence layer across their devices.

A native iOS companion is planned so users can reach their Fermix instance directly from their phone instead of depending only on third-party messaging applications.

Real-time multimodal interaction is another major focus. Fermix can currently listen, speak, use tools, and control the computer. The next step is continuous visual understanding so it can see what is happening on the screen and respond without requiring the user to narrate every detail.

The memory system will continue evolving toward deeper personalization, better taste acquisition, stronger user controls, and clearer visibility into what the agent has learned.

Computer Use will move from an experimental feature toward a more reliable and permission-aware system, with stronger interruption handling, clearer action visibility, and broader platform support.

I also plan to expand the contributor community, improve documentation and extension APIs, add more integrations, strengthen automated testing and pull-request review, and continue hardening the security boundaries around plugins and tools.

The long-term goal is to give every user a personal AI that is actually theirs: one that understands them, improves with them, works across their digital environment, and remains under their control.

Built With

• chatgpt
• claude
• codex
• elixir
• llm
• rust