ModelForge

Inspiration

Every AI startup faces a major dilemma: skyrocketing API bills versus the fear of production downtime.

Early-stage AI companies spend thousands of dollars each month on frontier models like GPT-4o. When new, open-weight, or cost-effective models such as Llama 3.2, Mistral, or DeepSeek become available, switching could cut API bills by 70% to 90%. Yet engineering teams hesitate to migrate because foundation models are unpredictable. A model switch can silently break production applications: tool-calling parameters shift from integers to strings, prompt formats drift, and enums fail validation.

Current practices boil down to either manual vibe checking or deploying blind and hoping nothing breaks. We built ModelForge to give founders and engineering teams the confidence to optimize their unit economics: an autonomous, self-healing migration rehearsal agent that stages model transitions in isolated sandboxes, auto-patches incompatibilities, and safely executes canary rollouts with human approval.

What it does

ModelForge automates the entire LLM migration lifecycle from end to end:

• Static Analysis and Schema Extraction: Scans the target codebase using AST and pattern analysis to map SDK imports, prompt locations, and tool call schemas.
• Empirical Baseline Benchmarking: Executes a deterministic benchmark against the live application to record ground-truth accuracy, p95 latency, and cost per request.
• Isolated Sandbox Staging: Clones the application into an isolated workspace sandbox, ensuring the production codebase remains completely untouched and immutable.
• Candidate Rehearsal and Differential Matrix: Boots the application with the new candidate model inside the sandbox, runs the identical benchmark suite, and compares results deterministically without relying on LLM self-evaluation.
• Autonomous Remediation: When tool-calling schema drift is diagnosed, ModelForge automatically patches the sandbox code to route complex tool calls to the incumbent model and general queries to the cheaper candidate model, re-testing until reaching 100% quality parity.
• Human-Gated Canary Deployment: Generates an immutable canary routing manifest (such as a 90/10 traffic split). Gated by a cryptographic HMAC-SHA256 human approval token, production routing cannot mutate until an engineer explicitly approves it on the dashboard.

How we built it

ModelForge was engineered as an architecture-first platform with modular subsystems:

• Executive Agent and MCP Architecture: Built on TrueForge with dual Streamable HTTP Model Context Protocol servers:
  • rehearsal-mcp (Port 8951): 10 inspection, sandbox copy-on-write, and evaluation tools.
  • gateway-mcp (Port 8952): 4 routing, canary, and human-approval gated tools.
• Deterministic Evaluation Engine: Pluggable adapter hierarchy (DemoAdapter, OpenAICompatibleAdapter, CustomRestAdapter) calculating p50, p95, and p99 latency percentiles, cost savings, and strict Zod schema validation.
• Durable State Engine: Powered by Node 22 native SQLite in WAL mode with foreign keys and atomic transactions for instant crash recovery.
• Cryptographic Guardrails: HMAC-SHA256 tokens binding the session ID, canary configuration, and manifest SHA hash to ensure zero-bypass human authority.
• Rigorous Testing: Validated by a comprehensive test suite of 180 passing tests out of 180 across 13 Vitest suites.

Challenges we ran into

• Separation of Ground Truth vs. Agent Reasoning: LLMs have significant self-grading bias. If an agent self-evaluates candidate models, it hallucinates success. We solved this by strictly isolating reasoning from evaluation: an independent, deterministic evaluation engine calculates pass or fail verdicts, leaving the LLM to act strictly as the orchestrator.
• Guaranteed Sandbox Isolation: Early prototypes ran the risk of polluting the host repository. We implemented copy-on-write directory trees with environment sanitization and full SHA-256 tree verification to ensure the source repository is never mutated during rehearsal.
• Preventing Prompt Injection Bypasses: To prevent an adversarial prompt from commanding the agent to push changes directly to production, the human approval gate was compiled into the TypeScript server layer. Without a cryptographically signed token, the server throws an ApprovalRequiredError (HTTP 403) that no prompt can bypass.
• Schema Drift Remediation: When a cheaper model fails specific tool schemas, brute-force migration fails. We engineered a pattern diagnosis engine that automatically detects schema mismatch and synthesizes hybrid routing patches to preserve reliability.

Accomplishments that we're proud of

• 100% Test Suite Verification: 180 out of 180 unit and integration tests passing across 13 test suites.
• 93% Cost Reduction with Zero Regressions: In our live benchmark demonstration, ModelForge safely transitioned an application from an expensive baseline to a hybrid configuration, slashing token costs by 93% while maintaining 100% test quality.
• Absolute Source Repository Immutability: Successfully verified through directory-level SHA-256 tree hashes that the source codebase remains completely untouched throughout rehearsal and patching.
• End-to-End Autonomous Pipeline: Delivering a complete, working solution that connects code analysis, ephemeral sandboxing, deterministic differential evaluation, and cryptographic canary deployment in a single cohesive platform.

What we learned

• Mission-critical AI agents require deterministic architectural guardrails, durable state persistence, and compiled boundary checks, not just system prompts.
• Model migration in production is rarely binary; hybrid routing unlocks the best balance between frontier model accuracy and open-weight cost efficiency.
• Designing pluggable evaluation adapters allows instant testing across different providers (OpenAI, Groq, local models) without changing application code.
• Human-in-the-loop workflows are most effective when paired with cryptographic non-repudiation, giving platform operators complete control over production changes.

What's next for ModelForge

• Production Gateway Connectors: Expanding beyond file-backed canary simulation to direct integrations with Envoy, Traefik, Kong, and LiteLLM.
• Firecracker MicroVM Sandboxing: Adding support for untrusted multi-tenant codebases with hardware-isolated microVMs.
• Continuous Integration GitHub Action: Packaging ModelForge into a GitHub Action that automatically rehearses and benchmarks every model change directly in a pull request comment.
• Multi-Model Orchestration: Supporting multi-candidate differential testing to simultaneously evaluate multiple providers and automatically choose the optimal price-to-performance configuration.

Built With

• ai-agents
• canary-deployment
• ci/cd
• devops
• groq
• hmac-sha256
• llama
• llm
• model-context-protocol-(mcp)
• node.js
• openai-api
• rest-api
• sqlite
• trueforge
• typescript
• vitest
• zod