Inspiration
What it does
How we built it
Challenges we ran into
Accomplishments that we're proud of# Foundry Core
One runtime. Any model. Any plugin. Any workflow.
Foundry Core is a multimodal AI runtime for embedding production-grade AI workflows into applications, services, and developer tools.
It accepts text or audio, routes each request to the appropriate plugin and action, selects the configured model, executes the workflow, and returns structured results and runtime events. Applications can let Foundry Core choose the right action automatically or invoke a specific action directly.
Foundry Core began before OpenAI Build Week. It was not created as a one-week prototype: I have been building it as infrastructure for real business workflows. Build Week gave me a reason to turn that work into a focused demonstration and show how GPT‑5.6 can operate as a powerful reasoning and execution layer inside a model-independent runtime.
Inspiration
Most AI integrations begin with a single API call. As the product grows, that call becomes a collection of prompts, tools, routing rules, provider-specific code, retries, model settings, and business logic spread throughout the application.
Adding a second model makes the architecture harder. Adding local inference, audio, plugins, or human approval steps makes it significantly harder again.
I wanted one clear boundary between an application and its AI capabilities:
- The host application owns users, data, UI, billing, durable storage, and business operations.
- Foundry Core owns model orchestration, routing, plugins, actions, structured outputs, tools, checkpoints, and execution events.
The goal is to make AI capabilities portable. A workflow should not have to be rewritten because it moves from a cloud model to a local model, from a CLI to a desktop application, or from one programming language to another.
What it does
Foundry Core provides a single runtime for several types of AI execution.
Multimodal input
The runtime can accept text directly or process audio through a local Whisper provider. Once audio is transcribed, it enters the same routing and plugin execution flow as text.
This means an application does not need a separate architecture for voice-driven workflows.
Intelligent or explicit routing
A host can invoke a known plugin action directly when the desired capability is already clear.
Alternatively, Foundry Core can inspect the request and choose the appropriate plugin and action automatically. Routing is treated as a dedicated runtime responsibility rather than prompt logic embedded throughout the host application.
Plugin-based capabilities
Every plugin declares the actions it can perform, the input and output contracts for those actions, the tools it requires, and the model slots it uses.
Foundry Core supports two primary plugin forms:
- Wasm plugins for capabilities that require executable code.
- Pipeline plugins for code-free workflows composed of Markdown skills, schemas, and ordered model steps.
Pipeline workflows can pause at checkpoints, request clarification or approval, and later resume from serializable state.
Multiple local and cloud models
Foundry Core can work with multiple model providers within the same runtime, including remote APIs and local GGUF models through llama.cpp.
The host can configure separate models for routing and default execution. Plugins can also declare their own model slots, allowing a specific action or pipeline step to use a different model.
For example, a lightweight local model can handle routing while GPT‑5.6 performs the workflow steps that require deeper reasoning and higher-quality structured output.
Embeddable runtime
The core is written in Rust and can run:
- as a Rust library;
- through a stable C ABI and JSON boundary;
- as a shared native library;
- or as a CLI/subprocess worker.
This makes the same runtime usable from Go, Python, Flutter/Dart, Swift, Kotlin, and other languages capable of calling a native library or managing a subprocess.
How I built it
Foundry Core is organized as a Rust workspace with separate components for:
- the core orchestration runtime;
- shared plugin contracts;
- the native FFI boundary;
- the CLI and process-host interface;
- local LLM execution through
llama.cpp; - local speech recognition through Whisper;
- and the
.aipplugin packaging tool.
Model providers implement a common runtime contract. Configuration identifies models by stable machine names, while routing, default execution, plugins, and individual actions refer to model slots instead of hard-coded providers.
Plugins are distributed as .aip packages. A package can contain a Wasm module, a pipeline definition, Markdown skills, schemas, and other resources. Before activation, Foundry Core validates package integrity, runtime compatibility, plugin requirements, and Wasm constraints.
Execution is observable through structured events covering transcription, routing, model calls, tools, plugin actions, pipeline steps, checkpoints, completion, and failures. This allows a desktop application to display progress while a service can send the same events to logs or monitoring systems.
For the Build Week demonstration, GPT‑5.6 is connected through the model-provider layer rather than embedded into one hard-coded workflow. This lets it participate in routing or power selected plugin actions while preserving the same runtime contracts used by other remote and local models.
Challenges
The hardest part was defining the boundary between the runtime and the host application.
It is tempting to make an AI framework own everything: databases, queues, user state, secrets, retries, and business actions. That quickly turns the framework into an inflexible application backend. Foundry Core instead keeps execution state explicit and returns serializable checkpoints, outputs, and events for the host to persist.
Another challenge was making local and remote models fit the same architecture. Local inference introduces model files, native acceleration, context limits, memory pressure, resident model state, and concurrency constraints that do not exist in the same form with hosted APIs.
Plugin execution introduced a different set of problems: package integrity, version compatibility, permissions, resource limits, predictable schemas, and safe execution of third-party code. Wasm isolation and declarative pipeline packages provide two different solutions for two different kinds of plugins.
Finally, AI routing cannot be treated as unrestricted free-form generation. The router must select only from registered capabilities and return a result the runtime can validate. Designing strict contracts around probabilistic model behavior was one of the most important parts of the project.
What I learned
The biggest lesson was that the model call is often the simplest part of an AI product.
The difficult work is everything around it: capability discovery, routing, structured data, tool boundaries, configuration, security, observability, checkpoints, local inference, and integration with existing applications.
I also learned that model independence does not mean treating every model as identical. Different actions need different cost, latency, privacy, and reasoning characteristics. Model slots allow the runtime to preserve a stable workflow while still choosing the best model for each responsibility.
GPT‑5.6 reinforced this architecture: a highly capable model becomes even more useful when it operates inside explicit actions, schemas, tools, and execution boundaries.
What’s next
Next, I want to expand Foundry Core with:
- more production plugins and reusable pipeline templates;
- higher-level SDK wrappers for additional languages;
- richer routing evaluation and model benchmarks;
- improved tracing, metrics, and developer tooling;
- more local inference backends and platforms;
- and a secure plugin ecosystem for distributing reusable AI capabilities.
The long-term goal is simple: applications should be able to add sophisticated AI workflows without rebuilding their architecture around every new model.
Log in or sign up for Devpost to join the conversation.