Inspiration

Most prompt tools I found started after the prompt had already been written. They could evaluate it, log it, or version it, but I was still responsible for guessing the wording, tweaking it, and trying to remember whether the latest edit had broken something that worked before.

In practice, my prompts were scattered across source code, configuration files, documents, and chat history. I kept copying them between places, running them manually, and comparing outputs by eye. Each prompt could use a different model, accept different inputs, and return a different structure, yet basic questions had no reliable answer:

• Which model does this prompt run on?
• What inputs does it need?
• What should it return?
• Which behaviors are non-negotiable?
• Who calls it?
• Did the last edit break anything?

That felt backwards. Applications treat prompts as executable instructions, but we often manage them as loose text. Anything that controls application behavior should have a defined contract and a repeatable way to verify that the contract still holds.

The idea behind Prompt Ninja was to invert the usual workflow: describe the behavior you expect and let the LLM engineer the prompt. The expectation becomes the source of truth, while the prompt text becomes an implementation that can be generated, tested, and safely rewritten as requirements change.

I looked for a lightweight, open-source tool built around that idea. The alternatives I found were either larger orchestration frameworks or hosted platforms that placed prompts behind someone else's service. I wanted something simpler: a versioned prompt artifact that developers own, commit, and test beside their code, with the same confidence and feedback loop that pytest gives ordinary Python behavior.

What it does

Prompt Ninja lets the LLM be the prompt engineer. You describe the behavior you need; it creates, tests, observes, and safely updates the prompt artifact.

1. Generate a prompt from an outcome

Install Prompt Ninja and describe what the LLM should achieve. You do not need to write the prompt instructions yourself.

pip install prompt-ninja

prompt-ninja generate \
    --goal "Extract every date an employee mentions in a chat message and return the dates as a list." \
    --output prompts/employee-dates.prompt.toml

Prompt Ninja turns that expectation into a versioned *.prompt.toml artifact. The generated file keeps the prompt and its operational contract together:

First, the output structure is defined once as a normal Pydantic model:

# my_app/models.py
from datetime import date

from pydantic import BaseModel


class EmployeeDatesResult(BaseModel):
    dates: list[date]

The prompt artifact references that model and declares each input's type, whether it is required, and any default value:

[metadata]
spec_version = "1.2"
name = "employee-dates"
description = "Extracts dates mentioned by an employee in a chat message."
version = "1.0.0"
output = "my_app.models.EmployeeDatesResult"

[llm_model]
provider = "openrouter"
name = "google/gemini-2.5-flash"

[prompt]
system = """
Extract only dates explicitly mentioned in the employee's message.
Do not infer or invent missing dates.
"""
user = """
Employee message: {{message}}
Locale: {{locale}}
Include weekday names: {{include_weekday}}
Ignore these dates: {{excluded_dates | json}}
"""

[[variables]]
name = "message"
type = "string"
required = true
description = "The employee's chat message."

[[variables]]
name = "locale"
type = "string"
required = false
default = "en-US"
description = "Locale used to interpret explicit dates."

[[variables]]
name = "include_weekday"
type = "boolean"
required = false
default = false
description = "Whether each result should include its weekday."

[[variables]]
name = "excluded_dates"
type = "list[date]"
required = false
default = []
description = "Explicit dates that should be omitted from the result."

[[tests]]
name = "extracts multiple dates"
variable.message = "I can work on July 24 and July 29."
expected_output = "An object whose dates field contains July 24 and July 29."

[[tests]]
name = "does not invent dates"
variable.message = "I will confirm my availability tomorrow."
expected_output = "An object with an empty dates field because no explicit date was provided."

Typed inputs without repetitive conditionals

Prompt Ninja applies these declarations before the provider is called:

• required = true rejects a missing message immediately.
• Optional values use their declared defaults, so callers do not need an if/else for every setting.
• string, integer, number, boolean, date, datetime, dict, JSON, Pydantic models, and list[TYPE] inputs are validated and rendered appropriately.
• Lists can be rendered naturally, as JSON with {{value | json}}, or as comma-separated text with {{value | csv}}.
• Invalid defaults and incorrectly typed test fixtures fail when the artifact is validated, not later in production.

The caller can provide only the required value:

from prompt_ninja import PromptNinja

prompt = PromptNinja.from_file("prompts/employee-dates.prompt.toml")
result = await prompt.run_openrouter({
    "message": "I can work on July 24 and July 29."
})

# result is already a validated EmployeeDatesResult instance.
print(result.dates)

There is no need to repeat JSON-format instructions throughout the prompt or manually decide where each output field should be written. Prompt Ninja converts the Pydantic model to a JSON Schema, adds that contract to the model request, parses the response, and validates the returned structure. A missing field, invalid date, wrong type, or malformed JSON fails the output contract instead of leaking into application code.

2. Test prompt behavior like code

The semantic tests live beside the prompt, so they can be committed and reviewed with the rest of the application. Run them with a pytest-like command:

prompt-ninja test --prompt prompts/employee-dates.prompt.toml

Prompt Ninja executes every [[tests]] case and uses an LLM judge to check the actual response against the expected behavior:

employee-dates
    PASS  extracts multiple dates      score 1.00
    PASS  does not invent dates        score 0.96

2 passed, 0 failed

This checks both structure and meaning. Every response must first validate as EmployeeDatesResult; then the semantic judge verifies that required facts remain present and behavioral constraints hold. Failures include a rationale plus suggestions for improving the prompt or test.

3. Observe real runs with hooks

Runtime hooks receive structured lifecycle events without changing the prompt caller. Different hooks can perform independent jobs after a response:

Hook	Observes	Example result	Caller impact
Semantic quality	Rendered prompt, input, and output	score: 0.94, with judge rationale	Non-blocking background evaluation
Usage and cost	Model and provider token counts	312 input, 48 output, $0.0007	Passive telemetry; no extra LLM call

The same mechanism can support logging, tracing, alerts, or automated repair while keeping those concerns separate from prompt behavior.

4. Update the prompt without rewriting its tests

Describe the behavior change in natural language:

prompt-ninja update prompts/employee-dates.prompt.toml \
    "Normalize explicit dates to ISO 8601, but preserve the rule that missing dates must never be inferred."

Prompt Ninja asks the LLM to revise the prompt implementation, preserves the existing test cases and pass threshold, and runs the complete semantic test suite against the candidate. The file is updated only if every protected test passes; otherwise, the original artifact remains unchanged.

5. Generate, test, and observe prompt artifacts in the Web UI

The Web UI turns a plain-language expectation into a tested Prompt Ninja artifact, provides a workspace for running and safely updating its contract, and shows runtime quality and cost through hooks. It is a visual demonstration of the same workflow exposed by the Python package and CLI, not a separate hosted prompt format.

Create your own prompt artifact

On the Board page, anyone can describe something they want to achieve with an LLM in ordinary language. They can also attach reference files and specify the expected output or important guardrails. No prompt-writing syntax is required.

The UI then walks through the complete generation process:

1. A requirements agent turns the request into a clear objective, inputs, expected output, and constraints.
2. Three creator agents independently propose prompt implementations.
3. A judge compares the approaches and synthesizes the strongest result.
4. The compiler validates the definition and packages the result as a Prompt Ninja artifact.
5. The user can inspect the generated prompt and download their own *.prompt.toml file.

The downloaded file is not just prompt text. It includes the model configuration, typed variables, required fields, defaults, output contract, test threshold, and generated semantic test cases. Users can commit it with their code and continue using it through the CLI or Python API.

Understand runtime hooks

The Hooks page demonstrates what can happen after an application runs a prompt. It explains the lifecycle in three steps: the prompt runs normally, hooks observe a structured event, and each hook performs its work separately without changing the caller's result.

The page includes two live examples:

• Semantic quality hook: receives the rendered prompt, input, and output, then asks a judge model to score the response and explain its decision. It runs in the background, so the application does not wait for the evaluation.
• Usage and cost hook: records the selected model, input tokens, output tokens, total tokens, and estimated cost. It is passive telemetry and does not make another LLM call.

After running the Board, visitors can open the Hooks page to see real evaluation records, judge rationales, token totals, and cost estimates from that run. Together, the Board and Hooks pages demonstrate both sides of Prompt Ninja: creating a prompt artifact before release and observing its behavior after it begins running.

How I built it

Prompt Ninja grew incrementally from that idea. The first version was a small tool that accepted a goal and asked an LLM to generate a prompt. Using it revealed the next problem: generated text alone was not enough. A usable prompt also needed a model, inputs, output structure, metadata, and configuration. Bringing those pieces together led to the human-readable *.prompt.toml format.

Once the prompt had a structure, I needed a way to know whether it still behaved correctly. I added semantic test cases directly to the artifact and built a pytest-like runner around them. That foundation made the next capabilities possible: typed variables, defaults, required inputs, Pydantic output contracts, deterministic validation, and natural-language updates that must pass the existing tests before replacing a prompt. Runtime hooks followed so applications could evaluate quality and track usage after deployment without coupling those jobs to the prompt call itself.

The Web UI began as a way to demonstrate generation, but it evolved into the Board of Prompts. Rather than asking one LLM for one prompt, the Board makes several agents work through the problem visibly: a requirements agent clarifies the goal, three creators propose different implementations, a judge synthesizes the strongest parts, and a compiler validates and tests the final artifact. The browser now brings generation, contract testing, protected updates, TOML downloads, and live hook activity into one workflow.

The backend is built with Python, FastAPI, Pydantic, Click, and the OpenAI-compatible Responses API through OpenRouter. TOML keeps artifacts readable and versionable, while Pydantic validates their structure, inputs, defaults, fixtures, and outputs. The React, Vite, and Chakra UI frontend exposes the same capabilities through the Board, contract workspace, and Hooks page.

I used Codex to generate most of the implementation code across both sides of the project. On the backend, Codex helped build the Python package and API, including the TOML models, validation, prompt runtime, CLI, semantic test runner, update safeguards, hooks, and FastAPI endpoints. On the frontend, I used it to implement the React and Vite application, including the Board of Prompts, contract workspace, TOML export flow, and Hooks page.

I directed that work rather than asking Codex to produce the entire application at once. I decided the product direction and implementation order, described the behavior and constraints for each capability, reviewed the generated code, ran the tests and frontend builds, and corrected the approach when it diverged from what I wanted. The loop was consistent: define the next capability, implement it with Codex, validate the result, tighten the requirements, and then move to the next layer.

Prompt Ninja also dogfoods its own approach. The prompts used by the Board, compiler, updater, repair flow, and semantic judges are themselves versioned *.prompt.toml artifacts with declared inputs, output contracts, and tests.

Challenges I ran into

One of the hardest ideas to communicate to Codex was the meta, dogfooding nature of the project. Prompt Ninja is a tool that uses prompts to generate, test, repair, and update other prompts, and those internal prompts should themselves use the Prompt Ninja format. Codex would sometimes treat them as ordinary Python constants or one-off strings, or move semantic behavior into Python instead of a versioned prompt artifact. I had to repeatedly clarify the boundary: prompt policy belongs in tested *.prompt.toml files, while deterministic guarantees such as schema validation, test preservation, and safe file promotion belong in Python.

Variable substitution was another difficult design problem. A simple string replacement works for text, but becomes inconsistent when values can be booleans, numbers, dates, lists, dictionaries, JSON, or Pydantic models. Optional variables, required variables, and defaults also created pressure to scatter conditional logic across every caller.

The trickiest part was moving safely from serialized values to real objects. A date may arrive as an ISO string, a list may contain typed items, a dictionary may represent a Pydantic input model, and an LLM response begins as JSON text but must become the declared output model. I eventually made this work through one conversion pipeline: parse the TOML, validate and coerce every input to its declared Python type, render it consistently, parse the model response, and validate it into the declared Pydantic object before returning it to application code.

Each variable now declares its type, whether it is required, and an optional default. Prompt Ninja automatically applies defaults, rejects missing required inputs, and uses consistent formatting rules and explicit str, repr, json, and csv filters. Templates are checked against their declarations, so unknown variables, invalid defaults, incorrectly typed fixtures, malformed model responses, and output schema mismatches fail clearly instead of producing surprising behavior later.

Testing LLM behavior presented a different challenge: how do I simulate a model call when model output is nondeterministic? Exact string assertions were too brittle, but mocking everything would prove only that the Python wiring worked. I solved this with two layers of testing. Deterministic unit tests inject fake provider executors to simulate success, errors, retries, structured responses, and hook events. These tests verify parsing, type conversion, rendering, validation, and control flow without making network calls.

The embedded [[tests]] cases cover the behavior that cannot be simulated with ordinary assertions. They run the real prompt and compare its response with a natural-language expectation through an LLM judge and pass threshold. Pydantic validates the response structure first; the judge then evaluates whether the meaning and constraints are correct. This gives me reliable unit tests for the engine and semantic regression tests for the prompt itself. It also lets protected updates run against the full behavioral contract before a candidate prompt is allowed to replace the current version.

Accomplishments that I'm proud of

The accomplishment I am most proud of is that Prompt Ninja became a usable framework rather than only a hackathon prototype. It is packaged as prompt-ninja and can be installed into an ordinary Python project:

pip install prompt-ninja

After installation, developers can generate a prompt artifact from a goal, validate its structure, run its semantic tests, update it safely, and load it through the Python API. The CLI and package operate on portable *.prompt.toml files that belong to the developer, so adopting Prompt Ninja does not require moving prompts into a proprietary hosted platform.

I am also proud that the Web UI makes the framework understandable and useful without hiding the artifact underneath it. A user can describe an LLM outcome in plain language, watch multiple agents discuss and construct the prompt, edit and run the generated tests, safely revise the implementation, and download the resulting TOML file for use in a real project. The Hooks page then demonstrates how the same prompt can be observed after it starts running.

Together, the package and Web UI make prompt management easier without treating prompts as disposable text. Model configuration, typed inputs, defaults, output schemas, tests, and runtime observations live in one workflow. Structural validation catches malformed artifacts, semantic tests catch behavioral regressions, and protected updates keep working expectations intact. That makes prompts easier to create and maintain, but more importantly, it makes their behavior more dependable as an application changes.

What I learned

Codex is a powerful tool for building projects faster. It handled a large amount of implementation work across the Python package, API, tests, and React/Vite frontend, and it was especially effective when each task had a clear behavioral goal and a focused validation step. It made it possible to move quickly from an idea to a working, installable framework while still iterating on the architecture.

The most important lesson was that speed does not remove the need for technical direction. At the beginning of a generated project, I need to define the packages, frameworks, versions, and tools I expect it to use. If those constraints are left open, Codex may choose a library that does not fit the project, use an outdated API, implement something manually when a proven package already exists, or select a method that works initially but becomes difficult to maintain.

For Prompt Ninja, explicitly choosing Pydantic for contracts, FastAPI for the API, Click for the CLI, OpenRouter through an OpenAI-compatible client, and React with Vite and Chakra UI for the frontend gave Codex a consistent foundation. It also helped to specify ownership boundaries, such as keeping semantic policy in prompt artifacts and deterministic enforcement in Python.

I learned to treat Codex as an implementation partner rather than an automatic architect: decide the stack and constraints first, provide work in a deliberate order, review the generated code, and run focused tests or builds after every meaningful change. With that direction, Codex can dramatically shorten implementation time without giving up maintainability or control over the final system.

The broader product lesson was that improving prompts is not only about finding better wording or choosing a stronger model. Changing how prompts are managed produced better results across the entire lifecycle. When expectations become the source of truth and prompts become structured, versioned, and tested artifacts, generation becomes clearer, failures become easier to diagnose, updates become safer, and runtime behavior becomes observable. The individual features reinforce one another, which is why treating prompt management as an engineering workflow had a greater impact than optimizing any single prompt in isolation.

What's next for Prompt Ninja

The first priority is better documentation. I want to add more complete guides, real application examples, framework integrations, prompt-file recipes, and migration paths for teams that currently keep prompts in source code or configuration files. The goal is to make the path from installation to a production-tested prompt artifact straightforward.

I also plan to build a TypeScript and JavaScript SDK so Node.js, browser, and full-stack TypeScript projects can use the same *.prompt.toml format and contract workflow. If there is interest from other language communities, the format could support additional SDKs while keeping artifacts portable and behavior consistent across runtimes.

The longer-term direction is safely self-improving prompts. Runtime hooks already provide quality scores, failure rationales, inputs, outputs, and usage data. Prompt Ninja could use that evidence to identify recurring failures and propose improved prompt implementations automatically. Those improvements would still be candidates rather than unchecked live edits: existing tests and thresholds would remain protected, new regression cases could be proposed from real failures, and a candidate would need to pass the complete contract before promotion.

The aim is not to let prompts rewrite themselves without control. It is to create a measurable improvement loop in which production evidence leads to a suggested change, tests verify the change, and developers retain ownership of the final artifact.

Built With

• codex
• docker
• fly.io
• gpt-5.6
• python
• typescript