Visual compiler

• 

  Compile AI once. Execute forever.

Visual Compiler — Project Story

Inspiration

I am not a professional software engineer. I am an anesthesiologist who spends a large part of his working life in environments where reliability, traceability, and predictable execution matter. I live in France (Bordeaux, city of wine).

Outside medicine, I have always been interested in software, automation, and artificial intelligence. Over time, I began automating some of the repetitive digital tasks surrounding my work: navigating administrative portals, extracting information, organizing documents, and interacting with web applications that often expose no usable API.

The problem looked simple at first.

I repeatedly had to perform sequences such as:

• locate a specific line in a table;
• identify a label;
• select the checkbox associated with that label;
• open a figure or modal window;
• extract or print content;
• repeat the same actions across several records. (the patients)

Traditional browser automation tools worked when the page structure was stable and when I could manually write precise selectors. But they struggled with instructions that are obvious to a human and surprisingly difficult to express deterministically:

Click the checkbox immediately after this label.

Select the first enabled control to the right of “Pending review”.

Click the text below the first green icon, reading from left to right and top to bottom.

These are not simply coordinate-based actions. They require an understanding of meaning, layout, visual hierarchy, and spatial relationships.

I then explored AI browser agents. They were much better at understanding such instructions, but I encountered the opposite problem: the model had to inspect the page and reason again during every execution.

A task that had already been understood once was being reinterpreted over and over.

That meant repeated inference cost, additional latency, inconsistent behavior, privacy concerns, and a new failure point at every click.

This led to the central question behind Visual Compiler:

Why should an AI reason again every time a workflow runs, when the workflow could be understood once and compiled into deterministic software?

Software developers do not ask a compiler to reinterpret their source code every time a processor executes an instruction. The expensive reasoning occurs during compilation, and the resulting program can then run repeatedly and predictably.

Browser agents seemed to be missing this separation.

That became the idea for Visual Compiler.

⸻

What Visual Compiler Does

Visual Compiler separates browser automation into two distinct phases.

1. Intelligent compilation

During the learning phase, GPT-5.6 analyzes:

• the user’s natural-language instruction;
• a demonstration of the task;
• the DOM;
• the accessibility tree;
• visible text;
• element states;
• bounding boxes;
• spatial relationships on the page.

The model does not directly control the browser indefinitely. Instead, it produces a structured semantic description of the intended workflow.

For example, a human instruction such as:

In the row containing “Pending review”, check the first enabled checkbox to the right of the status text.

can be represented as a semantic target:

{ "action": "check", "anchorText": "Pending review", "relations": [ { "type": "same-row" }, { "type": "right-of" } ], "elementType": "checkbox", "state": "enabled", "ordinal": 1 }

The compiler then evaluates deterministic locator strategies, including:

• accessibility roles and names;
• label associations;
• text anchors;
• DOM relationships;
• same-row and same-column relationships;
• geometric proximity;
• OCR anchors;
• image-based fallbacks.

It ranks the candidates, validates target uniqueness, generates assertions, and produces an executable Playwright workflow.

1. Deterministic execution

Once compilation is complete, the LLM is removed from the execution path.

The compiled workflow runs using local deterministic logic:

Compiled workflow ↓ Locator engine ↓ Playwright ↓ Assertions and telemetry

No model is asked what to do next.

No OpenAI API key is required.

The same compiled workflow can be replayed repeatedly, with:

• predictable behavior;
• lower latency;
• no runtime inference cost;
• clearer auditability;
• explicit failure conditions;
• improved privacy.

The core principle is simple:

Compile AI once. Execute forever.

⸻

How I Built It

I used Codex as a development partner throughout the project, while retaining responsibility for the product thesis, architecture, constraints, and technical decisions.

I began by reducing the idea to one testable claim:

A multimodal model should be able to interpret a complex browser instruction once, compile it into a semantic intermediate representation, and produce a workflow that continues to run without any LLM call.

Rather than attempting to automate the entire web, I built a controlled demonstration environment containing the types of ambiguity that originally motivated the project:

• repeated labels;
• multiple icons with different colors;
• checkboxes positioned before and after text;
• enabled and disabled controls;
• repeated table rows;
• unstable IDs and CSS classes;
• two layout variants with preserved semantic relationships.

The project was structured around a strict separation between compile-time and runtime components.

Page capture ↓ Normalized page model ↓ GPT-5.6 semantic interpretation ↓ Validated Semantic IR ↓ Deterministic locator generation ↓ Playwright workflow ↓ Local runtime with zero LLM calls

The semantic intermediate representation became the contract between intelligence and execution.

GPT-5.6 is allowed to interpret intent and resolve human ambiguity, but its output must conform to a validated schema. It does not directly emit unchecked browser code.

The deterministic compiler is responsible for translating semantic intent into locator candidates, scoring them, checking uniqueness, and rejecting workflows when confidence is insufficient.

This distinction became one of the most important design decisions in the project:

The model may explain what the user means, but deterministic software must decide whether the resulting automation is safe and executable.

The runtime was deliberately isolated from the OpenAI SDK. It can load a previously compiled workflow and execute it with the API key removed.

The interface makes this separation visible by displaying metrics such as:

Compile-time model calls: 1 Runtime model calls: 0

This is not merely a cost optimization. It is the product.

⸻

Challenges

Translating human spatial language into executable rules

Humans naturally use expressions such as “next to”, “below”, “the first”, or “in the same row”. Browsers do not expose these concepts directly.

A major challenge was translating these descriptions into deterministic geometry and DOM relationships.

For example, “same row” cannot simply mean that two elements have identical vertical coordinates. Real interfaces include padding, nested containers, wrapped text, and elements of different heights.

The implementation therefore uses tolerances, bounding-box overlap, reading order, anchor relationships, and candidate scoring rather than a single brittle rule.

⸻

Choosing stable locators

An element can often be targeted in several ways:

• generated CSS class;
• element ID;
• visible text;
• ARIA role;
• label association;
• XPath;
• relative position;
• absolute coordinates.

The easiest selector is not necessarily the most stable one.

Visual Compiler must therefore reason about selector quality rather than merely finding a selector that works once.

The compiler prefers semantic and accessible relationships over implementation details, and treats absolute coordinates as a last resort.

⸻

Handling ambiguity honestly

An AI agent can guess and continue.

A compiler should not.

If two checkboxes are equally plausible, Visual Compiler should report ambiguity instead of silently choosing one.

This required designing explicit confidence thresholds and failure modes.

Rejecting a compilation can feel less impressive than completing every task, but it is essential for a tool intended to produce reliable and auditable automation.

⸻

Proving that the runtime is genuinely model-free

It would have been easy to claim that the runtime was deterministic while retaining a hidden model fallback.

Instead, the architecture enforces the separation technically:

• the runtime package has no OpenAI dependency;
• compiled workflows contain no API secrets;
• workflows run with OPENAI_API_KEY unset;
• network interception can confirm that no OpenAI endpoint is contacted;
• runtime telemetry explicitly reports zero model calls.

This proof is central to the demonstration.

⸻

Balancing ambition with a Build Week deadline

The long-term vision includes arbitrary websites, visual recording, OCR, image templates, automatic repair, and enterprise deployment.

Attempting to implement all of this would have produced a broad but unreliable prototype.

The most difficult product decision was to reduce the project to a polished vertical slice that proves the new architecture.

The MVP focuses on:

1. one complex instruction;
2. one compilation step;
3. one deterministic generated workflow;
4. two related page layouts;
5. successful execution without an API key.

The scope is intentionally limited, but the architectural claim is real.

⸻

What I Learned

Intelligence and execution do not have to be coupled

Most current browser agents treat intelligence as a continuous runtime dependency.

This project taught me that many workflows can instead be divided into:

\text{Intelligent interpretation} \rightarrow \text{Deterministic execution}

The model is most valuable when the task is ambiguous and needs to be understood. Once that ambiguity has been resolved, repeatedly invoking the model may add more uncertainty than value.

⸻

Structured intermediate representations are essential

Natural-language instructions are too ambiguous to become executable code directly.

The Semantic IR acts as a safety and engineering boundary between the model and the runtime.

It makes workflows:

• inspectable;
• testable;
• versionable;
• explainable;
• portable across execution engines.

This was one of the strongest architectural lessons from the project.

⸻

Accessibility metadata is valuable automation infrastructure

ARIA roles, accessible names, labels, and semantic HTML are not only useful for assistive technologies. They are also among the most stable and meaningful signals available to deterministic automation.

A page with good accessibility semantics is easier to compile reliably than a visually similar page built entirely from anonymous containers.

⸻

Reliability sometimes means refusing to act

In medicine, an uncertain decision should not be disguised as a confident one.

I found the same principle useful in automation.

A reliable compiler must sometimes say:

I cannot identify a unique stable target.

That is preferable to generating a workflow that appears successful during the demo but fails unpredictably in real use.

⸻

Codex was most useful as an engineering collaborator

Codex accelerated repository setup, schema design, test creation, refactoring, and implementation of repetitive components.

But the most important product choices remained human decisions:

• separating compile-time from runtime;
• prohibiting model calls during execution;
• prioritizing explainability over autonomy;
• rejecting low-confidence compilation;
• building a controlled demonstration rather than pretending to support the entire web.

The collaboration worked best when Codex handled execution speed and implementation breadth while I maintained a clear product thesis and quality threshold.

⸻

What Comes Next

The current MVP is a proof of architecture rather than a universal automation product.

The next stages would include:

• browser-extension-based recording;
• richer visual demonstrations;
• OCR and image-template fallback engines;
• workflow versioning;
• automatic regression testing;
• safe recompilation when interfaces change;
• local vision models for private environments;
• a Visual Compiler SDK for agent developers;
• an enterprise workflow registry;
• a marketplace for reusable compiled automations.

The broader vision is that AI agents should not have to remain in the execution loop forever.

Foundation models could increasingly act as compilers that transform human intent into durable, inspectable, deterministic software.

Visual Compiler is an early exploration of that idea.

We are not building another browser agent.

We are building the compiler browser agents have been missing. Compile AI once. Execute forever.

Built With

• chatgp
• codex
• vibecoding