GhostLoop 👻

Your first users, before your first real user.

Inspiration

As a engineer and designer, one of our teammates, Emma, has worked on products where user testing was nearly impossible. On MoffittStatus, a crowd meter for Berkeley’s main library, she had no users to test with before launch. Working with pre-seed startups, the pattern repeated: UIs were picked hastily and pushed, often because there was no one to test it on. Additionally, on a QRE dashboard project with Microsoft, she manually UX-tested everything herself, clicking every button, trying every feature, checking every screen, even for simple tasks.

The problem is the same everywhere: A/B tests need traffic, and pre-launch products have none. So usability problems ship with the product, and the first real users are the ones who find them.

Existing tools can flag issues, but feedback alone doesn’t fix anything. Someone still has to decide what to change, build it, and check whether it helped. We wanted to close that loop: simulate users, see where they get stuck, propose a small design change, and retest it, with the designer approving every change.

So we built GhostLoop: your first users, before your first real user.

What it does

Building an app is one thing. Knowing whether people can actually use it is another.

For early-stage startups, getting meaningful feedback can be difficult, especially when you don't have enough users to run traditional A/B tests. And sometimes, poor design choices, confusing navigation, or broken workflows can cost you your very first customers.

That's why we built GhostLoop.

GhostLoop lets us test a prototype with simulated users before putting it in front of real ones.

We start by importing a Figma Make app and describing a task in plain language, like "Order one flat white for pickup and get the order confirmed." GhostLoop then deploys AI agents that interact with the application through real browsers, attempting to complete that task just like a user would.

From there, we close the loop:

• Watch what happens. Every trial is recorded step by step, including screenshots, actions, the agents' reasoning, and validation errors. We can see exactly where agents succeed, struggle, or get stuck. For instance, a seemingly harmless "Continue" button might unexpectedly redirect users back to the menu.

• Measure success, not just impressions. By default, a separate Claude judge evaluates each trial against the task description. For more reliable evaluation, designers can use our "Suggest Checks" feature to generate verifiable success conditions grounded in the application's actual text and routes. These might include reaching /order/confirmed or displaying "Order confirmed". When these checks are enabled, GhostLoop verifies them directly against the final page rather than relying on an AI's judgment.

• Get evidence-backed suggestions. Claude analyzes the interaction traces across trials, identifies recurring patterns, and proposes targeted wording improvements. Each suggestion is linked to the exact location in the application's source code.

• Stay in control. Nothing changes without the designer's approval. Once a suggestion is approved, GhostLoop creates a separate candidate version of the application. The original remains untouched.

• Retest and compare. GhostLoop runs the same experiment on the revised application, using the same frozen task definition and evaluation criteria. We then compare the results and determine whether the proposed change actually helped.

One of our favorite moments from building GhostLoop was when the system rejected its own AI-generated improvement.

Claude proposed a seemingly reasonable button-label change. But when we retested the application, the simulated success rate actually dropped from 16.7% to 0%.

Instead of blindly accepting the AI's recommendation, GhostLoop measured the regression, rejected the candidate, and preserved the original design.

And that's exactly the kind of behavior we wanted to build.

How we built it

GhostLoop brings together Figma Make, browser automation, multimodal AI inference, Anthropic Claude, and SpacetimeDB into one experimentation pipeline.

Figma Make and sandboxed application previews

We wanted GhostLoop to work with applications beyond our own sample prototype.

Designers can upload a code ZIP exported from Figma Make. Before extracting anything, we validate the archive in memory to protect against path traversal, symlinks, and ZIP bombs.

We then install, build, and preview the application inside hardened Docker containers. These containers run as non-root users, with restricted capabilities, CPU and memory limits, execution timeouts, and no network access during the build.

Each version receives its own isolated preview origin, and uploaded application code never executes directly on the host.

Simulated users with Bitdeer and Playwright

Our simulated users are powered by a vision-language model hosted on Bitdeer AI Model Studio.

We use Playwright to launch real Chromium browser sessions and observe the application. At each step, the agent receives a screenshot, its assigned task, and a list of interactive controls discovered from the rendered page.

The model decides what to do next, and Playwright executes the action.

Because we discover controls dynamically, developers don't need to add special test IDs or modify their applications for GhostLoop to work.

Every trial begins with a fresh browser session and application state, allowing us to compare results across experiments.

Analysis and evaluation with Anthropic Claude

We use Claude for three distinct responsibilities:

1. Analyzing interaction traces. Claude identifies recurring patterns across simulated sessions and proposes targeted wording changes.
2. Judging task completion. When deterministic checks aren't configured, a separate Claude judge evaluates whether a trial completed the user's task.
3. Suggesting verifiable success checks. Claude proposes success conditions grounded in the application's actual routes and text, which our code can verify independently.

Importantly, the agent performing the task never gets to declare its own success.

We also distinguish between judged and verified results throughout the interface, so designers know exactly what kind of evidence they're looking at.

Safe, reversible code modifications

We deliberately limited our initial design modifications to small wording changes.

Using the TypeScript compiler, we map editable JSX text to its exact source file and position.

When a designer approves a suggestion, GhostLoop applies the patch only if the original file still matches the expected content.

We then validate the modified application and verify that the resulting patch matches the approved candidate byte for byte.

This gives us a controlled way to experiment with changes without allowing an AI model to arbitrarily rewrite the application.

Recursive experimentation

At the heart of GhostLoop is a resumable experiment state machine.

Our workflow follows:

Baseline → Observe → Propose → Await Approval → Build → Validate → Retest → Compare

Each candidate is evaluated against the original experiment using fixed rules implemented in code.

Claude can suggest what to change, but it cannot decide whether its suggestion deserves to be accepted.

That decision comes from the measured results.

Real-time experiment tracking with SpacetimeDB

We use SpacetimeDB to store experiments, simulation runs, and individual agent actions.

As agents navigate an application, their interaction events are written to SpacetimeDB and streamed to our React and Vite dashboard through real-time subscriptions.

This allows us to observe experiments as they happen, inspect individual agent traces, and track the progress of each testing session.

It also provides the shared state connecting our simulation workers with the frontend.

Challenges we ran into

Making agents work with arbitrary applications

Our original simulator was designed around our own sample application. It relied on predefined test IDs and screen names, which made navigation relatively straightforward.

But we quickly realized that this wouldn't work for arbitrary Figma Make exports.

We had to rethink how agents interact with applications.

Instead of relying on hardcoded selectors, we built a system that discovers interactive controls directly from the rendered page.

The model selects from those controls, and we verify that the selected element is still valid immediately before executing the action.

This made our simulation engine much more generalizable.

Running untrusted applications safely

Supporting arbitrary uploaded applications introduced a completely different set of engineering challenges.

We needed to isolate the code without breaking legitimate application builds.

One particularly frustrating issue involved a native compiler dependency, @swc/core, silently failing because Docker mounted temporary directories as non-executable.

Getting the security restrictions and build compatibility to work together required several iterations.

It was a reminder that building a secure execution environment is considerably harder than simply running npm install inside a container.

Defining what success actually means

This was probably one of the most important lessons from the project.

The easy approach would have been to ask an AI agent whether it completed the task and trust its answer.

But that creates an obvious problem: the same model performing the task could incorrectly conclude that it succeeded.

We separated the acting agent from the evaluator.

Even then, we discovered another limitation. Because many of our proposed modifications change wording, an AI judge reading the application might be influenced by those same wording changes.

That's why we added optional deterministic success checks and made the distinction between judged and verified results explicit.

Cleaning up browser and container resources

We also encountered an unexpected shutdown issue.

When stopping the application with Ctrl+C, the signal was sometimes delivered multiple times through npm and tsx. This caused our worker to exit before cleaning up its preview containers.

We eventually traced the issue and fixed the shutdown behavior while preparing our demonstration.

Not the flashiest feature, but definitely an important part of making the system reliable.

Accomplishments that we're proud of

We built the complete experimentation loop

Our biggest accomplishment is that GhostLoop works end to end with real models, real browsers, and real application builds.

We can:

Import → Preview → Define Task → Run Baseline → Analyze → Suggest → Approve → Build Candidate → Retest → Compare

And importantly, the original application remains untouched throughout the process.

Our best demo result was a failed improvement

One of the most satisfying moments was watching GhostLoop reject a suggestion that sounded perfectly reasonable.

Claude recommended changing a button label, but the retest showed that simulated task completion dropped from 16.7% to 0%.

Rather than treating every AI recommendation as an improvement, GhostLoop identified the regression and retained the better-performing original.

For us, that demonstrated why measurement matters.

We made testing accessible without complicated setup

Designers can describe what they want users to accomplish in plain English.

They don't need to write test scripts, configure browser automation, or understand the underlying agent infrastructure.

And when stronger evidence is needed, they can use suggested success checks to move from AI-judged outcomes to directly verifiable ones.

We prioritized safety and transparency

We wanted GhostLoop to be a tool designers could trust.

That meant making deliberate choices throughout development:

• Uploaded code runs in isolated containers.
• Design changes require approval.
• Every candidate is reversible.
• Source modifications are validated against exact file contents.
• Task success is evaluated separately from the acting agent.
• Results clearly distinguish judged from verified outcomes.
• Experiment playback distinguishes live runs from recordings.

These weren't features we added at the end. They shaped how we built the system.

What we learned

AI suggestions aren't improvements until they're tested

Claude produced thoughtful explanations and reasonable design recommendations.

But our experiments showed that something that sounds like a good design change can actually make an application harder to use.

The only way to discover that was to run the experiment again.

That reinforced the central idea behind GhostLoop: AI should generate hypotheses, not unquestionable conclusions.

How we define success matters as much as how we simulate users

We initially focused heavily on the agents themselves.

But over time, we realized that the evaluation system was just as important.

A flexible natural-language judge makes usability testing accessible. Deterministic checks make results more trustworthy when the application exposes verifiable success conditions.

Both have a place, but designers should always know which one they're relying on.

Small, controlled changes are surprisingly powerful

It's tempting to let an AI agent redesign an entire interface.

Instead, we focused on small, verifiable wording changes.

This made it possible to preserve the original application, show exactly what changed, obtain explicit approval, and run fair comparisons.

We learned that a narrower, more reliable optimization loop is often more useful than a broad but unpredictable redesign system.

What's next for GhostLoop

We're excited about where we could take GhostLoop beyond the hackathon.

Support more Figma Make applications

We've already demonstrated that a real Figma Make export can be imported, built, and previewed.

Our next step is running complete experiments across a wider variety of exported applications and workflows.

Make design recommendations more context-aware

Our demo highlighted an important limitation: Claude suggested improving a button label without fully understanding where the button actually led.

We want to enrich our analysis with the functional behavior of each control, including navigation destinations and state transitions.

That should help the system propose more informed changes.

Expand beyond wording changes

Our current approach focuses on safe JSX text modifications.

We'd like to support more design properties, including styling, layout, element ordering, and visibility, while maintaining the same approval process and validation safeguards.

Close the loop back to Figma

We also want to bring approved, experimentally evaluated modifications back into Figma as editable design proposals.

This would allow designers to move more naturally between visual design, executable prototypes, and measured experimentation.

Simulate more diverse usage scenarios

Finally, we'd like to expand the range of synthetic-user scenarios.

That includes different familiarity levels, accessibility-focused tasks, mobile viewports, error-recovery behaviors, and more complex navigation workflows.

Our goal isn't to claim that these agents perfectly represent real users. It's to give designers more ways to explore potential usability problems before launch.

---

Final Thoughts

We started GhostLoop with a simple question:

What if startups could learn from their first users before those users even existed?

What we ended up building taught us something equally important.

Simulating users is only part of the challenge. The harder part is collecting meaningful evidence, interpreting it carefully, and knowing whether a proposed improvement actually made anything better.

That's the loop we're trying to close.

GhostLoop | Your first users, before your first real user. 👻

Built With

• anthropicclaude
• bitdeer
• claudecode
• figma
• figmaagent
• figmamake
• figmamcp
• figmaslides
• github
• node.js
• playwright
• react
• spacetimedb
• typescript
• vite
• vitest
• zod