Inspiration
Small businesses in Latin America sell on WhatsApp, and more of them are plugging an AI sales bot into that chat, often built in an afternoon with n8n or a no-code tool. Nobody tests it. The failures are quiet and expensive: the bot invents a price, accepts a discount "the owner promised me", misses a complaint that needed a human, or misreads a customer who writes klk, cuanto e el carton d huevo across three split messages. Owners don't have an evaluation team or a test set. I wanted to give them one, built from their own price list.
What it does
Cliente Fantasma ("Ghost Customer") turns a business's facts into synthetic customers that attack its sales bot, then hands back an evidence-based report card.
- Business profile. The owner pastes their facts or a website URL. Tavily search and extract supplement public information; NVIDIA Nemotron Ultra extracts products, prices, delivery rules and policies. The owner reviews and edits the profile before anything runs. That profile is the answer key.
- Ghosts. Nemotron Ultra generates customers across four axes: intent (ready to buy, complaint, wholesale, haggling…), attack (social engineering, prompt injection, fake promo), dialect (Dominican, Mexican, Colombian, Argentine, Spanish) and writing style (typos, emojis, split messages, no accents).
- Conversations. Each ghost chats with the bot. Three starting points: Business facts only (we generate a prototype bot on Nemotron Nano with simulated catalog, order, FAQ and human-handoff tools), I already have instructions (test your own prompt), or I have a live bot (we call your real webhook; an n8n connector is included).
- Judge. Deterministic checks (invented numbers, tool use, language, length, handoff) plus a Nemotron Ultra judge grounded in the approved profile.
- Fix, but only if it holds up. Nemotron Ultra proposes one focused prompt change at a time. It is accepted only if it beats the current version on training cases without breaking a critical case that used to pass, and the before/after is reported on held-out cases the optimizer never saw.
The report shows coverage first, then the three worst failures with the exact message where the bot went wrong, the business fact it contradicted, a score by kind of customer, and the prompt proposal with its validation status.
How we built it
All inference runs on Nebius Token Factory through its OpenAI-compatible API (api.tokenfactory.nebius.com/v1), with a different NVIDIA model and reasoning setting per role:
| Role | Model |
|---|---|
| Sales bot under test | NVIDIA Nemotron 3 Nano 30B-A3B |
| Ghost customers, test cases, business profile | NVIDIA Nemotron 3 Ultra 550B-A55B |
| Judge and prompt optimizer | NVIDIA Nemotron 3 Ultra 550B-A55B |
The app is Python 3.12 + FastAPI + Jinja with live conversations over Server-Sent Events, Pydantic schemas for every structured output, SQLite for runs and accounts (Argon2id passwords, revocable sessions), and Docker behind a Cloudflare Tunnel on my homelab. Every model request reserves a conservative budget before it is sent, so a run, a visitor and the whole demo each have hard spending caps; a model without a known price fails before inference. The webhook mode resolves and pins public IPs to block SSRF into private networks. 423 offline tests cover the engine, the budget accounting and the HTTP layer.
Challenges we ran into
- Reasoning ate the output budget. Nano and Super reason by default; with modest
max_tokensthe answer came back empty./no_thinkin the prompt did nothing;chat_template_kwargs: {"enable_thinking": false}per role fixed it. - Latency. Nemotron Super swung from 1.4 s to 77 s to timeouts on the same request, so the Ultra roles carry the synthetic-data and evaluation work and Nano stays the fast bot under test.
- The optimizer copied its own context. Early proposals pasted the rubric and critical rules back into the "improved" prompt, which made them too long and taught the bot fake example prices. Isolating the current prompt and cutting anything after a context heading fixed it: proposals went from 11–25 KB to a few hundred bytes.
- Not faking a fix. With one repetition per case, one case flipping moves the score about 1/train. We raised the minimum improvement accordingly, compare before/after on held-out cases only, and never let an interrupted or partial run call a bot "ready".
Accomplishments that we're proud of
In our sample run the generated prototype scored 41/100: talking to a Mexican customer about delivery, it invented a full product table with names and prices that don't exist (a US$42 "Adventure Nylon" backpack). In a full run on a prompt seeded with two common mistakes (no human handoff, long bulleted answers) it scored 18/100; both proposed fixes were clean and sensible, and both were rejected because they broke a critical case or scored lower. Saying "no" honestly is part of the product.
We're also proud that it is safe to leave open to the public: hard spending caps per run, per visitor and per day, a reserved budget for judges, SSRF protection for the live-bot webhook, and 423 offline tests.
What we learned
A score alone convinces nobody; the quoted message beside the business fact does. Owners act on "here is the exact message your bot sent, and here is the price list it contradicted", not on a number. We also learned that an evaluation loop needs as much skepticism as the bot it evaluates: small test banks are noisy, so improvements must be checked on held-out cases, and a run that stops early must never approve anything.
What's next for Cliente Fantasma (Ghost Customer)
Voice notes and product photos (they dominate WhatsApp; there is no Nemotron omni model on Token Factory yet), Tavily-backed checks of live competitor prices as an attack, per-fact source links in the profile, and a human-calibrated judge benchmark.
Log in or sign up for Devpost to join the conversation.