Storefront Fraud Investigator

An AI fraud analyst for online stores: ask in plain English, and it searches orders, sessions and known fraud patterns to give a verdict, a next step and its evidence.

Comment

Inspiration

Fraud rarely sits in one row. An account takeover shows up as a first-time device in the sessions table, a password reset in the events table, a new delivery address, and an order over 100× the customer's usual. Each signal looks harmless alone. Today an analyst joins those tables by hand to see the case. However, these is a very outdated of doing things, especially when agents exist and can perform investigations in a much faster pace, which aligns with what security is suppose to be. Quick and precise.

I have already built a log-analysis RAG prototype, and wanted to see whether an agent could do an analyst's investigation work, not just retrieve text, and show its evidence well enough to be trusted.

What it does

An analyst asks a question in plain English: "Was there anything unusual about a specific soemone's most recent order?" and the agent investigates:

  • Picks its own tools, up to 3 calls: read-only SQL on Postgres, hybrid search over session history (Qwen3 embeddings + BM25 in Qdrant), and Tavily web search to check whether the behaviour matches a known fraud scheme.
  • Compares against baselines: the customer's own history, or the rest of the store.
  • Answers verdict-first: a one-line verdict, a suggested next step (e.g. hold order 196, contact the customer), then the evidence and a short trace.
  • Shows its work: every tool call opens beside the answer with its real SQL or search input and output, plus a panel showing which Nemotron model handled each step and how long it took.
  • Doesn't cry wolf: an ordinary customer comes back as "Nothing unusual found", with the checks it ran.

The data is a fictional store with five planted incidents (account takeover, card testing, refund abuse, promo code sharing, credential stuffing). Nothing is pre-flagged, the agent works it out from the raw rows.

How I built it

  • NVIDIA Nemotron on Nebius Token Factory for every LLM call, routed by task:
    • Nemotron 3 Super runs the reasoning loop (is more data needed? which tool?) and writes the SQL and search arguments.
    • Nemotron 3 Ultra writes the answer only when it has to weigh 2+ tool results. The tier is chosen from what the investigation needed, not by a classifier. A typical investigation makes 8 LLM calls, of which 1 is on Ultra.
    • Nemotron 3.5 Lightning writes follow-up suggestions and out-of-scope replies.
    • Super also acts as an LLM judge, scoring each answer for faithfulness, coverage and tool routing.
  • LlamaIndex workflow: reason → read schema → write arguments → execute, looping up to 3 times, then answer and suggest follow-ups.
  • Guards in code, not just prompts: the model's SQL runs as a Postgres role with SELECT on the 9 store tables only, read-only sessions and a 15 s timeout; a rejected query gets a free repair with Postgres's error location; web searches are stripped of customer data before sending.
  • Versioned prompts in Postgres, so prompt changes can be rolled out and rolled back without a redeploy, and compared using the judge's scores.
  • FastAPI + Chainlit, Redis for conversation memory and rate limits, Docker Compose and Caddy for the hosted demo.

Challenges we ran into

  • Generic SQL that didn't fit this data. The model wrote SQL that would be fine on a typical database but was wrong for ours, so we had to spell out how this dataset behaves, in the schema prompt and in code:

    • Time: the store data is a fixed snapshot, but the model used now(), so "last 7 days" fell after the data ended. The agent is now told when the data ends, and code rejects any query that reads the database clock.
    • Table shape: one session can have many orders and many payment attempts. Joining both on the session copied rows (8 orders × 46 attempts = 368 rows) and inflated every count. The schema prompt now explains this and tells it to count each table separately before combining them.
    • History: activity only covers the last ~30 days, but accounts go back years, so an ordinary customer was called "dormant for 2 years". The prompt now says where the data starts.

  • Repairs that didn't repair. Given only "missing FROM-clause entry for table o", the repair step resubmitted the identical query four times. Passing Postgres's error line and hint fixed it: 3/3 correct runs afterwards.

  • False positives. "Unusually high refund rate" flagged a customer with 1 refund on 1 order (100%). Requiring at least 3 orders and showing the counts ("9 of 11 orders refunded") left only the real abuser.

  • Trust in a clean result. An ordinary customer was called "dormant for 2 years" because the activity data only covers 30 days. The agent is now told where the data starts.

Accomplishments that I'm proud of

  • It finds the planted account takeover from scratch: new device and country, address change, an order 112× the customer's usual.
  • Every number in an answer traces to a query you can open and read.
  • Routing is visible and cheap: only the hardest step goes to Ultra.

What I learned

  • Guards in code beat instructions in prompts: a read-only database role can't be talked out of being read-only.
  • Error messages are prompts too. Giving the model the failing line turned a loop into a fix.
  • A clean "nothing found" is as important to get right as catching the fraud.

Limitations

  • Synthetic data only: one store, ~30 days of activity, five planted incidents.
  • Answers vary between runs; the path (SQL only vs. SQL + session search) can differ.
  • At most 3 tool calls per question, so broad questions can stop short.
  • It investigates and recommends; it never acts (no blocking, refunds or account changes).

What's next for Storefront Fraud Investigator

  • could be turned into a real time investigator and actively monitor instead of a chatbot.
  • offline evaluation to further evaluate the investigator with reference dataset
  • saved investigation so we can link multiple investigations together

Built With

Share this project:

Updates

Submission history