PlayMind Labs Scoring AI

Inspiration

PlayMind Labs is my company — we build behavioral risk infrastructure. Prediction markets just turned on fees ($1.26B combined revenue at Polymarket + Kalshi), and their biggest revenue leak is tilted traders who spiral, blow up, and never come back. Nobody scores emotional state in-session; the behavioral vendors that exist are compliance products built for sportsbooks, and they can't sell into CFTC-regulated venues. I came to this hackathon to rebuild our production scoring architecture from zero, solo, in one day — and to make it smarter than the version I started with.

What it does

One REST API: a platform streams session events (trades, deposits, losses) and gets back a 0–100 stability score, a state classification, and an intervention tier (nudge / cooldown / re-engagement hold) in under 100ms. Fail-open by design — it can never block a trade.

The architecture separates state from onset. Each of 7 behavioral signals maintains an EWMA:

$$S_t = \alpha \, x_t + (1 - \alpha) \, S_{t-1}$$

The weighted composite answers "what state is this trader in." A two-sided CUSUM on the composite answers "did that state just shift":

$$C_t^- = \max\left(0,\; C_{t-1}^- + (\mu_t - x_t) - k\right), \quad \text{alarm when } C_t^- > h$$

EWMA is smooth but laggy; CUSUM catches small sustained drifts. Together, the alarm fires while the spiral is forming — in our demo, at score 79, twelve events before the trader hits critical.

The part I'm proudest of: nothing is hard-coded

An LLM-driven calibration agent (running on Nebius, Kimi-K2.7-Code) tunes every signal weight and tier threshold against the platform's own historical data. To give it real labels, I pulled trade histories for 2,067 real Polymarket wallets from public on-chain data the night before, labeled 90-day blow-up churn outcomes, and backtested the signal set: behavior from a trader's first 14 days predicts blow-up churn at 0.74 AUC (5-fold CV) — and prevented blow-ups are worth ~30.7x in downstream fees.

Then, live today, the agent tuned the runtime engine from 0.56 → 0.67 AUC in five iterations — and independently rediscovered my backtest's strangest finding: frequency acceleration inverts on real data. The gambling literature says fast trading = tilt. On real prediction-market wallets, fast trading predicts retention. The agent crushed that weight (0.14 → 0.01) on its own. Literature assumptions don't transfer between markets untuned — which is exactly why the calibration agent has to exist.

How I built it

• Engine: Fastify 4 / TypeScript. 7 signal scorers → per-signal EWMA → weighted composite → CUSUM onset detector → tier + intervention. In-memory session store, hashed identifiers only.
• Calibration agent: scores all 2,067 wallets per iteration, builds a confusion matrix, sends it to the LLM, parses the proposed config (with fence-stripping, retry, and a deterministic fallback so the loop can never stall), logs every iteration to a trace.
• Demo layer: a session generator whose four trader archetypes are parameterized from the real wallet feature distributions — the tilt spiral you watch is shaped like actual blow-up wallets — plus a live dashboard: score gauge, signal bars, alarm banner, and the calibration trace rendering the agent's tuning run.

Challenges

1. The cold-start alarm. After adding CUSUM warm-up suppression, my "boring" control archetype started alarming on recovery — scores climbing back from a dip. Fix: directional alarms. Interventions fire on deterioration only; improvement re-anchors silently. A bug that became a product principle.
2. Reasoning tokens ate my budget. The Nebius catalog models emit hidden reasoning tokens that count against max_tokens — my first API call returned content: null with the entire budget spent thinking. Diagnosed via the usage fields, fixed with a bigger budget and strict JSON-only prompting.
3. Honest metrics are harder than good ones. The 0.74 is an unconstrained offline benchmark; the deployable monotonic-weight composite measures 0.67 on the same labels. I ship both numbers. Closing that gap — including letting calibration learn signal signs, not just weights — is the roadmap.

What I learned

That the gap between a research result and a deployable system is where the actual product lives — and that an agent tuning against real ground truth can find things you didn't tell it to look for.

What's next

Kalshi replication of the backtest, shadow-mode pilots with live platforms (silent scoring, 2-day integration), and intervention-efficacy measurement — proving the alarm doesn't just predict the blow-up, it prevents it.

Built With

• fastify
• kimi
• nebius
• node.js
• python
• react
• typescript
• vite