DEF: Dynamic Engagement Framework

Video games get boring, and ppl out grow them. DEF runs on any game, and monitors heartbeat, dopamine, and keystrokes in real-time, auto-adjusting speed, obstacles, and difficulty for max engagement.

Comment

What inspired me

Lots of games have static difficulty, and it's kind of boring. I wanted to build something that could sit on top of any web game, read the player in real time, and tweak the right things to keep it fun without touching the game's code at all.

What we learned

The biggest thing was figuring out how to run inference on the CPU at low latency while the game is running. The browser only gives you one main thread, so everything was fighting for it. I learned to pull vitals sampling off the animation frame onto setTimeout loops so the game's rAF never stalls.

How we built it

I used a gemini agent to parse any game. A 500ms tracker builds an 18-dimensional vector (vitals, score, deaths, inputs, face cues) on user behavior every second. and feeds an engagement classifier. Then two decision layers, Thompson sampling and a UCB1 bandit to pick which mods to apply through the game's existing hooks.

How the AI architecture changed

We started thinking we'd use one big model for everything. That was a mistake lol. Too slow, too complex. We went layered and lightweight by using rule-based classifier on the hot path, Thompson sampling for mod selection, UCB1 for policy bundles. Kept it CPU-friendly and actually predictable.

The main challenge: speed

This was my first solo hack so I had no idea what I was getting into. Keeping the game smooth while running all of this at once was genuinely hard, webcam capture, FFT for rPPG, the tracker, RL engine, UI, all on one thread. I fixed it by moving vitals off rAF and throttling graph redraws. Game runs at full speed now and the engagement graph still feels live.

Built With

  • attention-based-policy-selection
  • bandpass-fft-for-signal-processing
  • bored
  • canvas-2d
  • css3
  • disengaged)
  • dotenv
  • engagement-classification-(flow
  • excited
  • face-roi-detection-(opencv-haar-cascade)
  • fetch-api
  • frustrated
  • gemini-2.5-flash-(generative-model-for-game-profile-generation)
  • getusermedia-(webrtc)
  • google-gemini-api
  • html5
  • in-browser-vitals-inference-(hr/br)
  • javascript
  • jquery
  • jsdelivr-cdn
  • localstorage
  • node.js
  • npm
  • opencv-haar-cascade-(face-detection-model)
  • opencv.js
  • pip
  • python
  • pytorch
  • reinforcement-learning-(rl)
  • rest
  • rppg-(remote-photoplethysmography)
  • server-sent-events-(sse)
  • thompson-sampling
  • training
  • ucb1-bandit
  • vitallens
  • vitallens-rppg-model-(in-browser-hr/br-inference)
Share this project:

Updates