• 

  BLAME OPENING PHOTO

Devpost submission — Blame

Project name

Blame

Tagline (short description)

An interactive lab that shows which past actions get the credit when a reinforcement-learning agent is finally rewarded.

Inspiration

The hardest idea in reinforcement learning isn't the math, it's the credit assignment problem: an agent takes hundreds of actions and gets one reward at the end, so how does it know which actions mattered? I work on credit assignment in my research, and every explanation I've read of it is a wall of equations. I wanted something you could watch.

What it does

Blame lets you draw a gridworld (walls, goals, pits, a tempting coin) and watch three classic learners solve it: Monte Carlo, Q-learning and SARSA(λ).

The centrepiece is the blame ribbon: a bar for every step of the current episode, showing how much credit that step received. The difference is immediate. On the first success in the long corridor, Q-learning credits exactly 1 step, SARSA(λ) credits about 6, and Monte Carlo credits about 58.

You can also:

• switch the grid between Blame, Memory (what the learner is currently holding responsible) and Value overlays, with policy arrows;
• tune α, γ, ε, λ, optimistic starts and a slippery floor while it runs;
• tap any cell to inspect its action values;
• load four guided experiments (one-step vs traces, Monte Carlo on a maze, on- vs off-policy on a cliff, and how γ decides between a near coin and a far goal);
• race all three learners on your map (6 seeds each) and compare their learning curves against the true optimum V*, computed by value iteration.

The motion design repeats the same idea everywhere, so the page teaches even when you're just scrolling:

• when a reward lands, a pulse sweeps backward along the path, lighting each step by the credit it got (one cell for Q-learning, the whole path for Monte Carlo);
• hovering the ribbon replays the episode: it rings the cell the agent stood on at that step and shows the move it made;
• behind the page, faint agents wander a grid leaving trails that fade like eligibility traces, flash credit backward when they find a reward, and chase your cursor as if it were one (click empty space to drop another);
• the "Blame" wordmark's letters take weight and colour by closeness to the cursor, like credit falling off with distance;
• smooth scrolling, a section bar whose progress line fades like a trace, hover previews and gap-free drag painting on the grid, and a chart that draws itself in with a hover crosshair. All of it switches off for people who ask their system for reduced motion.

How we built it

• Plain JavaScript and Canvas 2D with no frameworks, so it runs anywhere as a single HTML file. The one library is Lenis, for smooth scrolling, and the page still works if it fails to load.
• core.js holds the environment, value iteration and the three learners with no DOM code, so the same file runs in the browser and in Node for testing.
• SARSA(λ) uses replacing eligibility traces with a sparse active list, which keeps updates cheap enough to train hundreds of episodes instantly in the browser.
• A headless test suite (node test.js) checks that every map is solvable, every learner improves, and that each experiment actually demonstrates what its text claims.
• The race runs in time-sliced chunks so the page stays responsive while 18 agents train.
• The background is a toroidal grid drawn on one canvas with parallax tied to scroll; agents speed up with scroll velocity.

Challenges we ran into

• An experiment that didn't work. The "coin or goal" lesson was meant to show that a higher γ makes the agent walk past a small nearby reward. In testing, it picked the coin 10 out of 10 times at both γ = 0.9 and γ = 0.99: random exploration always stumbled into the coin first and the agent never discovered the far goal. The fix was optimistic initial values, which push the agent to try unexplored actions. After that, γ = 0.9 chose the coin and γ = 0.99 chose the goal in 10/10 seeds. That became a setting in the lab.
• Honest claims. The cliff lesson originally said Q-learning "learns the path along the edge." Testing across seeds showed the effect is real but noisier, so the text now says what the data supports.
• Making credit visible required tracking credit per time step, not just per state, including for traces where a state is revisited.
• Smooth-scroll jumps landed in the wrong place. Section links stopped 68 px short because the scroll library already applies the page's scroll padding and I was adding my own offset on top. Lesson jumps were still 19 px off after that: loading a lesson changes text above the viewport, the browser's scroll anchoring shifts the page, and the jump had been computed before the scroller noticed. Waiting two frames before jumping fixed it.

Accomplishments that we're proud of

• The blame ribbon turns an abstract concept into something you understand in five seconds.
• Every lesson is backed by an automated test, so the teaching claims are verified, not assumed.
• Works on a phone, in light and dark mode, with no install.

What we learned

• Exploration and credit assignment are tangled together: a learner can't assign credit to a reward it never finds.
• The gap between on-policy and off-policy learning is visible but noisy in practice; one run can mislead you, which is why the race uses multiple seeds.
• Designing an interface that makes an idea obvious is harder than implementing the algorithm.

What's next

• True online TD(λ) and n-step methods alongside the current three.
• Delayed-reward tasks that need memory (a key that opens a distant door).
• A shareable link that encodes a custom map.

Built with

javascript, html5, css3, canvas, node.js, lenis

Links

• Code:
• Demo:

Built With

• agent
• javascript
• rl