Hop-It

Draw levels on paper, play it and let AI remix it.

Comment

Inspiration

Recent studies highlight a "creativity crisis" in children, impacting their ability to problem-solve and express themselves. Hop-It tackles this head-on by lowering the barrier to game development. We provide a frictionless pipeline for children to see their handwritten ideas come to life, fostering a sense of agency and technical curiosity.

What it does

Hop-It is a web app that converts hand-drawn grid paper into playable platformer levels. Within its "play view" and editor, it provide the user with tools and feedback from AI agents to further enhance the game based on what they're interested in.

How we built it

  • Started with a Node/Express backend + React/Vite frontend scaffold Integrated Gemini Vision API to convert hand-drawn grid-paper photos into level JSON (50×35 grid with platforms, spikes, player spawn, goal)
  • Built a canvas-based platformer engine with gravity, jump buffering, coyote time, friction
  • Added a live level editor (paint tiles, undo, save), physics sliders, coordinate overlays, and dev mode hitbox visualizer
  • Integrated K2 Think V2 (MBZUAI reasoning model) via SSE streaming to verify level solvability, return a solution path, flag bottlenecks, and suggest design improvements
  • Built an AI autoplay bot that follows K2's solution path as waypoints, with a per-frame safety sensor (edge/spike detection using physical hitbox edges), physics-derived jump durations, gap landing checks, and stuck detection

Challenges we ran into

  • Accurately mapping the grid-paper sketch to the level JSON
  • Getting the AI autoplay bot to behave reliably without dying in easy levels
  • Running out of credits for Gemini API
  • Implementing a senseCAP for handheld device gameplay (ultimately scrapped)

What we learned

  • Vision-to-structured-data pipelines: prompting a multimodal LLM to transcribe gid paper into a precise JSON, including handling ambiguous marks, punch holes, and margin lines
  • Physics engine from scratch: implementing AABB collision, coyote time, jump buffering, friction deceleration, and diagonal velocity normalization without a library
  • LLM API rate limits and model availability: hands-on experience with 429 quota errors, 404 model-not-found errors, and the gap between marketed model names and actual API IDs
  • Iterative AI behavior tuning: learned that small sensor timing errors compound badly (bot dies same way repeatedly), and that correctness requires reasoning about worst-case pixel alignment, not just average behavior

Built With

Share this project:

Updates