Inspiration

Our approach with Hackathons is to try and build something that will be of greater benefit to those around us. Our greatest inspiration came from Mr Wolfram's interview in the opening ceremony where he advised us that the best projects to build are those that we'd use ourselves. That formed the basis of our idea and made us think of what technology we wished existed. Having survived the recent midterm season, it was clear what we all wanted: a way to make education more effective and engaging for us.

What it does

Instead of the usual teacher persona with an llm wrapper, we opted for educating using the Protege effect. The user journey is simple, a student uploads the material that they want a better understanding of, and they get their own mini student (lovingly named Pip) whom they get to teach and bestow their knowledge upon. To make it engaging we gamified the process. We break down the uploaded materials into smaller chunks of concepts. The student teaches Pip one concept at a time. After each concept, Pip goes into a dungeon where it must pass a fight using ONLY the knowledge it learnt from the student's teachings. During the fight, the student can view how Pip battles each question and also the reasoning it uses for them, in order to identify potential gaps in understanding. Pip can also randomly ask for the student's help in certain questions in order to make sure they also solve some problem questions instead of just being a Teacher. The final dungeon is supposed to be a boss fight representing the exam the student will be giving and Pip's remaining health serves as an approximate grade the student can expect to receive based on their current understanding of the material. We also added cool soundtracks to make it even more engaging!!

How we built it

Pip’s Notebook is a TypeScript game with a Phaser AE browser frontend and a separate Node.js backend. Its central design choice is that the system building and grading the exam can read the uploaded course material, while Pip answers from the notebook the player helped write. That separation makes the player’s teaching matter in combat. Frontend: Phaser AE renders the pixel-art dungeon, sprites, and animations. HTML and CSS handle uploads, teaching conversations, and notebook pages. A typed HTTP client sends player actions to the backend and displays the resulting game state. Frontend integration Game server: Node.js and Fastify run the API. A server-side state machine controls lessons, combat, retries, and scoring. Shared Zod schemas define the data exchanged with the frontend, while answer keys and hidden misconceptions stay on the server. AI and retrieval: The backend uses unpdf to extract PDF text, splits the material into page-linked passages, and uses embeddings to retrieve relevant evidence. OpenAI or Google Gemini helps turn those passages into lessons and verified questions, then grades open-ended answers against the source. A deterministic mock provider supports offline demos. Storage: A local MongoDB database stores the courses, question banks, embeddings, and saved runs by default, with Atlas Vector Search available for retrieving relevant passages. Stored page and section references let the results screen point players back to what they missed.

Challenges we ran into

  1. Refining the Teaching loop for an effective Protege effect. This required a lot of pedagogical research and we really wanted to make our system effective for our users. We struggled a lot with coming up with an effective process and our initial approach of teaching Pip only once before each dungeon and letting it figure things out by itself left a lot out on the table, while also hindering the student's progress and building onto their frustration. After a lot of debate amongst ourselves, we were happy to land on our final approach of a back and forth conversation between the student and Pip, ensuring a comprehensive understanding of the topic, as well as letting the student meaningfully progress without a headache.
  2. Struggled with animating sprites and game art as it was our first time building a game. As cool as it is, game development is still a niche sector in Computer Science and doesn't normally fall under regular Software Development. Two of our group members have always wanted to get into game dev but never really got the opportunity. So we decided to try something new this time and explore what we normally don't get to. We started small with just making Sprites and background art for our various scenes. From there we progressed slowly, with the help of a bunch of tutorials, into making new animations for Pip. Our first ones were just reading and attacking animations. Then we branched into more complex ones such as walking and talking.
  3. Perfecting the System Prompts to make sure Pip doesn't use external knowledge, and the dungeons specifically target potential gaps in understanding. Writing strict JSON structure to keep Pip friendly yet still have deterministic understanding from an inherently stochastic model was particularly challenging, along with making sure that the dungeon questions targeted only misconceptions, instead of testing out-of-syllabus knowledge .
  4. Prompt engineering agents to give us our desired output with deterministic results. Coordinating multiple agents and subagents can be difficult, especially with a complex idea like ours, where precise details do a lot of the heavy lifting and minor differences in interpretation can lead to vastly different outcomes with different efficacies . This is something we had to deal within our own team while brainstorming ideas and that obstacle carried over to our coding agents as well. So we had to spend a considerable amount of time learning how to be concise and structured with our prompts.
  5. Implementing a speech to text function in order to make the teaching phase simpler and more accessible. We wanted to implement a lightweight Speech to text model in order to accommodate more users, but the trade-offs for running it on device was not worth it, so we implemented the browser's built in Speech to Text pipeline which uses Google's servers.

Accomplishments that we're proud of

  1. We are super proud of the way we have been able to polish and perfect the user experience in our final product, especially as first time game devs. We have been able to craft smooth animations and deliver calming soundtracks that complement our intended experience.
  2. We are doubly proud of the innovation we have been able to bring to the education space for ourselves and our fellow students, moving away from a generic LLM wrapper presented with a teaching persona, to something more engaging, exciting, and novel.

What we learned

From a technical viewpoint, we will be leaving this hackathon having learnt multiple new skills which are impressive in their own rights: from game development, to coordinating a large codebase populated with numerous branches and git commits, organizing various different agents and subagents, learning animation, and some artistic skills. However, thats not the only thing we are taking away with us. We also carry a newfound respect and love for the spirit that hackathons promote and the culture they cultivate. Mr Wolfram's evergreen advice of working on something that YOU'd love using and mixing your life and interests with your work is what gave us the confidence in letting go of traditional builds that would pad up our portfolios, to building something that aligns more with what we love and stand for. This perspective not only made our time more enjoyable but also ensured that we looked forward to perfecting our approach every step of the way and ultimately accumulated into a project that we'd be proud to say is ours. Thank you HackPrinceton for giving us this opportunity and offering more than just a hackathon.

What's next for Pip's Notebook

We'd love to keep working on Pip's Notebook and implementing features that we unfortunately didn't have the time to get to.

  1. Implement progression beyond just studying, such as some customizability options based on milestones or quests to make the studying loop even more rewarding.
  2. Expand on our collection of sprites, animations and background scenes.
  3. Also greatly expand on the potential scope of studying, going from individual slide decks to all course materials, leading to massive 30-40 room gauntlets that can be mastered as the student progresses through the course.

Built With

Share this project:

Updates

Submission history