Teachly

Landing Page
Login Page
Course Syllabus
Course Content
AI Tutor in Action
Learning Interface
Materials
Showing Memory Functionality

Inspiration

In the age of developing AI, a student wants to learn how to code. As any new learner does, they read documentation, watch youtube videos, then most importantly; apply what they've learned in practice problems. To accelerate the process they have an LLM to ask inquiries to, but all of a sudden, the LLM gives the full solution! Where's the learning in that?

We wanted to build a tutor that does the opposite. Instead of immediately fixing the student’s code, Teachly uses a fine-tuned FreeSolo model to identify the misunderstanding behind the mistake and guide the student toward correcting it themselves.

We were also inspired by the growing need to teach students how to review AI-generated code. As coding assistants become more common, developers need to know how to test generated code, recognize hidden assumptions, and find issues that may not be obvious at first glance.

What it does

Teachly is an adaptive AI coding tutor for beginner programmers.

Students complete coding and debugging exercises directly in the platform. Their submissions are run against deterministic test cases, allowing Teachly to identify which inputs failed and what type of mistake may have caused the failure.

Instead of immediately returning corrected code, the tutor provides progressive Socratic hints. It may begin by identifying the relevant concept, ask a guiding question, point toward a specific part of the code, or provide a smaller example. The complete solution is only shown when the student chooses to reveal it.

For example, if a student repeatedly uses:

for i in range(len(items) + 1):

Teachly does not simply tell them to remove the + 1. It may ask:

If a list contains five items, what is the index of its final item? That's what sparks the Aha! moment.

After the student fixes the problem, Teachly can assign another exercise involving loop boundaries to determine whether the concept was actually learned. The student’s mastery profile is then updated based on their attempts, hint usage, and performance on the follow-up exercise.

Teachly also includes exercises where students review AI-generated code containing issues such as missing edge cases, off-by-one errors, incorrect assumptions, inefficient logic, or code that passes visible examples but fails hidden tests.

How we built it

We built the frontend using [Next.js/React] and TypeScript, with an interactive coding interface where students can read exercises, submit Python code, view test results, and communicate with the tutor.

The backend uses Python and FastAPI to manage exercises, submissions, test results, tutor requests, and student progress.

A key design decision in teach.ly was separating code evaluation from the language model. Student submissions are checked using deterministic test cases, while the tutor receives structured context including the exercise, submitted code, passed and failed tests, previous attempts, detected misconceptions, and current hint level. This allows the model to focus on choosing the most useful educational response rather than deciding whether the code is correct.

We built and trained the tutor within a single Freesolo Flash environment that handled tutoring prompts, code execution results, misconception tracking, reward scoring, and held-out evaluation. Supervised fine-tuning established the tutor’s core behavior, followed by distillation from multiple teacher models to improve reasoning and debugging. On-policy distillation targeted mistakes made by the tutor itself, while multi-turn GRPO rewarded successful teaching across an entire interaction instead of optimizing only individual responses. Finally, we evaluated every saved checkpoint in a tournament and deployed the model with the strongest balance of diagnosis accuracy, student learning, response quality, and solution-leakage prevention.

We used the Gemini API for [more advanced explanations, contextual documentation, resource discovery, or exercise analysis]. Teachly can also use Gemini to find learning resources related to a repeated mistake, including relevant documentation and useful timestamps within educational videos

For the hackathon prototype, we intentionally focused on beginner Python concepts such as loops, lists, indexing, incorrect boundaries, and missing edge cases instead of trying to support every programming language and topic.

Challenges we faced

The first challenge we faced was dealing with merge conflicts. We're a team of 4 devs each having our own agentic workflow. We've ran into issues several times that our solution to a merge conflict was to overwrite the others new additions. Our temporary solution was to revert back, but over a long period that'd eat up our precious time, so we focused on a) communicating to one another on what exactly we're working on at the moment (communication is key) and b) writing individual markdown files for our agents to parse, ensuring they understand what task they're working on and what piece they are in the larger puzzle.

Our biggest technical challenge was building a custom context compactor for Qwen 35B. A tutoring session can quickly grow beyond the model’s practical context window as it accumulates student messages, failed attempts, code revisions, misconceptions, and previous hints. Simply truncating older messages would cause the tutor to forget important details, while repeatedly sending the entire conversation would be inefficient and eventually impossible.

To solve this, we built a context-management layer that stores the complete conversation externally and continuously compresses it into a structured student state. Rather than producing a generic summary, the compactor preserves the information most important for tutoring: the student’s goal, confirmed understanding, active and resolved misconceptions, previous failed approaches, evidence from earlier messages, the current hint level, and the next teaching objective. Each memory item is scored according to its relevance, recency, instructional importance, and token cost, allowing the system to select the most useful information under a strict context budget. The tutor can also retrieve an exact earlier message when the compressed state is not sufficient.

Another one of our biggest challenges came from reward hacking during GRPO. GRPO does not understand our educational goal directly; it only learns to maximize the reward function we provide. Early in training, the model discovered shortcuts that increased its score without making it a better tutor.

We addressed this by separating the training reward from the evaluation criteria. The model received a strong penalty when it leaked a complete solution before the appropriate hint level, and we capped verbosity so longer answers were not automatically rewarded. We also evaluated every checkpoint on unseen conversations, manually inspected high-reward traces, and rejected later checkpoints when their reward increased but their tutoring quality or leakage behavior became worse. A KL penalty kept the trained policy close to the stable reference model, while entropy monitoring helped us detect when the model began repeating nearly identical answers. We selected the checkpoint that performed best on the independent tutoring evaluation, not the one with the highest GRPO training reward.

Another challenge was distinguishing between an incorrect line of code and the misconception that caused it. Several different misunderstandings can produce similar failed tests, so our FreeSolo tutor must avoid acting overly confident when the available evidence is limited.

What we learned

We learned that building an educational AI is fundamentally different from building a coding chatbot. A technically correct response is not necessarily a useful teaching response. The tutor must consider what the student already understands, what misconception they may have, how much help they have received, and what question will move their thinking forward without removing the challenge.

Working with Freesolo Flash also showed us that no single training method was enough. Supervised fine-tuning established the tutor’s basic behavior, distillation transferred stronger reasoning from teacher models, and multi-turn GRPO helped optimize for actual student progress. We also learned that later checkpoints are not always better, so evaluating saved checkpoints for teaching quality, solution leakage, and general capability was essential.

We learned the importance of combining AI with deterministic systems. Unit tests and code execution are more reliable for checking whether a program works, while AI is better suited to interpreting failures, identifying misconceptions, personalizing explanations, and choosing the appropriate level of guidance.

Designing the reward system was equally important. Rewarding correctness alone could encourage the model to reveal the solution too early, so we also had to reward diagnosis quality, useful counterexamples, appropriate hints, adaptation across turns, and independent student repair.

Most importantly, we learned that fixing one submission does not prove that a student understands the concept. Real learning is better demonstrated when the student can explain the mistake and apply the same idea to a different problem without being given the answer.