ClassPilot

• 

  ClassPilot in action

How I built it

• Slack Bolt SDK (JavaScript, Socket Mode) to run the agent inside a Slack developer sandbox — no public URL needed since Socket Mode only makes outbound connections
• Custom MCP server exposing tools for searching the question bank, retrieving formulas, and logging/summarizing topic gaps
• Google Gemini (gemini-flash-latest) as the reasoning model deciding which tool to call and how to answer
• Supabase (Postgres) storing the question bank and the resolved/unresolved log
• Block Kit for formatting answers and capturing feedback (resolved / still stuck) directly in the response

The question bank was seeded across multiple 200-level Systems Engineering courses spanning both semesters — not just one course — using Gemini to extract topics, subtopics, and formulas from real scheme-of-work material, giving the first version broad, real content to answer from instead of starting empty.

Challenges I ran into

Getting the Slack CLI installed was the first blocker — my default network's DNS resolver couldn't resolve Slack's CDN domain at all, failing with a server-side error rather than a normal "not found." Switching to a public DNS resolver fixed it, but it cost real time before a single line of the agent existed.

Gemini's function-calling had a subtle, undocumented-until-you-hit-it failure mode: when the model made more than one tool call in a single turn, only the first call reliably carried the "thought signature" the API requires to validate the next request — the second call would come back without one, and the whole request would get rejected. The fix was patching in Google's own documented placeholder signature for any tool call missing one, which isn't something you'd guess without digging into their changelog mid-build.

Getting the agent to actually use its tools consistently was its own problem — for questions it was "confident" about, it would sometimes answer straight from its own knowledge and skip calling search_questions entirely. That's fine for the answer, but it meant nothing got logged for gap-tracking, which defeats the whole point of the tool. Fixing this took an explicit system-prompt rule forcing a tool call on every academic question, not just the ones the model felt unsure about.

Slack's message format ("mrkdwn") doesn't support LaTeX or Markdown headers, and Gemini defaults to writing both — formulas came back as raw $$...$$ and ### symbols instead of clean text the first few times, until the prompt explicitly spelled out Slack's formatting rules.

Even sandbox access had a last-mile snag: the default sandbox template pre-loads seven placeholder users, filling the seat cap before you can invite anyone real — including the judges' emails required for submission. Fixing it meant deactivating those placeholder accounts at the organization level, not just removing them from the workspace.

What I learned

The value of MCP wasn't the protocol itself, it was the separation it forces — the agent's reasoning stays clean because it doesn't need to know how the question bank is stored, only that a tool exists to search it. That made swapping and iterating on the data layer painless.

I also learned that gap-tracking is only as good as the taxonomy behind it, and only as good as the model's willingness to actually call the tools that feed it. The technical build (agent, MCP server, Slack integration) turned out to be the easier half — getting the model to behave consistently, and getting the data categorized in a way that's actually useful to a lecturer, took more iteration than the code itself.

What's next for ClassPilot

Expanding the question bank further, piloting it with a real class group actively migrating off WhatsApp, and closing the loop on the gap data — surfacing not just "students are stuck on X" but a suggested resource or a lecturer nudge when a gap keeps recurring, rather than just a raw count.

Built With

• gemini
• node.js
• postgreesql
• slack
• supabase