Skillpilot

An intelligent system that creates questions on the fly, spots where someone needs help right away, and makes unique learning plans.

Comment

Inspiration

Most "skill assessments" online are the same thing wearing different skins: a fixed question bank, a score at the end, and nothing about what to study next. You can pass a Python quiz on loops and still not understand recursion, and the quiz won't tell you that's the gap.

We wanted the opposite: treat a subject like a dependency graph instead of a flat trivia list, work out in real time which pieces you have solid and which ones are shaky, and point at exactly what to look at next. Not a grade. A map.

What it does

You start by picking a subject: Python, JavaScript, SQL, Git, Docker, Linux, or something else technical. SkillPilot's AI breaks that subject into a concept blueprint, a prerequisite tree of the ideas you need and how they depend on each other, instead of handing you a random grab bag of questions.

From there, it pulls questions in real time, mostly from QuizAPI.io and topped up with AI-generated ones whenever a concept is too niche for the question bank, grades your answer on the spot, and keeps a running proficiency estimate for every concept on a 0 to 1 scale. Each concept gets labeled STRONG, DEVELOPING, WEAK, or UNCERTAIN based on that estimate, and you can watch it shift live on an interactive skill graph as you go.

Once there's enough signal, SkillPilot puts together a personalized report on where you stand, concept by concept. There's also a tutor chat sitting right on the dashboard, so if something comes back WEAK, you can start asking about it immediately instead of opening a new tab. You don't need an account to try any of this, either: guests get a cookie-based session so they can poke around before committing to signing up.

How we built it

SkillPilot is a Next.js 14 app end to end (App Router, Server Actions), written in TypeScript, with Postgres behind Prisma for everything from user accounts to per-concept proficiency history.

The AI side is split across two providers on purpose. Groq handles the fast, frequent calls, generating individual questions and classifying whether an answer is right, where low latency matters more than depth. Featherless AI handles the heavier reasoning: building the initial concept blueprint for a subject and writing the end-of-session report, where waiting a few extra seconds for a better result is a fair trade. We think of it as a router more than a single AI backend.

Getting reliable structured output out of either provider took more work than we expected. Every AI response that's supposed to come back as JSON gets validated against a Zod schema. When it doesn't pass, maybe because a model added a stray sentence before the JSON or produced almost-valid JSON with a formatting slip, we retry. If that fails too, a fallback extractor tries to salvage the structured data out of the raw text rather than failing the whole request.

The proficiency model itself is Bayesian rather than a simple percent-correct counter: each concept carries a running estimate of P(mastery | responses so far) somewhere between 0 and 1, recalculated after every answer. That's what lets a concept move between STRONG, DEVELOPING, WEAK, and UNCERTAIN as evidence comes in, instead of staying pinned to whatever your first few answers happened to be.

On the interface side, answer grading is a plain string match, so it can respond in well under 100ms, while the more expensive work, updating the Bayesian estimate and writing analytics, runs in the background instead of blocking the click. The skill graph is ReactFlow, proficiency charts are Recharts, and the rest of the interface leans on Tailwind, Radix UI, and Framer Motion for the transitions. Auth runs through NextAuth.js with a Prisma adapter, plus a guest-cookie path for people who don't want to make an account just to try it.

Challenges we ran into

The two-provider setup was the trickiest thing to get right. Groq is fast, but the models available there aren't always the strongest at multi-step reasoning; Featherless gives us access to bigger models at higher latency. Deciding which calls go where, and keeping the slower path from feeling broken while it's thinking, took a few iterations to get comfortable with.

The bigger headache was getting AI models to reliably produce clean JSON. A concept blueprint or a graded response coming back with a stray sentence before the braces, or a trailing comma, was common enough early on that we couldn't just trust the raw output. That's the whole reason the Zod validation and retry layer exists: without it, one slightly-off response would quietly break the concept graph or the grading pipeline downstream, and it wasn't always obvious where in the chain things had gone wrong.

Keeping the skill graph readable was its own problem. A subject with a genuinely tangled set of prerequisites can turn ReactFlow's default layout into a hairball fast, so we had to think carefully about how deep a concept blueprint should be allowed to nest before it stops being useful to look at.

We also had to be honest about the question supply. Live questions from QuizAPI.io cover well-known basics fine, but for a niche concept the bank sometimes doesn't have enough, or has nothing that maps cleanly to what we're testing. The AI top-up fallback exists because we ran into that wall more than once.

Accomplishments that we're proud of

The whole pipeline holds together: blueprinting a subject, sourcing or generating its questions, grading in real time, updating a live proficiency model, and writing a final report, all running across two different AI providers without the seams showing on the user's end. For a two-person team on a hackathon clock, that's a lot of moving parts to keep synced.

We're also proud that the app feels fast even with that much AI happening behind it. Answer feedback comes back in well under 100ms because grading never waits on the heavier analytics, which run afterward in the background instead.

The reliability layer around AI output, the Zod schemas, retries, and fallback extractor, isn't glamorous, but it's the reason the app doesn't fall over the first time a model has an off day. And letting people try SkillPilot as a guest, with no signup wall in front of the actual product, was a deliberate choice we're glad we made.

What we learned

Working with two LLM providers back to back taught us to code defensively around AI calls: validate everything coming back, never assume the first response is usable, and always have a fallback path instead of a crash. That habit will probably outlast this project.

We also learned that "adaptive" is a real modeling problem, not a feature you bolt on. It would have been easy to fake adaptivity with a simple percent-correct score, but that doesn't hold up once you want to say something meaningful like STRONG or UNCERTAIN about a specific concept. Getting that right meant sitting down with the Bayesian estimation properly instead of approximating it.

On the engineering side, we got a much better feel for where to draw the line between synchronous and background work in a Next.js app. Not everything needs to finish before the UI responds, and figuring out which parts of a request sit on the critical path made a bigger difference to how the app feels than any amount of visual polish would have, something we probably figured out later than we'd like to admit.

What's next for Skillpilot

A few things we want to build next:

  • More subjects and deeper trees. The concept blueprints currently cover things like Python, JavaScript, SQL, Git, Docker, and Linux; we want to go deeper on each one and add subjects people have asked for.
  • Spaced review instead of one-off assessment. Right now, a WEAK or UNCERTAIN concept just sits there until you come back to it. We'd like SkillPilot to nudge you back on a schedule that adapts to how quickly you tend to forget things.
  • A shareable skill profile, something a learner could put in front of a recruiter or mentor that means more than a resume bullet point, since it's backed by an actual assessment instead of a self-rating.
  • A classroom or team mode, where an instructor or team lead can see gaps across a whole group instead of one profile at a time.
  • A tutor with memory. The AI tutor chat doesn't carry context between sessions yet; giving it a memory of what you've struggled with before is an obvious next step.
  • Smarter routing between Groq and Featherless, picking per request based on cost or load instead of a fixed assignment.

Built With

Share this project:

Updates

Submission history