Inspiration
Honestly, this started with us being annoyed at ourselves. Every time we studied, it looked the same: a PDF in one window and three or four AI chats in the others. Copy a paragraph, paste it into a chat, explain which page you're on, get an answer that ignores the diagram, switch to another tab, paste it all again. By the time the concept finally clicked, we had no idea where we were in the chapter anymore.
So we asked a simple question: what if the book could just answer back? No tabs, no copy and paste, no re-explaining what you're reading. You circle the part that confuses you and the help shows up right there on the page, from a tutor who already knows what you're looking at.
What it does
Halo is an AI study buddy for iPad.
- Circle and ask. Draw a loop around anything on the page with Apple Pencil and ask your question out loud or by typing. Halo answers in a note next to the page, in handwriting with little diagrams, while a calm voice talks you through it and highlights the exact line in your book it's talking about.
- Teach me. Pick a chapter and Halo turns it into a short roadmap, so a long chapter becomes 2 or 3 stops. For each one it asks for your guess first (a little rude, we know), teaches from your book's own pages, and checks that you actually got it before moving on.
- Go deeper. Want more? Halo reads the web, finds videos, watches them for you, and skips you straight to the moment that answers your question.
- Notes that get better. Save a lesson as notes, let Halo fill in missing steps, a worked example and a diagram, then turn it all into flash cards or a quiz. After a quiz it tells you what you're good at and what to work on next.
- Halo watches you work. Solve a problem by hand. When you pause, Halo checks each line. If one is wrong it circles it and gives you a hint, never the answer. Still stuck? Tap "Why am I stuck?" and it finds the idea you're missing and the page in your book that explains it.
- Explore in 3D. Spin a real human brain model with 25 touchable parts, tap one, ask about it, and Halo turns the model to show you what it's explaining.
Halo also remembers what tripped you up before, so if you make the same slip twice it'll say something like "this is the same thing that caught you last time."
How we built it
- iPad app: built natively in SwiftUI, with Apple Pencil support for circling and writing. Halo's answers are drawn in a handwriting style so they feel like notes from a real tutor.
- The brain behind it: Gemini reads the page and your question, and explains things one step at a time, so Halo starts talking right away instead of making you wait for the whole answer.
- Voice: ElevenLabs gives Halo one calm, consistent voice that reads each step as it appears.
- Mistake checker: a LangChain Deep Agent reads your handwritten work, then double checks every step with real math before saying anything. It only flags a mistake when it's sure, so a messy value never interrupts a student who is actually right.
- "Why am I stuck?": a LangChain agent that searches through your chapter to find the exact sentence that teaches what you missed, and remembers that gap for next time.
- Lessons: a lesson agent that plans the key ideas of a chapter, checks your answers in your own words, and explains things a different way when you're only partly right.
- 3D: an interactive brain model built with Three.js from a real anatomy atlas, right inside the app.
- Storage on AWS and MongoDB: your documents, Halo's diagrams and the screenshots behind your questions are stored on AWS S3, and MongoDB Atlas keeps track of every file so Halo can find and reload them later.
- Deploy: the server is hosted on Render.
Challenges we ran into
- Where should the answer go? Writing directly on the book made busy pages impossible to read, and floating sticky notes were too small and kept getting in the way. We ended up with a note that sits right next to the page, plus quick highlights on the book that fade away once Halo moves on.
- Speed. Waiting for a full answer feels like forever when you're stuck. We made Halo explain one step at a time, so it starts talking almost right away instead of making you stare at a loading spinner.
- Not crying wolf. Early on, Halo sometimes told students they were wrong when they weren't, usually because it misread messy handwriting. We taught it to double check the math before saying anything, and now it only speaks up when it's actually sure.
- Making the voice sound like a person. Our first voice sounded robotic and kept changing mid-answer. Keeping one consistent voice and having Halo talk like a friend sitting next to you made a huge difference.
- Getting everything to talk to each other. Connecting our hosted server to the database and file storage took more trial and error than we expected, and we had to make sure saving files never slowed down an answer.
- Wrangling the AI. Answers arrive bit by bit, and sometimes a piece showed up half finished. We had to make Halo patient enough to handle that gracefully instead of breaking.
Accomplishments that we're proud of
- The main loop genuinely feels like magic: circle, ask, and handwriting and voice show up together, pointing at your own book.
- A mistake checker that gives hints instead of answers, and only speaks up when it's actually sure.
- "Why am I stuck?" connects a wrong line in your work to the exact page in your textbook that teaches what you missed.
- Halo has a personality. It's an ink blue little creature with a coral halo that naps, winks and cheers you on, without ever getting in the way of studying.
What we learned
- AI gets a lot more trustworthy when it double checks itself with real math instead of guessing.
- Explaining one step at a time isn't just faster. It changes how an explanation feels, because it unfolds the way a real person would teach it.
- Good tutoring is mostly holding back: ask before you tell, hint before you solve, and remember what the student struggled with last time.
- Design matters as much as the AI. Where the answer shows up on screen changed how helpful it felt more than any prompt tweak we tried.
What's next for Halo
- Sign in with Apple, with notes and learner memory synced across devices.
- More 3D models beyond the brain, neuron and cell, and more subjects beyond calculus and biology.
- Group study, where classmates' questions and Halo's answers live in the same margins.
- A teacher view showing which pages the whole class keeps circling, so they know exactly what to go over again. ```
Built With
- amazon-web-services
- backboard
- deepagents
- elevenlabs
- esbuild
- express.js
- gemini
- ipados
- javascript
- langchain
- langgraph
- mathjs
- mongodb
- mongodb-atlas
- node.js
- pdfkit
- pencilkit
- render
- sqlite
- swift
- swiftui
- three.js
- typescript