Inspiration
A teacher should be able to sketch an idea and bring it to life without leaving the whiteboard. A student should be able to turn confusing notes into something they can explore. A tutor should be able to build an explanation around the learner in front of them.
MagiBoard starts with a familiar infinite canvas for handwriting and note-taking, then adds a Magic Pen: point at part of the board and ask AI to create or change something there. The vision is an endlessly capable whiteboard for learning, teaching, and anything else you can imagine.
What it does
MagiBoard is a native iPad app that combines Apple Pencil handwriting with editable text, geometry, graphs, equations, images, and interactive objects on one expanding canvas.
With Magic Pen, you select a point or reference area and type or speak a request. Ask for a triangle, change its angles or color, create a graph, or build an interactive example with controls. The assistant uses the surrounding canvas as context and places its work alongside your notes. Interactive objects can expose sliders and other controls, and can create or update real objects on the board through a structured interface.
Students can import PDF references, place pages on the canvas, and annotate them. Teachers and tutors can reuse objects from a local catalog, export notes as images or PDFs, and share editable notebooks. An optional Databricks object catalog saves versioned object snapshots so useful learning materials can be loaded into other notebooks.
The goal is to make the whiteboard feel like a place where you can build an explanation, experiment with it, and keep thinking without switching between separate apps.
How we built it
The iPad app is written in Swift with SwiftUI and UIKit. PencilKit handles handwriting; PDFKit supports document references; WebKit hosts interactive HTML and JavaScript objects. We built the canvas object model, persistence, graph parser, and a native equation renderer supporting a subset of LaTeX.
A Python Mac companion connects the iPad to the Codex runtime. Requests include structured canvas context and a focused screenshot; validated document actions bring the results back onto the board. Voice uses Apple's speech recognition and speech synthesis around the text-based AI requests.
Databricks Unity Catalog volumes provide the optional cloud object library. Saving uploads an immutable object snapshot; loading imports a fresh local copy with its editable properties. This integration stores reusable objects, rather than automatically uploading notebooks or running SQL analytics.
The app and companion were authored for this project with AI coding assistance. Existing whiteboard and note-taking products inspired the interaction design; their application code was not copied or forked.
Challenges we ran into
The hardest part was making a flexible AI canvas still feel like a dependable notebook. Handwriting, objects, panning, zooming, and selection all need to agree on coordinates. AI changes also need validation and undo support so that an assistant does not disrupt the user's work.
Another challenge was making generated objects interactive while keeping their access to the document structured and bounded. Native math layout, voice interaction, and the connection between the iPad and Mac each added their own integration work.
Accomplishments that we're proud of
We brought native handwriting and AI-created, editable objects together in a working iPad prototype. Magic Pen gives requests a spatial reference, so the assistant works where the user is already thinking. We also implemented a reusable object catalog and verified a real save/list/load round trip through Databricks.
What we learned
A useful educational AI tool needs more than good answers: it needs direct manipulation, reliable state, clear feedback, and a way to keep useful results. Giving AI a structured document interface makes its output easier to edit, reuse, and understand.
What's next for MagiBoard
We want to refine Pencil and navigation interactions, reduce AI latency, expand mathematical notation, and build a richer searchable library of educational objects. Longer term, we envision easier sharing and collaboration between teachers, students, and tutors.
This is a hackathon prototype. AI currently requires the paired Mac companion and internet access; equations support a LaTeX subset, and interactive objects use a restricted HTML/JavaScript environment. The long-term ambition is broad, while the current focus is a solid foundation for an expressive learning canvas.
Built With
- codex
- databricks
- html
- javascript
- openai
- pdfkit
- pencilkit
- python
- swift
- swiftui
- webkit
- xcode
