Onoki

• 

  Self portrait

• 

  Explaining a pulley 1/2

• 

  Telling stories

• 

  Explaining biology (cell membranes)

• 

  Explaining a pulley 2/2

• 

  Explaining an important invention of Archimedes

Inspiration

My children use ChatGPT, but it is not automatically in a learning companion role. I wanted to create an app that my children and others could use to learn about any topic.

What it does

Onoki is an app with support for on-device user profiles. You can create learning topics and goals per profile. You can enter a session for a topic/goal or a free session. This starts an agent loop with speech input and speech output optimized for low latency. The agent loop works transport agnostic on SST->LLM->TTS engines, and on realtime sessions (OpenAI Realtime API and Gemini Live API). The agent has a range of tools for tracking learning progress, memory about the user, suggesting goals, and most importantly for creating and updating visual scenes using a custom, token-efficient DSL. The DSL supports state and animations, and additional syntax exists for geometry constructions and charts/diagrams. Onoki uses the "Socratic" method, where the agent guides, figures out what the student knows, and then asks the right question to build true understanding. It activates thinking and arriving at the solution.

How we built it

I am working on this since Opus 4.5 came out. During the hackathon timeframe, I used GPT 5.6 to build a harness for testing and optimizing the DSL end-2-end. Without knowing the DSL first, GPT 5.6 Sol came up with learning situations, then learned the capabilities of the DSL and identified gaps. It built an evaluation test set to run in the harness and validate the improvements it made and new DSL features implemented.

Challenges we ran into

A custom DSL is tailored to the use-case, but leans heavily on in-context learning. We trade low latency for reduced ability of the model to map the task (learning situation) onto a good output (in the DSL syntax).

Accomplishments that we're proud of

The test harness and end-2-end validation is a substantial improvement and paves the way for largely unattended self-improvement iterations.

What we learned

You gain a lot of efficiency and leverage by "closing the loop". Different LLMs have very different capabilities to create 2D scenes and build up the learning session step by step.

What's next for Onoki

Refinement and App Store release (currently in TestFlight).

Built With

• rust
• swift
• tauri
• typescript