• 

• 

Kokage: a better 3D AI character chat app

Kokage is an app where you talk with AI characters you can see and hear. It's available now on the App Store. What makes it better is that it has two sides. The local side runs the whole character on your own device, for free. The cloud side is optional and makes the character's performance more vivid. Together, local inference and a highly efficient cloud keep it affordable however long you use it. Every character is yours to customize and share, and it remembers you. The core libraries behind it are open source.

Inspiration

The project started as a game idea. We wanted to make a game whose characters were driven by AI models running on the player's own device, so they could talk and react freely instead of following a script.

While working on it, we realized that the character's performance had to come first. A character is only convincing if it can speak, move and react like someone who is really there. So we set the game aside and focused on making one AI character perform well. From there, the project naturally became a chat app: the simplest place to meet a character and talk to it.

The best AI models available today can make a character feel vivid, with its own voice, look and movement. But that only matters if people can afford to keep talking. So we combined on-device inference with an efficient cloud to keep Kokage affordable for long-term, everyday use, and added memory so the experience gets better the longer you stay with a character.

The name Kokage (木陰) is Japanese for "the shade of a tree": a quiet place you keep coming back to.

What it does

Local: the whole character runs on your device

Kokage puts a whole character on your phone: its mind, its ears, its voice and its body. After a one-time model download, you can talk to it by voice or text, and on devices with enough memory you can show it things through the camera.

Even without the cloud, the character performs. It's a 3D avatar that listens, thinks with a hand on its chin, and speaks with lip-sync and gestures.

The local side is free. It works offline and without an account, and nothing you say has to leave the device.

Cloud: models that make the performance more vivid

Kokage Cloud is separate and optional. It runs models that are too large for a phone: three tiers of stronger conversation models, plus ARDY motion, voice design and character model generation.

ARDY motion uses a motion-generation model called ARDY to create the character's body movement live, as it talks, so its gestures fit what it's saying.

Voice design lets you describe the voice you want, listen to a few candidates, and keep the one that fits your character. The voice can then deliver a line in different moods, from happy to whispering.

Character model generation turns a description, a picture or both into a complete 3D character model, rigged and ready to move and speak in the app.

Every character is yours to customize and share

You can write your own character, with its own personality and look. You can import characters in the card formats the character community already uses, and export yours in the same formats. You can also start more than one relationship with the same character, and each keeps its own history, memory and look.

Local and cloud together keep it affordable

Local inference carries the everyday conversation, and replies generated on your device cost nothing.

The cloud is used only where it adds something a phone can't do, and it's built to do that at low cost. We chose the models that give the best performance for their cost, and kept the cloud architecture lean and efficient. It's pay-as-you-go, with no subscription: you buy Leaves, the currency for cloud features, when you need them, and they don't expire.

It remembers you

For each character you talk to, Kokage keeps a long-term memory that you can read, edit and delete, and a running summary of your earlier conversations. This history is stored on your device, so the character you've come to know is still there next month.

How we built it

Kokage is a Flutter app for iOS and Android, and it also runs on macOS. To run a whole AI character on a phone, we wrote the core libraries ourselves and open-sourced them at github.com/projectkokage:

• fllamer runs language models on the device, using llama.cpp.
• flvtterm loads and animates VRM 3D avatars.
• fonix runs speech models through ONNX Runtime.
• misakid is a Dart port of the Misaki pronunciation engine, which the voice needs to speak English.

Kokage depends on these four libraries in the shipped app, so they're proven in real use. Any Flutter developer can use them to build on-device AI characters of their own.

Everything is sized to the device, so a phone with less memory gets a smaller model.

The cloud is a single, deliberately simple server. It sends chat to the models we chose and hands voice design, character model generation and ARDY motion to GPU workers.

Purchases go through RevenueCat. We sell Leaves as consumable packs instead of a subscription, because people who mostly chat locally shouldn't pay a monthly fee for a cloud they rarely use. The Android version is still in store review.

Challenges

The hardest part was fitting a whole character into a phone. A language model, speech recognition, a voice and a 3D body all compete for the same limited memory, and each had to give up a little so they could run together.

Small on-device models drift out of character and sometimes remember the wrong things. We learned to treat everything a model says as a suggestion, so the app checks it before letting it change anything.

Our first animations were hand-coded, and they looked stiff. We threw them all out and replaced them with motion clips made ahead of time by ARDY, the same model the cloud uses live.

What we learned

Running locally is not a compromise. Once replies don't depend on a server, privacy, offline use and free everyday conversation all follow.

Small models need good structure around them more than they need to be smarter.

The same words land very differently when they are spoken, timed to the lips and matched with a gesture than when they sit in a text bubble.

What's next

The character's performance comes first: more natural motion, more expressive voices, and better generated character models. We're also making the cloud features more responsive, so motion, voices and generated models arrive with less waiting. Alongside that, we're working on stability, so the app runs smoothly on more devices and through long conversations.

Built With

• 3d
• flutter
• google-cloud
• llm
• local
• rag
• rust
• speech
• voice-design