X68 StackChan

A custom firmware that transforms the official M5Stack Stack-chan into an anthropomorphized Retro-PC SHARP X68000 character.

Comment

Inspiration

X68 StackChan began with a simple idea: transform the official M5Stack Stack-chan into Pekeko-chan, a character inspired by the SHARP X68000.

We wanted to create more than a visual reskin. The goal was to give Pekeko-chan a real sense of presence through X68000-inspired startup effects, expressive faces, head movement, voice conversations, and reactions to being petted.

Another important goal was to avoid dependence on cloud AI services. Speech recognition, language-model inference, and speech synthesis all run locally on a PC or Mac.

What it does

X68 StackChan is a fully local AI conversation agent built around the M5Stack CoreS3 SE.

Its features include:

  • Push-to-talk voice input using the touchscreen
  • Wake-word detection for phrases such as “Pekeko” and “Stack-chan”
  • Speech recognition with faster-whisper
  • Local conversations using an LLM running on Ollama
  • Speech synthesis using Irodori-TTS-Lite or VOICEVOX
  • Three-stage lip-sync based on the audio RMS level
  • 36 facial expressions
  • Yaw and pitch head movement using two Feetech SCS0009 servos
  • Head-pat detection using the top touch sensor
  • Breathing-light and emotion effects using 12 RGB LEDs
  • Random blinking and subtle idle expressions
  • Sleep breathing and occasional sleep-talking
  • Scheduled speech using cron expressions
  • External speech requests from Discord bots or other applications
  • A Human68k-inspired startup screen and X68000-style sound effects

An offline mode is also provided. Without Wi-Fi or a host computer, the CoreS3 can still demonstrate its startup sequence, expressions, blinking, head-pat reactions, and LED effects.

How we built it

The system is divided into two parts: the CoreS3 device and a host PC or Mac.

The CoreS3 firmware was developed with PlatformIO, Arduino, M5Unified, M5Stack Avatar, and M5StackChan-BSP. It handles recording, audio playback, display rendering, lip-sync, servo control, touch detection, and LED effects.

The host-side server was built with Python and FastAPI. A typical conversation follows this pipeline:

  1. The CoreS3 records 16 kHz mono audio using its PDM microphone.
  2. The WAV data is sent to the FastAPI server over Wi-Fi.
  3. faster-whisper converts the audio into text.
  4. A local LLM running through Ollama generates a response.
  5. Irodori-TTS-Lite or VOICEVOX synthesizes the response.
  6. The resulting WAV audio is returned to the CoreS3.
  7. The CoreS3 plays the audio while synchronizing its mouth and head movements.

The API provides separate endpoints for conversations, wake-word detection, short speech synthesis, scheduled utterances, and external push requests.

A Windows 11, WSL2, and NVIDIA GPU configuration uses Irodori-TTS-Lite. Systems without CUDA, including macOS, can use VOICEVOX instead. Both backends work with the same CoreS3 firmware.

Challenges we ran into

Combining embedded hardware, networking, image processing, and local AI created several integration problems.

PlatformIO builds were unstable on Windows because of missing ESP32 Arduino 3.x SDK files and long paths containing Japanese characters. We adopted the pioarduino platform fork and moved builds to the native WSL2 filesystem.

The CoreS3 touchscreen did not behave correctly through M5Unified’s virtual BtnA interface. We solved this by reading the touchscreen state directly through M5.Touch.getDetail().

Direct I²C access to the Si12T top sensor caused repeated transmit-buffer errors. Switching to the APIs provided by M5StackChan-BSP resolved the problem.

Wi-Fi initialization also produced periodic resets and RTOS assertions. These were caused by network requests before initialization and repeated calls to WiFi.begin(). We introduced centralized connection-state management and prevented TCP connections while Wi-Fi was unavailable.

The source images had different face positions and margins. This caused visible movement and white lines during expression changes. We created a tool that analyzes and aligns all 36 facial images in two dimensions.

Audio required careful validation as well. We had to handle recording-buffer limits, WAV headers, PDM sampling, speech-synthesis latency, and synchronization between PCM playback, facial animation, and servo movement.

Accomplishments that we're proud of

Our biggest achievement is integrating speech recognition, an LLM, and speech synthesis without using cloud APIs while still providing responsive physical character animation.

X68 StackChan does more than answer questions. Pekeko-chan uses 36 expressions, three-stage lip-sync, head movement, blinking, micro-expressions, head-pat reactions, RGB lighting, sleep breathing, and occasional sleep-talking.

We also support both Windows with WSL2 and CUDA, and macOS or other systems using VOICEVOX. The TTS backend can be changed without modifying the CoreS3 firmware.

The offline mode made development substantially easier. Hardware behavior can be tested independently before introducing Wi-Fi, the FastAPI server, speech recognition, the LLM, or speech synthesis.

What we learned

Separating AI processing from physical interaction proved to be an effective architecture.

Instead of forcing all AI workloads onto the CoreS3, the device focuses on recording, playback, touch input, animation, and servo control. The host computer performs the computationally expensive AI tasks. This division provides better response quality without sacrificing physical expressiveness.

We also learned that personality does not come from the LLM alone. Small behaviors—brief blinks, synchronized mouth movement, changing expressions, and different reactions depending on how long the character is petted—have a major effect on perceived presence.

For debugging, testing each layer independently was essential. Our most effective sequence was offline hardware testing, server connectivity, speech recognition, LLM inference, and finally speech synthesis. Debugging AI, networking, and hardware simultaneously is an excellent way to create a mystery rather than solve one.

What's next for X68 StackChan

We plan to use the Si12T sensor’s swipe-direction detection so that normal and reverse petting gestures produce different reactions.

We also need to finalize Pekeko-chan’s voice by selecting a suitable reference recording or Voice Design model for Irodori-TTS-Lite.

Other possible improvements include:

  • Stronger coordination between emotion, facial expressions, lip-sync, and head movement
  • Long-term conversational memory
  • Expanded integration with Discord and other external services
  • Autonomous speech based on sensors, schedules, or time of day
  • More X68000-inspired startup sounds and visual effects
  • A simpler installation and firmware-writing process
  • Additional features that work without a host computer

The long-term goal of X68 StackChan is to connect X68000 culture with modern local AI technology and bring Pekeko-chan out of the screen as a character that lives and interacts on a real desk.

Built With

  • arduino
  • codex
  • m5stack
  • m5unified
  • platformio
Share this project:

Updates