Elder Buddy

Inspiration

One of my relatives loves playing chess. However, his significant other recently passed away, and he's told us how lonely with no one to play any games with. So we thought about his problem, and we wanted to provide a solution.

What it does

Elder Buddy is a conversational AI companion that plays chess and chats in real time. It listens, speaks, and reacts after each move, and you can converse with it about your moves. It keeps a short memory of mood/topics to steer warmer conversation, and also consistently tries to prompt memory recall to defend against cognitive decline. It's enthusiastic, kind, and infinitely patient.

How we built it

Frontend: single HTML page split with CSS for style: left = chessboard, right = voice panel. Chess: chessboard.js + chess.js, with Stockfish WASM running locally as a Web Worker (no CORS headaches). Realtime AI: WebRTC peer connection to OpenAI Realtime via a FastAPI /openai_realtime_proxy and data channel for session.update (voice, instructions) Backend: FastAPI app with routes: Static hosting for stockfish.js/wasm + piece sprites. Glue logic: On each user move, we send SAN + FEN to the LLM; engine computes a reply move; assistant acknowledges in one friendly line, then plays.

Challenges we ran into:

We originally wanted to use a different voice AI model, but found it too difficult to implement for our uses so we pivoted to a separate one.

When trying to use the js Stockfish files for the chess bot online, we found out browsers block cross-origin workers, so we had to figure out how to host Stockfish locally. Similar problems applied to a lot of things like images for the chess pieces, so we had to download a lot of files locally.

It was difficult figuring out how to implement a good front end with the complicated JS and HTML file setup we had.

Figuring out how to get the chess moves as an input into the model while also allowing for conversation to occur at the same time was really difficult. We had to write our own way to send a message into the model and if the chess move was made during a conversation, it had to be cached to be interpreted after the conversation was over.

Accomplishments that we're proud of

We're really proud of being able to build a fully playable, fast chess with on-device Stockfish + real-time voice that feels alive and is fine tuned to respond with kindness and purpose. Additionally, our front end looks really good as well, allowing for non-technical users to utilize our products simply.

What we learned

Despite never really having encountered things like WebRTC and TTS and not being an expert in JS and HTML, we learned a lot through this process on how to utilize these tools. We also learned integrating personalized AI components was far tougher than we previously thought.

What's next for Elder Buddy

We could add more games that the conversational AI can play, successfully put the application online, and finally if possible implement in nursing homes to help others.

Built With

• css
• html
• javascript
• openai
• tts
• webrtc