Nemes Spark

Inspiration

We are very interested in the human-computer interface and we believe that it can be used to improve many aspects of our lives, specifically in mental and emotional well-being. We believe that providing specialized and individual psychosomatic feedback, we can improve in ways that are not detrimental to our bodies.

What it does

This specific project teaches chess to improve focus. We use publicly available content such as youtube clips to create training session that are then monitored and evaluated via AI.

How we built it

This weekend we created the following components :

1. A specialized video analyzer agent for youtube chess videos built using Google ADK
2. A specialized voice bot agent built on the Pipecat framework that uses the video analysis to drive the learning sessions
3. A web UI that displays the lessons and allows the student to interact with the voice bot and the chess board
4. Two WandB dashboards to trace the two agents and to evaluate the models used for video analysis

Challenges we ran into

1. Properly defining the agent descriptions and tools using ADK

Accomplishments that we're proud of

1. The video analyzer provides very in depth useful data that can drive real world teaching sessions

What we learned

1. Profiling LLM models is critical. We learned that for some steps of our system a 7B model performed as well as larger models for a fraction of the price or free.

What's next for Nemes Spark

This is just a building block to create a full fledged feedback system that will help user in various aspect of their lives, combining biometrics, environmental factors and user inputs to gain a deeper understanding of our habits and behaviors and provide suggestions and steps to improve them

Built With

• adk
• flutter
• pipecat
• python
• stt
• tts
• wandb