Thaumazein

Adurino set-up
Server Logs of demo video
Server Logs of tests
Server Logs of tests
Server Logs of tests

Our project introduces Thaumazein, an AI chatbot that integrates speech recognition with facial recognition to create a multi-user conversational partner. The name “Thaumazein” comes from the Greek word for “wonder” and “astonishment,” often associated with academic and scientific discoveries. This reflects the intellectual stimulation and amazement we hope our users feel while interacting with our chatbot.

Inspiration

AI has become increasingly integrated into our daily lives, yet there remains a noticeable gap between AI and the humans who use it. Chat bots forget who we are, fail to remember our preferences, and every interaction with AI lacks continuity and meaningful connection between the users and chat bot. In addition, most AI systems assume that there is only one user conversing with it, meaning that every user interacts under the same profile. This leads to confusion and mixing up of preferences, and sometimes private information can appear to the wrong person. We aim to address the limitation that AI systems often fail to truly recognize who they are interacting with and do not adapt to an individual’s identities, preferences, and habits. Our goal is to reimagine AI like a friend instead of a tool.

What it does

When a user speaks, the system transcribes the audio, identifies the speaker’s face, retrieves their individual conversation history, and generates tailored responses that are then spoken back to the users through text-to-speech processing. We also added an interactive LED display that reflects Thaumazein’s listening, thinking, and speaking states in animation form, mimicking real-life conversations between humans. Instead of using generic AI conversational tools, Thaumazein remembers users and responds to them as individuals, bridging interaction between humans and AI and making AI chatbots one step closer to feeling truly human.

How we built it

Our team is divided into two subgroups, with one group focused on the hardware Arduino portion, and the other focused on the AI prompt engineering portion.

The backend AI system was built with a pipeline consisting of several models and components, with the Google Gemini API acting as the main Large Language Model. With carefully crafted prompt engineering and system instructions, the model’s behavior was tuned to replicate daily human conversations. The same Gemini API is also used to extract crucial user information from queries. These user profiles were mapped to a FAISS memory vector after processing through an embedding model by BAAI, creating a dynamic memory retrieval system to ensure every API query reminded the model of key relevant information about the user. Since each user are assigned a unique user profile document and memory vector, this retrieval system assists the model in remembering details about the user beyond name.

The program code on the Arduino is mainly written in Python, but the LED matrix display animations are controlled using C/C++ sketch code. The program uses OpenAI’s Whisper model to convert human speech into text which gets sent to our backend AI system. We used OpenCV to train a small AI model that can differentiate between our group members’ faces from our USB webcam. We send the most recently seen face to the AI backend so that it can differentiate between the group members. Finally, we use yapper-tts to convert the AI chatbot text into natural-sounding human speech.

Challenges we ran into

Throughout development, we faced several technical challenges. At one point, our USB speakers appeared connected in the operating system but produced no sound, which required us to reflash the Arduino’s firmware to resolve the issue. Slow internet significantly delayed updates, and the Arduino IDE proved unstable, frequently crashing or freezing, especially when checking for updates. To fix a persistent network glitch, we had to access the shell and manually switch Wi-Fi networks using nmtui. Because of these stability issues, we relied heavily on VS Code’s Remote-SSH feature to edit and manage our code more reliably. Additionally, the Arduino’s limited CPU performance made development slower, and we needed multiple USB connections, requiring extra USB hubs. Two facial recognition libraries we initially selected had incompatible dependencies with the Arduino, forcing us to pivot. Limited hardware resources also restricted our ability to fully utilize GPIO pins and sensors, so we ultimately used the Arduino more like a lightweight Raspberry Pi for processing rather than for extensive hardware integration.

Accomplishments that we're proud of

Functional System:

We were able to get real-time face recognition with a webcam
Identify new and returning users and have different profiles for each person
Loads different responses per person

Multi-User Capability:

Multiple people can share one device
Preventing profile confusion
Consistent behavior across sessions

Reliability:

Created a chatbot that is fast enough to respond to a natural conversation
Can handle multiple users at the same time

What we learned

We learned that creating AI for human interactions involved more than generating good responses. Real-time recognition requires balancing speed and accuracy, as well as external factors such as the camera angle and the background that the user is in, meaning that small technical errors can quickly become noticeable to users. We also learned that personalization depends on persistent identity and memory rather than intelligence alone. Finally, the project shows us how important reliability is for interactive AI since even small recognition mistakes can break the flow of the conversation and make the system feel unnatural. Having consistency and responsiveness matter just as much as accuracy.

What's next for Thaumazein

Moving forward, we envision a smarter emotional intelligence layer by detecting user emotions through users' facial expressions, vocal tone, and speaking rate, allowing for more nuanced and empathetic responses. We also plan to expand personalization features, including optional login or password protection to strengthen privacy and user-specific data management. Beyond individual use, the system could be applied across multiple fields, such as education, where it could support personalized instructional design and facilitate multi-student collaborative learning sessions.