Context Avatar Duel

## Inspiration

The project was inspired by the need to empirically quantify the value of context in Large Language Models. While generic models like GPT-5 are powerful, they often lack the depth, specificity, and authenticity that come from personal context. This project was designed to create a framework for evaluating and visualizing the difference in response quality between a generic model and a context-aware agent.

## What We Learned

(a) How to use W&B Weave to create a robust evaluation framework for LLMs. By defining custom scoring functions and using Weave's tracing capabilities, we were able to objectively measure metrics like context utilization, evidence density, and factual grounding. (b) API Integration Challenges: Integrating multiple APIs (OpenAI, ElevenLabs, Ready Player Me, and a custom backend for Craig's integration) presented significant challenges, particularly with CORS and authentication.

## How we built It

The project is composed of two main parts: an iOS application and a web-based evaluation platform.

The iOS App: Built with Unity, the iOS app integrates Ready Player Me avatars and OpenAI's GPT-4o to create an interactive chat experience. It features a dual-personality system, allowing users to switch between a friendly AI and the more argumentative "Craig" mode, which leverages personal context from the Signal app.
The Evaluation Platform: The web platform, built with React and deployed on Vercel, serves as a "demo stand" for comparing the generic and context-aware models. It visualizes the conversation with animated avatars and provides a side-by-side comparison of the responses. The backend is a Python service that uses W&B Weave to score the responses based on a set of predefined metrics.

## Challenges:

CORS and API Integration: As mentioned, integrating multiple third-party APIs from a client-side application was a major hurdle. The solution was to create a backend-for-frontend (BFF) using Vercel serverless functions to proxy all API calls.
Real-time Lip-Sync: Achieving believable lip-sync required a deep dive into the Web Audio API and careful synchronization of the audio stream with the avatar's blend shapes.
Objective Evaluation: Defining a set of objective metrics to evaluate the quality of an LLM's response was a significant challenge. I settled on a combination of quantitative measures (e.g., citation count) and qualitative scores (e.g., emotional authenticity) to create a comprehensive evaluation framework.