Tesseract

Proposed training pipeline
seamless AR AI experience
Examples of fine-tuning input data (manually generated in this case)
Python fine tuned AI model

❒ A Tesseract is a 4D Geometric Shape, also known as a hypercube. Our goal is to Transforming Customer Service with Tesseract: Shaping a New Dimension of Interactive Assistance!

Inspiration

We asked 15 small restaurants in downtown Toronto what their number one problem is and four of them said customer service. For example, customers being rude or getting to the front and not knowing what they want, or leaving because the restaurant is busy with no one to answer their questions. From our own experiences working as an ice cream shop employee and front desk agent, customer service is indeed difficult, especially during peak hours.

What it does

Introducing Tesseract, our innovative application that revolutionizes the way questions are answered. With its cutting-edge technology, Tesseract provides realistic and interactive responses through the seamless integration of text-to-speech and speech-to-text functionalities. Powered by state-of-the-art language models, this app delivers an unparalleled user experience.

How we built it

Our Unity-powered platform integrates augmented reality (AR) and frontend capabilities seamlessly. Leveraging the power of API calls to 11 Labs and OpenAI, we ensure a robust and dynamic user experience.

Challenges we ran into

Achieving accurate synchronization between the character's mouth movements and audio proved to be a challenging task. Integrating the fine-tuned model with Unity posed difficulties as the embeddings were generated in Python, hindering the connection. Additionally, incorporating the subscriptions API into Next JS was problematic due to CORS errors.

Accomplishments that we're proud of

This marks our debut in app development with AR features, and we're thrilled to announce that the app is now up and running flawlessly! Additionally, we're proud to present a visually stunning website that complements the app's capabilities.
We take immense pride in utilizing AWS Lambda functions to enhance our restaurant-specific models, with Magic Noodles being a prime example. By extracting information from the restaurant's website through web scraping, we have fed this data into our model, tailoring it to match the unique characteristics of the business.

What we learned

How to develop apps in Unity and use AI in a product.

What's next for Tesseract

We successfully deployed the Python code to backend systems like Firebase Functions and AWS Lambda, enabling additional features. Overcoming challenges, we ensured the functionality of the subscriptions API. We also dedicated ourselves to enhancing the model's performance by fine-tuning it with customer data, training realistic voices in-house, and creating a restaurant portal that allows customers to upload their data and 3D models. As part of our continuous improvement efforts, we're actively working on reducing API response times, potentially by developing our own solution. Furthermore, the AR aspect of our app can be expanded to incorporate geofencing, similar to Pokemon GO, where special 3D objects are tied to exciting offers like coupons and sales.