AMD RyzenAI Challenge

Inspiration

In the generation of AI, we wanted to contribute to the best use cases of AI chips, and AMD provided us the opportunity to do so.

What it does

It's an interactive AI assistant designed to provide information about CU Boulder. It combines: Face detection using computer vision. Speech recognition that automatically activates when a face is detected Natural language processing to answer questions about CU Boulder (we used the quantized model that we built with the help of AMD quark and the LLM uses hybrid model to use iGPU and NPU whenever necessary). Text-to-speech response capabilities too.

How we built it

Frontend: React (with Vite) for the user interface WebRTC for accessing the webcam Web Speech API for speech recognition and synthesis Context API for state management across components Tailwind CSS for responsive design Backend: Flask server handling API requests from the frontend AMD Ryzen AI NPU (when available) to run DeepSeek on the edge. PDF processing using PyMuPDF to extract information from CU Boulder documents Vector embeddings (FAISS) for semantic search capabilities LLM integration (using Ollama) for natural language understanding and response generation

Model Quantization

AMD Quirk to quantize DeepSeekR1

Challenges we ran into

Establishing reliable communication between the frontend and Flask backend Implementing proper error handling for network issues.

For running the quant model, the template code did not work so we had to write our own code to cater to it.

During the training, we ran out of disk space for Kaggle and Google Colab, so we had to buy Google Colab Pro for larger memory (RAM).

Accomplishments that we're proud of

Hardware Acceleration: Successfully integrated with AMD Ryzen AI NPU.

General Error Handling Developed comprehensive error handling to prevent crashes and blank screens Implemented detailed logging for better debugging and development

What we learned

Browser APIs for Natural Interfaces: Mastered the Web Speech API and its nuances across different browsers Learned techniques for managing webcam access and processing video frames

Hardware-accelerated AI: Gained experience integrating with specialized AI hardware (AMD Ryzen AI NPU) Understood the performance benefits and implementation challenges of hardware acceleration

React State Management: Refined our approach to handling complex state across multiple components Learned best practices for context providers when sharing critical status information

Error Resilience: Developed deeper understanding of defensive programming techniques Learned how to predict and handle various failure modes in web applications

What's next for AMD RyzenAI Challenge

Help AMD advance in the field of AI chips: Build developer mindshare for their AI acceleration capabilities. Create differentiation against both Intel and NVIDIA in the rapidly growing edge AI computing market. Expand from consumer and gaming markets into professional, commercial and specialized vertical markets.