-
-
AI face
-
UI
Inspiration
I wanted to build a recommendation system that everyone could access and run without paying for expensive AI APIs, commercial vector databases, or subscription-based services.
From my own experience as a developer, many powerful recommendation systems are difficult for students and independent builders to reproduce because they depend on proprietary models, paid infrastructure, or private platforms. I wanted to prove that a useful, intelligent, and engaging shopping assistant could be created using free, open-source or open-weight technology.
That idea became NAmazon — Voice-First AI Shopping. I built it entirely with free, open-source or open-weight technologies, including Qwen, Sentence Transformers, FAISS, BM25, Faster-Whisper, LiveKit, LiveTalking, and Wav2Lip. Together, they create a complete recommendation experience that can be self-hosted locally without paid AI APIs or commercial vector databases. NAmazon understands shopping intent, remembers user preferences, retrieves relevant products, speaks its responses, and interacts through a talking AI avatar.
My goal is to make modern recommendation technology more open, transparent, and accessible—especially for students, small teams, and developers who want to learn or build without paying per request. Running the project locally requires no paid model API or commercial database; users only need their own computer, while optional cloud infrastructure may still have a separate cost.
What it does
NAmazon is a conversational shopping assistant with two distinct flows:
- Buying Flow: Helps customers who already have requirements such as category, budget, size, color, material, brand, style, or desired features.
- Browsing Flow: Helps customers explore products, categories, styles, and shopping ideas when they have not made a final decision.
For each message, NAmazon:
- Checks whether the request is related to shopping or products.
- Identifies the current Buying or Browsing intent.
- Extracts multiple preferences from a single message.
- Searches the product catalog through BM25, FAISS semantic search, and metadata routes.
- Combines candidates using weighted reciprocal-rank fusion.
- Checks whether there is enough catalog evidence.
- Asks a focused clarification question when information is insufficient.
- Uses Qwen to rerank grounded candidates when evidence is strong.
- Returns ordered product recommendations without inventing catalog items.
Customers can interact through text or microphone input. LiveKit transports browser audio over WebRTC, and Faster-Whisper converts English or Vietnamese speech into text. The response is delivered through LiveTalking, speech synthesis, and a Wav2Lip-powered talking avatar.
When customers confirm a product and size, NAmazon stores an anonymous preference signal. This behavioral memory can improve recommendations in future sessions without storing names, email addresses, or raw microphone recordings.
How we built it
I built NAmazon as a multi-stage conversational recommendation system with separate Buying and Browsing flows.
The retrieval pipeline combines:
- BM25 for exact keywords, product names, and attributes
- Sentence Transformers for semantic product embeddings
- FAISS for fast vector similarity search
- Metadata filtering for category, size, material, color, and budget
- Weighted reciprocal-rank fusion to merge and rank candidates
Before generating an answer, NAmazon checks whether the request is related to shopping and whether enough product evidence is available. If the request is vague, it asks a focused follow-up question instead of generating unsupported recommendations.
When enough evidence is available, the open-source Qwen model reranks retrieved products and produces a grounded response. If Qwen is unavailable, NAmazon falls back to deterministic hybrid ranking.
The system also maintains semantic conversation memory. It remembers user preferences, previous requirements, intent changes, and confirmed products to improve later recommendations.
For voice interaction, I connected:
- LiveKit for browser microphone audio over WebRTC
- Faster-Whisper for local speech-to-text
- LiveTalking for real-time avatar delivery
- Wav2Lip for lip synchronization
- Edge TTS for English assistant speech
I created a supervisor script that validates the required models, dependencies, catalog, and avatar assets before starting LiveKit, LiveTalking, and the NAmazon frontend/backend together.
Challenges we ran into
One of the biggest challenges was balancing exact product constraints with semantic relevance. BM25 works well for exact words and attributes, while semantic embeddings understand vague descriptions and synonyms. However, either method used alone could miss the correct product. I addressed this by combining multiple retrieval routes and merging their results with weighted reciprocal-rank fusion.
Another challenge was preventing the language model from recommending unsupported or nonexistent products. I added a scope classifier and evidence gate so Qwen is called only after relevant products have been retrieved from the catalog. When evidence is insufficient, NAmazon asks the user for more information.
Managing conversational state was also difficult. Microphone transcripts, intent changes, extracted requirements, and remembered preferences could update simultaneously. I designed a single-writer state reducer and checkpointed session state to prevent conflicting updates.
Finally, integrating LiveKit, Faster-Whisper, LiveTalking, speech synthesis, and Wav2Lip required coordinating several real-time services. I created one startup script and a preflight checker to make the full system easier to run and debug.
Accomplishments that we're proud of
I am proud that NAmazon is more than a product-ranking script. It is a complete voice-first shopping experience that can listen to a customer, understand their goal, remember preferences, retrieve products, generate a grounded response, speak, and animate a talking AI avatar.
On all 200 released public evaluation sessions, NAmazon achieved:
- Hit Rate@10: 0.69
- MRR: 0.481871
- Mean Turns to Completion: 4.865
- Technical Score: 0.612261
The Qwen-enabled evaluation recorded:
- Prompt tokens: 2,494,783
- Completion tokens: 76,797
- Total tokens: 2,571,580
The deterministic fallback achieved a Hit Rate@10 of 0.675 without Qwen, demonstrating that the core recommendation pipeline remains functional when the model endpoint is unavailable.
I am also proud that the system is built with free, open-source or open-weight technologies. It can be self-hosted without a paid AI API or commercial vector database.
What we learned
I learned that improving retrieval recall is more important than immediately optimizing the LLM reranker. If the correct product does not enter the candidate pool, even a powerful model cannot recover it.
I also learned that BM25 and semantic search solve different problems. BM25 is effective for exact product terminology, while Sentence Transformers and FAISS handle synonyms, paraphrases, and vague shopping requests. Combining them produces stronger results than using either independently.
Voice interfaces should minimize unnecessary conversation turns. NAmazon therefore extracts multiple requirements—such as category, budget, and desired features—from a single customer message whenever possible.
Most importantly, I learned that an LLM should not be responsible for the entire recommendation process. Using Qwen as a grounded component inside a controlled retrieval pipeline makes the system more reliable, reproducible, and resistant to hallucination.
What's next for NAmazon — Voice-First AI Shopping
The next steps for NAmazon are:
- Connect to live product inventory, pricing, and checkout APIs
- Add streaming speech recognition and interruption handling
- Fine-tune the embedding model for product-search language
- Learn retrieval-fusion weights from evaluation data
- Support more product categories and multilingual conversations
- Add verified first-party product images
- Give users controls to view, edit, or delete remembered preferences
- Improve avatar quality and lip synchronization
- Measure per-turn p50 and p95 latency
- Deploy the system securely with TLS, authentication, monitoring, and rate limiting
My long-term goal is to make NAmazon an accessible, privacy-conscious, and fully open-source AI shopping assistant that anyone can run, study, and improve.
Log in or sign up for Devpost to join the conversation.