polyglot-palate

A real-time, Gemini-powered live restaurant concierge agent that allow customers talking to the menu in any foreign lanuages

Comment

Inspiration

I love exploring diverse, authentic restaurants, but I often run into a significant barrier: the more culturally authentic the spot, the harder it is to understand the menu. Whether it’s a language barrier or a lack of dish descriptions, the "mime" act of pointing at random pictures is a subpar experience. I wanted to build a bridge that allows anyone to walk into any restaurant in the world and have a natural, spoken conversation about the food in their own native language.

What it does

PolyglotPalate is an AI-powered, real-time voice concierge for diners.

Menu Ingestion: Restaurant owners upload a PDF menu, which the AI uses as its exclusive knowledge base.

Multilingual Voice Chat: Customers can speak to the agent in their native tongue to ask about ingredients, spice levels, or recommendations.

Intelligent Interaction: Powered by the Gemini Multimodal Live API, the agent supports natural interruptions and automatic language detection.

Grounded Responses: Unlike generic chatbots, it is strictly grounded to the uploaded menu, ensuring it never "hallucinates" dishes the restaurant doesn't actually serve.

How we built it

My development followed a high-velocity "vibe coding" workflow:

Prototyping: I used Google AI Studio to rapidly iterate on the system instructions and test the multimodal capabilities of Gemini 2.0 Flash.

Development: I used the Antigravity code editor paired with AI agents to build out the React frontend and Node.js backend, allowing me to focus on the "vibe" and logic while the agents handled the boilerplate.

Deployment: I managed the source code via GitHub and used the gcloud CLI to containerize the application and deploy it as a serverless service on Google Cloud Run, ensuring it scales effortlessly.

Challenges we ran into Handling Real-Time Multimodal Interactions: Implementing low-latency, interruptible voice-to-voice communication is notoriously difficult. We resolved this by leveraging the Gemini 2.0 Flash Multimodal Live API, which natively handles audio streaming and interruptions via WebSockets.

AI Context Hallucinations: To prevent the AI from suggesting items not on the menu, I implemented strict System Instructions and a preprocessing layer that extracts text from the PDF to constrain the AI’s knowledge.

Client-Side Authentication & Security: Initially, I hit 403 PERMISSION_DENIED errors because Vertex AI SDKs block project-based authentication in browsers. I resolved this by moving the authentication logic to a Node.js backend proxy, ensuring API keys are never exposed to the client.

Environment Variable Management: I discovered that Vite/React files served via Nginx couldn't read environment variables at runtime. I replaced the Nginx setup with an Express server to dynamically inject configuration variables into the frontend during execution.

IAM Permissions: Deployment to Cloud Run originally failed due to permission gaps. I manually configured the default service account with storage.objectAdmin, artifactregistry.writer, and logging.logWriter roles via gcloud commands to ensure a smooth CI/CD flow.

Accomplishments that we're proud of

The most rewarding moment was hosting the app publicly and taking it into the wild. I tested PolyglotPalate at a local Italian restaurant, and it performed flawlessly—translating complex dish names and handling questions about allergens in real-time. Seeing "vibe coding" transition from a local prototype to a functional, real-world tool was incredible.

What we learned

The biggest takeaway? "Vibe coding" a frontend is easy, but deployment into production is hard. While AI can help you stand up a UI in a few prompts, moving to the cloud requires deep consideration for security, API key management, environment variables, and long-term maintainability through version control. I learned that the "vibe" gets you to 80%, but engineering discipline gets you across the finish line.

What's next for PolyglotPalate

The goal is to turn this from a translator into a full-service dining assistant:

Order Management: Integrating with restaurant POS systems so customers can submit their orders directly through the voice chat.

Vision Integration: Leveraging Gemini's multimodal nature to allow users to point their camera at a dish or an ingredient they are allergic to for instant recognition.

Payments: Adding a checkout layer so users can settle their bill without waiting for a physical check.

Built With

Share this project:

Updates