AI Travel Activity Planning Assistant

By processing speech on-device, sales reps get real-time insights and guidance while protecting privacy — combining human empathy and AI intelligence to deliver outstanding customer service.

Comment

AI Travel Activity Planning Assistant

💡 Inspiration

The inspiration came from a long-standing challenge in retail and customer service — balancing AI innovation with privacy protection. Many companies wanted to leverage customer interactions for insight, yet real-world conversations often contain personal information that cannot safely be sent to the cloud.

When Chrome Built-in AI was announced, we saw a turning point. For the first time, we could process conversations on-device, keeping sensitive data secure while enabling real-time understanding. That sparked our idea: to build a solution that empowers front-line sales reps through human-AI collaboration — using Chrome’s built-in intelligence to make every customer interaction smarter, faster, and more personal.

🛠️ What it does

AI Travel Activity Planning Assistant is an intelligent assistant that supports travel agents during customer consultations using Chrome’s Built-in AI.

✅ Privacy-First Conversation Processing: All AI analysis runs entirely on-device using Gemini Nano (Prompt API / Summarizer API). Conversation data never leaves the device, ensuring full GDPR compliance.

✅ Real-Time Staff Guidance: The Prompt API instantly analyzes ongoing dialogues, extracting key travel requirements and suggesting the next best question (e.g., “Ask about activity level” or “Confirm dietary restrictions”).

✅ Intelligent Mentoring: The AI functions as a digital mentor, helping even junior agents deliver expert-level consultations from day one — eliminating missed questions and overlooked details.

✅ Human-AI Collaboration: The system balances human strengths — empathy and relationship building — with AI capabilities in data processing and coverage. Staff stay focused on meaningful conversations while the AI handles the rest.

✅ Smart Activity Search: The Summarizer API converts natural dialogue into structured data for our backend API. Gemini 2.5 Flash, enhanced with Google Search Grounding, finds personalized travel activities complete with images and references from platforms like Klook, Trip.com, and Expedia.

🏗️ How we built it

We implemented a secure human-in-the-loop architecture, balancing on-device privacy with cloud-powered search.

Frontend (Privacy Layer):

  • Chrome Built-in AI APIs: Prompt API for real-time analysis; Summarizer API for structured summary generation.
  • Web Speech API: Used for on-device voice transcription.
  • Vanilla JavaScript: Used for lightweight and high performance.

Technical Architecture:

  1. On-Device AI Processing: Customer conversations are analyzed locally using Prompt and Summarizer APIs; zero data transmission.
  2. PII (Personal Identifiable Information) Removal: Prompts are engineered to automatically exclude personal names from summaries.
  3. Real-Time Analysis: Requirements are extracted and suggestions generated at 5-second intervals during active conversations.
  4. Human-in-the-Loop: The AI provides suggestions only, ensuring staff retain control for final decision-making.

Backend (Search Layer):

  • Node.js + Express: Used as the REST API server.
  • Vertex AI (Gemini 2.5 Flash): Utilized Google Search Grounding for multi-platform activity search.
  • Open Graph Scraper: Used to extract image data (og images) from activity URLs.

🚧 Challenges we ran into

Speech Recognition Accuracy: Relying solely on the Web Speech API’s on-device processing caused issues in noisy environments. We needed an additional layer to compensate for the limitations of the Web Speech API and improve transcription quality.

Balancing Privacy and Functionality: We had to find the precise balance of information required to return optimal activity search results without compromising the privacy-first principle of the consultation.

🎉 Accomplishments that we're proud of

High-Accuracy On-Device Transcription: We streamed audio chunks to the Prompt API for transcription. By combining the real-time speed of the Web Speech API with the precision of the Prompt API, we achieved fast and accurate speech recognition while offsetting each API’s limitations.

Human-AI Collaboration Model: We designed an AI system that empowers agents instead of replacing them. This hybrid model automates repetitive tasks while preserving the essential human elements of empathy and emotional connection.

Low-Cost Application Development: Leveraging Chrome’s Built-in AI, we implemented high-quality transcription and summarization entirely on-device — without incurring Cloud API costs. and maintaining user privacy.

📚 What we learned

Prompt Engineering is Critical: Achieving reliable and structured outputs requires meticulous prompt design. Clearly defining response formats, anticipating edge cases, and testing iteratively all contribute to higher consistency and stability.

The Transformative Power of On-Device AI: We discovered that on-device AI unlocks new possibilities once limited by privacy concerns. By focusing its role and optimizing prompts, even lightweight on-device models can deliver highly effective results.

🚀 What's next for AI Travel Activity Planning Assistant

Client Site Implementation: Deploy the solution to client environments across travel, recruitment/HR, car dealers, and insurance sectors — industries that share similar consultation challenges.

Multi-Language Support/Translation: Utilize the Translate API for real-time multilingual translation, enabling high-quality customer service even when language barriers exist.

CRM Integration: Enable seamless export of anonymized conversation summaries to major CRM systems such as Salesforce and HubSpot to maintain consistency and enhance the customer experience.

Tool Calling Integration: Once Built-in AI supports tool calling, we plan to automatically initiate activity searches at the optimal moment — eliminating the need for manual agent input.

Built With

Share this project:

Updates