World Movers AI-agent multimodal customer service/marketing

✈️🌐Introducing the World Movers AI Agent– your intelligent logistics partner that understands you however you communicate. Engage via text, voice, document uploads, images, or even live screen

Comment 2

🤔 Inspiration

World Movers Phils Inc. operates in a fast-paced logistics environment where customers need quick, accurate information across various channels. We were inspired by the need to revolutionize customer service and marketing engagement beyond traditional methods. We envisioned an intelligent assistant capable of understanding user needs for support, service inquiries, and even marketing interactions, regardless of how they're presented – whether typed, spoken, shown in a document, or even visible on a screen. The goal was to streamline communication, provide instant customer support, personalize marketing outreach, and directly connect inquiries with actionable steps, like initiating quote requests efficiently. We saw the potential of Salesforce's Einstein Studio, leveraging its Bring Your Own Model (BYOM) capabilities and Agentforce concepts, combined with practical tools like email integration, to create a truly helpful agent for both customers and the World Movers team, all within the trusted Salesforce ecosystem.

✨ What it does

The World Movers AI Assistant is a comprehensive, multimodal support agent built as a web application, designed to significantly enhance customer service operations and marketing effectiveness. It demonstrates the power of integrating advanced AI capabilities within a business platform. It allows users and the World Movers team to:

  1. Engage in Smart Conversations: Ask questions about services (Air, Sea, Domestic Freight, Customs, Warehousing, Trucking), request quotes, get general support, and receive personalized information relevant to marketing campaigns.
  2. Analyze Diverse Content: Upload PDFs, DOCX files, text files, or images. Users can ask specific questions about the content, such as summarizing documents for customer understanding, extracting information from labels for service efficiency, comparing files, or even analyzing marketing collateral for feedback.
  3. Use Voice Commands: Speak commands directly, which are transcribed and processed for seamless customer interaction and marketing inquiries.
  4. Monitor Live Feeds (Webcam): Periodically analyzes a live webcam stream to identify logistics-relevant objects, text, or scenes, aiding in real-time operational awareness for better service.
  5. Interpret Shared Screens: Users or agents can share screens and ask the AI questions about the visible content, facilitating collaborative customer support or reviewing marketing materials.
  6. Analyze Captured Pictures: Take pictures using the device camera for quick analysis, useful for customer issue reporting or visual product/service queries.
  7. Leverage Internal Knowledge: Provides accurate answers based on World Movers documents (Terms & Conditions, new client guides), ensuring consistent customer service and reliable marketing information.
  8. Streamline Quote Requests: When quote details are gathered, the assistant compiles and emails this to the quoting team, accelerating the sales and service cycle.
  9. Support Marketing & Sales Communication: Leverage its understanding of services and documents to help draft responses for common marketing inquiries, provide quick information to sales teams, or suggest summarizations of service benefits, streamlining communication and supporting marketing message consistency.

🛠️ How we built it

  • Core AI Platform for Enhanced Engagement: Leveraged Salesforce Einstein Studio's Bring Your Own Model (BYOM) capability. This allowed integration of a powerful, pre-selected multimodal large language model (details classified), managed securely within Salesforce, to power sophisticated customer service and marketing interactions. The agent architecture is inspired by Agentforce concepts, enabling it to take actions.
  • Application Framework: for the interactive web application interface.
  • Multimodal Input Handling:
    • Text: Standard chat input.
    • Documents: (for PDFs), (for DOCX), standard file handling (for TXT).
    • Images: (PIL) for image processing and handling uploads/captures.
    • Voice: library (using a selected speech-to-text service) and audio capture widget.
    • Webcam/Screen:` for accessing camera/screen feeds and processing video frames.
  • External Knowledge (RAG): to fetch content from URLs (Terms & Conditions, PDF) and for parsing the HTML of the terms page. Content is cached`. This demonstrates grounding the BYOM with specific external data for accurate customer service and marketing information.
  • Action Integration (Email): to securely send formatted quote requests via SMTP, showcasing how AI insights can trigger business processes relevant to customer service and sales.
  • Output Generation: (for Text-to-Speech) and (for downloadable response formats).
  • Prompt Engineering & Orchestration: A detailed system prompt guides the integrated BYOM's behavior. Context from various modes and RAG sources is orchestrated and sent to the model endpoint managed via the Salesforce platform, ensuring responses are tailored for customer service and marketing effectiveness.

🚧 Challenges we ran into

  • Integrating Diverse Libraries: Harmonizing real-time components (WebRTC, audio) and asynchronous AI calls (via the BYOM endpoint) within execution flow.
  • Robust HTML Parsing (RAG): Reliably extracting relevant text from external websites for the RAG component to ensure accurate information dissemination for customer service.
  • Multimodal Prompt Orchestration: Structuring the payload correctly for the BYOM endpoint, combining text (system prompt, query, RAG, file context) and image data effectively for comprehensive understanding in customer interactions.
  • Real-time Analysis & State Management: Managing state for live feeds without overwhelming the application or the AI endpoint.
  • Error Handling & Dependencies: Ensuring resilience against API errors, file processing issues, and managing Python dependencies.
  • BYOM Integration: Ensuring seamless communication and authentication with the BYOM endpoint hosted or managed via the Salesforce platform.
  • Consistent & Appropriate Agent Behavior: Iteratively refining prompts and logic to ensure the AI agent reliably follows complex instructions, like quote handling, and maintains a helpful, professional tone suitable for customer service and marketing contexts.

🎉 Accomplishments that we're proud of

  • True Multimodal Integration for Superior Service & Engagement: Successfully demonstrating the integration of text, voice, document, image, webcam, and screen analysis using a powerful external model securely managed through Salesforce Einstein Studio, transforming how we approach customer service and marketing interactions.
  • Actionable AI for Business Impact: Implementing the crucial email forwarding feature, showcasing how Agentforce concepts can connect AI insights to real business actions like streamlined quoting, directly benefiting customer service and sales processes.
  • Functional RAG for Informed Interactions: Successfully grounding the integrated model with specific external company documents via RAG, ensuring accurate information for customer queries and marketing communications.
  • End-to-End Workflow for Efficiency: Showcasing a practical workflow from diverse inputs through sophisticated AI analysis (via BYOM) to a concrete business output (email), demonstrating a tangible improvement in customer service responsiveness and marketing lead handling.
  • Platform-Centric AI Application: Building a powerful AI tool that leverages the capabilities and security of the Salesforce platform to enhance customer service and marketing functions.

🧠 What we learned

  • The power and flexibility of using Einstein Studio's BYOM feature to integrate state-of-the-art multimodal models for diverse customer service and marketing use cases.
  • Practical application of Agentforce principles in building AI agents that perform actions crucial for enhancing customer support and marketing workflows.
  • Techniques for integrating real-time components within while interacting with external model endpoints.
  • The critical importance of prompt engineering and context orchestration when working with powerful, versatile LLMs via BYOM to ensure accurate and context-aware responses for customer interactions and marketing content assistance.
  • Effective state management and error handling are crucial for building robust AI applications.
  • The value of RAG for grounding even powerful foundation models in specific, timely business context, crucial for reliable customer service and factual marketing.

🚀 What's next for World Movers AI Assistant

  • Deeper Salesforce Integration for Customer & Marketing Workflows: Connect the agent more tightly with Salesforce objects (e.g., create Leads/Cases from quote requests instead of just email, log interaction summaries), boosting sales, service efficiency, and marketing intelligence.
  • Refine RAG for Richer Content & Insights: Utilize Salesforce Data Cloud or Einstein Search capabilities for more advanced, scalable RAG implementation, enabling access to broader marketing materials, detailed service knowledge bases, and competitor information for market positioning.
  • Expanded Agent Actions for Proactive Engagement: Explore adding more Agentforce actions beyond email, such as initiating proactive customer service follow-ups based on interaction history, or triggering targeted marketing campaigns/alerts based on identified customer needs or trends.
  • Improved UI/UX within Salesforce for Agent & Customer: Potentially embed the app (or rebuild key components) directly within Salesforce Lightning pages for a more seamless user experience for both internal teams managing customer service/marketing and for customers interacting with the AI.
  • Hyper-Personalization for Customer Journeys & Marketing: Leverage Salesforce CRM data to personalize interactions, dynamically tailor marketing messages, and offer highly relevant service recommendations.
  • Analytics for Service & Marketing Optimization: Utilize Salesforce reporting and dashboards to track agent usage, performance, and effectiveness, providing insights for refining customer service strategies, marketing campaigns, and product offerings.
  • Proactive Marketing Content Assistance: Develop features to assist the marketing team in drafting initial content (e.g., social media posts, email templates, product descriptions, FAQs for new services) based on approved information and recent customer interaction trends.
  • Sentiment Analysis & Trend Spotting: Integrate sentiment analysis of customer interactions to flag urgent issues for human agents, provide feedback for improving service and marketing messaging, and identify emerging customer needs or market trends.
  • Multilingual Support: Expand capabilities to support multiple languages for broader customer service reach and international marketing efforts.

Built With

  • agentforce
  • artificial-intelligence
  • byom
  • einstein-studio
  • salesforce
  • worldmovers
Share this project:

Updates