World Movers AI-Agent Drone, Customer Service, & Marketing

💡 Inspiration

In today's fast-paced logistics industry, customers expect instant, accurate, and accessible support. We were inspired to create an AI-powered solution that could not only answer customer queries 24/7 but also empower the World Movers Phils Inc. team with advanced tools. The goal was to bridge the gap between traditional customer service and modern AI capabilities, offering a seamless, multimodal experience that enhances efficiency and customer satisfaction. We also saw an opportunity to integrate a responsible marketing tool, provide access to essential freight calculation utilities, offer basic estimation tools for conceptual understanding, visualize potential future drone operations, and keep stakeholders informed with a live industry news feed, all to help the business operate more effectively and innovatively.

🚀 What it does

The World Movers AI-Agent, Customer Service, Marketing, & Operations Hub is a comprehensive Streamlit application designed to revolutionize customer interaction, internal workflows, and operational insights for a logistics company. It offers:

• Multimodal Conversational AI 🗣️📄🖼️:
  • Chat Assistant: Answers questions about services (Air, Sea, Domestic Freight, Customs, etc.), company policies (Terms & Conditions, New Client Info), and internal procedures (importer.docx). Users can upload documents/images for context.
  • Document/Image Analysis: Allows users to upload files (PDF, DOCX, TXT, JPG, PNG) and ask specific questions to extract, summarize, or analyze their content.
  • Voice Commands: Users can speak their queries, which are transcribed and processed by the AI.
• Interactive Media Analysis 👁️💻📸:
  • Live Webcam Analysis: Periodically analyzes a live webcam feed to identify logistics-relevant items or scenes.
  • Screen Analysis: Users can share their screen, and the AI answers questions about the shared content.
  • Take Picture Analysis: Captures a still image via webcam for the AI to analyze based on a user prompt.
• Automated & Assisted Actions ⚙️📧:
  • Quote Request Initiation: Gathers necessary details from users and can automatically prepare and forward a quote request summary to the World Movers team via email.
  • Marketing Email Tool:
    • Enables uploading contact lists (Excel/CSV) or manual entry.
    • ✨ AI-Powered Content Generation: Users can provide a prompt to have the AI draft the email subject and body, which can then be edited.
    • Allows composing personalized email campaigns with placeholders.
    • Sends emails via SMTP, with progress tracking and results.
• Logistics Utilities & Simulators 🛠️🧮🚁:
  • Freight Calculators (External Tools): Provides an interface to access embedded versions of essential freight calculation tools from Freightos.com (Rate, Transit Time, Import Duty, Container Cost, Emissions), with direct links as fallbacks.
  • Basic Estimators (Illustrative): Offers simplified tools for conceptual understanding of Volume/Volumetric Weight, basic Transit Time ideas, conceptual Duty percentages, and rough Emissions ideas, all with strong disclaimers regarding their illustrative nature.
  • Drone Operations Simulation (Conceptual): A visual simulation allowing users to assign targets to conceptual drones, "deploy" them on a map, simulate flight steps, and monitor basic status/battery. This feature demonstrates potential future operational capabilities.
• Market Intelligence & News Feed 📰:
  • Fetches the latest news articles relevant to logistics, freight, and supply chain management from a live news API.
  • Performs a simplified sentiment analysis on each article, flagging it as positive (🟢), neutral (⚪), or negative (🔴).
  • Provides a continuously updating, scrollable feed to keep users informed of market trends and events.
• User-Friendly Features ✨:
  • Text-to-Speech (TTS): AI responses can be read aloud.
  • Downloadable Content: AI responses and analyses can be downloaded as DOCX or PDF files.
  • Custom Themed UI: A visually appealing and branded interface.
  • Contextual Awareness: The AI leverages uploaded content, internal documents, and fetched web data to provide informed responses.

🛠️ How we built it

The World Movers AI-Agent, Customer Service, Marketing, & Operations Hub was built using a stack focused on Python and leveraging powerful AI and web technologies:

1. Core Framework:
   • Streamlit: For building the interactive web application and user interface rapidly.
2. Artificial Intelligence:
   • Google Gemini Pro (Multimodal Models like gemini-2.5): The brain of the assistant, handling natural language understanding, image analysis, document comprehension, response generation, and drafting marketing email content based on user prompts. We utilized the google-generativeai Python SDK.
3. Backend Logic & Data Processing:
   • Python 3: The primary programming language.
   • Pillow (PIL): For image manipulation and preparation for the AI.
   • pypdf & python-docx: For extracting text from PDF and DOCX files respectively.
   • Pandas & openpyxl: For handling Excel/CSV files in the Marketing Email Tool and for structuring data for map displays in the Drone Operations Simulation.
   • Custom Sentiment Logic: A simple keyword-based analysis was implemented in Python for the News Feed feature.
4. APIs & External Data:
   • requests & BeautifulSoup4: For fetching and parsing content from web URLs (Terms & Conditions).
   • NewsAPI (newsapi-python): For fetching real-time news articles for the "News Feed" mode.
5. Media & Interaction Libraries (Optional Features):
   • streamlit-webrtc & av: For integrating live webcam streaming, screen sharing, and picture capture capabilities.
   • audiorecorder & SpeechRecognition: For the voice command input and transcription (using Google Web Speech API).
   • gTTS (Google Text-to-Speech): For converting AI text responses into speech.
6. Communication & Output:
   • smtplib & ssl (Python Standard Libraries): For sending emails (quote requests, marketing campaigns) via SMTP.
   • fpdf: For generating PDF versions of AI responses.
7. Interface Components & Simulation:
   • Streamlit Components (st.tabs, st.map): Used for organizing calculator/estimator UIs and displaying the drone simulation map.
   • HTML/CSS for iframes: Used in the "Freight Calculators" mode to embed external tools.
8. Configuration & Secrets Management:
   • Both environment variables (for Cloud Run) and Streamlit's built-in secrets management (secrets.toml) (for local/Streamlit Cloud development) to securely store API keys and credentials.
9. Deployment:
   • Docker: For containerizing the application for portable and scalable deployment.
   • Google Cloud Run & Cloud Build: For building the container and deploying it as a serverless web application.

챌린지 Challenges we ran into

• Multimodal Context Aggregation 🧠: Effectively combining diverse inputs (text, multiple file types, images, live feeds, internal knowledge documents) into a coherent context for the Gemini API required careful prompt engineering.
• Real-time Media Processing 🎥: Integrating streamlit-webrtc for live video/screen analysis and ensuring smooth, non-blocking operation involved careful handling of asynchronous tasks and state management.
• Dependency Management 📦: Correctly identifying and configuring all necessary system-level dependencies (like those for pyaudio) within the Dockerfile for a successful cloud build.
• News Feed Relevancy and Sentiment Accuracy 📰: Tuning the NewsAPI query to pull relevant articles without too much noise required careful keyword selection. Additionally, the current sentiment analysis is a simplified keyword-based model, so managing user expectations about its accuracy was a key design consideration.
• SMTP Reliability & Deliverability 📨: Sending emails directly via SMTP can be prone to issues with spam filters, rate limits, and authentication complexities (like App Passwords for Gmail).
• Parsing AI Output Reliably 📝: For features like AI-generated marketing emails, ensuring the AI consistently outputs structured data (e.g., distinct "Subject:" and "Body:") and robustly parsing this output was a key challenge.
• Balancing Realism with Simplification for Simulators/Estimators ⚖️: For the "Basic Estimators" and "Drone Operations Simulation," clearly communicating their illustrative nature while still providing an engaging conceptual tool was crucial.

🎉 Accomplishments that we're proud of

• True Multimodal AI Assistant ✨: Successfully integrating text, document, image, voice, webcam, and screen analysis capabilities into a single, cohesive application.
• Effective Use of Internal Knowledge 📚: Enabling the AI to leverage company-specific documents to provide relevant and accurate answers.
• Comprehensive Feature Set 🛠️: Building out a wide array of useful modes, from basic chat and advanced media analysis to a functional marketing tool, a live news feed, and conceptual simulators.
• Real-time Market Insights 📈: Integrating a live news feed with sentiment analysis, providing users with immediate insights into the logistics industry's current events.
• Practical Business Application & Innovation Showcase 🏢🚁: Creating a tool that can bring real value to a logistics company by improving customer service, efficiency, marketing outreach, and demonstrating forward-thinking concepts.
• Successful Cloud Deployment ✅: Overcoming challenges related to dependencies, configuration, and containerization to successfully deploy the application on Google Cloud Run.
• Secure Credential Management 🔑: Implementing a dual strategy for handling secrets that works both for local development (secrets.toml) and cloud deployment (Secret Manager).

🧠 What we learned

• The Power of Multimodal LLMs 🚀: Gemini's ability to seamlessly process and understand combined text, image, and document data is incredibly powerful for building sophisticated AI applications.
• Importance of Prompt Engineering 🎨: The quality of AI output is heavily dependent on well-crafted system prompts and dynamic contextual information provided in each request.
• Effective API Caching and Graceful Degradation ⏳: The importance of using Streamlit's caching (@st.cache_data) to manage API usage and costs for the news feed, and designing features to be entirely optional if an API key isn't provided.
• The Nuances of Containerization 🐳: A deep dive into Dockerfile configurations, including the management of system-level dependencies (like portaudio19-dev) needed for certain Python libraries.
• Cloud Deployment is an Iterative Process ☁️: Debugging on a cloud platform like Cloud Run relies heavily on interpreting logs to diagnose issues ranging from missing secrets and files to syntax errors.
• The Need for Responsible AI Implementation & Clear Expectations 🙏: It's important to build in considerations for privacy and to clearly set user expectations about the capabilities and limitations of illustrative tools like the estimators and simulators.

🚀 What's next for World Movers AI-Agent, Customer Service, Marketing, & Operations Hub

• Database Integration 💾: Store chat histories, marketing campaign results, and news sentiment trends over time.
• Enhanced AI Capabilities 🧠:
  • Function Calling: Enable the AI to call external APIs directly (e.g., for live shipment tracking).
  • AI-Driven News Analysis: Replace the simple keyword sentiment with a more sophisticated analysis using Gemini to summarize articles, identify key players, and detect more nuanced sentiment.
• User Accounts & Personalization 👤: Allow users to save preferences, common queries, and customize their news feed keywords.
• Advanced Analytics & Reporting 📊: For the marketing tool, estimator usage, and summaries of news trends.
• More Sophisticated Drone Simulation 🚁: Implement automated drone movement, draw flight paths, and add conceptual mission objectives on the map.
• Deeper Integration with World Movers Systems 🔗: Connect directly to internal databases or APIs for real-time data lookup.

Built With

• docx
• drone
• drone-control
• dronestream
• freight-calculator
• freight-estimation
• freight-forwarding
• gemini-2.5
• gitlab
• gmail
• google-cloud
• google-gmail-oauth
• gtts
• pdf
• python
• streamlit
• worldmovers