loews

• 

  RAG Chat

Inspiration

The idea for the Loews project was inspired by the recurrent locust outbreaks in Kenya, which have had devastating impacts on agriculture and food security. Witnessing the severe consequences of these outbreaks on farmers and the economy, I was motivated to create a solution that leverages technology to provide early warnings and mitigation strategies. My background in geospatial data analysis and machine learning provided the foundation for developing a system that could predict and visualize locust breeding grounds, thereby aiding stakeholders in taking timely and effective action.

What I Learned

Building the Loews project was an enriching experience that deepened my understanding of several key areas:

• Geospatial Data Analysis - I enhanced my skills in processing and analyzing geospatial data to identify patterns and trends.
• Machine Learning - Training the Random Forest Model on geospatial data points helped me understand the intricacies of feature selection and model optimization using hyperopt and mlflow.
• Data Visualization - Utilizing the Folium Library for mapping and visualizing data provided insights into creating intuitive and informative visual representations.
• Chatbot Development - Implementing the Retrieval Augmented Generation (R.A.G) technique for the chatbot functionality taught me about vector stores, embeddings, and integrating external data sources to counteract AI limitations.
• Synthetic Data Generation - Using the Faker Library to simulate farmer data showcased the importance of robust data generation methods in testing and validating systems.

How I Built the Project

Data Collection and Processing

• Synthetic Data - Generated synthetic farmer reports on locust sightings using the Faker Library for visualization purposes.

Model Training

• Random Forest Model- Trained on the collected geospatial data to predict locust breeding grounds.
• Model Optimization - Utilized MLflow for tracking model parameters and optimizing the Random Forest model using hyperopt/collab.

Visualization

• Folium Library - Created interactive maps to visualize potential breeding grounds with markers indicating land sizes and probabilities.
• Analytics Page - Developed visualizations using synthetic data to provide insights on locust sightings and trends.

Chatbot Functionality

• R.A.G Technique- Implemented using FAISS for vector store simulation, OpenAI Embeddings, and Langchain Library.
• Information Retrieval- Designed to provide updated information on Highly Hazardous Pesticides (HHP) in use in Kenya, with plans to extend to locust outbreak updates and mitigation strategies.

Local and Docker Setup

• Local Setup - Detailed instructions are provided for setting up the project locally, including creating a virtual environment, configuring API keys, installing libraries, migrating databases, and running the server.
• Docker Setup - Instructions for building and running the Docker container for easier deployment and management.

Challenges Faced

• Data Availability- Finding reliable and comprehensive geospatial data was challenging, hence the use of Synthetic Data.
• Chatbot Reliability- Ensuring the chatbot provided relevant and up-to-date information without hallucinations was difficult, an Issue I could not fully overcome.

What's next for Loews

• Automated ETL Process - Setting up an automated ETL (Extract, Transform, Load) process using Airflow to streamline data collection and processing. Real-time Predictions - Implementing models on AWS for real-time predictions and integrating them into the app for immediate visualization and alerts.
• Multilanguage Support - Incorporating support for multiple languages to make the system accessible to a broader audience.
• Enhanced Referencing - Improving the method of referencing chat outputs to include exact text chunks and associated citations for better context and reliability.

By overcoming these challenges and continuously improving the system, I aim to make Loews a valuable tool in combating locust outbreaks and supporting the agricultural community in Kenya.

Built With

• django
• faiss
• folium
• gpt3.5-turbo
• hyperopt
• mlflow
• plotly
• python
• sqlite