Remote Sensing ChatGPT

Inspiration

• Need to make remote sensing interpretation more accessible to non-experts
• Challenge of automating remote sensing task planning
• Impressive performance of large language models like Meta Llama 3
• Potential to leverage LLMs for task planning and execution in remote sensing

What it does

• Understands user requests for remote sensing image interpretation
• Plans and executes appropriate remote sensing tasks using AI models
• Generates interpretation results and language feedback -The tasks that the current A.I is capable of performing are: Satellite Image Object Detection. Satellite Image Instance Segmentation. Image Captioning and Classification. Landuse Segmentation Vegetation Health Index Edge Detection

How we built it

• Used Meta Llama 3 as the core language model
• Integrated various AI-based remote sensing models
• Developed a workflow including:
  • Prompt template generation
  • Task planning
  • Task execution
  • Response generation
• Used BLIP model to caption images and provide visual cues
• Trained task-specific models on public remote sensing datasets

Challenges we ran into

• Meta Llama 3 inability to directly perceive visual concepts in remote sensing images
• Handling unsupported categories or tasks
• Preventing Meta Llama 3 from imagining answers when information is insufficient
• Balancing between using multiple tasks for reasoning and identifying essential tasks

Accomplishments that we're proud of

• Achieved 94.9% accuracy in task planning using Meta Llama 3
• Successfully handled simple and complex queries
• Created a system that can potentially automate remote sensing image interpretation
• Improved accessibility of remote sensing techniques to non-experts

What we learned

• Potential and limitations of using LLMs for remote sensing task planning
• How to integrate LLMs with domain-specific AI models
• Importance of providing visual cues to language models for image-related tasks
• Varying capabilities of different LLMs model in understanding complex instructions

What's next for Remote Sensing GPT

• Develop open-vocabulary remote sensing foundation models
• Explore parameter-efficient fine-tuning of LLMs
• Expand the range of supported tasks
• Incorporate more sophisticated models
• Work towards fully-automated remote sensing interpretation for environmental monitoring and disaster response

Built With

• cuda
• llama
• nvidia
• python