Multi-Agent Form Processing System

A multi-agent system for processing government forms. It uses the Google ADK to orchestrate multiple AI agents, each with a specialized function, to analyze and route forms efficiently.

Comment

Inspiration

Government form processing remains a complex and often inefficient bureaucratic process. Citizens struggle with navigating complex paperwork, and government employees spend countless hours manually processing, categorizing, and routing forms. This reality inspired us to create AutoGov—an intelligent form processing system that uses the power of collaborative AI agents to transform how government forms are handled.

The recent release of Google's Agent Development Kit (ADK) presented a perfect opportunity to reimagine this process. By leveraging multiple specialized AI agents working together, we could create a system that not only extracts information from forms but understands their context, classifies them appropriately, and routes them to the correct departments automatically.

What We Built

AutoGov is an end-to-end form processing system built using Google's Agent Development Kit (ADK). It demonstrates how multiple specialized AI agents can collaborate to process government forms, extract relevant information, classify document types, and determine the appropriate routing.

Our system consists of five specialized agents:

OrchestratorAgent: The central manager that coordinates the entire workflow OCRAgent: Extracts text from form images using computer vision NERAgent: Extracts structured data (entities) from text ClassifierAgent: Determines the type of form (FIR, tax return, complaint, etc.) RouterAgent: Suggests the appropriate government department for routing These agents work together seamlessly to process forms and provide structured, actionable information. Users can either upload images of forms or directly input text, and our system handles the entire processing pipeline.

How We Built It

Building AutoGov was an exercise in leveraging the right tools and structuring a complex AI system effectively:

Agent Architecture: We designed a clear architecture where each agent has a specific responsibility. This modular approach follows the single responsibility principle and makes the system easier to maintain and extend.

Tool Development: Each agent has access to specialized tools built on Google's Gemini 1.5 Flash model. For example, the OCRAgent uses a vision tool to extract text from images, while the NERAgent uses a specialized prompt to extract structured entities.

Orchestration Logic: We implemented a sophisticated orchestration pattern where the OrchestratorAgent manages the workflow, passing data between specialized agents and tracking the complete processing chain.

API Development: We built a FastAPI backend with endpoints for both image and text processing, ensuring a flexible interface for integration with various frontend systems.

Frontend Interface: We created a user-friendly Next.js frontend that allows users to upload forms or input text directly, and then visualizes the processing results in an intuitive way.

Cloud Deployment: We deployed the entire system to Google Cloud Run using GitHub Actions for continuous delivery, ensuring a scalable and resilient service.

Challenges We Faced

Building AutoGov wasn't without its challenges:

ADK Learning Curve: As the Google ADK is relatively new, we had to quickly learn its concepts and patterns. This involved understanding how agents interact with tools and how to structure a multi-agent system properly.

Model Deprecation: Midway through development, we discovered that the Gemini Pro Vision model we were using had been deprecated. We had to quickly pivot to using the newer Gemini 1.5 Flash model, which required adjusting our API calls and payload structures.

API Format Inconsistencies: The Gemini API sometimes returned responses in unexpected formats, particularly for JSON outputs. We implemented robust parsing logic to handle these inconsistencies gracefully.

Deployment Complexity: Setting up a secure deployment pipeline with GitHub Actions required careful management of secrets and service accounts to ensure the system could be deployed without exposing sensitive credentials.

Cross-Agent Communication: Ensuring that data flowed correctly between agents and that errors were properly handled required careful design of the orchestration logic.

What We Learned

This project was a tremendous learning experience:

Agent-Based Design: We gained practical experience with designing systems where multiple AI agents collaborate to solve complex tasks.

Advanced Prompting: We refined our skills in prompt engineering to get reliable, structured outputs from large language models.

Cloud Deployment: We learned best practices for deploying AI systems securely to cloud infrastructure.

Error Handling: We developed better strategies for handling unexpected model outputs and API failures in production AI systems.

Frontend-Backend Integration: We improved our skills in building integrated systems where frontend interfaces effectively communicate with AI-powered backends.

Built With

Share this project:

Updates