DossierAgent

Privacy-first multi-agent desktop application powered by Gemini Flash for intelligent document classification, secure archiving, gap detection, and automated administrative drafting.

Comment

Inspiration

Administrative correspondence from government agencies and other organizations—such as tax authorities, healthcare institutions, pension providers, and employment agencies—often arrives fragmented across emails, PDF attachments, scanned documents, and physical letters.

Extracting important information, monitoring deadlines, and organizing these documents into a reliable archive is time-consuming. Sending complete administrative dossiers to external cloud-storage systems can also create privacy and data-sovereignty concerns.

We wanted to build an autonomous, privacy-first desktop solution that transforms chaotic administrative correspondence into structured, traceable local dossiers while keeping users in control of their sensitive information.

What it does

DossierAgent is a local-first, multi-agent Windows desktop application that supports the end-to-end processing of administrative correspondence.

  • Ingestion and synchronization: Connects securely to the user's Gmail environment to retrieve relevant administrative emails and attachments.
  • Multimodal extraction: Uses Gemini 3.5 Flash to analyze digital PDFs and scanned documents and extract structured information such as dates, reference numbers, organizations, deadlines, and monetary amounts.
  • Agentic classification and routing: Classifies correspondence into administrative categories such as tax matters, employment and benefit records, legal correspondence, and invoices.
  • Newsletter separation: Separates commercial messages and newsletters from important administrative correspondence and provides direct unsubscribe actions when available.
  • Local-first archiving: Stores documents, extracted metadata, and audit information in an organized local dossier structure. Google Cloud AI is used for processing, not as permanent document storage.
  • Archive validation and gap detection: Audits dossier timelines and flags potentially missing payment periods, administrative inconsistencies, and approaching deadlines.
  • Source-grounded document generation: Generates factual administrative drafts—including objections, requests for clarification, and payment-arrangement proposals—using dossier metadata and source references.
  • Human approval and safety controls: Keeps generated documents in draft form for user review and blocks output when the available sources do not support the requested document.
  • Traceable output: Saves generated PDF documents together with supporting metadata, source references, and audit information.

How we built it

  • Desktop application: C# and .NET 10 with a WPF/MVVM architecture, providing a responsive native Windows experience.
  • AI and agent pipeline: Gemini 3.5 Flash is used for multimodal document analysis, prompt orchestration, classification, and schema-constrained JSON extraction.
  • Google agent framework: The Google GenAI SDK connects the Cloud Run gateway to Gemini through Vertex AI.
  • Google Cloud infrastructure: A Node.js gateway is deployed on Google Cloud Run in europe-west4 and communicates with Gemini through the global Vertex AI model endpoint.
  • Authentication: Scoped OAuth 2.0 authentication supports secure interaction with the Gmail API.
  • Local data management: SQLite and the local filesystem provide metadata indexing, dossier organization, synchronization, auditability, and offline access.
  • Reliability controls: Gmail message IDs prevent duplicate processing, attachment hashes verify file integrity, and atomic local registers help protect against incomplete or corrupted state.
  • Environment separation: LIVE, DEMO, and isolated TEST modes keep real user data separate from synthetic testing workflows.
  • Repository access: The project source code is maintained in a private GitHub repository and has been shared with the designated Devpost and Google Cloud judging accounts.
  • Reproducibility: The repository includes setup, deployment, testing, and architecture documentation for the Windows application and Google Cloud gateway.

Challenges we ran into

  • Document variability: Administrative documents use inconsistent layouts and may contain low-resolution scans, complex tables, handwritten elements, or multiple pages. We combined prompt engineering, schema validation, and explicit agent boundaries to reduce extraction errors.
  • Privacy and boundary control: We needed to maintain a clear separation between temporary AI processing and permanent local document storage.
  • Reliable structured output: Legal and financial documents require factual and traceable extraction. Schema-constrained responses, source references, and fail-closed safety controls help prevent unsupported conclusions.
  • Asynchronous desktop workflows: Coordinating background ingestion, AI processing, file operations, and real-time status updates without blocking the WPF interface required careful task management.
  • Duplicate prevention: Repeated synchronization must not create duplicate dossiers or archives. We therefore use Gmail message IDs and persistent processing records.
  • Human oversight: Generated administrative documents must remain reviewable. DossierAgent assists with drafting but keeps the user responsible for verification and submission.

Accomplishments that we're proud of

  • Building a working local-first desktop workflow covering ingestion, classification, archiving, archive validation, gap detection, and document drafting.
  • Combining a native Windows application with Gemini 3.5 Flash, the Google GenAI SDK, Vertex AI, and Cloud Run while retaining local control over permanent dossier storage.
  • Producing structured metadata and professional PDF drafts linked to their underlying dossier records and source evidence.
  • Detecting missing administrative periods and inconsistencies instead of functioning only as a passive document archive.
  • Implementing deduplication, attachment integrity verification, audit metadata, isolated testing, and source-grounded document generation.
  • Deploying and validating the supporting gateway on Google Cloud Run.
  • Maintaining a reproducible private GitHub repository that has been shared with the designated hackathon judges.

What we learned

  • Gemini 3.5 Flash's multimodal capabilities make it suitable for extracting structured information from varied administrative documents.
  • Explicit agent responsibilities and schema-constrained output substantially improve reliability.
  • Source-grounded generation and fail-closed controls are essential when working with legal, financial, and administrative documents.
  • A local-first architecture can still benefit from cloud AI when processing, storage, permissions, and retention boundaries are clearly separated.
  • Administrative automation is most useful when every result remains traceable and subject to human review.

What's next for DossierAgent

  • Local folder monitoring: Automatically ingest and classify new PDFs and scanned documents added to designated folders.
  • Deeper timeline reconciliation: Cross-reference benefit periods, payment specifications, decisions, and correspondence across complete dossiers.
  • Advanced dashboards: Provide interactive views of deadlines, missing records, unresolved actions, payments, and historical trends.
  • Expanded connectors: Support additional email providers and administrative data sources.
  • Stronger verification controls: Add confidence indicators, source highlighting, approval steps, and more detailed audit logs for sensitive actions.

Built With

Share this project:

Updates