PantryPilot

• 

  1-dashboard_before_scanning

• 

  1-dashboard_before_scanning-2

• 

  1-dashboard_before_scanning-3

• 

  6-dashboard_search_in_action

• 

  1-dashboard_after_scanning

• 

  5-dashboard_after_approval

• 

  9-sample receipt

• 

  10-architecture

• 

  7-frontend_started

• 

  8-reset_demo

Inspiration

The inspiration for PantryPilot came from observing the severe operational bottlenecks faced by local food banks. While AI is frequently used for consumer-facing chatbots, its potential to automate the unglamorous but critical back-office logistics of non-profits is vastly underutilized.

Food banks operate on razor-thin margins and rely heavily on volunteer staff. However, the donation intake process is highly fragmented: donors send unstructured messages via SMS, email, and voicemail. Staff must manually transcribe audio, parse details, log inventory, and generate IRS-compliant tax receipts. This administrative burden leads to burnout, data entry errors, and lost future donations. I wanted to build a system that doesn't just "read text," but acts as a reliable, compliant, and transparent administrative co-pilot for non-profits.

What it does

PantryPilot is an autonomous, multi-agent AI platform that automates the entire donation lifecycle. It ingests unstructured multimodal inputs (SMS, Email, and Voice messages) and uses Qwen's multimodal models to transcribe and parse the intent. The setup was intentionally designed to make changing the model very easy.

Instead of acting blindly, the system presents a transparent "Agent Reasoning" card to the staff, detailing exactly what it plans to do. Once the staff member clicks "Approve & Log" (Human-in-the-Loop), downstream agents automatically trigger:

1. Proactive Engagement: Sending an SMS to request missing donor emails for tax receipts.
2. Inventory Forecasting: Analyzing incoming stock to flag potential shortages.
3. Automated Compliance: Instantly generating and delivering a beautifully formatted, IRS-compliant PDF tax receipt.

How I built it

I built PantryPilot using a decoupled, event-driven architecture to ensure scalability and maintainability:

• Backend & Orchestration: Built with Python and FastAPI. We implemented a durable, file-based queue system (received_messages/) that acts as a crash-resistant message broker. The core logic is split into specialized agents (Intake, Dispatch, Logistics) managed by a central Orchestrator that strictly enforces Human-in-the-Loop (HITL) boundaries.
• AI & Multimodal Processing: I utilized Qwen's DashScope APIs for the heavy lifting. I used qwen3-asr-flash for high-accuracy voice transcription, cosyvoice-v3-flash to generate realistic demo voice notes, and a qwen3.7-plus compatible endpoint for structured Pydantic data extraction.
• Provider-Agnostic Design: All AI configurations are abstracted behind MAIN_*, ASR_*, and TTS_* environment variables, allowing me to swap between Qwen, OpenAI, or AWS Bedrock without changing a single line of code.
• Frontend Dashboard: Built with Next.js 16, React, Tailwind CSS, and Framer Motion. The DecisionCard component provides a beautiful, responsive UI for HITL approvals, while the ObservabilityDashboard tracks real-time system health.
• Document Generation: We used fpdf2 to dynamically generate IRS-compliant PDF tax receipts upon human approval.

Challenges I ran into

• Voice Model Compatibility (Error 418): During initial testing, the Qwen CosyVoice API returned a 418 InvalidParameter error for several voice names on the international endpoint. I solved this by refactoring the script to standardize on the universally supported longanyang voice and implementing robust fallback mechanisms.
• Safety Guardrail False Positives: A donor's voice note saying, "we'd hate to throw away perfectly good food," was blocked by my input safety guardrails because the AI flagged the words "hate" and "throw away" as toxic. We resolved this by refining the guardrail prompts to account for non-profit donation semantics, turning a frustrating bug into a great demo point about domain-specific AI tuning.
• Observability Metrics Inflation: The observability dashboard initially showed "2 Total Requests" even though 6 messages were processed, because it was only tracking the queue scan endpoint. I fixed this by injecting metrics.record_request() into the per-file processing loop in file_queue.py, providing granular, per-message tracking.

Accomplishments that I am proud of

I am incredibly proud of building a truly production-grade Human-in-the-Loop system. Most AI demos are "black boxes" that hallucinate actions; PantryPilot explicitly shows its reasoning and waits for human approval before executing real-world actions (like sending SMS or generating legal documents).

Furthermore, I am proud of my granular observability. Tracking metrics at both the macro level (queue scans) and micro level (individual message intake latency) provides enterprise-grade monitoring visibility that is rarely seen in hackathon prototypes. Finally, achieving a seamless, end-to-end flow—from a raw, unstructured voice note to a downloaded, compliant PDF tax receipt in seconds—was a massive engineering milestone.

What I learned

Building PantryPilot transformed my understanding of what it takes to move from a simple "AI script" to a robust multi-agent system:

1. Transparent AI Builds Trust: Showing the user exactly what the AI parsed and why it made certain decisions is crucial for user adoption in critical workflows.
2. HITL State Management: Pausing an asynchronous agent workflow for human approval without losing context requires careful state serialization. The orchestrator must maintain a persistent state dictionary that survives UI pauses.
3. The Math of Observability: I learned to track the average processing latency $L_{avg}$ for multimodal messages by breaking it down into its constituent parts: $$L_{avg} = \frac{1}{N} \sum_{i=1}^{N} (t_{ASR}^{(i)} + t_{LLM}^{(i)} + t_{DB}^{(i)})$$ Tracking these individual components proved invaluable for debugging bottlenecks in the pipeline.

What's next for PantryPilot

• Cloud-Native Migration: I plan to migrate the agent tools to AWS Lambda and deploy the orchestration logic on Amazon Bedrock AgentCore. This will provide serverless auto-scaling to handle sudden spikes in holiday donation traffic, along with native conversational memory.
• Donor Portal: Building a lightweight web portal where recurring donors can log in, view their past donation history, and download their annual tax summary.
• Advanced OCR Integration: Expanding the ocr_mcp.py tool to process image-based receipts and handwritten donation slips from local grocery stores, further reducing manual data entry.

Built With

• ai-agents
• dashscope
• fastapi
• fpdf2
• framer-motion
• generative-ai
• human-in-the-loop
• llm
• lucide
• multi-agent-systems
• next.js
• openai-api
• pdf-generation
• pydantic
• python
• qwen
• react
• rest-api
• speech-to-text
• tailwind-css
• text-to-speech
• twilio
• typescript
• uv