Zubaan — Voice-Powered Digital Khata

Zubaan Pipeline filters out background market noise to prevent invalid audio entries from logging into the khata ledger.
Real-time voice ingestion supporting multi-language selection (Urdu, English, Punjabi, Arabic) for regional shopkeepers.
Automated digital khata dashboard parsing cash, udhaar, customer details, and total balances in real time.

Inspiration

Millions of micro-retailers across South Asia rely on traditional paper ledgers ("Khata") to track daily credit (udhaar) and cash sales. In fast-paced, noisy market environments, manual logging leads to untracked debts, revenue leakage, and language friction (Urdu, Roman Urdu, Punjabi). Zubaan was built to solve this—a high-throughput, voice-first AI pipeline that converts informal spoken shopkeeper dictation into structured financial ledgers in real time.

What it does

Zubaan acts as an intelligent voice co-pilot for shopkeepers:

Market Noise Isolation & Voice Capture: Captures raw audio input via a mobile browser and streams it to an automated AI ingestion pipeline.
Multi-Lingual Automatic Speech Recognition: Transcribes regional multi-lingual dictation (Urdu, English, Punjabi, Arabic) with specialized vocabulary tuning for Pakistani retail terms (udhaar, cash, rupay, chini, aata, doodh).
Deterministic Information Extraction: Extracts key transaction entities—Customer Name, Item Names (standardized into universal English equivalents), Amount (PKR), and Transaction Type (cash_sale vs udhaar).
Smart Debt Reconciliation Engine: Automatically detects returning customers settling debts. Instead of creating redundant logs, it updates existing unpaid credit entries in real-time, subtracting partial payments or marking records as (Paid/Cleared) upon full settlement.
IP-Isolated Dynamic Ledger Storage: Allocates distinct, secure client-isolated JSON databases per user IP address to preserve privacy across remote testing environments.
Real-time Analytics: Computes live metrics for Total Cash Received, Outstanding Udhaar Balance, and Most Demanded Products (equipped with tie-breaker thresholds and plural normalization).

How we built it

Frontend: Responsive, mobile-first Web Interface constructed with HTML5, Tailwind CSS, MediaRecorder API, and asynchronous JavaScript fetch pipelines.
Backend Architecture: High-performance FastAPI server running ASGI uvicorn engines with custom CORS middleware and static file serving routes.
Speech-to-Text Pipeline: Integrated Groq Inference Engine API utilizing whisper-large-v3-turbo with customized domain prompts for ultra-low latency transcription.
Natural Language Extraction: Powered by Llama-3.1-8b-instant with strict JSON-mode response formatting and system-level guardrails against hallucinations or background noise.
Tunnelling & Remote Access: Deployed local-to-global network tunnels via Ngrok with custom header handling (ngrok-skip-browser-warning) for cross-device mobile execution.

Challenges we faced

Background Noise & Hallucinations: Silent recordings or background bazaar noise initially caused ASR/LLM models to invent fictitious transactions. We implemented string-length guardrails, threshold filters, and strict JSON validation rules to reject non-transactional audio.
Plural & Synonymous Item Merging: Items like "egg" and "eggs" or localized terms like "aata" vs "flour" fragmented the analytics engine. We engineered custom normalization rules, plural stripping logic, and commodity standardization mapping.
Debt Duplication vs Reconciliation: Preventing the backend from creating duplicate entries when a customer pays off an old debt required developing an automated ledger reconciliation algorithm that parses active unpaid credit entries by customer name before appending new logs.

Accomplishments that we're proud of

Zero-Download Cross-Platform Access: Users can access the full app on mobile browsers directly via FastAPI static serving without needing app store installations.
Sub-2-Second End-to-End Latency: Audio capture to full LLM entity extraction and database state sync executes in under two seconds.
Privacy-First Isolation: Complete client-level data separation achieved dynamically without complex database authentication overhead.

What we learned

Building Zubaan gave us deep insights into low-latency ASR tuning, edge-case guardrailing for small language models, real-time client-server audio streaming, and financial ledger reconciliation logic under unstructured speech inputs.

What's next for Zubaan

Offline Mesh Syncing: Integrating peer-to-peer local mesh networking (Bluetooth / Google Nearby Connections) for offline transaction sync during internet outages.
Text-to-Speech (TTS) Voice Confirmations: Synthesizing native voice responses in the shopkeeper's dialect to confirm transactions out loud.
WhatsApp Ledger Receipts: Automated SMS/WhatsApp receipt dispatch directly to end customers upon debt logging.