Inspiration

During Hajj, a campaign supervisor walks through Mina responsible for hundreds of pilgrims. Questions like "how many from group 3 haven't boarded the bus?" or "who is still missing?" need answers in seconds, but the data sits in systems built for desks, not crowds. Supervisors are on the move, hands full, and many think and speak in Saudi dialect, not in SQL or dashboards.

What it does

MATS Agent is a real-time, bilingual (Arabic / English) voice agent for Hajj campaign operations.

Talk, don't type: push-to-talk or hands-free mode with voice activity detection and barge-in. Start speaking while the agent is answering and it stops and listens.
Understands real speech: Saudi dialect, misspellings and follow-ups ("and group 5?") are resolved against the conversation.
Verified answers: every number comes with the generated SQL and the result table, so supervisors can check it.
Learns from use: successful question→SQL pairs become few-shot examples; supervisors mark answers right or wrong.
Live context: weather, announcements and traffic in the holy sites come from Tavily search and are combined with campaign data.
Safe by design: generated SQL runs read-only behind three guard layers.

All data in the demo is fully fictional (2,000 synthetic pilgrims, 20 groups, camps, buses, trips and incidents).

How we built it

Every language-model call runs on NVIDIA Nemotron models through Nebius Token Factory, with each step routed to the model that fits it:

Step	Model
Intent routing + follow-up rewriting	Nemotron 3.5 Lightning (Nano fallback)
Text-to-SQL + self-repair	Nemotron 3 Super
Complex analysis + escalation when SQL fails twice	Nemotron 3 Ultra
Streamed spoken answer	Nemotron 3.5 Lightning
Speech recognition	Browser Web Speech API

A "How it answered" panel shows, for every turn, which Nemotron model ran each step and how long it took. The backend is FastAPI with SSE streaming; the frontend is a voice-first RTL/LTR web app; the database is SQLite opened read-only with a SQLite authorizer and a time budget.

Challenges we ran into

Keeping voice conversation natural: we stream tokens and speak sentence by sentence instead of waiting for the full answer.
Dialect and spelling variation in Arabic: we normalize text and give the models a domain glossary.
Making text-to-SQL trustworthy: retry with the database error, escalate to Ultra, and always show the query.

Accomplishments that we're proud of

A supervisor can ask a question out loud in Saudi dialect and hear a checked answer back. It runs on open NVIDIA models with routing that keeps the fast path cheap, and Ultra is used only when it's needed.

What we learned

Model routing matters more than model size: the fastest Nemotron handles most turns, and the bigger models earn their cost only on hard steps.

What's next

Nemotron 3.5 ASR on a Nebius Serverless Endpoint, to replace browser speech recognition. NVIDIA fine-tuned it on SDAIA's SADA dataset for Najdi and Hijazi speech.
A pilgrim-facing mode where pilgrims ask in their own language (Urdu, Indonesian, English) about their camp and bus.
Integration with real campaign systems under proper data agreements.