Inspiration
Ordering groceries on a phone means typing, scrolling and tapping through a catalogue. In a hurry, one-handed, in a language the catalogue was not written in, it is miserable. People here already speak to their phones for everything else, so we wanted to see how far that could go: say the list, and have the shopping and the payment both done.
What it does
You speak a list in Hindi, Hinglish, English or any of the Indian languages Sarvam supports. PayGrow transcribes it, works out the items, finds real products, shows them as numbered cards, and reads the list back to you. You say "card 2", it creates a PayPal order, opens the approve page, and captures the payment when you confirm.
Greetings and questions get answered as conversation. An order is created only when you actually ask to buy something.
How we built it
The backend is FastAPI. Sarvam Saaras handles speech to text and reports the language it heard, so nothing is hardcoded: speak Tamil and the answer comes back in Tamil. Sarvam's chat model plans the turn and writes the spoken summary, with Groq as a fallback and plain rules as the last resort. Spoonacular supplies the catalogue, Open Food Facts takes over when its daily quota runs out. Bulbul speaks the answer sentence by sentence while the words appear on screen in step with the audio.
Payments use PayPal Orders v2 on the sandbox: create, approve, capture. The order id and payer email come back to the screen, and the app can read the order state at any point.
The app is Flutter for Android: email login with a one time code, a chat screen, a voice window that runs the whole turn by itself, and an order history drawer. The same backend serves a browser version, so the flow can be tried without installing anything.
Challenges we ran into
Spoonacular's search endpoint returns no price at all, and its product endpoint has a price whose unit nobody documents. The sample values look like cents and are often zero. We stopped guessing and made every product carry where its price came from, simulated or from the API, and the app prints it on the card.
PayPal rejects itemised orders with fractional quantities, and "2.5 kg" is normal in a grocery list, so the order carries the total instead of the lines.
Capturing an order that had not been approved returned a wall of JSON. We turned that into a plain sentence, added an order status endpoint, and put the approve link on screen, because on a phone the PayPal tab was sometimes blocked and the shopper never knew it had not opened.
Sarvam's model occasionally answers with an empty body, which used to crash the whole turn with a 500. It now moves to the next model and falls back to rules.
Render's free plan blocks outbound SMTP, so an SMTP key for login emails silently fails. Brevo's HTTPS API works.
Our own laptop ran out of memory building the APK, because Gradle was asking for more heap than the machine had.
Accomplishments that we're proud of
A full loop that actually ran: spoken sentence, product cards, PayPal approval, captured payment, with the capture id and payer email on screen.
Language detection that is not hardcoded, so a Tamil request answers in Tamil and a Hinglish one answers in Hinglish.
An agent that knows the difference between "hi" and "I want milk".
110 backend tests, including the paths that only run when keys are present, so a missing key degrades the demo instead of killing it.
What we learned
Real APIs disagree with their own documentation, and the honest move is to label what is simulated rather than fill the gap quietly.
Voice interfaces live or die on timing. The pause before sending is 450 ms, and audio has to be cut under the provider's limit before it is sent, not after.
Payment flows have more steps than they look like they have. Creating an order is not paying for it, and a demo that only does the first looks fine until someone presses capture.
What's next for PayGrow
Store the history in Postgres per account, so a shopper keeps their chats across devices. The schema and the endpoints are already written.
"Pull real prices from Channel3 across merchants instead of labelling them simulated, and give the order ledger an AG Grid table so past orders can be sorted and filtered.
Save a payment method with PayPal Vault so a repeat order is one tap.
Move to a real Indian catalogue. Swiggy publishes an MCP server for Instamart, and production access needs their approval, which is the honest route to real inventory and real prices.
Add a proper price source so the simulated labels can go away.