Gruhasthi

Inspiration

We use multiple Apps for ordering deliveries - scheduled deliveries as well as on demand deliveries - and for making payments. Each App requires an OTP or a PIN or a PASSWORD and the latter two have to be rotated regularly with the added constraint that we cannot reuse the previous three Passwords or PINs.

The App UI changes regularly: for example, every time we need to adjust the scheduled delivery, we find ourselves adapting to a new flow. Every interaction amounts to a micro-stress, even for the IT guy in the team.

It doesn't help that we need to work with a tiny keyboard. The occasional voice interface is poor.

On top of that, our transactions are not really private.

This led us to explore building one App that can help us manage and simplify, if not eliminate, our daily grind and help minimize the data that gets leaked by our daily transactions.

What it does

The initial version of the App aims to make the daily grind less stressful by centralizing all the delivery and payments through a single App, all enabled by Voice mode (with keyboard support provided as a fallback). Specifically, this initial version support sending orders by WhatsApp, which is supported by all the neighbourhood grocery stores.

For example, our neighbourhood convenience store, that serves over 3000+ apartments, requires us to send delivery requests by WhatsApp: using this initial version of the App, Users can send delivery requests to our neighbourhood convenience store using WhatsApp.

Users can:

Create a list of preferred stores to order from. This can be done using Voice commands or via Web Search.
- Users can create separate grocery lists for each Store.
- Users can submit orders via WhatsApp
Create a list of Contacts to be paid to.
- Payments by WhatsApp will be added in the next version.

How we built it

We wanted one App that would work for Android as well as iOS devices. From a prior work, we know that Flutter and React Native were the two popular options. Flutter won primarily on account on a richer user interface and the fact that it is backed by Google.

The App was built completely using Codex, down to the README file on GitHub. Over 95% of the App was built using GPT 5.6 Terra: GPT 5.6-Luna was used as a fallback when GPT 5.6 Terra was not available for a brief time, to keep the momentum going.

Voice Support

This was two step:

Speech To Text conversion was done primarily by Android Speech to Text.
The App has a set of stock commands that were parsed by the App and actioned to keep the App responsive
For more flexible conversations, the App supports an optionally deployable on device Gemma 4 E2B/E2B model is used a fallback. To support this, LiteRT Run Time is embedded within the App.

All data is stored on the device. The only thing that leaves the device is the grocery list for now.

Challenges we ran into

There were several:

There were quite a few in handling the speech to text
- It had difficulty handling pauses, leading to capture of only trailing part of the speech
- The same requests were handled differently on Home Screen versus other screens that supported voice input
- Often the speech itself was not captured correctly: for example, add would be captured as had. This problem still exits and requires users to try again.
The transcribed speech would not scroll when it got bigger than the window, and so the user couldn't see how the speech was being transcribed.
There was difficulty in laying out the home screen:
- The initial design had the home screen icons arranged in an arch over the microphone. It couldn't get it right.
- It took multiple iterations to arrange the Microphone glyph at the right place.
Design Issues
- Most of the buttons and help text had poor contrast, by default, making it difficult to read.
- Navigation had a back arrow (<-) on some screens and at other places there was it was "< Back"
The initial approach suggested for downloading the model required the user to access the HuggingFace site and download the model, which would have been difficult for most users.
The infrastructural plumbing require to test the App: flutter and android sdk, hot loading, connectivity to a phone, Google Places API. Nothing major, but it took up some of our time.

Accomplishments that we're proud of

Getting the voice mode "mostly" right.
Integration with Google Places API
Ability to have an on-device model working for processing flexibly worded requests.

What we learned

Integration with a payment system is a non-trivial effort and involves more than just coding.
Coding is cheap: it takes effort to tell GPT-5.6 what to code for.