NPU-Piggy

Inspiration

We've all had that moment: checking the bank account at the end of the month and it becomes a crime scene investigation. Who authorized this? When did this happen? For young adults want to save the first bucket of gold but occasionally shopping online at 2am, budgeting isn't a knowledge problem. It's a visibility problem! There are many existing budget apps that attempt to solve this problem by asking you to sync your bank and hand over your most sensitive data to the cloud server. But your transaction history is more than numbers! It reveals where you eat, how you live, and what you’re going through. That data is personal. It should not leave your device! That is why we built NPU-Piggy, a fully on-device AI budget planner that turns a copy of a receipt into structured financial insight, with zero cloud dependency.

What it does

NPU-Piggy is an end-to-end expense tracking tool that runs entirely on your local machine. You scan any receipt, physical or digital, and NPU-Piggy will:

1. Extracts the text using on-device OCR
2. Parses it into structured JSON data using a local LLM
3. Stores your expenses in React local storage
4. Visualizes your spending patterns on a personal dashboard
5. Chats with you through an interactive AI assistant for personalized budget planning feedback and suggestions
6. No accounts. No clouds. No sharing your privacy with a third party.

How we built it

Our pipeline runs in the following stages:

1. Capture: Users photograph a receipt via PC camera.
2. OCR Extraction: EasyOCR, optimized and deployed on-device through Qualcomm AI Hub (NPU powered detector, CPU powered recognizer), extracts raw text from the receipt image.
3. LLM Parsing: The raw OCR text is passed to a local Mistral 7B model via Ollama, orchestrated with LangChain. A structured prompt with explicit rules converts messy text into validated JSON. Output is validated with a Pydantic schema. Retry logic and fallback mechanisms handle malformed or incomplete responses gracefully.
4. Storage & Insights: Parsed expense data is stored locally using React local storage, keeping all financial data on-device with no database server required. A React frontend renders the spending dashboard with interactive visualizations, and hosts the local LLM chat interface for personalized budget planning suggestions. All within a clean, responsive GUI.

Challenges we ran into

The OCR deployment through the Qualcomm AI hub and the LLM integration part are by far the most challenging step. EasyOCR originally runs on CPU, which we find to be very time consuming. Learning from Qualcomm mentors that their AI Hub provides an excellent solution to build local NPU-optimized runtime, we swiftly decided to adopt it after troubleshooting various technical challenges. After enabling NPU, the processing of a single image is speeded up by about 60%. Later when we needed to implement LLM text to json feature, we naturally followed the same logic and thought using the more advanced NPU compute would be always better, so we looked into NEXA AI SDK, which natively supports NPU. However, soon we realized that although the NPU feature accelerates Model inferencing generally, the downside of NEXA SDK is that there is a narrower range of LLM models to choose from, and it's hard to find a model with a balance of weight and the output efficiency. Therefore, in the end, we chose the more traditional Ollama solution that runs on CPU, but supports a very appropriate model to our use case.

Accomplishments that we're proud of

We feel very proud that we can locally deploy the NPU optimized EasyOCR to our specialized Qualcomm powered PC, and actually witness how important it is to “build the appropriate wheel” so that the exciting part of AI and ML can work more efficiently. Especially because all of us have an interest and passion for AI infra career in the future, we see this as a really motivating experience.

What we learned

The experience of choosing Ollama over NPU-compatible Nexa SDK taught us a meaningful lesson: when it comes to data science tooling, it is not always “the fancier, the better. Instead, we should analyze the use case and trade-off rationally and critically, instead of blindly chasing what is commonly perceived as “more advanced”.

What's next for NPU-Piggy

Looking ahead, our primary focus is on pushing NPU-Piggy toward a seamless mobile experience. We aim to optimize the pipeline to increase throughput and reduce latency so that expense tracking feels instant rather than effortful. On the hardware side, we plan to accelerate mobile deployment by fully leveraging Qualcomm's NPU capabilities for on-device inference. To make tracking truly effortless, we want to expand beyond manual receipt scanning by automatically capturing transactions from on-device sources like Apple Pay, keeping privacy intact. And on the model side, we aim to fine-tune a multimodal LLM that processes receipt images directly, collapsing our current OCR and LLM parsing steps into a single end-to-end model that goes from raw image to structured JSON in one shot.

Built With

• cpu
• easyocr
• flask
• langchain
• mistral
• npu
• onnx
• python
• react
• react-native