Inspiration
Most expense-tracking apps ask people to complete forms, select categories, enter dates, and organize every transaction manually. Penny takes a more natural approach: Azerbaijani speakers can simply type a message, send a voice note, or photograph a receipt, then receive structured expense records and clear reports.
Penny keeps AI in the background. The user should experience a simple conversation—not a complicated financial tool.
However, that simplicity creates an important responsibility. If AI misunderstands a merchant, reporting period, correction, or deletion request, the resulting financial record or report may be wrong. A report is only as reliable as the interpretation behind it.
During Build Week, we wanted to make Penny’s AI understanding testable before new behavior reaches real users.
What it does
Penny is an Azerbaijani-language AI finance assistant currently being tested in close beta. Through a familiar Telegram interface, users can:
- record expenses using natural text;
- submit voice messages;
- extract expense information from receipt images;
- correct previously recorded expenses;
- search for specific transactions;
- receive clear spending reports.
These capabilities existed before Build Week.
During Build Week, we meaningfully extended Penny with the Penny Trust Preview & Eval Lab.
The standalone lab evaluates how GPT‑5.6 Sol interprets synthetic Azerbaijani financial messages. For every message, it displays and validates:
- user intent;
- amount and currency;
- financial category;
- merchant or note filter;
- date or reporting period;
- correction target;
- ambiguity and clarification requirements;
- whether the message represents a possible financial write action.
The lab cannot create, update, or delete an expense. Every write-like result remains a non-executable candidate.
For example:
- “Jet xərclərim” (“Show my Jet expenses”) remains a read-only report filtered specifically by Jet.
- “5 manat taksi” (“5 AZN for a taxi”) becomes a proposed transport expense, but nothing is saved.
- “Taksi və Jet xərclərim” (“Show my taxi and Jet expenses”) contains multiple filters, so the lab requests clarification.
- “Hamısını sil” (“Delete everything”) is recognized as a dangerous bulk-delete request that requires confirmation and cannot be executed inside the lab.
How we built it
We created a standalone, default-offline command-line evaluation system completely isolated from Penny’s production database, Telegram flow, and production behavior.
GPT‑5.6 Sol processes synthetic Azerbaijani inputs through the Responses API using Strict Structured Outputs. Deterministic software then:
- verifies the requested and returned model identity;
- validates every field against a strict schema;
- compares the complete semantic result with the expected result;
- enforces ambiguity and financial safety rules;
- prevents silent fallback to another model;
- records privacy-safe and reproducible evidence.
The evidence includes hashes of the prompt, fixture set, schema, and rendered request input. It excludes API keys, personal information, raw model responses, production data, and local system paths.
Codex was used throughout the Build Week engineering process to:
- audit Penny’s existing AI architecture;
- identify production isolation risks;
- compare architecture options;
- design and implement the standalone lab;
- define the strict semantic contract;
- build synthetic Azerbaijani fixtures;
- create deterministic validation and privacy controls;
- add focused and full regression tests;
- review the implementation and verify repository safety.
GPT‑5.6 Sol provides Azerbaijani semantic interpretation. Codex helped engineer the system that verifies, constrains, tests, and documents that interpretation.
Challenges we ran into
The biggest challenge was separating useful AI reasoning from financial execution.
Valid JSON is not automatically a correct financial interpretation. The lab therefore compares every meaningful field—including explicit null values, merchant filters, reporting periods, corrections, and ambiguity—against an expected result.
Penny’s production bot also has database, Telegram, environment, and model-client dependencies that should not be imported into an isolated safety lab. We therefore built the laboratory as a separate system rather than modifying the production bot.
Another challenge was producing useful evidence without storing sensitive information. The final design uses synthetic inputs, strict model verification, redaction, bounded live execution, and hashed evidence.
Accomplishments that we're proud of
- Built a production-isolated Trust Preview & Eval Lab
- Created 17 synthetic Azerbaijani financial fixtures
- Passed 36/36 focused tests
- Passed 17/17 offline evaluation cases
- Passed Penny’s complete 53/53 standalone test suite
- Completed GitHub Actions Python Tests run #30 successfully
- Passed one controlled GPT‑5.6 Sol live sample
- Verified that requested and returned model IDs matched exactly
- Prevented silent fallback to another model
- Kept every financial write candidate non-executable
- Accessed no production database, Telegram flow, or real user data
- Stored no secrets, personal information, raw responses, or production artefacts
The controlled live result is one synthetic, nondeterministic sample. It demonstrates that the integration works, but it is not presented as a complete production correctness guarantee.
What we learned
Trustworthy financial AI requires both model intelligence and deterministic engineering.
The model can propose a semantic interpretation, but software must verify the model, enforce the schema, identify ambiguity, protect write boundaries, and preserve evidence.
We also learned that Azerbaijani needs dedicated evaluation. Letter casing, decimal formats, merchant names, reporting phrases, and short correction commands cannot be covered adequately by generic English benchmarks.
Finally, we learned that model evaluation should remain separate from production execution. A successful test is useful evidence, but it should not automatically authorize a production change.
What's next for Penny
The laboratory is not deployed to production and does not replace Penny’s current parser.
Our next step is to expand the Azerbaijani evaluation set using approved and anonymized product observations. We want to turn recurring misunderstandings and user corrections into regression cases that can be tested before future model or prompt changes.
We also plan to explore a user-facing trust preview that can clearly show how Penny interpreted a message before a sensitive financial action is confirmed.
Any production integration will require a separate safety review, explicit approval, and real-user validation.
Log in or sign up for Devpost to join the conversation.