Inspiration

Many Kenyans abroad pay for a house being built back home, often through a relative or a foreman they can't watch. They send money for cement, the foreman sends a photo, and they hope the photo is real. Some aren't: the same "foundation done" photo gets sent twice, or it was taken at someone else's site. We wanted the money to move only when the work is real, without asking the owner to trust anyone.

What it does

Hakika (Swahili for "certainty") sits between the site and the owner's money.

  1. The foreman sends a quote on WhatsApp, as a photo, a voice note in Swahili or English, or text. Hakika reads it and checks every line against reference prices and verified suppliers.
  2. The owner gets an approval card: what was asked for, what it should cost, the problems found (for example "18% above the verified median, supplier not verified") and a cheaper verified supplier.
  3. The owner approves in PayPal, and PayPal pays the supplier directly. The order is created with the verified supplier as the payee, so Hakika never holds anyone's money.
  4. Every delivery and milestone needs proof: a one-time 4-character code, written on paper or on the delivery and photographed at the site within 30 minutes.
  5. Code checks the proof: the code, the time, the location, and whether the photo was used before (a perceptual hash). The vision model adds soft checks: is it a photo of a screen, does it show what was paid for, is it the same site.
  6. If anything fails, the next payment is held and the owner sees exactly why. A Bryntum Gantt shows each milestone and the labour payment it unlocks. When fresh proof passes, the fundi (mason) is paid through PayPal.
  7. Ask Hakika answers the owner's questions ("Why is the foundation payment held?") from the project's records, with links to the records it used.

Judge Mode runs the whole story without a phone: an inflated quote, a valid delivery, and a recycled photo sent from 37 km away that gets caught and holds the payment.

How we built it

  • PayPal Orders v2 with the supplier or fundi as payee; the owner approves with the PayPal JavaScript SDK button, and the server captures the order with an idempotency key. Prices are kept in Kenyan shillings, and orders are in USD at a daily rate saved with each payment.
  • PayPal Webhooks, signature-verified, keep the money trail in sync (completed, pending, declined, refunded and reversed captures), with duplicates ignored.
  • PayPal Agent Toolkit gives Ask Hakika one read-only tool, get_order, for orders that belong to the project.
  • Amazon Nova 2 Lite on Amazon Bedrock reads quote photos and messages, inspects proof photos, words the owner's summary, and runs Ask Hakika with tool calls.
  • Groq Whisper large-v3 translates Swahili voice notes into English.
  • Twilio's WhatsApp sandbox carries messages to and from the site.
  • Bryntum Gantt shows the milestones with their payment gates.
  • Next.js, TypeScript, Drizzle, Neon Postgres and private Vercel Blob, hosted on Vercel; sharp for image hashing; about 500 Vitest tests.

The rules that keep money safe

  • Code decides whether money can move. The models only read, compare and explain; no model output can trigger a payment.
  • Only the owner's PayPal approval moves money. The server computes every amount from the database; the browser never sends a total.
  • WhatsApp text and text in photos are data, never instructions. People on site only ever get fixed English and Swahili templates.
  • Ask Hakika is read-only. Its tools are bound to one project. Only links to records a tool returned are shown, every number in an answer must appear in those records, and questions are capped.

Challenges we ran into

  • Paying a third party directly. We tested early that a sandbox order with a supplier as payee could be approved by the buyer and land in the supplier's account. The whole idea depended on it.
  • The Kenyan shilling isn't a normal PayPal presentment currency, so we keep KES for prices and convert to USD at a daily rate stored with each payment.
  • Models obey text in photos. In testing, a quote photo with "ignore previous instructions" written on it changed what the model read. Now a photo whose text reads like instructions is flagged as suspicious and nothing read from it is used.
  • Models round, add and invent numbers. Every amount the owner reads is checked against the database; an answer with a number that isn't in the records is replaced by a fixed sentence.
  • WhatsApp strips photo metadata, so location comes from the capture page or a shared WhatsApp pin, never from photo metadata.

What we learned

The model is most useful when it's not in charge. It reads handwriting, Swahili and messy quotes far better than any rule could, but the decisions that move money belong in plain code that can be tested, and every model output needs a check that doesn't trust it.

What's next

Owner sign-in and more than one project, supplier onboarding with business checks, video proof, and asking two verified suppliers for prices before the owner approves.

Built With

  • ai-sdk
  • amazon-bedrock
  • amazon-nova
  • bryntum
  • drizzle
  • groq
  • neon
  • next.js
  • paypal
  • paypal-agent-toolkit
  • paypal-javascript-sdk
  • paypal-orders-api
  • paypal-webhooks
  • postgresql
  • react
  • sharp
  • tailwindcss
  • twilio
  • typescript
  • vercel
  • vercel-blob
  • vitest
  • whatsapp
  • whisper
Share this project:

Updates

Submission history