Shuks AI

Connect with local artisans, finalize jobs on a call, and let our AI Mediator turn your spoken agreement into a milestone-protected PayPal Escrow contract with proof-of-work.

Comment

Inspiration

In Hebrew, "Shuk" (שׁוּק) means marketplace—the bustling neighborhood bazaar where local craftspeople, artisans, and buyers meet face-to-face, negotiate a fair price out loud, and shake hands on a deal.

While digital freelancers on platforms like Fiverr or Upwork enjoy written contracts, milestone tracking, and escrow protection, the real-world shuk—hiring a local plumber, electrician, carpenter, jeweler, or mechanic—still runs on quick phone calls, Google Meet links, or WhatsApp voice calls. Nobody writes a contract for an emergency pipe repair.

This creates a massive "Verbal Trust Gap":

  1. "He Said / She Said" Disputes: Verbal promises made on a call are forgotten or disputed once the work begins.
  2. The Payment Fear Trap: Customers are afraid to pay cash upfront because an artisan might disappear or do incomplete work; artisans are afraid to buy materials out of pocket and travel across town only for the customer to refuse to pay.
  3. Forgotten Pricing Conversations: People often discuss the technical problem on a call, agree on an arrival time, and hang up without ever locking in a price or milestone structure.

We built Shuks AI to bring the trust of PayPal Escrow to any spoken agreement anywhere in the world—without forcing people to change how they talk.

What it does

Shuks AI is an autonomous AI Meeting Mediator and PayPal Escrow orchestrator that turns spoken conversations into milestone-protected contracts:

  • Silent 3rd-Party Call Listener: Shuks AI joins live calls (Google Meet, Zoom, LiveKit, or uploaded call recordings) as a silent note-taker. It never interrupts the conversation—it simply listens and diarizes what the Buyer and Artisan say.
  • Post-Call Actionable Checklists & Smart Branching: As soon as the call ends, Shuks AI messages the Buyer with a structured summary, a Client Checklist, and an Artisan Checklist, then branches intelligently based on what happened on the call:
    • Branch A (Fee Mentioned on Call): If a price was discussed, Shuks AI asks the Buyer for permission to turn those exact figures into a PayPal Escrow Milestone Plan and send the formal offer to the Artisan.
    • Branch B (No Fee Mentioned on Call): If the parties forgot to discuss money, Shuks AI catches the gap and proactively asks the Buyer if they want help structuring a financial budget and milestone plan before work starts.
  • PayPal Escrow Locking: Once approved, the Buyer funds the plan via PayPal Checkout, locking the money in escrow before the artisan travels to the job site.
  • Multimodal Proof-of-Work Release Gate: The Buyer cannot prematurely release escrow funds while work is pending. To unlock a payout, the Artisan uploads a Proof-of-Work photo on site. Shuks AI Vision cross-checks the photo against what was promised on the original call, posts the photo and verification report into the shared chat for both parties to see, and unlocks the Buyer's Release button.
  • Artisan PayPal Connect, Change Orders & AI Dispute Arbitration: Artisans log in with their PayPal account to withdraw released funds, can submit mid-job Change Orders when unexpected repairs arise, and if a disagreement occurs, the Shuks AI Dispute Arbiter cross-examines the call recording, checklists, and photo evidence to compute a fair integer-cent PayPal split settlement.

(Note: The Shuks web application we built serves as an interactive simulation studio with a **Side-by-Side Buyer + Artisan Split View* so judges can watch both sides of the transaction simultaneously. In real-world deployment, Shuks AI is platform-agnostic and communicates with both parties directly via WhatsApp, Email, and shareable contract links.)*

How we built it

  • Multi-Agent Orchestration (AutoGen + LangGraph): We used Microsoft AutoGen (RoundRobinGroupChat & AssistantAgent) to coordinate specialized agents (ScribeAgent, ContractMediatorAgent, ChangeOrderMediator, and DisputeArbiterAgent), wrapped inside an asynchronous LangGraph (StateGraph + MemorySaver) state machine that enforces human-in-the-loop permission gates (DISCOVERY -> IN_CALL -> ANALYZING_CALL -> AWAITING_DRAFT_PERMISSION -> AWAITING_BUYER_FUNDING -> ESCROW_FUNDED -> IN_PROGRESS -> COMPLETED).
  • Multimodal Audio & Vision Intelligence (Gemini 3.8 Flash): Raw call audio (.wav / .webm / .mp3) is normalized via ffmpeg and passed to Gemini 3.8 Flash for speaker diarization and transcription, while completion photos are inspected against the call transcript using Gemini Vision.
  • Autonomous Meeting Bot & WebRTC (LiveKit + Chrome CDP + PipeWire): We built an in-app LiveKit WebRTC call stage alongside a server-side autonomous meeting bot (meeting_bot.py) that uses headless Chrome DevTools Protocol (CDP) and an isolated PipeWire/PulseAudio virtual audio sink (pactl) to dial into external Google Meet, Zoom, and Jitsi links with its microphone muted and camera off.
  • Payments & Escrow (PayPal JS SDK, Orders API v2, Payouts API v1): All monetary calculations use strict integer minor units (USD cents) to prevent floating-point rounding errors, backed by a SQL escrow ledger (PostgreSQL / SQLite).
  • Frontend Simulation Studio (Next.js 15, TypeScript, Tailwind CSS, React-Leaflet / OpenStreetMap): Provides the interactive split-screen workspace, non-blocking Server-Sent Events (SSE) call completion stream, and shareable artisan contract pages (/contract/[artisanId]).

Challenges we ran into

  • Non-Blocking Audio Processing at Call Hangup: Transcoding and diarizing multi-megabyte .wav call recordings plus running a multi-turn AutoGen group chat takes several seconds. Initially, ending a call blocked the UI until analysis finished. We redesigned /analyze-call to use FastAPI BackgroundTasks (HTTP 202 Accepted in under 50ms) paired with a Server-Sent Events (SSE) stream (/jobs/{artisan_id}/events) so the call closes instantaneously and the summary pops into chat when ready.
  • External Meeting Bot Anti-Bot Guards & Virtual Audio Routing: Joining external Google Meet rooms from a backend process required stripping HeadlessChrome client hints via CDP before navigation, routing browser audio into an isolated module-null-sink PipeWire monitor so host audio isn't leaked, and intercepting getUserMedia so the bot joins with its camera off and microphone silent.
  • PayPal Sandbox Regional Account Constraints: Discovering that PayPal Sandbox accounts defaulted to certain regions (NG) operate as send-only accounts (PAYEE_ACCOUNT_LOCKED_OR_CLOSED) required configuring US Sandbox merchant/buyer flows and building an explicit Artisan PayPal login & payout connection workflow.

Accomplishments that we're proud of

  • Zero Hardcoded Transcripts—True Audio-to-Contract Intelligence: You can feed raw 2-speaker .wav audio files (conversation.wav where a fee is agreed upon, and conversation_2.wav where pricing is never mentioned) into Shuks AI and watch it carve out the speakers, extract the exact items discussed, and adapt its post-call questions automatically.
  • End-to-End Multimodal Loop (Voice $\rightarrow$ Contract $\rightarrow$ Vision $\rightarrow$ PayPal Payout): Connecting what was promised in an audio call to what is shown in a Proof-of-Work photo before unlocking PayPal Escrow funds.
  • Side-by-Side Buyer & Artisan Split View: Allowing judges to see the Buyer fund PayPal Escrow on the left and watch the Artisan upload Proof-of-Work, log into PayPal, and withdraw released earnings on the right in real time.

What we learned

  • AI Agents Need Strict Human-in-the-Loop Gates When Money Is Involved: An AI mediator should never move money or lock a contract on its own during a call. Having the agent listen silently and ask the Buyer for explicit permission in chat after the call makes the system trustworthy.
  • Escrow Must Be Portable, Not Locked to a Single Website: Real-world service deals happen across phone calls, WhatsApp, and Google Meet. Building shareable contract links (/contract/[artisanId]) solved the marketplace cold-start problem by letting a buyer bring any artisan into a PayPal-protected milestone agreement.

What's next for Shuks AI

  • Native WhatsApp Business & SMS Bot Integration: Delivering post-call summaries, Proof-of-Work photo uploads, and PayPal Escrow approval/withdrawal links directly inside WhatsApp threads and SMS so neither party ever needs to open a web browser.
  • PSTN / Regular Phone Call Dial-In (Twilio + LiveKit SIP): Allowing users on a normal cellular phone call with a local plumber or mechanic to 3-way merge +1-800-SHUKS-AI into the call.
  • Automated Hardware & Parts Supplier Split Payouts: Allowing Milestone 1 ("Materials & Parts") to pay a local hardware supplier directly via PayPal while locking Milestone 2 ("Labor") for the artisan.

Built With

Share this project:

Updates

Submission history