AgentCart AI - Autonomous Multi-Modal Commerce

Snap or type any real-world product to instantly authorize and capture orders via Google Gemini, biometrics, and PayPal.

Comment

AgentCart AI is an autonomous, multi-modal mobile commerce agent that bridges visual perception, machine reasoning, and payment infrastructure. It transforms how users shop in the physical and digital world by collapsing the multi-step checkout funnel—product search, merchant discovery, authentication, card entry, and address confirmation—into a single biometrically authenticated, one-click PayPal transaction.

Inspiration

What if commerce was as seamless as looking at an item in the physical world, pointing your camera, and confirming with your fingerprint? We were inspired to build an autonomous agent that perceives physical objects or natural-language product queries, maps them to live merchant channels, confirms the transaction via hardware-backed biometrics, and executes the entire purchase lifecycle via PayPal's REST v2 APIs in few seconds.

What it does

AgentCart AI delivers an end-to-end ambient shopping workflow:

Multi-Modal Product Recognition:

  • Utilizes Android CameraX to snap live physical items. The image is evaluated by Google Gemini Flash Vision, extracting the brand, model and later passed to Channel3 API to identify commercial category, and real-time USD market pricing.
  • Direct Specification Mode: Allows users to type real product specifications and exact prices.

Review & Confidence Scoring:

  • Presents an aggregated product dossier including model confidence score , category tags, and merchant routing notes.

Biometric 1-Click Payment Confirmation:

  • Intercepts payment dispatch with Android's BiometricPrompt (fingerprint/face recognition), guaranteeing user intentionality before funds move.

2-Step PayPal Order Creation & Capture:

  • Calls the backend POST /api/execute-paypal to negotiate an OAuth2 bearer token and create an official v2/checkout/orders transaction with CAPTURE intent.
  • Directs the buyer to an approval window and uses deep linking (agentcart://paypal/return?token={orderId}) to return execution control back to the Android app.
  • Automatically executes POST /api/capture-paypal to capture the approved funds and receive a confirmed transaction hash.

Post-Purchase Ledger: Records the completed transaction in a real-time, persistent ledger.

How we built it

AgentCart AI was built across a decoupled, cloud-and-mobile multi-agent architecture:

Client-Side Mobile App (Android & Jetpack Compose)

  • Jetpack Compose & Material 3: Designed an edge-to-edge UI featuring custom surface elevations, UI Components
  • CameraX library to capture image.
  • AndroidX Biometric for biometric prompt flows on physical devices and emulators.
  • Navigation & Deeplink: Configured Android intent filters to capture deep-link approval tokens (agentcart://paypal) via Activity.onNewIntent() to trigger order completion.
  • Networking: Integrated Retrofit 2 and OkHttp 3 with custom JSON serialization and robust HTTP error parsers.

2. Microservice Backend (Python & Flask)

  • Google Gemini Multimodal API: Integrated gemini-flash with engineered prompt templates:
  • Channel3 REST API to fetch the product details from market.
  • PayPal REST v2 Integration: Engineered custom OAuth2 token caching, order creation (/v2/checkout/orders), and capture lifecycle endpoints.

Challenges we ran into

Q: When redirecting the user to PayPal Sandbox in the browser, the Android app process could pause or drop memory state before returning via deep link. Solution: Engineered a resilient AICheckoutViewModel state machine backed by StateFlow. We stored active orderId tokens in memory and configured singleTop launch modes in AndroidManifest.xml to resume gracefully in onNewIntent.

Accomplishments that we're proud of

  • Successfully demonstrating an end-to-end journey from a physical camera snapshot to an authenticated PayPal capture in few seconds.
  • Real Production-Ready Payment Rails: Every checkout goes through live PayPal OAuth2 token exchanges, creates real sandbox order IDs, and returns authentic capture hashes.
  • Biometric Trust Layer: Seamlessly pairing Android's biometric security with PayPal payment approval to ensure payments remain safe and authorized.

What we learned

  • Intent-Driven Mobile Architecture: Decoupling the intent capture (camera/text) from the settlement layer (PayPal) allows an AI agent to handle any catalog or product type dynamically.
  • Deep Link Hygiene in Jetpack Compose: Properly managing task affinities, backstacks, and single-top launch configurations is essential when orchestrating browser-to-app payment handoffs.

What's next for AgentCart AI - Autonomous Multi-Modal Commerce

  • Autonomous Price Comparison & Merchant Arbitration: Expanding the agent to query multiple merchant nodes simultaneously to find the lowest price, fastest shipping, and best coupon codes before checkout.
  • Voice-Activated Ambient Purchasing: Enabling continuous voice intent (e.g., "AgentCart, reorder my favorite coffee beans"), handling reorders hands-free via PayPal pre-approved billing agreements.
  • Cross-Border Multi-Currency Settlement: Supporting live FX currency conversion directly in the UI for international shoppers.
  • AR Viewfinder Overlays: Overlaying interactive 3D bounding boxes and product specs directly in the augmented reality camera feed prior to snapshot capture

Built With

Share this project:

Updates

Submission history