How we built it
OnPoint is a Next.js monorepo with reusable middleware for agent controls,
state persistence, commission splits, and suggestion toasts. AI providers
(Venice, Gemini Live, OpenAI) implement a unified interface for vision
analysis and real-time streaming. The agent loop follows a perceive →
reason → act pattern: camera input routes through the AI provider
abstraction, style scoring applies sentiment-weighted analysis, and
suggestions flow through a time-bounded toast system with auto-approve
thresholds. When catalog searches fail, a Python FastAPI bridge using
Browser Use Cloud autonomously browses whitelisted fashion sites.
Blockchain integration via RainbowKit/Wagmi enables multi-chain payments
(Celo, Base, Ethereum, Polygon) with four-tier commission splits, while
Auth0 Token Vault handles secure OAuth token exchange via RFC 8693 for
third-party API access.
Challenges we ran into
Balancing agent autonomy with user control was the hardest problem — we
needed the agent to act independently without making users feel like
they'd lost agency. We solved this with configurable spending thresholds,
time-bounded suggestion toasts, and approval workflows that interrupt only
when necessary. Real-time video streaming with Gemini Live required
managing WebSocket connections, canvas capture rates, and adaptive polling
to keep latency under 3 seconds. Building the 3-tier web discovery engine
(catalog → API aggregation → autonomous browsing) meant handling rate
limits, structured data extraction from unstructured pages, and surfacing
live browsing URLs in the UI so users could watch the agent work.
Integrating Auth0 Token Vault required mapping on-chain wallet identities
to Auth0 user IDs and implementing just-in-time consent flows for new
retailers.
Accomplishments that we're proud of
We built a production-ready AI agent that genuinely helps people shop —
not a demo or prototype, but a live app with real photo analysis,
real-time streaming sessions, and actual on-chain transactions. The
transparent decision-making system means every suggestion comes with a
visible reasoning trail, and cryptographic audit trails stored on IPFS
make agent actions verifiable. We're proud of the Auth0 Token Vault
integration that lets the agent access external services without ever
touching credentials, the 3-tier web discovery engine that autonomously
finds products when our catalog falls short, and the spending control
system that gives users confidence the agent won't go rogue. We've also
open-sourced the entire codebase under MIT so other builders can reuse our
middleware modules.
What we learned
Building autonomous agents requires a fundamentally different approach to
UX — you're not designing interfaces, you're designing trust relationships
between humans and AI. We learned that transparency isn't optional; users
need to see why an agent makes decisions, not just what it decides. The
Auth0 Token Vault taught us that secure agent-mediated API calls are
possible without exposing credentials, but it requires careful identity
mapping and consent flow design. We discovered that multi-provider AI
architectures with automatic fallback are essential for production
reliability, and that Redis-backed state persistence with in-memory
fallbacks keeps the system resilient. Most importantly, we learned that
the best agent UX feels like a conversation, not a command —
interruptible, explainable, and always reversible.
What's next for OnPoint
We're expanding multi-chain support to Base and Polygon with cross-chain
transaction aggregation, building analytics dashboards to track user
journeys and conversion funnels, and completing the Auth0 Token Vault
integration with granular retailer permissions and step-up authentication
for high-value purchases. Short-term, we're adding premium persona
unlocks, saved looks galleries, email notifications for agent purchases,
and a mobile app prototype. Medium-term, we're building a creator
marketplace with stylist profiles and custom revenue splits, custom
persona creation that learns your style, multi-agent collaboration between
stylists and shoppers, and regional payment integrations for African
markets. Long-term, we're working toward agent-to-agent commerce where AI
agents buy from each other in an open agentic economy.
Log in or sign up for Devpost to join the conversation.