Post-Mortem: ICP Refinement Failure

Objective

Refine the Ideal Customer Profile (ICP) to improve lead targeting and qualification.

What Happened

We were unable to successfully refine the ICP during the project timeline. The system consistently failed to process and generate refined customer profiles as expected.

Diagnosis

Using Comet's traces, we were able to track the execution flow and identify where the system was breaking down. The traces revealed that the model was failing to properly parse and validate the complex data structures we were using for ICP definitions.

Root Cause

GPT-4o-mini was unable to handle our complex Zod schemas.

The ICP refinement logic relied on structured outputs with nested Zod schemas to ensure type safety and data validation. While this approach works well with more capable models, GPT-4o-mini consistently struggled with:

Nested object validation
Complex union types
Conditional schema logic

This resulted in malformed outputs that failed validation, causing the refinement pipeline to fail.

Key Learnings

Model selection matters for structured outputs - Not all models handle complex schema validation equally
Observability is critical - Comet traces were instrumental in identifying the issue quickly
Simplify or upgrade - Either simplify schemas for smaller models or use more capable models for complex validation

Next Steps

Upgrade to GPT-4o or Claude Sonnet for ICP refinement tasks
OR simplify Zod schemas to be more compatible with smaller models
Add model capability checks before schema validation

Lead Finder Architecture

AI-Powered B2B Lead Generation Platform

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────────────┐
│                          LEAD FINDER ARCHITECTURE                            │
│                   AI-Powered B2B Lead Generation Platform                    │
└─────────────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────────────┐
│  PRESENTATION LAYER (Next.js 15 + React 19)                                 │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  Landing Page (Unauthenticated)          Main App (Authenticated)           │
│  ┌──────────────────────┐                ┌────────────────────────────┐    │
│  │ • Hero               │                │ Navbar + Theme Toggle      │    │
│  │ • Features           │   Auth0 ──────▶│                            │    │
│  │ • How It Works       │   Login        │ Routes:                    │    │
│  │ • Tech Stack         │                │  /                 (Home)  │    │
│  │ • CTA                │                │  /me              (Profile)│    │
│  └──────────────────────┘                │  /icps/*          (ICPs)   │    │
│                                          │  /icps/[id]/[idx]/prospects│    │
│  UI Components (shadcn/ui + Radix)       └────────────────────────────┘    │
│  ┌────────────────────────────────────────────────────────────────────┐    │
│  │ Button, Card, Input, Table, Accordion, Sheet, Badge, Skeleton...   │    │
│  │ SearchInput, ICPCard, ChatPanel, CompaniesTable, EmailComposer...  │    │
│  └────────────────────────────────────────────────────────────────────┘    │
│                                                                              │
│  Styling: Tailwind CSS 4 + Framer Motion                                    │
└──────────────────────────┬───────────────────────────────────────────────────┘
                           │
                           │ Real-time Sync
                           ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│  API LAYER (Next.js API Routes + Convex Functions)                          │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  Next.js API Routes (/app/api/*)              Convex Backend (/convex/*)    │
│  ┌──────────────────────────────────┐         ┌─────────────────────────┐  │
│  │ POST /api/generate-icps          │◀────────│ icpGeneration.ts        │  │
│  │  └─ Orchestrates ICP generation  │         │  └─ Job management      │  │
│  │                                   │         │                         │  │
│  │ POST /api/chat-icp                │◀────────│ icpChats.ts             │  │
│  │  └─ Streaming AI refinement      │         │  └─ Chat storage        │  │
│  │                                   │         │                         │  │
│  │ POST /api/prospect                │◀────────│ prospecting.ts          │  │
│  │  └─ Company & contact research   │         │  └─ Prospect jobs       │  │
│  │                                   │         │                         │  │
│  │ POST /api/outreach/generate-email │◀────────│ outreach.ts             │  │
│  │ POST /api/outreach/email          │         │  └─ Campaign tracking   │  │
│  │ POST /api/outreach/call           │         │                         │  │
│  │  └─ Email & voice outreach       │         │ icpSets.ts, icps.ts     │  │
│  └──────────────────────────────────┘         │ userProfiles.ts         │  │
│                                                └─────────────────────────┘  │
│  Auth: Auth0 Middleware                        Auth: Auth0 JWT Tokens      │
│  Timeout: 5 minutes                            Real-time: Reactive Queries │
└──────────┬────────────┬─────────────┬──────────────────┬────────────────────┘
           │            │             │                  │
           │            │             │                  │
           ▼            ▼             ▼                  ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│  DATA LAYER (Convex Real-time Database)                                     │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  Tables (schema.ts):                                                         │
│  ┌──────────────────┐  ┌──────────────┐  ┌─────────────────┐               │
│  │ icpGenerationJobs│  │  icpSets     │  │  icpChats       │               │
│  ├──────────────────┤  ├──────────────┤  ├─────────────────┤               │
│  │ • status         │  │ • 3 ICPs     │  │ • messages[]    │               │
│  │ • searchQueries  │  │ • userId     │  │ • timestamps    │               │
│  │ • searchResults  │  │ • createdAt  │  │ • linked to ICP │               │
│  │ • tempICPs       │  │ • indexes    │  └─────────────────┘               │
│  └──────────────────┘  └──────────────┘                                     │
│                                                                              │
│  ┌──────────────────┐  ┌──────────────┐  ┌─────────────────┐               │
│  │ prospectingJobs  │  │ outreachCampaigns │ userProfiles   │               │
│  ├──────────────────┤  ├──────────────┤  ├─────────────────┤               │
│  │ • 5 companies    │  │ • type       │  │ • whatBuilding  │               │
│  │ • 3 prospects ea.│  │ • status     │  │ • userId        │               │
│  │ • status/company │  │ • resendId   │  └─────────────────┘               │
│  │ • enriched data  │  │ • vapiCallId │                                     │
│  └──────────────────┘  └──────────────┘                                     │
│                                                                              │
│  Indexes: by_userId, by_status, by_createdAt, by_icpSet, by_prospect        │
└─────────────────────────────────────────────────────────────────────────────┘


┌─────────────────────────────────────────────────────────────────────────────┐
│  EXTERNAL SERVICES & INTEGRATIONS                                           │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  AI & Search                  Communication              Observability      │
│  ┌──────────────┐             ┌──────────────┐          ┌──────────────┐   │
│  │ OpenAI GPT-4o│             │ Resend       │          │ OpenTelemetry│   │
│  ├──────────────┤             ├──────────────┤          ├──────────────┤   │
│  │• ICP gen     │             │• Email send  │          │• Trace all   │   │
│  │• Chat        │             │• Tracking    │          │  LLM calls   │   │
│  │• Email copy  │             └──────────────┘          │• Token usage │   │
│  └──────────────┘                                       │• Latency     │   │
│                                ┌──────────────┐          └──────────────┘   │
│  ┌──────────────┐             │ Vapi AI      │                 │           │
│  │ Exa API      │             ├──────────────┤                 ▼           │
│  ├──────────────┤             │• AI calls    │          ┌──────────────┐   │
│  │• Neural      │             │• Transcripts │          │ Opik         │   │
│  │  search      │             │• Recording   │          │ Dashboard    │   │
│  │• Companies   │             └──────────────┘          └──────────────┘   │
│  └──────────────┘                                                           │
│                                                                              │
│  ┌──────────────┐             ┌──────────────┐                              │
│  │ Hunter.io    │             │ Auth0        │                              │
│  ├──────────────┤             ├──────────────┤                              │
│  │• Email find  │             │• Login/OAuth │                              │
│  │• Contact data│             │• JWT tokens  │                              │
│  └──────────────┘             └──────────────┘                              │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

Key Data Flows

1. ICP Generation Flow

Multi-step AI pipeline with real-time updates:

User Query → CreateJob → Exa Search (6 queries) → AI Generation → SaveICPSet → Display 3 ICPs
                              ↓
                    [Real-time updates at each step]

Detailed Steps:

User enters search query + "What are you building" context
Frontend: SearchInput component submits to /api/generate-icps
Convex: Create icpGenerationJobs record with 6 search queries
Frontend: Navigate to /icps/generating/[jobId] with real-time updates
API Route: Execute 6 Exa searches sequentially (250ms delay between)
AI: OpenAI GPT-4o generates 3 ICPs with structured output (Zod schema)
Convex: Save to icpSets table
Frontend: Auto-navigate to /icps/[icpSetId] to display results

2. ICP Refinement Flow

Streaming AI chat with tool calls:

User Message → Stream AI Response → Tool Calls (updateICP) → Convex Mutation → Real-time UI Update

Detailed Steps:

User clicks ICP → Chat panel opens
User types message → POST /api/chat-icp
API: Stream response from OpenAI with streamText
AI calls updateICP tool during streaming (e.g., update value proposition)
onFinish callback: Apply tool calls to Convex
Frontend: ICP view updates automatically via reactive query

3. Prospect Research Flow

Multi-API orchestration with status tracking:

ICP → Exa Search (5 companies) → Hunter.io (3 prospects/company) → Prioritize by Title → Display Table
                                        ↓
                              [Sequential with status tracking]

Detailed Steps:

User clicks "Research Prospects" on ICP
POST /api/prospect with ICP data
Exa: Search for companies matching ICP criteria
Convex: Create prospectingJobs with 5 companies (status: pending)
For each company sequentially:
- Update status to "prospecting"
- Hunter.io: Domain search for emails
- Prioritize by decision-maker titles (CEO, VP, etc.)
- Take top 3 prospects
- Update status to "complete"
- Rate limit: 1 second delay
Frontend: Display companies table with prospects

4. Email Outreach Flow

AI-generated personalized emails:

Prospect → AI Generate Email → User Edit → Resend API → Track Status
                ↓
    [Context: ICP messaging + previous emails]

Detailed Steps:

User clicks "Email" on prospect → EmailComposer modal opens
POST /api/outreach/generate-email with prospect + ICP + previous emails
AI: Generate email body (150-200 words) + subject line (8 words max)
Frontend: Display draft email (user can edit)
User clicks "Send" → POST /api/outreach/email
Convex: Create/update outreachCampaigns (status: sending)
Resend API: Send email
Convex: Update status to "sent" with Resend ID
Frontend: Success toast

5. Voice Call Flow

AI-powered phone calls with context:

Prospect → Build Context → Vapi API → Store Call ID → Track Transcript
                ↓
    [Context: ICP + emails + solution]

Detailed Steps:

User clicks "Call" on prospect → CallInterface modal opens
POST /api/outreach/call with prospect + ICP + previous emails
Build assistant context string (solution, prospect info, value prop, email history)
Vapi API: Initiate phone call with:
- assistantId from environment
- customer.number (hardcoded for demo)
- assistantOverrides with personalized first message
Convex: Update campaign status to "completed" with vapiCallId
Frontend: Show call status
Optional: GET /api/outreach/call?callId={id} for call details

Tech Stack Summary

Frontend

Framework: Next.js 15 with App Router
React: Version 19.1.0
Language: TypeScript 5
Styling: Tailwind CSS 4
UI Components: Radix UI primitives + shadcn/ui patterns
Icons: Lucide React, Tabler Icons
Animations: Framer Motion 12
Theme: next-themes (dark/light mode)
Notifications: Sonner (toast)
Validation: Zod 4.1.12

Backend

Database: Convex (real-time serverless database)
API: Next.js API Routes (Edge + Node.js runtimes)
Auth: Auth0 (@auth0/nextjs-auth0)

AI & ML

Primary Model: OpenAI GPT-4o
AI SDK: Vercel AI SDK (@ai-sdk/openai, @ai-sdk/react)
Search: Exa API (neural semantic search)

External Services

Email: Resend (transactional email)
Voice: Vapi AI (AI phone calling)
Contacts: Hunter.io (email discovery)

Observability

Telemetry: OpenTelemetry (@opentelemetry/sdk-node)
Tracing: Opik (opik-vercel) - AI observability
Instrumentation: @vercel/otel + auto-instrumentations

Build & Deploy

Package Manager: pnpm
Build Tool: Turbopack (Next.js 15)
Deployment: Vercel (Serverless Functions + Edge Network)
Linting: ESLint 9

Architectural Characteristics

1. Real-time Reactive

Convex Reactive Queries: Frontend automatically re-renders when database changes
Optimistic Updates: Mutations return immediately, UI updates optimistically
No REST APIs: Direct function calls from frontend to Convex backend
Type Safety: Full TypeScript types from database schema to frontend

2. Streaming AI

Progressive Enhancement: UI updates as AI generates text chunk-by-chunk
Tool Calls: AI can call functions during streaming (e.g., updateICP)
Telemetry Integration: All AI calls traced for observability
Error Recovery: Graceful fallback if AI stream fails

3. Multi-step Async Workflows

Job Pattern: Create job record → Update status → Store results
Progress Tracking: Real-time status updates visible in UI
Error Isolation: Individual step failures don't crash entire workflow
Rate Limiting: Delays between API calls to avoid rate limits

4. Schema-first Development

Zod Schemas: Define data structure once, use everywhere
AI Structured Output: GPT-4o generates JSON matching Zod schema
Database Validation: Convex validates all writes against schema
Type Generation: TypeScript types derived from schemas

5. Composition Over Inheritance

Component Composition: Small, reusable UI primitives
Provider Composition: Nested context providers for global state
Layout Nesting: App Router layouts provide shared UI structure
Utility Functions: Pure functions for reusable logic

Project Structure

/Users/ferran/workspace/hackathons/ai-by-the-bay/
│
├── app/                              # Next.js 15 App Router
│   ├── layout.tsx                    # Root layout (fonts, metadata, providers)
│   ├── globals.css                   # Global Tailwind styles
│   │
│   ├── (main)/                       # Authenticated application routes
│   │   ├── layout.tsx                # Navbar, theme toggle, logout
│   │   ├── page.tsx                  # Home: search input or landing page
│   │   │
│   │   ├── icps/                     # ICP management
│   │   │   ├── page.tsx              # List all ICPs
│   │   │   ├── generating/[id]/      # ICP generation progress
│   │   │   │   ├── page.tsx          # Real-time generation status
│   │   │   │   └── [index]/          # Individual ICP detail during generation
│   │   │   │       └── page.tsx
│   │   │   └── [id]/                 # Completed ICP sets
│   │   │       ├── page.tsx          # List ICPs in set
│   │   │       └── [index]/          # Individual ICP detail
│   │   │           ├── page.tsx      # View ICP + chat
│   │   │           └── prospects/    # Prospect management
│   │   │               ├── page.tsx  # List prospects
│   │   │               └── outreach/ # Outreach interface
│   │   │                   └── page.tsx
│   │   │
│   │   └── me/                       # User profile
│   │       └── page.tsx              # "What are you building?" settings
│   │
│   └── api/                          # Next.js API Routes (serverless functions)
│       ├── generate-icps/            # ICP generation orchestration
│       │   └── route.ts              # POST: Search + AI generation (5min timeout)
│       ├── prospect/                 # Prospect research
│       │   └── route.ts              # POST: Exa search + Hunter.io enrichment
│       ├── chat-icp/                 # ICP refinement chat
│       │   └── route.ts              # POST: Streaming AI chat with tool calls
│       └── outreach/                 # Outreach automation
│           ├── generate-email/       # AI email generation
│           │   └── route.ts          # POST: Generate personalized email
│           ├── email/                # Email sending
│           │   └── route.ts          # POST: Send via Resend
│           └── call/                 # Voice calling
│               └── route.ts          # POST: Initiate Vapi call, GET: Call status
│
├── components/                       # React components
│   ├── ui/                           # Reusable UI primitives (shadcn pattern)
│   ├── landing/                      # Landing page sections
│   ├── SearchInput.tsx               # ICP search/generation input
│   ├── ICPCard.tsx                   # ICP preview card
│   ├── ChatPanel.tsx                 # ICP refinement chat UI
│   ├── CompaniesTable.tsx            # Prospect companies table
│   ├── EmailComposer.tsx             # Email draft/send interface
│   └── CallInterface.tsx             # Voice call interface
│
├── convex/                           # Convex backend (real-time database)
│   ├── schema.ts                     # Database schema definitions
│   ├── auth.config.ts                # Auth0 integration config
│   ├── icps.ts                       # ICP CRUD operations
│   ├── icpSets.ts                    # ICP set management
│   ├── icpGeneration.ts              # Generation job management
│   ├── icpChats.ts                   # Chat message storage/updates
│   ├── prospecting.ts                # Prospect job management
│   ├── outreach.ts                   # Outreach campaign tracking
│   └── userProfiles.ts               # User profile management
│
├── lib/                              # Utility libraries
│   ├── auth0.ts                      # Auth0 SDK initialization
│   ├── utils.ts                      # Helper functions
│   └── schemas/                      # Zod schemas
│       └── icp.ts                    # ICP validation schemas
│
├── providers/                        # React context providers
│   ├── providers.tsx                 # Provider composition
│   ├── convex-provider.tsx           # Convex client provider
│   └── theme-provider.tsx            # Theme context provider
│
├── middleware.ts                     # Next.js middleware (Auth0)
├── instrumentation.ts                # OpenTelemetry setup
└── components.json                   # Shadcn UI config

External Service Dependencies

Service	Purpose	Critical	API Key Required
OpenAI	ICP generation, chat, email writing	Yes	`OPENAI_API_KEY`
Exa	Company search, market research	Yes	`EXA_API_KEY`
Auth0	User authentication	Yes	`AUTH0_*`
Convex	Real-time database	Yes	`CONVEX_DEPLOYMENT`
Resend	Email delivery	High	`RESEND_API_KEY`
Hunter.io	Contact discovery	High	Implicit in code
Vapi	AI phone calls	Medium	`VAPI_API_KEY`
Opik	AI observability	Low	`OPIK_API_KEY`

Rate Limits Handled:

Exa: 250ms delay between searches
Hunter.io: 1 second delay between companies
OpenAI: Managed by Vercel AI SDK (auto-retry)

Security

Authentication

Auth0 handles all credential management
Session cookies with httpOnly flag
No passwords stored in application database

Authorization

User-scoped queries via Convex indexes
Server-side session validation in middleware
API routes validate Auth0 tokens

Data Protection

API keys stored in environment variables (not in code)
Sensitive data (emails, phone) hardcoded for demo (not production-safe)
Convex handles encryption at rest

Performance Optimizations

Frontend

React Server Components for initial render
Turbopack for fast builds (Next.js 15)
Code splitting via dynamic imports
Image optimization via Next.js Image component

Backend

Convex indexes for fast queries
Real-time updates avoid polling
Parallel API calls where possible

AI

Streaming responses reduce perceived latency
Cached Exa results (15-minute cache)
Telemetry for performance monitoring

Deployment

Target Platform: Vercel (Serverless)

Deployment Components:

Next.js App: Vercel Edge Network (global CDN)
API Routes: Vercel Serverless Functions (5-minute max duration)
Convex Backend: Convex Cloud (separate infrastructure)
Static Assets: Vercel CDN (public/ folder)

Environment Variables (Production):

AUTH0_BASE_URL → Production domain
APP_BASE_URL → Production domain
All API keys → Production credentials

Continuous Deployment:

Push to GitHub → Automatic Vercel build
Preview deployments for pull requests
Main branch → Production deployment

Built With

auth0
comet
exa
nextjs
opik
vapi

Post-Mortem: ICP Refinement Failure

Objective

What Happened

Diagnosis

Root Cause

Key Learnings

Next Steps

Lead Finder Architecture

Architecture Diagram

Key Data Flows

1. ICP Generation Flow

2. ICP Refinement Flow

3. Prospect Research Flow

4. Email Outreach Flow

5. Voice Call Flow

Tech Stack Summary

Frontend

Backend

AI & ML

External Services

Observability

Build & Deploy

Architectural Characteristics

1. Real-time Reactive

2. Streaming AI

3. Multi-step Async Workflows

4. Schema-first Development

5. Composition Over Inheritance

Project Structure

External Service Dependencies

Security

Authentication

Authorization

Data Protection

Performance Optimizations

Frontend

Backend

AI

Deployment

Built With

Updates

Submission history