GTM Research Agent
My goal was to understand my market. Who is already selling to the people I want to reach, what they teach, what they sell, and what their audience says about it.
This agent does that research for me. Give it a YouTube channel and it watches the content, reads everything the creator links to, profiles their business, and scores how my product fits. Then it goes looking for more channels like them. 60+ creators are profiled in there today. I chat across all of them, or hand the agent a goal and get back a cited research report.
Inspiration
I'm building a product for creators who monetize their expertise, launched with the X Prize Gemini Hackathon. I want to be able to profile and research creators, and see if their a fit for my product.
I was doing this already by hand. Open a channel, watch a few videos at 2x, click every link in the description, skim the sales page, form an opinion. Most of an hour per creator, and the answer lived in my head. After sixty of those I had a pile of opinions I couldn't check and couldn't query. Asking "what does the audience complain about across creators who teach AI tools" meant starting the whole thing over.
So before writing any code I spent a day away from the keyboard working out what an ideal research partner would do.
Whiteboard, paper, then a mindmap to turn the mess into stages and tools.
Two decisions came out of that day and made into the build. The first is that the work splits in half. Profiling a channel is the same steps every time, so it runs as a deterministic workflow and the model never chooses the order.
Finding new channels and answering research questions is open-ended, so those run as agentic loops where the model decides when it has enough.
The second decision is where I sit: approving or rejecting candidates in an Inbox. Everything before that is the agent's job, and every call I make there becomes memory it uses on the next pass.
What it does
One YouTube URL goes in. Three stages come out.
Stage 1. Profile a partner. A fixed ingestion workflow: pull the channel, pull video transcripts, shorts and comments, harvest every link in every description, have Gemini classify each link (own site, landing page, affiliate, social), read the pages worth reading through Jina, then chunk and embed the whole thing into a Firestore vector index. Then it writes the profile. Custom sections generated from prompts I control, structured properties the model fills in so I can filter and group later, an audience digest built from comments, and a predicted fit score that learns from the ratings I've already given other partners. Every stage, token count and dollar is logged.
Stage 2. Find look-alike partners. The moment a profile lands, a discovery run starts. Gemini writes search queries from that specific profile and its video titles, runs them against Google Search and the YouTube API, groups and dedupes the results, then triages what's left against my fit criteria and every channel I've previously rejected. Survivors get a snapshot profile and land in my Inbox with a score and a one-line reason. I approve or reject; both answers are stored and shape the next pass. 83% of the partners in the database were found this way.
Stage 3. Chat and deep research. With the corpus in place I can ask it things. Chat runs RAG over the Firestore index with two context dials: which partners (all, a filtered group, a hand-picked few) and how much of each one (whole profile, one section, or raw content and transcripts). Deep research takes a goal instead of a question. If the goal is vague, the agent pushes back and shapes it with me before it plans anything. It then breaks the goal into 3–6 sub-questions and runs each as its own tool loop, using profile vector search, section reads, transcript retrieval, and Google Search with URL context when the knowledge base has nothing concrete. Each loop stops when it has enough, so it never has to touch every partner. The output is a written report with citations, plus an optional infographic from Gemini's image model. I can comment on any report and open a chat with that report as context.
How I built it
Models. gemini-3.5-flash for every generation and tool loop, through
Genkit. gemini-embedding-001 for retrieval. gemini-3-pro-image and
gemini-3.1-flash-image for report infographics.
Framework. Genkit, running flows for ingestion, discovery, chat and reports.
The agentic surfaces are Genkit tool loops with Zod-typed tools: searchProfiles,
getProfile, retrieveContent, createImage, plus Gemini's built-in
googleSearch and urlContext. Research loops are capped at 12 turns per
sub-question.
Infrastructure. Cloud Run hosts an Express API (1 CPU / 1 GiB, max 5 instances, 1800s timeout). Firestore holds documents, the vector index and the run log. Cloud Tasks runs everything that outlives a request. Third-party data comes from the YouTube Data API, ScrapeCreators and Jina Reader. The front end is Next.js 16 in a Turborepo monorepo, sharing wire types and a typed API client with the backend.
Control surfaces. Profile sections and their prompts are stored as data. I can add a section, test its prompt against a live profile, save it, and backfill it across every partner. Same for structured properties. Every agent area's system instructions are editable in the admin, with the shipped defaults as fallback.
Observability. Every run and event is written to Firestore with duration, tokens and cost, including third-party credits. Prices are applied at write time, so a later price change doesn't rewrite history. The admin has an active-runs view, a live event tail, a per-run cost table, and retry on any failed run.
Security. The API refuses to boot on Cloud Run without API_SECRET. The
browser never talks to it: the Next.js app checks a signed role cookie and
proxies with a bearer token and an asserted role, and the API enforces
admin/viewer separately from the UI. Cloud Tasks handlers are exempt from the
bearer gate and verify a per-path OIDC token instead. Firestore rules deny all
client access.
Challenges I ran into
Cloud Run kills background work. CPU is only allocated while a request is in flight, so a fire-and-forget ingest stalls the moment the response is sent and dies on scale-in. The fix was Cloud Tasks. Every long run is enqueued as a task that POSTs back to the same service, and the whole run executes inside that request. The service timeout and the task dispatch deadline both had to move to 1800 seconds to match.
Rate limits, three kinds of them. The free tier ran out on day one at 20 requests. Per-minute embedding limits then broke ingests that embed hundreds of chunks with several runs in flight, which needed jittered exponential backoff up to 64 seconds, plus a check that separates a 429 that will heal from a 429 that means the prepaid credits are gone. Retrying the second kind just burns seven attempts. Running report sub-questions in parallel tripped the spend-tier limit too, so they run one at a time. $200+ CAD spent on Gemini Credits.
Losing the logs of the thing that failed. Run events were buffered, and Cloud Run gives about 10 seconds after SIGTERM. An interrupted run gets retried by the queue, but its lost log events are gone for good, so shutdown now flushes every in-flight run log before exiting.
Everything links to everything. Creator descriptions are full of zoom.us,
google.com and link-shorteners. Without a blocklist and a broken-link check the
scraper spent real credits reading Google's homepage.
Accomplishments that I am proud of
- It ran itself. 200 autonomous runs, 8,163 logged steps, 1,047 channels evaluated by the discovery loop with 792 hard-filtered and 148 dropped on low scores before a human saw anything. The agent found 83% of the partners in the database.
- The knowledge base. 65 channels reviewed, 3,189 videos, 897 social posts, 1,181 links read and classified, 12,773 embedded chunks, ~27,600 Firestore documents, ~210 MB, ~14.5 million words.
- Every run is auditable. Each one carries a cost, a step list and a retry button. I can see what a partner profile cost to build, down to the individual scraper credit.
- The agent changes without a deploy. A new profile section, property or system prompt gets written in the admin, tested against a live profile, then backfilled across sixty partners.
- I actually use it. It's shaping how I build my product right now: which partners to approach, what sales copy lands, which features matter, what to charge.
What I learned
Narrower context beats bigger context. The biggest quality jump in both chat and research came from choosing which slice of a profile goes into the prompt. Audience section only, or transcripts only. Stuffing the whole profile in made the answers vaguer.
Store the rejections. Every channel the agent passed on is written down with its reason, and that record is what makes each discovery pass narrower than the last. Without it the loop searches the same ground every time and I re-reject the same channels.
Costing every call changes the design. Once each run showed a dollar figure, some of my choices became obviously wrong. Re-embedding on every profile edit, scraping link shorteners, triaging the same channel from three different seeds. The 30-day seen-window and the batched single triage call both came from staring at the cost table.
What's next for my GTM Agent?
- Outreach. The profiles already contain what to say. The next step is drafting the actual message and tracking what happened after it was sent.
- The partner portal. The
apps/portalskeleton is in the repo. It shows a creator their own profile, so onboarding starts from research the agent already did. - Beyond YouTube. The ingestion stage is the only part that knows what a YouTube channel is. Podcasts and newsletters are the same pipeline with a different first step.
Architecture
The full pipeline: problem and goal, the fixed ingestion workflow, the agentic discovery loop, chat and research, and the admin and observability surfaces around them.
Log in or sign up for Devpost to join the conversation.