GEO Experiment Platform

Inspiration

As search engines rapidly evolve into conversational AI agents, traditional SEO is dying. We are entering the era of Generative Engine Optimization (GEO), but e-commerce vendors and researchers are completely in the dark. How do you optimize product listings for an autonomous AI agent?

This platform was inspired by my doctoral thesis research. I needed a rigorous, data-driven laboratory to execute massive A/B tests on product catalogs, simulating how a conversational agent reacts to AI-optimized data. Traditional databases weren't built for the autonomous scale required to process hundreds of thousands of LLM calls, which led to the creation of the GEO Experiment Platform.

What it does

The GEO Experiment Platform is an end-to-end laboratory, survey, and data collection platform built to assess the real-world efficacy of GEO. It perfectly embodies the theme of agents that think, act, and remember reliably at scale:

• Massive Scale A/B Testing (Agents that Act): Researchers can upload massive e-commerce catalogs (up to 1,000,000 products). The platform autonomously iterates through the database, acting on the data by applying generative "GEO treatments" (like factual restructuring or statistical enrichment) to a stratified 50% split.

• AI Shopping Assistant (Agents that Think): Survey respondents interact with a live AI Shopping Assistant powered by a Retrieval-Augmented Generation (RAG) pipeline. The agent dynamically reasons over user queries and complex catalog data to provide grounded, expert shopping recommendations.

• Efficacy Tracking (Agents that Remember): Behind the scenes, the platform relies on CockroachDB to act as a persistent memory layer for the agent. It stores everything from raw product embeddings to granular conversation history, user context, and task states. By tracking this memory globally and reliably, the platform yields hard empirical data on whether GEO-optimized products were recommended more often than control products, proving that persistent memory is the exact mechanism that makes this agentic research possible in production.

How we built it

The platform consists of a React/Vite frontend and a Python FastAPI backend deployed on AWS infrastructure, but the absolute heart of the system is CockroachDB, which serves as our resilient, agentic memory layer:

• Persistent Agentic Memory: We used CockroachDB to store the complex web of conversational threads, task states, survey data, and A/B cohort designations. Because the agent relies heavily on past context, this data needed to be strictly consistent and always online.
• Distributed Vector Indexing: We utilized CockroachDB's native pgvector support to store thousands of product embeddings right alongside our relational data. When a user asks the AI a question, we execute semantic similarity searches directly within CockroachDB to retrieve product context. Having no separate vector store to maintain eliminated data consistency gaps entirely.
• Resilient Background Workers: We built custom Python scripts to process hundreds of thousands of LLM calls in the background, utilizing strict batching and commit intervals to ensure zero data loss.

Challenges we ran into

When attempting to apply generative AI treatments to a massive dataset (600,000+ Amazon products), we quickly ran into cloud load-balancer idle timeouts. Because generating batches of 500 records via OpenAI took over 30 minutes, our database connections were being unexpectedly severed while sitting idle in the pool.

We had to architect a much more resilient agentic memory pipeline. We solved this by implementing SQLAlchemy pool_pre_ping to handle aggressive serverless timeouts and reduced our batch sizes to 20 records. This forced our script to commit to the SQL tables and index the Vectors every two minutes, keeping the connection perfectly healthy and making the pipeline practically bulletproof.

Accomplishments that we're proud of

• Zero-Sync Architecture: By using CockroachDB's Distributed Vector Indexing, we successfully merged highly structured A/B test data with unstructured semantic vectors in a single query system. We are incredibly proud of how fast our RAG pipeline runs without maintaining a standalone vector database.
• Scale: Building a background optimization script that can gracefully chunk and process 1,000,000 agentic API calls without crashing or losing data.
• Academic Milestone: Creating a platform robust enough to actually serve as the primary data collection tool for a Doctoral Thesis!

What we learned

We learned that agentic systems demand fundamentally different memory architectures than traditional CRUD apps. If an agent's memory goes offline halfway through processing 10,000 API calls, it doesn't degrade gracefully, it crashes. Relying on CockroachDB's persistent memory proved that a distributed, fault-tolerant system of record is the only way to put AI agents into production safely.

What's next for GEO Experiment Platform

Next, we plan to fully containerize our agent execution environment on AWS ECS, expand our AI treatments to test specific prompt engineering strategies (like evidence-first formatting), and finally launch the platform to live survey respondents to gather the empirical data needed for my thesis publication!

Built With

• agentic-memory
• amazon-web-services
• cockroachdb
• fastapi
• llm
• pgvector
• postgresql
• python
• react
• semantic-search
• sqlalchemy
• tailwind-css
• typescript
• vector-database
• vite