UserSim

What's next for UserSim

Inspiration

Every founder wants to know the same thing: when a real buyer compares my product with the alternatives, which one do they pick, and why? Finding out usually means weeks of recruiting, interviews and usability sessions, so most teams guess. We wanted an answer for any URL, in minutes, backed by evidence instead of opinions.

What it does

You paste in a product URL. UserSim reads the site, works out who the buyers are, and picks three real competitors. It then writes 5 buyer personas and 5 realistic tasks, and sends simulated users to try every task on your product and on each competitor. That comes to 100 agents per study.

The agents behave like real buyers. They sign up with real email addresses, confirm the verification email, get past captchas and SMS codes, and then do the task inside the product, such as finding at-risk accounts or checking the price for 50 seats. A judge model scores each attempt from what the agent actually saw.

The report leads with what matters: which product each persona would choose, a ranked list of where you win and lose (each with the competitor, the task, both scores, a quote and a screenshot), and a list of fixes.

How we built it

• Planner: Gemini reads the landing page and writes the segment, competitors, personas and tasks in a few seconds, so the first agent starts acting within about 10 seconds of submitting the URL.
• Agent loop: our own text-only Playwright loop. Each step, the agent reads the page's accessibility tree (no screenshots) and picks one action. That keeps it fast and cheap and lets it work on any site.
• Signup: Gmail plus-addresses give every agent a fresh, real inbox. The agent reads verification emails over IMAP, CapSolver handles captchas, and SMS codes are forwarded from a phone to an ntfy topic that the agent reads.
• Browsers: Browserbase runs the cloud browsers, with a queue so 100 agents share 25 browsers.
• Judging: Gemini scores each run by level. Doing the task inside the product scores highest, clear evidence on the website comes next, and vague marketing or a wall scores lowest. Ties go to the level first and then to less friction. Buyer picks are made blind, with shuffled, unlabeled products.
• Stack: Python and FastAPI on a GCP VM, Vertex AI, and a Vercel front end with live progress.

Challenges we ran into

• Signups are the hardest part. Sites throttle verification emails, reject disposable inboxes, show image captchas, and ask for a phone number for 2FA. We moved to real Gmail aliases, added captcha solving, and built the SMS-to-ntfy relay.
• Fairness. Early studies favored the product being tested. We fixed that with identical task wording on every site, blind shuffled picks, and leaving out runs that our own harness stopped.
• Planning mistakes. A landing page that opens with testimonials can make the planner pick the wrong category and the wrong competitors, so we had to give it better context from the page.
• Speed at scale. Getting 100 agents through a study in about 10 minutes on 25 shared browsers meant queueing, preopening browsers and starting on a fast plan first.
• Knowing when a task is done. Agents would declare success too early or keep going past the answer. Each task type needed its own clear stopping point.

Accomplishments that we're proud of

• Full 100-agent comparison studies with every agent running, and the first action in about 5 seconds.
• Live signups on real products with real email and SMS verification.
• A report that answers "who would pick us, and why" with a quote and a screenshot behind every claim.

What we learned

• For a buyer, the value is the comparison, not whether a task was completed.
• An accessibility-tree agent is faster and more reliable than a screenshot agent for most web tasks.
• Every special case you hardcode eventually breaks on some site. The fewer rules the agent has, the better it generalizes.
• Evidence-backed scoring (a quote and a screenshot for every score) is what makes people trust the results.

What's next for UserSim

• Better scoring for competitors that only offer a demo, so website-only runs aren't all scored the same.
• More browsers so 100-agent studies finish under 8 minutes.
• Scheduled studies that alert you when a competitor ships something that changes buyer picks.
• Let teams supply their own personas and tasks, and invite real users to compare their answers with the simulated users.

Built With

• browser-use
• browserbase
• capsolver
• fastapi
• gemini
• gmail-api
• google-cloud
• imap
• javascript
• ntfy
• playwright
• python
• vercel
• vertex-ai