Nori | Devpost

Inspiration

Many product teams discovers confusing UX flows, overlooked edge cases, or expensive bugs only after launch when real users beta test the software. Nori was inspired by the idea to help early stage teams like startups to accelerate and scale their early beta testing and take some pressure off the human testers by putting an alternative in place.

Nori looked into the opportunity to create a solution, where the product could be tested at any time by a diverse virtual beta panel. Using AI synthetic agents that would be programmed to think like users with different goals, behaviours, confidence levels and pain points.

What it does

Nori is beta-testing powered by AI synthetic agents. Rather than giving generic feedback, each agent acts as a single user who approaches the product from a specific point of view and navigates through the product to fulfill their personal objective.

For example, there could be an agent representing a time-constrained first-time user who wants to complete the onboarding as quick as possible. Another agent could represent a more experienced but cautious user who wants to check every setting and permission to avoid any issues. Each agent is able to recognise failure points* identify confusing parts in the interface and report them in a way a product team could understand.

Nori aims to complement the existing research process with early and continuous feedback. It is not meant to replace human research and beta-testing, but instead give teams the ability to find potential issues before they are expensive to fix by simulating the most common discovery funnel of a new product user. We believe that this level of analysis could help save precious resources on unproven ideas and provide a solid foundation before a product is released to a larger audience.

How we built it

We built Nori as a full-stack web application using Next.js, React, TypeScript, and Node.js. The frontend provided a focused workspace where product teams could define a beta test, add product context, select the type of user journey they wanted reviewed, and examine feedback generated by their synthetic testing agents. We used JavaScript, HTML, CSS, and Unframer to create the product interface and interaction patterns.

On the backend, we used Next.js server-side capabilities and Node.js to manage test creation, agent orchestration, authentication, results processing, and API requests. PostgreSQL stored core application data, including user accounts, test configurations, agent personas, test runs, observations, and generated feedback. Clerk handled secure user authentication and account management.

Nori's synthetic agents were powered by the Anthropic API. Each agent received a structured prompt containing a defined persona, a product-testing objective, relevant context, and behavioural constraints. Instead of asking one model for a broad review, we created multiple specialised agents, for example, a first-time user, a time-constrained user, or a detail-oriented power user, and asked each one to work through a specific product journey. Their responses were collected, compared, and synthesised into clearer themes, risks, and next steps for the builder.

We used Playwright to automate browser-based testing flows, allowing agents to interact with the product experience rather than evaluating only a written description. This made it possible to test concrete steps such as navigating pages, completing forms, attempting sign-up or onboarding, and encountering error states. The AI layer could then interpret what happened during the journey and explain where a user was likely to become confused, blocked, or frustrated.

To keep the application reliable as it evolved, we used Zod for runtime validation of user input, API payloads, and structured agent outputs. Docker gave us a consistent development and deployment environment, while npm managed project dependencies. We maintained code quality with ESLint and Prettier, and used GitHub Actions to automate checks in our development workflow.

This stack let us combine real browser interaction with structured AI reasoning: Playwright captured what happened during a test, synthetic agents interpreted the experience from a particular user perspective, and Nori transformed those observations into actionable beta-test feedback.

Challenges we ran into

The most challenging aspect of the project was avoiding generic feedback that may sound good but does not provide actionable insight. The criticism of the products that the AI-provided had to be precise enough to give clear direction on what and how to improve. To resolve this issue, the researchers used personas and journeys as guides and asked the agents to identify a specific moment of truth, articulate why it is a problem, and suggest what can be done to resolve the issue.

The other challenging aspect of the project was ensuring that the feedback provided by different agents was varied enough to be useful but not exaggerated to the point of lying. Essentially, the AI had to provide different tones of advice and instruction without creating artificial disagreements between the agents, undermining their voices as real users. To address this issue, the researchers positioned synthetic agents as a first-line-of-defense usability test and a quality-assurance tool that complimented traditional research methods but did not replace them. The criticism provided by the AI-agents was not viewed as a replacement for user testing, including heuristic evaluation, accessibility audits, security assessments, and end-user feedback.

The last but not least challenge was to make sure that the feedback provided by the agents was actionable. Since multiple agents provided feedback, it was possible to get lost in the details, distracted by minor suggestions while missing a significant problem. To resolve this issue, the product had to prioritise issues according to their impact, identify common themes, recognize patterns, and provide clear direction on what to do next.

Accomplishments that we're proud of

We built an end-to-end beta-testing workflow that moves from defining a product scenario to receiving structured, agent-generated feedback.

We combined automated browser interaction through Playwright with AI reasoning through the Anthropic API, allowing agents to assess a real product journey rather than only a written prompt.

We created a multi-agent approach, where different synthetic users can test the same flow with distinct goals, expectations, and levels of product knowledge.

We turned individual agent observations into clearer themes, such as onboarding friction, unclear copy, missing states, navigation issues, and potential drop-off points.

We built Nori as a production-minded full-stack application using Next.js, React, TypeScript, Node.js, PostgreSQL, Clerk, Zod, Docker, and automated quality checks through GitHub Actions.

We designed the platform to be useful for fast-moving builders: create a test, run it against a real experience, review the findings, and use them to decide what to validate or improve next.

What we learned

We've learned that the quality of the agent feedback depends on the context. The most valuable feedback loops were created when a particular user persona was defined, a specific goal was set, and a concrete scenario was outlined. For instance, a situation when a user who has never used the product before tries to complete the onboarding flow and sees what UX challenges arise is much more informative than a vague question about improving the UX without context.

We've learned that browser automation combined with the power of reasoning behind AI is more effective at finding issues than either method alone. While using Playwright, one can easily see if a particular task is possible within the application. However, it becomes unclear what UX problems can arise and why the user might have abandoned the flow at some point. On the other hand, while reasoning about the experience, one can, indeed, imagine some obstacles but lack the actual user interaction context.

We've learned the importance of the feedback structure. We've seen how the structured schema allows for the validation of the agent's answers, making it easier to compare different models' performance while also presenting the results to the user in a more accessible way. It would eliminate some repetitive feedback and focus on the actual issues.

Most importantly, we've learned that synthetic agents can help design better user research. We've seen multiple examples of how asking questions with specific context, goals, and users in mind can help simulate the real user experience and highlight potential issues. However, we've also seen that real users are still integral to the process and that the performance has to be validated in front of human users.

What's next for Nori

We want to make Nori's agent personas more configurable to be representative of the target users a product team wants to test, including their goals, technical proficiency, urgency, accessibility needs, device preferences, and industry-specific context, to make agent testing more valuable for the audience they are building for.

Then, we will think about broader testing scenarios in the future like onboarding journeys, logged-in testing, user roles, purchasing and payments flows, responsive designs, and regression suites comparing variations of a product version.

Prioritisation is another significant area. Instead of presenting the observations as unordered entries, grouping them by their impact level (likelihood of occurrence, severity, and criticality) on the user journey and product team's priorities while providing context-specific remedial suggestions.

We plan to add richer reporting, including journey replays, screenshots at the point of failure, links to the relevant page or interaction, and clearer recommendations for the team. We also see an opportunity to integrate Nori into existing product workflows, such as GitHub pull requests, issue trackers, and CI/CD pipelines, so synthetic beta testing can become part of the process before a feature is released.

Finally, we want Nori to become an always-on pre-release feedback layer: a fast way for product teams to test assumptions, spot likely friction, and enter human beta testing with a stronger, more reliable product.