testX

AI-powered exploratory testing that thinks like a real user.

Comment

Inspiration

Building software is becoming faster, but testing it still requires a large amount of repetitive manual work.

Traditional automated tests are good at checking known paths, but they often miss the kinds of problems that real users encounter:

  • A notification remains visible after it has been read.
  • A user receives something but cannot find it again.
  • Two different pages appear to represent the same feature.
  • A workflow technically works, but the product logic feels incomplete.
  • A new release fixes one issue while quietly breaking another flow.

These problems are not always code errors. Many of them are product consistency, state-management, and user-experience problems.

I created testX to explore whether an AI agent could test a product more like a real user: understanding the intended experience, navigating complete workflows, identifying inconsistencies, and producing actionable reports instead of only reporting failed assertions.

What it does

testX is an AI-assisted product testing agent for web and mobile applications.

A user provides:

  1. A product description or testing objective.
  2. The application URL, local development environment, or simulator.
  3. Optional requirements, screenshots, or known regression cases.

testX then creates a structured testing plan and evaluates the product across several layers:

  • Core user journeys
  • Feature completeness
  • UI state transitions
  • Notifications and unread states
  • Data persistence
  • Navigation and entry-point consistency
  • Empty, loading, error, and edge states
  • Regression scenarios
  • Product logic and usability issues

Instead of producing only a list of technical failures, testX generates a product-focused report containing:

  • Reproduction steps
  • Expected behavior
  • Actual behavior
  • Severity
  • Affected workflow
  • Screenshots or evidence
  • Possible root cause
  • Recommended acceptance criteria
  • Suggested regression cases

The goal is not to replace engineers or QA specialists. The goal is to help small teams discover product problems earlier and turn informal testing into a repeatable process.

Example

Suppose an application contains a notification center.

A traditional test may confirm that the notification page loads successfully.

testX would continue investigating:

  • Does a read notification disappear from the unread area?
  • Is it still available in notification history?
  • Does the unread count update immediately?
  • Can the user find the related content from another entry point?
  • Is the same state reflected after restarting the application?
  • Does the behavior remain correct after receiving multiple notifications?

This allows testX to find problems that exist across an entire user journey rather than within a single screen.

How I built it

The project is designed around four main components.

1. Test Objective Interpreter

The user describes the product and the goal in natural language. An OpenAI model converts this information into:

  • Product assumptions
  • Core entities
  • User roles
  • Important workflows
  • Test scenarios
  • Acceptance criteria
  • Risk areas

2. Exploration Agent

The agent navigates the application step by step.

For a web application, it can interact with pages through browser automation. For mobile applications, the same architecture can later be extended to simulator-based testing.

The agent records each action, the visible state, and the reason for selecting the next action.

3. Evidence and State Recorder

testX stores structured observations for every testing session:

  • Page or screen
  • User action
  • Previous state
  • Resulting state
  • Visible text
  • Screenshot
  • Network or console error
  • Related test objective

This makes the final findings traceable and reproducible.

4. Report Generator

The recorded evidence is analyzed by an OpenAI model and converted into a concise testing report.

Similar findings are grouped together, duplicate issues are removed, and every issue is connected to a user workflow and a suggested regression case.

Challenges

One of the main challenges was distinguishing between a real defect and an intentional product decision.

An AI agent can observe that something changed, but it cannot automatically know whether that behavior is correct without understanding the product requirements. To address this, testX separates:

  • Confirmed requirement violations
  • Likely product inconsistencies
  • Usability concerns
  • Open questions requiring product confirmation

Another challenge was preventing the agent from wandering through the application without a clear purpose. The exploration process therefore uses a hierarchy:

  1. Testing objective
  2. User journey
  3. Scenario
  4. Action
  5. Observation
  6. Conclusion

This keeps the testing process focused and makes the results easier to explain.

A further challenge was evidence quality. A useful testing report must show not only that a problem exists, but also how to reproduce it. testX therefore treats screenshots, application state, and action history as first-class data rather than optional attachments.

What I learned

This project reinforced that product testing is not just about verifying buttons and APIs.

The most valuable problems are often found between features:

  • A state changes on one page but not another.
  • A notification exists without a clear destination.
  • Data is created but cannot be reviewed later.
  • The same concept is represented differently across the product.
  • A workflow is technically complete but does not match user expectations.

I also learned that an AI testing agent becomes more reliable when it is required to separate observations from conclusions. Instead of immediately declaring something a bug, the agent first records what happened, compares it with the expected behavior, and then assigns a confidence level.

Accomplishments

The most important accomplishment is defining a testing workflow that combines AI reasoning with structured, reproducible evidence.

Rather than using an AI model as a simple bug-report writer, testX uses it throughout the testing process:

  • Understanding the product
  • Designing scenarios
  • Selecting meaningful actions
  • Comparing states
  • Identifying inconsistencies
  • Producing regression cases

The resulting report is intended to be useful to product managers, designers, developers, and QA engineers.

What's next for testX

The next steps include:

  • Supporting authenticated testing sessions
  • Importing product requirements and user stories
  • Running scheduled regression tests
  • Comparing results between product versions
  • Connecting findings to GitHub Issues
  • Supporting iOS and Android simulators
  • Generating executable automated test cases
  • Learning from issues that teams accept or reject
  • Creating reusable testing memory for each project

The longer-term vision is for testX to become a persistent product-quality assistant that understands how an application is supposed to behave and continuously checks whether that behavior remains consistent.

Built With

Share this project:

Updates