NovaTest

Inspiration

I was working on a Zapier style app that allowed users use blocks to create workflows - much like Apple Shortcuts but specifically tailored to students. After implementing the core apps, I realised that the app was utterly unmaintainable for two reasons:

1. There are a lot of apps - using an API based system means individual implementation. For a team of one (just me) this is not scaleable quick enough, and would take up all my time.
2. Many learning tools do not provide APIs or the API is terrible. This means grabbing data would take additional web scraping, which is again app specific, requiring individual implementation. The best approach would a combination of both. Fast, reliable APIs and a dynamic web based agent for additional data. Then as I was implementing the new system, I ran into hundreds of visual bugs as code blocks were being flung all over the page. I thought it would be fun to automate the testing of UI elements, because doing so would save hours down the line.

What it does

NovaTest is a GitHub app that allows you to register a project using an existing GitHub repository containing a web app or website. In the project dashboard you can launch tests - fleets of agents that explore your website and attempt to find UI/UX edge case errors as well as loading time errors. These errors are very difficult to catch in unit tests, as they are usually caused by weird user input combinations. In the test run interface, you can view ongoing tests, see faults detected, and rerun tests.

How we built it

The frontend was build with Next.js, which made routing and styling the site easy. The Next.js api handles three things:

1. Supabase storage and retrieval - all user data including past tests and custom agents live here
2. GitHub login and app installation for fetching user repositories and profile information - username, avatar, etc.
3. Python Server communication, for starting NovaAct runs with a target configuration The server itself has two main parts: The fastapi app hosting which reads messages from the client and starts and act. The fastapi endpoint sends the configuration to the act_runner class providing context to the run itself (target URL, agent actions, context, temperature, etc.). While running, the logs emitted by nova act are captured and sent directly to Supabase. The client listens in to Supabase and relays the changes to the UI, which is how live logging works in the test detail view.

Challenges I ran into

The original idea for SDK-client communication was to use web-sockets that send commands to each other. The SDK would stream live metadata and runtime logs to the client to be displayed, and could also request human approval from the client. The client sends commands (start, cancel, etc.). I realised that my web-socket system was not persistent - refreshes and navigation would clear the event handlers, essentially cutting off the client. The socket system needed a middleman that lives on a server to buffer and store data, even when the client is closed. I decided to make the server send test run logs directly to a Supabase table storing events associated with an run_id. Then I could subscribe to this database on the client providing live persistent logging, albeit with slightly more latency. I also removed the logs column from the original test run database, which was a growing text array - notoriously inefficient to store.

Accomplishments that I'm proud of

I'm extremely proud that I was able to stick with this project throughout all of its 4-5 refactors - there were times when the project seemed hopeless, but it forced me to reread and simplify, and now I have a clean and simple system for starting and tracking NovaAct runs.

What we learned

I learned a ton about production level client-server communication, and also about agentic models and how tools, model context, and data fetching and preparation work. I'm getting closer fluent in the NovaAct SDK, and realised there are so many features that I'm yet to explore further, and would love to continue working on, possibly through another app that serves NovaAct for the average user, maybe as a supplement to an agentic system like OpenClaw.

What's next for NovaTest

Next I want to implement a multi-agent process where the act run provides context to another agent which locates the cause of the error in the repository and makes a PR suggesting fixes for the faults. I would also like to do more model fine tuning, e.g. provide error cases and fixes so the agent is more familiar with what to report. Finally, I would like to implement my CI system, which would require creating npm commands to launch agents. These changes would allow the flow to run autonomously on GitHub actions.

Built With

• amazon-web-services
• nextjs
• python
• supabase
• typescript