Inspiration
During my six-month internship at Hong Leong Bank as an Athena Regression Tester, where I also took on responsibilities as the senior intern overseeing the launch of a new app, I gained a deep understanding of UAT testing and the importance of flagging bugs before release. With hundreds of functional test cases to cover and multiple test accounts spread across different app interfaces, it quickly became clear that a team of just ten testers could never complete all the scripts within a week, much less on a daily basis. That's when I started to research on Automated test tools. This challenge re-sparked my interest in exploring automated testing tools, as I realized automation was the only way to achieve both speed and thoroughness at scale.
What it does
UATest is a TikTok hybrid UI testing tool that blends AI and robotic automation. It executes quickly recorded test flows while dynamically adjusting with MLLMs in response to unforeseen popups, delays, or crashes.
How we built it
For fast, scriptable browser control, we build it using the library from pyautogui. We also use it for perception/ UI state capture by taking screenshots for test success rate and analysis purposes. For MLLM, we use GPT-4o-mini to analyse image and for text grounding. We also use it for LLM Reasoning such as few-shot prompts for "what to do next, what's the reason it failed". Lastly, we use simple python dictionary for storing seen UI states and actions.
Challenges we ran into
I think that balancing between speed and intelligence was one of our toughest obstacles. The tests took longer when AI inference was run at each stage, but if it was skipped, the system was not always able to handle unforeseen problems like blank screens or overlapping text. In order to address this, we devised a method for creating a trustworthy "state signature" that would enable us to determine with certainty when the application was on the correct screen. This allowed us to be confident in the accuracy and effectiveness of our tests.
Accomplishments that we're proud of
We designed a self-healing test framework that could intelligently retry, undo, or skip steps instead of failing outright. Using a combination of perceptual hashing and OCR, we developed consistency checks to strengthen it and make sure the system could consistently identify screens and content. These enhancements made the entire process much more resilient by enabling our functional tests to adapt, recover, and continue operating after the initial error.
What's next for UATest
One objective is to make the system more adaptable and widely applicable by adding support for other dynamic apps in addition to TikTok. Simultaneously, intelligent caching and lightweight LLMs can be used to optimise inference, maintaining flexibility while greatly increasing speed. Adding an analytics dashboard that makes it simple for developers to monitor and examine tickets that have been marked as "crucial" is another important step that gives them a clear picture of the most urgent problems. Last but not least, investigating parallel test execution will be crucial for scalability, allowing larger test suites to operate more quickly and effectively without compromising reliability.
Log in or sign up for Devpost to join the conversation.