Flaky Test Detector and Mitigator

In the world of QA automation, flaky tests—those that unpredictably pass or fail without code changes—are a persistent headache. They erode trust in test suites, waste time on false failures, and slow down CI/CD pipelines. For the Kyiv QA Automation Summit 2025 Hackathon, we present FlakeBuster, an innovative tool designed to detect and mitigate flaky tests, aligning with the theme "Smarter Testing for a Smarter Future." FlakeBuster tackles a critical pain point: identifying and resolving test flakiness efficiently. Built in Python, it integrates seamlessly with CI/CD systems like Jenkins or GitHub Actions, pulling test execution logs to analyze historical data. Using metrics such as execution time variability, failure frequency, and environment-specific patterns, FlakeBuster employs a lightweight Random Forest Classifier to predict which tests are flaky. Beyond detection, it goes a step further by suggesting actionable fixes—whether it’s adding retries for unstable timing, isolating environment issues, or investigating race conditions. The tool’s core strength lies in its practicality. QA engineers can run FlakeBuster as a standalone script or embed it into their CI/CD workflows, receiving a concise JSON report that highlights flaky tests and mitigation strategies. For example, a test with high time variability might get a recommendation to "add retry logic," while one with intermittent failures prompts a deeper look into concurrency issues. This empowers teams to act quickly, reducing debugging time and improving test reliability. FlakeBuster’s innovation stems from its blend of automation and intelligence. While traditional approaches rely on manual log reviews or basic heuristics, our solution leverages machine learning to uncover subtle flakiness patterns, making it scalable and adaptable to diverse test suites. It’s also extensible—future iterations could incorporate real-time monitoring or auto-apply fixes via code generation. For real-world adoption, FlakeBuster requires minimal setup (just a log source and Python dependencies), supports popular testing frameworks like pytest, and provides clear documentation. Its impact is immediate: faster pipelines, fewer false positives, and more confidence in automated testing. By addressing a universal QA challenge with a smart, data-driven approach, FlakeBuster paves the way for more reliable software delivery in today’s fast-paced development landscape.

Tech Stack Language: Python Libraries:

• pandas for data analysis
• scikit-learn for ML model
• pytest for test execution and log collection
• requests for CI/CD integration (e.g., fetching logs from Jenkins) CI/CD Integration: Compatible with Jenkins or GitHub Actions via API Output: JSON report and actionable suggestions

Directory Structure

FlakeBuster/
├── README.md
├── requirements.txt
├── src/
│   ├── flake_detector.py  # Core logic for flakiness detection
│   ├── data_collector.py  # Fetches and processes test logs
│   ├── mitigator.py       # Suggests fixes for flaky tests
│   └── utils.py           # Helper functions
├── tests/
│   ├── sample_tests.py    # Sample test suite with flaky tests
│   └── test_logs.json     # Sample historical test data
└── run_flakebuster.py     # Entry point script

Built With

• pandas
• pytest
• requests
• scikit-learn