Inspiration
Green tests can lie. A test suite can pass while it checks almost nothing, and a pull request with a green build can still carry a bug no test would ever notice. I built FROM WITHIN to prove what tests really catch. It breaks Python code on purpose, one small change at a time, and runs the tests after each change. Every change the tests do not notice is a hole. FROM WITHIN Gate brings that proof into the place where code is accepted, the Bitbucket pull request.
What it does
- A pull request panel shows the FROM WITHIN verdict for the exact source commit of the pull request. It lists the counts and every change no test noticed, with file, line, the code before and after, and the attack id.
- A custom merge check, FROM WITHIN verdict is SALE-READY, passes only for a SALE-READY report of that exact commit, with 0 changes not caught and the original files untouched. A new commit needs a new report. On Bitbucket Premium the check can be Required and then a failing check stops the merge.
- A repository settings page gives the web trigger URL and a token for the repository. Only the sha256 of the token is stored.
- The CI runs FROM WITHIN and sends a short summary to the app. The app checks the token and every field before it stores anything.
- A built-in example shows a real report without any setup.
How I built it
- Forge with UI Kit for the panel and the settings page, a Bitbucket merge check module with the on-merge trigger, a static web trigger and the Forge key value store.
- The rules live in one small module with unit tests. Tokens, storage keys, input checks and the merge decision are tested on their own.
- Key facts were measured on the live install. The settings page context was read from the Forge logs before the check that guards the token was written.
- The web trigger is static, so its replies are fixed texts in the manifest. forge eligibility reports the app as eligible for Runs on Atlassian.
- The demo pull request uses FROM WITHIN 1.1.0 for real. Three weak tests give NOT READY with 3 changes not caught. Two more tests give SALE-READY with 20 attacks caught and 0 missed.
Challenges I ran into
- The repository settings page does not send a moduleKey in its resolver context. The live logs showed extension.type instead, and the check was rebuilt on that measured value.
- A dynamic web trigger blocked the Runs on Atlassian badge. Rebuilding it as a static trigger with fixed outputs made the app eligible.
- On the free Bitbucket plan a custom merge check is saved as Not required. The texts in the app now say exactly what Bitbucket does on each plan.
Accomplishments that I'm proud of
- FROM WITHIN attacked its own code before release. Version 1.1.0 is SALE-READY on itself, 3219 attacks caught and 0 not caught.
- The gate stores no user data, sends nothing out of Atlassian and trusts no field it did not check.
What I learned
Measure the live platform before writing a line that depends on it. The documentation and the running system did not always say the same thing, and the running system decides.
What's next for FROM WITHIN Gate
- A Rovo agent that explains each change no test noticed and proposes the test that would catch it.
- A Bitbucket Pipelines step that runs FROM WITHIN and sends the report in one line.
How it meets the main technology requirement
FROM WITHIN Gate is built on Forge and integrates with Bitbucket Cloud. It uses a Bitbucket pull request overview panel and a Bitbucket repository settings page, both in UI Kit, a Bitbucket custom merge check that runs on merge, the Bitbucket REST API to read the source commit of the pull request, a static Forge web trigger and the Forge key value store.
Runs on Atlassian
The app has no remotes, no external fetch permissions and no Connect modules. Its only web trigger is static, so it can only answer with the fixed replies in the manifest. All data is kept in Forge hosted storage. forge eligibility on the production version reports it eligible for the Runs on Atlassian program.