Allure Extension for VSTS

Automated tests reporting made for humans

Comment 1

Inspiration

There are lots of cool testing frameworks for different programming languages. Unfortunately only few of them can provide good representation of test execution output. Allure is an open-source framework designed to create test execution reports clear to everyone in the team.

Our team members at Molecula have already used Allure framework while working on the other projects in the past. We used it with CI tools like Jenkins and Team City which are very popular in Java world. When we started working with Team Services we realized that such a great tool was still missing here. That's why we developed Allure Extension for VSTS which brings all the power of Allure framework to Visual Studio Online.

Why we like Allure

Reporting plays one of the most important roles in automation testing. If the author of the automated tests code could often understand the results of his tests without any report, for other team members it could be difficult. But this understanding is very valuable for all team members, even for managers.

In order to make the tests relevant it is required to be able to know when there's a failure and what caused it. Report must have human readable format that can help to diagnose test failures quickly. For example, report may contain not only the stacktraces of the code, but also the screenshots, video or the parts of the log file. It is also very useful (especially for the team managers), when you can inspect tests results grouped by features.

The general approach is to create an HTML report with test results and failure screenshots intertwined. But this is easier said than done since there is no proven standard - leaving us to code our own approach based on the test framework we are using.

By leveraging a language agnostic open-source library like the Allure Framework, we can easily generate a robust, human-readable HTML report from each of our test runs.

Please, check following video if you would like to know more about Allure Framework: https://www.youtube.com/watch?v=xrlS-7mLm0E

How it works

How Allure works

To use Allure, it is required to gather the information about tests and then to generate a report. Gathering information about tests is performed during the automated tests execution. This information is saved to XML files with a predefined format. For this purpose, Allure provides so-called adapters - small libraries that are attached to particular test frameworks and know how to extract test information to XML. Allure already supports adapters for the most of popular languages and test frameworks.

How Allure works 2

Allure report generation is performed using one of the available report building tools. In basic scenario there is a command line tool to generate the Allure report by running a console command. There are also the plugins for some popular build systems and tools, such as Jenkins, TeamCity, Bamboo, Maven, Gradle, etc, that allow to add the test report generation as an additional build step.

Allure report supports some additional features, such as grouping the tests by features, files attachments, issues and test management system links, assertions over a set of custom steps distribution, etc. All these features are available in Allure core, but their usage may differ from one Allure adapter to another.

How it works in Team Services

In order to bring Allure to Team Services we created an extension which uses few extension points of Team Services:

  1. Custom Build Task Extension for report generation
  2. Custom Action for completed Builds to allow to view generated report
  3. Custom Settings hub in Project settings

You can see how it works on the video and screenshots above.

Challenges we ran into

1. The biggest challenge was to figure out how to store the report. For instance in Jenkins there is an option to save static files generated during the build and open them later in the browser using special link - that is exactly what we needed. Unfortunately, there is no such an option in Team Services, so we had to find another way. In the end we decided to move this out of scope of the extension and now you would need to set up your own website to store allure reports. We have added a new suggestion on UserVoice and if the API evolves we may change this behaviour in the future.

2. There is another strange problem with the build definition editing. Sometime if you add a custom build task installed with the third party extension (Allure for instance), save the build and open the edit form again after some time the custom task would just disappear. Unfortunately, we can not figure out how to reproduce it and it happens occasionally.

3. There is very serious limitation in the current version of Team Services API for custom tabs. At the moment if you implement ms.vss-web.action with the target = ms.vss-build-web.completed-build-menu Team Services will pass the context about the selected build to the action handler. Using the context you can get all the required information about the build like BuildNumber, etc. Unfortunately, it is not the case for ms.vss-build-web.build-results-tab with the target = ms.vss-build-web.build-results-view. We have also submitted this case as a suggestion for improvement on Team Services User Voice.

Built With

Share this project:

Updates