Methoda AIR Check

Generate Gherkin Scenarios
Generate Test Steps

Inspiration

We have been an Atlassian partner for over 10 years. We have assisted thousands of customers in realising their vision for an agile, high-quality, and sustainable software development cycle. All of Methoda's activities are centred on supporting our clients' core business processes, helping them emerge as industry leaders.

Methoda sees the latest innovations in artificial intelligence as an opportunity to accelerate the realisation of our customers' visions.

We aim to automate and optimise the software development process, to increased efficiency, improved test coverage, reduced errors, and ultimately contributing to the overall quality and success of software development projects.

What it does

AIR Check stands as an indispensable tool as part of our Methoda AIR Platform, leveraging the magic of automation to create manual test scripts seamlessly. This intelligent tool empowers users to effortlessly generate a test script according to their issue description. The users can choose to generate the test script in a standard format or opting for the elegant precision of Gherkin scenarios.

How we built it

The forge application collects the context from the Jira issue, including summary, description and linked issues. The data is passed to a chain of interactions with various Open AI models to infer the testable aspects of the Jira issue, and then create a test flow for it.

Our backend is based on an implementation built around Langchain. We have managed to design a chain of prompts that is efficient in token consumption while being able to integrate user inputs. In addition, a series of unique prompts was designed to generate test scenarios in both Gerkin and Standard formats. Generating tests required multiple prompts to implement the logic of a Test script creation, to cover a feature or a system capability.

The Forge application verifies the user identity and provides the client with a token for accessing our backend. We create and sign a JWT token in the Forge function (including the installment ID, Atlassian user ID and license validity), pass it back to the Custom-UI element, and use it to authenticate with our backend platform.

Challenges we ran into

During the development of our app, we encountered several challenges that required creative solutions.

Firstly, Forge does not support many standard Node.js interfaces, thus incompatible with the library we choose (Langchain). We therefore had to rely on our existing implementation of the backend on another platform, rather than port it to Forge.

Also, Forge currently does not support authenticating with 3rd party backends. Since our backend runs on another platform, We had to implement our own version of authentication to verify the validity of the API calls.

Moreover, time constraints were a constant pressure, demanding efficiency and adaptability in our development process. This requires careful prompt engineering to minimize token generation by the AI model, and running some of the inference flows in parallel.

Accomplishments that we're proud of

We are proud of our ability to prove test scenarios and scripts can be generated by AI.
We have taken a great step in the understanding of the ecosystem around AI and large language models
We have managed to build an efficient team of developers working together in tight schedules
We have built a nice UX/UI inside Forge. Just look at it.

What we learned

Expanding our knowledge in Forge!
Langchain model of building LLM-based applications
Prompt engineering
While developing features addressing BDD, we were able to understand the great potential of this methodology and plan to implement it for our day-to-day workflow as an agile team
Asymmetric JWT authentication