Git Integration for Jira: Private Servers

Inspiration

Back in 2016, we launched Git Integration for Jira Cloud to connect a variety of git servers to Jira Cloud. Customers configure “integrations” with the git server url, credentials and some settings and our indexer is then able to connect to the git server to index the commits/branches/pull requests. One concern that we had (in 2016) was that the market for a Jira Cloud app that would connect to locally hosted git servers would either be small or admins would not approve of such connections. Fortunately, a sizeable number of Cloud customers have found great utility in our offering.

But there are two groups that we have not been able to serve yet:

Customers hosting their git server on a private network
Customers who cannot (for privacy, security, and other reasons) grant access to source code directly.

We have had almost 100 separate customers ask us for some kind of work around. We expect the total market to be considerably larger.

Our solution: Git Integration: Private Servers.

What it does

Building on our existing product, Git Integration: Private Servers, we have built the ability to configure webhooks on the git server to send basic commit, branch and pull request information that our application can index.

When configuring a new Private Server integration - the Jira admin obtains a unique and secret url to be used in configuring webhooks on the git server. The git server will begin transmitting webhooks to the unique URL where our indexer can index the webhooks.

The two advantages of indexing webhooks in this fashion are:

Many more self-hosted git servers will be able to send development information to Jira Cloud.
Webhooks sent by the major git hosting providers do not contain source code.

Challenges we ran into

We had two main challenges to overcome In creating the Private Servers edition of Git Integration for Jira: Durability and Completeness.

When a webhook is sent to us by a customer’s git server, the webhook is sent once and only once. If our application is down for any reason (maintenance, code updates, too high load, etc) then that webhook is lost forever.

This problem is compounded by the Completeness problem. Webhooks are only ever sent for current activity - you cannot send webhooks for past activity. This means that our application (unlike our current feature set) cannot request missed activity.

This means we need to have virtually 100% uptime as we aim to reach enterprise customers in addition to our traditional SMB customers.

How we built it

We already host Git Integration for Jira Cloud on Amazon Web Services and to solve the challenges detailed above we are employing several special AWS services.

When a Private Server integration is created we generate a unique/secret url recognized by the AWS API Gateway. The webhook is then is placed in an AWS Simple Queue Service (SQS). By relying on these industrial strength AWS services, we are employing highly-redundant and performant services to capture the webhook traffic sent to us.

Our application server can then collect the messages from SQS and dispatch them to our indexing service where they are delivered to the customer’s data store. The indexer extracts and stores Git information from the webhook payload such as: commits, authors, file names, branches, pull requests and so forth. With this information - we display commit, branch and pull request information in the Jira Cloud UI for our app.

Additionally - we upload this information directly to Jira Cloud using the Development Information API. By uploading this information to Jira Cloud, our customers can take advantage of such powerful features such as JQL searching, workflow triggers, Release Hub, and other features offered by Atlassian.

Accomplishments that we're proud of

We’re proud that we’ve built an application that can process high webhook loads with enterprise quality uptime while giving us the flexibility to update the application in rapid fashion. We’re also very excited to reach out to a whole new class of customer that has not been able to use Jira Cloud for tracking development activity without this feature set.

What we learned

Git servers send the information that they send and sometimes the information is different between git servers. Additionally - git servers will truncate the activity sent if there are large changes. We will have to be excellent at educating our customers on the limitations of this type of integration.

We will be building out some features for Jira admins to troubleshoot issues on their own without having to reach out to us.

We also will need to educate our users on the difference of the two types of integrations: Private Servers (using webhooks) or traditional integrations (using APIs) as some features are not available to Private Server integrations since they do not provide source code at the indexing stage.

What's next for Git Integration for Jira: Private Servers

Currently the Git Integration: Private Servers app in the scope of our Codegeist submission is only supporting GitHub.com repositories. We have extensive experience supporting GitHub Enterprise, GitLab, AWS CodeCommit, all Microsoft git based servers and more and will be rolling out support for their webhook payloads in the coming weeks. We will also be setting up various customer demos to show off our progress and gather feedback on prioritizing features.

We hope to launch publicly on the Atlassian Marketplace in the next month.

Known issues with Codegeist submission

Only GitHub.com repositories supported
Not all features of our current app are supported (some due to data missing in webhooks and others will be finished in coming weeks)