IssueOrb

Skip sprint planning meetings; IssueOrb triages automatically

Comment

About the Project

Inspiration

I spent my Friday afternoons doing the same thing every week: pulling up our GitLab project, scrolling through 150+ open issues, and manually applying labels, assigning them to teammates, and trying to spot duplicates. Then on Monday mornings, we'd spend 2 hours in sprint planning trying to guess how long everything would take. The worst part? Issues would arrive completely unlabeled and unassigned, piling up faster than we could triage them.

One sprint, we accidentally assigned the same feature to two different people. Another week, we missed an obvious duplicate that wasted a week of engineering time. That's when I realized: this shouldn't be a human job. The information needed to triage an issue — which component it touches, who knows that code best, how similar it is to existing issues, how long similar work took before — was all right there in our codebase and issue history. We just needed a way to query it intelligently.

That's when I discovered GitLab Orbit and Duo agents, and everything clicked.

What it does

IssueOrb is a GitLab Duo agent skill that automates the entire issue triage and sprint planning workflow:

Auto-triage (runs on every new issue):

  • Scans for duplicates using semantic similarity ($>85\%$ structural/textual match)
  • Walks the codebase knowledge graph to infer the right component labels
  • Finds the top contributor to affected components and ranks them by current workload
  • Estimates effort by aggregating historical time-to-close for similar issues
  • Posts a triage comment with all recommendations and applies labels, assignee, and story points

Sprint planning (runs every Friday at 17:00 UTC):

  • Scores all open triaged issues by priority weight + staleness
  • Uses a greedy assignment algorithm respecting per-person story-point capacity
  • Generates a "Sprint N Proposal" issue with full rationale and team visibility

The result: Monday sprint planning shrinks from 2 hours to 10 minutes of review-and-approve.

How we built it

Key technical decisions

Orbit integration: Rather than building custom similarity detection or expertise ranking, we leaned on Orbit's pre-built query types. The search query handles duplicate detection; traversal queries with BELONGS_TO and AUTHORED_BY edge types let us navigate the code-definition graph in milliseconds. This meant we could focus on orchestration rather than reinventing knowledge graphs.

Effort estimation: Instead of guessing, we aggregated historical data: for every new issue, we find all resolved issues with the same labels and compute the $p_{90}$ of their time-to-close. A simple bucketing function ($\leq 1\text{ day} = S$, $\leq 3\text{ days} = M$, etc.) gives us reliable estimates trained on the team's actual velocity.

Sprint assignment: We model this as a bounded knapsack problem. Each team member has a story-point capacity (e.g., 8 points/sprint). We score issues by priority label weight (critical $\times 4$, high $\times 2$, etc.) plus a staleness factor for issues waiting $>2$ weeks. Then we greedily assign issues in order until everyone hits their cap. This ensures balanced load and prioritizes urgent work.

Webhook + scheduler: The triage runs on every new issue via a GitLab webhook that triggers a CI pipeline. The sprint planner runs on a fixed schedule (Friday 17:00 UTC via .gitlab-ci.yml schedules). Both write back to GitLab as comments and issue updates, keeping everything in one place—no external dashboard to check.

Challenges we ran into

1. Orbit query performance at scale

Initially, our duplicate detection was too slow. Querying Orbit for semantic similarity against every open issue took 10–15 seconds. We fixed this by:

  • Caching Orbit query results for 1 hour (issues don't change that fast)
  • Limiting duplicate checks to issues opened in the last 30 days
  • Using Orbit's limit parameter to cap results early

Result: sub-1-second duplicate checks.

2. Handling edge cases in expert identification

If a component had only one author, or that author was on leave, we'd suggest someone with stale knowledge. We solved this by:

  • Ranking assignees by recency of contribution (last 90 days weighted higher)
  • Falling back to the next top contributor if the top choice already has $>10$ open MRs
  • Surfacing uncertainty in the triage comment ("low confidence" if $<3$ prior commits)

3. Effort estimation with sparse data

Some components had only 1–2 prior issues, making estimates unreliable. We implemented:

  • A minimum sample size (show estimate only if $\geq 3$ similar issues exist)
  • Fallback to team-wide median effort for new components
  • Confidence intervals (p50 and p90) so people know the range

4. GitLab API rate limits

Posting triage comments to 20+ issues in a sprint planning run would hit rate limits. We batched API calls and implemented exponential backoff with jitter.

Accomplishments that we're proud of

  • Sub-second triage comments — the full pipeline (duplicate search, label inference, assignee ranking, effort estimation, and comment posting) completes in under 1 second per issue.

  • Zero false-positive duplicates — by setting the similarity threshold at 85%, we have confidence in flagging genuine duplicates without overwhelming teams with noise.

  • Reproducible sprint proposals — because the algorithm is deterministic and logged, sprint proposals are auditable. The team can see exactly why an issue was assigned to someone and debate the tradeoffs.

  • Integration into Duo Chat — users can now type "Triage issue #88" or "Generate a sprint plan" directly into GitLab Duo Chat, making the tool frictionless.

What we learned

  1. Knowledge graphs are powerful — before Orbit, I thought we'd need to build a custom similarity detector and mine commit history manually. Orbit's pre-built semantic search and traversal queries saved weeks of work.

  2. The 80/20 rule applies to automation — we didn't need perfect effort estimates or duplicate detection. A 85% confidence threshold and historical p90 estimation cover most real-world needs and let humans make the final call.

  3. Context matters more than raw data — knowing who authored a piece of code (via Orbit edges) is more useful than a list of names. The context of a contributor's recent activity and workload is what makes a good assignment.

  4. Transparency builds trust — teams initially hesitated to trust automated assignments. Once we started showing the reasoning in triage comments, adoption skyrocketed. People want to understand why.

What's next for IssueOrb

  • Machine learning for priority inference — currently, priority is manual. We could train a classifier on past issue resolutions to auto-detect critical vs. low-priority based on description, labels, and past severity.

  • Burndown predictions — integrate with MR merge time data to predict sprint completion and flag risks early.

  • Slack/Teams integration — post triage summaries and sprint proposals to team channels for async review, not just GitLab issues.

  • Custom scoring rules — let teams define their own priority weights and capacity models via a config file.

  • Multi-project support — extend the skill to handle cross-project dependencies and shared backlogs.

Built With

  • gitlab
  • gitlab-ci/cd
  • gitlab-duo
  • gitlab-orbit
  • gitlab-rest-api
  • orbit-rest-api
  • python
Share this project:

Updates