IssueMatcha

IssueMatcha Showing strong and good matches to a user interested in AI with computer vision

Inspiration

Our initial explorations in the tech world began with contributing to open-source repositories a few years ago. We spent a lot of time scouring for relevant repositories, joining IRC chat rooms, and pinging maintainers to understand what would be a good first issue to solve. This is a time-consuming process for both maintainers and new contributors.

Many open-source projects are in dire need of more contributions from the community. Similarly, we see many folks who are interested in contributing to open-source projects but don't know a good starting point. Finding repositories and issues aligned with a user's interest and skills is often difficult when the user doesn't have a good understanding of the codebase. With the amazing capabilities of LLMs to work on such high-level concepts of preference, we thought personalized GitHub issue discovery could finally be achievable.

We want to match open-source enthusiasts with relevant open issues on GitHub using our LLM-powered tool, IssueMatcha.

What it does

IssueMatcha recommends open Github issues to anyone interested in contributing to open-source projects. We use the preferences and interests of the user along with their previous contributions (if any) to match them to the relevant open issues.

How we built it

We scrape Github Issues, Issue body, Issue comments, and Readme files of repositories and use this information to generate tags for each Issue and a summary of the repository by prompting the Mistral models.
We set up a vector database with this information to help with the candidate generation step. We use weaviate as our vector db. Since the LLM does a good job at understanding issue titles and body, we use the vector db primarily to limit our search space. We filter a set of candidates using semantic search against generated tags.
We prompt the Mistral-large model with the user preferences and list of retrieved issues from our vector database. We ask it to semantically label each retrieved issue based on how relevant it is to the user preferences. We use this to filter and rank the final set of recommendations for the user.
We have a Streamlit app where a user can enter their preferences and they can view the recommended Github issues along with an explanation for why we think this is a good issue for them to solve.

Challenges we ran into

Figuring out how to categorize issues : How should we label and tag the issues that we fetch from Github.
Prompt engineering: Making the model adhere to the output requirements, while at the same time ensuring that it generates labels while adhering
Handling big context windows: Github issue bodies and comments can often run very long, so we did multi-step prompting where we summarized the information and sent it to the next prompt.
Hallucinations in recommendations: We had to specify specific steps the model had to follow to decide whether it should recommend an issue or not.

Accomplishments that we're proud of

Worked around the token limit by summarizing comments separately
Good recommendations!
Providing value to open-source community !

What we learned

Hallucinations can be largely worked around by careful prompting and providing options to choose from The traditional IR retrieval + ranking pipeline can be replaced with LLMs in certain situations