board2ticket

Ahaan Kanaujia ahaank2@illinois.edu Shiv Trivedi shivvt2@illinois.edu Aditya Kunte akunte2@illinois.edu

Inspiration

Working in Grainger Library, we often found ourselves sketching out ideas, debugging algorithms, and brainstorming solutions on whiteboards—only for those insights to be erased and lost forever. We wanted to build a tool that could preserve these discussions and seamlessly transform them into structured, actionable GitHub tickets.

What it does

board2ticket captures whiteboard discussions and automatically converts them into structured GitHub issues. It integrates computer vision algorithms to extract whiteboard content and natural language processing (NLP) to analyze spoken discussions, ensuring that valuable ideas don't get lost.

How we built it

We developed board2ticket using a combination of computer vision, audio processing, and NLP techniques:

Computer Vision: Used OpenCV to extract and cluster whiteboard content.
Audio Processing: Leveraged pydub for silence detection and OpenAI Whisper for transcription.
NLP & Semantic Analysis: Used OpenAI embeddings to cluster discussions and generate structured GitHub tickets.
Multimodal Integration: Combined audio and visual data through timestamp alignment to ensure contextual accuracy.

Challenges we ran into

Accurate Bounding Box Detection: Ensuring that whiteboard content was properly segmented and grouped.
Temporal Alignment: Matching whiteboard updates with corresponding spoken discussions.
Clustering Discussions: Determining how to effectively group related content into structured tickets.

Accomplishments that we're proud of

Successfully developed a system that automates ticket creation from whiteboard discussions.
Built a robust multimodal processing pipeline integrating audio and visual data.
Created an efficient content tracking algorithm that preserves whiteboard sessions even after they’re erased.

What we learned

More planning is crucial before implementing a complex multimodal system.
Fine-tuning clustering parameters (DBSCAN, K-means) significantly improves content grouping.
Speech-to-text accuracy is critical—choosing the right transcription model makes a big difference.

What's next for board2ticket

Live Whiteboard Capture: Implement real-time processing to track changes dynamically.
Better OCR & Handwriting Recognition: Improve text extraction accuracy from handwritten content.
Direct Integration with GitHub Repos: Enhance ticket creation by linking directly to relevant code files.
Collaboration Features: Allow multiple users to contribute to the same session and refine ticket details.