ExSAMine

Inspiration

When researching the impact of digital forensics in the real world, we came across a high-impact case in Indiana in which a suspect was implicated because the murder weapon and his bracelet were found in a Facebook photo. The process of finding and matching that photo to evidence took 2 years. Given the advancement in AI models recently, we aim to use computer vision and segmentation (SAM3 model) to give forensic analysts a fast and easy way to extract, contextualize, and generate detections from photo and video evidence, helping them quickly process case data to match it to leads.

What it does

A fully agentic AI platform that segments and generates detections from case data (images and videos). In the demo video, the user creates a new case and adds case context and data as well as photo and video evidence pertinent to the case. Then the user asks to analyze the photos and the Gemini API takes case context. Gemini then returns the segmented images, using the provided context to generate further leads that can be pinned and explored later by the analyst.

How we built it

Used NextJS and TypeScript for frontend, with a Python and FastAPI backend. We used modal to run the SAM3 model and perform the inference, and used the Gemini API to chat with 3.0 and use its multi-modal capabilities to the best.

Challenges we ran into

Given we're working with 2 different AI models, sending visual data through requests was complicated, and sometimes slow. Also, setting up the actual API keys was quite difficult at first.

Accomplishments that we're proud of

We properly got both Gemini and SAM to stream good responses, prompt properly, but most proud of being able to provide such complex context to these models via just simple markdown files.

What we learned

We learned how to run virtual gpu containers with AI Models and I personally learned how to use the python Fast API along with the Google Gemini API. I also learned about the capabilities of SAM 3,