Deep Learning Microscopy For Dummies

Bios:

J. Langley - Founder of HSV.ai
Benjamin Etheredge - AI / ML Expert
Geoffrey Sizemore - Local AI enthusiast, aspiring Deep Learning student

Inspiration / Background

Team is interested in fostering the AI knowledge community of the Tennessee Valley and Huntsville. This challenge was the most open-ended challenge and the team members are more software engineers than Bioinformatics experts. We appreciate open source tools and like to learn new things.

LLM

J really wanted to push the boundaries of generative AI prompts in order to try and complete this challenge, if you review the github repo and specifically the Perplexity prompt markdown page, you'll see the depth and breadth of knowledge that the team sought to retrieve from the Large Language Model ChatGPT4.

What our solution does

Leverages open source libraries built around Python and CLIP. Demonstrates a very rough prototype example where the BiomedCLIP model was retuned on some DIBAS data.

How we built it

Sourced information from the internet and then did a proof of concept demonstration using both a server-based instance and multiple laptop instances of different LVMs.

An excursion was done into leveraging Llama-3.2-vision for simple image recognition but the team ran into issues accelerating the model with GPUs locally. We can demonstrate a brief example of that model running locally but the speed of the model is too slow to use in practice. A follow-on effort could be done to take advantage of server-based GPU models and Llama-3.2-vision.

Challenges we ran into

Tuning the model with new data wasn't a trivial exercise. The generative AI prompts relied on very explicit directions. As an example, there are two variants of CLIP (clip vs open-clip) and the LLM responses were often a mixture of functionality from those two python libraries.

Accomplishments that we're proud of

Doing most of the work almost exclusively based on generative AI prompts. A future goal might be to take a system like this and put it on a smaller form factor like the Raspberry Pi so that parts of the world without access to high performance computing assets can reap the benefits of the new technology.