We're excited to introduce our project, Tio SAM, developed for HackUPC 2023. 🎉 Please keep in mind that due to the rapid nature of hackathons, there may be areas that could benefit from further refinement.
"Tio SAM" employs the powerful "Segment Anything Model" (SAM) from Meta and a comprehensive dataset from restbai. These tools have facilitated the segmentation of thousands of images, revealing intricate structures within each one.For added sophistication, we integrated BLIP, an advanced image captioning model, into our project. This, coupled with GPT-2, allowed us to generate relevant captions for each segmented image.To further enrich our image descriptions, we also utilized the captions provided by the restbai API. 📊
Taking into account the unique types of input images supplied by restbai, we've incorporated ControlNet into our system. ControlNet, a neural network structure, guides diffusion models by introducing additional conditions. This feature empowers our model to generate innovative interior design improvement ideas, suggesting aesthetic enhancements while preserving the existing structural integrity of a house. 🏠
Built With
- blip
- controlnet
- gpt-2
- pytorch
- restb.ai
- sam
Log in or sign up for Devpost to join the conversation.