Project Story: Breaking Down Barriers (Knowron)
Inspiration
- We chose this particular challenge because we could relate to the problem we were trying to solve, making it an engaging and meaningful project for us.
- Our inspiration for using technologies such as Azure services and Streamlit came from attending deep dive sessions, which introduced us to their potential benefits and use cases.
What We Learned
- Throughout the project, we learned the importance of teamwork and how to effectively divide tasks among team members.
- We also gained valuable experience in learning to work with and integrate new technologies in a short amount of time.
Implementation Steps
Frontend
- Implement the user interface using Streamlit for rapid prototyping and easy deployment
- Allow users to input text, voice memos, and images
Image Dense Captioning
- Use Azure Computer Vision API for extracting dense captions from images
- Process the captions and prepare them for integration into the final report
Speech-to-Text & Translation
- Convert voice memos into text using Azure Speech-to-Text
- Translate the extracted text and user-entered text into English using Azure Translator
Information Extraction
- Use Azure's Natural Language Generator AI for information extraction
- Fill a general task report template with the extracted information
Report Generation
- Generate the final report as a PDF document
- Ensure the report has sections and clear descriptions in English.
Challenges Faced
- While Streamlit saved us a significant amount of development time, we encountered difficulties in customizing the framework due to its inflexibility.
- During the testing phase, we found that Azure's language processing capabilities were not up to our expectations, delivering inaccurate and incorrect results. This prompted us to explore alternative solutions for better performance.
Log in or sign up for Devpost to join the conversation.