Readme-png

Inspiration

The idea for Readme-PNG was born from the growing demand for accessible and multilingual product information in the global market. As more consumers around the world, particularly those interested in Korean beauty products and K-pop merchandise, encounter language barriers, it became clear that there was a need for a solution that could bridge this gap. Moreover, making detailed product information accessible for visually impaired users through TTS was another crucial aspect that drove us to create this service.

What it does

Readme-PNG allows users to upload PNG images of product detail pages (e.g., cosmetics) and uses OCR to extract text from these images. This extracted text is then translated into multiple languages using OpenAI's GPT API and converted into a clean, easy-to-read Markdown format. Additionally, the service generates audio files via TTS for users to listen to the content. The platform also includes an AI-powered chatbot to answer any questions about the product, using the translated text as its knowledge base.

How we built it

We used Streamlit as the primary platform for the user interface, providing a smooth and intuitive user experience. Tesseract or Google Vision API was employed for OCR to accurately extract text from the uploaded images. OpenAI GPT API handles the translation of the extracted text into multiple languages, and the content is then formatted in Markdown. For accessibility, we integrated Google TTS API to convert translated text into audio files. The chatbot functionality was built using OpenAI's API to create a personalized assistant that can answer product-specific questions.

Challenges we ran into

One of the major challenges we faced was ensuring the OCR worked well across multiple languages and fonts, especially with product detail pages that often contain various fonts and layouts. We also had to fine-tune the chatbot's responses to ensure it provided concise and relevant answers without overwhelming the user. Ensuring the quality of translations, especially in more complex languages, also required extensive testing and refinement.

Accomplishments that we're proud of

We’re proud of successfully creating an integrated system that makes product details more accessible to a global audience while also providing support for visually impaired users. The seamless combination of OCR, translation, Markdown formatting, TTS, and AI chatbot functionalities into one streamlined service is a key accomplishment. Additionally, building an intuitive user interface that allows users to upload, translate, and interact with the platform easily was a big win for our team.

What we learned

Through this project, we learned a great deal about the complexities of OCR across different languages and the importance of ensuring accurate translation when dealing with product-specific terminology. We also gained insights into building user-friendly, accessible features such as TTS and chatbot functionalities that could assist users in real-time.

What's next for Readme-PNG

The next step for Readme-PNG includes expanding our support for more complex product pages, including those with dynamic content or interactive elements. We also plan to enhance the chatbot functionality to make it more conversational and versatile, as well as improve the customization options for TTS to cater to different user preferences. Integrating with more e-commerce platforms and expanding our translation language options are also on the roadmap, making Readme-PNG a truly global solution.

Built With

• gpt
• gtts
• openai
• pil
• python
• streamlit