knowron_reporTUM

Logo

Inspiration

We were inspired by the opportunity of being able to help many technicians that have moved countries for better employement offers and that were struggling due to the language.

What it does

How we built it

We based our whole stack on python and related libraries. We used streamlit to build the front-end and the main app logic. We used azure services for handwriting recognition and automatic language detection and translation. We used the popular Whisper API to transcribe text from audio recordings. For the natural language processing tasks such as summarization and text embedding we used the freely available Cohere.ai API keys and models for inference. We used standard libraries like pdfkit to generate the pdf report based on our custom HTML report template and used SQLAlchemy for the database backend.

Challenges we ran into

Connecting the whole pipeline was a little challenging because there are a lot of moving parts. Also dealing with large language models (LLMs) requires patience. Having only high-level access to the models and not being able to fine-tune them with examples forced us to tune several hyperparameters and try out different prompts for the summarization models.

The LLMs are really finnicky and their outputs are not very robust to the input text given to summarize.

Accomplishments that we're proud of

We are proud to have brought all the different moving parts together and create an app.

What we learned

We learnt how sensitive LLMs are to the inputs and the prompts. We also learnt valuable skills in app development as our main experience was based in deep learning and machine learning.