NetOne Business Document LLM

Inspiration

When we built the RAG system, we find the information on the diagrams was lost. At the same time, the open-source Text to image model was released, which led us to come up with this system.

What it does

The Text to image feature allows diagrams to be converted into text and made searchable in RAG.

How we built it

We choose to build the solution on Databricks platform, and this solution is mainly divided into 2 parts in development progress.

The first part is the Retrieve Data and Convert to Text part, which is the Image-to-Text generation part. In this part, we choose the llava-v1.5-7b model for the image-to-text generation. The second part is the User Prompt and Query part, which is the part mainly faces to the end users, and therefore we built a chatbot UI with Gradio in Databricks Notebook as for the demonstration. In this part, it mainly supported by Vector Indexing DB with Langchain, together with the llama-2-7b model as the LLM in this solution.

Challenges we ran into

For the devolpment, the first challenge was to choose the appropriate model for developing the solution we designed. Luckily in these days, there are many open source models on huggingface platform that can be used on development, and we finally found the suitable models for our solution at last.

On the top of that, it was the first time for us developing a solution on Databricks platform with MLflow, and we are also new to the model serving endpoint provided. It does take some time to go through those tutorials and documentations, as well as the time for try-and-error to learn how to make request to the endpoint and get response successfully.