Ben - The ChatBot

Output for a simple question
Ben is typing - the model is processing
Output showing the Conversation

Inspiration

Serve as virtual assistants, making them a compelling tool for businesses and individuals looking to leverage technology for improved interactions, customer service, and information dissemination.

What it does

Streamline communication, enhance user experiences, and provide efficient solutions in various domains using a conversational approach

How we built it

We built this chatbot using a pretrained Llama 2 model and streamlit to deploy it locally. Llama 2 is a collection of pretrained and fine-tuned generative text models ranging in scale from 7 billion to 70 billion parameters. This chatbot is built using the 7B pretrained model

Challenges we ran into

We initially tried deploying a 13b and 70b parameter Llama 2 model. Given that we are using a Local PC, we did not have enough memory to run the huge models. Later, we tried using a 7b parameter trained model which is using GGML version. GGML is a machine learning library designed to handle large models and deliver high performance on standard hardware. It uses a quantized representation of model weights, which essentially means it uses ‘approximated’ parameters as opposed to the full version, resulting in slightly less accuracy. However, the trade-off is a model that requires 4x less RAM, 4x less RAM bandwidth, and offers faster inference on the CPU. This makes GGML an ideal starting point for most local machines, particularly those not equipped with GPUs for machine learning or with limited RAM.

Accomplishments that we're proud of

We are proud of setting up local environment and run the Llama2 model locally and develop a decent user interface to interact with the model using Streamlit.

What we learned

-Deploy Streamlit application locally -Use Large Language Models to develop a chatbot application -Concept of LLM, Deep Learning and Natural Language Processing