Team104_1J_Smart Cookbook System

Inspiration

We observe that many students at Nanyang Technological University (NTU) crave delicious food but they tend to have little experience in cooking. even if some of them are willing to learn, they have little time outside of their various commitments. So we want to streamline the learning process by providing real-time feedback, leveraging on the trending LLM technologies

What it does

Our project is a retrieval-augmented generation (RAG) system, specifically targeted towards cooking recipes from PDF documents. It aims to teach people to cook by providing them with a detailed recipe on dish preparation along with a picture of the desired final product. In short, it is an interactive AI agent with image generation capabilities on top of simple conversation exchanges via texts

How we built it

The program trains GPT-4 with our customised data (a cookbook found online), employing common Natural language Processing (NLP) techniques such as vector embedding and document splitting to extract the relevant content from the training data in response to the user’s questions. We also employed chromadb, a database to store vector values and initialising them. This part is crucial for creating a searchable database of embeddings that can be used for similarity searches and other retrieval tasks. The text and image data will be saved locally in the form of a pdf file.

Challenges we ran into

We have little experience in image-generating algorithms and saving outputs in a pdf to store it locally, so we spent a lot of time figuring out how to accomplish the above tasks

Accomplishments that we're proud of

We are elated that the system is able to produce clear, relevant, and contextualised answers that can meet the needs of students

What we learned

We learned how to build an interactive AI chatbot using ChatGPT API, as well as training it with our data. We also learned to save our Python program's output in a pdf and store it locally. More importantly, we have learned effective teamwork and communication skills that are essential to complete a complex project.

What's next for Smart Cookbook System

Tuning the algorithm so that it can accept different types of user prompts, including very specific ones. Increase the size of the training dataset to make the responses more comprehensive