A(I)yushi - Your Nutrition and Dietary Health AI Assistant

Know-Before-You-Buy & Know-Before-you-Eat. "Aiyushi", your personalized Nutrition and Diet AI assistant, empowers you to make the right decisions for health and longevity.

Comment

Inspiration

Firstly, I am personally big on nutrition, diet and strive for a disease free today and tomorrow for myself and for my family. I read nutrition labels and ingredients before buying food products. I do not buy heavily processed food and cook at home as well. While food is considered a basic need, health which has a direct influence on what you buy and consume, is put to the side often. Health and health care is a huge issue in the US. There's lot of information on health-care crisis but instead of complaining about health-care we should focus on personal health so we dont have to depend on hospitals and insurance. Secondly, there is a wealth of information on nutrition and diet from US government agencies the USDA, FDA, CDC, NIH, etc. This was an eye opener to me. I can across this treasure at dietaryguidelines.gov, an USDA website which lays out guidelines for each life stage, gender, calorie profiles. Then I found the USDA database on pretty much all commercial food products containing every bit of information on the products. I am sure most Americans are not familiar with it. Some of it targeted towards health-care professionals. All this lead to this product idea to make this wealth of credible, authoritative well researched information available literally at the hands of the common person (as a mobile app). _ (Btw, I took permission of USDA by email as well to use the database and guidelines for this purpose)._

What it does

Most food consumption journey starts with buying a food product. Therefore it is important to first buy the right food. The application, uses information available on the food product itself to bring that wealth of information to the user. It can be the picture of nutrition label, the list of ingredients or the UPC barcode. Gemini-pro-1.6 multi-model LLM enabled automatic inference of the type of input image and analysis of the input as well. As UPC barcode provides all the details of the product, it helps with personalization of the analysis and recommendation, again based on credible and authoritative information I previously mentioned. The app takes Age and Gender as input for this personalized output. User can select from 'canned' analysis options or they can choose to chat with the application (again based on Gemini-pro-1.5). The model in this case has been already prompted with the personalized analysis and recommendation to provide contextual chat for the user. Please note: The chat assistant is not working consistently at the huggingface hosting space. It works better with the direct gradio link link but this link is available only for 72 hours. Its a gradio thing.

How we built it

The foundation of this application is the Gemini multi-model LLM (1.5-pro) which enables auto-detection of image type (used function calling feature) and further analysis. The model performed extremely well for this first step task. Lot of time was spent on to ensure this part worked to satisfaction. The next step, if image is a barcode, is extracting information from USDA database. Their API was used to retrieve information on the food product. The last step is nutritional and healthy (/unhealthy) aspects of the food product and producing recommendation on how much one can consume, what are the nutritional gaps. The detailed analysis option (input) produces dietary guidelines as well (this part is based on the dietaryguidelines.gov PDF document). Python was used for the backend. I am not a professional software developer; Gemini LLM was used again as python coding assistant Gradio was used for the frontend as the low-code option for quickly making this app available for demo. Gemini was useful for this part as well. Especially the model garden was useful. Huggingface spaces was used to host this gradio app. This was also a huge plus in bring the app out quickly. (I did not have time for productizing it as mobile app before the May 2 deadline.) BTW, the repo is available at the same huggingface hosting space - link

Challenges we ran into

  1. Main challenge was handling the data and using it for the output analysis, be it the PDF document with its plethora of table formatted information or the data retrieved from the USDA database. I had to extract the tables as CSVs. LLMs, in general, are limited by its capacity in analyzing CSVs so it was a challenge in continues to be a challenge. Full transparency, getting the chatbot assistant to work with gradio, especially with pre-installing the chat-history on a loaded gr.ChatInterface was a challenge and the chatbot does not get enabled in a consistent manner. Sometimes the chat-history is not loaded. Response accuracy needs some work as well.
  2. Handling google cloud/gemini API rate-limiting issues and unknown internal errors, etc was a challenge as well.
  3. Have to say this - doing the demo under 3-mins was a challenge for sure :) The video quality for some reason (most likely has to do with screen sharing with zoom) did not come out good. I couldnt find an alternative way of doing this unfortunately. Sincere apologies. Hope it does not affect judging the application. Lot of work has gone into this and its for a great cause as well. TIA!

Accomplishments that we're proud of -

Doing this project itself as a 'non-programmer' is a major accomplishment in my opinion! Overcoming the data handling challenges and making this product (demo) available in time especially with the chatbot feature is something I am proud of. I did think about taking in a team-mate but with the availability of low-code options so far I didnt need one. I might need one to productize as full-fledged mobile app.

What we learned

** Everything to do with this application was pretty much a learning experience.** This is the first time I was working on multi-model LLM, handling images with APIs/SDK of a model. Havent used Gemini before (used Palm model) as well.

What's next for A(I)yushi - Your Nutrition and Dietary Health AI Assistant

According to this Mckinsey report: link, "...wellness products and services to be more than $450 billion in the United States and growing at more than 5 percent annually." There is great commercial potential for this product. I believe this would be useful for insurance companies, government agencies, hospitals to integrate this into their systems. This can also be integrated into kitchen appliances such as refrigerators (I know Google already has a product for a commercial appliance, but not sure if the scope of that product is the same or if there are overlaps TBH )

As for the next step, the possibilities are endless. Firstly I have to make it a mobile app.

Here are some of the things in my roadmap :- 1. Fine-tuning gemini-1.5-pro model (when available) on the data and use it as the base for this application. 2. Highlight unhealthy/redflag-nutrients or ingredients live on the phone camera - this may require fine-tuning image models 3. Connect to market places/food delivery app for ordering 4. Use local model (running on mobile phone) for faster processing 5. Recommend food products based on dietary preferences (diary-free, gluten-free, etc)

6. Set nutritional filters (carb limits, etc) when searching for food products.

Built With

Share this project:

Updates