Soul Food Heart Bot

Get modern, healthy soul food recipes to improve the heart health of African Americans with high blood pressure using the latest fine-tuned OpenAI model.

Comment

Inspiration

When I signed up for the AfroTech hackathon, I wondered what could I create that could be useful/helpful to the black community in North America since I'm first gen Chinese in Canada. I grew up in the inner city in Toronto in a public housing complex raised by a single mom on welfare and I had many black classmates growing up. I lost touch with all of them but I remember at lunch their food smelled so tasty. I know African Americans have a great culinary tradition like Chinese do, so I thought why don't I create a soul food recipe app powered by OpenAI. But I thought what's so special about using AI to generate recipes?

Then I realized I share something life-impacting in common with many blacks in the US and Canada and that's high blood pressure. I've had HBP for a while now and I've had to change my diet and take medication to avoid the risk of stroke as I get older. I decided my app will focus on healthy soul food recipes to address HBP and related health issues, not only recipes generated from OpenAI's training data which has a cutoff of Oct 2023, but also modern, healthy versions of traditional recipes from 2024 which I searched for 2 weeks ago. OpenAI had free fine tuning of its models up until last Monday so I took advantage of the promotion and fine-tuned OpenAI 4o-mini with 50 healthy recipes I found in my search through various websites covering soul food.

I also realized my app, "Soul Food Heart Bot", needs to be personalized so I implemented a dietician interview with the user that creates a dietary health profile for them, and then recipes are generated using my fine-tuned 4o-mini model based on their unique profile. As an incentive for the user to try the healthy recipes, I used OpenAI DALL-E 3 to generate an appetizing image for each recipe, with the images (saved in AWS S3) appearing in the recipe book the user downloads from the app. Since we're only human and we all love treats now and then, I included an optional desserts feature which adds healthy soul food dessert recipes based on the user's profile, and these can be updated in the user's recipe book for download.

My hope is that this app helps members of the black community reduce or control high blood pressure and related health issues by incorporating healthier recipes in daily eating which still maintain a vital part of African American culture and history, soul food!

Technical Challenges

Surprisingly, getting the images to work was the biggest technical hurdle. At first I tried using DALL-E 2 to generate the images since it's much cheaper than DALL-E 3 and has a much, much, much higher rate limit (number of images I can generate per minute). But I found DALL-E 2 rejected many requests for images of recipes containing chicken breast for example, which is a healthier choice due to lower fat. So I had to retry with different prompts that would not trigger DALL-E 2's "content_policy_violation" error (with message "Your prompt may contain text that is not allowed by our safety system"). But since the prompt sometimes had to omit "unsafe" words, the resulting image wouldn't reflect the recipe accurately and also I didn't think the images looked that great. The need to retry prompts until eventually an image was generated slowed down the display of the recipes in the app, making the user wait even when the recipes themselves were ready to be displayed.

So I decided to try switching from DALL-E 2 to DALL-E 3, which I read automatically rewrites the image gen prompts to be "safe". I noticed my image gen requests no longer failed due to "content policy violation" on chicken breasts, etc., and the resulting images did look higher quality, which would make the recipes more appetizing to the user. But the image generation itself was slow and the slowness was made worse due to the low rate limit of only a few images per minute.

To avoid making the user wait minutes for the recipes with images to be displayed, I decided in the app itself to not display the images. Instead the images would be linked in the recipe book downloaded by the user. Each recipe would get a unique image URL initialized to a "coming soon" placeholder image stored in AWS S3. When DALL-E 3 eventually generates the real image, my code would upload it and overwrite the "coming soon" placeholder. This allows the user to see the images as they're generated by refreshing their downloaded recipe book (which is a HTML file).

To support this async flow I had to implement a queue for the image gen requests so the UI of my app wouldn't be blocked, with the queued requests processed using DALL-E 3 in a background worker thread. Also I had to use tenacity to back off and retry on rate limit errors, which could easily occur since for just one user my app typically generates 8 meal recipes plus an optional 3 or 4 dessert recipes.

That's why initially I wanted to use DALL-E 2 to keep my costs down and generate lots of images per minute no matter how many users used my app at the same time. But with my code changes and switching to DALL-E 3, as you can see by trying "Soul Food Heart Bot", the user experience is pretty cool as long as you download your recipe book.

Built With

Share this project:

Updates