Campuskost is a platform for recipe creation and sharing. The app has been my long-time project and every now and then, I tackle a new challenge, setting up CI/CD, adding SEO, offline mode.. This Hackathon was a perfect opportunity to add AI capabilities.
The client application is a progressive web app built with React. Before starting this effort for Azure AI Hackathon, the following features were available:
- uploading recipes by manually entering ingredients and descriptions.
- liking recipes, creating and saving recipes to lists. Following users, etc.
I'd like recipe management to be seamless, motivating more people to find joy in cooking. 🍲🥑 Moreover, I want to ensure that only valid food images are uploaded to facilitate the task of being an administrator to an application open for public use.
What it does
The added AI capabilities allow the users to:
- add recipe ingredients by voice
- add recipe description by voice
And ensures a good vibe on the platform by:
- validating uploaded images with Computer Vision API (looking at the "adultScore" and "goryScore" )
How I built it
- First, I created the Azure resources needed for the AI capabilities:
- Speech resource (for speech-to-text)
- Computer Vision (provides a broad range of analysis features)
Then the major task was to integrate the AI capability into the app logic. In this case, it was the upload procedure that I wanted to improve, and thus buttons for voice recording were added when listing both ingredients and descriptions.
Challenges I ran into
- most examples use prediction by image URL, and I had some issues making sure my image data was in a valid format.
- at first, when users were to record ingredients, they had to press the recording button for each new entry as the API returned the result as soon as it detected a full sentence. It was a little challenge to rewrite the code to run continuous analysis and update the recipe form on asynchronous analysis events.
Accomplishments that I'm proud of, and what I learned
I'm glad that the AI features already add value to the application in production mode. During this project time, I learned how to work with the speech-to-text API and making use of the Computer Vision resource.
What's next for Campuskost
- Adding other image analysis models. One could be to extract information about recognized ingredients, helping to automatically create descriptions of nutritional statistics.
- Create recipe recommendations (can also be an AI enhancement!)