Inspiration
I went on a 14-mile hike once when I was just a jit, and we ran out of food halfway through. I found some nice looking berries but my mom tells me till this day that they were poisonous. When I saw the offline hosting capabilities of Zetic Melange, I was reminded of the berries, and how I would never know for sure if they were edible.
What it does
Checkit is an offline-first plant safety assistant for campers and hikers, leveraging the power of multiple multimodal models locally.
- Off-line: real time object detection, classification, and reasoning to check if object is edible.
- On-line: converse with the app to prepare for a trip, and parallelly compiles a region-specific knowledge pack for local models.
Benefits of off-line models:
- No network in the wilderness.
- Low latency real-time inference.
- Private travel plans and locational data.
How we built it
We built Checkit in SwiftUI using Cursor for rapid development. For model deployment and inference, we utilized the Zetic Melange platform and its hosted quantized models built for on-device, offline processing. The pipeline combines YOLO11 (detection), Apple Vision (plant/not plant classification), custom Plant300K classifier (plant species classification), and Gemma4 (reasoning), to take real time camera input, identify if it is a plant and what kind, compare against prepared database, and outputs edibility. We also utilized Whisper-tiny's encoder and decoder models to let users converse with Gemini 2.5 Flash model through Google's API to help them prepare for upcoming adventures. Simultaneously, Gemini searches the web for regional information and compiles them into a plant knowledge pack that can be accessed once the user is on their journey and without network. We also leveraged Figma MCP for page planning, widget templates, and design imports, as well as Swift MCP for SwiftUI documentation lookup.
Challenges we ran into
This project was harder than a 14-mile hike without food. We ran into several issues:
- Our biggest challenge was balancing accuracy and latency on-device. We had to iteratively redesign the runtime flow so camera inference stays responsive while deeper plant reasoning happens on tap.
- We also needed to avoid overconfident outputs by adding multiple fallbacks and uncertainty paths to ensure user's safety.
- Multimodal support was also tricky, especially due to hardware constraints where larger, more advanced audio models could not be hosted on our devices.
- It was also our first time building an end-to-end MVP using Swift, and Xcode complains more than an adolescent, with error messages more perplexing than a physics textbook.
Accomplishments that we're proud of
We’re proud of integrating multiple multimodal, on-device models into one smooth offline + online experience. We also really like the parallel between the frontend user experience and the backend region-aware workflow that makes plant safety guidance more context-specific. We also spent a lot of time working on audio input and processing, and it was rewarding to see it work.
What we learned
We learned a lot about SwiftUI architecture, mobile inference optimization, and how to coordinate multiple computer vision and language components under real-time constraints. More important, we learned the importance of having a complete and detailed set of requirements, instructions, and objectives. Having a good plan is a project half done.
What's next
We want to expand Checkit into a broader offline camping assistant: first aid triage support, fire-building guidance, campsite setup help, and more real-time outdoor safety workflows.
Log in or sign up for Devpost to join the conversation.