Bucketfly

Inspiration

We wanted to create an easy to use app to help technicians in all conditions complete inspections.

What it does

Bucketfly turns a smartphone into an AI-powered equipment inspector. Operators point their camera at a machine and narrate what they see — the app automatically identifies components, grades their condition (pass / monitor / fail), and generates a complete daily inspection report mapped to the official CAT TA1 safety checklist. It also provides real-time guidance during recording, telling operators what to look at next and flagging issues on the spot.

How we built it

Frontend: React with Vite, using the browser's native MediaRecorder and Web Speech APIs for video capture and text-to-speech guidance Backend: Flask REST API AI: Google Gemini 2.5 Flash Lite for multimodal analysis Memory: Supermemory for persistent storage

Structure

You submit a video with audio -> audio is transcribed via gemini and keywords + timestamps are identified. -> timestamps are used to pull frames from video to submit to gemini for analysis Analysis -> There are four ratings- Green (pass), Orange (take caution), Red (Failure), Grey (not enough information given) -> gemini will use the audio transcript and the frames selected to inspect the conditions of the parts and map them to their proper section in the report. -> gemini will check that the audio and video analysis agree with each other. If it does not the program will flag this item as either orange or red depending on the criticality of the part -> gemini will compare the current evaluation with the evaluation of the specific part from previous inspections. It will observe differences and any change in condition e.g. cleaned or rusted. -> the app will inform the user of any orange, red, or grey status and give a report on why -> the app will complete the report

Challenges we ran into

Gemini's free-tier rate limits were our biggest bottleneck, we had to carefully optimize how many frames we send (2 FPS), batch our API calls, and design the real-time identification endpoint to be as lightweight as possible. We also faced challenges getting the AI to output consistently structured JSON across wildly different equipment components and conditions, which we solved with few-shot prompt engineering and training data examples.

Accomplishments that we're proud of

History comparison of parts. Having a history both allows us to have a more precise evaluation of the part and observe trends of these conditions. This inspection history is powered by Supermemory, you can track how a component degrades over time. Real-time live feedback overlay that identifies components as you point your camera, with spoken guidance via TTS. A complete, professional-looking inspection report generated automatically

What we learned

We learned how critical project structure is when building with AI, clean separation between visual analysis, audio processing, and cross reference reasoning made the system more reliable than trying to do everything at once. We also learned the importance of proper git workflow and incremental testing when multiple systems are talking to each other.

What's next for Bucketfly

More machines: Currently supports the Medium Wheel Loader. But it is a very modular build. Additional machines could easily be added to define more depth to the program. Fleet dashboard: A manager/admin web view that shows inspection status across an entire fleet in real time Long term analysis: Show how parts degrade or hold up for specific conditions. Could provide information for Caterpillar to help create condition resistant or increase warranty of specific parts.