StreamSentinel

Inspiration

OneAquaHealth asks citizens to observe urban streams and fill in a form in their app. While exploring the app, we noticed that the form is quite long (9 steps) and that some questions are hard to answer if you are not an expert. So the observations are often uneven and sometimes wrong. We wanted to see if AI could help citizens give better answers, without taking the decision away from them.

What it does

StreamSentinel takes a photo of a stream and pre-fills part of the OneAquaHealth form. First it checks that the photo is usable and that it really shows a stream. If not, it asks the citizen to take another one. Then it suggests answers using the same codes as the OneAquaHealth app. When the model is not sure, it says so and leaves the question to the citizen instead of guessing.

It also estimates a health risk level, one for humans and one for animals, by combining the photo with weather data (Open-Meteo) and river flow data (Hub'Eau). The citizen checks and corrects every answer. If the risk is high, the observation goes to a manager who validates it before any alert. The final record can be exported in FHIR R4 format.

How we built it

We did not train a big model from scratch. We combined existing models: SegFormer to find the water, vegetation and banks in the image, CLIP to check that the photo shows a stream, OWLv2 to detect pipes and animals, and Qwen2.5-VL-3B, an open source vision language model, to answer the form questions. The only model we trained ourselves is a YOLO11s litter detector, fine-tuned on about 2000 images of floating waste.

The risk score uses simple rules with points. When possible, each rule is linked to the OneAquaHealth indicator factsheets. When there was no published threshold, we say clearly that it is our own assumption.

The models run on a GPU in Google Colab behind a FastAPI server, and the interface is a Streamlit app with three parts: one for citizens, one for managers, and one that explains how the AI works.

Challenges we ran into

Our first result was 86.7% of correct answers, but we realised it was measured on the same photos we used to build the system. When we tested on new photos, it dropped to 71%. It was a bit disappointing but much more honest.

For four questions (channel shape, vegetation on each bank and overall condition), the AI was not better than chance, so we decided to stop pre-filling them.

We also had problems with false detections. A public dead fish model detected fish in clouds and guard rails, and our language model sometimes took plastic bottles for dead fish. We removed that model and rewrote the prompt, which reduced the false alarms from 5 to 1.

Finally, we did not have a GPU on our computers, so we had to run everything on Colab and add a demo mode that works without a GPU.

Accomplishments that we're proud of

We are proud to have a complete prototype that works from the photo to the final export. We tested each model on photos it had never seen. On 21 photos that we annotated by hand, the AI pre-filled 61% of the questions and 71% of its answers were correct. The scene check accepted 92% of real stream photos and rejected 79% of off-topic photos.

We are also proud that the human stays in control at every step, and that the export uses the same codes as the OneAquaHealth app, so the data could be used directly by the project.

What we learned

We learned that testing on new data really changes the results, and that a model that can say "I am not sure" is more useful than one that is always confident. We also learned a lot about putting several AI models together in one working application.

What's next for StreamSentinel

We would like to test the tool on real photos taken by citizens with their phones, have the risk rules checked by OneAquaHealth experts, improve the litter detector (on new images it only finds about 43% of the litter), add face blurring for privacy, and host the models on a stable server.

Demo: https://streamsentinel.streamlit.app Code: https://github.com/momo25bend/streamsentinel