FISHazam

Winner of Fishackathon London
Live Demo

Inspiration

One out of every three fish is incorrectly labelled. Fishmongers regularly try to pass off widely available cheaper fish as more expensive sustainably managed fish. When fish is in the form a fillet, it is very difficult to tell the difference. Human vision, let alone its computerized version, simply fails despite extensive training. There is a significant cost to sustainability. First, fishers who fish sustainably often have relatively additional costs. If other types of fish can be easily substituted, it disincentivizes fishers from fishing sustainably. Second, there is a consequently skewed global picture of the health of fisheries. Relying on economic sales data as an indication of fish stocks is simply not reliable as a guide for fishing regulations and strategies. The level of another wise at risk fish species may be artificially inflated.

What it does

FISHazam derives a spectrogram via infrared spectroscopy. The resulting spectrogram is passed through a fingerprint algorithm that allows a more efficient identification process. This is analogous to music identification services. The main difference is that infrared waves rather than sound waves are used in our case to build the spectrogram.

How We built it

We were a relatively small team of two. This was both an advantage and a disadvantage. It was an advantage in the sense of less time spent building a consensus and coordinating activities. But it was a disadvantage due to our relatively greater individual workload. Suffice it to say, sleep was just not on the agenda throughout Fishackathon. We narrowed down the problem into two fundamental features : spectrometer and fingerprint. Each one of us was responsible for one feature. However, we practiced pair programming intermittently - coding along each other with real time feedback.

What We learned

To our surprise, two scientific researchers had previously pursued the avenue of fish fillet authentication via infrared spectroscopy. Their findings confirmed our hunch, that infrared spectroscopy would be a reliable and accurate methodology for the problem. This gave us further motivation.

Challenges We ran into

Unfortunately, the study was limited in scope. No public database of spectrograms for fish seems to be available. So we could not conduct large scale data tests. Furthermore, we did not have access to a professional spectrometer or hundreds of types of fish in fillet form either. We hope to rectify these issues gradually in the future as part of the Road Map.

Accomplishments that We Are proud of

But we were nonetheless proud of our ability to have both the spectrometer and fingerprint features working within the confines of our limited time and resources. Our spectrometer feature used WebRTC to call upon the camera directly from a browser on any platform. Combined with our physical DIY Spectrometer, we were able to derive spectrograms. Our fingerprint algorithm received these spectrograms successfully and was able to match patterns.

Road Map - What's next for FISHazam

Phase 1 - FISHazam Coalition
Fishing problems are inherently global problems. We will try to assemble a coalition of both governmental and non-governmental institutions which share a belief that our approach can tackle fish fraud and promote sustainability. Coalition members can be involved in different ways. They can provide a grant for FISHazam DB. They can provide feedback during beta testing. They can share their exisiting data sets. They can discuss their first hand knowledge of issues.

Phase 2 - FISHazam DB
Our first order on the agenda will be to establish FISHazam DB - a free public repository of infrared spectrograms of commonly eaten fish. With a grant, we would be able to use professional lab-grade spectrometers that can provide the benchmark for future spectrograms.

Phase 3 - FISHazam API for Hardware
The next phase will involve integrating various spectrometer hardware providers into a FISHazam API. Our objective will be to facilitate the submission of spectrograms from various spectrometers around the world.

Phase 4 - FISHazam Fingerprint
With enough data, we will be able to apply the fish fingerprint algorithm to a large data set. Subsequently, pattern recognition can be optimized to the point where only a few seconds are necessary to identify the species of fish.

Phase 5 - Self Sufficiency
At this level of efficiency, the hope is that there will be wider use. FISHazam will then be able to achieve self-sufficiency via a referral service of fish providers with low instances of fish fraud.

More Info

You will find a general overview with links on our website. For those that are technical minded, you will find code repos at our github along with snippets and explanations in the documentation. On our YouTube channel, you will find the presentation we made at the hackathon, screencasts discussing code, as well as playlists of good videos explaining the scientific underpinnings of our project. And if you have any further questions, feel free to drop us an email to info at fishazam.