The inspiration for participation in Helvar challenge came from team's common interest in the new high tech thermal sensors. We saw multiple opportunities how to utilize thermal cameras capabilities to detect human presence. First priority was, of course, to solve problems with processing the raw sensor data and utilize machine vision techniques to detect people in thermal image to, for example, count them. Helvar utilizes this system in active lightning control to predict people movement in offices and corridors and turn on only required lights with added option of light intensity control. Secondly, we saw possibility to use same sensor to count people in robot bus, in Helsinki, while still maintaining privacy of passengers. Third challenge is from GE whereby we tried to count people and estimate the heat emitted from their bodies. Using this data we could optimize the central heating system to use less heat and hence save energy and money. We felt that all three problems can be approached by a common solution/method. And this is the use of thermal imaging and then using the acquired data to move towards solution to above mentioned challenges. There are no privacy concerns. The solution are generally cheap to deploy and the prospects are HUGE!
What it does?
The FLIR Lepton thermal camera is relatively cheap piece of hardware, opening new opportunities to sense presence of people. It was provided by Helvar with required openCV and python software on raspberry pi 3 development board. Code was written to detect the number of people by keeping in view the limited resolution of the camera. After detecting the number of people and the amount of heat radiated by them, we can use this data for all the above mentioned challenges.
How we built it?
- Sending picture stream from camera to server that is doing the actual machine vision.
- Detection of the contours, using existing algorithms from openCV (this give us the number of people).
- Detection of entrance/exit of the thermal image (real time tracking of people number).
- Creating ID for each contour inside the image.
- Designed 3D printable device enclosure, inspired by junction logo.
This data could be used to save power as per power emitted by human body. (attached in files). https://docs.google.com/presentation/d/1TinyfOhpKdX5xsJUNUu2CLumtHkMTZK9wm0YNdeWPSY/edit?usp=sharing
The consumption data could be sent to utility companies and then they could schedule the balance between demand and supply in real time.
Saves a lot of assets in terms of fuels or operation costs.
The system is intelligent and will learn over time.
Challenges we ran into
First challenge was the lack of instructions to control camera, we were forced to develop 'tape and glue' solution to fix certain colors to certain temperatures to ease detection of human body temperature. For example hot coffee mug caused automatic gain control to render human body temperature same color as surrounding structures i.e. whole image saturation. Hardware was otherwise quite straight forward. Programming, however, caused quite a few problems due to installation of the programs and finding correct tools with preferred language, we were supposed to use existing tools, not to create anything new due to short time.
Accomplishments that we're proud of
Although we did not quite reach goal, we went according to the plan and got done everything we tried. The system has been implemented in a basic form.
What we learned
Thermal cameras are getting more advanced year by year but are not quite there yet. Lepton is interesting project but does not yet work with intended usages, mostly due to resolution and narrow field of view.
What's next for Meltdown
Core of the team will continue working as a team due to same working place, we will wait for future hackathons and perhaps continue working with new stuff.