With vehicle automation and development of smarter vehicles, a significant amount of driving related safety concerns have been addressed. The sensors such as proximity sensors, cameras, night vision cameras, blind spot vision, driver monitoring systems, and corresponding algorithms have enabled and revolutionized the self-driving technology. At the same time, they have also guaranteed improved safety while the vehicles are being driven. However, not a lot of effort has been made to prevent vehicle and property damage when the vehicles are not driven or occupied. Although the damages caused during such scenarios are comparatively of less intensity and happen less frequently, the problem itself is not of negligible scale. According to CNN, there were 1.85 million car break in incidents in US leading to a property damage/loss of $1.255 billion in 2009 (source: “http://www.cnn.com/2009/LIVING/wayoflife/ 04/15/aa.avoid.car.break.ins/”). Parking lot hit and runs, and door dings have happened to almost all of us and, unfortunately, are routine incidents for some. Similarly, vehicle vandalisms of different natures such as keyed paints, dented cars, and slashed tires are not uncommon.
What it does
The presented solution is a monitoring system that senses any contact made with the car, and records the incident. Furthermore, license plate detection and facial recognition features are added along with capability to send text messages with pictures. Hence, in an event of violation, pictures will be taken, transmitted to internet to utilize IOT image recognition services for license plate and facial recognition. The results will then be conveyed to the end user in the form of text messages with relative information. The text result from license plate recognition will include alert notification with plate number and a picture of the license plate of the vehicle involved. The text result from facial recognition algorithm will include an alert notice with a picture that displays the face of the individual involved. It is to be noted that the project has focused on prototype development for proof of concept, keeping in mind that the ultimate implementation of the proposed system would be in a modern smart car with access to cameras and sensors installed by the manufacturers. This will provide adequate amount and variety of data to make this system more effective.
How we built it
First made independent sections work
Arduino 101 + Camera + SD card The first part of the system is capturing the pictures from the cameras at desired rate, and storing them in SD card in correct format. Two camera modules – Arducam mini 2mp OV2640 have been selected, however, the number of type of the camera is not a limitation. Two cameras were chosen to keep the project economically reasonable, although more cameras (at least four), and wide angle lenses would be preferable. Moreover, the future goal of the application is to access the multiple cameras that modern smart cars are equipped with.
Arduino 101 IMU trigger The IMU mounted in Arduino 101 has been used as the sensor to detect the resultant shock of another vehicle, person or object hitting the parked vehicle. The IMU is operated in shock detection mode. For final implementation, it will be beneficial to access vehicle mounted accelerometers. The trigger function generated currently is based on the amplitude of the shock. Although this method is satisfactorily effective for shock detection, adequate data collection of amplitude and frequency of vibration can lead to implementation of machine learning algorithms to accurately determine general location of impact. Hence, the data only from the respective camera can be stored and analyzed for efficient operation.
IMU + Camera + SD Card +BLE functionality The camera, SD card, and IMU trigger have to work in sync for this application to work. The implemented logic involves continuous capture of events using camera into the limited slots of SD card as determined by the script. Once all the slots are full, the old images are over written by new ones if the trigger is not generated. This help minimizing the memory requirements. However, when the trigger is detected, images from a minute before and after the trigger are stored and transferred to WiFi module via BLE. Hence, both past and future data is provided.
BLE Arduino – Vehicle communication The images stored in SD card will be accessed, deconstructed to byte arrays and transmitted to the vehicles through BLE. The WiFi devices present in the cars will further transmit the images to corresponding APIs. Our vehicles lacked such capability and we faced complications with transferring images using BLE. Therefore, we manually transferred images from SD card using a laptop, however, the laptop can easily be replaced by other WiFi enabled modules to develop retrofit products for older cars.
Vehicle – API communication The WiFi enabled vehicles will transmit the images received from Arduino to corresponding APIs for image recognition. The WiFi capability for this project is provided by Laptop. The laptop can be replaced with WiFi enabled modules like Raspberry Pi to make this product feasible for retro-fit purposes in older vehicles.
Image Recognition Each image transmitted to Vehicle (Raspberry Pi) through Arduino 101 will be sent to two APIs. The two APIs use are discussed below: OpenALPR, License plate number detection – OpenALPR utilizes Local Binary Pattern (LBP) algorithm followed by Optical Character Recognition (OCR) algorithm. Hence, future steps can include development of independent API. (Source: “https://github.com/openalpr/openalpr /wiki/OpenALPR-Design”). Face API by Microsoft Azure, Facial recognition – LBPs and OCRs are also utilized by most of the facial recognition software. Although, Microsoft has not revealed the algorithm behind the Face API, plenty of resources are available for development of a facial recognition API.
API – Mobile device communication A python script has been written to receive the results from the APIs and communicate to mobile devices in form of texts and pictures. The text message includes alert message, the license plate number in text, and figures that include the license plates and faces that were nearby during the incident. It is to be noted that content of the message can be modified as needed. A simple pattern was selected for experimentation. Finally we integrated all these functionalities for the prototype to operate as designed.
Challenges we ran into
- BLE communication - Transferring image files through BLE using Arduino 101 is the biggest challenge we faced during this project. Although we could establish BLE connection successfully and transmit characters and strings, we were unable to transfer images as it involved deconstructing images into byte array in Arduino IDE and reconstructing using a different platform (python in our case).
- Access to vehicle computer, sensors and cameras – Without consent and assistance from the vehicle manufacturers, it will be very challenging to integrate the on board vehicle components with this system. Hence, application with modern smart cars might require manufacturer collaboration and will need further integration efforts.
Accomplishments that we're proud of
We are proud of the hardware which works exactly as planned, but more so of the cloud part of the project where an image is sent to APIs and image recognition is performed and result is sent to the mobile devices.
What we learned
- Establishing communication between hardware and sending large files like images or videos is harder than we had assumed. Especially if the software platforms are different.
- A solution that gets visualized during design phase can go through significant diversions as the project proceeds.
What's next for UNOCCUPIED VEHICLE SAFETY AND MONITORING
• Adding a proximity sensor trigger
• Improving IMU sensing with machine learning algorithms utilizing amplitude and frequency as features, instead of amplitude dependent trigger
• Complete BLE-BLE-WiFi configuration for image transfer
• Implementation in a smart car, experimentation with and integration of vehicles’ OEM sensors
• Improve the cameras and their FPS
• Develop independent image detecting algorithms
• Include other applications such as detecting baby and pets locked inside a hot car