When I had to drive through Texas I got into an accident with deer. It appeared so fast that I couldn't react to it. Due to deer vehicle collisions, there are about 1.5 million claims, 211 deaths, 29,000 injuries, and 1 billion dollars worth of property loss per year. Deers travel in packs, so there are probably more nearby.
So we decided to build a solution that prevents the death of people, deers, and avoids damage to your car. Of course our solution can be used for a number of other driver-assistance cases to avoid different accident scenarios(for example accidents out of sight, like electric scooter, bike, etc.), emergency medical assistance services support, etc.
What it does
Our Intelligent Emergency Warning Solution is based on collecting information from moving vehicles and applying Machine Learning to help decision-making for Drivers in different accident scenarios (for our demo we have chosen "Deer on a Road" scenario).
- The Android application sends real-time information from the Camera combined with GPS location, and sensors: accelerometer, gyroscope (need to improve location accuracy and detect hard-brake events).
- Server side (backend) is deployed on the AWS Wavelength processing video stream to detect potential collisions/accidents/emergency situations and sends notification events in real-time to potentially affected cars (nearby cars in a range of 0.5 km, configurable).
- Android App shows the notification on a map with event location and type of accident. Why do we need AWS Wavelength? - Such connected vehicle services required low-latency (milliseconds) telemetry transmission and processing in real-time.
How we built it
Core microservices deployed to AWS Wavelength EC2 g4dn.2xlarge (GPU) instance type:
● Custom Geo-Spatial Message Broker for location-based realtime Vehicles Communication: core business logic to route real-time updates/events with connected vehicles within target range. (WebSockets, Java, Micronaut). Custom Geo-Spatial Message Broker Implemented using Publisher - Subscriber pattern where Topic Ids based on geo-location bins (hexagons). Publisher and Subscriber switching between sets of topics during changing location, depending on current vehicle locations, very similar to handover procedure in mobile networks.
● Custom Deep Learning model to detect and classify different objects on a road (cars, pedestrians, bikers, deers, etc). Why do we need AWS Wavelength? - Such connected vehicle services require low-latency (milliseconds) video-processing in real-time, thats why we have chosen GPU based VM instances for Deep Learning acceleration - g4dn.2xlarg deployed on the edge using AWS Wavelength.
● The Android application is designed to gather device sensor data and real time video feed. In the demo we mainly use position related data such as location, speed, azimuth (device bearing) and accelerometer data. The video processing is optimized so that we can process each frame individually as image data. All the inferred objects by the model are displayed in near real time on the device's screen and a warning notification is simultaneously transmitted to every other device in the configured range. For the demo we implemented an ability to send frames of a pre-recorded video instead of a camera video feed, so that we can see the application in action even in the Nova lab.
Challenges we ran into
At first, we had some image pre-processing on the device (extracting every frame's features to pass to ML model) as 4G and high latency to machine learning service deployed in AWS implied we can't send unprocessed frames of 30 fps video feed from camera and receive insights in near real time. With AWS Wavelength and Nova testing environment we've discovered that 5G and AWS edge processing has enough performance (low latency & high bandwidth) to allow near real time processing of 30 fps video frames by our ML model as is, without pre-processing.
Accomplishments that we're proud of
We have a working prototype!!!
What we learned
Configuring and deploying backend to AWS Wavelength, and of course fighting with Nova service)
What's next for SmartRoad5G
MVP, and then go to market)