CloseCalls

• 

  A foggy scene, intended to add slight noise to the data

• 

  A night scene, intended to challenge models to detect dark vehicles and pedestrians in low light scenes

• 

  A rainy scene, common almost everywhere in the world, challenges models through water reflection and dark skies

• 

  A sunny scene, considered ideal for computer vision, a baseline

• 

  Example conversation with Gemini about city planning based on our traffic data

Inspiration

We find that ML and computer vision will be a pivotal tool in improving road safety for both drivers and pedestrians. However, finding and processing computer vision datasets can be costly and rare to find useful data. We look for an alternative such as nuScenes feature large cities, but they limit in flexibility despite the vast amount of labeled data they offer. CloseCalls was a way to add new traffic scenarios and environments for training Machine Learning models in an easily configurable way. Allowing us expand they opportunities for computer vision implementations in smaller towns.

What it does

CloseCalls uses the CARLA simulator in Unreal Engine 4 to generate realistic traffic intersection data. Every scene is automatically labeled with bounding boxes, object attributes, and actions. Expanding on traditional tasks such as predicting vehicle speed or location, we add the capability to detect car crashes or scenarios in which a car nearly hits a pedestrian/bicyclist. This provides a way to simulate vehicle-vehicle and vehicle-human interactions without compromising real people. This information can be used for city planning, determining where in a city may be dangerous and in need of increased development for pedestrian safety. After generating data in various scenes such as clear-sunny days, a foggy morning, and a night, we employ a OpenTAD Machine Learning Model to train and test using our synthetic datasets using the AMD Developer Cloud.

How we built it

We used the CARLA Drive Sim, configured it in Python and ran a Python script to automate capturing frames including raw images, semantic segmentation images, and various target metadata. By compiling a variety of scenes, we made a foundation dataset useful for many machine learning tasks. Using the AMD Developer Cloud, we were able to train our models using ROCm without compromising our performance on Unreal Engine 4. These models can now be applied to testing environments within Unreal Engine 4 and other existing datasets such as nuScenes.

Challenges we ran into

Our biggest challenge was resource limitation, with only one high end PC sponsored by AMD that could run Unreal Engine simulations, we often had to wait for data generation or large file installations and processing scripts. After the first few hours of setup, and once we got our first synthetic driving dataset generated, we could work on multiple tasks at a time. As a team, we could simultaneously work on data generation, model training, and Gemini processing at the same time. Our initial delays became the catalyst that allowed us to finish strong after the first 12 hours.

Accomplishments that we're proud of

We were able to not only run a Machine Learning model with custom features, we generated our own synthetic data that we were able to preprocess and convert into our own custom dataset. It could easily be configured to add/remove different assets, weather, and autonomous "actors" that run on their own. Additionally, we bring a processing script that allows users to extract more metadata than we provided, limited only by the amount of processing they are willing to perform

What we learned

Prior to this, none of us had experience with Unreal Engine or working with the tiered inheritance structure brought by Carla. This project required us to learn fast in spite of our initial delays and generate a sample dataset to begin training Neural Networks. We learned about what features make a good image dataset for multiple tasks. We then had to learn about segmented image classification in order to train our model. To prove our dataset is useful, we learned how to implement a DeepLabV3 model that can accurately segment raw traffic data on 29 classes with only 0.8 test loss at the time of writing.

What's next for CloseCalls

We want to build a report about the data we gather and go through an Exploratory Data Analysis Dashboard, that would congregate all the data we generated plus all the metadata in order to effectively discover insights of how we could improve our datasets and traffic safety. Additionally, we hope to address challenges in generating skewed data labels for machine learning. For example, we want to provide a setting to generate more car crashes and close calls with pedestrians. This will build more robust models for use in real world environments, providing insights to machine learning engineers and civil engineers alike.

Built With

• amd
• carla
• gemini
• opentad
• python
• rocm
• unreal-engine