Currently, insurance companies use photos for review of a claim as well as physically having to go to inspect the property damage in person. It can be quite a lengthy process for an agent to review and it takes time out of the clients day as well as having to meet with an agent. With the easy accessibility of Augmented Reality and using the latest generation mobile devices, users would have the ability to make claims by creating a virtual 3D version of their covered loss, increasing the ease of use for both the customer and company.
What it does
EasyAR Claims allows for a customer of an insurance company to make a claim easily by taking a video of their damages (be it home or auto) and submitting it to their agent for a quick in-depth review that is more reliable than photos for the nitty-gritty details and leaves no questions unanswered. It does this by taking the video, breaking it down into its frames, using those frames as 2D images, and then finally stitching them together to create a 3D model that can be rendered.
How we built it
We have developed an android app to be used for taking videos of objects and turning them into a collection of images used for submitting to our backend service via a React front-end, which then renders the 3d model in the app through the mobile app wrapper.
The React app consists of two React apps, React 16 for the UI and submitting the photos to our service and React 15 at another endpoint displayed in an iFrame within React 16 to allow for the rendering of 3d models in one application.
For the image processing, we capture video using an android device, break that video into its frame components then uniformly selecting 30 frames from beginning to end, convert each frame to a bitmap which we then use to store as a .jpg file on the device's internal storage. We then send the file batch to Sourceforge's API to be processed, which then is retrieved as a .obj file (with its textures) from an AWS S3 bucket to display in our web app.
For the augmented reality we use Unity and Magic Leap to render the same results so the model can be viewed in a much more interactive way for the agent.
Challenges we ran into
Unfamiliarity with React and researching different packages that would allow for rendering a 3D model in a web browser. None of us had ever worked with Magic Leap before or Sourceforge's API, so understanding the data flow and structure for images was a massive challenge.
Accomplishments that we are proud of
Being able to render 3D Models on a web interface and learning ReactVR for web. Although we ended up going with OpenGL instead. This is also one of the most ambitious projects we have attempted with so many moving parts and that we were able to mostly bring everything together is something we are very proud of.
What we learned
How to use React and OpenGL for displaying 3D Models in a web application. How to best stage images to have them pieced together from a 2D to a 3D space and have that rendered in augmented reality.
What's next for EasyAR Claims
Next we would like to build out user accounts for agents and clients and build a full working claim system where we can start building out a database of these damage models and possibly start doing data analysis over these datasets when they start to build out.