Paint-mate AR

Use AR to accurately estimate the amount of paint you need.

Comment

Inspiration

Three facts: 1) humans are terrible at area estimation; 2) AR tech has the potential to be excellent at area estimation; and 3) approximately 1/6th of paint is wasted (presumably in relation to 1) which is both an environmental and commercial problem.

This suggests that if we can harness the power of AR technology, we could equip domestic and commercial users with a fast and accurate way to save money, and the environemnt.

What it does

Paint-mate has three key functions: i) it measures the area of selected walls; ii) it previews the painting of those walls in a selected colour, iii) it reports how much paint (of that colour, from a chosen manufacturer) that paint job requires.

How I built it

We built on top of the spatial understanding functionality within the Mixed Reality Toolkit (MRTK) for Unity. We started with the base functionality, and adapted each element to fit the desired UX.

We designed the following original funcitonality directly through the MRTK in Unity: Area calulation and readout, area subtraction for windows/doors (incomplete), colour-selection, colour application to a surface, and an output report (incomplete).

UI was designed first with sketches on a surface pro, then in Sketchbox on the Vive, then prepared in illustrator and imported into Unity.

Challenges I ran into

No-one in the team had HoloLens or MRTK experience before we began, and simply deploying anything to the device proved problematic. On day one we lost the majority of our time to this issue. This turned out to be an issue with the network that we were using at the hackathon and, once resolved, we started to make rapid progress (though intermittent glitches hindered us on day two).

Once we had harnessed the spatial understanding in MRTK, we found that it made certain simplifying assumptions about corners that affected the accuracy of the area calculation (c.93% accuracy). Repeated testing showed that this could be mitigated through a post-hoc adjustment on area calculations, and we now have a tiny +/-2% error average over dozens of tests.

A further challenge arose in relation to the removal of windows and doors and the like from the area calculation. This required the designing of a subtraction tool, but the task of calculating the subtraction of one mesh from another was more than we were able to undertake in the remaining time. Instead we opted to identify an area on the known plane, subtract that from the initial area calculation, and use the subtracted area as the basis of an object that masks the painted area of the wall that it covered. The result will be that all and only the area that should be painted is included in the area calculation, or shown in the preview. (We ran out of time to complete this - feature redacted for demo.)

Also: Neil and Yannick were jetlagged, Aiden fell whilst skating and cut his hands, Neil's GPU fan on his computer came loose from its bearings, frying his laptop, our loaner Vive laptop didn't support Vive-grade VR, and we were evicted from our room just as we got Sketchbox working.

Accomplishments that I'm proud of

The primary accomplishment is achieving the primary goal of the app: to harness (available) AR tech to accurately measure the area of a wall. This functional benefit to users has extraordinary reach and value.

The second major accomplishment was to work out how we could allow users to identify not-to-be-painted areas without requiring mesh subtraction. We ran out of time to fully implement, but this is perhaps one of the major estimation errors that contributes to waste, and so a major featrue planned for Paint-mate.

The UI/UX flow is also worth being proud of. AR is novel, and the UI/UX paradigms are still evolving. That makes even simple funcitonality challenging, and good experiences have proven elusive. We are proud of the user experience and interaction in Paint-mate.

What I learned

How to use GitHub, DevPost, and Slack (Neil) How to use MRTK! How to deploy to a HoloLens!(hint: make a local hotspot and do not rely on institutional networks) How the spacial understanding abilities of HoloLens can (and cannot) be deployed. (A normal wall is a good size. A small alcove is less accruately captured, and high ceilings are unlikely to be well captured.) That USC is a dry campus.

What's next for Paint-mate AR

The big vision is to 1) white label paint-mate as a plug-in that paint manufacturers can licence for their own apps, and 2) to develop a stand alone tool that draws from existing BIM databases.

To that end, we need to develop/refine the following additional functionality:

  • Add multi-wall support. Users should be able to identify more than one wall per estimate.
  • Add support for non-standard wall shapes.
  • Add subtraction support.
  • Add true mesh-addition support, for areas of painted surfact that are occluded from scan by furniture etc.
  • Add support for non-standard subtraction shapes.
  • Add API support.
  • Add back-end support for storage/recall of previous estimates.
  • Add order integration.
  • Refine preview colour rendering/ambient light integration.
  • Add support for different lighting contexts emulation.
  • Develop for ARKit and ARCore as vertical plane detection becomes fully supported.

Built With

Share this project:

Updates