The inspiration came from reimagining how stores can advertise to pedestrians in real time. Rather than having static, bland ads that offer no real value to consumers, we thought we could create a basic version of an interactive advertisement that offers real discounts to consumers as well as create value for store owners who want to learn about their clients and drive sales.

What it does

The product runs by displaying an advertisement that engages pedestrians by allowing them to use leap motion gestures to, in this case, move the "Roots Beaver" to a shopping basket. The engaging ad promises to give customers 10% off their next purchase, and once the challenge is completed, a unique code is revealed to the pedestrian. While this whole engagement is occurring, an embedded camera (in this case webcam) will snap one photo of the pedestrian. Using Clairifai and Azure, we process the image to understand the gender, ethnicity, age and emotional status of the pedestrian, thereby connecting this data to whether or not the person completed a sale.

How we built it

We build it mainly using Javascript written inline in an HTML file which displays the advertisement. Simultaneously we run a express server that accesses the webcam, takes a photo, sends it to Azure/Clairifai, and we then process/aggregate the information returned from both imagine recognition tools. Each interaction loops this process.

Challenges we ran into

Xesto. We invested a huge amount of time into using Xesto to process more complex gestures, to deal with more complex customer engagements, but unfortunately Xesto broke, resulting in our shifting our work to dealing directly with Leap Motion.

Accomplishments that we're proud of

Making it past the challenges listed above, working together with a completely new set of friends, being able to stay awake through it all (;

What we learned

Simple things are never really as simple as you might imagine. Also the winter is wise.

What's next for Store Front Engaging Ad Platform

We are looking to develop a more comprehensive prototype of the idea, and potentially work with Xesto moving onwards to create a proof of concept for them to bring before major commercial partners.

Share this project: