Smyle

Cashier Waiting
Analytics Page
Processing an Order

Inspiration

It all started when we saw saw an elderly man in a wheelchair at a Tim Hortons struggling to pay for his order. He had to be helped by staff to stand up and take out his wallet to make the payment. When we talked to him, he told us that he had severe arthritis and often struggled to use his hands, and it was particularly difficult when he had to take out his wallet to pay.

This got us thinking– what if we could build a system that empowers people in similar situations and makes it easier for people with reduced mobility to make payments.

What it does

Smyle doesn’t focus what you’ll be paying with, but instead how you’ll be paying. It cuts out the need for any kind of wallet, card or phone. Instead, we use your most unique and identifying part, your face.

Everything you need to pay is in your smile. Cutting out the needs for a physical card enables people with disabilities to pay by themselves like never before. People who have had arthritis, are paralyzed, have dementia or alzheimers can now pay by themselves with just their face. And the list of disabilities we can help just keeps on going.

We also saw that our systems tends to dramatically reduce the amount of friction at each touchpoint. We’ve made sure Smyle will make payments super quickly! The identity transfer that Smyle provides also enables powerful features like recommendations and personalized customer service.

How we built it

We made 4 different source bases for Smyle, each necessary in their own way.

Our cashier terminal was built with Vue.js and provides the cashiers appropriate tools to take orders and even look at business statistics.
Our payment terminal was built with Swift and Firebase SDK and works hand in hand with the cashier terminal. It uses Firebase to detect facial expressions quickly on device.
Our user app was built with Vue.js and helps the user enroll and manage their account.
Our backend was built with Ktor and Azure. It connects all our solutions together and handles most of the facial identification with Azure Cognitive Services. It was hosted on Google Cloud Platform.

Challenges we ran into

For many of us, this was our first run with AI. We decided to study a bit on the API beforehand, but we were taken by complete surprise by how in depth Azure Cognitive Services are. Occasionally, our backend wouldn’t tell us if it couldn’t access the image we were sending it or if it couldn’t understand our request. We had to do lots of guessing to get the whole system working, but when we finally got it to identify us.

Accomplishments that we're proud of

The “smile to confirm” part of the payment terminal works like magic! It’s amazing how easy it is, and it’s incredible how the computer recognizes it so quickly. Every time we confirm and the payment went through, our small smiles we used to trigger the machine became completely genuine.

Our UI turned out to be much better than we expected when we walked into the project. But we took important feedback and spent hours working it into our interface. We came out with some of the best work we have ever done.

What we learned

We all learned a lot about facial recognition, how it works and how to make sure it is consistent and safe to use. Each of us experimented with a new technology we have never tried before which was super cool! This time we researched and incorporated design patterns, tips and principals into our project.

We all can’t wait to use there technologies in the future!

What's next for Smyle

We see smile as more than just a payment system– we see at as a way to completely reinvent the way we interact with the physical world. We see public transit cards, airplane tickets, concert passes, and more being completely replaced by your face. We believe that by connecting all of your identification and payment details together, we can create a powerful tool for both consumers and business to improve experiences.

We didn’t have access to an IR camera, but in the future a depth map could be constructed from user’s faces to make the service even more secure and reliable.