Flux

Flux is a privacy focused digital billboard concept aimed to personalize the shopping experience in physical spaces. Giving shopping malls the necessary edge over e-commerce.

Comment

Inspiration

Personalized experiences on e-commerce services has improved shopper experience drastically. If you and your friend visits Amazon.com, you are likely to see a very different homepage than one another. Usually populated with appealing items for each of your personal taste. Advertising/recommendations play a significant role improving discovery.

Unfortunately, one of the most engaging shopping experiences; physical spaces, boutiques, showrooms, malls lose the battle in providing a personalized experience. This leads to advertising billboards in physical spaces that leave bad taste with shoppers. There is room for digital tech to be implemented in these areas, but at the same time much more attention is needed on respecting and preserving shopper's privacy.

With technological advancements in Machine Learning and Edge Computing, we are able to conceptualize a solution that help drive personalized experiences to shoppers without being too intrusive.

What it does

Flux, is a smart billboard that allows retail owners to improve customer footfall through a personalized experience for their shoppers. This is achieved by...

  • Allow retail owners to upload an inventory of offers and recommendations for shoppers
  • Allow shoppers to volunteer for the experience and ensure no personally identifiable data is stored during this process
  • Recommend products and keep track of metadata that allows the recommendations to improve over time

How we built it

For the purpose of the concept, the solution has been hosted on cloud services such as Vercel and Azure where the billboard and the control panel can be loaded in a web browser. The same solution can be run on a RaspberryPi or Nvidia Jetson devices to perform PII computing on the edge.

The current process is as follows

  1. Retails owners upload offers and recommendations along with tags and purchase information to Flux.
  2. The kiosk prompts a shopper standing in front of it to take a picture
  3. The picture is captioned using models hosted on Azure AI service. Captions look like "Woman with long hair carrying a yellow purse"
  4. The kiosk queries the offers and recommendations uploaded by retail owners.
  5. OpenAI GPT services are used to identify an offer that is best suited for the caption. Example "Sunglasses at 20% discount"
  6. OpenAI GPT service is further used to generate a custom sales pitch for the shopper. Example "Looking fab! Complete your sunny day look with our premium sunglasses now at 20% discount"

Challenges we ran into

Preserve privacy of the photos

The primary challenge has been to ensure through this process personally identifiable information (name, email, photo, etc.) is not retained without user consent. The current implementation, being on the cloud has to upload the photo to cloud services. However, we have identified models such as BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation that can be run on edge devices ensuring that shopper's photos are not leaving the physical space.

The only information leaving the device is the caption, which is reduced to metadata. The device can still connect to Flux APIs to fetch the offer and recommendation inventory from Cloud services using this caption.

Database at Scale

In order to find offers and generate the best sales pitch for the shopper, Flux has to interact with a database with large queries. Unpredictable demand patterns dictated the need for a highly scalable database. This where TiDBs Serverless offering proves helpful. Furthermore, to analyse high performing tags; the database has to be queried to generate reports of tags being hit and missed. TiFlash column store replicas are able to ensure these parallel use cases scale well.

This requirement of hybrid of OLAP and OLTP usecases makes TiDB the ideal service to power Flux while keeping query times below 5ms.

Accomplishments that we're proud of

The billboard has been able to generate on-point promotions for the audience. The effectiveness of GPTs few-shot learning process has been an eye-opener.

What we learned

It is important to establish a feedback loop in such a system (recommender) to ensure that the quality of promotions keeps on increasing. With this is mind, we extended the features of Flux to fetch the historical sales pitches generated against the audience caption. This can help in fine-tuning the models to be much more effective.

What's next for Flux

  • Implement a Edge Device with localized Captioning
  • Automate the step of taking a picture to automatically take a picture if a shopper is looking at the screen for than 5 seconds
  • Fine-tune the GPT model against 100+ pitches and promotions to improve accuracy and model creativity

Built With

  • .netcore
  • azure
  • c#
  • functions
  • gpt
  • next.js
  • openai
  • vercel
Share this project:

Updates