SimbaGPT

We turn digital screens into intelligent ad platforms using machine learning, enabling real-time, hyper-targeted content delivery that boosts engagement and ROI across urban and retail spaces.

Comment 1

🗺️ Project Story: Location-Aware Intelligent Advertising with Machine Learning

🚀 About the Project

Our project transforms public digital screens into intelligent, location-aware advertising platforms using machine learning, markerless motion capture, and Google Maps Platform APIs. Inspired by the underutilization of physical ad spaces across urban Africa, we set out to create a solution that responds to the real-time environment, audience dynamics, and location context—delivering smarter ads and higher engagement.

🔍 What Inspired Us

Billboards and public screens are everywhere, yet most play content without context—regardless of the time of day, local events, or who’s watching. We asked: What if outdoor screens could think like online ads? That question led to SimbaGPT, a system that brings hyper-local intelligence to physical media using maps, computer vision, and ML.

⚙️ How We Built It

We used the following components:

  • Google Maps Platform for location data, geofencing, and contextual triggers (e.g., traffic, places, weather overlays).
  • Luxonis OAK-D PoE Cameras and Moverse markerless motion capture for real-world visual sensing.
  • ML models trained on behavioral data to choose the most relevant ad based on audience density, time, and location.
  • A backend system that combines Maps API outputs with camera data to make real-time ad decisions.

Mathematically, we approached ad optimisation as a contextual multi-armed bandit problem:

$$ \text{Ad}_{t} = \arg\max_a \left( \mathbb{E}[r_t(a) \mid x_t] + \beta \cdot \sqrt{\frac{\ln t}{n_t(a)}} \right) $$

Where $r_t(a)$ is the expected reward for ad $a$, and $x_t$ is the location + visual context.

💡 What We Learned

  • Location APIs can unlock new dimensions of personalisation in offline spaces.
  • ML inference must happen at the edge to ensure low latency.
  • Map-based insights (e.g., peak hours, popular venues) are critical for predicting audience intent.
  • Creatives perform better when adapted not just to the user, but to the place.

🧗 Challenges We Faced

  • Balancing privacy with perception: we ensured all sensing is anonymised and GDPR-compliant.
  • Delays in procuring specialised hardware impacted live deployments.
  • Training models to adapt across different regions, languages, and cultures took time and iteration.

🌍 Why It Matters

Our platform enables real-time, intelligent storytelling in physical spaces, combining the power of maps, AI, and motion. It can serve municipalities, retail networks, transportation hubs, and event venues — anywhere dynamic, location-relevant messaging matters.

We’re currently in early-stage rollout across Nairobi, Cape Town, and Kigali, with cloud backends hosted on Paperspace by Digital Ocean and integrated with Google Maps APIs during live campaigns and Google Adwords for analytics.

Built With

  • c++
  • custom-ad-serving-api-*-**browser-interface**:-rendered-through-google-chrome-###-?-v2.0-?-current-system-*-**edge-inference**:-nvidia-jetson
  • depthai-sdk-*-**orchestration**:-docker
  • firebase-*-**apis**:-google-maps-platform-(maps-javascript-api
  • flask-*-**cloud-services**:-digitalocean-(compute
  • geocoding-api)
  • google-cloud-functions-*-**apis**:-google-maps-platform-(places-api
  • javascript-*-**frameworks**:-tensorflow
  • kubernetes-*-**monitoring**:-grafana
  • luxonis-oak-d-cameras-*-**vision-ai**:-markerless-human-pose-estimation-via-moverse-studio-*-**languages**:-python
  • node.js-*-**cloud-services**:-google-cloud-platform-(bigquery
  • prometheus
  • react-(ui)-*-**frameworks**:-pytorch
  • reflecting-the-evolution-from-v1.0-to-v2.0:-##-?-built-with-###-?-v1.0-?-initial-deployment-*-**video-processing**:-nvidia-deepstream-sdk-*-**hardware**:-hikvision-ptz-cameras-with-25x-optical-zoom-*-**languages**:-python
  • roads-api)
  • storage)
  • vertex-ai)
Share this project:

Updates

posted an update —

Contextual Bandit: Code Example (Epsilon-Greedy)

import numpy as np

class ContextualBandit:
    def __init__(self, n_actions, n_contexts):
        self.n_actions = n_actions
        self.n_contexts = n_contexts
        self.Q = np.zeros((n_contexts, n_actions))  # Estimated rewards
        self.N = np.zeros((n_contexts, n_actions))  # Visit counts

    def select_action(self, context, epsilon=0.1):
        if np.random.rand() < epsilon:
            return np.random.randint(self.n_actions)
        else:
            return np.argmax(self.Q[context])

    def update(self, context, action, reward):
        self.N[context, action] += 1
        alpha = 1 / self.N[context, action]
        self.Q[context, action] += alpha * (reward - self.Q[context, action])

# Simulated environment
def generate_reward(context, action):
    # Toy reward matrix: rows = contexts, cols = actions
    true_rewards = np.array([
        [0.1, 0.9, 0.2],
        [0.8, 0.1, 0.4],
        [0.3, 0.3, 0.7]
    ])
    return np.random.rand() < true_rewards[context, action]  # Bernoulli trial

# Setup
n_contexts = 3
n_actions = 3
bandit = ContextualBandit(n_actions, n_contexts)

# Run simulation
n_rounds = 10000
rewards = []

for _ in range(n_rounds):
    context = np.random.randint(n_contexts)
    action = bandit.select_action(context, epsilon=0.1)
    reward = generate_reward(context, action)
    bandit.update(context, action, reward)
    rewards.append(reward)

print(f"Average reward over {n_rounds} rounds: {np.mean(rewards):.4f}")

Notes:

  • This example uses a toy reward function to simulate different contextual response probabilities.
  • The bandit learns which action performs best for each context.
  • The exploration is controlled using the epsilon parameter (epsilon-greedy).

Log in or sign up for Devpost to join the conversation.

posted an update —

Contextual Bandit Theory

What Is a Contextual Bandit?

A contextual bandit is a machine learning model used to make sequential decisions where side information (context) is available before choosing an action. It balances exploration (trying new actions) and exploitation (choosing the best-known action), conditioned on observed context.

Formal Definition

At each time step $t = 1, 2, \dots, T$:

  1. Observe context $x_t \in \mathcal{X}$
  2. Choose action $a_t \in \mathcal{A}$
  3. Receive reward $r_t(a_t)$
  4. Update policy to improve future choices

The objective is to maximise cumulative reward or minimise regret over time.

Why Context Matters

Unlike traditional multi-armed bandits, contextual bandits condition decisions on environmental signals or user data, enabling personalised recommendations or targeted actions.

Example: In online advertising:

  • Context: User profile, location, device
  • Actions: Ads to display
  • Reward: Click or conversion

Algorithms

1. Epsilon-Greedy

  • Explore randomly with probability $\varepsilon$, otherwise exploit best action.

2. Upper Confidence Bound (UCB)

  • Choose:

$$ a_t = \arg\max_a \left( \hat{r}_t(a) + \beta \cdot \sqrt{\frac{\ln t}{n_t(a)}} \right) $$

  • Balances estimated reward and uncertainty.

3. Thompson Sampling

  • Use Bayesian posterior over reward models.
  • Sample from the posterior to guide exploration.

4. Policy Gradient (Neural Bandits)

  • Approximate a policy $\pi(a \mid x)$ using a neural network.
  • Update parameters based on observed reward and gradient feedback.

Regret Analysis

Regret quantifies the loss due to not always choosing the optimal action:

$$ \text{Regret}T = \sum{t=1}^T \left[ \max_{a \in \mathcal{A}} \mathbb{E}[r_t(a) \mid x_t] - \mathbb{E}[r_t(a_t)] \right] $$

Goal: Keep regret sublinear, e.g. $\mathcal{O}(\sqrt{T})$, so average regret tends to zero over time.

Applications

  • AdTech: personalised ad targeting
  • News & media: content recommendations
  • Healthcare: adaptive clinical trials
  • Robotics: sensory-based action selection
  • E-commerce: price and product selection

Theoretical Foundations

  • Online Convex Optimisation
  • PAC Learning Theory
  • Bayesian Decision Theory
  • Reinforcement Learning (simplified case)

Related Concepts

  • Reinforcement Learning (RL): Contextual bandits are RL problems with a single state and no long-term planning.
  • Causal Inference: Bandits operate under a causal action → reward assumption.
  • Meta-Learning: Improve learning efficiency using past contextual interactions.

References

  • Lattimore, T., & Szepesvári, C. (2020). Bandit Algorithms
  • Agarwal, A., et al. (2014). A Theory of Contextual Bandits with Applications to Personalized Learning
  • Russo, D. et al. (2018). A Tutorial on Thompson Sampling
  • Sutton, R., & Barto, A. (2018). Reinforcement Learning: An Introduction

Log in or sign up for Devpost to join the conversation.

posted an update —

Sources & References

Core Technologies

Maritime Logistics

Aviation APU Optimisation

Autonomous Vehicles and Waypoints

Case Study & Demonstration

Log in or sign up for Devpost to join the conversation.

posted an update —

System Architecture

Below is the high-level architecture of the motion capture solution, combining edge AI processing, cloud storage, and a browser-based playback interface.

System Architecture

Key Components

  • Capturing Setup: Utilises multiple Luxonis OAK-D cameras to record human motion markerlessly from multiple angles.
  • Local GPU Processing: Real-time inference happens on an edge workstation powered by NVIDIA DeepStream and Jetson SDKs.
  • Cloud-based AI Processing & Storage: Captured data is uploaded and archived for further training, analysis, or remote access.
  • Browser-Based Review & Playback: Users can interact with sessions remotely using a Chrome-based web UI.
  • Flexible Equipment Integration: Non-restrictive to specific hardware—compatible with widely available peripherals and sensors.

This architecture supports low-latency motion capture workflows suitable for animation, biomechanics, and smart city surveillance infrastructure.

Log in or sign up for Devpost to join the conversation.

posted an update —

The Breathing Billboard – Real-Time Environmental Messaging Powered by AI

As a live demonstration of SimbaGPT’s motion capture and contextual intelligence capabilities, the Breathing Billboard project showcased the impact of workflow optimisation in public communication. In partnership with Dentsu, UNEP, and Safaricom, the system visualised real-time air pollution levels across Nairobi using adaptive digital billboards that changed colour based on PM2.5 data.

Rather than displaying scientific units like “µg/m³,” pollution levels were translated into intuitive colour gradients — from green (safe) to brown (hazardous) — providing immediate health cues to pedestrians, athletes, and commuters. These billboards were placed in high-traffic zones to maximise visibility and behavioural impact.

Cross-Industry Integration

  • Health & Environment – UNEP
  • Telecom & IoT Infrastructure – Safaricom
  • Creative Advertising & Visualisation – Dentsu
  • AI & Geospatial Intelligence – Rydlr / SimbaGPT

This initiative proved the viability of fusing Google Maps Platform APIs, AI-driven pose estimation, and real-time sensor data to deliver hyperlocal, context-aware messages. The result: accessible environmental data for millions, powered by a scalable ML pipeline built for responsiveness and clarity.

Demo Video: Watch on YouTube

This project now informs future deployments of interactive billboards across Africa — expanding from environmental use cases to commerce, education, and transportation.

Log in or sign up for Devpost to join the conversation.

posted an update —

SimbaGPT v2.0 – Markerless Motion Capture Meets Smart Advertising

We’ve officially transitioned from prototype to production-ready.

V1.0 of SimbaGPT relied on a combination of Hikvision PTZ cameras and OBS Studio, enabling basic crowd interaction through motion-based overlays. It served as our initial proof-of-concept for integrating physical audience behaviour into dynamic, screen-based advertising.

Today, with V2.0, we're introducing:

  • Markerless motion capture using Luxonis OAK-D PoE cameras
  • NVIDIA Jetson-accelerated edge inference for ultra-low latency ad switching
  • Google Maps Platform APIs for contextual, location-based engagement
  • A proprietary ML pipeline for real-time human pose estimation and attention tracking
  • Integration with Rydlr’s Ad Decision Engine for adaptive campaign delivery

Our solution no longer requires wearable sensors — we use computer vision alone to understand audience behaviour and trigger content transformations on public-facing screens. It's real-time, scalable, and geographically aware.

Built initially with NVIDIA DeepStream (V1.0), the system now supports Jetson NX and OAK edge hardware, making it deployable in urban mobility hubs, retail stores, and sports arenas.

Deployed across Nairobi, Cape Town, and Kigali, SimbaGPT is redefining out-of-home advertising in emerging markets through intelligent interactivity.

Created by: Ted Iro Opiyo
Built with: Google Maps Platform APIs, Luxonis OAK-D PoE, Jetson Xavier NX, NVIDIA DeepStream, Digital Ocean, Python, ONNX, React
Organization: Rydlr Cloud Services Ltd.

SimbaGPT #MachineLearning #GoogleMapsPlatform #ComputerVision #MotionCapture #Devpost

Log in or sign up for Devpost to join the conversation.