Introduction
We are a group of avid gamers who have noticed a dramatic shift in how modern games are designed and experienced. Games today are more immersive and visually diverse than ever, yet much of this immersion is focused on sight and sound, leaving other sensory cues (like scent) underexplored. Our project aims to address this gap by building a model that learns to associate “scent profiles” with specific game scenes. In other words, we want the model to look at a screenshot from a game and infer which scents might be present (e.g., “muddy,” “fresh,” “chemical,” “bloody”), and at what relative intensities. We view this primarily as a multi-label classification task, because the output is a normalized set of intensities across multiple scent categories. By predicting these scent distributions, we hope to lay a foundation for enhancing player immersion through more nuanced in-game feedback—potentially influencing how games could respond dynamically to a player’s sensory preferences.
Relative Works
In multi-label scene understanding and fine-grained image classification tasks, recent advances in deep learning architectures have proven especially beneficial. A key example is the Inception architecture introduced by Szegedy et al. (2015) in the paper “Going Deeper with Convolutions.” By stacking multiple filter sizes and pooling operations in parallel (the so-called “Inception modules”), their network captures features at various scales simultaneously, improving accuracy without significant computational overhead. This multi-scale approach has influenced numerous models aimed at handling complex scenes or objects—an essential aspect when assigning multiple overlapping labels (as in our project’s “scent” categories). In addition, the breakthrough work presented in “Deep Residual Learning for Image Recognition” by He et al. (2016) complements this perspective by addressing the degradation problem common in very deep networks; their introduction of residual connections enables more stable training and the successful scaling of network depth, further enhancing performance while maintaining efficiency.
DATASET
Our primary dataset comes from this Hugging Face repository, which originally contained 52,726 images from various video games. Each entry includes a screenshot and textual metadata (e.g., game name, captions, etc.). These images are diverse in content, spanning open-world titles, stealth games, fantasy RPGs, and more. Because not all images are equally informative (some are blurry or lack clear context), we perform a filtering step to remove low-quality screenshots. Afterward, we generate or refine captions and “scent” labels for each remaining image. The final dataset used for model training will include roughly 1,289 high-quality images, each with a detailed caption and a normalized scent distribution label (e.g., {“muddy”: 0.4, “bloody”: 0.3, …}). This labeled data then drives our multi-label classification experiments. Minimal additional preprocessing is required beyond our quality checks and label generation, so the dataset is otherwise used as-is, leveraging the diversity of its original game images to ensure broad coverage of potential “scents.”
Methodology
Training Dataset Preparation The original dataset consisted of 52,726 images. However, many of these images were uninformative—they were either too blurry or lacked sufficient contextual details. To address this, we applied a two-step filtering process:
Quality Screening：We first used a blurriness detection algorithm (based on Laplacian variance) and semantic evaluation via CLIP to discard images that were either too blurry or did not exhibit rich environmental details. This screening reduced the dataset to 1,289 high-quality images that met both clarity and content criteria.
Caption Generation and Scent Labeling: Since the GPT API does not process images directly, we decided to generate detailed captions for each of the 1,289 images using the BLIP model. For each of these images, we generated a detailed caption using the BLIP model, which transforms the visual content into descriptive text and then those captions are fed into ChatGPT with a carefully constructed prompt that instructs it to select relevant scent labels from a preset collection and assign normalized intensity scores (summing to 1) for each label, thereby producing our final labeled dataset.
Model Architecture and Training We use a pre-trained CNN model (ResNet50) to extract visual features from each game scene image. Rather than employing the conventional approach with Sigmoid activations (which would produce independent probability values that are not normalized), we modify the network’s final layer to output a normalized probability distribution over 21 predefined scent labels. This is achieved by replacing the final fully connected layer with a sequential block that includes a Softmax activation function. The Softmax layer ensures that the output is a 21-dimensional vector where the sum of all values is 1. Thus, each value directly represents the relative weight (or intensity) of the corresponding scent in the image.
Metrics
For our project, success is defined not only by correctly identifying the relevant scent categories but also by accurately estimating their relative intensities. In our case, “accuracy” is not sufficient on its own; we will also assess the fidelity of the predicted distribution against ground truth distribution by the following metrics (since our model outputs two types of values—a categorical label component and continuous intensity values—we will use a combination of metrics to comprehensively evaluate performance):
Categorical Metrics: We plan to use Top-k accuracy, binary precision, and recall to determine how well the model identifies the relevant scent categories. Continuous Metrics: We plan to use Cosine similarity and mean absolute error (MAE) to measure how closely the predicted intensity distribution aligns with the ground truth, along with KL divergence loss during training.
To thoroughly evaluate our model, we plan to run those experiments: We will try to test different network architectures and fine-tune their hyperparameters to get better results; We will also try to evaluate how different thresholds for converting continuous outputs to binary predictions affect precision and recall; We will include visual interpretability methods (i.e., Grad-CAM) to determine how the model makes its predictions.
Goals
Base Goal Develop and train a CNN-based model that, given a single game image, reliably predicts a normalized distribution over predefined scent categories (e.g., an output such as {'muddy': 0.4, 'bloody': 0.3, 'clean': 0.2, 'chemical': 0.1}). Success here is measured by high top-k accuracy, high cosine similarity, low MAE, and low KL divergence between predicted and ground truth distributions.
Target Goal Add visual interpretability using Grad-CAM or saliency maps to highlight which parts of the image contribute to each scent so that it can provide insights into why the model made its predictions.
Stretch Goal Implement a user-feedback loop where natural-language scent preferences (e.g., “avoid chemical smells”) are accepted and processed by a lightweight LLM (such as GPT-3.5) to adjust the predicted scent distribution. This personalized prediction capability will further align the model’s output with individual user preferences, making the system more adaptable and user-centric.
Ethics
What Is Your Dataset? Are There Any Concerns About How It Was Collected, or Labeled? Is It Representative? What Kind of Underlying Historical or Societal Biases Might It Contain? Our dataset started with 52,726 game scene images from Hugging Face, which were then filtered down to 1,289 high-quality images based on clarity and rich content. The labeling process relies on generated captions (using the BLIP model) and subsequent scent label selection via ChatGPT. And there are potential biases contained in this dataset: It may contain Collection Bias because the original image dataset may over-represent certain game genres or art styles (e.g., action, fantasy, or photorealistic settings), while under-representing others like stylized, abstract, or indie game visuals. As a result, the trained model may perform better on specific types of images and fail to generalize across the full spectrum of game scenes; It may also contain Labeling Bias because the scent labels are generated from image captions produced by a visual-language model (BLIP) and interpreted by a language model (ChatGPT). Any inaccuracies, limitations, or stereotypes present in these models such as misinterpreting visual elements or associating certain scenes with stereotypical scents may propagate into the final dataset and affect the model's predictions.
Who are the major “stakeholders” in this problem, and what are the consequences of mistakes made by your algorithm? The key stakeholders include game developers, content creators, and ultimately the players. Game developers and designers might use the model to enhance immersion by integrating realistic scent profiles in virtual environments. If the algorithm makes mistakes—such as mispredicting the scent distribution or missing important sensory cues—it could lead to a less engaging or even jarring user experience. For example, if a scene that is meant to evoke a “grassy” and “fresh” atmosphere is mistakenly labeled with “chemical” or “burnt” scents, it might disrupt the game’s immersion and affect user satisfaction. In a broader context, any inaccuracies could also impact the credibility of the technology when integrated into more advanced, immersive experiences.
Log in or sign up for Devpost to join the conversation.