Inspiration and the Project

The four of us are veteran players of the classic tabletop RPG, Dungeons & Dragons. As any D&D player knows (or will soon discover), it's all about the DICE. We love our click clack math rocks, coming in 6 different varieties including the iconic d20 as well as the more universal six-sided-die. This is one of the most popular games in the world, and has been evolving and growing since it was created in the 1970s. Upon hearing the theme "retro," our immediate thought was to do something D&D-related, and we are excited to take part in this trend of adding content and cultivating the best aspects of different styles of The World's Greatest Roleplaying Game.

Since in-person gatherings were rapidly diminished last year and continue to be discouraged, our D&D campaigns have shifted to an online setting. This has had some benefits, mostly including the conveniences of the virtual character sheet; when making a roll for a certain action, ordinarily a player would look in their paper character sheet for the desired stats to be sure they roll the correct set of dice, and they must again reference the sheet for any modifiers to the result. The typical online version allows simply choosing the action and clicking a button; the server handles everything else, including generating random numbers for the roll itself.

A big downside of the online experience is being unable to use our dice! We came up with this project as a way to get the best of both worlds. A player chooses their action, the mobile web client prompts them with the exact number and type of dice to roll, and the dice tray is recorded and screen-shared with the rest of their adventuring party. We then use computer vision to automatically identify the results of the dice roll, and add any necessary modifiers from the player's character sheet. We tried to do something involving CV with dice rolls at last year's Hacklahoma, with Dungeons and Mad Libs, but we were not able to do much, and were not satisfied with it. We've learned a lot since then, so we decided to start from scratch with the dice CV and tackle a much more in-depth project!

We also wanted to interface with the technology people are already using, rather than adding many extra steps to the process. The most common way that online D&D players make and update their character sheets is using the free website D&D Beyond. Our project can be run on a mobile or desktop browser, and uses a webcam connected to the computer. We wanted to get this working with a phone camera as well, but did not have time to do this alongside finishing the development of the main app. When logging into the site, a user's ID will be saved and tied to a few bits of information they initially provide, such as their name and a link to their D&D Beyond profile. This means the player can then easily use our sleek interface to initiate any type of roll their character can make, and they get to roll their physical dice while our app does all the character-specific calculations for them!

Note about viewing the site

For now, it can be accessed here, but we will get a more dedicated URL in the future.

This site is designed to work on mobile. To get the best experience when viewing it on a computer, press Ctrl+Shift+M to enter Responsive Design Mode. This will format the site to look as it would on a smartphone. At the top, select "iPhone X/XS iOS 12" to make sure you see it the way we designed it!

DnD Crash Course:

If you’ve never played Dungeons & Dragons, this project description may be a little confusing at times. The gist of the game is that each player has a character sheet with all the information about their character. This character is someone the player has made, and is sort of their avatar in the fantasy world of the campaign. One player serves as the Dungeon Master (DM), and rather than having one character, they control the monsters, NPCs, and everything else about the world. A campaign is a series of sessions in which the DM creates stories, and the players roleplay, fight, explore, and improv their way through.

Throughout all these different encounters, the players and DM will be rolling dice! D&D 5th edition uses 6 different dice, represented by the number of faces: d4, d6, d8, d10, d12, and d20. The character sheet has info including ability scores, skill checks, and proficiency bonuses. These are the simplest ones, but it can get complicated. There’s a lot of math in D&D! If a player with high charisma (let’s say +4 CHA) is trying to tell a lie to an NPC, they may roll a Deception check, which is 1d20 plus their charisma modifier. If they are proficient in Deception, they would also add their proficiency bonus. So, all in all, they may have a result of 1d20 + 4 + 2. You can see a lot goes into this. That’s why we’re trying to help players avoid doing this manually every time!

How we built it: Overview

This project is a hodge-podge of different components that we were each interested in working with. We setup a Python server using socket.io and Flask, which allows us to interface with both the client and our computer vision (CV) module. The server is hosted in an Ubuntu 20.04 VM on Google Cloud Platform. The client is a JS/TypeScript module running React, which provides our HTML interface functionality by setting off socket.io events. These events involve updates to the player/campaign lists (via join/disconnect), requests for information to display (i.e, roll history, player list), and the more complex roll initiations. The CV module receives a stream of images from the client’s camera, and runs OCR to identify results of dice rolls. To stream these images, we use a similar software that the Octopi uses for monitoring our 3D printer; we essentially update a local site to replace a still-frame image whenever the camera sends a new one to the client, and the CV module checks this website constantly while searching.

Our main goal with the interface was to have expected information readily available, and nothing more. We carefully storyboarded all the different screens of our app in order to make sure it's never too busy, and made sure to keep in mind that the game is not being played exclusively through our app. This is a supplement intended specifically to help with dice rolls, and we often had to reel our scope back in to keep it manageable. As such, the dice rolls are the main focus, and the main screen shows only the players and the active roll. Our "character sheet" page contains not the full character sheet, but only those aspects which can be directly pressed to initiate a roll.

We started out by each building our modules individually (the front-end static display, the back-end server, the React client, and the OpenCV dice reader), and then some of us continued adding features while the others worked on netting everything else together.

How we built it: Importing Character Sheet

When a player joins, they get a unique ID. We map this ID to other features of the player, such as the campaign they have joined, their name, and a link to their D&D Beyond profile. With this link, we run a custom script that grabs all the information we need from their character sheet and passes it back to the client as a JSON object. We need only keep information that affects dice rolls, so all we need are the ability scores, skill and saving throw proficiencies, and special class- or race-specific modifiers. Anything extremely specific the player needs can be done within our app by setting up presets; a player may have a custom-made weapon dealing 2d6+5 damage that we as developers have no way of knowing about, but a player can easily scroll down and create a named preset for this roll, and can simply tap the button for this roll in the future.

How we built it: Updating the Displays

Requests for information work on what we think of as a "leaderboard system," although there is no actual score in D&D. Our two "leaderboards" are the player list and the roll history. The player list displays on the main page, and shows all players in the current campaign. When a player joins or disconnects, this is automatically updated. A cool aspect that we're really proud of is the player avatar system; when they enter their character sheet URL, we also snag a link to their character's portrait they've set in D&D Beyond, and use this as their player icon in the app. We usually draw or search thoroughly for our character portraits, so it's cool to see them pop up on the app without having to upload an image to our server. The roll history is a running list of all roles that have taken place so far, and this is displayed on the main screen underneath the camera feed. This is a cool way to look back on several rolls done in a row, and to go check anything that has previously happened in the game. Each roll is shown alongside the name of the player who made it and the type of roll done.

How we built it: Dice Rolls

Initiating a roll requires much more back-and-forth between the modules than the previously mentioned events. 1) The user tells the client the roll they want to make. 2) The client (which stores the character sheet) identifies which dice need to be rolled and what the modifier is. 3) It then directs the user to make the roll, and triggers a server event that tells the CV module to begin looking for dice with the camera. This event also brings important information to the server, such as the roll purpose and the modifier. 4) As it initiates the roll, the server changes every player's display to show the camera feed of the player currently rolling their dice, and it gets ready to update everything when the roll result is detected by the CV and sent back. 5) The server uses the result from CV and the provided modifier to calculate the final result, and it then returns this to the client. The server also stores all the information for that roll in the roll history, and updates that display accordingly. 6) When the roll has finalized, no camera feed will be shown, so the display must change back to showing the "waiting on a roll" screen.

The most complicated part of this process is the computer vision (CV). A stream of images at 1Hz are sent from the user’s camera through to the CV module, which can then use OpenCV on each still image to perform canny edge detection. We check adjacent frames for similarity to ensure the dice have come to rest, and we also zoom in on the area the dice occupy to improve performance. We tried out using both Tesseract and Google Cloud Vision AI for their Optical Character Recognition (OCR), and determined that Google’s is far more effective at identifying the numbers on the dice. However, we want our module to only identify the number on the main, top face, and ignore all other faces. Google’s Vision AI is extremely effective, and in fact sees more numbers than we want. For this reason, we kept Tesseract as our main OCR method.

We improved our accuracy in this area further by intentionally lowering the resolution, which helps our code to focus on the top face of the die, since it becomes further unable to read the numbers on side faces. This code works for single dice as well as multiple dice being rolled together at once.

Challenges we ran into

By far our hardest conceptual problem relates to the difficulty of reading dice values in the CV module. It at first seemed a simpler task, since OpenCV allows us to clearly and easily find edges and contours. Even when these are clean and precise, however, we have loads of trouble actually identifying the characters these contours represent. This became even harder when multiple different types of dice were rolled at once. Using OCR, we could tweak our thresholds to be able to identify and read the numeric values off a rolled die fairly accurately, but there were many more constraints than we initially realized. The performance changed as it got later in the day, since the sun setting influenced the light levels in the room. Different sets of dice have different colors, patterns, and font sizes, so it is hard to develop a standard.

We wanted to make this an app that could be easily accessed by people in different locations, and have an online template that can be easily printed out and assembled into a uniform, predictable dice tray. This would enable anyone to reliably setup the software to work, as the template would have included somewhere to set the phone or webcam to best view the dice. As it stands now, there is so much unpredictability in the noise, exposure, and contrast caused by differences in one environment, so getting this to work with any camera in a huge variety of environments turned out to be basically impossible. This has been a really fun personal project for us, so we are excited to keep chipping away and thinking about ways to make this more universal and less of a novelty.

Accomplishments that we're proud of

We felt the most proud around 2:30 AM when we first accessed the server independently, pasted our character sheet URL, and saw all of our character's stats and photo pop up on the screen almost immediately. This meant our client successfully sent information to the server, which used that to pull data from D&D Beyond, and sent it back to the client, who then updated all the displays correctly.

Getting the character sheet loaded in was certainly not the part we were most worried about while brainstorming at the beginning, but it ended up being a long struggle with so many different moving parts that we were ecstatic to finally reach a point where even if nothing else gets achieved, we had at least made a neat mobile-friendly web app that can display important aspects of any character sheet.

What we learned

We learned a lot during the process of working on this project. The things we learned were not only technical skills, but abilities like forseeing areas of difficulty, preserving a manageable project scope, and prioritizing the common case. In our past couple of years at Hacklahoma, we’ve scrambled to settle on a project idea, and end up randomly working on scattered, massive projects, resulting in a huge grind to get anything submitted. This time, we spent the whole first hour just discussing our ideas, setting up a shared Miro page for whiteboard and storyboarding, mocking up the different page layouts, and figuring out who would work on which component. This meant we could jump straight in and get to work on one ambitious project, rather than several foolishly impractical ones. Often, we had cool ideas for additional features, or were stuck on one nonessential aspect for a long time, and we practiced restraint by holding fast to our original concepts, and favoring time spent on essential, basic tasks over spur-of-the-moment whims. This was certainly a strange hackathon experience, being our first virtual one, but we came away having learned and grown a lot, and having made a project we were genuinely excited to get working from the very beginning.

What's next for Fantasy Dice

We had many ambitions with this project, and had to really push ourselves to keep our scope small enough to be achievable. We'd like to tune the computer vision to work in a wider variety of environments/conditions and be less finicky. A feature that works some but not all of the time is when a roll contains multiple different types of dice, such as a d6, d10, and d20 all rolled at once. We didn’t have time to work to perfect this behavior, and decided to hold off on pursuing it until we can make the CV more stable.

As the app stands now, the webcam feed is hosted on our local wifi network, rather than on the web. This means for us in our house, the video sharing works fine, but it does not yet share the feed across the internet. We unfortunately did not have time to develop both the main server and the video-handling server, so moving forward we would like to get this working as well. We will look into hosting them both in the same VM, but suspect we will need two separate web servers to make it streamlined.

As we’ve mentioned throughout this project page, we really envisioned this as a universally accessible tool that works on any smartphone, and we were sad not to be able to integrate the phone camera itself. We made some progress on this front towards the end of the hackathon, when we finally discovered how to access the iPhone camera from within a browser, but we decided to work on improving the rest of the project and keep it running from a webcam for now. Our top priorities moving forward will be improving the CV and adding phone camera support.

We'd also like to implement some other quality-of-life features, such as a turn counter which shows on the player list who's turn it is, and displays them in initiative order. This didn't get implemented during the hackathon since we couldn't decide how to handle the DM's monsters and NPCs having places in the order but not appearing in the app, but we believe with fresh eyes and less impending deadlines we could make it work. We similarly wanted to dive deep into the weeds of character sheets and D&D rules and exceptions, but had to limit ourselves to the common cases. It would be neat to expand this over time, especially tuning it to specific classes and characters we ourselves may be playing.

We have so many more ideas for this project, and we’re just getting started!

Share this project:

Updates