We are gamers who love streaming and watching high-level "Magic: The Gathering" gameplay on Twitch. One of the challenging things about streaming a game like Magic is having to explain a ton to get viewers to follow along. What are the actual cards being played on the table? What decks are the players playing? Even for regular fans of the game, it can be hard to understand what’s happening.

From a content creator perspective, we believe there are ways to empower ALL Magic streamers to make their channels more interactive and fun to watch. We also want to give them dynamic tools to develop solid streamer-viewer relationships that don’t feel forced, and is organic to what’s happening on-stream.

We know that new streamers can use some help to monetize their stream in ways that feel genuine. As a first step, we want to give viewers more interesting content to watch -- and eventually tie card purchases ON Twitch to what's happening, without viewers leaving the stream.

Bottom line – streamers can tell a better Magic story, and viewers can watch a better story!

What it does

Our Twitch extension uses a basic service for Magic cards object detection and identification, such that viewers can click on the physical outlines of cards on-stream to see what the cards do.

We have also built a Magic decklist upload option, so that content creators can display the hidden contents of a deck uniquely designed for a tournament.

To demonstrate all of this, we've recorded a Magic demo video on constant loop in our Twitch channel, so that the judges can view it 24/7. While the video runs, our image detection service will make detected cards “clickable.” There are also pop-ups on both sides of the screen to expose the players’ decklists.

We're more than happy to demo everything "live" with a physical Magic stream setup, if need be.

How I built it

Project consists of few layers:

  • Extension Backend Service (“EBS”) - this is the main backend for the extension. It keeps information about the decklists presented on-stream. It is responsible for card parsing from the text, and providing card image details. We wrote this in node.js and use Redis as a store. Hosted on AWS EC2.
  • EBS Pub/Sub service - passes information about the cards detected by image recognition service to all the viewers. Written in node.js.
  • Twitch extension - frontend application written in Preact (javascript). It presents information about the detected cards and decklists to all the viewers across the Internet.
  • Image recognition service (“IRS”) - this is the backend app responsible for detecting cards on-stream. We wrote the IRS in Python with ffmpeg and OpenCV. Hosted on highly efficient EC2 machines (c5n.large).

During the stream, the IRS constantly analyses the video stream in order to find videos. Algorithm consists of 3 steps:

  1. Preprocessing of video frame - noise removal, contrast enhancement, and more.
  2. Detecting possible cards positioning.
  3. Matching detected cards against Magic cards database.

Results over processing are send via EBS Pub/Sub to the extension, for all the viewers.

Challenges I ran into

Starting with OpenCV/Python for basic implementation and then converting it to read from the video stream, taking into account the delays in reading from stream. The last part was very tricky because Twitch is a global service, and video delays differ for each user. We had a breakthrough when we discovered that Twitch provides information to the extension in the form of the ‘hlsLatencyBroadcaster’ parameter -- allowed us to properly account for delays.

Accomplishments that I'm proud of

We’re super proud of the fact that we put several components together in limited time. We’ve learned to integrate with Twitch in several ways -- visual overlays, image recognition for a running stream, and so forth.

What I learned

We learned a lot about how generally image processing and image recognition works.

We also learned about, from the ground up, how to work with the Twitch extensions ecosystem: requirements, limitations, dev tools, working with Twitch API, the extensions review process.

What's next for Image recognition & card analysis for tabletop games

The next steps are to refine image recognition to be more accurate for a broader pool of cards, since we've only trained a small subset of cards for the demo. We need to refine the overall object detection to be stronger at recognizing the contours/shapes of cards, and this problem is magnified with playmats, dice, notepads, and other common objects on the stream.

As Twitch extensions support richer integrated purchasing capabilities (i.e. without leaving the Twitch channel), we would be looking to take advantage of new features. Building a shopping cart and checkout system within the extension would be super cool. This will help streamers monetize their streams in a way that feels genuine and not forced like ads.

We definitely want to "branch out" and provide image recognition and eventually integrated sales for other collectible trading card games, such as Keyforge, Yugioh, and Pokemon.

We want this product to be used for as many tabletop games as possible, so it’s also conceivable we could branch out into board games like “Settlers of Cataan” and use card detection for game pieces detection.

Share this project: