Rainer spielt Schafkopf

Inspiration

Schafkopf is a really fun Bavarian card game. But to play it, you need at least three friends. If you don't have friends or they cannot play Schafkopf, you're out of luck. We introduce Rainer, a Schafkopf bot who can jump in for one of the players.

What it does

Rainer can recognize his own cards and the cards played by other players and play accordingly. A LED strip indicates which card he wants to play. In Schafkopf, trash talking is part of the fun, so likes Rainer to comment on the state of the game and sometimes tease the other players.

How we built it

We used hardware provided by Infineon, including cameras, ESP-32 microcontrollers and LED stripes. We designed rigs to hold the camera and rainers cards and 3D printed them. We used the openschafkopf library, which we adapted for our needs, to create the Schafkopf AI. We trained our own YOLO model to detect Schafkopf cards by creating a custom synthetic training data set. We used a pre-trained model to create and synthesis a selection of voice lines for Rainer.

Challenges we ran into

There does not seem to exist reliable object detection model for Bavarian playing cards. The openschafkopf library is extremely powerful but not well documented so we really had to dig into the code to adapt it for our needs.

Accomplishments that we're proud of

Although the limited time, the card object detection model we trained works very well and we are confident, we are able to further improve it in the future

What we learned

YOLO is really good at object detection, even when only trained on synthetic data.

What's next for Rainer spielt Schafkopf

We already started creating other characters including Markus Söder and we want to add them soon.

Built With

• c++
• python
• rust
• rvc-voice-synthesis
• yolo