DungenDrive

An immersive hardware-software augmented tabletop adventure that transforms classic D&D into an interactive experience with smart figurines, sensors, AI narration, and real-world actions

Comment

Dungeon Drive: Bringing Dungeons & Dragons into the Physical-Digital World

Inspiration

Dungeons & Dragons is one of the most beloved tabletop role-playing games, but it often requires players to manually track character positions, movement, combat, and game state. We wondered: What if the physical board itself could become intelligent?

Inspired by the idea of blending traditional tabletop gaming with modern embedded systems, AI, and real-time tracking, we set out to create Dungeon Drive — a hardware-software augmented D&D experience where combine the advantages for game stat tracking on digital worlds with the hands-on-feeling of moving figurines.

Our goal was not to replace the tabletop experience, but to enhance it. We wanted players to keep the excitement of moving real miniatures on a real board while benefiting from intelligent automation, live tracking, and AI-powered assistance.

What We Built

Dungeon Drive combines physical hardware, embedded systems, and modern web technologies to create an interactive smart game board.

Players move magnetic D&D miniatures across the board. Beneath the board, a magnetic sensor continuously track their positions and transmit the information to a software platform that maintains a live digital representation of the game state.

The system includes:

  • 3D-printed D&D miniatures
  • Magnetic position tracking
  • Infineon TLx493D 3D magnetic sensors
  • XY gantry system with stepper motors
  • FastAPI backend
  • React frontend dashboard
  • AI-powered gameplay recommendations

The result is a "phygital" experience that bridges the gap between traditional tabletop gaming and modern interactive systems.

How We Built It

Hardware Layer

We designed a board capable of detecting the location of magnetic miniatures using Infineon's TLx493D magnetic sensors. Each miniature contains a magnet that can be detected and tracked in real time.

An XY gantry system driven by stepper motors enables future automated interactions and movement across the board.

Embedded Software

Using Arduino-based firmware and embedded C/C++, we developed software to:

  • Stream board state updates
  • moving the figurines
  • detecting player movement

The PSoC Edge AI Kit provides a platform for future edge-AI features such as intelligent move prediction and game-state analysis.

Backend

The backend was built using FastAPI and Python.

Its responsibilities include:

  • Receiving sensor updates
  • Maintaining the game state
  • Synchronizing player actions
  • Exposing REST APIs
  • Providing AI-generated recommendations

Frontend

The frontend was developed with React and Vite.

It visualizes:

  • Character positions
  • Monster locations
  • Board state
  • AI recommendations
  • Game events

This creates a real-time digital twin of the physical board.

Challenges We Faced

Reliable Position Tracking

Our biggest challenge was accurately tracking miniature positions using magnetic sensing. Small variations in magnet placement, sensor noise, and physical movement introduced inconsistencies that required extensive tuning.

Getting to work the old Gantry

One of our biggest challenges was getting the gantry system working reliably. Delays in receiving hardware, our limited experience with motion-control systems, and the need to reverse-engineer parts of the electronics significantly slowed development.

Hardware-Software Integration

Integrating embedded hardware with a modern web application proved more difficult than expected. We had to ensure that sensor updates propagated through multiple layers—from hardware to firmware, backend APIs, and finally the frontend dashboard—without introducing noticeable latency.

Mechanical Design

Designing and assembling the gantry system while keeping the board practical and portable required multiple iterations of both the mechanical and electronic components.

What We Learned

This project taught us valuable lessons about:

  • Embedded systems development
  • Sensor calibration and signal processing
  • Having a common ground :) (lost approx 4h due to that :/)
  • Full-stack web development
  • Hardware-software co-design
  • Rapid prototyping under hackathon constraints

Most importantly, we learned how powerful the combination of physical interaction and digital intelligence can be. Players naturally engage with tangible objects, and augmenting those interactions with AI and real-time software creates experiences that neither hardware nor software could achieve alone.

Future Work

Future versions of Dungeon Drive will include:

  • Automatic dice recognition using computer vision
  • AI Dungeon Master support (using edge-AI capabilities on the PSoC Edge platform)
  • Multi-player online synchronization
  • Additional smart miniatures and accessories
  • Automated monster movement through the gantry system

Conclusion

Dungeon Drive transforms a traditional tabletop game into an interactive smart gaming experience. By combining magnetic sensing, embedded systems, AI, and modern web technologies, we created a platform that preserves the magic of physical D&D while unlocking entirely new possibilities for gameplay.

We believe this is only the beginning of what hardware-software augmented tabletop gaming can become.

Built With

Share this project:

Updates