dota2 ai

Inspiration

Dota 2 bots usually rely on Valve's Bot Scripting API, which provides exact information about heroes, abilities, items, and the game world. We wanted to explore a different and more challenging approach: Can an AI understand Dota 2 using only what it can see on the screen, just like a human player?

This idea inspired us to begin building an external, vision-based AI that observes the game through screen capture instead of reading process memory or using internal game data. The existing Lua Dota 2 bot in the repository is not the main product; it serves as a reference for studying gameplay decisions such as farming, combat, ability usage, item usage, and team strategy.

What It Does

The project is an early-stage technical prototype for a vision-based Dota 2 agent. Its intended pipeline is:

Screen Capture -> Visual Perception -> State Estimation -> Decision-Making -> Action Output

The current prototype focuses on the foundations required for that pipeline:

• Detecting and capturing the Dota 2 window
• Calibrating HUD regions across different screen layouts
• Extracting visible information through OCR and computer vision
• Defining interfaces for object detection and tracking
• Combining uncertain observations into an estimated game state
• Recording data for offline evaluation and annotation
• Testing basic strategy and input abstractions

The project defaults to a safe dry_run mode, which reports proposed actions without controlling the keyboard or mouse.

How We Built It

We built the prototype primarily in Python and divided it into independent modules for screen capture, HUD analysis, OCR, object detection, tracking, state fusion, strategy, and input handling.

This modular architecture allows us to test each part separately before attempting complete gameplay automation. We also created tools for HUD calibration, observation recording, dataset preparation, offline policy replay, and quality evaluation.

The long-term decision system will adapt ideas from mature Lua bot strategies. However, those strategies cannot be copied directly because Lua bots have access to exact game-state APIs. They must be redesigned to operate on incomplete visual evidence and confidence scores.

Challenges We Faced

The greatest challenge is the difference between structured game data and raw pixels. Important information can be obscured by effects, camera movement, overlapping units, UI scaling, or resolution changes.

Reading small HUD values is also difficult. Health, mana, cooldowns, and item quantities change rapidly and may be stylized or partially hidden. Reliable object detection will require a representative dataset of Dota 2 gameplay images, which still needs to be collected, labeled, and evaluated.

Another major challenge is uncertainty. A visual agent cannot assume that every observation is correct. It must track confidence over time and avoid risky decisions when the available visual evidence is insufficient.

What We Learned

We learned that a visual game AI requires much more than a single object-detection model. Reliable perception depends on calibration, temporal tracking, OCR quality, state fusion, representative training data, and explicit handling of uncertainty.

We also learned that development must happen incrementally. Before building advanced gameplay strategies, we first need dependable screen capture, stable HUD recognition, useful datasets, and measurable perception quality.

What's Next

This project is still at an early stage and is not yet a complete Dota 2-playing AI. Our next milestones are to collect and annotate real gameplay data, improve HUD and object recognition, evaluate perception accuracy, and gradually connect reliable observations to increasingly capable decision-making policies.

Our long-term goal is to create an AI that can observe, understand, and eventually interact with Dota 2 through the same visual interface available to a human player.

Built With

• computer-vision
• dota
• jsonl
• mss
• ocr
• pillow
• pydantic
• pytest
• python