HumanCraft

HumanCraft Thumbnail
Human beside a minecraft player
Punching a skeleton
Devin Contribution

Inspiration

We built HumanCraft because we thought putting real people into Minecraft would be really cool and really funny. There’s something ridiculous about seeing your actual friend standing in a world made of blocks, and we wanted to see if we could make it happen.

What it does

HumanCraft brings real people into Minecraft as colored 3D point clouds, with body tracking and hitboxes for their arms, hands, legs, and feet. Our initial focus is getting the person to render properly and making their hitboxes match their body, and then making interactions with how hitboxes move (such as doing a uppercut with your hands)

How we built it

We combine iPhone LiDAR and RGB capture with a Python pipeline for point-cloud reconstruction and body landmark detection. A custom Java Fabric mod takes that data and renders the person and their hitboxes inside Minecraft.

How we use Sentry

Our pipeline crosses phones, computer vision processing, networking, and a Minecraft mod. When a hand appears in the wrong place, it isn’t immediately obvious which part caused it.

We integrated Sentry across HumanCraft’s iOS capture app, Python FastAPI reconstruction service, and Java Minecraft mod. We used Issues and Logs to track errors and capture quality, and Traces to inspect reconstruction stages such as detection, calibration, fusion, and publishing. We also used the Sentry MCP to query unresolved issues, search logs, and inspect events and traces directly from our coding agent.

For example, when a reconstructed character appeared split in Minecraft, we used the MCP to connect the failure with upstream depth.missing, views_misaligned, and scan_quality_poor events. This helped us trace the problem back to missing depth data and misaligned scans, making it easier to debug across services before our final demo.

We go more in depth about how we used it and examples at around 2:05 in the video.

Cognition: best use of Devin

All of our team used Devin as our agent to develop HumanCraft (until we ran out of usage). The agentic work was run through the cloud, and we primarily used the modes Ultra and SWE-2 Max!

Devin contributed 33k lines of code.

Challenges we ran into

Calibrating the two phones to provide one cohesive point cloud of the human

Getting the cameras and Minecraft to agree on scale, position, and orientation is tricky. Small errors become especially noticeable around hands and feet. Keeping the point cloud and hitboxes synchronized adds another challenge: both can look reasonable individually but still represent different moments.

Accomplishments that we're proud of

Taking a funny idea and working through the engineering needed to make it possible. Connecting phone sensors, computer vision, and Minecraft modding gave us a reason to learn several things we probably wouldn’t have put together otherwise.

What we learned

Rendering a person and accurately locating their body are different problems. We learned how much calibration and timing matter, and how useful it is to follow a piece of data across the entire system when debugging.

What's next for HumanCraft

Smoother movement, more reliable hand and foot tracking, and multiple people in the same world. After that, we want to build minigames around it and see what ridiculous things our friends come up with.