HeritageArena

Inspiration Cultural heritage belongs to everyone, yet museum experiences rely heavily on visual observation, excluding visually impaired individuals and creating passive learning for general audiences. Meanwhile, centuries-old murals, frescoes, and sculptures face ongoing degradation from environmental factors. Inspired by the desire to merge adaptive technology with historic preservation, Tactile Heritage was created to transform static art conservation into an interactive, multi-sensory web experience—allowing users to feel, explore, and digitally restore ancient masterpieces.

What it does Tactile Heritage is an interactive WebXR and 3D web platform that enables users to inspect, repair, and study damaged historical artifacts.

3D & XR Restoration Canvas: Users interact directly with damaged 3D models (such as worn sculptures or faded frescoes) using gesture-controlled scalpels and digital brushes.
Generative AI Inpainting: When users brush over cracked or missing surfaces, an integrated image-to-image AI engine predicts and fills in lost details based on historical pigment and style contexts.
Multi-Sensory Feedback: Utilizes the Web Audio API and game-controller haptics to simulate the physical tactile sensation of brushing paint or chiseling stone.
Educational Layer Overlay: Toggles between degraded original states, AI-predicted restorations, and contextual annotations about historical techniques, making art theory accessible and engaging.

How we built it

Front-End & XR: Built using Three.js and the WebXR API to deliver smooth, cross-device 3D rendering and VR/AR support in the browser without requiring external installations.
AI Inpainting Engine: Integrated a Stable Diffusion / ControlNet API pipeline running on a lightweight Python backend to process texture regions selected by the user in real time.
Mesh & Spatial Processing: Used Open3D and Trimesh scripts on the backend to handle mesh segmentation, crack detection, and point-cloud alignments.
Audio-Tactile Experience: Structured custom sound synthesizers using the Web Audio API alongside spatialized audio triggers mapped to cursor coordinates.

Challenges we ran into

Latency in AI Inpainting: Running diffusion models in real time during continuous brushstrokes created lag. We resolved this by generating localized low-resolution preview patches first before rendering high-fidelity textures asynchronously.
Cross-Device WebXR Compatibility: Ensuring touch controls, mouse interactions, and VR motion controllers behaved identically across different browsers and hardware required extensive custom input abstraction layers.
Preserving Historical Accuracy: Preventing AI models from hallucinating inaccurate historic elements required constraining prompt engineering using curated metadata datasets from open museum archives.

Accomplishments that we're proud of

Seamless Web Performance: Successfully loaded and manipulated dense 3D photogrammetry models inside the browser while keeping frame rates stable at 60 FPS.
Accessible Multi-Sensory Design: Engineered a functional audio-tactile feedback loop that allows users to "feel" surface textures through spatialized sound cues and haptic patterns.
Bridging Art & Tech: Built an end-to-end working pipeline that respects art history conservation principles while leveraging cutting-edge generative tools.

What we learned

Multi-Sensory Modality: Discovered how combining spatial sound design with visual feedback dramatically increases immersion and accessibility for educational art platforms.
AI as a Collaborative Tool: Realized that generative AI works best in art restoration not as an automated replacement, but as an interactive "brush" guided by human intent and historical context.
Web-First XR Feasibility: Gained deep insights into building lightweight WebXR applications that democratize access to high-end 3D spatial tools directly in the browser.

What's next for Tactile Heritage

Haptic Hardware Integration: Expand support for physical haptic gloves and specialized micro-actuator styluses for enhanced accessibility for visually impaired museum visitors.
Community-Sourced Restorations: Allow art history students and conservators to upload custom photogrammetry scans, create guided restoration lessons, and collaborate in real time.
Publication & Expansion: Refine project documentation to submit a book chapter proposal for the Binnovative Innovation Book series and partner with local heritage non-profits to host virtual restoration workshops.