Cultura

Cultura is an AI-powered, real-time voice-interactive 3D platform that transforms indigenous cultural heritage into immersive, living, two-way conversations — making cultures alive for digital natives

Comment

Inspiration

The world is losing its cultural memory faster than we can document it.

Over the past decade, the rapid advancement of Generative AI and interactive multimedia has fundamentally transformed the paradigm of human–computer interaction. Projections from Gartner (2022) forecast a sharp global increase in the adoption of conversational AI, signalling an irreversible transition from passive visual interfaces toward dynamic, two-way voice-based interaction systems. Yet despite these technological breakthroughs, one of humanity's most irreplaceable assets indigenous cultural heritage is disappearing at a rate that demands urgent, scalable intervention.

UNESCO's Atlas of the World's Languages in Danger, supported by youth demographic data from Indonesia's Badan Pusat Statistik (BPS), confirms that countless local languages and cultural traditions are experiencing a critical decline in everyday family use. The evidence is unambiguous: digital globalisation is actively eroding the intergenerational transmission of indigenous identity.

The root cause? Cultural materials are still overwhelmingly delivered through passive formats text, video, and static museum displays formats that are fundamentally misaligned with how digital natives learn, engage, and retain information. According to Edgar Dale's Cone of Experience, supported by modern e-learning research, memory retention from passive methods sits at a mere 10–20%. In stark contrast, immersive, participatory simulations push that retention figure to 75–80%. These findings are further reinforced by landmark studies from Thomas M. Connolly et al. (2012) and James Paul Gee (2003), which conclusively demonstrate that digital interactive environments significantly enhance both learning motivation and long-term engagement.

This is the crisis that Cultura was engineered to address.

As Software Engineering students with a profound personal connection to the Minangkabau heritage of West Sumatra, Indonesia, we witnessed first-hand how rich philosophical traditions the "Adat Basandi Syarak, Syarak Basandi Kitabullah" principle, the matrilineal kinship system, and the architectural legacy of heritage sites like Istano Basa Pagaruyung were fading from the daily lives of younger generations. We recognised that the challenge was not a lack of cultural knowledge, but a fundamental failure of the medium through which that knowledge was being delivered.

We asked ourselves a defining question: What if you could not just read about a culture, but actually have a conversation with it?

Cultura is our answer: an AI-powered, real-time voice-interactive 3D cultural exploration platform that transforms passive, one-directional learning into a living, breathing, two-way dialogue with culture itself. While our inaugural proof-of-concept focuses on the Minangkabau culture of Indonesia as a deeply researched and contextually rich starting point the platform's architecture is engineered from day one to scale across any indigenous heritage worldwide.

By converging Generative AI, real-time voice processing, Natural Language Processing, and immersive 3D character technology, Cultura does not merely present cultural information it embodies it, making cultural education as natural, engaging, and impactful as a conversation with a wise elder.

What it does

Cultura is a real-time, AI-driven immersive cultural exploration experience. At its core, users engage in fully natural, two-way voice conversations with Datuak Rajo Alam a richly animated 3D virtual character powered by Conversational AI set inside an authentic digital recreation of a Rumah Gadang, the iconic traditional longhouse of the Minangkabau people of West Sumatra, Indonesia.

The platform is built on a single, powerful premise: cultural knowledge is best transmitted through dialogue, not documentation.

Core Features

Real-Time Conversational Voice Interaction Users speak naturally in their own words. The virtual cultural guide listens, comprehends, and responds in real-time through a fully integrated Speech-to-Text (STT) → Natural Language Processing (NLP) → Text-to-Speech (TTS) pipeline, powered by the Convai conversational AI engine. There are no menus to navigate, no multiple-choice prompts just a genuine, free-form conversation.

Deep Generative AI Cultural Knowledge Base The AI engine is trained on a comprehensive, curated Minangkabau cultural knowledge base encompassing traditional customs (adat), philosophical principles, matrilineal kinship structures, traditional architecture, regional cuisine, performing arts, language, and historical narratives. The system is capable of understanding contextual questions and delivering nuanced, culturally accurate responses not just keyword-matched answers.

Synchronized 3D Character Animation Datuak Rajo Alam delivers every response through a fully synchronized combination of natural audio output, real-time lip-sync animation, contextual facial expressions, and purposeful body gestures creating the immersive sensation of conversing with a real, living cultural guide rather than interacting with a chatbot interface.

Guided Immersive Cultural Exploration The virtual character actively navigates users through the Rumah Gadang environment, explaining the cultural significance, architectural meaning, and social function of each space from the central hall to the private bilik rooms weaving historical context and living tradition into every step of the journey.

Fully Modular Cultural Framework Architecture The platform's most strategically significant feature is its modular knowledge base layer. Every cultural dataset, every character identity, and every linguistic context is abstracted from the core system architecture. This means the same proven technology can be redeployed for any new culture simply by configuring a new knowledge base and character model without rebuilding the underlying system.

Intuitive User Interface The experience is designed for accessibility across all demographics, featuring a clean Main Menu with Start and Exit navigation, a Culture Selection Menu for future multi-culture browsing, a fully functional Pause Menu (Resume, Restart, Quit), and an unobtrusive real-time dialogue interface that keeps the immersive environment at the forefront.

The Paradigm Shift

Cultura moves cultural learning from passive consumption to active participation a transformation supported by decades of educational psychology research. When a user asks Datuak Rajo Alam about the significance of a Rumah Gadang's curved roof, they are not reading a paragraph they are engaging in the same kind of knowledge transfer that has preserved culture across generations: human conversation.

How we built it

Technology Stack

Languages:

  • C# Primary scripting and application logic language within the Unity 3D environment, used for all game systems, API integration management, UI logic, and voice interaction pipeline orchestration.

Frameworks & Libraries:

  • Unity 3D The core real-time 3D interactive application engine, selected for its robust capability in handling advanced 3D rendering, character animation systems (including real-time blend shape and skeletal animation), spatial audio management, scene management, and flexible external API integration all critical requirements for AI-driven interactive development.
  • Convai SDK for Unity The official integration library that creates a direct, low-latency bridge between the Convai Conversational AI cloud platform and the Unity runtime environment. This SDK is the technical backbone enabling real-time character responsiveness to voice input.

Platforms & Cloud Services:

  • Convai Cloud-based Conversational AI platform providing the complete Speech-to-Text (STT), Natural Language Processing (NLP), and Text-to-Speech (TTS) pipeline infrastructure. Convai manages the conversational memory, contextual dialogue flow, knowledge base querying, and character audio response generation all in real-time.

Tools:

  • Blender Primary 3D digital creation suite used for character concept modelling, mesh sculpting, UV unwrapping, rigging, weight painting, and facial expression shape key animation for Datuak Rajo Alam.
  • Gemini (Google) Utilised for documentation linguistic refinement, bilingual translation structuring, and professional report composition.
  • ChatGPT (OpenAI) Utilised during initial brainstorming phases for concept exploration, research framework ideation, and structural logic refinement of the project's foundational architecture.

System Architecture

The Cultura system is built around a continuous, AI-driven real-time voice interaction loop a closed pipeline where every stage flows seamlessly into the next with minimal perceptible latency:

[1] Voice Input (Microphone)
        ↓
[2] Speech-to-Text (STT) — Audio signal → Digital text transcription
        ↓
[3] Natural Language Processing (NLP) — Intent recognition, context analysis
        ↓
[4] Convai AI Engine + Cultural Knowledge Base — Query processing, response generation
        ↓
[5] Text-to-Speech (TTS) — Text response → Natural audio output
        ↓
[6] 3D Character Response — Audio + Lip-Sync + Facial Expression + Body Gesture
        ↓
    [Back to Step 1 — Continuous loop]

The architectural innovation that makes Cultura genuinely scalable lies at Stage 4: the Cultural Knowledge Base is a hot-swappable dataset layer, entirely decoupled from the core processing pipeline. The same STT, NLP, TTS, and animation systems that currently serve the Minangkabau module can serve a Javanese, Sundanese, Balinese, or any other cultural module by reconfiguring this single layer without touching the underlying code architecture.

Implementation Process

The development was executed across six iterative phases to ensure optimal technological integration, stability, and performance:

  1. 3D Character Pipeline Concept design, mesh modelling, skeletal rigging, blend shape expression animation, and Unity import optimisation for Datuak Rajo Alam.
  2. Convai Platform Configuration Knowledge base structuring and population with the Minangkabau cultural dataset, covering traditions, historical narratives, philosophical principles, architectural knowledge, cuisine, and performing arts. Dialogue flow scripting for natural conversational cadence.
  3. Unity–Convai API Integration Deep integration of the Convai SDK within the Unity runtime, establishing the real-time bidirectional communication channel between the 3D character and the cloud AI engine.
  4. Voice Interaction Pipeline End-to-end connection and testing of the STT → NLP → Dialog AI → TTS modules, with latency benchmarking and optimisation at each node.
  5. Animation–Audio Synchronisation Implementation and fine-tuning of real-time lip-sync driven by TTS audio waveform analysis, integrated with idle and responsive body gesture animation state machines.
  6. Iterative Quality Assurance Systematic testing cycles covering dialogue accuracy, response relevance, animation synchronisation fidelity, application stability, and end-to-end latency under real-world interaction conditions.

Challenges we ran into

1. Real-Time Synchronisation Architecture The most formidable technical challenge was achieving genuinely seamless, low-latency synchronisation between the cloud-based voice processing pipeline and the local 3D character animation system. Natural-feeling lip-sync requires that audio playback, mouth shape animation, facial micro-expressions, and body gestures all initiate and resolve in concert with a response lag that remains imperceptible to the user. This demanded deep iterative engineering of the Convai–Unity API integration layer, including custom audio buffering logic and animation state machine timing calibration.

2. Cultural Knowledge Base Design for Natural Conversation Structuring the Minangkabau cultural knowledge base to produce conversationally natural responses rather than encyclopaedic recitations required significant iterative design work. The AI needed to respond not merely with factual accuracy, but with the appropriate cultural tone, storytelling cadence, contextual depth, and philosophical nuance. Translating complex concepts like the "Adat Basandi Syarak, Syarak Basandi Kitabullah" principle or the social significance of the bilik system into digestible, engaging conversational responses required careful prompt engineering and knowledge base architecture.

3. Building Genuine Modularity from the Ground Up Designing an architecture that was truly scalable not just architecturally claimed to be required deliberate engineering discipline. Every database query pattern, every dialogue management flow, every character behaviour trigger, and every UI element needed to be fully abstracted from culture-specific data. This level of rigorous decoupling was essential to ensure that adding a new cultural module in the future would require only data configuration, not system re-engineering.

4. Immersive 3D Environment and Character Authenticity Ensuring that Datuak Rajo Alam and the Rumah Gadang environment were culturally accurate and authentically representative demanded extensive research into Minangkabau visual culture, traditional dress, architectural proportions, interior spatial arrangements, and symbolic motifs. Cultural authenticity was treated not as an aesthetic consideration, but as a core product requirement because the platform's educational and social credibility depends entirely on its accuracy.

5. Bridging Technical Research Gaps Our review of existing scientific literature including studies by Wang (2024), Song & Xiong (2025), Zhang et al. (2025 ViBES), and Huang et al. (2025) revealed that while virtual humans have advanced significantly in NLP quality and visual expression, most research develops these components in isolation, without direct integration inside interactive game engine environments. We were therefore building not just an application, but filling a documented gap in the state of the art which meant there were few established integration blueprints to follow.

Accomplishments that we're proud of

  • First-of-its-kind integration achievement Successfully built a full, stable, real-time Conversational AI pipeline (STT → NLP → TTS) operating natively inside a Unity 3D interactive application directly addressing a gap identified across multiple peer-reviewed research publications (Wang 2024, Huang et al. 2025, Zhang et al. 2025).
  • Living cultural avatar Brought Datuak Rajo Alam to life as a fully functional AI cultural guide capable of holding contextually rich, nuanced, open-ended dialogues about Minangkabau history, philosophy, traditions, architecture, and arts not as a scripted chatbot, but as a genuinely intelligent conversational agent.
  • Low-latency real-time performance Achieved stable, imperceptibly low-latency two-way voice communication between users and the 3D character under real-world interaction conditions.
  • Proven modular architecture Demonstrated genuine architectural scalability through complete decoupling of the cultural knowledge layer from the core system, validated by the clean abstraction boundary between Minangkabau-specific data and the underlying application logic.
  • Scalable social impact blueprint Established a fully replicable, technology-proven framework for AI-driven cultural preservation that can be extended to any indigenous heritage laying the groundwork for a comprehensive digital cultural ecosystem across the region and beyond.
  • Dual SDG alignment The platform directly advances SDG 4: Quality Education by making indigenous cultural knowledge accessible, interactive, and engaging for digital natives, and SDG 11: Sustainable Cities & Communities by providing a scalable tool for the digital preservation of intangible cultural heritage.

What we learned

Building Cultura taught us that the most significant engineering challenges are almost never purely technical they live at the intersection of technology, human experience, cultural sensitivity, and systems thinking.

Making a 3D character feel genuinely alive and trustworthy as a cultural guide is not a graphics challenge. It is simultaneously a systems architecture problem (low-latency pipeline design), a linguistic design problem (natural conversational flow), a cultural research problem (accuracy and authenticity), and a user experience problem (creating the psychological safety for users to engage freely). Every one of these dimensions had to be solved in concert.

We learned that modular system design is not an optimisation it is a first-class architectural principle. The decision to fully decouple the cultural knowledge layer from the core processing pipeline was the single most consequential design choice we made, and it transformed Cultura from a Minangkabau-specific application into a genuinely universal cultural preservation platform.

We learned the critical importance of grounding technology development in rigorous research. By conducting a thorough review of state-of-the-art literature identifying the limitations of Wang (2024), the gap in ViBES (Zhang et al. 2025), and the shortcomings of Huang et al. (2025) we were able to position Cultura not as an incremental improvement, but as a direct solution to documented, peer-validated research gaps.

Most profoundly, we learned that Generative AI reaches its highest purpose not when it generates content autonomously, but when it becomes the living conduit through which human cultural wisdom is transmitted across generations, across geographies, across time. Technology does not preserve culture. People preserve culture. But the right technology can make that preservation dramatically more accessible, engaging, and enduring.

The heart of Cultura is not its AI pipeline. It is the cultural data itself painstakingly curated, contextually structured, and carefully verified to honour the communities whose heritage we are privileged to steward.

What's next for Cultura

The vision for Cultura extends far beyond a single cultural module or a single region. We are building infrastructure for cultural survival a platform designed from its foundation to scale, evolve, and endure.

Near-Term Roadmap

  • Multi-culture expansion across ASEAN and beyond Rapidly onboard new cultural datasets Javanese, Sundanese, Balinese, Dayak, Torajan, and beyond each with their own culturally accurate 3D virtual guides, authentic environments, and deeply curated knowledge bases. The same proven technology, infinitely multiplied.
  • Indigenous language NLP support Extend the speech recognition and NLP pipeline to process, understand, and respond in local indigenous dialects beginning with the Minangkabau language delivering an unprecedented level of cultural authenticity and linguistic preservation.
  • Advanced emotional synchronisation and expression fidelity Integrate deeper facial expression mapping using blend shape morphing, dynamic contextual body gesture systems, and emotional prosody modulation in TTS output to create character interactions that feel indistinguishably human.

Medium-Term Roadmap

  • Institutional deployment partnerships Establish partnerships with schools, digital museums, cultural heritage centres, tourism boards, and ed-tech platforms to deploy Cultura as a permanent, live educational infrastructure moving from proof-of-concept to real-world impact at scale.
  • Large-scale cross-regional empirical research Conduct rigorous, longitudinal user studies across diverse demographics to quantitatively measure the platform's impact on cultural knowledge acquisition, engagement scores, comprehension depth, and long-term retention generating peer-publishable research data.
  • Expanded interactive cultural environments Develop additional culturally authentic 3D environments beyond the Rumah Gadang interior including traditional market scenes, ceremonial spaces, and natural landscapes enabling multi-scene, narrative-driven cultural exploration journeys.

Long-Term Vision

  • A unified cross-cultural digital ecosystem Build a connected platform where users can journey seamlessly between cultural modules, guided by a network of AI-powered virtual cultural ambassadors representing the full diversity of regional heritage.
  • Formal educational curriculum integration Partner with ministries of education and international educational organisations to integrate Cultura into formal learning pathways as a certified, interactive cultural learning tool.
  • Intangible heritage preservation archive Develop a structured, open digital archive of curated cultural knowledge bases ensuring that the rich knowledge encoded into each Cultura module is permanently preserved, openly accessible, and continuously expandable by communities themselves.

Cultura is not the end of a development journey. It is the beginning of a movement a proof of concept that Generative AI can become one of the most powerful tools humanity has ever developed for preserving and sharing the irreplaceable diversity of human cultural expression.

AI Use

Was more than 70% of the code generated by AI?

No. The complete system architecture, all Unity–Convai API integration logic, the character animation pipeline, the cultural knowledge base structural design, and the full user experience design were conceived, engineered, and implemented by our team. AI tools played a defined supporting role: Convai powers the application's core conversational feature as a platform service; ChatGPT assisted initial concept brainstorming and framework logic structuring; Gemini supported documentation linguistic refinement and bilingual composition. The primary application codebase, system design decisions, and cultural data curation were entirely human-authored.

Clarification: Convai functions as the AI engine that powers the application's core interactive feature it is a platform dependency, not a code generation tool.

Built With

Share this project:

Updates

Submission history