HeirloomOS

Artisans lose everything when content creators profit from their stories. HeirloomOS flips that giving craftspeople their own school, shop, and legacy, generated entirely through AI conversation.

Comment

HeirloomOS

The Incident That Made Us Build This

A few years ago, a prominent content creator with over 20 million followers launched a paid online educational course featuring a centenarian traditional tattoo artist from an indigenous tribe in Southeast Asia.

Almost immediately, the artist's family sounded the alarm. They revealed that the elder had not provided true informed consent — that her thumbprint on a contract did not mean she understood or agreed to have her cultural heritage packaged and monetized by a foreign tech platform. The backlash was massive. The course was pulled.

But here is the part that stayed with us long after the controversy faded: the course itself — the idea of digitizing her knowledge, making it accessible to students worldwide, building a school around her craft — was actually the right instinct. The knowledge deserved to be preserved. The craft deserved a global audience. The problem was not the medium. The problem was that she was the subject, not the owner.

This is not just about one bad actor. It is a pattern that repeats endlessly: someone with a massive digital platform swoops in, packages an artisan's indigenous knowledge into a product, captures the profits, and moves on. The artisan remains exactly where they were before — no website, no students, no international buyers. Just a subject in someone else's story, briefly visible, then forgotten.

We kept thinking: what if the artisan owned the platform?

What HeirloomOS Does

HeirloomOS gives any artisan — regardless of technical literacy — a full digital ecosystem generated through a guided AI conversation. No third-party creator. No cultural intermediary. No contract with fine print. Just the artisan and an AI that listens, documents, and builds.

The Knowledge Archive

ERNIE conducts a structured multi-turn interview across five stages: materials, tools, process, history, and products. It does not ask generic questions from a fixed script. After every answer, it generates contextual follow-ups that reference what the artisan just said — if they mention sheesham wood, the next question is about sheesham wood specifically, not wood in general.

Photos the artisan uploads are processed by PaddleOCR, which extracts text from labels, handwritten notes, pattern diagrams, and certificates. That extracted text is passed directly back to ERNIE as context — the AI reads the artisan's physical world and weaves it into the conversation. The entire interview is stored permanently in a structured profile that all downstream modules read from.

The Online School

When the interview is complete, HeirloomOS auto-generates a fully structured online course — chapters mapped to interview stages, step-by-step lessons built from the artisan's own process descriptions, photos embedded as illustrations, and a quiz generated for each chapter. The artisan sets their price through conversation. One button publishes it to a live, shareable public URL.

The artisan does not build a course. They have a conversation. The course exists as a byproduct.

The Global Shop

ERNIE generates product listings in English, Hindi, and Chinese — not translated robotically, but culturally adapted for each market. The Chinese description is written for luxury craft buyers and uses vocabulary associated with heritage and authenticity. The English description leads with the artisan's story and process. An enquiry form routes buyer messages directly to the artisan by email or WhatsApp.

Every product listing, in every language, is owned entirely by the artisan.

How We Structured Conversations with MeDo

As developers, we know how complex it is to build an interconnected database, a multi-step stateful interview UI, a course generation pipeline, and an e-commerce frontend that talks to all of them. We wanted to test MeDo's limits by generating this entire full-stack architecture through multi-turn natural language conversation.

We structured our MeDo prompts in three deliberate phases:

Phase 1 — The Architecture Prompt

We did not start with the UI. Before describing any interface, we instructed MeDo to establish a central ArtisanProfile data model — one JSON record that every module would read from and write to. The prompt was explicit:

"Create a central artisan profile data structure with the following fields: profile (name, craft type, location, languages, bio), knowledge_stages (array of stage objects each containing the question, raw answer, structured answer, photos, OCR extracted text, and completion flag), interview (current stage, conversation history, completed stages), course (title, chapters array, price, enrolled count, published flag), and products (array with multilingual name and description fields, price in local currency and USD, and inquiry count). Every module we build after this must read from and write to this exact schema."

This upfront schema definition was the single most important architectural decision in the project. Because every subsequent module prompt referenced named fields in this schema, MeDo connected them correctly. The interview writes to knowledge_stages. The school reads from knowledge_stages to generate course.chapters. The shop reads from the products array. The public profile reads from all of them simultaneously. There is no manual data wiring — the connections were described before the modules existed.

Phase 2 — The Logic Prompts

We iteratively refined the ERNIE integration across multiple conversation turns with MeDo. The first iteration of the AI interviewer was too generic — it asked the same follow-up questions regardless of what the artisan said. We corrected it with a targeted multi-turn prompt:

"Update the ERNIE prompt logic for the interview module. After every artisan answer, ERNIE must generate exactly two follow-up questions that directly reference a specific word or phrase from the artisan's previous answer. If the artisan's answer contains fewer than 20 words, ERNIE must first ask for elaboration before generating follow-ups. ERNIE's tone must acknowledge the answer warmly before pivoting — never ask the next question cold."

The result was a conversational system with genuine adaptive behaviour. We ran similar iterative correction cycles for the PaddleOCR-to-ERNIE integration loop, the ERNIE translation quality for each market, and the interview resume logic.

Phase 3 — The UI Prompts

We described the frontend components using the visual editor, iterating on the public profile page specifically to ensure it felt like a dignified portfolio — not a generic SaaS template. The public profile needed to feel like something an artisan would be proud to share with an international buyer or a journalist, so we pushed MeDo hard on typography, whitespace, image treatment, and the language toggle between English, Hindi, and Chinese.

The Most Impressive Feature MeDo Generated

The automated state management and data routing across modules.

We described a genuinely complex relationship to MeDo: take unstructured conversational output from a multi-turn ERNIE interview, parse it into typed educational modules, and auto-populate a separate e-commerce frontend with translations — without any of these modules knowing about each other at build time, only through the central schema.

MeDo handled the underlying state connections without us writing a single line of backend routing code. When the interview completes, the school generation triggers automatically. When the school publishes, it appears on the profile automatically. When a product is added in the shop, it appends to the profile automatically. We described the connections. MeDo built them.

The AI Pipeline

The complete data flow in HeirloomOS runs like this:

Artisan speaks or types (any language)
         ↓
Web Speech API — audio transcribed in real time
         ↓
ERNIE — processes answer, generates 2 contextual follow-ups
referencing the artisan's exact words
         ↓
Artisan uploads craft photo
         ↓
PaddleOCR — extracts text from labels, diagrams, handwritten notes
         ↓
ERNIE — receives OCR output as next-turn context
("I see from your photo label that you use natural indigo
from Rajasthan — how does that compare to synthetic dye
in terms of colour fastness under monsoon washing?")
         ↓
All Q&A pairs written to ArtisanProfile.knowledge_stages
         ↓
Interview completion triggers course generation:
knowledge_stages → course.chapters (ERNIE writes lesson content)
         ↓
ERNIE translates all product fields to EN + HI + ZH
(market-adapted, not literal translation)
         ↓
Public profile URL live — all modules populated,
language toggle active, analytics tracking

Challenges We Faced

The translation quality problem. When we first asked ERNIE to translate product descriptions to Chinese, the output was flat and literal — technically accurate, commercially useless. We went back to MeDo with a more specific brief: translate for Chinese buyers of handmade luxury goods, use vocabulary that conveys heritage and authenticity, do not translate word for word. The difference in output quality was significant. Market-adapted copy is a fundamentally different task from translation, and ERNIE needed to know which task it was doing.

The PaddleOCR-to-ERNIE context window. Passing OCR output back to ERNIE as next-turn context required careful prompt engineering. If the OCR extraction was long or contained irrelevant text, ERNIE's follow-up questions drifted. We resolved this by instructing MeDo to have ERNIE extract only the most semantically relevant phrase from OCR output before incorporating it — the artisan's craft context, not the full raw text dump.

Making the interview feel like a conversation, not a survey. This took the most iteration. The difference between "What materials do you use?" asked generically and the same question asked after the artisan just described their grandmother teaching them the craft is enormous. Getting ERNIE to acknowledge what the artisan just shared before pivoting to the next question — and to treat short answers as invitations to go deeper rather than moving on — required multiple refinement passes.

The resume logic. An artisan completing a five-stage interview in one session is unrealistic for our target user. They need to leave, return the next day, and continue exactly where they left off — including the full conversation history. Implementing session-persistent state for the interview, where ERNIE picks up the thread rather than starting fresh, was the most technically demanding requirement we brought to MeDo, and the most satisfying to see working correctly.

Why This Category

We chose Surprise Us! because HeirloomOS does not fit neatly inside any other track — and that is the point. It is a productivity tool for people who have never seen a productivity tool. It is a learning platform for teachers who do not know what an LMS is. It is an e-commerce platform for sellers whose customer acquisition strategy is a WhatsApp message.

It serves people who exist outside the assumptions baked into every other category. That felt like exactly the spirit the "Surprise Us!" track was made for.

On Informed Consent and Ownership

We want to be clear about one design principle that shaped every decision in HeirloomOS.

The artisan owns everything. Their interview data, their course content, their product listings, their public profile — none of it is licensed to us, scraped for training data, or resold in any form. The platform exists to give them a presence they control and can take anywhere.

The incident that inspired this project was not just about consent being ignored. It was about a system where the person with the platform always captures more value than the person with the knowledge. HeirloomOS is an attempt to invert that. Whether we succeed will depend on whether artisans actually use it — but the architecture was built to serve them, not to extract from them.

Accomplishments that we're proud of

We are most proud of successfully generating a stateful, multi-module architecture without writing backend code. In traditional development, connecting a dynamic interview chat, an OCR processing pipeline, an LMS (Learning Management System), and an e-commerce storefront requires significant routing and database work.

By carefully structuring our prompts in MeDo, we managed to create a single central Artisan Profile data record that perfectly syncs all these components. The fact that an artisan can simply talk to ERNIE, upload a photo, and watch a fully structured, multi-lingual school and shop populate automatically is a massive technical and UX win for us. We're also deeply proud that the final UI feels like a premium, dignified portfolio—not a generic form or a charity tool.

What we learned

The biggest lesson was mastering "Architecture via Conversation."

We learned that when using MeDo to build complex apps, prompt engineering isn't just about generating good text; it's about defining data flows. We realized that if we explicitly told MeDo how modules should interact ("The interview module writes to the central record; the school module reads from it"), the platform would handle the underlying state connections perfectly.

We also learned the profound difference between literal translation and cultural adaptation. When we initially asked ERNIE to translate the shop to Chinese, it was flat. When we adjusted the prompt to target "Chinese buyers of luxury handmade crafts," the output instantly elevated, proving that LLMs can handle deep cultural nuance if prompted correctly.

What's next for HeirloomOS

  1. WhatsApp Integration: While the web interface works beautifully, most artisans in the global south run their lives on WhatsApp. We want to integrate MeDo with a WhatsApp API so the entire ERNIE interview and shop management can happen via voice notes in the chat app they already use.
  2. Video Processing: Expanding beyond PaddleOCR's image capabilities to allow artisans to upload short videos of their weaving or carving, letting the AI automatically extract key steps for the course chapters.
  3. Real-World Pilot: We built a robust demo to prove the architecture. Our next immediate step is to take this platform into the field and onboard a pilot group of real traditional craftspeople to battle-test the UX and refine the voice-recognition workflows in noisy, real-world environments.

Built With

  • api
  • ernie
  • medo
  • paddleocr
Share this project:

Updates