Inspiration
Researchers who code interviews alone have no second coder, and many are not allowed to send recordings to cloud AI tools. Their codebooks are never tested against another reader.
What it does
Codebook Second Reader gives one interview recording on a UGREEN AI NAS a second reading against the researcher's own codebook. On a first visit it transcribes the recording into a numbered transcript, which the researcher codes. On a second visit it shows a Disagreement Queue: the passages where the second reading and the researcher's coding differ, each with its time, its words, both readings and the kind of disagreement. The researcher decides every item. The decisions are saved as an adjudication log and, on request, a revised codebook. A per-code stability table grows across interviews. It is meant for codebook-based approaches, not for reflexive ones.
How I built it
A Markdown-only Uliya Skill: SKILL.md and six reference files. The parent Skill delegates to file-agent, knowledge-agent and document-agent. Code definitions are saved once as a small code-card file; after that the local model answers yes or no about five passages per command, and fixed rules do the comparing. Before any reading, the model must pass an exam on unseen examples from the codebook; a code that fails is withheld.
Challenges
A desk pre-test on Qwen3-4B (not on a NAS) showed that asking a small model to pick passages out of a whole transcript does not work: no code passed the exam. The Skill was rebuilt around small checks. With the submitted wording all five codes passed the exam; 5 of 5 planted disagreements reached the queue in the synthetic interview used for tuning and 4 of 4 in a second one, together with 4 items that were not planted. Giving the interviewer's question as context made the reading worse, so it was left out. On public counselling dialogues coded by ten practitioners (AnnoMI), with a two-code codebook built from random examples, the exam withheld both codes. Read anyway as a check, the model agreed with the reference annotator on 77% of decisions, while each of nine human annotators agreed on 85 to 96%, so withholding was the right outcome there. A second test used think-aloud monologues rated sentence by sentence by four people, a case chosen because its codes can be judged from one sentence. Three of six codes passed the exam. On those, the model's agreement with the reference rater (Cohen's kappa 0.26) was inside the range of the three human raters (0.20 to 0.51), near the low end, and 15 of the 19 queue items fell on decisions where at least one of the three other human raters disagreed with the reference rater. A third test used real research interviews coded by one researcher, with definitions written from her category names: one of six codes passed the exam and there was almost nothing to compare. The figures and their limits are in references/pre-test.md.
What I learned
A small local model is useful as a nominator when each question is small, the rules are fixed and the output says what was not read. It also needs a gate: the codebook exam is what kept an unreliable reading away from the researcher in the first human-coded test. The exam is strict. In three tests on other people's data it withheld between half and all of the codes, so what the Skill can read depends on how well the codebook's definitions and examples teach each code.
What's next
Next: testing on the iDX6011 Pro (whether transcription returns times and speaker labels, how the document expert reads the card file, approval prompts and run time), and a test on research interviews coded by two people, using a researcher's own codebook and chosen examples.
Disclosure
AI tools used and third-party components are disclosed in references/disclosures.md inside the package.
Built With
- markdown
- ug-document
- ug-filemgr
- ug-filesystem
- ug-knowledge
- ugos
- uliya-skill