Themeforge

Inspiration

I have benefited from open-source research tools such as Engauge Digitizer, which make specialized research methods more accessible to people who may not have access to expensive commercial software.

At the same time, seeing what Codex could produce gave me the confidence to turn ideas that previously existed only in my head into a tangible research product. ThemeForge grew from the belief that researchers should be able to transform methodological knowledge into usable tools, even without a traditional software-development background or substantial funding.

What it does

ThemeForge analyzes rich-text sources and identifies potential thematic categories within the data. It then associates each category with relevant direct quotations, allowing researchers to examine the evidence supporting each proposed theme.

The system helps researchers: -Import and organize qualitative text data -Identify potential thematic categories from multi text sources -Populate categories with supporting direct quotations -User can review, revise, merge, or reject suggested categories -Maintain a visible connection between interpretations and source evidence

ThemeForge does not treat automated output as a final analysis. Instead, it provides an evidence-linked starting point that researchers can critically evaluate and refine.

How we built it

ThemeForge was developed through an iterative process combining methodological review, algorithm design, pilot testing, and user-experience evaluation.

I reviewed qualitative research articles and thematic-analysis guidance to determine how established analytic principles could be translated into computational procedures. These materials were then used to guide Codex in developing the application and its underlying analysis workflow.

The initial product was repeatedly revised through pilot use. I examined how researchers interact with widely used commercial qualitative-analysis programs, including NVivo, and incorporated familiar organizational and workflow features to improve accessibility for researchers already working in the field.

Development therefore involved more than generating code. It required continuously translating theoretical concepts into explicit rules, testing whether those rules produced meaningful results, and revising the system based on observed user experience.

Challenges we ran into

One of the most difficult challenges was designing an algorithmic structure that remained consistent with thematic-analysis theory without generating excessive false-positive results.

Qualitative documents often contain headings, labels, formatting elements, repeated phrases, and structural text that may appear important computationally but have little analytic meaning. The system therefore needed to distinguish between meaningful conceptual patterns and document features that should not become thematic categories.

Another challenge was determining how much evidence should be required before suggesting a category. Rules that were too permissive produced irrelevant themes, while rules that were too restrictive risked excluding less frequent but conceptually important ideas.

Implementing theoretical guidance within an algorithm also required a multiperspective approach. Each rule had to be considered from methodological, computational, and user-experience perspectives to reduce errors and produce outputs that researchers could understand, inspect, and revise.

Accomplishments that we're proud of

Internal validation testing showed at least 95% agreement between ThemeForge’s identified thematic categories and the expected categories in the test data.

This result was particularly meaningful because the system was designed not only to identify categories, but also to connect those categories with relevant direct quotations. Maintaining this link between interpretation and evidence makes the output more transparent and useful for researcher review.

What we learned

The central lesson from developing ThemeForge was that methodological theory cannot be transferred directly into an algorithm without substantial interpretation.

Concepts that appear straightforward in research guidance often require multiple operational definitions when implemented computationally. Each definition can affect what the system recognizes, excludes, or prioritizes.

The development process reinforced that automation is most useful when it strengthens human analysis rather than concealing or replacing it.

What's next for Themeforge

The next stage of development will focus on further optimizing the thematic-analysis algorithm, expanding validation across more diverse qualitative datasets, and improving the system’s ability to distinguish meaningful concepts from structural or irrelevant text.

ThemeForge will also continue expanding its functionality so that it can provide an experience comparable to established commercial qualitative-analysis platforms. Planned development includes stronger coding workflows, improved category refinement, more transparent validation information, better quote management, and additional tools for reviewing and reporting findings.

The long-term goal is to provide a rigorous and accessible qualitative-analysis platform for researchers, students, and practitioners who may not have institutional funding for expensive commercial software.

Built With

• bm25
• natural-language-processing
• python
• sentence-transformer