-
-
Authentic AI
AuthenticAI — Project Story
Inspiration
The problem started with a question I couldn't answer.
I was reading through Turnitin's documentation on AI detection when I noticed something unsettling: the system flags submissions by comparing them against a distribution of AI-generated text. If a student writes clearly and concisely, they risk a false positive. If a non-native English-speaking student writes in a more formal register than their peers, they risk a false positive. The tool is modeling AI output, not the student.
This struck me as the wrong framing entirely. The right question is not "does this look like AI wrote it?" It is "does this look like this student wrote it?"
Every person has a writing fingerprint: characteristic sentence rhythms, vocabulary patterns, pronoun habits, and structural tendencies that stay consistent across different topics and assignment types. These are the features that are hardest to consciously fake, because they operate below the level of deliberate style choices. A ghost-writer cannot easily replicate them. AI paraphrasing tools flatten them.
The parallel to fraud detection was immediate. Transactions are not flagged because they look suspicious in general; they are flagged because they look anomalous for this cardholder. Behavioral fraud detection is longitudinal and identity-anchored. Academic integrity detection should be, too.
What I Built
AuthenticAI is a full-stack Flask application implementing a longitudinal writing identity framework for academic integrity. It builds a persistent, evolving stylometric profile for each student and scores new submissions against that student's own history, not against an external AI distribution.
The Three-Call Pipeline
The system uses three LLM calls per submission, each with a distinct role:
Call 1 — Feature Extractor. The LLM reads the raw text and returns a JSON-formatted 9-feature stylometric vector. Five features form Signal 1 (assignment-invariant): function-word ratio, pronoun distribution, sentence-length variance, type-token ratio, and punctuation rhythm. Four features form Signal 2 (course-aware): average sentence length, passive voice ratio, transition word density, and Flesch reading ease. Using the LLM for this rather than brittle keyword heuristics means the features are semantically grounded.
Call 2A — Profile Updater. The LLM updates the student's writing identity profile \(\Psi\) using a prompted LSTM-inspired mechanism. The profile has a cell state (long-term stable identity across all courses), a hidden state (per-course recent tendencies), and a forget candidate list. The LLM acts as the gating function. This keeps the context window cost constant regardless of the length of the submission history.
Call 2B — Anomaly Explainer. Only fires when a flag is raised. The LLM produces a plain-language explanation for the lecturer describing exactly which style dimensions shifted and how. Cost proportional to anomaly frequency.
Two-Signal Scoring
Signal 1 (\(\alpha_Q\)) computes a z-score of each invariant feature against the student's full submission history across all courses. Signal 2 (\(\alpha_S\)) computes a z-score against only prior submissions to the same course, because a lab report is expected to use more passive voice than a personal reflection. \(\alpha_S\) is suppressed until at least 3 same-course submissions exist.
$$\alpha_Q = \frac{1}{5} \sum_{j \in S_1} \frac{f_j - \bar{\mu}_j^{(n)}}{\sigma_j + \varepsilon}$$
A submission is flagged when \(|\alpha_Q| \geq \theta\) or \(\alpha_S \geq \theta\), where \(\theta = 2.5\) by default. Both signals are z-scores; a value of 3.8 means the submission sits 3.8 standard deviations above the student's own baseline.
Two-Tier Baseline Architecture
Flagged submissions are never automatically incorporated into the student's permanent profile. The system maintains two tiers:
- \(P_\text{perm}\) — the verified permanent baseline, updated only on clean submissions or lecturer-confirmed genuine work
- \(P_\text{temp}\) — a provisional update held until the lecturer decides: genuine promotes it to \(P_\text{perm}\), violation rolls it back
This guarantees the permanent fingerprint can only advance via verified evidence.
Multi-Tenant Architecture
The system supports multiple institutions running simultaneously with complete data isolation. Each institution has its own students, courses, profiles, and users. The same student ID at two institutions represents two completely independent people with independent profiles and independent sliding windows \(W\) and \(W_\tau\).
Challenges
The cold start problem. A longitudinal system needs history to score against, but students start with none. The Verified Baseline Acquisition Framework addresses this by treating the first two submissions as a supervised baseline-building period. Lecturers submit known-genuine work (in-class writing, timed assessments) to seed \(P_\text{perm}\) before coursework begins. The system transitions automatically from cold start to full scoring at submission 3.
Prompt engineering for structured consistency. Getting the LLM to return the same JSON schema reliably across different text types required careful prompt design. The feature extractor prompt provides explicit value-range guidance for each feature, and every feature value is defensively validated before use; dict/list returns from the LLM fall back to sensible defaults rather than crashing the pipeline.
The \(\sigma\) floor problem. When a student's first two baseline submissions are very similar, the sliding window standard deviation \(\sigma_j \to 0\), causing z-scores to explode on any deviation, however small. A relative \(\sigma\) floor (\(2\%\) of \(|\mu|\)) dampens this without over-suppressing real deviations.
Course-scoped Signal 2. Early versions keyed the \(W_\tau\) window on assignment type as a flat dict. When the design moved to course codes, the window needed to be restructured as a nested dict {course_code: {feature: [values]}}. ESSAY and REFLECTION windows silently overwrote each other until the nesting was correct; a subtle bug that caused \(\alpha_S = 0\) across all submissions, regardless of the actual deviation.
SQLite concurrency. Flask's debug reloader holds connections open across threads. Switching to WAL journal mode with a 30-second timeout resolved the database is locked errors that appeared during batch submission processing.
What I Learned
The most important insight was that the right abstraction for academic integrity is behavioral, not content-based. Generative AI is improving at mimicking human style faster than detection tools can adapt. A system anchored to a specific student's longitudinal fingerprint is inherently more robust; the target is stable (the student's own history) rather than a moving one (the frontier of AI writing quality).
I also learned that prompted LSTMs are a surprisingly powerful design pattern. Using natural language to describe the gating mechanism, rather than implementing it mathematically, lets the LLM bring semantic understanding to profile updates. When a student's vocabulary suddenly becomes more complex, the LLM doesn't just record the number. It understands that "formal academic register" displaced "casual first-person voice", and expresses that in the profile in a way that makes the eventual diff table legible to a non-technical lecturer.
Finally: building for real users forces architectural discipline. The multi-tenant design, role system, batch upload flow, and progress indicators were not included in the original specification. They emerged from thinking through what a lecturer at a real institution would actually encounter when sitting down to use this system.
Built With
- Python / Flask: backend and routing
- SQLite: persistence with WAL journal mode for concurrency
- Anthropic Claude / Azure OpenAI: three-call LLM pipeline (provider-configurable via env vars)
- Flask-Login / Werkzeug: authentication and password hashing
- Vanilla HTML / CSS / JavaScript: dark-theme frontend, no framework dependencies
