Human AI Dyad

Inspiration

This project comes out of our post-doctoral research at Michigan State University with Stephanie J. Nawyn, on how organisations produce workforce skill inequality during the generative AI transition.

The construct at the centre is AI-Induced Skill Ineptness (AISI): a structurally reproduced condition in which a worker's skill repertoire becomes misaligned with what AI-integrated work demands. It is not a skills gap, not digital illiteracy, and not technological unemployment. Four characteristics define it — velocity (ineptness is continuously renewed rather than closed), organisational production (organisations that deploy AI without contemporaneous training manufacture the incompetence they then attribute to workers), structural inequality (it lands hardest on women and lower-paid occupational categories), and consequential asymmetry (workers absorb the cost, organisations capture the benefit).

The Skill Ineptness Reproduction Model (SIRM) explains how AISI forms across macro, meso, and micro levels. We tested it against five secondary workforce datasets — OECD PIAAC 2023 ($n = 52{,}847$), BLS-ETA CPS ($n = 28{,}419$), Randstad ($n = 12{,}429$), Interface-EU ($n = 1{,}592{,}384$), and the WEF Future of Jobs Employer Survey ($n = 1{,}041$). The predictions held. Adjusted gender odds ratios for AI skill deficit classification ran 1.63 to 2.47 across four independent datasets, net of education, occupation, and tenure. Peer network effects were 1.9 to 2.4 times stronger for women. Training investment moderated the deployment-to-skills-gap relationship, with the simple slope falling from $\beta = .58$ under low training investment to a non-significant $\beta = .12$ under high investment.

Then we wrote the limitations section, and it told us what to build:

"There are no validated measures of AI-Induced Skill Ineptness as defined by the SIRM... The development and validation of a purpose-built AISI scale should be carried out for primary data collection."

Three propositions were supported. Three could not be tested at all, because no instrument exists. The theory was empirically grounded and had nowhere to go.

A fixed-item scale was never going to work, for a reason internal to the theory. One of SIRM's own micro-mechanisms — identity-based competence discounting — distorts self-report. Interface-EU's odds ratio of 2.47 sits far above PIAAC's 1.71 precisely because listing AI skills involves self-presentation, and women systematically under-list. A questionnaire inherits that distortion with no way to detect it. Something that can follow up on a hedge, notice a gap between a claimed skill and a described behaviour, and separate "I can't do this" from "I stopped doing this" cannot be built out of Likert items.

So we built the instrument as an agent.

What it does

Human AI Dyad evaluates the dyad itself — the working partnership between a person and their AI tools — and asks whether the division of labour inside it is sustainable. It runs three phases.

1. Interview across four dimensions of Human-AI Collaborative Proficiency. Our companion paper on Human-AI Integration Capital defines HACP as spanning technical skills (AI tool use, evaluating AI-generated output) and metacognitive skills (critical judgment of AI, human oversight and the ability to override an AI decision). The agent assesses four dimensions drawn directly from that specification:

Dimension	What it probes
Reliance	How much cognitive work has been delegated, and on which tasks
Verification	How AI outputs get checked, and whether the user could still catch an error
Judgment	Whether AI is shaping the decision or informing it, and whether override still happens
Skill development	Whether independent capability is still accumulating, or only being consumed

This is not a form. Each follow-up is generated from what the user just said, so admitting you no longer read stack traces gets you pushed on debugging specifically, while claiming you still write your own first drafts gets probed on whether that holds under deadline pressure. Fixed questions let people perform the answers they think are expected. Follow-ups do not.

2. Diagnosis. The agent analyses the resulting profile for skill ineptness risk, identifying which dimensions have thinned and how the pattern reproduces itself. Reliance without verification is a different condition from reliance without skill development, and they call for different remediation.

3. Coaching. It produces a Dyad Rebalancing Plan, which operationalises the individual level of the Strategic Augmentation Architecture from Paper 2 — AI agency development, experience accumulation pathways, and AI-inclusive peer learning. The plan is concrete and time-boxed. Not "use AI less," but closer to "run the first twenty minutes of every debugging session unassisted, three times this week, and log what you found."

Every turn, diagnosis, and plan is written to Firestore against a user ID supplied at the start of the session. Because velocity — continuous renewal of ineptness — is the first defining characteristic of AISI, a single measurement contradicts the construct it is measuring. A returning user is assessed against their own prior session, not from zero.

How we built it

Layer	Choice
Model	Gemini 3.5 Flash, via Vertex AI's global endpoint
Agent framework	Google ADK — LlmAgent, ADK Runner, ADK Dev UI
Persistence	Cloud Firestore, native mode
Hosting	Cloud Run
Language	Python 3.11

A single LlmAgent carries the full instruction prompt, which translates the papers' theoretical specification into interview logic: what evidence each HACP dimension requires, how response patterns map to ineptness risk, and which SAA individual-level mechanism a given remediation belongs to.

Three Firestore tools are bound to the agent:

• save_assessment_turn() — persists each question/response pair as the interview proceeds
• get_assessment_history() — loads prior sessions once the user supplies their ID
• save_saa_plan() — writes the finished diagnosis and rebalancing plan

Gemini decides when to call these. They are not fired on a schedule by application code — the model recognises that a returning user needs history loaded, or that the interview has produced enough evidence across all four dimensions to commit a plan.

The data model is deliberately flat:

assessments/{user_id}/turns/{turn_id}
  dimension, question, response, timestamp

plans/{user_id}/{plan_id}
  diagnosis, saa_actions, created_at

Development ran locally under adk run in Cloud Shell, then deployed to Cloud Run from source.

Challenges we ran into

Translating a theoretical construct into interview logic

The hardest problem was not software. HACP's dimensions are analytically clean on the page and slippery in conversation. Verification and judgment in particular collapse into each other unless the questions are carefully separated — checking whether an output is correct is a different capability from deciding whether to act on it, and most people describe both as "I look it over."

The related difficulty is that the thing being measured resists direct questioning. Nobody can accurately self-rate how much their judgment has shifted, because the shift is what would have flagged it. The agent has to infer from described behaviour rather than accept self-assessment, and keeping the instruction prompt from drifting back toward "rate your AI skills from 1 to 5" took more iterations than the entire deployment pipeline.

The three-agent design lost to the one-agent design

We started with three ADK agents mapped to the three phases — interviewer, diagnostician, coach. Clean separation, and it mirrored the paper structure.

It was worse in every measurable way. Under real rate limits the handoffs added latency without improving output, because the valuable artefact was the accumulated interview context and every handoff meant re-serialising it. Worse, the diagnosis degraded. Distinguishing eroded verification from eroded judgment depends on conversational texture — hesitation, qualification, what someone volunteers unprompted — and that texture does not survive being summarised and handed to a separate agent. Collapsing to a single LlmAgent with three tools made the diagnosis better, not merely faster.

It worked perfectly locally and 404'd in production

Locally: flawless multi-turn interviews, Firestore writes landing, real Gemini responses. Deployed to Cloud Run:

404 NOT_FOUND: Publisher model
projects/.../locations/us-central1/publishers/google/models/gemini-3.5-flash
was not found

We read this as a routing bug — the container falling through to Vertex AI instead of the Gemini Developer API. Granting roles/aiplatform.user flipped a 403 into the 404 above, confirming it really was hitting Vertex.

The actual answer was more mundane: gemini-3.5-flash is not served from us-central1 on Vertex. It is available on the global endpoint and several other regions, but not that one. The 404 was accurate. It only read like a bug because we had assumed model availability was uniform across regions. Setting GOOGLE_CLOUD_LOCATION=global fixed it — the Cloud Run service still runs in us-central1, only the model endpoint is global.

Configuration baked into the container silently outranked configuration set on the service

Before finding the above we tried to force the routing three ways: GOOGLE_GENAI_USE_VERTEXAI=FALSE as a Cloud Run environment variable, then os.environ[...] in code, then that line moved above the google.adk import.

All three had exactly zero effect. Not a different error — no change at all. The cause was a .env file inside the agent directory, copied into the container at build time and loaded with override semantics at agent-load time. Everything set on the service was being overwritten at startup by a file we had forgotten was there.

Local success was weak evidence of deployed success

The Firestore helpers lived in a separate tools/ package. It resolved fine under adk run and broke on import inside the container. Inlining them into agent.py removed the failure mode entirely.

Accomplishments that we're proud of

The theory has a working instrument. All three papers end at the same blocker: SIRM Propositions 3, 4, and 5 cannot be tested without a construct-valid AISI measure that does not exist. This is a first deployed attempt at the individual-level half of that instrument, taking real sessions today.

The interview adapts, and that solves a measurement problem rather than a UX one. The gap between PIAAC's adjusted OR of 1.71 and Interface-EU's 2.47 is substantially a self-presentation artefact. An interviewer that follows up on a hedge is doing something a fixed scale cannot do by construction. This is the clearest argument for why the instrument had to be an agent.

It distinguishes conditions, not just severity. Heavy reliance with intact verification is a stable dyad. Heavy reliance with eroded verification is a failure waiting for its first confident-but-wrong output. Same reliance score, different diagnosis, different plan.

We killed our own architecture when the evidence said to. Discarding a working three-agent design for a simpler one was uncomfortable and correct, and we can explain precisely why diagnosis quality improved.

It ships and it remembers. A live Cloud Run URL, real Firestore documents accumulating per user, and a returning-user path that recalls a prior assessment. Given that continuous renewal of ineptness is the first defining characteristic of the construct, persistence is not a convenience feature — it is a requirement of measuring the thing at all.

What we learned

• Conversational assessment solves a measurement problem fixed scales structurally cannot. Self-report distortion is not noise here; it is one of the mechanisms the theory names. Only an instrument that can probe an inconsistency has any purchase on it.
• Multi-agent architecture is a cost centre until proven otherwise. For a task where accumulated conversational context is the diagnostic signal, decomposition destroyed the thing we needed.
• Model availability is regional, and the error message will not tell you. A 404 on a publisher model is worth checking against the region table before assuming an auth or routing fault.
• When three independent, individually-correct fixes all produce zero change, stop fixing and start verifying what is actually running. Three correct fixes failing identically is not three bugs. It is one wrong assumption about which code is executing. The most transferable lesson of the build.
• Theory-to-prompt translation is its own discipline. Constructs that are cleanly separated in a paper blur in an interview, and only building it surfaces where the definition is underspecified. That has fed back into the research.

What's next for Human AI Dyad

Instrument validation. Administering the agent alongside established measures, then confirmatory factor analysis to test whether the four dimensions are empirically separable as theorised. A validated scale unblocks SIRM Propositions 3, 4, and 5 and the primary data collection described in Zia and Nawyn (2027).

The organisational half of the instrument. This version measures the dyad from the individual's side. AISI's defining claim is that it is organisationally produced — the same reliance pattern means something different in an organisation that trained its workforce than in one that deployed first and blamed after. Capturing that requires the two organisational metrics our research specifies and no existing dataset contains: the AI Deployment Velocity Ratio

$$\text{ADVR} = \frac{r_{\text{deployment}}}{r_{\text{training}}}$$

and the Co-Skilling Quality Index. An HR-facing companion agent could capture both, making the threshold prediction directly testable — below some critical $\text{ADVR}^*$, deployment should produce ineptness rather than augmentation, consistent with the moderation we observed: {SkillGap} = \beta_0 + \beta_1\text{Deploy} + \beta_2\text{Train} + \beta_3(\text{Deploy} \times \text{Train}) + \varepsilon, \quad \hat{\beta}_3 = -.29,\; p = .002$$

Pairing individual and organisational instruments is what would let the Dyad Rebalancing Plan extend from individual practices to team and organisational actions — which is where, by our own theory, most of the responsibility actually sits.

Longitudinal panel design. All five secondary datasets were cross-sectional, which is why none of those findings support causal inference. Firestore already stores timestamped per-user sessions. Adding Cloud Scheduler and Pub/Sub so the agent initiates re-assessment turns that into the panel data the field does not have.

Granular gender measurement and intersectional extension. Four of the five secondary datasets treat gender as binary and Interface-EU infers it from name analysis. Primary collection through the agent can do better immediately. Beyond that, extending to compounding barriers across race, class, and disability — the intersectional extension all three papers flag and none is yet equipped to test.

Built With

• cloud-run
• firestore
• gemini
• gemini-3.5-flash
• google-adk
• google-cloud
• python
• vertex-ai