When You Cannot See Inside the Teacher

In 1995, two researchers named Corbett and Anderson asked a simple question: does this student know this concept? They modeled the answer as a hidden coin — either the student had learned it, or they hadn't — and described how that hidden state updated with every quiz attempt. They called it Knowledge Tracing.

For twenty years, researchers made the coin more sophisticated. In 2015, a neural network replaced the coin. Deep Knowledge Tracing passed hundreds of thousands of interactions through an LSTM, letting the network discover its own representation of student knowledge. It got better at predicting the next answer. So did the self-attention version. So did the graph neural network.

Every model got better at answering: “what will the student do next?” None of them could answer: “what is the system actually doing to help?”

Consider what an adaptive system actually does. At every moment, it chooses which concept to show a student next. That choice is implicitly governed by up to six competing forces: how much did the student's mastery just change? What do they already know that connects to this? How hard is the concept relative to their current ability? How uncertain is the system about what they know? Are they in their zone of proximal development? What new concepts might this unlock?

These are the six dimensions of what we call governance. And as of 2024, there existed no instrument to characterize which of those six dimensions were carrying real signal in any deployed system — and which were decorative. Two systems with identical prediction accuracy might be governed completely differently, and no existing evaluation method could detect the difference.

This was the state of adaptive learning governance in 2024. The inciting question was not “can we build a better predictor?” It was: can we build a system auditable enough that we could actually tell whether its governance dimensions were working?

The design constraint was unusual: build an adaptive learning system where every single decision could be traced back to its inputs. Not a black box. Not a latent embedding. Every recommendation had to be auditable — traceable from the Kafka event, through the prerequisite graph lookup, through the six-dimensional Joint Trajectory score, to the mastery update. If you couldn't explain it, it didn't go in.

For mastery estimation, we turned to a tool from a completely different domain. In 1960, Rudolf Kálmán published a paper about estimating the state of a linear system from noisy measurements — a problem arising in aerospace navigation. His filter, now called the Kalman filter, became the backbone of every GPS system, every spacecraft navigation computer, every autonomous vehicle.

For representation selection — which format to present a concept in — we borrowed from a completely different field again: clinical trials. In medicine, when you don't know which treatment works best for a patient, you use adaptive randomization. Thompson Sampling, a Bayesian bandit algorithm, does exactly this. For each arm (text, MCQ, video, audio, code), maintain a belief about how effective it is for this student on this concept. Sample from that belief. The arm that gets sampled most becomes the recommendation. The arm that worked gets reinforced. Uncertainty drives exploration automatically.

The system was built. All 96,727 interactions were logged. Every Kafka event was traced. Every JT score component was recorded. We had our auditable machine. Then we turned the instrument on.

The instrument's first act was to find a mistake in itself.

The normalizer — the function that scales the JT score before it feeds into the governance decision — was a sigmoid centered at zero. A sigmoid has an asymptote: it never quite reaches zero at the bottom. Evaluated at −2.5, it returns approximately 0.076. This meant that even a genuinely null governance dimension — one whose signal was zero — would contribute approximately 0.076 to the JT score. Not zero. A floor artifact, invisible until the instrument looked.

The instrument filed a correction against its own verdict. The categorical “ACTIVE” classification for the transfer dimension was marked as threshold-fragile. A new control was needed.

This was the design of the shuffled-DAG control. Keep the same concept graph — same number of nodes, same number of edges, same degree distribution. But randomize which concepts the edges connect. The prerequisite from Algebra to Calculus now points to, say, Ancient History. The prerequisite from Addition to Multiplication now points to, say, Geology. The structure of connection is preserved; only the meaning is destroyed.

The experiment ran at full corpus scale: 1,980,000 first-encounter learning events. The shuffled map produced essentially zero learning transfer effect. The real map produced a causal gain of approximately +0.053 — a within-learner, time-placebo-corrected, fixed-effects estimate of the durable effect that explicit prerequisite topology has on learning outcomes. Permutation p-value: less than 0.01 across 100 shuffled-DAG null draws.

About two-thirds of the effect is curriculum proximity — the sequencing advantage of same-session learning along connected concepts. The remaining third is a durable causal component that survives a time-placebo test: comparing students who mastered a prerequisite before attempting the target against students who mastered it after. The after-group eliminates the causal mechanism; their association drops by 42%. What remains is the irreducible transfer signal from correct prerequisite topology.

The map is not decoration. The roads lead somewhere real. A student who masters a prerequisite concept genuinely learns successor concepts better — not because good students do both, but because the connection itself is doing causal work.

This seems obvious in hindsight — of course algebra helps with calculus. But “obvious in hindsight” is exactly what distinguishes a folk belief from a scientific result. The graph-based knowledge tracing literature had shown that adding prerequisite graphs improves AUC. None of those papers ran the shuffled-DAG control. None separated correct topology from graph presence alone. The causal mechanism was assumed, not demonstrated.

And then there is the instrument itself — the ADC — whose defining property is not the finding it produced but the finding it was willing to revise. A scientific instrument that only confirms its own predictions is not a scientific instrument. It is a mirror. The ADC found the normalizer floor, flagged it as a structural issue, and forced a re-derivation under a stricter control. The result held. But the holding mattered less than the testing.

The cold-start result added a practical dimension to the theoretical one. An embedding-free system — with no offline training, knowing nothing about a new student before their first interaction — beat the standard Bayesian benchmark at every cold-start window on the credible evaluation dataset. The mechanism was the Kalman filter's uncertainty term: high variance in early interactions drove exploration that BKT's EM-fitted parameters couldn't match in sparse-data regimes.

The thesis delivers three things. A system (HCIE) that works from the first interaction without embeddings. An instrument (the ADC) that characterizes which governance dimensions are active — and is willing to report when its own verdict is wrong. And a calibrated causal finding: explicit prerequisite topology contributes real, measurable, cross-concept transfer to learning outcomes at a scale that rules out coincidence.