案例漂移分两条路走,只量窄的那条会把它判成不存在

Written by

in

Language

Evidence level: C

材料自己在变(换季、换批次、保存条件变)这件事对评分的影响分两条路:全体读者都觉得变难,走 occasion 主效应;只有少数读者特别敏感,走案例×读者交互。上一条判据只量了后者,于是最该防的那一种漂移被系统性判成「不存在」。

正文

先接着上一轮那笔账。上一条留下的判据是:把同一批材料给几位读者在几个时点各评一遍,做一次方差分解,看「案例与时点纠缠项」(σ²(o:r))占总方差几成——不到 5% 就认为案例漂移可忽略,原来那个共同模的点估计可以保留。

这条判据有两个独立的毛病,一个比一个深。

第一个毛病:量错了项。 设同一批材料、k 位读者、O 个时点各评一遍。材料本身在变,这件事对评分的影响可以分成两半。一半是共同的:所有人到了第三个月都觉得这批材料难读了——这落进 occasion 主效应 σ²(o)。另一半是读者特异的:只有某两位读者对某些时点格外敏感——这落进交互项 σ²(o:r)。

「材料变质」最典型的样子恰恰是前一半。而我们只量后一半。

仿真把这件事说得很清楚。设共同漂移 12 分、读者特异漂移 2 分(共同漂移占总方差 10.9%,两者差六倍),四组设计(80×5×3、80×8×3、80×10×4、120×10×5 到 200×12×6)跑下来,σ²(o:r) 的估计中位数都在 1.9–2.0,形态完好;然后按那条判据判——99.3%、100%、100%、100% 的样本都落进「<5%,可忽略」档。而真实的案例漂移占了 10.9%。

判据在它最该报警的情形下,百分之百判「没事」。这不是精度不足,是瞄错了靶。

第二个毛病:分母没写。 就算量对了项,还有一层。同一个 σ²(o:r)=2.0,换个分母就是另一档结论:占总方差 2.2%(落 <5% 档)、占读者侧方差(读者主效应+案例×读者+人×读者)15.4%(落 5–20% 档)、占读者主效应一项 33%(落 >20% 档)。同一个数,三个档全占。

判据原文只说「占比」,没说占谁的比。于是「跑出来落第几档」取决于跑的人选了谁当分母。这不是统计精度问题,是可复现性问题——两个人拿同一批数据、跑同一套代码,可以合法地得出「可忽略」和「不可辨识」两个相反结论。

第三件事更硬:这个量的自由度挂在哪儿。 σ²(o:r) 的自由度是 R×(O−1),与病例数 P 完全无关。仿真里把病例从 80 加到 800(十倍),区间相对宽度只从 156% 缩到 147%,几乎没动;把读者从 5 位加到 20 位,156% 缩到 83%;把时点从 3 个加到 12 个,缩到 72%。

换句话说,加病例对这个量一点帮助都没有。研究者的直觉是「多收点材料就更准」,在这个量上是错的。真正稀缺的资源是时点,其次才是读者数。

而 σ²(o) 主效应更惨——它的自由度只有 O−1。O=3 时自由度是 2,真值 12 分,5–95 区间是 [0.13, 35.56]:上界是真值的三倍,下界几乎贴零。就算做到 O=12,区间仍是 [4.93, 20.35]。也就是说,共同漂移这一项,在能落地的设计里根本估不出来。 承认这一点,比硬报一个点估计有用得多。

最狠的一处:占比和后果脱钩。 固定「案例漂移占总方差 5%」不变,只改人的真差异 S² 和读者噪声 T,共同模点估计的改变量 Δc 会从 5.2 个百分点一路变到 13.6 个百分点——同样 5%,后果差 2.6 倍。原因很直白:Δc = A/(S²+A+T/k),分母里挂着 S²。人差异越大,那个占比就被稀释得越厉害,但后果一分不减。

这条教训和上一轮「偏倚的离散该用离散-均值比、不该用方差占比」是同一件事的两面:占比类判据在人差异大的场景里,系统性低估后果。

何况「可忽略」这条线画在 5% 上,可真实后果早就越线了——A/S²=0.05 时 Δc 已有 4.0–4.6 个百分点,A/S²=0.10 就到 7.6–8.8。若把「影响结论」定在 5 个百分点,这条线在 A/S²≈0.06 处就被跨过,恰好压在「可忽略」档的边界上。

所以该怎么办:别报档,直接报那个数。 要问的从来不是「案例漂移占几成」,而是「扣与不扣,共同模的点估计差几个百分点」。这个量就是 Δc = 100·A/(S²+A+T/k),单位是百分点,它本身就是答案,不需要三档阈值去翻译。真要设线,就设成:Δc>5 个百分点,等价于 A > 0.053×(S²+T/k)。门槛随场景自己走,不借别人的量纲。

外部证据也支持「这一项必须以本批材料自测」。一项 135 名学生、2 位读者、3 个时点的概化研究里,occasion 方差可忽略、student-occasion 交互也小,而 student-rater 交互占 17.7%、三阶含残差占 27.7%,Φ 系数 0.75(PMID 32581621)——注意它报的是「小」,但那是特定结构下的「小」。另一项跟踪 458 名教师两年的观察评分研究给出反面:评分者漂移在最初几天非常大、且持续近两年,评分者不但没有收敛到共同的严厉度,彼此差异反而随时间扩大;趋势项相对评分者效应、特定课次上的评分误差与残差算是「中等」而非主项(PMID 29795823)。而重评抽样方式本身会实质影响漂移检测结论,边际齐性检验在位移跨类别平衡时功效很低(PMID 29881127)。44 项研究的系统综述加 meta 分析给的汇总数是:人的方差 44.2%、G 系数 0.65,并明确点出「情境因素会影响方差成分,报告严谨性仍需加强」(PMID 33951677)。

一句话:案例漂移不是一个可以查表的常数,它是每一批材料自己的属性;而判定它的那一项、那个分母、那个自由度,三处都不能照抄。

证据与局限

已验证:σ²(o:r) 的自由度是 R×(O−1)、与病例数无关,这是方差分解的代数事实,仿真中把 P 从 80 加到 800 区间宽度仅从 156% 缩到 147% 完全吻合。Δc 的恒等式 A/(S²+A+T/k) 亦可直接推导。概化理论在评分研究中的用法与方差成分的异质性,有系统综述级的外部支撑(PMID 33951677、32581621、29795823)。

部分验证:共同漂移进 occasion 主效应、特异漂移进交互项,这一拆分在完全交叉设计下成立;但若实际设计是「每人对每读者每时点只评一次」,人×读者项与三阶残差完全混杂(仿真中 σ²(p×r) 真值 5.0、估计中位 −26 到 −7,100% 为负),此时 σ²(o:r) 本身还会被系统性压低 19%–66%。三档阈值的具体数值(5%/20%)来自上一条判据的原始表述,本文只论证其分母与选项目标有误,未独立验证阈值本身的来源。

待验证:① 真实研究里「共同漂移」与「特异漂移」的实际比例,目前只有零星研究报告 occasion 方差「可忽略」,缺乏跨研究的分布;② Δc 的 5 个百分点门槛是否与实际决策后果(是否改变「共同模存在」这一结论)对应,尚无可校准的外部标准;③ 若漂移是单调趋势而非随机波动,方差分解会把趋势摊进残差,此时本文的拆分式需改写为带趋势项的增广模型。

相关阅读

证据等级: C
参考文献: PMID 32581621, PMID 29795823, PMID 29881127, PMID 33951677
L1 网络状态:边表 EDGE_TABLE_science-yixue.md 中「概化/方差/漂移/时点/重测/评定者/kappa/信度/一致/误差/校正/异质」关键词命中数全为 0,该主题尚未进入 L1 对应网络;网络侧最接近的是 E-060 标签前提(KB-317/333/334,状态 conflict 未裁决),故本文证据等级封顶 C。

证据等级:A=有PubMed直接支撑,B=科学类比/间接证据,C=传统文化共识/初步探索


免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。

Case drift splits into two paths, and measuring only the narrow one will classify it as nonexistent.

Evidence level: C

Material itself changes (seasonal shifts, batch changes, storage conditions). The impact of this change on scoring follows two paths: if all readers perceive increased difficulty, it enters the occasion main effect; if only a few readers are particularly sensitive, it enters the case×reader interaction. The previous criterion measured only the latter, so the most dangerous type of drift was systematically judged as “non-existent.”

Main Text

Continuing from the previous calculation. The criterion left by the prior section was: evaluate the same batch of materials by several readers at several time points, perform variance decomposition, and examine what proportion the “case×occasion entanglement term” (σ²(o:r)) constitutes of the total variance—if less than 5%, case drift is considered negligible, and the original common model point estimate can be retained.

This criterion has two independent flaws, one deeper than the other.

First flaw: measuring the wrong term. Suppose the same batch of materials, k readers, and O time points each evaluate once. The material itself changes, and the impact of this change on scoring can be split into two halves. One half is common: everyone finds the materials harder to read by the third month—this falls into the occasion main effect σ²(o). The other half is reader-specific: only certain readers are particularly sensitive at certain time points—this falls into the interaction term σ²(o:r).

The most typical manifestation of “material degradation” is precisely the first half. Yet we only measure the second half.

Simulation makes this clear. With common drift of 12 points and reader-specific drift of 2 points (common drift accounts for 10.9% of total variance, six times the other), running four designs (80×5×3, 80×8×3, 80×10×4, 120×10×5 to 200×12×6), the median estimate of σ²(o:r) remains between 1.9–2.0, with intact structure; yet according to that criterion, 99.3%, 100%, 100%, 100% of samples fall into the “<5%, negligible” category. Meanwhile, the true case drift accounts for 10.9%.

The criterion fails completely in the very situation where it should alarm. This is not a precision issue; it is aiming at the wrong target.

Second flaw: missing denominator. Even if we measure the right term, there is another layer. The same σ²(o:r)=2.0 yields different conclusions depending on the denominator: 2.2% of total variance (falls into <5% category), 15.4% of reader-side variance (reader main effect + case×reader + person×reader) (falls into 5–20% category), or 33% of reader main effect alone (falls into >20% category). The same number occupies three categories.

The original criterion merely states “proportion,” without specifying proportion of what. Thus, which category the result falls into depends entirely on who the researcher chooses as the denominator. This is not a statistical precision issue; it is a reproducibility issue—two people using the same data and running the same code can legitimately arrive at opposite conclusions: “negligible” versus “unidentifiable.”

Third issue is harder: where does the degrees of freedom hang? The degrees of freedom for σ²(o:r) is R×(O−1), completely independent of the number of cases P. In simulation, increasing cases from 80 to 800 (tenfold) shrinks the relative interval width only from 156% to 147%, almost no change; increasing readers from 5 to 20 shrinks it from 156% to 83%; increasing time points from 3 to 12 shrinks it to 72%.

In other words, adding cases does not help this measure at all. Researchers intuitively believe “more materials yield better accuracy,” but this is wrong for this measure. The truly scarce resource is time points, followed by the number of readers.

The σ²(o) main effect fares even worse—its degrees of freedom is only O−1. When O=3, the degrees of freedom is 2, with true value 12 points, the 5–95 interval is [0.13, 35.56]: the upper bound is three times the true value, the lower bound nearly zero. Even with O=12, the interval remains [4.93, 20.35]. Thus, the common drift term cannot be estimated in feasible designs. Acknowledging this is far more useful than reporting a point estimate.

Most severe point: proportion decouples from consequence. Fixing “case drift accounts for 5% of total variance,” but varying the true person difference S² and reader noise T, the change in common model point estimate Δc ranges from 5.2 percentage points to 13.6 percentage points—same 5%, yet consequences differ by 2.6 times. The reason is straightforward: Δc = A/(S²+A+T/k), where the denominator includes S². As person differences grow, this proportion becomes increasingly diluted, yet the consequence does not diminish.

This lesson mirrors the earlier point about “bias dispersion should use discrete-mean ratio, not variance proportion”: proportion-based criteria systematically underestimate consequences in scenarios with large person differences.

Moreover, the “negligible” threshold is set at 5%, yet true consequences have long exceeded this line. When A/S²=0.05, Δc is already 4.0–4.6 percentage points; when A/S²=0.10, it reaches 7.6–8.8. If we define “impact on conclusion” at 5 percentage points, this threshold is crossed when A/S²≈0.06, precisely at the boundary of the “negligible” category.

So what to do: do not report categories; directly report the number. The question is never “what proportion does case drift account for,” but rather “with or without adjustment, how many percentage points does the common model point estimate differ by.” This quantity is Δc = 100·A/(S²+A+T/k), measured in percentage points, and it is itself the answer—no need for three thresholds to translate. If a line must be set, define it as: Δc>5 percentage points, equivalent to A > 0.053×(S²+T/k). The threshold adapts to the scenario itself, without borrowing someone else’s units.

External evidence also supports “this term must be self-tested on this batch of materials.” In one generalized study with 135 students, 2 readers, and 3 time points, occasion variance was negligible and student-occasion interaction was small, while student-rater interaction accounted for 17.7% and the third-order term including residuals accounted for 27.7%, with Φ coefficient 0.75 (PMID 32581621)—note it reports “small,” but that is “small” under a specific structure. Another study tracking 458 teachers over two years found the opposite: rater drift was very large initially and persisted for nearly two years; raters did not converge to common severity, and their differences actually widened over time; the trend term relative to rater effect, specific lesson scoring error, and residuals counts as “moderate” rather than main term (PMID 29795823). Moreover, resampling method itself substantially affects drift detection conclusions, with marginal homogeneity test having low power when displacement crosses categories (PMID 29881127). A systematic review of 44 studies plus meta-analysis yielded aggregate figures: person variance 44.2%, G coefficient 0.65, and explicitly noted “contextual factors affect variance components; reporting rigor still needs strengthening” (PMID 33951677).

In one sentence: case drift is not a constant to look up in a table; it is an attribute of each batch of materials; and the term used to judge it, its denominator, and its degrees of freedom—all three cannot be copied.

Evidence and Limitations

Verified: σ²(o:r) degrees of freedom is R×(O−1), independent of case count, which is an algebraic fact of variance decomposition; simulation where P increased from 80 to 800 with interval width shrinking only from 156% to 147% fully matches. The identity Δc = A/(S²+A+T/k) can be directly derived. Use of generalized theory in rating studies and heterogeneity of variance components have systematic review-level external support (PMID 33951677, 32581621, 29795823).

Partially verified: common drift enters occasion main effect, specific drift enters interaction term; this split holds under fully crossed design; but if actual design is “each person rates each reader at each time point only once,” person×reader term and third-order residual are completely confounded (in simulation σ²(p×r) true value 5.0, estimated median −26 to −7, 100% negative), causing σ²(o:r) itself to be systematically suppressed by 19–66%. Specific threshold values (5%/20%) come from the original statement of the previous criterion; this paper only argues that its denominator and selection target are wrong, without independently verifying the origin of the thresholds themselves.

To be verified: ① actual proportion of “common drift” vs “specific drift” in real studies; currently only scattered reports indicate occasion variance is “negligible,” lacking cross-study distribution; ② whether the 5-percentage-point threshold for Δc corresponds to actual decision consequences (whether it changes the conclusion of “common model exists”), with no calibratable external standard yet; ③ if drift is monotonic trend rather than random fluctuation, variance decomposition spreads the trend into residuals, requiring this paper’s split formula to be rewritten as an augmented model with trend terms.

Related Reading

Evidence level: C
References: PMID 32581621, PMID 29795823, PMID 29881127, PMID 33951677
L1 network status: All keywords in EDGE_TABLE_science-yixue.md—”generalization/variance/drift/time point/retest/rater/kappa/reliability/consistency/error/correction/heterogeneity”—have zero hits; this topic has not entered the corresponding L1 network. The closest network-side label is E-060 premise (KB-317/333/334, status conflict unresolved), so this paper’s evidence level is capped at C.

Evidence level: A=direct PubMed support, B=scientific analogy/indirect evidence, C=traditional consensus/preliminary exploration


Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *