「评分者越判越松」还是「及格线越划越严」?一个可能无法从数据里回答的问题

Written by

in

Language

Evidence level: C

一句话:漂移压低合格率这个「方向结论」,可能整个建立在「切点不动」这个没人验证过的假设上——而漂移与切点移动在观测上可能根本分不开。

正文

前一篇已经算出一个很漂亮的方向结论:当评分者的尺度出现漂移(同一批人越判越不齐)时,合格率的期望会被单向压低——在漂移标准差 0.515、真实合格率 0.95 的设定下,期望合格率被拽低 2.18 个百分点,这个凹性缺口占到一阶项的 41.1%。结论听起来很干净:漂移是坏东西,它系统性压低你看到的合格率。

但这个结论里藏着一个前提:切点是固定的。合格线划在那里不动,评分者的尺度在动,于是两者错位,合格率被压低。

问题在于,另一篇文章已经证明了一条更麻烦的事:读者位置与切点组之间存在一条不含任何信息的平移共线——「读者整体变松」与「切点整体下移」在数据上完全等价,你看到的分数分布一模一样。如果把这条等价用到底,就会得到一个颠覆性的可能:漂移根本不是一个独立发生的量,它可以全部被吸收进切点的移动里。此时合格率的期望不变,凹性缺口归零,设计效应退化为 1。换句话说,那个「单向拽低」的方向,取决于研究者把漂移记在读者账上还是切点账上——而这两种记法在观测上不可分。

这不是本领域独有的纠缠。kappa 系数的老毛病就是同一件事的经典形态:Cohen’s kappa 同时受评定者偏倚与类别流行度影响,二者无法从单一的 kappa 值里分离出来(PMID 8501467);著名的「高一致但低 kappa」悖论,可以完全由边际分布失衡造成,与真实一致程度无关(PMID 2348207)。甚至有人反过来主张流行度依赖不构成对 kappa 的实质批评(PMID 15939215)——但注意其论证前提恰是「改变流行度必然同时改变敏感度与特异度」,也就是说,反方同样默认了两本账拆不开。三方争论了几十年,共同点只有一个:没人承诺能把两种记账从同一份数据里分开。

那这个方向判断还救得回来吗?有一条窄缝:设计效应 k 依赖读者间的方差 Var(b),而纯粹的切点移动不产生这一项。k 可能是唯一能把两种记账拆开的统计量。

那条窄缝为什么存在?机制上可以说明白:设计效应 k 的分子里有读者间方差 Var(b)——它量的是「不同评分者的合格率彼此差多远」。纯粹的切点移动是所有人共享的一次平移:每个评分者相对别人的位置纹丝不动,Var(b) 一点不变;而读者漂移是各人往不同方向散开,Var(b) 必然被撑大。所以在「分数分布相同」这一约束下,两种记账若要维持 k 一致,就必须在 Var(b) 上做不一样的文章——这正是唯一可能留下指纹的地方。同理,若观察到 k 显著大于 1 而分数分布与「纯切点移动」版本无法区分,就可以把「漂移属读者侧」从假设升级为有观测内容的判断。

可做的实测很直接:模拟三个模型——① 只有读者漂移(切点固定);② 只有切点移动(读者固定);③ 两者按不同比例混合但总效应相同——让三者在完全相同的观测分数分布下生成数据,然后比较三个读数:期望合格率 m̂、设计效应 k、通过率。

判据写死:若三者在 m̂、k、通过率上全部一致(差 <0.5 个百分点),则「漂移压低合格率」这个方向断言不可证伪,必须从结论里撤回,改报「在该等价类下合格率的期望不变」——这是一种诚实的降级。若只有 k 能区分三者的记账方式,则 k 必须上升为必报项:以后任何报合格率的研究,不同时报 k 就等于没交代漂移记在哪本账上。若三者可分、且读者漂移版的缺口显著大于切点版(差 >1 个百分点),则原方向结论成立——但必须显式写出它的前提:漂移确属读者侧,这个前提本身需要独立证据,不能靠数据反推。

对易学判读研究的含义很实际:五脏辨证、脉象分级这类多评定者判读,若发现「合格率随年份下降」,先别急着归因于「传承在退化」——先问一句:是判读者的尺度散了,还是验收线被悄悄挪了?在拿出 k 之前,这两种解释的分量是一样的。

证据与局限

已验证:三个 PMID 确立了 kappa 的偏倚-流行度等价类是真实的、且无法从单一统计量分离(PMID 8501467, 2348207, 15939215)。
部分验证:平移共线本身来自同系列前篇的推导(#1420),尚未对真实判读数据做过直接演示。
待验证:三模型模拟的全部判据(三读数是否可分、k 是否唯一拆分统计量、缺口差是否 >1pp);「漂移记在读者账上」这一前提目前没有任何独立证据。

相关阅读

证据等级: C
参考文献: PMID 8501467, PMID 2348207, PMID 15939215

L1 网络状态:该主题尚未进入 L1 对应网络。逐词检索(漂移/切点/合格率/等价类/kappa/信度/评定者)未见对应边;最接近的是 E-060 标签前提(conflict,未裁决):「中医医师间诊断一致性在开放人群中『低』,作者称为『令人担忧』(KB-292,PMID 30383398);偶然校正一致性统计常被误用、建议 Gwet AC2(KB-317/333/334)」——该冲突链只覆盖「标签信度低」这一前提,不含漂移方向与切点等价问题。


免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。

Is the grading curve becoming more lenient or is the passing threshold becoming stricter? A question that may not be answerable from the data.

Evidence level: C

In one sentence: The directional conclusion that “drift lowers the pass rate” may rest entirely on the unverified assumption of a “fixed cut-off”—while drift and cut-off movement may be observationally inseparable.

Body

The previous article derived a clean directional conclusion: when scoring scales drift (the same group of people becomes increasingly inconsistent), the expected pass rate is unidirectionally depressed. Under settings of drift standard deviation 0.515 and true pass rate 0.95, the expected pass rate is pulled down by 2.18 percentage points; this concave gap accounts for 41.1% of the first-order term. The conclusion sounds neat: drift is bad; it systematically depresses the pass rate you observe.

But this conclusion hides a premise: the cut-off is fixed. The passing line is drawn and stays put, while the scoring scales move, creating misalignment and depressing the pass rate.

The problem is that another article proved a more troublesome fact: there exists a translation collinearity containing no information between reader positions and the cut-off group—”the whole reader set becomes looser” is data-equivalent to “the cut-off shifts downward.” If this equivalence holds, it yields a disruptive possibility: drift is not an independent quantity; it can be entirely absorbed into the movement of the cut-off. In this case, the expected pass rate remains unchanged, the concave gap vanishes, and the design effect degenerates to 1. In other words, that “unidirectional pull-down” direction depends on whether the researcher attributes drift to the readers or the cut-off—and these two attributions are observationally inseparable.

This entanglement is not unique to this field. The classic trouble of kappa coefficients is the canonical form of the same issue: Cohen’s kappa is simultaneously affected by rater bias and category prevalence, and neither can be separated from the single kappa value (PMID 8501467); the famous “high agreement but low kappa” paradox can be entirely explained by marginal distribution imbalance, regardless of true agreement (PMID 2348207). Even some argue that prevalence dependence does not constitute a substantive critique of kappa (PMID 15939215)—note that their argument premise is precisely “changing prevalence necessarily changes sensitivity and specificity,” meaning the opposing side also defaults to the two accounts being inseparable. Three parties have debated for decades, with one common point: no one promises to separate the two bookkeeping methods from the same data.

So can this directional judgment be salvaged? There is a narrow gap: the design effect $k$ depends on the variance between readers, $\text{Var}(b)$, whereas pure cut-off movement does not produce this term. $k$ is the only statistic that can separate the two bookkeeping methods.

Why does this narrow gap exist? Mechanistically, it can be explained: the numerator of the design effect $k$ contains $\text{Var}(b)$, which measures “how far apart the pass rates of different raters are.” Pure cut-off movement is a shared translation for everyone: each rater’s position relative to others remains unchanged, so $\text{Var}(b)$ does not change; reader drift scatters individuals in different directions, inevitably inflating $\text{Var}(b)$. Under the constraint of “identical score distributions,” if both bookkeeping methods are to maintain consistent $k$, they must differ in $\text{Var}(b)$—this is the only place a fingerprint could leave a trace. Similarly, if $k$ is observed to be significantly greater than 1 while the score distribution cannot be distinguished from the “pure cut-off movement” version, then the assumption that “drift belongs to the reader side” can be upgraded into an observationally grounded judgment.

The practical test is straightforward: simulate three models—① only reader drift (fixed cut-off); ② only cut-off movement (fixed readers); ③ both mixed in different proportions but with identical total effect—generating data under completely identical observed score distributions, then comparing the three readings: expected pass rate $\hat{m}$, design effect $k$, and pass rate.

The criteria are fixed: if all three are identical in $\hat{m}$, $k$, and pass rate (difference < 0.5 percentage points), then the directional assertion “drift lowers the pass rate” is unrefutable and must be withdrawn from the conclusion, replaced by “under this equivalence class, the expected pass rate remains unchanged”—this is an honest downgrade. If only $k$ can distinguish the three bookkeeping methods, then $k$ must become a mandatory reporting item: any future study reporting pass rates without $k$ fails to disclose where drift is attributed. If all three are separable, and the gap in the reader-drift version significantly exceeds that of the cut-off version (difference > 1 percentage point), then the original directional conclusion holds—but it must explicitly state its premise: drift indeed belongs to the reader side, a premise requiring independent evidence, not inferred from the data**.

The implications for Yijing Studies are practical: multi-rater assessments such as five-organ syndrome differentiation or pulse diagnosis grading, if they show “pass rates declining over years,” should not immediately attribute this to “degenerating transmission.” First ask: is the scale of the raters scattering, or is the acceptance line being quietly shifted? Until $k$ is reported, the weight of these two explanations is identical.

Evidence and Limitations

Verified: Three PMIDs establish that the kappa bias-prevalence equivalence class is real and cannot be separated by a single statistic (PMID 8501467, PMID 2348207, PMID 15939215).
Partially verified: The translation collinearity itself derives from derivations in the preceding articles of the same series (#1420); it has not yet been directly demonstrated on real assessment data.
To be verified: All criteria of the three-model simulation (whether the three readings are separable, whether $k$ is the unique splitting statistic, whether the gap difference exceeds 1pp); the premise “drift attributed to the reader side” currently has no independent evidence.

Related Reading

Evidence level: C
References: PMID 8501467, PMID 2348207, PMID 15939215

L1 network status: This topic has not yet entered the L1 corresponding network. Keyword search (drift/cut-off/pass rate/equivalence class/kappa/reliability/rater) yields no corresponding edges; the closest is the E-060 label premise (conflict, unresolved): “Diagnostic consistency among TCM physicians is ‘low’ in the open population, termed ‘worrying’ by the author (KB-292, PMID 30383398); occasional correction consistency statistics are often misused; Gwet AC2 is recommended (KB-317/333/334)”—this conflict chain covers only the premise of “low label reliability,” not the drift direction and cut-off equivalence issue.


Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *