Evidence level: C
想量「两人漂移的相关系数」,用两个人的分数是量不出来的——三个量里只有两个独立信息。要量它,必须往设计里加一个不是人的东西。
一、先把这个设问的算式摆正
上一批文章里有一条结论:实测漂移是临界限的 1.5 到 3.8 倍,撞线概率约七成。这条结论站在一个式子上——两人分差的标准差等于单个漂移标准差乘根号二。根号二从哪来?从「两人各漂各的、互不相干」这个前提来。
这个前提从没被检验过,一松就全翻。若两人一起往同方向松(同一套培训、同一时段、同一份标准一起走样),分差被削掉大半,撞线概率掉一个量级。
但有一件事必须先说清:「量这个相关系数」这个动作,用两位判读者自己的数据做不到。
二、三个量、两个方程:为什么不可辨识
把每次评分的严厉度拆成三块:读者自己的常数偏倚、随时间一起走的共同成分、读者与时间的交互(各漂各的)。可观测的只有两个统计量——两人分差的方差只由交互那一块决定,共同成分在里面被差掉了;两人均值的方差则由共同成分、交互成分的一半、以及案例难度漂移三者相加。
两个方程、三个未知数。解不唯一。数值核算:固定交互项 0.30,让共同成分与案例漂移从零往上走,估出的相关系数会从 0.000 涨到 0.500、再到 0.667,而真值一直是 0。只要案例难度也在漂,估出来的「共同模」有一半是案例漂移冒充的。
这正是多面 Rasch 模型里反复记录的一件事:严厉度与题目难度的漂移在观测上混在一起,看分数分不开(PMID 37283589 指出忽略评分顺序会产生有偏的模型结构与能力估计;PMID 36284065 给严厉度加马尔可夫先验,正因时点独立假设不够用)。
三、原文那两个数字对不上
顺手核了一下上一批给出的两个数:分差标准差 0.44、共同模下 0.098。按根号二关系反解,0.098 对应相关系数 0.9504,不是文中暗示的 0.9。而按七成撞线率反解临界限(约 0.170),相关系数 0.9 对应的分差标准差是 0.139,撞线率落在 22.3%,也不是 8.8%。
方向没错——共同模确实把撞线率大幅压低——但幅度被夸大了。这类「方向对、数字没核」的算式,正是这条支线反复出问题的地方。
四、外部证据怎么说
一项持续近两年、458 名教师的观察评分研究,直接测了「评分者会不会收敛到同一水平」:结论是没有收敛,评分者之间的差异在整个研究期内反而在扩大(PMID 29795823)。这支持「各漂各的」这一侧。
另一项 16 名考官的前后对照研究,把严厉度反馈给考官本人,期望他们向共同标准靠:干预没有显著效应,且严厉度在干预前就已不稳定;作者写「校准对考官而言似乎很难」(PMID 28587597)。这削弱了「共享干预产生共享漂移」这条机制。
还有一项指出:漂移检测依赖「重评旧材料」的设计,而重评样本的抽样方式会实质影响结论(PMID 29881127)。连带「漂移有多大」这个上游量都还没被稳稳测住。
五、那要怎么量
加人。 三位以上判读者、构成连通网络,共同成分才可能从两两分差的方程组里分离出来。
加锚。 同一批材料在不同时点由固定一组校准读者重评,用他们的漂移当共同成分读数。但锚自己也在漂,这一步只是把不可辨识变成条件可辨识。
加非人通道。 案例难度漂移若能由材料本身的客观指标(时长、复杂度、客观检查条目数)预估,就能从均值方差里扣掉,剩下的才归给共同模。
判据建议:若三条路都做不到,「共同模有多大」在任何单一研究中都不该给点估计,只能报区间,且区间须宽到同时包含「完全独立」与「完全共模」两个端点。 这不是保守,是数据里真的没那么多信息。
证据与局限
已验证:分差方差只反映交互成分、不反映共同成分,这是代数事实(本文数值核算,固定交互项 0.30 时真值 0 可估出 0.500);持续两年的观察评分研究实测到差异扩大而非收敛(PMID 29795823);严厉度反馈干预无显著效应(PMID 28587597)。
部分验证:漂移确实存在、量级不小,这在多面 Rasch 与概化理论的多次应用中反复出现(PMID 36284065、PMID 37283589);但这些研究都没把共同成分单独分离出来报过。
待验证:共同模的真实大小;案例难度漂移与评分者漂移的相对量级;加入第三位读者或客观锚之后区间能收窄多少。三项全部未被实测,本文所有数字均为模型推演,不可当实测值引用。
相关阅读
证据等级: C
参考文献: PMID 29795823, PMID 28587597, PMID 37283589, PMID 36284065, PMID 29881127L1 网络状态(原文摘录,E-060 标签前提,CORRESPONDENCE_NETWORK.md 第 202 行):「KB-292(conflict):中医医师间诊断一致性在开放人群中『低(low)』,作者称为『令人担忧』(PMID 30383398)……但训练可改善(PMID 19538100)、量表化可行……→ 『用五脏标签做分组』这一步本身带噪声」,层级「冲突(未裁决)」,状态 conflict。该边只裁决「标签整体信度低不低、训练能不能改善」,从未涉及「漂移是共同还是独立」这一层,故本文证据等级封顶 C。
免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。
Are the two raters drifting together or drifting apart: this correlation coefficient is not \”unmeasured,\” but \”impossible to measure from the two people’s data.\
Evidence level: C
To quantify the “correlation coefficient of drift between two individuals,” using the scores of the two people alone is insufficient—only two independent pieces of information exist among three quantities. To measure it, one must introduce a non-human element into the design.
I. Correcting the Equation for This Question
A conclusion from a previous article states: measured drift ranges from 1.5 to 3.8 times the threshold, with a collision probability of approximately seventy percent. This conclusion rests on an equation—the standard deviation of the difference between two raters equals the single drift standard deviation multiplied by the square root of two. Where does the square root of two come from? It stems from the premise that “the two raters drift independently, without influencing each other.”
This premise has never been tested; once relaxed, everything collapses. If the two drift together in the same direction (same training regimen, same time period, same standard deforming simultaneously), the difference is reduced by half, and the collision probability drops by an order of magnitude.
However, one thing must be clarified: measuring this correlation coefficient cannot be done using data from the two raters themselves.
II. Three Quantities, Two Equations: Why They Are Unidentifiable
Let us decompose the severity of each rating into three parts: the rater’s own constant bias, the common component drifting over time, and the interaction between rater and time (drifting independently). Only two statistical quantities are observable—the variance of the difference between raters is determined solely by the interaction component, as the common component cancels out in the difference; the variance of the mean is composed of the common component, half the interaction component, and the case difficulty drift.
Two equations, three unknowns. The solution is not unique. Numerical calculation: fixing the interaction term at 0.30 and letting the common component and case drift increase from zero, the estimated correlation coefficient rises from 0.000 to 0.500 and then to 0.667, while the true value remains 0. As long as case difficulty is also drifting, half of the estimated “common mode” is actually masquerading as case drift.
This is precisely what has been repeatedly recorded in the Multi-Facet Rasch model: severity and item difficulty drift are confounded in observation, making scores indistinguishable (PMID 37283589 points out that ignoring rating order produces biased model structure and ability estimates; PMID 36284065 adds a Markov prior to severity precisely because the independence-of-time assumption is insufficient).
III. The Original Two Numbers Do Not Match
Checking the two numbers given previously: difference standard deviation 0.44, common mode under 0.098. Solving backwards via the square root of two relationship, 0.098 corresponds to a correlation coefficient of 0.9504, not the 0.9 implied in the text. Solving backwards via the seventy percent collision rate (threshold approximately 0.170), a correlation of 0.9 yields a difference standard deviation of 0.139, placing the collision rate at 22.3%, not 8.8%.
The direction is correct—the common mode does indeed significantly suppress the collision rate—but the magnitude is exaggerated. Such “correct direction, unverified numbers” equations are precisely where this branch repeatedly fails.
IV. What Do External Evidence Say?
A two-year observational study of 458 teachers directly measured “whether raters converge to the same level”: the conclusion is no convergence; differences between raters actually widened over the study period (PMID 29795823). This supports the “drifting independently” side.
Another before-and-after study of 16 examiners, where severity feedback was given to the examiners themselves, expecting them to move toward the common standard: the intervention had no significant effect, and severity was already unstable before the intervention; authors wrote “calibration seems difficult for examiners” (PMID 28587597). This weakens the mechanism of “shared intervention producing shared drift.”
There is also a study indicating: drift detection depends on the design of “re-rating old materials,” and the sampling method of the re-rated sample substantially affects conclusions (PMID 29881127). Even “how much drift there is,” this upstream quantity, has not been firmly measured.
V. How Then to Measure It?
Add people. With three or more raters forming a connected network, the common component can only be separated from pairwise difference equations.
Add anchors. The same batch of materials is re-rated at different times by a fixed set of calibrated raters, using their drift as the common component reading. But anchors themselves also drift; this step merely transforms the unidentifiable into conditionally identifiable.
Add non-human channels. If case difficulty drift can be estimated by objective indicators of the material itself (duration, complexity, number of objective check items), it can be subtracted from the mean variance, leaving only what belongs to the common mode.
Judgment recommendation: If none of these three paths is feasible, “how large the common mode is” should not be given a point estimate in any single study; only intervals should be reported, and these intervals must be wide enough to encompass both “completely independent” and “completely common mode” endpoints. This is not conservatism; it is simply that the data contains far less information than assumed.
Evidence and Limitations
Verified: Variance of differences reflects only the interaction component, not the common component, which is algebraic fact (this article’s numerical calculation shows that with fixed interaction 0.30, true value 0 yields estimated 0.500); the two-year observational study measured widening rather than convergence of differences (PMID 29795823); severity feedback intervention had no significant effect (PMID 28587597).
Partially verified: Drift does exist and is substantial, repeatedly appearing in applications of Multi-Facet Rasch and Generalizability Theory (PMID 36284065, PMID 37283589); however, none of these studies separately reported the common component.
Pending verification: True magnitude of the common mode; relative magnitude between case difficulty drift and rater drift; how much the interval narrows after adding a third rater or objective anchors. All three remain unmeasured; every number in this article is model-derived and cannot be cited as observed values.
Related Reading
- How Large Is Position Drift: From “Did They Drift” to “How Much Did They Drift”
- Does “Using Senior as Reference” Go Wrong? The Question Is Not About Frequency
- Resonance Consciousness Field Research Pool
Evidence Level: C
References: PMID 29795823, PMID 28587597, PMID 37283589, PMID 36284065, PMID 29881127L1 Network Status (excerpt from original text, E-060 tag premise, CORRESPONDENCE_NETWORK.md line 202): “KB-292 (conflict): Diagnostic consistency among TCM physicians in open populations is ‘low’ (PMID 30383398), termed ‘concerning’ by authors… but training can improve (PMID 19538100), scaling is feasible… → ‘grouping by five-organ labels’ itself introduces noise’, level ‘conflict (unresolved)’, status conflict. This edge only adjudicates ‘whether label reliability is low and whether training can improve’, never addressing ‘whether drift is common or independent’, hence this article’s evidence level caps at C.”
Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.
Leave a Reply