Evidence level: C
让更多人来读,能把「读的人自己抖」平均掉,却一分钱也平均不掉「被读的东西自己在变」——前者按人数的平方根收敛,后者是一个乘在真值上的常数。
一、「多找几个人」这句话,其实混了两件不同的事
一致性研究里最常听到的处方是「加人」。两位读者算不出一致性,就找四位;四位还不够,就找八位。理由听起来很硬:人多,噪声就被平均掉了。
这个理由对一半。把这件事拆成两层看,加人的效果完全不一样:
第一层是精度。 每位读者都带一份自己的抖动——今天手松、明天手紧、同一张片子看两遍给两分。这份抖动彼此独立,k 位读者一起读,它就被平均成原来的 1/√k。加人确实让估计更稳,这是真的。
第二层是准头。 如果所有读者其实都在同一件「会变的东西」上做判断,那 k 位读者加起来仍然只有那一件东西。加人只会把它更精确地量出来,不会让它变成另一件东西。精度上去了,准头不动。
一条老规矩早就说清楚:一致性系数的置信区间与样本量,必须按「评定者数」和「病例数」两个方向分别算(PMID 15733050)。可实践里几乎没人分开算,于是「加人」被当成包治百病的动词用。
二、把两条路写成一个式子,就看出加人管用在哪一段
设每位读者对每个病例给出的读数,由三块叠加而成:
- 一块是所有人共同感受到的信号——那个我们真正想测的「共同模」,方差记作 S²;
- 一块是读者各读各的抖动,方差 T(含读者自己的特异偏移与偶然误差);
- 一块是病例本身在变——材料难度漂移、时点效应、样本异质性,方差记作 A。
读者随机、病例随机、两者完全交叉时,任意两位读者读数之差(一切一致性系数赖以计算的原料)只含中间那一块:差的方差是 2T,A 被彻底抵消了。这是完全交叉设计最漂亮的地方——病例怎么变都无所谓,因为大家读的是同一批病例。
再看「k 位读者取平均」这条路。平均值的方差是 S² + A + T/k。用平均值方差扣掉从分差里估出来的 T/k,剩下 S² + A,开根号就是 √(S² + A)。
答案就在式子里:k 只出现在 T/k 那一项,S² + A 里没有任何 k。
加多少人都只能把 T/k 压薄,压不掉 A。估出来的「共同模」天生是 √(S²+A),不是 √S²。
仿真实测也一致:N = 200、真值 S = T = 1,只要 A 不为零,无论读者从 2 位加到 50 位,估计值都稳定偏高同一个百分比——A 占 S 的四成六时偏高 10%,占七成一时偏高 22%,正好等于 √(1+(A/S)²)−1。覆盖率在 A=0 时能从 63% 爬到 94%;A 一旦达到 S 的 0.71 倍,就一路从 14% 掉到 2.6%:加人越多,越精确地停在一个错的位置上。
三、「加人」的门槛可以算死:n=200 时 k 的边际收益断崖
A = 0 的干净情形下(N = 200,覆盖率 = 点估计落在真值 ±10% 内的比例):
| 读者数 k | 2 | 3 | 4 | 5 | 6 | 10 | 20 | 50 |
|---|---|---|---|---|---|---|---|---|
| 覆盖率 | 63% | 75% | 80% | 84% | 85% | 91% | 93% | 94% |
三件事:第三位读者最值钱(一步抬 12 个百分点);第五位之后基本到头(第 5→6 位只加 2 个百分点);40 位也换不到 95%——天花板不在人身上,在病例数。N = 200 时相对标准误的理论地板是 1/√(2(N−1)) ≈ 5%,实测 5.2%,两边对得上;N = 50 时覆盖率只有 67%,N = 800 才到 100%。「加人」有个隐形上限,跨过去之后要多的是病例,不是读者。
按覆盖率达到八成算门槛:A = 0 时 k = 4;A 达到 S 的四分之一时 k ≈ 7;A 达到三分之一时 k ≈ 16;A 到 0.46 倍时连 N = 200 加 6 位读者都做不到(覆盖率 53%);A 到 0.71 倍时无论如何过不了 5%。
也就是说:加人这条处方只在「病例几乎不变」时便宜好用;A 只要超过 S 的三分之一,它就从「省钱补救」变成「花大钱买精度、还买错了方向」。
四、一条更扎心的推论:换设计也换不来多一个方程
有人会说,那就别让所有人读同一批病例,改成两半区组、中间留一批重叠病例当桥——这是不完全设计的标准做法,看起来像给「共同模」多添了一条独立方程。
仿真实测(N = 200、8 位读者分两组、重叠 20 例):A = 0 时覆盖率 71%,A = 0.71 时 14%,偏差一分没减。
原因是式子没变:分组后每组各算一份平均值方差、再用组内分差扣掉各自的 T/g,剩下的仍是 S² + A。桥接只让「两侧可比」成立,不让 A 从式子里消失。 不完全设计的真本事在别处:它能把病例难度当成一项可估参数估出来(PMID 37810544 的模拟显示,稀疏设计下多面 Rasch 与概化理论是否还能读准,本身就是要检验的问题,不是自动成立)。而「估出来」这一步,恰恰需要有人先给病例难度一个独立于读者读数的量尺,不能靠读者自己再读一遍。
五、这个式子同时把「扣掉」的代价算清楚了
既然加人不行,剩下的路就是:先量出 A 有多大,再从平均值方差里扣掉 A²。
代价可以精确算。真 A/S = 0.6、读者 50 位、N = 400 时,标定完全准则覆盖率 95%;高估 10% 掉到 87%,高估 20% 掉到 60%,高估 50% 只剩 1%;低估 20% 掉到 78%。这里的不对称值得记住:高估 A 比低估 A 危险得多,因为高估会把 S² 扣成负的、压低估计值,而压低看着不像错误,更像一个「更保守」的结果。
这也解释了为什么「共同模」在实践里总是偏乐观:不扣减它一定偏高,做了扣减又极易扣过头。
证据与局限
已验证:这是代数结论,不是经验发现。在「读者随机、病例随机、完全交叉」的模型下,任意两位读者之差的方差恒等于 2T,病例项 A 必然抵消;k 位读者平均值的方差恒为 S² + A + T/k。由此「加人只压 T/k、不压 A」是恒等式级别的结果,不依赖任何模拟。上述覆盖率数字来自 4000–6000 次重抽的仿真(S = T = 1,N = 200),与解析式偏差一致到小数点后一位。可靠性与一致性研究报告规范历来要求同时报样本量与区间估计(PMID 21130355),「两个方向分别算」这件事在方法学上早有出处(PMID 15733050)。
部分验证:病例/时点项的方差占比在真实数据里到底有多大,现有文献给出的数差得很远——某项持续两年的课堂观察研究做方差分解后认为趋势项只占中等份额(PMID 29795823),而概化研究里读者×时点交互既可能接近零、也可能占到可观的份额(PMID 32581621)。这说明 A 不是可以一律当零处理的量,但也说明「A 通常多大」目前没有可用的先验。稀疏设计下病例难度参数能否读准,已有模拟指出需要专门检验(PMID 37810544);多面 Rasch 里严厉度随时间的漂移必须显式建模、独立性假设下估计效率很低,需引入马尔可夫先验才稳(PMID 36284065),这也从另一个角度说明「时间维」不能被当成噪声吸收掉。
待验证:第一,本文全部计算建立在「A 对所有读者是同一份」的假定上。若不同读者面对的病例漂移不同(例如分批读、批次间材料换过),则 A 不再是一个标量,而是一个与读者下标相关的结构,此时式子要重写,结论可能从「常数偏高」变成「读者间不可比」。第二,本文把「共同模」当成一个可加分量。真实判读里共同模更可能作用在判断门槛上而非读数上(读者一起变松或紧),此时它进入的是分差而非均值,抵消关系整个反过来——这条已另立课题。第三,A 的独立量尺从哪来(客观难度代理、重复测量、外部分级),本文只指出必须有,没有给出可行性边界。
L1 网络状态:L1 对应网络中未检索到该主题对应的边。原文摘录(E-060 标签前提行):「KB-292(conflict):中医医师间诊断一致性在开放人群中「低(low)」,作者称为「令人担忧」(PMID 30383398)」;同一边另载「KB-317(conflict):闸门规格撞上方法学反例,两处对不上」。全网络 1956 行内「kappa」0 次、「可辨识」0 次、「共同模」0 次,与本文相关的仅「评定者」4 处、「信度」12 处,且全数落在 E-060 的标签信度前提上,状态为 conflict(未裁决)**,无任何 A 级证据支撑本文的代数结构。故本文证据等级封顶 C,缺口已登记至 l1-gaps 第 62 条。
相关阅读
- 两位判读者是一起漂还是各漂各的:这个相关系数不是「没人量过」,而是「两个人的数据里量不出来」
- 加锚这件事最怕的不是锚不准,而是锚和读者错在同一处
- 位置参数的可辨识性:为什么「平移」这种最朴素的漂移反而最难估
- 池1-共振意识场:测量与被测量之间的接口
证据等级: C
参考文献: PMID 15733050, PMID 21130355, PMID 29795823, PMID 32581621, PMID 37810544, PMID 36284065
免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。
Adding people improves precision but not truth: readers suppress noise, yet cannot suppress the material itself changing.
Evidence level: C
Increasing the number of readers averages out the “jitter” of the readers themselves (e.g., hand tremor, fatigue), which converges at the square root of the number of readers. However, it cannot average out the fact that the material being read is changing, which acts as a constant multiplier on the true value.
I. “Finding more people” mixes two distinct issues
The most common prescription in consistency studies is “add more raters.” If two readers cannot calculate agreement, find four; if four are not enough, find eight. The rationale sounds solid: more people average out the noise.
This rationale is only half true. Breaking this down into two layers reveals that adding raters affects the two components differently:
The first layer is precision. Each reader brings their own jitter—loose hands today, tight hands tomorrow, or reading the same image twice and giving it two different scores. These jitters are independent; with $k$ readers, the jitter averages out to $1/\sqrt{k}$. Adding raters does indeed make the estimate more stable; this is true.
The second layer is accuracy. If all readers are judging the same drifting thing, then $k$ readers combined still only measure that one thing. Adding raters merely measures it more precisely, it does not turn it into a different thing. Precision goes up, accuracy stays the same.
An old rule already clarifies this: the confidence interval for the consistency coefficient must be calculated separately along two directions—”number of raters” and “number of cases” (PMID 15733050). Yet in practice, almost no one calculates them separately, so “adding raters” is used as a catch-all verb.
II. Writing the two paths into one equation reveals where adding raters helps
Let each reader’s reading for each case be composed of three stacked components:
- One component is the signal perceived by everyone—the “common mode” we truly want to measure, with variance denoted $S^2$;
- One component is each reader’s individual jitter, with variance $T$ (including each reader’s specific bias and random error);
- One component is the case itself drifting—material difficulty drift, timing effects, sample heterogeneity, with variance denoted $A$.
When readers are randomized, cases are randomized, and they cross completely, the difference between any two readers’ readings (the raw material for all consistency coefficients) contains only the middle component: the variance of the difference is $2T$, $A$ is completely canceled out. This is the most elegant feature of the fully crossed design—the cases can drift however they like, because everyone is reading the same set of cases.
Now consider the path of “averaging $k$ readers.” The variance of the average is $S^2 + A + T/k$. Subtracting the $T/k$ estimated from the pairwise differences leaves $S^2 + A$, and taking the square root gives $\sqrt{S^2 + A}$.
The answer lies in the equation: $k$ appears only in the $T/k$ term; there is no $k$ in $S^2 + A$.
Adding more raters only thins out the $T/k$ term; it cannot reduce $A$. The estimated “common mode” is inherently $\sqrt{S^2 + A}$, not $\sqrt{S^2}$.
Simulation results confirm this: with $N = 200$, true values $S = T = 1$, as long as $A \neq 0$, regardless of whether readers go from 2 to 50, the estimate remains consistently biased upward by the same percentage—10% when $A$ is 46% of $S$, 22% when $A$ is 71% of $S, exactly matching $\sqrt{1 + (A/S)^2} – 1$. Coverage rises from 63% when $A = 0$ to 94%; once $A$ reaches $0.71 \times S$, coverage plummets from 14% to 2.6%: the more raters you add, the more precisely you stop at the wrong place.
III. The threshold for “adding raters” can be calculated precisely: marginal benefit cliffs at $n = 200$
In the clean case where $A = 0$ ($N = 200$, coverage = proportion of point estimates falling within $\pm 10\%$ of the true value):
| Number of raters $k$ | 2 | 3 | 4 | 5 | 6 | 10 | 20 | 50 |
|---|---|---|---|---|---|---|---|---|
| Coverage | 63% | 75% | 80% | 84% | 85% | 91% | 93% | 94% |
Three takeaways: the third rater is most valuable (jumping 12 percentage points); after the fifth rater, it’s basically done (5→6 adds only 2 percentage points); even 40 raters cannot reach 95%—the ceiling is not in the number of people, but in the number of cases. With $N = 200$, the theoretical floor for relative standard error is $1/\sqrt{2(N-1)} \approx 5\%$, observed at 5.2%, which matches; with $N = 50$, coverage is only 67%, and $N = 800$ is needed for 100%. “Adding raters” has an invisible ceiling; beyond it, you need more cases, not more raters.
Setting the threshold at 80% coverage: when $A = 0$, $k = 4$; when $A$ reaches one-quarter of $S$, $k \approx 7$; when $A$ reaches one-third of $S$, $k \approx 16$; when $A$ reaches $0.46 \times S$, even adding 6 raters to $N = 200$ fails (coverage 53%); when $A$ reaches $0.71 \times S$, you cannot cross 5% under any circumstances.
In other words: the prescription of “adding raters” is only cheap and effective when cases hardly change; once $A$ exceeds one-third of $S$, it shifts from “saving money to fix accuracy” to “spending big money on precision but getting the wrong direction.”
IV. A more sobering inference: changing the design does not add another equation
One might argue, then do not have everyone read the same cases; split them into two blocks with an overlapping set of cases as a bridge—this is the standard approach for incomplete designs, appearing to add an independent equation for the “common mode.”
Simulation results ($N = 200$, 8 raters split into two groups, overlap of 20 cases): when $A = 0$, coverage is 71%; when $A = 0.71$, it is 14%, with zero reduction in bias.
The reason is that the equation does not change: after grouping, each group calculates its own average variance, then subtracts $T/g$ from the within-group differences; what remains is still $S^2 + A$. The bridge only ensures comparability between the two sides; it does not make $A$ disappear from the equation. The real power of incomplete designs lies elsewhere: it allows case difficulty to be estimated as an additional parameter (PMID 37810544’s simulation shows that under sparse designs, whether multi-facet Rasch and Generalizability Theory can still estimate it accurately is itself a question to be tested, not automatically true). And this “estimation” step precisely requires someone to provide a scale independent of reader readings for case difficulty, not relying on readers reading it again.
V. This equation also calculates the cost of “subtracting” $A$
Since adding raters is not viable, the remaining path is: first measure how large $A$ is, then subtract $A^2$ from the average variance.
The cost can be precisely calculated. With true $A/S = 0.6$, 50 raters, and $N = 400$, the calibration criterion coverage is 95%; overestimating $A$ by 10% drops it to 87%, overestimating by 20% drops it to 60%, overestimating by 50% leaves only 1%; underestimating by 20% drops it to 78%. This asymmetry is worth remembering: overestimating $A$ is far more dangerous than underestimating it, because overestimation subtracts $S^2$ into negative values, depressing the estimate; this depression looks less like an error and more like a “more conservative” result.
This also explains why the “common mode” is always optimistically biased in practice: not subtracting it makes it high, while subtracting it risks subtracting too much.
Evidence and Limitations
Verified: This is an algebraic conclusion, not an empirical discovery. Under the model where “readers are randomized, cases are randomized, and they cross completely,” the variance of any two readers’ difference is always $2T$, so the case term $A$ must cancel out; the variance of the average of $k$ readers is always $S^2 + A + T/k$. Thus “adding raters only reduces $T/k$, not $A$” is an identity-level result, independent of any simulation. The coverage numbers come from simulations with 4000–6000 resamples ($S = T = 1$, $N = 200$), matching the analytical formula to one decimal place. Reliability and consistency reporting standards have long required reporting both sample size and interval estimates (PMID 21130355), and “calculating separately along two directions” has methodological precedent (PMID 15733050).
Partially verified: How large the variance of the case/timing term actually is in real data varies widely across literature. One two-year classroom observation study decomposed variance and found the trend term accounts for only a medium share (PMID 29795823), whereas in generalizability studies, the reader×timing interaction may be near zero or occupy a substantial share (PMID 32581621). This shows $A$ is not a quantity that can always be treated as zero, but also indicates there is currently no prior available for “how large $A$ usually is.” Whether case difficulty parameters can be estimated accurately under sparse designs has been shown via simulation to require specific testing (PMID 37810544); in multi-facet Rasch, drift in severity over time must be explicitly modeled, and under independence assumptions estimation efficiency is low, requiring Markov priors to stabilize (PMID 36284065), which also shows from another angle that the “time dimension” cannot be absorbed as noise.
To be verified: First, all calculations in this paper assume $A$ is the same for all readers. If different readers face different case drifts (e.g., batch reading with material changes between batches), then $A$ is no longer a scalar but a structure dependent on reader index, requiring rewriting of the equation; conclusions may shift from “constant upward bias” to “incomparable readers.” Second, this paper treats the “common mode” as an additive component. In real judgment, the common mode more likely acts on the judgment threshold rather than the reading itself (readers collectively loosen or tighten), so it enters the difference rather than the mean, reversing the cancellation relationship—this is a separate research topic. Third, where does the independent scale for $A$ come from (objective difficulty proxy, repeated measurement, external grading)? This paper only notes it must exist, without specifying feasibility boundaries.
L1 Network Status: L1 corresponds to edges not retrieved in the network for this topic. Original excerpt (E-060 label premise line): “KB-292 (conflict): Diagnostic consistency among TCM physicians is “low (low)” in open populations; authors call it “concerning” (PMID 30383398); the same edge also carries “KB-317 (conflict): Gate specifications clash with methodological counterexamples; two sides do not match.” Across the entire network of 1956 lines, “kappa” appears 0 times, “identifiable” 0 times, “common mode” 0 times; only “raters” (4 occurrences) and “reliability” (12 occurrences) are related to this paper, all falling under E-060’s label reliability premise, status conflict (unresolved), with no A-level evidence supporting the algebraic structure of this paper. Hence, the evidence level for this paper is capped at C, and the gap has been registered as item 62 in l1-gaps.
Related Reading
- Are two raters drifting together or drifting separately: this correlation coefficient is not “unmeasured,” but “cannot be measured from the two people’s data”
- The hardest part about adding anchors is not inaccurate anchoring, but the anchor and reader erring in the same place
- Identifiability of position parameters: why “translation,” the simplest drift, is hardest to estimate
- Pool 1—Resonance Consciousness Field: Interface between measurer and measured
Evidence Level: C
References: PMID 15733050, PMID 21130355, PMID 29795823, PMID 32581621, PMID 37810544, PMID 36284065
Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.