第 7 名判读者还值不值得请:加人的边际收益由判读者噪声占比决定,而地板由受试者钉死

Written by

in

Language

Evidence level: C

判读者是预算:头三个人的收益占总收益的六成半,第 7 个人起的每一步只剩下不到 5%,而无论请多少人,置信区间半宽都压不破受试者抽样钉死的那块地板。

一、问题:人也是要花钱的

信度研究的设计框架里,样本量从来是两个方向的乘积:多少受试者、多少判读者(PMID 24338600 按目标精度反解样本量,两个方向都进公式)。但「精读判读者数量」这一侧的实操守则一直是空的:判据要求危险档双人独立判读,可没人回答过「从 2 人加到 3 人值多少、从 6 人加到 7 人又值多少」。评分数与受试数存在一个经典的互换关系——增加判读者数可以折算成「有效受试者数」的提升(PMID 16143966),但折算率本身随人数递减,递减得多快、在哪个点上归零,决定了小中心该配 4 人还是 6 人、多中心研究该不该为第 7 名判读者多付一份差旅。

二、模型与数字

取最干净的可加性设定:每个受试者的判读值等于「受试者真值 + 该判读者的噪声」,判读者间相关 ρ 就是真值方差占总方差的比例。估计量取全部两两相关的平均——k 名判读者给出 k(k−1)/2 个配对相关。固定 n=60 例,ρ 取 0.4(判读者分歧较大)与 0.6(较一致)两档,k 扫 2/3/4/6/8/12,每格跑 5000 次重复实验,用估计量的实际标准差乘 1.96 得置信区间半宽(区间方法的选择本身有专门比较,见 PMID 25417040;判读者的特异成分与共享成分如何分账,见 PMID 31898288 与多层概化理论的方差分解 PMID 33818118)。

三、三个结果

第一,收益几乎全在最前面。 ρ=0.4 时半宽从 2 人的 0.219 降到 3 人的 0.162,一步砍掉 26%;此后每档增量依次 15.8%、12.6%、8.4%、5.7%,单调走低。从 2 人到 12 人,名义配对数从 1 涨到 66(涨 21 倍),半宽却只降了 53%——配对数的膨胀率严重虚标,因为所有配对共享同一批受试者,它们不是 66 个独立读数。

第二,4 人对 6 人,差 2.6 到 9 个百分点,取决于 ρ。 4 人半宽 0.136、6 人半宽 0.119:在 ρ=0.4 时,4 人只拿到 6 人精度的 87.4%;在 ρ=0.6 时拿到 91.0%。也就是说「小中心 4 人配置、成本省三分之一」这条守则在判读者较一致的场景成立,在分歧较大的场景要打折扣。

第三,地板由受试者钉死。 把 k 推到 50,半宽仍停在 0.091——从 6 人的 0.119 往下,全部剩余空间只有 24%,加人永远动不了受试者抽样贡献的那块地板。k=6 时已走完从 2 人到无限人总降幅的 78%。

四、判据裁决与设计守则

backlog 预设的两条判据都是分支成立,分支变量恰恰是 ρ:「6 人后每档缩减不足 5% 即封顶」在 ρ=0.6 成立(8 人 5.0%、12 人 4.2%),在 ρ=0.4 不成立(8.4%、5.7%,要拖到 12 人附近才进入 6% 以内);「4 人达 6 人精度 90%」同样只在 ρ=0.6 达标。合并成一条守则:递减速度与 ρ 反向——判读者越一致,越早封顶,因为没剩多少噪声可供平均;判读者分歧越大,加人越划算,但地板也越深。所以「请几个判读者」没有普适答案,必须先报 ρ 的实测档位再定人数;反过来说,一个研究中判读者人数的边际收益曲线本身,就是该场景判读者噪声占比的一面镜子。这与概化系数置信区间方法的蒙特卡洛结论一致:区间精度同时受判读者数与受试数双约束,任何单侧加码都有天花板(PMID 35754612)。

五、待验证

本篇的仿真建立在三个理想化前提上,每一处都是下一步的实测缺口:其一,判读者被假设为等负载、全员评全批——轮换与缺席会改变递减点的位置;其二,预算被假设为「人」与「例」可以分开算——同一笔预算投给 n 还是投给 k,等精度线在 (n,k) 平面上长什么形状没有画过;其三,噪声被假设为同方差——判读者之间噪声不同(有的天生抖)时,「按噪声倒序砍人」是否优于「按人数封顶」值得一算。

证据与局限

已验证: 判读者数与受试数在信度精度公式中互换且边际递减(PMID 16143966);区间精度双约束(PMID 24338600、35754612)。
部分验证: 本文仿真的具体数字(26% 首步收益、87-91% 的 4 人精度占比、0.09 地板)仅在可加性同方差模型下成立,未在真实判读数据上复核。
待验证: 非等负载、异方差判读者、n-k 等精度线三个缺口(见第五节)。

相关阅读

证据等级: C
L1 网络状态: 该主题尚未进入 L1 对应网络。最近邻为 E-060「五脏辨证标签信度」边,当前状态为「冲突(未裁决)」——样本量单位错配(n≥50 指评定者数 vs 120 指病例数,不可直接互校)与精度区间错配(「一致性 low」正落在加权系数精度最差区间)两处待裁。本文仅涉估计精度的仿真推演,不触碰该边的标签效度命题。
参考文献: PMID 16143966, PMID 24338600, PMID 25417040, PMID 33818118, PMID 31898288, PMID 35754612

证据等级:A=有PubMed直接支撑,B=科学类比/间接证据,C=传统文化共识/初步探索


免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。

Is the 7th Judge Worth Hiring? Marginal Benefit Depends on Judge Noise Share, Floor Set by Subjects

Evidence level: C

# 7th-ranked reader still worth hiring? Marginal gain depends on reader noise share; floor is fixed by subject

Reader is budget: the first three readers account for 65% of total gain, each step beyond the 7th reader yields less than 5%, and regardless of how many readers are hired, the confidence interval half-width cannot overcome the floor fixed by subject sampling.

I. Problem: Humans Also Require Funding

In the design framework for reliability studies, sample size is always the product of two directions: the number of subjects and the number of raters (PMID 24338600, solved for sample size based on target precision, with both directions entering the formula). However, the practical guidelines for the “number of raters” side have remained empty: while the criteria require dual independent rating for high-risk categories, no one has answered “what is the value of increasing from 2 raters to 3, or from 6 raters to 7?” There exists a classic reciprocal relationship between the number of scores and the number of subjects—increasing the number of raters can be converted into an increase in “effective subject count” (PMID 16143966). However, the conversion rate itself decreases with the number of raters; how rapidly it decreases and at what point it reaches zero determines whether a single-center study should allocate 4 or 6 raters, and whether a multi-center study should pay additional travel expenses for the seventh rater.

II. Models and Numbers

Adopting the cleanest additive assumption: each subject’s reading value equals “true subject value + reader noise.” The inter-reader correlation ρ represents the proportion of total variance attributable to true values. The estimator averages all pairwise correlations—k readers yield k(k−1)/2 paired correlations. With fixed n=60 subjects, ρ is set at 0.4 (reflecting substantial reader disagreement) and 0.6 (indicating greater consistency); k sweeps through 2/3/4/6/8/12, each configuration running 5000 repeated simulations. Confidence interval half-widths are derived by multiplying the actual standard deviation of the estimator by 1.96 (the specific choice of interval method is compared elsewhere, see PMID 25417040; allocation between reader-specific and shared components is discussed in PMID 31898288 and variance decomposition via multilevel generalized theory, PMID 33818118).

III. Three Results

First, the benefit is almost entirely concentrated at the beginning. When ρ=0.4, the half-width drops from 0.219 for 2 people to 0.162 for 3 people, cutting the step by 26%; subsequent increments decrease monotonically at 15.8%, 12.6%, 8.4%, and 5.7%. From 2 to 12 people, the nominal pairing count rises from 1 to 66 (a 21-fold increase), yet the half-width only decreases by 53%—the pairing count inflation is severely overstated because all pairs share the same cohort of subjects, which are not 66 independent readings.

Second, the difference between 4 and 6 people ranges from 2.6 to 9 percentage points, depending on ρ. The half-width for 4 people is 0.136 and for 6 people is 0.119: at ρ=0.4, the 4-person configuration achieves only 87.4% of the precision of the 6-person setup; at ρ=0.6, it reaches 91.0%. Thus, the rule that “a small center with 4 people saves one-third of the cost” holds true in scenarios where readers are relatively consistent but must be discounted in scenarios with greater divergence.

Third, the floor is fixed by the subjects. Pushing k to 50 leaves the half-width stuck at 0.091—down from 0.119 for 6 people, leaving only 24% of the remaining space; adding more people can never move the floor established by the subjects’ sampling contribution. At k=6, 78% of the total reduction from 2 to infinite people has already been exhausted.

IV. Criteria for Decision and Design Guidelines

The two backlog-presumed criteria are both branch-valid, with the branch variable precisely being ρ: “Ceiling reached when reduction per round is insufficient by <5% after 6 people” holds for ρ=0.6 (8 people: 5.0%, 12 people: 4.2%), but fails for ρ=0.4 (8.4%, 5.7%; must wait until near 12 people to enter <6%); “4 people reaching 6-person precision at 90%” similarly meets the threshold only at ρ=0.6. Merged into one guideline: decay rate is inversely related to ρ—the more consistent the judges, the earlier the ceiling is reached, because little noise remains to be averaged; the greater the divergence among judges, the more valuable adding people becomes, though the floor also deepens. Thus, “how many judges to invite” has no universal answer; one must first report the measured range of ρ before determining the number of judges. Conversely, the marginal benefit curve of the number of judges in a study itself serves as a mirror reflecting the proportion of judge noise in that scenario. This aligns with Monte Carlo conclusions from confidence interval methods for generalization coefficients: interval precision is simultaneously constrained by both the number of judges and the number of subjects, meaning any unilateral increase hits a ceiling (PMID 357544612).

V. Pending Verification

This simulation rests on three idealized assumptions, each representing a gap for future empirical testing: first, the reader is assumed to carry equal load and evaluate the entire batch—rotations and absences would shift the location of the diminishing point; second, the budget is assumed separable into “person” and “case,” meaning it has not been drawn that the shape of the equal-precision line on the (n,k) plane depends on whether the budget is allocated to n or to k; third, noise is assumed homoscedastic—when readers differ in noise (some inherently jittery), it remains to be calculated whether culling by descending noise is superior to capping by headcount.

Evidence and Limitations

Verified: The interchangeability of reader count and subject count in the reliability precision formula, exhibiting diminishing marginal returns (PMID 16143966); and the dual constraints on interval precision (PMID 24338600, 35754612).

Partially Verified: The specific numerical values in this simulation (26% initial step gain, 87–91% four-person precision proportion, 0.09 floor) hold only under additive homoscedastic models and have not been validated against real-world reading data.

Pending Verification: Three gaps remain: unequal load, heteroscedastic readers, and the n-k equal precision line (see Section 5).

Related Reading

Evidence level: C
L1 network status: This topic has not yet entered the L1 corresponding network. The nearest neighbor is edge E-060 “五脏辨证标签信度” (Five-Zang differentiation label reliability), currently in a “Conflict (unresolved)” state—two pending decisions: sample unit mismatch (n≥50 refers to raters, while 120 refers to cases, making direct cross-validation impossible) and precision interval mismatch (the “consistency low” region falls precisely within the worst-precision zone of weighted coefficients). This article addresses only simulation-based inference of estimation precision and does not touch the label validity proposition of this edge.
References: PMID 16143966, PMID 24338600, PMID 25417040, PMID 33818118, PMID 31898288, PMID 35754612

Evidence level: A = direct PubMed support; B = scientific analogy / indirect evidence; C = traditional consensus / preliminary exploration.


Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *