那条「超过 10% 就判死」的法则,里面的 0.05 从来没人论证过:换个公认基准,边界宽出六到九倍

Written by

in

Language

Evidence level: C

判读一致性研究里流传一条极好用的法则:两位读者的主档占比一差,把这个差除以「1 减主档占比」,过了 10% 就宣布这篇研究的可补偿性已被边际差锁死。两个数就能算,所以到处在用。但 10% 是把「可接受带宽」设成 0.05 反解出来的,而 0.05 从没被单独论证过。代回公式后有三件麻烦事:隐含带宽不是常数而是随主档从 0.083 漂到 0.053;换成公认的半个标准差基准,边界宽出六到九倍;而这两条边界的差距在高一致区连一个误差地板都填不满——这个争议当前根本判不出来。

一、那个 0.05 是反推出来的

一致性天花板有条干净写法:主档差记作 δ,偶然一致率记作 p_e,天花板等于 1 减去 δ 除以(1 减 p_e)。要求它不低于目标 κ₀,等价于要求 δ 不超过可接受带宽 w 乘以(1 减 p_e),其中 w 就是 1 减 κ₀。

现行法则把判据写成 δ 除以(1 减主档占比)不超过 0.10。分母换了:标准式用(1 减 p_e),它用(1 减主档占比)。这不是等价变换,代价是隐含带宽变成主档的函数——主档 0.60 时隐含 w 是 0.0832、0.75 时 0.0666、0.95 时 0.0526。同一条「10%」,低一致区只要求天花板 0.917,高一致区要求 0.947,两头严格程度差六成,而这漂移只是分母替换的副产品。

二、换成半个标准差,边界宽出六到九倍

把 w 当自由参数反推边界。路径一(现行 0.05):主档 0.60 时边界 4.00 个百分点、0.75 时 2.50、0.95 时 0.50。

路径二(半个标准差):最小重要差异文献里最常被引的经验法则,有过专门复核(PMID 19807551)。换成切点偏移代进阈值模型,切点相差半个标准差时反推出的 w 是 0.380 到 0.466,对应边界 19.7 到 7.6 个百分点,是现行法则的 4.7 到 8.9 倍。

反查现行法则在说什么:反解 w 等于 0.05 只需切点偏移 0.046 到 0.062 个标准差。它其实等价于允许两位读者隐含切点差十分之一个标准差——比公认的半个标准差严五倍以上。 两个基准都不是随手挑的:一侧是反复复核过的分布法经验值,另一侧是判读文献惯用的 κ 目标。差距不是误差,是两套语言从未对齐。

三、这个边界比测量误差小一个数量级

前篇算过:判读偏倚的误差地板是 5.31 个百分点,且不随样本量稀释。以此为尺量 w 的争议区间(0.02 到 0.15):

主档占比 下端边界 上端边界 位移 位移占几个误差地板
0.60 0.96 7.24 6.28 1.18
0.75 0.75 5.65 4.90 0.92
0.85 0.51 3.84 3.33 0.63
0.90 0.36 2.71 2.35 0.44
0.95 0.19 1.43 1.24 0.23

判「位移显著」需接近两个地板。w 在 0.02 到 0.15 之间怎么取,边界都移不到可辨幅度。 要达可辨所需的带宽:0.60 时 0.219、0.90 时 0.572;0.95 时无论 w 取多大都到不了——在高一致区,「w 该取多少」在现有精度下连被提出来的资格都没有。

边界绝对大小更难看:主档 0.60 的边界 4.00 个百分点占误差地板的 75%,0.95 时 0.50 个百分点只占 9%。这条法则的判别式,比噪声小一个数量级。

四、完全对齐的系统,九成概率被判死

设两位读者其实完全对齐,观测的差纯由测量噪声产生,被判「已锁死」的概率是:主档 0.60 单侧 22.6%、0.75 时 31.9%、0.85 时 38.9%、0.90 时 42.5%、0.95 时 46.2%;双侧是 45.1%、63.8%、77.8%、85.1%、92.5%。一个没有任何系统性分歧的系统,在主档 0.95 时有九成以上概率被判「上限已锁死」。 这不是偏严,是判别式已埋进噪声,判出来的东西和抛硬币同源。

五、三条必须改的口径

第一,w 必须当显式报告项。现行 10% 隐含的 w 随主档从 0.083 漂到 0.053,报告里看不见。应报 w 的来源(κ 目标、半个标准差锚、专家共识),并按 w 给敏感性带而非单一边界。

第二,不许跨主档搬运边界。同一个 w 在 0.60 与 0.95 给的边界差四倍以上,现行法则靠隐含 w 漂移把这差掩盖了;改显式后掩盖失效,必须按主档分档给。

第三,当 w 的不确定性造成的位移小于两个误差地板时,应判「该研究不具备判定可补偿性的条件」,而不是给出判定。这条最要紧:不是改数字,是承认某些场合下判据没有分辨力。

六、L1 网络里这条边还没被裁决

研究内核里最接近的边是「五脏辨证作为分组标签」对「标签信度」的前提边(E-060 标签前提),挂冲突状态、至今未裁决。冲突双方并列:一侧是中医医师间诊断一致性在开放人群中被评为「低」、作者称其「令人担忧」(PMID 30383398);另一侧指出偶然校正类统计量常被误用、建议改用 Gwet AC2,且训练可改善一致性(PMID 31730403、31634000、19538100)。

该冲突谈的是「分组标签可不可靠」,不是「阈值该取多少」,但它点出本篇处境:标签带噪声时,一致性数值同时承载标签噪声与判读噪声,此时拿未经论证的带宽去划定「可不可补偿」,等于在两端都没校准的坐标上画线。

L1 网络状态:该主题尚未进入 L1 对应网络。CORRESPONDENCE_NETWORK.md 与 EDGE_TABLE_science-yixue.md 全表检索「判读 / 一致性 / 信度 / 可接受 / 带宽 / 阈值 / 标称 / 量表 / kappa / agreement / reliab / tolerance」均无对应边;唯一结构相关命中为 §3.8「E-060 标签前提」(层级「冲突(未裁决)」、状态 conflict,内容为「中医医师间诊断一致性低(PMID 30383398)对 偶然校正一致性统计常被误用(PMID 31730403 / 31634000 / 19538100)」),其语义是分组标签可不可靠,不是判据阈值该取多少、这个阈值有没有依据。已登记至 science-yixue-l1-gaps.md。

证据与局限

已验证:现行 10% 法则的分母是主档占比而非偶然一致率,故隐含带宽恒等于主档的函数,随主档从 0.0832 单调降到 0.0526(代数恒等,无近似);同一条法则在全主档区间对应两套天花板,差异达 0.03。

部分验证:与误差地板的比较沿用 5.31 个百分点这个地板,它建立在「两位读者漂移相互独立」这个尚未检验的前提上。半个标准差这条锚有专门复核(PMID 19807551),但它原本是「组间均值差」的尺度,搬到「两位读者切点差」上需正态潜变量与单切点假设,这一步是类比而非实测。

待验证:① 真实研究里 w 的实际取值从未被反推过;② w 是否随情境变(影像判读与功能评定可能不同)无实证;③ 全部计算基于二分类与正态潜变量,多档有序量表下 1 减 p_e 与主档的关系不是简单比值,边界公式须重推。

相关阅读

证据等级: C
参考文献: PMID 19807551, PMID 15733050, PMID 2348207, PMID 38319488, PMID 37933798, PMID 30383398, PMID 31730403

证据等级:A=有PubMed直接支撑,B=科学类比/间接证据,C=传统文化共识/初步探索


免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。

The \”Rule of 10% or Death\” Never Proves the 0.05 Threshold: Two Accepted Benchmarks Reveal an Eightfold Boundary Difference

Evidence level: C

In consistency assessment research, a highly practical rule of thumb circulates: if the difference between two readers’ primary agreement proportions (δ) divided by “1 minus primary agreement proportion” exceeds 10%, the study’s upper limit is deemed irreversibly locked by its own marginal difference, regardless of added samples. Since only two numbers are required to calculate this, it is widely used. However, the 10% threshold was derived by inverting an “acceptable bandwidth” of 0.05, yet 0.05 itself was never independently proven. This paper substitutes it back into the formula, revealing two complications: the bandwidth implicitly assumed by the current rule is not 0.05, but rather 0.05 divided by the primary agreement proportion, drifting from 0.091 when primary agreement is 0.091 to 0.053 when it is 0.95; and reversing the calculation using two other established benchmarks from the literature yields boundaries differing by eight times—far exceeding any measurement resolution.

I. A Drifting Bandwidth Buried in the Current Rule

The consistency ceiling has a clean formulation: let the difference in primary agreement proportions be δ, and the chance agreement rate be p_e; the ceiling equals 1 minus δ divided by (1 minus p_e). For this not to fall below the target κ, it is equivalent to requiring δ not to exceed the “acceptable bandwidth” w multiplied by (1 minus p_e).

The current rule replaces the denominator with (1 minus primary agreement proportion), writing the criterion as δ divided by (1 minus primary agreement proportion) not exceeding 0.10. Under binary classification, this expression can be precisely simplified, resulting in equivalence to δ not exceeding twice “0.05 multiplied by primary agreement proportion” times (1 minus primary agreement proportion). Comparing with the standard form, the implied bandwidth is w equal to 0.05 divided by primary agreement proportion.

This number is not constant: when primary agreement is 0.55, the implied value is 0.091; at 0.65 it is 0.077; at 0.75 it is 0.067; at 0.85 it is 0.059; and at 0.95 it is 0.053. The same “10%” actually requires a target consistency of only 0.909 in the low-agreement region but 0.947 in the high-agreement region—a difference of more than double in strictness, yet this drift was never proven.

II. Three Benchmarks Differ by Eight Times

Treating w as a free parameter, we reverse-calculate the boundary along three paths:

Path One (Current 0.05): At primary agreement 0.75, the boundary is 1.88 percentage points; at 0.90 it is 0.90; and at 0.95 it is 0.48.

Path Two (Half a Standard Deviation): The most frequently cited empirical rule in minimum important difference literature is “half a standard deviation,” which has its own verification (PMID 19807551). Substituting this as cutoff offset into the threshold model: when two readers’ cutoffs differ by 0.5 standard deviations, primary agreement 0.50 yields w of 0.397, 0.75 gives 0.403, and 0.90 gives 0.439. This is eight times the value of 0.05.

Path Three (True Meaning of Current Rule): Solving for what cutoff offset yields w equal to 0.05 gives an answer of 0.046 to 0.063 standard deviations. Thus, the current 10% rule is equivalent to “allowing an implicit cutoff difference of one-tenth of a standard deviation between two readers.” Meanwhile, a 0.10 standard deviation offset produces w of 0.080 to 0.096—exactly falling back near 0.10. Going in a circle, the current rule is self-consistent within its own units, yet it sets “clinically negligible” at 0.1 standard deviations, five times stricter than the widely accepted 0.5 standard deviations.

Neither benchmark was arbitrarily chosen: one side is the distribution-based empirical value repeatedly verified, while the other is the κ target conventionally used in assessment literature. The eightfold gap is not error; it stems from these two languages never aligning.

III. This Boundary Is Smaller Than Measurement Error by One to Two Orders of Magnitude

Previous calculations established: the error floor for assessment bias is 5.31 percentage points, and it does not dilute with sample size. Using this as a scale to measure the disputed range of w (0.02 to 0.15):

  • Primary agreement 0.60: the boundary shifts from 0.96 to 7.20 percentage points, a displacement of 6.24, equal to 1.18 error floors;
  • Primary agreement 0.75: displacement is 4.87, or 0.92 error floors;
  • Primary agreement 0.85: displacement is 3.32, or 0.62 error floors;
  • Primary agreement 0.90: displacement is 2.34, or 0.44 error floors;
  • Primary agreement 0.95: displacement is 1.24, or 0.23 error floors.

To judge “significant displacement,” one needs to approach two error floors. Consequently, across all primary agreement intervals, whether w is set at 0.02 or 0.15, the boundary never shifts by an amount discernible to measurement. Reversing the calculation to find the bandwidth change required for displacement reaching discernibility: 0.60 requires 0.217, 0.75 requires 0.278, 0.85 requires 0.408, 0.90 requires 0.578, and 0.95 requires 1.096. The last exceeds 1, yet bandwidth by definition cannot exceed 1—in the high-agreement region, “what w should be taken” lacks even the qualification of being posed under existing measurement precision.

The boundary itself is smaller: at primary agreement 0.60, the boundary is only 23% of the discernible threshold (10.41 percentage points), and at 0.95 it is merely 4.6%. The discriminant of this rule is a quantity smaller than noise by one to two orders of magnitude.

IV. A Fully Aligned System Has Over 90% Chance of Being Declared Fatal

Continuing the calculation yields the most unsettling number in this paper. Suppose two readers’ systems are actually perfectly aligned (true δ equals zero), with deviation arising purely from measurement noise. The probability of being judged “already locked” by this rule is: 32.6% for primary agreement 0.60, 36.2% at 0.75, 40.5% at 0.85, 43.3% at 0.90, and 46.4% at 0.95; the two-sided thresholds are 65.1%, 72.4%, 81.0%, 86.5%, and 92.9% respectively.

A reading system with no systematic divergence has over 90% probability of being declared “its upper limit is irreversibly locked by itself” when primary agreement is 0.95. This is not due to the rule being overly strict; rather, the discriminant has been embedded within noise, rendering its output equivalent to coin tossing.

V. Two Calibers That Must Be Revised

First, w must be explicitly reported, not implicitly inherited. The current 10% is the product of “0.05 divided by primary agreement proportion,” a number that drifts with primary agreement, becoming stricter as primary agreement rises, yet invisible in reports. The standard should be revised to: report the source of w (κ target, 0.5 standard deviation anchor, or expert consensus), and provide sensitivity bands based on w rather than a single boundary.

Second, boundaries must not be transported across primary agreement intervals. The same w yields boundaries differing by five times between 0.60 and 0.95; the current rule masks this gap via “implicit w drift.” Using explicit w renders this masking ineffective, so boundaries must be assigned by primary agreement tier.

Third, when boundary displacement caused by w uncertainty is less than two error floors, the study should be judged “lacking conditions for compensability,” not given a verdict. This is more critical than the previous two: it is not about changing numbers, but acknowledging that under certain circumstances this criterion lacks discernibility.

L1 Network Status: This topic has not yet entered the L1 corresponding network. Searching CORRESPONDENCE_NETWORK.md and EDGE_TABLE_science-yixue.md for terms like “assessment / consistency / reliability / acceptable / bandwidth / threshold / nominal / scale / kappa / agreement / reliab / tolerance” yields no corresponding edges; the only structurally related hit is §3.8 “E-060 Label Premise” (level “Conflict (Undecided)”, status conflict, content “Low diagnostic consistency among TCM physicians (PMID 30383398) leads to statistical misuse causing underestimation (PMID 31730403 / 19538100)”), whose semantics concern “whether grouping labels are reliable,” not “what threshold value to take and whether this threshold has basis.” It has been registered in science-yixue-l1-gaps.md.

Evidence and Limitations

Verified: The current 10% rule under binary classification is precisely equivalent to “δ not exceeding twice 0.05 multiplied by primary agreement proportion times (1 minus primary agreement proportion),” hence its implied bandwidth is constantly 0.05 divided by primary agreement proportion, monotonically decreasing from 0.091 to 0.053 as primary agreement rises (algebraic identity, no approximation). Reverting to explicit w yields a flip rate of 6.0%, with zero cases of over-stringency in six million-point sampling.

Partially Verified: Comparing boundaries with error floors used the 5.31 percentage point floor from the previous paper, which rests on the untested premise of “independent drift between two readers” (listed as a pending verification topic). The 0.5 standard deviation anchor is repeatedly verified in literature (PMID 19807551), but it originally scales “mean difference between groups”; applying it to “cutoff difference between two readers” requires normal latent variables and single-cutoff assumptions, making this step an analogy rather than direct measurement.

Pending Verification: ① In real studies, the actual value of w has never been reverse-calculated—whether published consistency studies’ difference distributions can infer a stable w remains unchecked; ② whether w varies by context (imaging assessment vs. functional evaluation may not share the same value) lacks any domain-specific evidence; ③ all calculations in this paper rely on binary classification and normal latent variables, while under multi-tier ordinal scales, the relationship between 1 minus p_e and primary agreement proportion is proven not to be a simple ratio by #1408, requiring the boundary formula to be re-derived.

Related Reading

Evidence Level: C
References: PMID 19807551, PMID 15733050, PMID 2348207, PMID 38319488, PMID 37933798

Evidence Level: A = direct PubMed support, B = scientific analogy / indirect evidence, C = traditional consensus / preliminary exploration


Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *