Category: 科学易学

  • 加人能提精度,但加不出真相:读者的作用是压噪声,压不掉材料本身在变

    Evidence level: C

    让更多人来读,能把「读的人自己抖」平均掉,却一分钱也平均不掉「被读的东西自己在变」——前者按人数的平方根收敛,后者是一个乘在真值上的常数。

    一、「多找几个人」这句话,其实混了两件不同的事

    一致性研究里最常听到的处方是「加人」。两位读者算不出一致性,就找四位;四位还不够,就找八位。理由听起来很硬:人多,噪声就被平均掉了。

    这个理由对一半。把这件事拆成两层看,加人的效果完全不一样:

    第一层是精度。 每位读者都带一份自己的抖动——今天手松、明天手紧、同一张片子看两遍给两分。这份抖动彼此独立,k 位读者一起读,它就被平均成原来的 1/√k。加人确实让估计更稳,这是真的。

    第二层是准头。 如果所有读者其实都在同一件「会变的东西」上做判断,那 k 位读者加起来仍然只有那一件东西。加人只会把它更精确地量出来,不会让它变成另一件东西。精度上去了,准头不动。

    一条老规矩早就说清楚:一致性系数的置信区间与样本量,必须按「评定者数」和「病例数」两个方向分别算(PMID 15733050)。可实践里几乎没人分开算,于是「加人」被当成包治百病的动词用。

    二、把两条路写成一个式子,就看出加人管用在哪一段

    设每位读者对每个病例给出的读数,由三块叠加而成:

    • 一块是所有人共同感受到的信号——那个我们真正想测的「共同模」,方差记作 S²;
    • 一块是读者各读各的抖动,方差 T(含读者自己的特异偏移与偶然误差);
    • 一块是病例本身在变——材料难度漂移、时点效应、样本异质性,方差记作 A。

    读者随机、病例随机、两者完全交叉时,任意两位读者读数之差(一切一致性系数赖以计算的原料)只含中间那一块:差的方差是 2T,A 被彻底抵消了。这是完全交叉设计最漂亮的地方——病例怎么变都无所谓,因为大家读的是同一批病例。

    再看「k 位读者取平均」这条路。平均值的方差是 S² + A + T/k。用平均值方差扣掉从分差里估出来的 T/k,剩下 S² + A,开根号就是 √(S² + A)。

    答案就在式子里:k 只出现在 T/k 那一项,S² + A 里没有任何 k。

    加多少人都只能把 T/k 压薄,压不掉 A。估出来的「共同模」天生是 √(S²+A),不是 √S²。

    仿真实测也一致:N = 200、真值 S = T = 1,只要 A 不为零,无论读者从 2 位加到 50 位,估计值都稳定偏高同一个百分比——A 占 S 的四成六时偏高 10%,占七成一时偏高 22%,正好等于 √(1+(A/S)²)−1。覆盖率在 A=0 时能从 63% 爬到 94%;A 一旦达到 S 的 0.71 倍,就一路从 14% 掉到 2.6%:加人越多,越精确地停在一个错的位置上。

    三、「加人」的门槛可以算死:n=200 时 k 的边际收益断崖

    A = 0 的干净情形下(N = 200,覆盖率 = 点估计落在真值 ±10% 内的比例):

    读者数 k 2 3 4 5 6 10 20 50
    覆盖率 63% 75% 80% 84% 85% 91% 93% 94%

    三件事:第三位读者最值钱(一步抬 12 个百分点);第五位之后基本到头(第 5→6 位只加 2 个百分点);40 位也换不到 95%——天花板不在人身上,在病例数。N = 200 时相对标准误的理论地板是 1/√(2(N−1)) ≈ 5%,实测 5.2%,两边对得上;N = 50 时覆盖率只有 67%,N = 800 才到 100%。「加人」有个隐形上限,跨过去之后要多的是病例,不是读者。

    按覆盖率达到八成算门槛:A = 0 时 k = 4;A 达到 S 的四分之一时 k ≈ 7;A 达到三分之一时 k ≈ 16;A 到 0.46 倍时连 N = 200 加 6 位读者都做不到(覆盖率 53%);A 到 0.71 倍时无论如何过不了 5%。

    也就是说:加人这条处方只在「病例几乎不变」时便宜好用;A 只要超过 S 的三分之一,它就从「省钱补救」变成「花大钱买精度、还买错了方向」。

    四、一条更扎心的推论:换设计也换不来多一个方程

    有人会说,那就别让所有人读同一批病例,改成两半区组、中间留一批重叠病例当桥——这是不完全设计的标准做法,看起来像给「共同模」多添了一条独立方程。

    仿真实测(N = 200、8 位读者分两组、重叠 20 例):A = 0 时覆盖率 71%,A = 0.71 时 14%,偏差一分没减。

    原因是式子没变:分组后每组各算一份平均值方差、再用组内分差扣掉各自的 T/g,剩下的仍是 S² + A。桥接只让「两侧可比」成立,不让 A 从式子里消失。 不完全设计的真本事在别处:它能把病例难度当成一项可估参数估出来(PMID 37810544 的模拟显示,稀疏设计下多面 Rasch 与概化理论是否还能读准,本身就是要检验的问题,不是自动成立)。而「估出来」这一步,恰恰需要有人先给病例难度一个独立于读者读数的量尺,不能靠读者自己再读一遍。

    五、这个式子同时把「扣掉」的代价算清楚了

    既然加人不行,剩下的路就是:先量出 A 有多大,再从平均值方差里扣掉 A²。

    代价可以精确算。真 A/S = 0.6、读者 50 位、N = 400 时,标定完全准则覆盖率 95%;高估 10% 掉到 87%,高估 20% 掉到 60%,高估 50% 只剩 1%;低估 20% 掉到 78%。这里的不对称值得记住:高估 A 比低估 A 危险得多,因为高估会把 S² 扣成负的、压低估计值,而压低看着不像错误,更像一个「更保守」的结果。

    这也解释了为什么「共同模」在实践里总是偏乐观:不扣减它一定偏高,做了扣减又极易扣过头。

    证据与局限

    已验证:这是代数结论,不是经验发现。在「读者随机、病例随机、完全交叉」的模型下,任意两位读者之差的方差恒等于 2T,病例项 A 必然抵消;k 位读者平均值的方差恒为 S² + A + T/k。由此「加人只压 T/k、不压 A」是恒等式级别的结果,不依赖任何模拟。上述覆盖率数字来自 4000–6000 次重抽的仿真(S = T = 1,N = 200),与解析式偏差一致到小数点后一位。可靠性与一致性研究报告规范历来要求同时报样本量与区间估计(PMID 21130355),「两个方向分别算」这件事在方法学上早有出处(PMID 15733050)。

    部分验证:病例/时点项的方差占比在真实数据里到底有多大,现有文献给出的数差得很远——某项持续两年的课堂观察研究做方差分解后认为趋势项只占中等份额(PMID 29795823),而概化研究里读者×时点交互既可能接近零、也可能占到可观的份额(PMID 32581621)。这说明 A 不是可以一律当零处理的量,但也说明「A 通常多大」目前没有可用的先验。稀疏设计下病例难度参数能否读准,已有模拟指出需要专门检验(PMID 37810544);多面 Rasch 里严厉度随时间的漂移必须显式建模、独立性假设下估计效率很低,需引入马尔可夫先验才稳(PMID 36284065),这也从另一个角度说明「时间维」不能被当成噪声吸收掉。

    待验证:第一,本文全部计算建立在「A 对所有读者是同一份」的假定上。若不同读者面对的病例漂移不同(例如分批读、批次间材料换过),则 A 不再是一个标量,而是一个与读者下标相关的结构,此时式子要重写,结论可能从「常数偏高」变成「读者间不可比」。第二,本文把「共同模」当成一个可加分量。真实判读里共同模更可能作用在判断门槛上而非读数上(读者一起变松或紧),此时它进入的是分差而非均值,抵消关系整个反过来——这条已另立课题。第三,A 的独立量尺从哪来(客观难度代理、重复测量、外部分级),本文只指出必须有,没有给出可行性边界。

    L1 网络状态:L1 对应网络中未检索到该主题对应的边。原文摘录(E-060 标签前提行):「KB-292(conflict):中医医师间诊断一致性在开放人群中「低(low)」,作者称为「令人担忧」(PMID 30383398)」;同一边另载「KB-317(conflict):闸门规格撞上方法学反例,两处对不上」。全网络 1956 行内「kappa」0 次、「可辨识」0 次、「共同模」0 次,与本文相关的仅「评定者」4 处、「信度」12 处,且全数落在 E-060 的标签信度前提上,状态为 conflict(未裁决)**,无任何 A 级证据支撑本文的代数结构。故本文证据等级封顶 C,缺口已登记至 l1-gaps 第 62 条。

    相关阅读

    证据等级: C
    参考文献: PMID 15733050, PMID 21130355, PMID 29795823, PMID 32581621, PMID 37810544, PMID 36284065


    免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。

    Adding people improves precision but not truth: readers suppress noise, yet cannot suppress the material itself changing.

    Evidence level: C

    Increasing the number of readers averages out the “jitter” of the readers themselves (e.g., hand tremor, fatigue), which converges at the square root of the number of readers. However, it cannot average out the fact that the material being read is changing, which acts as a constant multiplier on the true value.

    I. “Finding more people” mixes two distinct issues

    The most common prescription in consistency studies is “add more raters.” If two readers cannot calculate agreement, find four; if four are not enough, find eight. The rationale sounds solid: more people average out the noise.

    This rationale is only half true. Breaking this down into two layers reveals that adding raters affects the two components differently:

    The first layer is precision. Each reader brings their own jitter—loose hands today, tight hands tomorrow, or reading the same image twice and giving it two different scores. These jitters are independent; with $k$ readers, the jitter averages out to $1/\sqrt{k}$. Adding raters does indeed make the estimate more stable; this is true.

    The second layer is accuracy. If all readers are judging the same drifting thing, then $k$ readers combined still only measure that one thing. Adding raters merely measures it more precisely, it does not turn it into a different thing. Precision goes up, accuracy stays the same.

    An old rule already clarifies this: the confidence interval for the consistency coefficient must be calculated separately along two directions—”number of raters” and “number of cases” (PMID 15733050). Yet in practice, almost no one calculates them separately, so “adding raters” is used as a catch-all verb.

    II. Writing the two paths into one equation reveals where adding raters helps

    Let each reader’s reading for each case be composed of three stacked components:

    • One component is the signal perceived by everyone—the “common mode” we truly want to measure, with variance denoted $S^2$;
    • One component is each reader’s individual jitter, with variance $T$ (including each reader’s specific bias and random error);
    • One component is the case itself drifting—material difficulty drift, timing effects, sample heterogeneity, with variance denoted $A$.

    When readers are randomized, cases are randomized, and they cross completely, the difference between any two readers’ readings (the raw material for all consistency coefficients) contains only the middle component: the variance of the difference is $2T$, $A$ is completely canceled out. This is the most elegant feature of the fully crossed design—the cases can drift however they like, because everyone is reading the same set of cases.

    Now consider the path of “averaging $k$ readers.” The variance of the average is $S^2 + A + T/k$. Subtracting the $T/k$ estimated from the pairwise differences leaves $S^2 + A$, and taking the square root gives $\sqrt{S^2 + A}$.

    The answer lies in the equation: $k$ appears only in the $T/k$ term; there is no $k$ in $S^2 + A$.

    Adding more raters only thins out the $T/k$ term; it cannot reduce $A$. The estimated “common mode” is inherently $\sqrt{S^2 + A}$, not $\sqrt{S^2}$.

    Simulation results confirm this: with $N = 200$, true values $S = T = 1$, as long as $A \neq 0$, regardless of whether readers go from 2 to 50, the estimate remains consistently biased upward by the same percentage—10% when $A$ is 46% of $S$, 22% when $A$ is 71% of $S, exactly matching $\sqrt{1 + (A/S)^2} – 1$. Coverage rises from 63% when $A = 0$ to 94%; once $A$ reaches $0.71 \times S$, coverage plummets from 14% to 2.6%: the more raters you add, the more precisely you stop at the wrong place.

    III. The threshold for “adding raters” can be calculated precisely: marginal benefit cliffs at $n = 200$

    In the clean case where $A = 0$ ($N = 200$, coverage = proportion of point estimates falling within $\pm 10\%$ of the true value):

    Number of raters $k$ 2 3 4 5 6 10 20 50
    Coverage 63% 75% 80% 84% 85% 91% 93% 94%

    Three takeaways: the third rater is most valuable (jumping 12 percentage points); after the fifth rater, it’s basically done (5→6 adds only 2 percentage points); even 40 raters cannot reach 95%—the ceiling is not in the number of people, but in the number of cases. With $N = 200$, the theoretical floor for relative standard error is $1/\sqrt{2(N-1)} \approx 5\%$, observed at 5.2%, which matches; with $N = 50$, coverage is only 67%, and $N = 800$ is needed for 100%. “Adding raters” has an invisible ceiling; beyond it, you need more cases, not more raters.

    Setting the threshold at 80% coverage: when $A = 0$, $k = 4$; when $A$ reaches one-quarter of $S$, $k \approx 7$; when $A$ reaches one-third of $S$, $k \approx 16$; when $A$ reaches $0.46 \times S$, even adding 6 raters to $N = 200$ fails (coverage 53%); when $A$ reaches $0.71 \times S$, you cannot cross 5% under any circumstances.

    In other words: the prescription of “adding raters” is only cheap and effective when cases hardly change; once $A$ exceeds one-third of $S$, it shifts from “saving money to fix accuracy” to “spending big money on precision but getting the wrong direction.”

    IV. A more sobering inference: changing the design does not add another equation

    One might argue, then do not have everyone read the same cases; split them into two blocks with an overlapping set of cases as a bridge—this is the standard approach for incomplete designs, appearing to add an independent equation for the “common mode.”

    Simulation results ($N = 200$, 8 raters split into two groups, overlap of 20 cases): when $A = 0$, coverage is 71%; when $A = 0.71$, it is 14%, with zero reduction in bias.

    The reason is that the equation does not change: after grouping, each group calculates its own average variance, then subtracts $T/g$ from the within-group differences; what remains is still $S^2 + A$. The bridge only ensures comparability between the two sides; it does not make $A$ disappear from the equation. The real power of incomplete designs lies elsewhere: it allows case difficulty to be estimated as an additional parameter (PMID 37810544’s simulation shows that under sparse designs, whether multi-facet Rasch and Generalizability Theory can still estimate it accurately is itself a question to be tested, not automatically true). And this “estimation” step precisely requires someone to provide a scale independent of reader readings for case difficulty, not relying on readers reading it again.

    V. This equation also calculates the cost of “subtracting” $A$

    Since adding raters is not viable, the remaining path is: first measure how large $A$ is, then subtract $A^2$ from the average variance.

    The cost can be precisely calculated. With true $A/S = 0.6$, 50 raters, and $N = 400$, the calibration criterion coverage is 95%; overestimating $A$ by 10% drops it to 87%, overestimating by 20% drops it to 60%, overestimating by 50% leaves only 1%; underestimating by 20% drops it to 78%. This asymmetry is worth remembering: overestimating $A$ is far more dangerous than underestimating it, because overestimation subtracts $S^2$ into negative values, depressing the estimate; this depression looks less like an error and more like a “more conservative” result.

    This also explains why the “common mode” is always optimistically biased in practice: not subtracting it makes it high, while subtracting it risks subtracting too much.

    Evidence and Limitations

    Verified: This is an algebraic conclusion, not an empirical discovery. Under the model where “readers are randomized, cases are randomized, and they cross completely,” the variance of any two readers’ difference is always $2T$, so the case term $A$ must cancel out; the variance of the average of $k$ readers is always $S^2 + A + T/k$. Thus “adding raters only reduces $T/k$, not $A$” is an identity-level result, independent of any simulation. The coverage numbers come from simulations with 4000–6000 resamples ($S = T = 1$, $N = 200$), matching the analytical formula to one decimal place. Reliability and consistency reporting standards have long required reporting both sample size and interval estimates (PMID 21130355), and “calculating separately along two directions” has methodological precedent (PMID 15733050).

    Partially verified: How large the variance of the case/timing term actually is in real data varies widely across literature. One two-year classroom observation study decomposed variance and found the trend term accounts for only a medium share (PMID 29795823), whereas in generalizability studies, the reader×timing interaction may be near zero or occupy a substantial share (PMID 32581621). This shows $A$ is not a quantity that can always be treated as zero, but also indicates there is currently no prior available for “how large $A$ usually is.” Whether case difficulty parameters can be estimated accurately under sparse designs has been shown via simulation to require specific testing (PMID 37810544); in multi-facet Rasch, drift in severity over time must be explicitly modeled, and under independence assumptions estimation efficiency is low, requiring Markov priors to stabilize (PMID 36284065), which also shows from another angle that the “time dimension” cannot be absorbed as noise.

    To be verified: First, all calculations in this paper assume $A$ is the same for all readers. If different readers face different case drifts (e.g., batch reading with material changes between batches), then $A$ is no longer a scalar but a structure dependent on reader index, requiring rewriting of the equation; conclusions may shift from “constant upward bias” to “incomparable readers.” Second, this paper treats the “common mode” as an additive component. In real judgment, the common mode more likely acts on the judgment threshold rather than the reading itself (readers collectively loosen or tighten), so it enters the difference rather than the mean, reversing the cancellation relationship—this is a separate research topic. Third, where does the independent scale for $A$ come from (objective difficulty proxy, repeated measurement, external grading)? This paper only notes it must exist, without specifying feasibility boundaries.

    L1 Network Status: L1 corresponds to edges not retrieved in the network for this topic. Original excerpt (E-060 label premise line): “KB-292 (conflict): Diagnostic consistency among TCM physicians is “low (low)” in open populations; authors call it “concerning” (PMID 30383398); the same edge also carries “KB-317 (conflict): Gate specifications clash with methodological counterexamples; two sides do not match.” Across the entire network of 1956 lines, “kappa” appears 0 times, “identifiable” 0 times, “common mode” 0 times; only “raters” (4 occurrences) and “reliability” (12 occurrences) are related to this paper, all falling under E-060’s label reliability premise, status conflict (unresolved), with no A-level evidence supporting the algebraic structure of this paper. Hence, the evidence level for this paper is capped at C, and the gap has been registered as item 62 in l1-gaps.

    Related Reading

    Evidence Level: C
    References: PMID 15733050, PMID 21130355, PMID 29795823, PMID 32581621, PMID 37810544, PMID 36284065


    Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.

  • 门限 0.95 这个数字是怎么定下来的:验收、证明与功效-样本量三维表

    Evidence level: C

    0.95 这个门限是在「点估计时代」定下来的,从未按「区间估计 + 指定功效」重算过——把「验收」和「证明」分开之后,门限-功效-样本量是一张三维表,每一格说的都是不同的话。

    正文

    任何一套「人工判读要过验收」的规则里,都藏着一个数字。在脉诊、舌诊这类一致性验收场景里,这个数字通常是 kappa ≥ 0.95。但很少有人追问:0.95 是谁、在什么前提下定的?

    先分清两件事:验收和证明。「证明」是研究者想说明真值有多高,报个点估计加置信区间就够了;「验收」是门禁——m̂ 高于门限才放行,这时你关心的不是真值是多少,而是这套门禁的误放率和漏放率。混淆两者的代价是:一个在「描述水平」语境下合理的数字,搬到「放行门槛」语境下可能完全不设防。

    三维表才是完整的账。门限 T、样本量 n、功效这三个量互相咬死:给定 n 和 T,真值恰好等于 T 时通过率恒等于显著性水平 α(3%–5%)——这意味着门禁在边界处形同虚设;要达到 80% 功效让真值过线,真值须到 0.996 以上,现实中不存在。反过来,若坚持 n=480 时让真值 0.95 有八成把握通过,门限须从 0.950 下调到约 0.921。这三个数构成一张 n × T × 功效的三维表,每一格都在回答不同的问题:这格的误放率是多少、漏放率是多少、要求的真值下限是多少、最危险的「真值比门限低 2 个百分点」档位穿透率是多少。

    文献里的先例都是二维的,而且各说各话。早期的 kappa 样本量表在点估计框架下工作,先给出门限再配一个够用的 n;后来的路线改用置信区间精度定标——预先指定 κ 的置信区间上下限宽度来反推样本量,这正是现代精确样本量算法的思路;还有工作专门研究「训练到某个 kappa 准则」需要多少测试病例,发现准则定在 0.60 下界、要 95% 把握时,每位评定者至少要测 5 个病例以上,且病例在难度档位上的分布会显著改变所需样本量。这些文献没有一篇算过「门限 × 功效 × 样本量」的全格表——它们各自固定了两个轴、只扫一个轴。

    为什么边界处通过率恒为 α?这反直觉但必然。把「验收」写成分位数的语言:精确下界方法给的是 m̂ 的置信区间下界,放行条件 = 下界 ≥ T。当真实一致性恰好等于 T 时,区间恰好有一侧概率质量骑在 T 上——在 α 的水平下,「下界超过真值」的概率由构造定义就是 α。也就是说:门禁只对「真值明显高于门限」的样本有拦截力,对「真值恰好在门限上」的样本近乎随机放行。这解释了为什么把惯例代入表后会出现「宣称而非验收」的尴尬:点估计时代只关心 m̂ 高不高,从不问「如果真值比门限低 2 个百分点,这套规则还拦得住吗」。穿底率这一格要的就是这个数——让「差一点点也放行」的漏洞可量化。

    0.95 的门限演化也有考古价值。经典分期常把 0.81 以上称作 almost perfect,0.95 并不来自这张分期表;它更像是「再高一点更保险」的直觉放大,加上样本量规范缺位,就从习惯变成了行规。文献层面从未有人论证过「为什么是这个数、配多大 n、保证什么功效」——本主题的三维表正是替这个缺位补账。

    把三条现行惯例代入这套表打分:「n=120 例 + 点估计 ≥0.95」「n=240 + 精确下界」「n=480 + 精确下界」。判据有三条:若任一惯例在「真值比门限低 2pp」处的通过率超过 20%,它就不构成验收、只构成宣称,须废止;若「n=240 + 精确下界」这一格在 80% 功效处所需真值已超 0.99,则本领域根本不该用 0.95 门限配精确区间,须改为「门限 0.90 + 精确区间 + n≥480」(宣称达标用)或「门限 0.95 + 点估计 + 强制报 CI」(描述水平用)两条路线择一;若功效 0.50 与 0.80 两档在 n≥480 处所需真值相差不足 1pp,功效这一轴可压掉,三维表退化成二维。

    这不是本领域特有的麻烦:序贯监测的模拟研究已明确报告,计划块数少时 I 类错误可显著膨胀、块数多或加校正后才能回到名义值——门限与设计参数的耦合是普遍现象。0.95 没什么神圣的,它只是从点估计时代 inherited 下来的一个习惯数;要么重算它,要么写清楚它在哪个功效下守护什么。

    证据与局限

    已验证:kappa 置信区间宽度随样本量变化可精确计算(PMID 24338600、22560852、1410963);「训练到 kappa 准则」的样本量-病例数关系已有模拟结果(PMID 20473138)。
    部分验证:门限边界处通过率恒为 α、功效-真值需求曲线的解析式成立,但本领域 n ∈ {60…1920} 全格表尚未有人算过。
    待验证:三条现行惯例的穿透率打分;「宣称/描述」双路线的适用边界;功效轴是否可压掉。

    L1 网络状态:该主题尚未进入 L1 对应网络(方法学/统计定标类主题,非易学-科学对应边)。L1 中最接近的边为 E-060 标签前提(conflict):「样本量单位错配(n≥50 指评定者 vs 120 指病例)与精度区间错配(『一致性 low』正落在加权系数精度最差区间)」——与本篇「门限数字从未按功效重算」同属验收规格错配家族,但不是同一条边,已登记缺口表。

    相关阅读

    证据等级: C
    参考文献: PMID 15733050, PMID 24338600, PMID 22560852, PMID 20473138, PMID 1410963

    证据等级:A=有PubMed直接支撑,B=科学类比/间接证据,C=传统文化共识/初步探索


    免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。

    Threshold 0.95: Acceptance, Proof, and Efficacy – A Three-Dimensional Table of Sample Size

    Evidence level: C

    The threshold of 0.95 was established during the era of point estimation and has never been recalculated using interval estimation plus specified power. Once acceptance and proof are separated, the threshold-power-sample size relationship becomes a three-dimensional table, where each cell conveys different information.

    Body

    Any set of rules requiring “artificial judgment to pass acceptance” contains a numerical threshold. In consistency-based acceptance scenarios such as pulse diagnosis or tongue diagnosis, this number is typically kappa ≥ 0.95. Yet few ask: who set 0.95, and under what conditions?

    First, distinguish two concepts: acceptance and proof. “Proof” refers to the researcher’s attempt to demonstrate how high the true value is; reporting a point estimate with a confidence interval suffices. “Acceptance” acts as a gate—m̂ must exceed the threshold to be allowed through. Here, concern is not with the true value itself, but with the false acceptance rate and false rejection rate of this gate. Confusing the two carries a cost: a number reasonable in a “descriptive level” context may become completely unguarded when shifted to a “release threshold” context.

    The three-dimensional table constitutes the complete account. Threshold T, sample size n, and power are interlocked: given n and T, if the true value exactly equals T, the pass rate is always equal to the significance level α (3%–5%)—meaning the gate is virtually non-existent at the boundary. To achieve 80% power for the true value to pass, the true value must exceed 0.996, which does not exist in reality. Conversely, if one insists on n=480 to give the true value 0.95 an 80% chance of passing, the threshold must be lowered from 0.950 to approximately 0.921. These three quantities form an n × T × power table, where each cell answers different questions: what is the false acceptance rate? What is the false rejection rate? What is the required lower bound of the true value? What is the penetration rate for the most dangerous “true value 2 percentage points below the threshold”?

    Precedents in literature are all two-dimensional, and each speaks its own language. Early kappa sample size scales operated under the point estimation framework, first setting the threshold then matching it with a sufficient n. Later approaches switched to calibrating via confidence interval precision—pre-specifying the width of κ’s confidence interval limits to back-calculate sample size, which is precisely the logic of modern exact sample size algorithms. Other work specifically studied how many test cases are needed to “train to a kappa criterion,” finding that with a lower bound criterion of 0.60 and 95% confidence, each rater must assess at least five cases, and the distribution of cases across difficulty levels significantly alters the required sample size. None of these papers calculated the full table of “threshold × power × sample size”—they each fixed two axes and scanned only one.

    Why is the pass rate at the boundary always α? This is counterintuitive but inevitable. Writing “acceptance” in terms of quantiles: exact lower bound methods provide the lower confidence limit for m̂, and the release condition is lower bound ≥ T. When the true consistency exactly equals T, the interval has precisely one side’s probability mass riding on T—at level α, the probability that “the lower bound exceeds the true value” is defined by construction as α. In other words: the gate only intercepts samples where the true value is clearly above the threshold; for samples where the true value lies exactly on the threshold, it lets them through almost randomly. This explains why substituting conventions into the table leads to the awkwardness of “claiming rather than accepting”: the point estimation era cared only whether m̂ was high, never asking “if the true value is 2 percentage points below the threshold, can this rule still catch it?” The “penetration rate” cell specifically requires this number—to quantify the loophole where “almost enough also passes.”

    The evolution of the 0.95 threshold also has archaeological value. Classic classifications often label anything above 0.81 as “almost perfect,” yet 0.95 does not originate from this classification table; it seems more like an intuitive amplification of “a bit higher is safer,” compounded by the lack of sample size standards, turning habit into convention. No literature has ever argued “why this number, matched with what n, may support what power”—this subject’s three-dimensional table precisely fills that gap.

    Substituting the three current conventions into this table for scoring: “n=120 cases + point estimate ≥0.95”, “n=240 + exact lower bound”, “n=480 + exact lower bound”. Three criteria apply: if any convention’s pass rate at “true value 2pp below threshold” exceeds 20%, it does not constitute acceptance but merely claims, and must be abolished; if the “n=240 + exact lower bound” cell requires a true value exceeding 0.99 at 80% power, this field should not use the 0.95 threshold with exact intervals at all, but instead choose between “threshold 0.90 + exact interval + n≥480” (for claims meeting standards) or “threshold 0.95 + point estimate + mandatory CI reporting” (for descriptive level use); if the difference in required true values between power levels 0.50 and 0.80 at n≥480 is less than 1pp, the power axis can be collapsed, reducing the three-dimensional table to two dimensions.

    This is not a trouble unique to this field: sequential monitoring simulation studies have clearly reported that with few planned blocks, Type I error can inflate significantly, and only with many blocks or correction does it return to the nominal value—the coupling of threshold and design parameters is universal. 0.95 is not sacred; it is merely a habitual number inherited from the point estimation era: either recalculate it, or clearly state under what power level it guards what.

    Evidence and Limitations

    Verified: kappa confidence interval width varies with sample size and can be precisely calculated (PMID 24338600, PMID 22560852, PMID 1410963); the sample size-case number relationship for “training to kappa criteria” has simulation results (PMID 20473138).
    Partially verified: the pass rate at threshold boundaries is always α, and the analytical expressions for power-true value demand curves hold, but no one has yet calculated the full table for n ∈ {60…1920} in this field.
    Pending verification: penetration rate scoring for the three current conventions; the applicability boundaries of the “claim/describe” dual routes; whether the power axis can be collapsed.

    L1 Network Status: This topic has not yet entered the L1 corresponding network (methodological/statistical calibration topics, not Yijing-science correspondence edges). The closest edge in L1 is the E-060 label premise (conflict): “sample unit mismatch (n≥50 refers to raters vs 120 refers to cases) and precision interval mismatch (‘consistency low’ falls precisely in the worst precision zone of weighted coefficient intervals)”—this belongs to the same family of acceptance specification mismatches as “threshold numbers never recalculated based on power,” but it is not the same edge; the gap has been registered in the deficit table.

    Related Reading

    Evidence Level: C
    References: PMID 15733050, PMID 24338600, PMID 22560852, PMID 20473138, PMID 1410963

    Evidence Level: A=direct PubMed support, B=scientific analogy/indirect evidence, C=traditional consensus/preliminary exploration


    Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.

  • AC1 减 κ 那个差根本不携带边际差的信息:它对边际差的偏导恒为零,而归零点压在表格边线上

    Evidence level: C

    两位读者分档边际差多少,不可能从 Gwet 的 AC1 与 Cohen 的 κ 之差里反解出来——因为 AC1 的期望一致率只含两边际的平均、不含它们的差,所以那个”差”全程只有 κ 一侧在动。

    上一轮留下一条诱人的路:两位读者各自的分档边际差了多少,可以从 Gwet 的 AC1 与 Cohen 的 κ 之差里反解出来。理由听上去很硬——这两个系数对边际差的敏感程度不同,κ 的期望一致率里含两个边际的乘积,差多少会直接改它;AC1 的期望一致率只用两者的平均。一相减,剩下的就该是边际差。上一轮还算出这个差在均值接近 0.43、边际差接近 19.8pp 时归零,据此判定指纹在最需要它的地方失明。方向对,理由不是它给的那个。

    先把 AC1 拆开。分子是观察一致率,是数据;分母里的期望一致率等于两倍均值乘一减均值——均值只管和,不管差。这意味着两位读者一个偏阳性、一个偏阴性,只要平均不变,AC1 一个数都不动。数值上钉死:均值 0.43、观察一致率 0.80 固定,边际差取 1pp、5pp、10pp、15pp 四档,AC1 全部等于 0.6076892899,小数点后十位完全一致;同一组四档下 κ 从 0.592045 走到 0.601157。对边际差求偏导,AC1 与 Scott 的 π 都是精确的零,κ 在 0.025 到 0.140 之间。

    这件事的性质得说清楚:不是差太小读不出来,是 AC1 这个量在定义上根本不含边际差。所谓”两系数之差携带边际差信息”,其实是 κ 那一半在单向运动,AC1 只是被减数位置上的一个常数。上一轮看到的”归零”,本质是一条只含 κ 的曲线穿过一条水平线。

    接着是更硬的一击。原题把一致率 0.85 上下、边际差 14 到 20pp 当成”判读研究最常见的量级”,这个组合在二乘二表上不存在。四格概率非负给出一条精确约束:左下格等于一减观察一致率再减边际差的一半,必须非负,于是边际差不超过一减观察一致率,与均值无关。观察一致率 0.85 时边际差最大 15pp,0.90 时只有 10pp。二十万张随机表逐张核对,违反零张。换言之”高一致率”和”大边际差”是两个互相排斥的处境:真的差到 20pp,一致率就不可能高过 0.80。

    归零点于是不再神秘。AC1 与 κ 归零的条件是边际差等于根号二乘二的均值减一的绝对值,均值 0.43 时正是 19.80pp,而它落在可行域内的前提是观察一致率不超过 0.802——此时左下格恰好等于零。所以那条线不是藏在参数空间深处的陷阱,它压在单纯形的一条边上。一致率一旦高过 0.802,这条线就出了可行域,差恒为正、永不归零。上一轮把”归零”当成普遍威胁,实际只在表格退化到边界时才发生。

    真正的救法不在差值,在两个水平值本身。边际均值只由 AC1 与观察一致率决定,解一元二次即得,闭式解是均值等于一减根号下一减二倍期望一致率再除以二;拿到均值后,κ 的期望一致率反解出边际差,闭式解是根号下二倍的中括号,中括号里是三个含均值的项减去观察一致率与 κ 决定的项。在 1188 组随机参数上回代,最大误差 8.3×10⁻¹⁴——这不是近似,是恒等。

    这条路的代价是精度,不是可解性。以均值 0.43、一致率 0.80、真边际差 6pp 做参数 bootstrap:样本量 60 时边际差的抽样标准差 4.68pp,200 时 3.01pp,400 时 2.24pp,800 时 1.56pp。要让 95% 区间半宽压到 3pp,大约需要八百例。作为对照,只用差值反解时同一批样本有 42% 到 61% 根本无解,且标准差始终卡在 4.1 到 4.8pp 不随样本量下降——因为差值本身丢掉了大部分信息。

    这条结论不只是二分类的性质。多类别下 AC1 的期望一致率等于各档边际占比乘一减占比之和再除以档数减一,仍然只含边际、不含任何跨档配对结构;而加权 κ 的期望一致率含跨档权重与边际乘积之积,配对结构才从那里进来。所以”AC1 对边际差零响应”在多档判读里同样成立,且越是把判读压成二分类、越是把人往高一致率推,这条零响应就越彻底。

    实践含义很直接:AC1 与 κ 并列报告是可以的,但两者之差不能当边际差的度量,只能当”κ 被边际结构惩罚了多少”的粗略指示。要边际差本身,报两档边际占比即可,成本为零。PMID 23627889 在九位精神科医师、六十七例人格障碍样本上报告 AC1 在所有条目上高于 κ(0.752 到 1.000 对 0 到 1.00),并把差异归因于患病率敏感性,正是这条零响应的临床痕迹;PMID 8501467 早年就要求偏差与患病率分报才能解读 κ;PMID 24338600 给出的按目标精度反解样本量,可直接用于上文那条”八百例换 3pp”的规划;PMID 36123642 抽样八十二项一致性研究,只有三成给了样本量依据,说明这条精度账在实务里基本没被算过。

    证据与局限

    已验证(数学恒等,非经验结论):AC1 的期望一致率等于两倍均值乘一减均值,只含边际之和、不含之差;对边际差求偏导精确为零,与均值、一致率取值无关。固定均值 0.43、观察一致率 0.80 时,边际差从 1pp 到 15pp 四档,AC1 恒为 0.6076892899。归零条件为边际差等于根号二乘二的均值减一的绝对值,其可达前提是观察一致率不超过 1 减该值。可行域约束”边际差不超过 1 减观察一致率”由四格概率非负直接推出,二十万张随机表零违反。两个水平值联立可闭式反解均值与边际差,1188 组随机参数回代最大误差 8.3×10⁻¹⁴。

    部分验证:反解精度由参数 bootstrap 给出量级——真边际差 6pp 时,样本量 60/200/400/800 对应抽样标准差 4.68/3.01/2.24/1.56pp;该量级依赖”观察一致率已知且无系统误差”这一理想条件。作为对照,仅用差值反解在同批样本有 42% 到 61% 无解且标准差卡在 4.1 到 4.8pp 不随样本量下降。

    待验证:真实判读研究的(均值,边际差,观察一致率)三元组分布尚未实测,本稿所有量级都建在参数网格上而非文献回填;”AC1 对边际差零响应”在多档加权场景下虽已由公式给出,但缺实测数据集核验档数增加时零响应是否仍精确;本文未讨论三人及以上评定者情形下 AC1 的期望一致率结构是否仍只含边际。

    证据等级: C
    参考文献: PMID 23627889, PMID 8501467, PMID 24338600, PMID 36123642, PMID 25090041, PMID 33853549

    相关阅读

    L1 网络状态:EDGE_TABLE_science-yixue.md 概念最高止于生理层(频率 407 篇/谐波 147 篇/脉象 73 篇),无统计一致性系数层条目;CORRESPONDENCE_NETWORK.md 中「AC1」命中 1 处、「Gwet」3 处、「Brennan」1 处,全部落在 E-060 标签前提(五脏辨证作分组标签|前提|标签信度,状态 conflict,未裁决);「可辨识」「识别性」「prevalence」「零分布」「系数对」命中均为 0。该主题尚未进入 L1 对应网络。


    免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。

    AC1 minus κ carries no information about the marginal difference: its partial derivative with respect to the marginal difference is identically zero, and its zero point lies on…

    Evidence level: C

    Two readers’ marginal differences cannot be reverse-engineered from the difference between Gwet’s AC1 and Cohen’s κ. This is because AC1’s expected agreement rate contains only the sum of the two margins, not their difference; therefore, throughout the calculation, only the κ side moves.

    The previous section left an enticing path: one could reverse-engineer how much the two readers’ marginal differences differ from the difference between Gwet’s AC1 and Cohen’s κ. The reasoning sounded solid—these two coefficients have different sensitivities to marginal differences. In κ’s expected agreement rate, the product of the two margins appears; changing them directly alters the value. In AC1’s expected agreement rate, only their average matters. Subtracting them should leave the marginal difference. The previous calculation showed this difference goes to zero when the mean is near 0.43 and the marginal difference is near 19.8 percentage points (pp), leading to the conclusion that fingerprints are blind where they are most needed. The direction was right, but the reason was not.

    Let’s break down AC1. The numerator is the observed agreement rate, which is data. The denominator’s expected agreement rate equals twice the mean multiplied by one minus the mean—the mean handles only the sum, not the difference. This means if one reader is biased positive and the other negative, as long as the average remains unchanged, AC1 does not move at all. Numerically fixed: mean 0.43, observed agreement rate 0.80, with marginal differences of 1pp, 5pp, 10pp, and 15pp, AC1 remains exactly 0.6076892899, identical to ten decimal places; under the same four scenarios, κ ranges from 0.592045 to 0.601157. Taking the partial derivative with respect to the marginal difference, AC1 and Scott’s π are precisely zero, while κ varies between 0.025 and 0.140.

    The nature of this result must be clarified: it is not that the difference is too small to detect, but that AC1 itself, by definition, contains no information about the marginal difference. The claim that “the difference between the two coefficients carries information about the marginal difference” is actually because only the κ half moves unidirectionally, while AC1 acts as a constant in the subtraction. The “zeroing” observed earlier is essentially a curve containing only κ intersecting a horizontal line.

    A harder blow follows. The original problem treats agreement rates between 0.85 and marginal differences from 14 to 20pp as “the most common scale in diagnostic studies,” yet this combination does not exist on a 2×2 table. Non-negative four-cell probabilities impose an exact constraint: the bottom-left cell equals one minus the observed agreement rate minus half the marginal difference, which must be non-negative; thus, the marginal difference cannot exceed one minus the observed agreement rate, regardless of the mean. When the observed agreement rate is 0.85, the maximum marginal difference is 15pp; at 0.90, it drops to 10pp. Checking 200,000 random tables individually, zero violations occurred. In other words, “high agreement rate” and “large marginal difference” are mutually exclusive states: if the marginal difference truly reaches 20pp, the agreement rate cannot exceed 0.80.

    The zeroing point is no longer mysterious. AC1 and κ go to zero when the marginal difference equals the absolute value of √2 times twice the mean minus one; at mean 0.43, this is exactly 19.80pp, but it falls within the feasible region only if the observed agreement rate does not exceed 1 minus this value. At that threshold, the bottom-left cell is precisely zero. Thus, the line is not hidden deep within parameter space; it lies on the edge of the simplex. Once the agreement rate exceeds 0.802, this line exits the feasible region, and the difference remains positive forever. The previous assumption that “zeroing” is a universal threat only occurs when the table degenerates to its boundary.

    The true solution lies not in the difference, but in the two level values themselves. The marginal mean is determined solely by AC1 and the observed agreement rate; solving the quadratic yields a closed-form expression: mean equals one minus √(one minus twice the expected agreement rate) divided by two. Once the mean is obtained, κ’s expected agreement rate allows reverse-solving for the marginal difference, yielding another closed form: √(twice the bracket), where the bracket contains three terms involving the mean minus the terms determined by the observed agreement rate and κ. Substituting back into 1,188 sets of random parameters yields a maximum error of 8.3×10⁻¹⁴—not an approximation, but an identity.

    The cost of this path is precision, not solvability. Using mean 0.43, agreement rate 0.80, and true marginal difference of 6pp as parameters for bootstrap: with sample sizes of 60, 200, 400, and 800, the sampling standard deviation of the marginal difference is 4.68pp, 3.01pp, 2.24pp, and 1.56pp respectively. To bring the 95% confidence interval half-width down to 3pp, approximately 800 cases are needed. By contrast, reverse-solving using only the difference leaves 42% to 61% of the same samples unsolvable, with standard deviations stuck between 4.1 and 4.8pp regardless of sample size—because the difference itself discards most information.

    This conclusion extends beyond binary classification. In multi-class settings, AC1’s expected agreement rate equals the sum of each category’s marginal proportion multiplied by one minus that proportion, divided by (number of categories minus one); it still contains only margins, no cross-category pairing structure. Weighted κ’s expected agreement rate includes products of cross-category weights and marginal products, introducing pairing structure. Thus, “AC1’s zero response to marginal difference” holds even in multi-class diagnosis, and the more one forces classification into binary form or pushes people toward high agreement rates, the more complete this zero response becomes.

    Practical implications are direct: reporting AC1 and κ together is acceptable, but their difference cannot serve as a measure of marginal difference; it can only act as a rough indicator of “how much κ is penalized by the marginal structure.” To obtain the marginal difference itself, report the two level proportions—the cost is zero. PMID 23627889 reports AC1 exceeding κ across all items in nine psychiatrists and 67 personality disorder cases (0.752 to 1.000 versus 0 to 1.00), attributing the difference to prevalence sensitivity—this is precisely the clinical trace of this zero response; PMID 8501467 early on required separate reporting of bias and prevalence to interpret κ; PMID 24338600 provides sample size reverse-solution based on target precision, directly applicable to the “800 cases for 3pp” planning above; PMID 36123642 sampled 82 consistency studies, yet only one-third provided sample size justification, indicating this precision accounting is largely overlooked in practice.

    Evidence and Limitations

    Verified (mathematical identity, not empirical conclusion): AC1’s expected agreement rate equals twice the mean multiplied by one minus the mean, containing only the sum of margins, not their difference; the partial derivative with respect to marginal difference is precisely zero, independent of mean and agreement rate values. With fixed mean 0.43 and observed agreement rate 0.80, as marginal difference ranges from 1pp to 15pp, AC1 remains exactly 0.6076892899. The zeroing condition occurs when marginal difference equals the absolute value of √2 times twice the mean minus one, achievable only if observed agreement rate does not exceed one minus this value. The feasible region constraint “marginal difference ≤ 1 − observed agreement rate” follows directly from non-negative four-cell probabilities, with zero violations among 200,000 random tables. Solving two level values jointly yields closed-form reverse solutions for mean and marginal difference, with maximum error of 8.3×10⁻¹⁴ upon substitution into 1,188 random parameter sets.

    Partially Verified: Reverse-solution precision is quantified via parameter bootstrap—when true marginal difference is 6pp, sample sizes of 60/200/400/800 correspond to sampling standard deviations of 4.68/3.01/2.24/1.56pp; this scale depends on the ideal condition that “observed agreement rate is known and free of systematic error.” By contrast, reverse-solving using only the difference leaves 42% to 61% of the same samples unsolvable, with standard deviations stuck between 4.1 and 4.8pp regardless of sample size.

    To Be Verified: The distribution of (mean, marginal difference, observed agreement rate) triplets in real diagnostic studies has not been empirically measured; all scales in this draft are built on parameter grids rather than literature backfill. While “AC1’s zero response to marginal difference” is given by formula in multi-class weighted scenarios, it lacks empirical dataset verification of whether zero response remains precise as the number of classes increases; this paper does not discuss whether AC1’s expected agreement rate structure still contains only margins in cases with three or more raters.

    Evidence Level: C
    References: PMID 23627889, PMID 8501467, PMID 24338600, PMID 36123642, PMID 25090041, PMID 33853549

    Further Reading

    L1 Network Status: EDGE_TABLE_science-yixue.md concepts stop at the physiological layer (407 articles on frequency, 147 on harmonics, 73 on pulse), with no entries for statistical agreement coefficients; CORRESPONDENCE_NETWORK.md shows “AC1” hits once, “Gwet” three times, and “Brennan” once, all falling under the E-060 label premise (organ differentiation as grouping label | premise | label reliability, status conflict, undecided); “identifiable,” “recognizability,” “prevalence,” “zero distribution,” and “coefficient pairs” all hit zero. This topic has not yet entered the corresponding L1 network.


    Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.

  • 案例漂移分两条路走,只量窄的那条会把它判成不存在

    Evidence level: C

    材料自己在变(换季、换批次、保存条件变)这件事对评分的影响分两条路:全体读者都觉得变难,走 occasion 主效应;只有少数读者特别敏感,走案例×读者交互。上一条判据只量了后者,于是最该防的那一种漂移被系统性判成「不存在」。

    正文

    先接着上一轮那笔账。上一条留下的判据是:把同一批材料给几位读者在几个时点各评一遍,做一次方差分解,看「案例与时点纠缠项」(σ²(o:r))占总方差几成——不到 5% 就认为案例漂移可忽略,原来那个共同模的点估计可以保留。

    这条判据有两个独立的毛病,一个比一个深。

    第一个毛病:量错了项。 设同一批材料、k 位读者、O 个时点各评一遍。材料本身在变,这件事对评分的影响可以分成两半。一半是共同的:所有人到了第三个月都觉得这批材料难读了——这落进 occasion 主效应 σ²(o)。另一半是读者特异的:只有某两位读者对某些时点格外敏感——这落进交互项 σ²(o:r)。

    「材料变质」最典型的样子恰恰是前一半。而我们只量后一半。

    仿真把这件事说得很清楚。设共同漂移 12 分、读者特异漂移 2 分(共同漂移占总方差 10.9%,两者差六倍),四组设计(80×5×3、80×8×3、80×10×4、120×10×5 到 200×12×6)跑下来,σ²(o:r) 的估计中位数都在 1.9–2.0,形态完好;然后按那条判据判——99.3%、100%、100%、100% 的样本都落进「<5%,可忽略」档。而真实的案例漂移占了 10.9%。

    判据在它最该报警的情形下,百分之百判「没事」。这不是精度不足,是瞄错了靶。

    第二个毛病:分母没写。 就算量对了项,还有一层。同一个 σ²(o:r)=2.0,换个分母就是另一档结论:占总方差 2.2%(落 <5% 档)、占读者侧方差(读者主效应+案例×读者+人×读者)15.4%(落 5–20% 档)、占读者主效应一项 33%(落 >20% 档)。同一个数,三个档全占。

    判据原文只说「占比」,没说占谁的比。于是「跑出来落第几档」取决于跑的人选了谁当分母。这不是统计精度问题,是可复现性问题——两个人拿同一批数据、跑同一套代码,可以合法地得出「可忽略」和「不可辨识」两个相反结论。

    第三件事更硬:这个量的自由度挂在哪儿。 σ²(o:r) 的自由度是 R×(O−1),与病例数 P 完全无关。仿真里把病例从 80 加到 800(十倍),区间相对宽度只从 156% 缩到 147%,几乎没动;把读者从 5 位加到 20 位,156% 缩到 83%;把时点从 3 个加到 12 个,缩到 72%。

    换句话说,加病例对这个量一点帮助都没有。研究者的直觉是「多收点材料就更准」,在这个量上是错的。真正稀缺的资源是时点,其次才是读者数。

    而 σ²(o) 主效应更惨——它的自由度只有 O−1。O=3 时自由度是 2,真值 12 分,5–95 区间是 [0.13, 35.56]:上界是真值的三倍,下界几乎贴零。就算做到 O=12,区间仍是 [4.93, 20.35]。也就是说,共同漂移这一项,在能落地的设计里根本估不出来。 承认这一点,比硬报一个点估计有用得多。

    最狠的一处:占比和后果脱钩。 固定「案例漂移占总方差 5%」不变,只改人的真差异 S² 和读者噪声 T,共同模点估计的改变量 Δc 会从 5.2 个百分点一路变到 13.6 个百分点——同样 5%,后果差 2.6 倍。原因很直白:Δc = A/(S²+A+T/k),分母里挂着 S²。人差异越大,那个占比就被稀释得越厉害,但后果一分不减。

    这条教训和上一轮「偏倚的离散该用离散-均值比、不该用方差占比」是同一件事的两面:占比类判据在人差异大的场景里,系统性低估后果。

    何况「可忽略」这条线画在 5% 上,可真实后果早就越线了——A/S²=0.05 时 Δc 已有 4.0–4.6 个百分点,A/S²=0.10 就到 7.6–8.8。若把「影响结论」定在 5 个百分点,这条线在 A/S²≈0.06 处就被跨过,恰好压在「可忽略」档的边界上。

    所以该怎么办:别报档,直接报那个数。 要问的从来不是「案例漂移占几成」,而是「扣与不扣,共同模的点估计差几个百分点」。这个量就是 Δc = 100·A/(S²+A+T/k),单位是百分点,它本身就是答案,不需要三档阈值去翻译。真要设线,就设成:Δc>5 个百分点,等价于 A > 0.053×(S²+T/k)。门槛随场景自己走,不借别人的量纲。

    外部证据也支持「这一项必须以本批材料自测」。一项 135 名学生、2 位读者、3 个时点的概化研究里,occasion 方差可忽略、student-occasion 交互也小,而 student-rater 交互占 17.7%、三阶含残差占 27.7%,Φ 系数 0.75(PMID 32581621)——注意它报的是「小」,但那是特定结构下的「小」。另一项跟踪 458 名教师两年的观察评分研究给出反面:评分者漂移在最初几天非常大、且持续近两年,评分者不但没有收敛到共同的严厉度,彼此差异反而随时间扩大;趋势项相对评分者效应、特定课次上的评分误差与残差算是「中等」而非主项(PMID 29795823)。而重评抽样方式本身会实质影响漂移检测结论,边际齐性检验在位移跨类别平衡时功效很低(PMID 29881127)。44 项研究的系统综述加 meta 分析给的汇总数是:人的方差 44.2%、G 系数 0.65,并明确点出「情境因素会影响方差成分,报告严谨性仍需加强」(PMID 33951677)。

    一句话:案例漂移不是一个可以查表的常数,它是每一批材料自己的属性;而判定它的那一项、那个分母、那个自由度,三处都不能照抄。

    证据与局限

    已验证:σ²(o:r) 的自由度是 R×(O−1)、与病例数无关,这是方差分解的代数事实,仿真中把 P 从 80 加到 800 区间宽度仅从 156% 缩到 147% 完全吻合。Δc 的恒等式 A/(S²+A+T/k) 亦可直接推导。概化理论在评分研究中的用法与方差成分的异质性,有系统综述级的外部支撑(PMID 33951677、32581621、29795823)。

    部分验证:共同漂移进 occasion 主效应、特异漂移进交互项,这一拆分在完全交叉设计下成立;但若实际设计是「每人对每读者每时点只评一次」,人×读者项与三阶残差完全混杂(仿真中 σ²(p×r) 真值 5.0、估计中位 −26 到 −7,100% 为负),此时 σ²(o:r) 本身还会被系统性压低 19%–66%。三档阈值的具体数值(5%/20%)来自上一条判据的原始表述,本文只论证其分母与选项目标有误,未独立验证阈值本身的来源。

    待验证:① 真实研究里「共同漂移」与「特异漂移」的实际比例,目前只有零星研究报告 occasion 方差「可忽略」,缺乏跨研究的分布;② Δc 的 5 个百分点门槛是否与实际决策后果(是否改变「共同模存在」这一结论)对应,尚无可校准的外部标准;③ 若漂移是单调趋势而非随机波动,方差分解会把趋势摊进残差,此时本文的拆分式需改写为带趋势项的增广模型。

    相关阅读

    证据等级: C
    参考文献: PMID 32581621, PMID 29795823, PMID 29881127, PMID 33951677
    L1 网络状态:边表 EDGE_TABLE_science-yixue.md 中「概化/方差/漂移/时点/重测/评定者/kappa/信度/一致/误差/校正/异质」关键词命中数全为 0,该主题尚未进入 L1 对应网络;网络侧最接近的是 E-060 标签前提(KB-317/333/334,状态 conflict 未裁决),故本文证据等级封顶 C。

    证据等级:A=有PubMed直接支撑,B=科学类比/间接证据,C=传统文化共识/初步探索


    免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。

    Case drift splits into two paths, and measuring only the narrow one will classify it as nonexistent.

    Evidence level: C

    Material itself changes (seasonal shifts, batch changes, storage conditions). The impact of this change on scoring follows two paths: if all readers perceive increased difficulty, it enters the occasion main effect; if only a few readers are particularly sensitive, it enters the case×reader interaction. The previous criterion measured only the latter, so the most dangerous type of drift was systematically judged as “non-existent.”

    Main Text

    Continuing from the previous calculation. The criterion left by the prior section was: evaluate the same batch of materials by several readers at several time points, perform variance decomposition, and examine what proportion the “case×occasion entanglement term” (σ²(o:r)) constitutes of the total variance—if less than 5%, case drift is considered negligible, and the original common model point estimate can be retained.

    This criterion has two independent flaws, one deeper than the other.

    First flaw: measuring the wrong term. Suppose the same batch of materials, k readers, and O time points each evaluate once. The material itself changes, and the impact of this change on scoring can be split into two halves. One half is common: everyone finds the materials harder to read by the third month—this falls into the occasion main effect σ²(o). The other half is reader-specific: only certain readers are particularly sensitive at certain time points—this falls into the interaction term σ²(o:r).

    The most typical manifestation of “material degradation” is precisely the first half. Yet we only measure the second half.

    Simulation makes this clear. With common drift of 12 points and reader-specific drift of 2 points (common drift accounts for 10.9% of total variance, six times the other), running four designs (80×5×3, 80×8×3, 80×10×4, 120×10×5 to 200×12×6), the median estimate of σ²(o:r) remains between 1.9–2.0, with intact structure; yet according to that criterion, 99.3%, 100%, 100%, 100% of samples fall into the “<5%, negligible” category. Meanwhile, the true case drift accounts for 10.9%.

    The criterion fails completely in the very situation where it should alarm. This is not a precision issue; it is aiming at the wrong target.

    Second flaw: missing denominator. Even if we measure the right term, there is another layer. The same σ²(o:r)=2.0 yields different conclusions depending on the denominator: 2.2% of total variance (falls into <5% category), 15.4% of reader-side variance (reader main effect + case×reader + person×reader) (falls into 5–20% category), or 33% of reader main effect alone (falls into >20% category). The same number occupies three categories.

    The original criterion merely states “proportion,” without specifying proportion of what. Thus, which category the result falls into depends entirely on who the researcher chooses as the denominator. This is not a statistical precision issue; it is a reproducibility issue—two people using the same data and running the same code can legitimately arrive at opposite conclusions: “negligible” versus “unidentifiable.”

    Third issue is harder: where does the degrees of freedom hang? The degrees of freedom for σ²(o:r) is R×(O−1), completely independent of the number of cases P. In simulation, increasing cases from 80 to 800 (tenfold) shrinks the relative interval width only from 156% to 147%, almost no change; increasing readers from 5 to 20 shrinks it from 156% to 83%; increasing time points from 3 to 12 shrinks it to 72%.

    In other words, adding cases does not help this measure at all. Researchers intuitively believe “more materials yield better accuracy,” but this is wrong for this measure. The truly scarce resource is time points, followed by the number of readers.

    The σ²(o) main effect fares even worse—its degrees of freedom is only O−1. When O=3, the degrees of freedom is 2, with true value 12 points, the 5–95 interval is [0.13, 35.56]: the upper bound is three times the true value, the lower bound nearly zero. Even with O=12, the interval remains [4.93, 20.35]. Thus, the common drift term cannot be estimated in feasible designs. Acknowledging this is far more useful than reporting a point estimate.

    Most severe point: proportion decouples from consequence. Fixing “case drift accounts for 5% of total variance,” but varying the true person difference S² and reader noise T, the change in common model point estimate Δc ranges from 5.2 percentage points to 13.6 percentage points—same 5%, yet consequences differ by 2.6 times. The reason is straightforward: Δc = A/(S²+A+T/k), where the denominator includes S². As person differences grow, this proportion becomes increasingly diluted, yet the consequence does not diminish.

    This lesson mirrors the earlier point about “bias dispersion should use discrete-mean ratio, not variance proportion”: proportion-based criteria systematically underestimate consequences in scenarios with large person differences.

    Moreover, the “negligible” threshold is set at 5%, yet true consequences have long exceeded this line. When A/S²=0.05, Δc is already 4.0–4.6 percentage points; when A/S²=0.10, it reaches 7.6–8.8. If we define “impact on conclusion” at 5 percentage points, this threshold is crossed when A/S²≈0.06, precisely at the boundary of the “negligible” category.

    So what to do: do not report categories; directly report the number. The question is never “what proportion does case drift account for,” but rather “with or without adjustment, how many percentage points does the common model point estimate differ by.” This quantity is Δc = 100·A/(S²+A+T/k), measured in percentage points, and it is itself the answer—no need for three thresholds to translate. If a line must be set, define it as: Δc>5 percentage points, equivalent to A > 0.053×(S²+T/k). The threshold adapts to the scenario itself, without borrowing someone else’s units.

    External evidence also supports “this term must be self-tested on this batch of materials.” In one generalized study with 135 students, 2 readers, and 3 time points, occasion variance was negligible and student-occasion interaction was small, while student-rater interaction accounted for 17.7% and the third-order term including residuals accounted for 27.7%, with Φ coefficient 0.75 (PMID 32581621)—note it reports “small,” but that is “small” under a specific structure. Another study tracking 458 teachers over two years found the opposite: rater drift was very large initially and persisted for nearly two years; raters did not converge to common severity, and their differences actually widened over time; the trend term relative to rater effect, specific lesson scoring error, and residuals counts as “moderate” rather than main term (PMID 29795823). Moreover, resampling method itself substantially affects drift detection conclusions, with marginal homogeneity test having low power when displacement crosses categories (PMID 29881127). A systematic review of 44 studies plus meta-analysis yielded aggregate figures: person variance 44.2%, G coefficient 0.65, and explicitly noted “contextual factors affect variance components; reporting rigor still needs strengthening” (PMID 33951677).

    In one sentence: case drift is not a constant to look up in a table; it is an attribute of each batch of materials; and the term used to judge it, its denominator, and its degrees of freedom—all three cannot be copied.

    Evidence and Limitations

    Verified: σ²(o:r) degrees of freedom is R×(O−1), independent of case count, which is an algebraic fact of variance decomposition; simulation where P increased from 80 to 800 with interval width shrinking only from 156% to 147% fully matches. The identity Δc = A/(S²+A+T/k) can be directly derived. Use of generalized theory in rating studies and heterogeneity of variance components have systematic review-level external support (PMID 33951677, 32581621, 29795823).

    Partially verified: common drift enters occasion main effect, specific drift enters interaction term; this split holds under fully crossed design; but if actual design is “each person rates each reader at each time point only once,” person×reader term and third-order residual are completely confounded (in simulation σ²(p×r) true value 5.0, estimated median −26 to −7, 100% negative), causing σ²(o:r) itself to be systematically suppressed by 19–66%. Specific threshold values (5%/20%) come from the original statement of the previous criterion; this paper only argues that its denominator and selection target are wrong, without independently verifying the origin of the thresholds themselves.

    To be verified: ① actual proportion of “common drift” vs “specific drift” in real studies; currently only scattered reports indicate occasion variance is “negligible,” lacking cross-study distribution; ② whether the 5-percentage-point threshold for Δc corresponds to actual decision consequences (whether it changes the conclusion of “common model exists”), with no calibratable external standard yet; ③ if drift is monotonic trend rather than random fluctuation, variance decomposition spreads the trend into residuals, requiring this paper’s split formula to be rewritten as an augmented model with trend terms.

    Related Reading

    Evidence level: C
    References: PMID 32581621, PMID 29795823, PMID 29881127, PMID 33951677
    L1 network status: All keywords in EDGE_TABLE_science-yixue.md—”generalization/variance/drift/time point/retest/rater/kappa/reliability/consistency/error/correction/heterogeneity”—have zero hits; this topic has not entered the corresponding L1 network. The closest network-side label is E-060 premise (KB-317/333/334, status conflict unresolved), so this paper’s evidence level is capped at C.

    Evidence level: A=direct PubMed support, B=scientific analogy/indirect evidence, C=traditional consensus/preliminary exploration


    Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.

  • 五步联合验证为什么几乎不可能通过:合取判据的乘法代价

    Evidence level: B

    先说结论

    一套”三步、每步四条判据、全过才算通过”的验证链,看起来只是”我们查了 12 项”。但把 12 项当合取(AND)之后,通过率按乘法塌下来:如果每条判据在真值恰好落在门槛上时只有一半机会判对,那么整链通过率是 0.5¹² ≈ 0.024%。即使每条判据有 80% 的正确率,整链通过率也只有 6.9%。

    更值得注意的不是”通过很难”,而是”作废很容易”:当同一套结构把”三步中至少三步失败”作为降级条件时,在每条判据毫无分辨力的极端情形下,被判”作废”的概率高达 82%,而被判”通过”的概率只有 0.024%。这两件事方向相反,叠加后体系会单向偏向”作废”。

    1. 结构长什么样

    这条链(下称 N19 链)由三步组成,每步四条判据,共 12 条:

    步 判据(摘要)
    站点交叉核对 相关系数 ≥ 0.7;外包络差 ≤ 0.15;变暖贡献 ∈ [30%, 50%];城市差异 ≤ 30%
    模式对比 四项 M 判据,形式同上
    城市化剥离 趋势差 < 0.05 ℃/decade;相关系数差 ≤ 0.07;占比 ∈ [30%, 41%];两套资料差 ≤ 0.10 ℃/decade

    结果处置是步级的,不是判据级:零步失败且上游基座达标 → 升级候选;一至两步失败 → 降一档;三步及以上失败 → 降两档并宣布原结论作废;任一步未启动 → 维持原等级。

    2. 乘法代价

    设单条判据在”真值恰落门槛”时的通过率为 p,则每步(4 条 AND)的通过率是 p⁴,三步全过是 p¹²:

    单条判据通过率 p 单步通过率 p⁴ 三步全过 三步全失败
    0.5 6.25% 0.024% 82.40%
    0.7 24.01% 1.38% 43.88%
    0.8 40.96% 6.87% 20.58%
    0.9 65.61% 28.24% 4.07%

    这张表的读法是:“查了 12 项”不等于”要求很严”,而是”把通过率压到 0.02%–7%”。如果对外说”我们做了五步联合验证”,读者会以为这是稳健的标志,实际是把一个本来就难的判定压到几乎不可能达成。

    3. 被最钝的一道决定

    如果四条判据的通过率不同,联合通过率并不是平均,而是被最低的那条主导。以 0.95 / 0.90 / 0.85 / 0.50 为例:

    • 联合 = 36.3%
    • 把最钝的一道从 0.50 提到 0.95 → 联合升到 72.7%,收益 2.00 倍
    • 把任一条从 0.90 提到 0.95 → 收益只有 0.95 倍(几乎为零)

    所以”多加几道判据更稳”和”平均提升每条判据”都不是正确动作。正确动作永远是:先找最钝的那一道,只修它。

    4. 双向不对称:通过稀薄、作废廉价

    上面两张表要一起看。在 p=0.5(判据没有分辨力)时:

    判"通过"(三步零失败):0.024%
    判"作废"(三步全失败):82.4%
    

    这不是”体系很保守”,而是体系在没信息的时候会大声说”作废”。原因在结构:每步是 4 连 AND,任一条判错整步就算失败;而”失败”与”没做对”在判据层面无法区分。于是精度不足会被自动翻译成”结论不成立”。

    这与”止损闸门不该把不可逆侧阈值设在目标值上”是同一条原则的多层放大版:单层时误杀一半,叠加两层后误杀变成单向偏置。

    5. 模板表漏了一格

    把处置规则穷举,16 种组合里有一种无处可落:三步全部实测通过,但上游基座(三源再分析交叉验证)未达 B+。

    这不是假想的边角情形,而是很常见的真实情形——下游三步跑通了,上游数据还没站住。按现行文本,执行者会卡住,然后临时自选一个模板(最可能选”升级候选”,因为”三步都过了”),于是”上游不足”被静默吞掉。

    修法是把这一格显式补上:三步零失败但上游未达标 → 维持原等级、不登记发布候选、先补上游。

    6. 一个和判据无关但更该先修的问题

    这条链引用的中国均一化气温数据集,编号写错了类型。

    文中把 11436746 标为 PMID。但把 11436746 送进 PubMed,返回的是一篇 2001 年关于门诊精神科服务满意度的论文;把它送进 PMC 数据库,才返回目标数据集论文(CUG-CMA CHDT,1840–2020 年中国日均温)。也就是说,这是 PMCID 的数字部分被当成了 PMID。

    同一篇文献有三个编号,必须分清:

    PMID  39333104
    PMCID PMC11436746
    DOI   10.1038/s41597-024-03758-3
    

    作者也写错了:文中两处写 Cao et al. 2024,实际第一作者是 Zheng X,任国玉为第三作者兼通讯;另有一处写 Xu et al.。

    为什么这类错误特别危险:11436746 是一个真实存在的编号,只是指向另一篇论文。任何”检查 PMID 是否存在”的校验都会通过——校验必须比对标题与作者,不能只验存在性。目前这个写法扩散在 9 个文件、22 处,其中同一文件内并存 PMC11436746(对)与 PMID 11436746(错)两种写法,说明是复制粘贴时前缀丢失,而不是理解错误,修起来是机械替换。

    另有一处引用(36089010,中国 86 城热岛研究)被用来支撑”站点被城市包围比例从 30% 升到 60%”。该文摘要给出的是热岛强度 0.33 ℃、趋势 0.013 ℃/10a、城乡法贡献 53%、观测减再分析 23%,没有”30%→60% 站点比例”这个数字。该数可能出自正文,但本轮未核到,应标”待核”,不应写成已引。

    7. 证据边界

    标 B。三条文献均逐字段回查(含 PMCID/DOI 三编号对齐):

    • PMID 39333104 / PMC11436746 / DOI 10.1038/s41597-024-03758-3:Zheng X, Ren Y, Ren G 等,Sci Data 2024;11:1045,中国 1840–2020 日均温均一化数据集。
    • PMID 36089010:Zhou D 等,Sci Total Environ 2023;854:158638,中国 86 城大气热岛多视角研究。
    • PMID 18457487:Meade 等,J Appl Psychol 2008,测量不变性中拟合指标的功效与样本量敏感性。

    没有一条直接研究”多层合取判据的联合功效”——那是本项目自建的结构,故只能标 B。

    本文所有通过率都建立在”12 条判据相互独立”的假设下。真实情形里,同一步的四条判据往往由同一批站点、同一段序列算出,彼此正相关,实际通过率会高于表中的值。所以本文主张的不是”真实通过率是 6.9%”,而是”必须报联合通过率;不报就等于把严格度藏起来”。

    “作废概率 82%”同理,是极端情形(判据完全无分辨力)的上界讨论,不是对 41% 归因结论的任何判断。它既不说 41% 为真,也不说 41% 为假,只说这套判据在现有设置下无法区分”通过了”和”没做对”。

    可执行修订

    1. 引用”五步联合验证”时,同时报判据总数、每条预期通过率、联合通过率。
    2. 找出并优先修最钝的那一条判据,不要平均加严。
    3. 模板表补一格:三步零失败但上游基座未达标 → 维持原等级、不登记发布候选。
    4. 全库把 PMID 11436746 改为 PMID 39333104(保留 PMC11436746 写法),作者改为 Zheng/Ren/Ren;Cao et al. 与 Xu et al. 同步更正。
    5. 36089010 支撑的”30%→60%”改标”待核”。
    6. 校验引文时同时比对标题与作者,不只验编号存在。

    免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。

    Why Five-Way Joint Verification is Almost Impossible: The Multiplicative Cost of the Conjunction Criterion

    Evidence level: B

    Conclusion First

    A verification chain described as “three steps, four criteria per step, all must pass” appears to be simply “we checked 12 items.” However, treating these 12 items as a conjunction (AND) causes the pass rate to collapse multiplicatively: if each criterion has only a 50% chance of being correctly judged when the true value lies exactly at the threshold, the overall chain pass rate is 0.5¹² ≈ 0.024%. Even if each criterion has an 80% accuracy, the overall pass rate drops to only 6.9%.

    What is more noteworthy is not that “passing is hard,” but that “being invalidated is easy”: when the same structure uses “failure in at least three of the three steps” as a downgrade condition, the probability of being judged “invalidated” reaches 82% in the extreme case where each criterion has no discriminative power, while the probability of being judged “passed” is only 0.024%. These two directions oppose each other, causing the system to unidirectionally bias toward “invalidation.”

    1. What Does the Structure Look Like?

    This chain (hereafter N19 chain) consists of three steps, each with four criteria, totaling 12:

    Step Criteria (Summary)
    Site Cross-Check Correlation coefficient ≥ 0.7; outer envelope difference ≤ 0.15; warming contribution ∈ [30%, 50%]; urban difference ≤ 30%
    Pattern Comparison Four M criteria, form as above
    Urbanization Removal Trend difference < 0.05 ℃/decade; correlation coefficient difference ≤ 0.07; proportion ∈ [30%, 41%]; two datasets difference ≤ 0.10 ℃/decade

    Result handling is step-level, not criterion-level: zero step failures with upstream base达标 → upgrade candidate; one or two step failures → downgrade one tier; three or more step failures → downgrade two tiers and declare original conclusion invalid; any step not initiated → maintain original tier.

    2. Multiplicative Cost

    Let the pass rate of a single criterion when “the true value lies exactly at the threshold” be p. Then the pass rate of each step (4 AND) is p⁴, and passing all three steps is p¹²:

    Single Criterion Pass Rate p Single Step Pass Rate p⁴ All Three Steps Pass All Three Steps Fail
    0.5 6.25% 0.024% 82.40%
    0.7 24.01% 1.38% 43.88%
    0.8 40.96% 6.87% 20.58%
    0.9 65.61% 28.24% 4.07%

    The reading of this table is: “Checking 12 items” does not mean “requiring strictness,” but rather “compressing the pass rate to 0.02%–7%.” If externally stating “we performed five-step joint verification,” readers will think this is a sign of robustness, actually compressing an already difficult judgment into almost impossible to achieve.

    3. Decided by the Dullest Criterion

    If the pass rates of the four criteria differ, the joint pass rate is not averaged but dominated by the lowest one. Taking 0.95 / 0.90 / 0.85 / 0.50 as an example:

    • Joint = 36.3%
    • Raising the dullest criterion from 0.50 to 0.95 → joint rises to 72.7%, gain 2.00 times
    • Raising any criterion from 0.90 to 0.95 → gain only 0.95 times (almost zero)

    Therefore “adding more criteria makes it more stable” and “averaging improvement of each criterion” are not correct actions. The correct action is always: first find the dullest one, fix only it.

    4. Bidirectional Asymmetry: Passing Thin, Invalidating Cheap

    The above two tables must be read together. At p=0.5 (criteria have no discriminative power):

    Judge "passed" (zero failures in three steps): 0.024%
    Judge "invalidated" (all three steps fail): 82.4%
    

    This is not “the system is very conservative,” but rather the system loudly says “invalidated” when there is no information. The reason lies in the structure: each step is 4-con AND, any single criterion failure makes the whole step fail; while “failure” and “not done right” cannot be distinguished at the criterion level. Thus insufficient precision is automatically translated into “conclusion not valid.”

    This is a multi-layer amplification of the same principle as “the stop-loss gate should not set the irreversible-side threshold at the target value”: single-layer mis-kills half, adding two layers makes mis-kill become unidirectional bias.

    5. Template Table Missed One Cell

    Exhausting the disposal rules, among 16 combinations there is one nowhere to fall: all three steps pass in measurement, but the upstream base (three-source reanalysis cross-validation) does not reach B+.

    This is not a hypothetical corner case, but a very common real situation—downstream three steps run through, upstream data not yet standing. According to current text, the executor will be stuck, then temporarily choose a template (most likely “upgrade candidate,” because “all three passed”), thus “upstream insufficient” is silently swallowed.

    The fix is to explicitly add this cell: three steps zero failures but upstream not met → maintain original tier, do not register publish candidate, first supplement upstream.

    6. One Issue Unrelated to Criteria but More Urgent to Fix

    The chain referenced the Chinese homogenized temperature dataset, with the number written wrong type.

    In the text 11436746 is marked as PMID. But sending 11436746 into PubMed returns an article from 2001 about outpatient psychiatry service satisfaction; sending it into PMC database returns the target dataset article (CUG-CMA CHDT, 1840–2020 China daily mean temperature). That is, this is the PMCID number part mistakenly treated as PMID.

    The same article has three numbers, must distinguish:

    PMID  39333104
    PMCID PMC11436746
    DOI   10.1038/s41597-024-03758-3
    

    The authors also wrote wrong: two places write Cao et al. 2024, actually first author is Zheng X, Ren Guoyu is third author and corresponding; another place writes Xu et al..

    Why such errors are especially dangerous: 11436746 is a real existing number, just pointing to another paper. Any “check whether PMID exists” validation will pass—validation must compare title and authors, not only existence. Currently this writing spreads in 9 files, 22 places, where within the same file both PMC11436746 (correct) and PMID 11436746 (wrong) coexist, indicating copy-paste lost prefix rather than understanding error, fixing it is mechanical replacement.

    Another citation (36089010, China 86-city heat island study) is used to support “site surrounded by urban proportion rose from 30% to 60%”. The abstract of that paper gives heat island intensity 0.33 ℃, trend 0.013 ℃/10a, urban-rural method contribution 53%, observation minus reanalysis 23%, no “30%→60% site proportion” number. This number may come from the main text, but not checked in this round, should mark “pending verification,” not written as cited.

    7. Evidence Boundary

    Mark B. Three literature sources checked field-by-field (including PMCID/DOI three-number alignment):

    • PMID 39333104 / PMC11436746 / DOI 10.1038/s41597-024-03758-3: Zheng X, Ren Y, Ren G et al., Sci Data 2024;11:1045, China 1840–2020 daily mean temperature homogenized dataset.
    • PMID 36089010: Zhou D et al., Sci Total Environ 2023;854:158638, China 86-city atmospheric heat island multi-perspective study.
    • PMID 18457487: Meade et al., J Appl Psychol 2008, power and sample size sensitivity of fit indices in measurement invariance.

    None directly studies “joint efficacy of multi-layer conjunction criteria”—that is the structure built by this project, so can only mark B.

    All pass rates in this paper are based on the assumption that “12 criteria are mutually independent.” In reality, the four criteria of each step are often calculated from the same set of sites and same sequence, positively correlated, actual pass rate will be higher than values in the table. Therefore this paper does not claim “real pass rate is 6.9%,” but rather “must report joint pass rate; not reporting equals hiding strictness.”

    “Invalidation probability 82%” similarly, is the upper bound discussion of extreme case (criteria completely non-discriminative), not any judgment on the 41% attribution conclusion. It neither says 41% is true nor false, only that under current settings this set of criteria cannot distinguish “passed” and “not done right.”

    Executable Revisions

    1. When citing “five-step joint verification,” simultaneously report total number of criteria, expected pass rate per criterion, joint pass rate.
    2. Find and prioritize fixing the dullest criterion, do not average tightening.
    3. Template table add one cell: three steps zero failures but upstream base not met → maintain original tier, do not register publish candidate.
    4. Change PMID 11436746 to PMID 39333104 throughout the entire database (keep PMC11436746 writing), authors corrected to Zheng/Ren/Ren; Cao et al. and Xu et al. simultaneously corrected.
    5. The “30%→60%” supported by 36089010 marked “pending verification.”
    6. When validating citations, simultaneously compare title and authors, not only check number existence.

    Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.

  • 疲劳到底把判读噪声放大了几倍:那个写死的1.3倍,决定一个实验是省事还是报废

    Evidence level: C

    一个写死在推演模型里的「疲劳使噪声乘1.3」,既是没人量过的设定值,又是「填充材料无害」这条结论唯一撑得住的支点——它若实际是1.1,规则可以简化;它若实际是1.6,同样一批填充可以直接把实验判定翻面。

    正文(约1150字)

    用实验去测「气功外气」这类微弱效应,最大的敌人从来不是效应本身,而是测量工具自己的抖动。前面那篇填充材料的分析(#1429)给出过一条设计规则:判读序列里塞进同难度的「填充项」,可以压住前后材料的对比效应,但代价是判读者的疲劳会悄悄放大单次判读的噪声。问题出在一处细节:那套推演全部固定「疲劳使噪声乘以1.3」——这个数是设定的,不是测的。

    这个设定值有多要命?把三种可能的实测结果摆在一起就明白了。若疲劳实际只把噪声放大1.15倍,一致性系数几乎纹丝不动(0.811到0.811),任何下游检验都看不出差别——「填充无害」成立,规则可以简化成「只要对比项可控就行」。若放大1.3倍,合计通过率下降约1.71个百分点,尚在容忍范围。若放大1.6倍,通过率要掉将近4个百分点——同一个设计,判定可以从「无影响」直接跨到「必须改设计」。一个没人量过的常数,决定了整条规则的生死。

    文献对疲劳效应的方向没有争议,对大小没有共识。持续注意的元分析确认:简单重复任务里,时间效应(警觉性随任务进程下降)是最稳定、最明显的现象之一,其神经机制包括注意网络活动的衰减与默认网络的代偿性激活(PMID 23163491)。动态环境中的警觉研究进一步表明,负荷的「事件率」和任务结构都会改变警觉下降的速度——同样是判读,连续快节奏的填充和留白间隔的填充,对噪声的影响未必一样(PMID 26844367)。落到真实判读场景:CT结肠造影的多中心研究发现判读时间受读者经验与疲劳状态共同影响,疲劳不只拖慢判读,还会改变判读的稳定性(PMID 16636802);乳腺筛查中判读过程被打断与医生负荷的研究则说明,负荷本身是一个可以被操纵、也必须被记录的变量(PMID 38056581)。方向都有——但没有任何一篇给出过「噪声放大倍数」这个数。方向确定、大小空白,正是本主题要补的位置。

    值得强调的是,这条主题属于共振意识场研究池的方法学地基:任何「外气影响判读」的判决实验,前提都是判读工具本身的噪声特性已知。工具在疲劳下的噪声曲线都不清楚,读数的任何波动都可以被归咎于疲劳,效应检测就永远差一口气。

    可做的实测设计并不贵:招募20位以上判读者,同一批60例以上主材料,随机分到两个负荷条件——先判40例同难度填充材料,或先判40例空白间隔——再判同一批主材料。每位读者对部分材料重复判读,用重复判读的离散度(或Rasch类模型的个案误差项)反解出每位读者在两种负荷下的噪声尺度,两者相除就是放大倍数。

    判据事先定死:实测放大倍数落在1.1以内,疲劳可忽略,填充规则简化为「只要对比项可控」;落在1.4以上,填充量本身必须当作剂量变量处理,做剂量-反应曲线;落在中间,则现行规则保留但标注适用边界。

    证据与局限

    已验证:警觉性随任务时间下降是元分析级别的稳定现象(PMID 23163491);负荷与任务结构改变警觉动力学(PMID 26844367);判读表现受疲劳影响在影像判读中有实证(PMID 16636802)。
    部分验证:疲劳会改变判读稳定性(时间/一致性),但「噪声放大倍数」作为可计算量在判读场景从未被报告。
    待验证:放大倍数在1.1/1.4两个阈值之间的具体落点;倍数是否存在个体差异、是否随材料难度调制;1.3这个设定值本身。

    相关阅读

    L1 网络状态:该主题尚未进入 L1 对应网络(EDGE_TABLE_science-yixue.md 无「疲劳/判读噪声/填充材料」对应边;CORRESPONDENCE_NETWORK.md 最近的相邻边为 E-060 标签信度 conflict,PMID 30383398,属中医诊断标签问题,非同一条边)。

    证据等级: C
    参考文献: PMID 23163491, PMID 26844367, PMID 16636802, PMID 38056581

    证据等级:A=有PubMed直接支撑,B=科学类比/间接证据,C=传统文化共识/初步探索


    免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。

    How much does fatigue amplify noise judgment: that hardcoded 1.3 factor decides whether an experiment is easy or scrapped

    Evidence level: C

    A hardcoded assumption in the deduction model—that fatigue multiplies noise by 1.3—is both an unmeasured parameter and the sole support for the conclusion that “filler materials are harmless.” If the actual multiplier were 1.1, the rules could be simplified; if it were 1.6, the same batch of fillers could flip the experimental judgment entirely.

    Main Text (approx. 1150 words)

    When experiments attempt to measure weak effects like “Qigong external Qi,” the greatest enemy is never the effect itself, but the jitter of the measurement tool. The previous analysis of filler materials (#1429) established a design rule: inserting same-difficulty “filler items” into the judgment sequence suppresses contrast effects between adjacent materials, but at the cost that reader fatigue quietly amplifies single-judgment noise. The problem lies in one detail: the entire deduction fixed “fatigue multiplies noise by 1.3″—this number is set, not measured.

    How critical is this set value? Placing three possible measured outcomes side-by-side clarifies the stakes. If fatigue actually amplifies noise only by 1.15 times, the consistency coefficient remains virtually unchanged (0.811 to 0.811), and any downstream tests show no difference—”fillers are harmless” holds, and the rule simplifies to “as long as contrast items are controllable.” If amplified by 1.3 times, the total pass rate drops by about 1.71 percentage points, still within tolerance. If amplified by 1.6 times, the pass rate falls nearly 4 percentage points—under the same design, the judgment could shift directly from “no impact” to “design must be changed.” An unmeasured constant determines the life or death of the entire rule.

    Literature agrees on the direction of fatigue effects but lacks consensus on magnitude. Meta-analyses of sustained attention confirm that in simple repetitive tasks, time effects (vigilance declining with task progression) are among the most stable and obvious phenomena, with neural mechanisms including attenuation of attention network activity and compensatory activation of the default network (PMID 23163491). Studies of vigilance in dynamic environments further show that load “event rate” and task structure alter the speed of vigilance decline—meaning for the same judgment task, continuous rapid-fillers versus spaced fillers may affect noise differently (PMID 26844367). In real judgment scenarios: a multicenter CT colonography study found that reading time is jointly influenced by reader experience and fatigue state; fatigue not only slows reading but also changes reading stability (PMID 16636802); research on breast screening interruptions and physician load indicates that load itself is a manipulable variable that must be recorded (PMID 38056581). Directions are clear—but no paper has ever reported the “noise amplification multiplier.” Direction is determined, magnitude is blank; this is exactly where this topic must fill the gap.

    It is worth emphasizing that this topic forms the methodological foundation of resonance consciousness field research: any judgment experiment on “external Qi affecting reading” presupposes known noise characteristics of the reading tool. Without clear noise curves under fatigue, any fluctuation in readings can be blamed on fatigue, and effect detection will always fall short.

    The feasible experimental design is not expensive: recruit 20+ readers, use the same batch of 60+ primary materials, randomly assign to two load conditions—first judge 40 same-difficulty filler materials, or first judge 40 blank intervals—then judge the same batch of primary materials. Each reader repeats judgment on partial materials; using the dispersion of repeated judgments (or case error terms from Rasch-type models) to back-calculate each reader’s noise scale under both load conditions, dividing the two yields the amplification factor.

    Judgment criteria are fixed in advance: if the measured amplification falls within 1.1, fatigue is negligible, and the filler rule simplifies to “as long as contrast items are controllable”; if it exceeds 1.4, the filler quantity itself must be treated as a dose variable, requiring dose-response curves; if it lies in between, current rules remain but with marked applicability boundaries.

    Evidence and Limitations

    Verified: Vigilance declines with task duration is a meta-analysis-level stable phenomenon (PMID 23163491); load and task structure alter vigilance dynamics (PMID 26844367); reading performance affected by fatigue has empirical support in imaging judgment (PMID 16636802).
    Partially Verified: Fatigue alters reading stability (time/consistency), but the “noise amplification multiplier” as a calculable quantity has never been reported in judgment scenarios.
    To Be Verified: Exact placement of the multiplier between thresholds 1.1 and 1.4; whether the multiplier varies by individual, whether it is modulated by material difficulty; and the validity of the set value 1.3 itself.

    Further Reading

    L1 Network Status: This topic has not yet entered the L1 corresponding network (EDGE_TABLE_science-yixue.md lacks edges for “fatigue/judgment noise/filler materials”; the nearest adjacent edge in CORRESPONDENCE_NETWORK.md is E-060 labeled “credibility conflict,” PMID 30383398, which pertains to TCM diagnostic label issues and is not the same edge).

    Evidence Level: C
    References: PMID 23163491, PMID 26844367, PMID 16636802, PMID 38056581

    Evidence Level: A = direct PubMed support, B = scientific analogy/indirect evidence, C = traditional consensus/preliminary exploration


    Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.

  • 「评分者越判越松」还是「及格线越划越严」?一个可能无法从数据里回答的问题

    Evidence level: C

    一句话:漂移压低合格率这个「方向结论」,可能整个建立在「切点不动」这个没人验证过的假设上——而漂移与切点移动在观测上可能根本分不开。

    正文

    前一篇已经算出一个很漂亮的方向结论:当评分者的尺度出现漂移(同一批人越判越不齐)时,合格率的期望会被单向压低——在漂移标准差 0.515、真实合格率 0.95 的设定下,期望合格率被拽低 2.18 个百分点,这个凹性缺口占到一阶项的 41.1%。结论听起来很干净:漂移是坏东西,它系统性压低你看到的合格率。

    但这个结论里藏着一个前提:切点是固定的。合格线划在那里不动,评分者的尺度在动,于是两者错位,合格率被压低。

    问题在于,另一篇文章已经证明了一条更麻烦的事:读者位置与切点组之间存在一条不含任何信息的平移共线——「读者整体变松」与「切点整体下移」在数据上完全等价,你看到的分数分布一模一样。如果把这条等价用到底,就会得到一个颠覆性的可能:漂移根本不是一个独立发生的量,它可以全部被吸收进切点的移动里。此时合格率的期望不变,凹性缺口归零,设计效应退化为 1。换句话说,那个「单向拽低」的方向,取决于研究者把漂移记在读者账上还是切点账上——而这两种记法在观测上不可分。

    这不是本领域独有的纠缠。kappa 系数的老毛病就是同一件事的经典形态:Cohen’s kappa 同时受评定者偏倚与类别流行度影响,二者无法从单一的 kappa 值里分离出来(PMID 8501467);著名的「高一致但低 kappa」悖论,可以完全由边际分布失衡造成,与真实一致程度无关(PMID 2348207)。甚至有人反过来主张流行度依赖不构成对 kappa 的实质批评(PMID 15939215)——但注意其论证前提恰是「改变流行度必然同时改变敏感度与特异度」,也就是说,反方同样默认了两本账拆不开。三方争论了几十年,共同点只有一个:没人承诺能把两种记账从同一份数据里分开。

    那这个方向判断还救得回来吗?有一条窄缝:设计效应 k 依赖读者间的方差 Var(b),而纯粹的切点移动不产生这一项。k 可能是唯一能把两种记账拆开的统计量。

    那条窄缝为什么存在?机制上可以说明白:设计效应 k 的分子里有读者间方差 Var(b)——它量的是「不同评分者的合格率彼此差多远」。纯粹的切点移动是所有人共享的一次平移:每个评分者相对别人的位置纹丝不动,Var(b) 一点不变;而读者漂移是各人往不同方向散开,Var(b) 必然被撑大。所以在「分数分布相同」这一约束下,两种记账若要维持 k 一致,就必须在 Var(b) 上做不一样的文章——这正是唯一可能留下指纹的地方。同理,若观察到 k 显著大于 1 而分数分布与「纯切点移动」版本无法区分,就可以把「漂移属读者侧」从假设升级为有观测内容的判断。

    可做的实测很直接:模拟三个模型——① 只有读者漂移(切点固定);② 只有切点移动(读者固定);③ 两者按不同比例混合但总效应相同——让三者在完全相同的观测分数分布下生成数据,然后比较三个读数:期望合格率 m̂、设计效应 k、通过率。

    判据写死:若三者在 m̂、k、通过率上全部一致(差 <0.5 个百分点),则「漂移压低合格率」这个方向断言不可证伪,必须从结论里撤回,改报「在该等价类下合格率的期望不变」——这是一种诚实的降级。若只有 k 能区分三者的记账方式,则 k 必须上升为必报项:以后任何报合格率的研究,不同时报 k 就等于没交代漂移记在哪本账上。若三者可分、且读者漂移版的缺口显著大于切点版(差 >1 个百分点),则原方向结论成立——但必须显式写出它的前提:漂移确属读者侧,这个前提本身需要独立证据,不能靠数据反推。

    对易学判读研究的含义很实际:五脏辨证、脉象分级这类多评定者判读,若发现「合格率随年份下降」,先别急着归因于「传承在退化」——先问一句:是判读者的尺度散了,还是验收线被悄悄挪了?在拿出 k 之前,这两种解释的分量是一样的。

    证据与局限

    已验证:三个 PMID 确立了 kappa 的偏倚-流行度等价类是真实的、且无法从单一统计量分离(PMID 8501467, 2348207, 15939215)。
    部分验证:平移共线本身来自同系列前篇的推导(#1420),尚未对真实判读数据做过直接演示。
    待验证:三模型模拟的全部判据(三读数是否可分、k 是否唯一拆分统计量、缺口差是否 >1pp);「漂移记在读者账上」这一前提目前没有任何独立证据。

    相关阅读

    证据等级: C
    参考文献: PMID 8501467, PMID 2348207, PMID 15939215

    L1 网络状态:该主题尚未进入 L1 对应网络。逐词检索(漂移/切点/合格率/等价类/kappa/信度/评定者)未见对应边;最接近的是 E-060 标签前提(conflict,未裁决):「中医医师间诊断一致性在开放人群中『低』,作者称为『令人担忧』(KB-292,PMID 30383398);偶然校正一致性统计常被误用、建议 Gwet AC2(KB-317/333/334)」——该冲突链只覆盖「标签信度低」这一前提,不含漂移方向与切点等价问题。


    免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。

    Is the grading curve becoming more lenient or is the passing threshold becoming stricter? A question that may not be answerable from the data.

    Evidence level: C

    In one sentence: The directional conclusion that “drift lowers the pass rate” may rest entirely on the unverified assumption of a “fixed cut-off”—while drift and cut-off movement may be observationally inseparable.

    Body

    The previous article derived a clean directional conclusion: when scoring scales drift (the same group of people becomes increasingly inconsistent), the expected pass rate is unidirectionally depressed. Under settings of drift standard deviation 0.515 and true pass rate 0.95, the expected pass rate is pulled down by 2.18 percentage points; this concave gap accounts for 41.1% of the first-order term. The conclusion sounds neat: drift is bad; it systematically depresses the pass rate you observe.

    But this conclusion hides a premise: the cut-off is fixed. The passing line is drawn and stays put, while the scoring scales move, creating misalignment and depressing the pass rate.

    The problem is that another article proved a more troublesome fact: there exists a translation collinearity containing no information between reader positions and the cut-off group—”the whole reader set becomes looser” is data-equivalent to “the cut-off shifts downward.” If this equivalence holds, it yields a disruptive possibility: drift is not an independent quantity; it can be entirely absorbed into the movement of the cut-off. In this case, the expected pass rate remains unchanged, the concave gap vanishes, and the design effect degenerates to 1. In other words, that “unidirectional pull-down” direction depends on whether the researcher attributes drift to the readers or the cut-off—and these two attributions are observationally inseparable.

    This entanglement is not unique to this field. The classic trouble of kappa coefficients is the canonical form of the same issue: Cohen’s kappa is simultaneously affected by rater bias and category prevalence, and neither can be separated from the single kappa value (PMID 8501467); the famous “high agreement but low kappa” paradox can be entirely explained by marginal distribution imbalance, regardless of true agreement (PMID 2348207). Even some argue that prevalence dependence does not constitute a substantive critique of kappa (PMID 15939215)—note that their argument premise is precisely “changing prevalence necessarily changes sensitivity and specificity,” meaning the opposing side also defaults to the two accounts being inseparable. Three parties have debated for decades, with one common point: no one promises to separate the two bookkeeping methods from the same data.

    So can this directional judgment be salvaged? There is a narrow gap: the design effect $k$ depends on the variance between readers, $\text{Var}(b)$, whereas pure cut-off movement does not produce this term. $k$ is the only statistic that can separate the two bookkeeping methods.

    Why does this narrow gap exist? Mechanistically, it can be explained: the numerator of the design effect $k$ contains $\text{Var}(b)$, which measures “how far apart the pass rates of different raters are.” Pure cut-off movement is a shared translation for everyone: each rater’s position relative to others remains unchanged, so $\text{Var}(b)$ does not change; reader drift scatters individuals in different directions, inevitably inflating $\text{Var}(b)$. Under the constraint of “identical score distributions,” if both bookkeeping methods are to maintain consistent $k$, they must differ in $\text{Var}(b)$—this is the only place a fingerprint could leave a trace. Similarly, if $k$ is observed to be significantly greater than 1 while the score distribution cannot be distinguished from the “pure cut-off movement” version, then the assumption that “drift belongs to the reader side” can be upgraded into an observationally grounded judgment.

    The practical test is straightforward: simulate three models—① only reader drift (fixed cut-off); ② only cut-off movement (fixed readers); ③ both mixed in different proportions but with identical total effect—generating data under completely identical observed score distributions, then comparing the three readings: expected pass rate $\hat{m}$, design effect $k$, and pass rate.

    The criteria are fixed: if all three are identical in $\hat{m}$, $k$, and pass rate (difference < 0.5 percentage points), then the directional assertion “drift lowers the pass rate” is unrefutable and must be withdrawn from the conclusion, replaced by “under this equivalence class, the expected pass rate remains unchanged”—this is an honest downgrade. If only $k$ can distinguish the three bookkeeping methods, then $k$ must become a mandatory reporting item: any future study reporting pass rates without $k$ fails to disclose where drift is attributed. If all three are separable, and the gap in the reader-drift version significantly exceeds that of the cut-off version (difference > 1 percentage point), then the original directional conclusion holds—but it must explicitly state its premise: drift indeed belongs to the reader side, a premise requiring independent evidence, not inferred from the data**.

    The implications for Yijing Studies are practical: multi-rater assessments such as five-organ syndrome differentiation or pulse diagnosis grading, if they show “pass rates declining over years,” should not immediately attribute this to “degenerating transmission.” First ask: is the scale of the raters scattering, or is the acceptance line being quietly shifted? Until $k$ is reported, the weight of these two explanations is identical.

    Evidence and Limitations

    Verified: Three PMIDs establish that the kappa bias-prevalence equivalence class is real and cannot be separated by a single statistic (PMID 8501467, PMID 2348207, PMID 15939215).
    Partially verified: The translation collinearity itself derives from derivations in the preceding articles of the same series (#1420); it has not yet been directly demonstrated on real assessment data.
    To be verified: All criteria of the three-model simulation (whether the three readings are separable, whether $k$ is the unique splitting statistic, whether the gap difference exceeds 1pp); the premise “drift attributed to the reader side” currently has no independent evidence.

    Related Reading

    Evidence level: C
    References: PMID 8501467, PMID 2348207, PMID 15939215

    L1 network status: This topic has not yet entered the L1 corresponding network. Keyword search (drift/cut-off/pass rate/equivalence class/kappa/reliability/rater) yields no corresponding edges; the closest is the E-060 label premise (conflict, unresolved): “Diagnostic consistency among TCM physicians is ‘low’ in the open population, termed ‘worrying’ by the author (KB-292, PMID 30383398); occasional correction consistency statistics are often misused; Gwet AC2 is recommended (KB-317/333/334)”—this conflict chain covers only the premise of “low label reliability,” not the drift direction and cut-off equivalence issue.


    Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.

  • 记忆不是均匀的:置信度门控下,ICC 的抵消点会往哪边走

    Evidence level: C

    判读者「记得上一遍答案」这件事若只在中低置信区生效,ICC 抵消点将整体左移——所有卡在门限附近的一致性读数都得重新解释。

    正文

    先摆清楚这盘棋。本系列前篇把判读一致性系数(ICC)的变动拆成两个反向机制:记忆效应(判读者按固定权重 λ 把观测值往上一轮的答案靠,把 ICC 的分子抬高)和漂移求稳(判读者系统性往中间收,把变异压小、也把 ICC 拉平)。两者相抵的那个漂移量叫「抵消点」——前篇在线性记忆假设下算出:ICC 锁在 0.18 时,抵消所需的漂移跨度从 0.075 到 0.503。这张等值线图后来成了本组设计判读实验的参照。

    但整张图立在一个从未被检验的设定上:记忆是线性平滑的——每一轮观测值按固定权重往上一轮靠,不管这轮答案离中心多远、判读者多有把握。真实判读不是这样。放射科医师重复判读的经典实测(PMID 22109344)给出一个关键结构:记忆效应主要集中在低置信区——24 位医师的实验里,最自信那一档(definitely included)的回忆数反而接近随机分配,也就是说判读者不是「均匀地记得」,而是「在某些条件下才调用上一遍的答案」。

    这就引出一个可计算的问题:若把记忆模型从线性平滑换成置信度门控——记忆只在偏离中心小于某阈值时生效,阈值与生效强度是两个独立参数——前篇那张等值线会发生什么?

    方向上的推演并不含糊。门控记忆意味着:只有中低置信区的判读会被上一轮「拽住」,ICC 分子被抬高的幅度比线性模型小得多;而漂移项的压制作用不挑区域、原样保留。两边一消一长,抵消点会往更小的漂移值移动。用大白话说:线性模型说「要 0.503 的漂移才能压平 ICC」,门控模型可能只要 0.3 甚至更小——因为记忆这边的力气本来就只有一半在干活。

    这对实践是致命的。如果门控版抵消点真比线性版低 30% 以上,那么前篇的 0.503 就是乐观上界:真实人群里更小的漂移就已把 ICC 压平,凡卡在门限附近的 ICC 读数都应默认解释为「两机制已抵消」,而不是「判读真的稳」。反过来,若门控版抵消点反而升高,线性模型就是保守设定,原结论可直接引用。

    还有一个更麻烦的第三种可能:抵消点随门控强度非单调——先升后降。若真如此,「记忆的门控结构」本身就成了第一个必须实测的量,单凭 ICC 观测永远定不了机制,任何等值线图都只是某个假设下的条件结论。

    用一个数字直觉把两版模型的差别摆到桌面上。线性记忆下,λ=0.6 时每个判读值有六成「重量」来自上一轮,ICC 分子被均匀抬高;门控版若记忆只在离中心一个噪声单位内生效,而真实材料里恰好六成判读落在这一带,则门控版的等效 λ 约为 0.36——记忆力气只剩六成。抵消点自然要按六成力气重算:线性版说 0.503 的漂移才能压平,门控版可能 0.35 上下就够了。反过来,若材料陡峭、九成判读都挤在紧邻中心的那一带,门控版的等效 λ 反而逼近线性值——门控效应的大小,取决于材料难度梯度的形状。这就是为什么敏感性分析必须把「阈值 × 材料分布形状」一起扫,只扫阈值会得出片面结论。

    怎么裁决?三步:① 把记忆模型换成门控形式,在与前篇相同的 λ 等效水平上重跑整张网格,逐档比较等值线位置与抵消点;② 做敏感性分析,看门控参数(阈值、强度)在合理区间内变动时抵消点移动多少——移动小于 10% 说明结论对门控结构不敏感,可以安心用线性近似;③ 用 PMID 31329615 给出的 ICC 分解框架核对:门控记忆下绝对一致与一致性 ICC 的差值会呈特定形态,这个差值正是「偏置是否存在」的现成判据,也是门控机制留下的可观测指纹。此外,PMID 11091941 提醒连续测量仪器的重复性数据同样受「数据非均匀来源」影响,等值线结论外推到仪器数据时须逐项声明假设。

    一句话收束:等值线不是地图,是某个记忆假设的投影。 记忆按置信度门控这个结构一日不实测,门限附近的 ICC 读数就一日要挂「假设待验」的牌子。

    证据与局限

    已验证: 判读存在记忆效应且按置信度分层(PMID 22109344 实测:记忆集中于低置信区);ICC 的统计分解与偏置判据(PMID 31329615)。
    部分验证: 线性记忆模型下 ICC 抵消点的数值结构(前篇计算,未经实测);门控记忆对抵消点方向的定性推演(逻辑自洽但无数据)。
    待验证: 门控记忆下抵消点的具体位置与线性版相差多少;抵消点对门控参数的敏感性;非单调情形是否存在。

    相关阅读

    L1 网络状态:该主题(置信度门控记忆 × ICC 抵消点)尚未进入 L1 对应网络;最接近的 E-060「标签前提」边处于 conflict 状态(中医辨证标签一致性低、加权系数精度区间错配),非同一条边。

    证据等级: C
    参考文献: PMID 22109344, PMID 31329615, PMID 11091941

    证据等级:A=有PubMed直接支撑,B=科学类比/间接证据,C=传统文化共识/初步探索


    免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。

    Memory is not uniform: the ICC cancellation point shifts under confidence gating

    Evidence level: C

    If the reader’s “remembering the previous answer” effect only operates within the low-to-mid confidence zone, the ICC cancellation point shifts leftward overall—requiring reinterpretation of all consistency readings hovering near the threshold.

    Body

    Let’s lay out the board clearly. The previous article decomposed variations in the Interpretation Consistency Coefficient (ICC) into two opposing mechanisms: memory effect (the reader applies a fixed weight λ, pulling current observations toward the previous answer, thereby inflating the ICC numerator) and drift-to-stability (the reader systematically pulls toward the center, suppressing variance and flattening the ICC). The net drift required to balance these two forces is called the cancellation point. Under the linear memory assumption, this cancellation point was calculated as ranging from 0.075 to 0.503 when the ICC locks at 0.18. This contour map later served as the reference for designing interpretation experiments in this series.

    However, this entire map rests on an untested assumption: memory is linearly smoothed—each round of observation is pulled toward the previous answer by a fixed weight, regardless of how far the current answer lies from the center or how confident the reader is. Real-world interpretation does not work this way. Classic empirical studies of radiologists re-reading cases (PMID 22109344) reveal a key structural feature: the memory effect is concentrated in the low confidence zone—in an experiment with 24 physicians, the recall rate for the most confident category (“definitely included”) was nearly random, indicating readers do not “remember uniformly,” but rather “invoke the previous answer only under certain conditions.”

    This raises a calculable question: if we replace the linear memory model with confidence-gated memory—where recall only applies when deviation from the center is below a certain threshold, with the threshold and activation strength being independent parameters—what happens to the previous contour map?

    The directional deduction is unambiguous. Confidence-gated memory means only interpretations in the mid-to-low confidence zone are “held back” by the previous round; the ICC numerator is inflated far less than under the linear model, while the drift term’s suppressing effect remains unchanged across all regions. The net result is that the cancellation point shifts toward smaller drift values. In plain terms: the linear model predicted “a drift of 0.503 is needed to flatten the ICC,” whereas the gated model may require only 0.3 or even less—because half of the memory force is effectively inactive.

    This has critical practical implications. If the gated cancellation point is indeed more than 30% lower than the linear one, then the previous 0.503 becomes an optimistic upper bound: in real populations, smaller drifts already flatten the ICC, and any ICC readings near the threshold should be interpreted as “the two mechanisms have canceled out,” not “interpretation is truly stable.” Conversely, if the gated cancellation point rises, the linear model represents a conservative setting, and its conclusions can be directly applied.

    There is also a more problematic third possibility: the cancellation point varies non-monotonically with gated strength—rising then falling. If true, the “gated structure of memory” itself becomes the first parameter requiring empirical measurement; ICC observations alone can never determine the mechanism, and any contour map is merely a conditional conclusion under specific assumptions.

    To make the difference between the two models concrete, consider a numerical intuition. Under linear memory with λ = 0.6, each interpretation draws 60% of its “weight” from the previous round, uniformly inflating the ICC numerator. In the gated version, suppose memory only activates within one noise unit of the center, and exactly 60% of real materials fall in this zone; then the equivalent λ for the gated model is approximately 0.36—memory’s force is only 60% effective. Naturally, the cancellation point must be recalculated based on this reduced force: where the linear model required 0.503 drift, the gated model might suffice with around 0.35. Conversely, if the material is steep and 90% of interpretations cluster tightly near the center, the gated equivalent λ approaches the linear value—the magnitude of the gated effect depends on the shape of the material difficulty gradient. This is why sensitivity analysis must jointly scan “threshold × material distribution shape”; scanning only the threshold yields partial conclusions.

    How do we adjudicate? Three steps: ① Replace the memory model with the gated form and re-run the entire grid under the same λ-equivalent level, comparing contour positions and cancellation points step-by-step; ② Conduct sensitivity analysis to see how much the cancellation point shifts as gated parameters (threshold, strength) vary within plausible ranges—a shift under 10% indicates the conclusion is insensitive to gated structure, justifying use of the linear approximation; ③ Verify against the ICC decomposition framework provided in PMID 31329615: under gated memory, the difference between absolute consistency and consistency ICC takes a specific form, serving as a ready indicator of “whether bias exists,” and constituting the observable fingerprint left by the gated mechanism. Additionally, PMID 11091941 reminds us that repeatability data from continuous measurement instruments is also affected by “non-uniform data sources,” so when extrapolating contour conclusions to instrument data, each assumption must be explicitly stated.

    In summary: The contour map is not a map; it is a projection of a memory assumption. Until the gated structure of confidence is empirically tested, ICC readings near the threshold must bear the label “hypothesis pending verification.”

    Evidence and Limitations

    Verified: Memory effect exists in interpretation and is stratified by confidence level (PMID 22109344 experimental data: memory concentrated in low confidence zone); statistical decomposition of ICC and bias criterion (PMID 31329615).
    Partially Verified: Numerical structure of ICC cancellation point under linear memory model (calculated previously, not yet tested experimentally); qualitative deduction of gated memory’s effect on cancellation point direction (logically consistent but lacks data).
    Pending Verification: Exact position of cancellation point under gated memory and its difference from the linear version; sensitivity of cancellation point to gated parameters; existence of non-monotonic cases.

    Related Reading

    L1 Network Status: This topic (confidence-gated memory × ICC cancellation point) has not yet entered the corresponding L1 network; the nearest E-060 “label premise” edge is in conflict state (low consistency of TCM syndrome differentiation labels, mismatched precision intervals of weighting coefficients), not the same edge.

    Evidence Level: C
    References: PMID 22109344, PMID 31329615, PMID 11091941

    Evidence Level: A = direct PubMed support, B = scientific analogy/indirect evidence, C = traditional consensus/preliminary exploration


    Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.

  • 置信度低于阈值才融合?门槛筛出来的样本,四分之三本来就答对了

    Evidence level: B

    池③ 里有一张卡,思路是这样:单模型判证候,置信度低于阈值时,触发 CNN/RNN 多模型加权平均,提升整体预测质量。

    这句话听起来无懈可击。但把门槛装上、把模拟跑完,会看到两件反直觉的事:门槛筛出的那批样本,四分之三本来就判对了;而真正需要融合的那批,只有四分之一落在门里。

    一、先把这张卡的做法写成可算的形式

    原设计(#75)有两个部件,各自选了一个变量:

    部件 作用 原设计用的变量
    门(gate) 决定”哪些样本要动用多模型” 单模型置信度 p̂ < τ(阈值 0.60)
    权(weight) 决定”动用时各模型话语权” 各模型自己的置信度,w ∝ p̂

    两个部件都指向同一个量:p̂。这一点后面会变成本文最硬的一条。

    模拟设置(全部本库自行执行,百万级样本):五个证候类,单模型准确率 0.90,三个融合模型之间错误相关性 ρ_e 取 0.5–0.9(脉象仪这类同一传感链、同一套谐波分解、同一批标注的模型,错误高度相关是常态而非例外)。ρ_e 的定义是”给定样本会错时,三个模型一起错的概率”。

    二、第一个数字:融合到底值多少

    先把”多模型融合”的总收益摆在桌上,单位是百分点(pp):

    ρ_e 单模型 等权 K=3 加权 K=3 等权增益 加权增益
    0.50 0.9249 0.9298 0.9382 +0.49 pp +1.33 pp
    0.70 0.9212 0.9247 0.9274 +0.35 pp +0.62 pp
    0.80 0.9159 0.9185 0.9191 +0.26 pp +0.32 pp
    0.90 0.9092 0.9097 0.9103 +0.06 pp +0.12 pp

    读法:ρ_e = 0.8 时,三个模型融合把准确率从 91.6% 抬到 91.9%。加权比等权多 0.06 pp。

    0.06 个百分点,在 n=200 的测试集上是 0.12 个样本。这不是”误差范围内”的问题,而是这个数比随机种子引起的波动还小。也就是说:#75 里”加权”这个动作,本身没有可测的收益。

    原因在第三节。

    三、加权为什么白做:门和权用了同一个数

    这是本轮唯一的新判断,写成一句可执行的话:

    同一个统计量,不得同时用于"准入"和"加权"。
    用在准入上,它决定"谁进来";用在加权上,它决定"进来后谁说话大声"。
    两处用同一个带噪量 ⇒ 同一份噪声被用了两次,且方向一致(p̂ 低者既被选中、又被降权)。
    于是子集内的权重差异,测的是"被选中的程度",不是"模型能力的差异"。
    

    实测:在 ρ_e = 0.8、K = 3 下,加权相对等权的输出改动量(RMS)是 0.0314,而融合相对单模型的收益量(RMS)是 0.3652——加权的全部动作只相当于融合收益的 8.6%,其余 91.4% 是把三个模型等权平均这件事本身给的。

    而这份 8.6% 的改动,注入的还是带噪置信度。模拟里权重偏离等权的 L1 距离是 0.096(ρ_e 噪声 0.05)到 0.141(噪声 0.20)——权重确实在动,动的方向来自置信度的随机起伏,不是能力。

    再补一刀:在触发子集内,p̂ 的平均值被系统性压低(E[p̂]=0.359 而 E[p] = 0.476)。因为门就是”p̂ 低”筛出来的,子集内的 p̂ 自然整体偏低。 这时候再用 p̂ 加权,等于在一个已经按 p̂ 排序过的子集里,再按 p̂ 排一次序。

    四、第二个数字:门槛筛出来的是谁

    这是本文最该记住的一张表。同一批数据,三种门,ρ_e = 0.8:

    门 触发率 门内单模型准确率 门内融合后 门内净增益
    置信度低 p̂ < 0.60 26.1% 0.7453 0.7538 +0.84 pp
    三模型不全一致 8.5% 0.0672 0.0965 +2.93 pp
    有模型偏离多数票 8.5% 0.0672 0.0965 +2.93 pp

    置信度门的门内准确率是 0.755——也就是说,被门槛判定为”需要救助”的样本里,四分之三单模型本来就答对了。

    而那些真正判错的样本(门内准确率 0.067,即 93% 是错的),只有 8.5% 落在分歧门上。

    更狠的是两个门的重合度:Jaccard = 0.226。置信度门里只有 24.4% 被分歧门覆盖。 两个门抓的是两批不同的人。

    五、为什么两个门不重合:低置信度不等于难样本

    机制很直白。

    分歧门测的是”三个模型给出了三个不同答案”——这件事只有在模型真的处理不了这个样本时才会发生。所以门内准确率 0.067,几乎是纯错误集合。

    置信度门测的是”单模型自己的 p̂ 低”。但 p̂ 有两个来源的波动:① 样本真的难;② 模型自己的随机性(树的切分、自助采样、特征子采样)。当 ρ_e 高的时候,三个模型在同一批样本上一起错、或者一起在同一个地方犹豫——这时候”三个模型一致地说’我不确定’”经常发生在一个它们其实答对了的样本上。

    于是:门内 74.5% 的样本,是”模型谦虚但实际上答对了”。融合处理它们的动作,是把已经正确的答案再确认一遍——这不是纠错,是重复确认。

    六、K 的天花板:加模型不是路

    等权融合的方差增益有解析式:

    增益(K) = K / [1 + (K-1)·ρ_e]        上限 = 1/ρ_e  (K → ∞)
    
    ρ_e 天花板(方差) K=3 取到 K=10 取到
    0.60 1.667× 81.8% 93.8%
    0.80 1.250× 92.3% 97.6%
    0.90 1.111× 96.4% 98.9%

    ρ_e = 0.8 时,三个模型已经吃掉了天花板的 92.3%;从 3 个加到 10 个,只多拿 5.3 个百分点。对应的准确率:K=3 是 0.9181,K=10 是 0.9198,+0.17 pp,而模型数翻了 3.3 倍。

    加模型不是提升路径。降 ρ_e 才是。 而 ρ_e 高恰恰是”同一台仪器、同一套特征、同一批标注”的必然结果——这正好是池③ 反复撞到的那堵墙(参见 #76 的跨设备迁移与分辨率下限)。

    七、那正确的结构长什么样

    把上面的数拼起来,可执行的版本是这样:

    门 = 分歧(三模型预测不一致)      触发率 8.5%,门内准确率 0.067
    权 = 等权                          不用 p̂ 加权
    K  = 3                             已吃掉 92.3% 天花板,不必再加
    

    它的总额外收益:8.5% × 2.93 pp ≈ 0.25 pp,与全域等权融合的 0.25 pp 几乎完全相同——但只动了 8.5% 的样本,计算量与调用次数降一个数量级。

    这就是本文能给出的最有用的东西:不是”要不要融合”,而是”融合只在分歧时触发,且触发后不要按置信度加权”。

    八、证据与局限(必读)

    证据等级 B,不是 A。 理由:所引 6 条 PMID 全部真实、全部经 NCBI E-utilities 逐条比对标题与作者回查通过(见下方核验表),但无一条直接研究”脉诊多模型融合的门控变量该用置信度还是分歧”。文献提供的是各组件(概率校准、弃权判据、集成多样性、门控混合专家)的一般性支撑,本文的拼接与全部数值是本库自行推导与模拟的结果。

    四条硬局限,不得省略:

    1. 全部是模拟,没有一次实测。 每个 pp 差值都是”在设定结构下必然出现的量级”,不是”实测提升”。真实脉象仪数据的 ρ_e、p̂ 的校准程度、五个证候的真实可分性,一个都还没测。
    2. ρ_e 是被假设的,不是被估计的。 本文横扫 0.5–0.9 是为了说明”结论对 ρ_e 的依赖形状”,不是因为知道脉象仪模型的 ρ_e 是多少。若实测 ρ_e = 0.5,加权确实有 +1.33pp(比等权多 0.84pp),本文”加权白做”的结论在这一区就弱化了。
    3. 错误结构是人为设定。 模拟里”会错的样本”是随机分配的,真实数据里难样本是聚集的(某些证候、某些年龄段、某些体型)。聚集会让分歧门的触发率与门内准确率都改变。
    4. p̂ 假定为”已校准”。 若 p̂ 本身严重失校准(这在随机森林上很常见,见 Mehrtash 2020、Murugesan 2023),则”置信度门”抓到的甚至不是”模型不确定”,而是”校准曲线在那一端的斜率”。这一层本文没有拆开算。

    不许做的推论(三条):

    • 不许把这篇文章读成”多模型融合没用”。它读作”在 ρ_e 高的情况下融合的边际很小,且加权几乎为零“。
    • 不许把这篇文章读成”置信度不可靠”。置信度作为报告不确定性的量仍然必要,问题只出在把它同时用于准入与加权。
    • 不许把 0.25 pp / 0.06 pp 当成”实测结论”引用。它们是假设结构下的量级推演。

    健康相关声明:本文涉及中医证候的机器判定,属高风险主题。任何模型输出都不构成诊断、不构成治疗建议、不得替代执业医师判断。中医证候的判定标准本身在学界尚未统一(这是池③ 长期未解的 #78 号课题),在这个前提下讨论”融合提升 0.25 pp”必须同时说明:被提升的那个”准确率”,其标签本身的可信度还没有被确立。

    九、引文核验(6 条,全部 2026-11-02 回查)

    编号 PMID 真值(比对标题 + 作者,不只验号存在) 用途
    A1 32746129 Mehrtash A, Wells WM, Tempany CM, Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation, IEEE Trans Med Imaging 2021;39:3868-3878, doi:10.1109/TMI.2020.3006437 置信度不等于正确率(校准层)
    A2 37146441 Murugesan B, Liu B, Galdran A, Calibrating segmentation networks with margin-based label smoothing, Med Image Anal 2023;87:102826 置信度需要显式校准(局限 4 的依据)
    A3 33571099 Perales-Gonzalez C, Fernandez-Navarro F, Carbonero-Ruz M, Global Negative Correlation Learning: A Unified Framework for Global Optimization of Ensemble Models, IEEE Trans Neural Netw Learn Syst 2022;33:4031-4042 集成增益取决于成员间负相关/多样性(第六节依据)
    A4 19171521 Soda P, Iannello G, Aggregation of classifiers for staining pattern recognition in antinuclear autoantibodies analysis, IEEE Trans Inf Technol Biomed 2009;13:322-329 医学分类器聚合的早期实证(融合确实有效,但未分离门控变量)
    A5 38322152 Hashimoto N, Hanada H, Miyoshi H, Multimodal Gated Mixture of Experts Using Whole Slide Image and Flow Cytometry for Multiple Instance Learning Classification of Lymphoma, J Pathol Inform 2024;15:100359 门控混合专家在医学分类的实际用法(对照)
    A6 38269963 Millar AS, Arnn J, Himes S, Uncertainty in Breast Cancer Risk Prediction: A Conformal Prediction Study of Race Stratification, Stud Health Technol Inform 2024;310:991-995 以”覆盖-精度”替代单点阈值报告不确定性的做法

    按 R11(零参数判据只可证伪、不可验证):本文的”分歧门优于置信度门”是在同一模拟结构下的内部比较,它证伪了”置信度门是最佳门控变量”这一默认假设,但不能反过来证明”分歧门是正确门控”——换一个错误结构(难样本聚集、三模型异构程度不同),排序可能翻转。这条界限必须与结论一起读。

    按 R18(断言充分性分层报):本文的 0.25 pp / 0.06 pp 全部建立在”ρ_e 已知且恒定”之上;而 ρ_e 恰恰是唯一没有实测的量。结论的精度远高于输入的精度——这正是池④ 立的 R26 说的那个病,本次在自己的推理上复发了一次,如实记下。


    本文是 draft,未标 accepted,未进 779,未发布。配套实验设计:08-ops/experiments/pool3-confidence-gate-vs-disagreement-gate-fusion.md(P3-EXP-060)。


    免责声明

    本文属传统知识与现代统计方法的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,也不构成诊断依据,不能替代执业医师的判断与诊疗。文中出现的准确率、百分点差值、触发率等全部数值,均为在文内明示的模拟假设结构下推得的量级,不是实测提升,更不代表任何脉象仪或模型输出具有诊断效力。个体情况差异很大;如有身体不适或健康疑虑,请及时就医并由专业医师判断。


    免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。

    Only fuse when confidence is below the threshold? Samples filtered by the threshold already got it right three out of four.

    Evidence level: B

    Pool ③ contains a card with the following rationale: the single model classifies the condition; if confidence falls below the threshold, a CNN/RNN multi-model weighted average is triggered to enhance overall prediction quality.

    This sounds flawless. However, once the threshold is implemented and the simulation runs, two counterintuitive facts emerge: three-quarters of the samples filtered out by the threshold were already correctly classified; conversely, only one-quarter of those truly requiring fusion fell within the threshold.

    I. First, Express the Card’s Procedure in Calculable Form

    The original design (#75) comprises two components, each selecting a variable:

    Component Function Variable Used in Original Design
    Gate (门) Determines “which samples require the multi-model approach” Single-model confidence p̂ < τ (threshold 0.60)
    Weight (权) Determines “the relative influence of each model when activated” Each model’s own confidence, w ∝ p̂

    Both components point to the same quantity: p̂. This convergence will become the most critical constraint of this paper.

    Simulation Setup (all parameters executed within this repository, using million-scale samples): five syndrome categories; single-model accuracy 0.90; the error correlation ρ_e among the three fusion models ranges from 0.5 to 0.9. (Note: For models sharing the same sensor chain, harmonic decomposition method, and annotation batch—such as pulse waveform analyzers—high error correlation is the norm rather than the exception.) ρ_e is defined as “the probability that all three models err simultaneously given a sample error.”

    II. The First Number: How Much Does Fusion Bottom Out to?

    First, place the total gain of “multi-model fusion” on the table, measured in percentage points (pp):

    ρ_e Single Model Equal-Weight K=3 Weighted K=3 Equal-Weight Gain Weighted Gain
    0.50 0.9249 0.9298 0.9382 +0.49 pp +1.33 pp
    0.70 0.9212 0.9247 0.9274 +0.35 pp +0.62 pp
    0.80 0.9159 0.9185 0.9191 +0.26 pp +0.32 pp
    0.90 0.9092 0.9097 0.9103 +0.06 pp +0.12 pp

    Reading: when ρ_e = 0.8, fusing the three models lifts accuracy from 91.6% to 91.9%. Weighted fusion gains 0.06 pp over equal-weighting.

    A gain of 0.06 percentage points corresponds to 0.12 samples on an n=200 test set. This is not a matter of “within error bounds”; rather, this magnitude is smaller than the fluctuation caused by random seed variation. Consequently: the “weighting” operation in #75 itself yields no measurable gain.

    The reason lies in Section III.

    III. Why weighting is done in vain: Gate and Weight use the same number

    This is the only new judgment for this round, stated as an executable sentence:

    The same statistic must not be used simultaneously for "admission" and "weighting."
    When used for admission, it determines "who enters"; when used for weighting, it determines "who speaks louder after entering."
    Using the same noisy quantity in both places ⇒ the same noise is applied twice, and in the same direction (low p̂ is both selected and down-weighted).
    Thus, within the subset, differences in weight measure "degree of selection," not "differences in model capability."
    

    Empirical test: Under ρ_e = 0.8 and K = 3, the output change due to weighting relative to equal weighting (RMS) is 0.0314, whereas the gain from fusion relative to single models (RMS) is 0.3652—the entire effect of weighting accounts for only 8.6% of the fusion gain, with the remaining 91.4% arising simply from averaging the three models equally.

    Furthermore, this 8.6% adjustment is injected via noisy confidence. In the simulation, the L1 distance of weights from equal weighting ranges from 0.096 (ρ_e noise 0.05) to 0.141 (noise 0.20)—weights do indeed shift, but the direction of shift stems from random fluctuations in confidence, not capability.

    To add one more point: within the triggered subset, the average of p̂ is systematically depressed (E[p̂]=0.359 versus E[p] = 0.476). Because the gate selects on “low p̂,” the p̂ values within the subset are naturally overall lower. Using p̂ for weighting then amounts to re-sorting an already p̂-sorted subset by p̂.

    IV. The Second Number: Who the Threshold Filters Out

    This is the most critical table to remember in this paper. Using the same dataset with three gates and ρ_e = 0.8:

    Gate Trigger Rate Single Model Accuracy Within Gate Accuracy After Fusion Within Gate Net Gain Within Gate
    Low confidence p̂ < 0.60 26.1% 0.7453 0.7538 +0.84 pp
    Three models not fully consistent 8.5% 0.0672 0.0965 +2.93 pp
    A model deviates from majority vote 8.5% 0.0672 0.0965 +2.93 pp

    The accuracy within the confidence gate is 0.755—meaning three-quarters of the samples flagged by the threshold as “requiring assistance” were already correctly answered by the single model.

    Those truly misclassified samples (gate accuracy 0.067, i.e., 93% were wrong) fall into the disagreement gate only 8.5% of the time.

    More striking is the overlap between the two gates: Jaccard = 0.226. Only 24.4% of the confidence gate samples are covered by the disagreement gate. The two gates capture entirely different populations.

    V. Why Two Gates Do Not Overlap: Low Confidence Does Not Mean Difficult Samples

    The mechanism is straightforward.

    Divergence Gate measures “three models yielded three different answers”—an event that occurs only when the models truly cannot handle this sample. Thus, the accuracy inside this gate is 0.067, almost a pure error set.

    Confidence Gate measures “a single model’s own $\hat{p}$ is low.” Yet $\hat{p}$ fluctuates from two sources: ① the sample is genuinely difficult; ② the model’s own randomness (tree splits, bootstrap sampling, feature subsampling). When $\rho_e$ is high, the three models err together on the same batch of samples or hesitate together at the same spot—in such cases, “three models consistently say ‘I am unsure’” often happens on a sample they actually answered correctly.

    Therefore: the 74.5% of samples inside this gate are “models being humble but actually correct.” Processing them involves re-confirming answers that were already right—this is not error correction, it is redundant confirmation.

    VI. K’s Ceiling: Adding Models Is Not the Path

    The variance gain from equal-weight fusion has an analytical form:

    Gain(K) = K / [1 + (K-1)·ρ_e]        Upper bound = 1/ρ_e  (as K → ∞)
    
    ρ_e Ceiling (variance) Achieved at K=3 Achieved at K=10
    0.60 1.667× 81.8% 93.8%
    0.80 1.250× 92.3% 97.6%
    0.90 1.111× 96.4% 98.9%

    When ρ_e = 0.8, three models already consume 92.3% of the ceiling; increasing from 3 to 10 yields only an additional 5.3 percentage points. Corresponding accuracies: K=3 is 0.9181, K=10 is 0.9198, +0.17 pp, while the number of models increases 3.3-fold.

    Adding models is not an improvement path; reducing ρ_e is. A high ρ_e is precisely the inevitable result of “the same instrument, the same feature set, and the same batch of labels”—this is exactly the wall repeatedly encountered in Pool ③ (see #76 on cross-device transfer and resolution limits).

    VII. What Does the Correct Structure Look Like

    Assembling the numbers above yields this executable version:

    门 = 分歧(三模型预测不一致)      Trigger rate 8.5%, accuracy inside the gate 0.067
    权 = 等权                          Equal weighting; do not use p̂ weighting
    K  = 3                             Already consumes 92.3% of the ceiling; no further increase needed
    

    Its total extra gain: 8.5% × 2.93 pp ≈ 0.25 pp, which is almost identical to the 0.25 pp obtained via global equal-weight fusion—yet it engages only 8.5% of the samples, reducing computational load and call count by an order of magnitude.

    This is the most useful contribution of this paper: not “whether to fuse,” but “fuse only when disagreement occurs, and once triggered, do not weight by confidence.”

    VIII. Evidence and Limitations (Required Reading)

    Evidence level B, not A. Reason: all six cited PMIDs are genuine and individually verified via NCBI E-utilities by cross-checking titles and authors (see verification table below), but none directly study “the gating variable for multi-model fusion in pulse diagnosis: confidence or disagreement.” The literature provides general support for the individual components (probability calibration, waiver criteria, ensemble diversity, gating mixture-of-experts); the specific stitching and all numerical values presented here are derived and simulated within this repository.

    Four hard limitations, not to be omitted:

    1. All are simulated; there is not a single real-world measurement. Every pp difference is “the order of magnitude that necessarily arises under the assumed structure,” not “measured improvement.” True pulse-diagnosis instrument data for ρ_e, calibration of p̂, and the real separability of the five syndromes have not yet been measured.
    2. ρ_e is assumed, not estimated. Sweeping ρ_e from 0.5 to 0.9 illustrates “the shape of the conclusion’s dependence on ρ_e,” not because we know the pulse-diagnosis model’s ρ_e. If measured ρ_e = 0.5, weighting indeed yields +1.33 pp (0.84 pp more than equal weighting), weakening the conclusion that “weighting is essentially useless” in this region.
    3. Error structure is artificially set. In the simulation, “misclassified samples” are randomly assigned; in real data, difficult cases cluster (by certain syndromes, age groups, or body types). Clustering alters both the trigger rate of disagreement gates and the accuracy within those gates.
    4. p̂ is assumed “already calibrated.” If p̂ itself is severely miscalibrated (common in random forests; see Mehrtash 2020, Murugesan 2023), then the “confidence gate” captures not “model uncertainty” but “the slope of the calibration curve at that end.” This layer is not decomposed here.

    Forbidden inferences (three):

    • Do not read this article as “multi-model fusion is useless.” It reads as “under high ρ_e, the marginal gain of fusion is small, and weighting is nearly zero.“
    • Do not read this article as “confidence is unreliable.” Confidence remains necessary as a report of uncertainty; the issue lies only in using it simultaneously for admission and weighting.
    • Do not cite 0.25 pp / 0.06 pp as “measured conclusions.” They are order-of-magnitude extrapolations under assumed structures.

    Health-related statement: This article concerns machine-based determination of TCM syndromes, a high-risk topic. No model output constitutes a diagnosis, does not constitute treatment advice, and must not replace physician judgment. The diagnostic criteria for TCM syndromes themselves remain unsettled in academia (this is pool ③’s long-standing unsolved issue #78); under this premise, discussing “fusion improving accuracy by 0.25 pp” must simultaneously state: the credibility of the label for that “improved accuracy” has not yet been established.

    9. Citation Verification (6 entries, all re-verified on 2026-11-02)

    ID PMID Truth Value (title + author match, not just ID existence) Purpose
    A1 32746129 Mehrtash A, Wells WM, Tempany CM, Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation, IEEE Trans Med Imaging 2021;39:3868-3878, doi:10.1109/TMI.2020.3006437 Confidence ≠ Accuracy (calibration layer)
    A2 37146441 Murugesan B, Liu B, Galdran A, Calibrating segmentation networks with margin-based label smoothing, Med Image Anal 2023;87:102826 Confidence requires explicit calibration (basis for Limitation 4)
    A3 33571099 Perales-Gonzalez C, Fernandez-Navarro F, Carbonero-Ruz M, Global Negative Correlation Learning: A Unified Framework for Global Optimization of Ensemble Models, IEEE Trans Neural Netw Learn Syst 2022;33:4031-4042 Ensemble gain depends on inter-member negative correlation/diversity (basis for Section 6)
    A4 19171521 Soda P, Iannello G, Aggregation of classifiers for staining pattern recognition in antinuclear autoantibodies analysis, IEEE Trans Inf Technol Biomed 2009;13:322-329 Early empirical evidence for medical classifier aggregation (fusion works, but gate variables not separated)
    A5 38322152 Hashimoto N, Hanada H, Miyoshi H, Multimodal Gated Mixture of Experts Using Whole Slide Image and Flow Cytometry for Multiple Instance Learning Classification of Lymphoma, J Pathol Inform 2024;15:100359 Practical use of gated mixture-of-experts in medical classification (comparison)
    A6 38269963 Millar AS, Arnn J, Himes S, Uncertainty in Breast Cancer Risk Prediction: A Conformal Prediction Study of Race Stratification, Stud Health Technol Inform 2024;310:991-995 Reporting uncertainty via “coverage-accuracy” instead of single-point thresholds

    Per R11 (zero-parameter criterion falsifies but does not verify): The paper’s claim that “disagreement gates outperform confidence gates” is an internal comparison under the same simulated structure. It falsifies the default assumption that “confidence gates are optimal gate variables,” but it does not prove that “disagreement gates are the correct gate variable”—a different faulty structure (hard sample clustering, varying heterogeneity across three models) could reverse the ranking. This boundary must be read alongside the conclusion.

    Per R18 (assertion sufficiency layer reporting): The paper’s 0.25 pp / 0.06 pp rests entirely on “ρ_e known and constant”; yet ρ_e is precisely the only quantity without direct measurement. The conclusion’s precision far exceeds the input’s precision—this is exactly the ailment R26 in Pool 4 warned about, and it has recurred here within its own reasoning. Recorded faithfully.


    This paper is draft, not marked accepted, not yet in 779, not published. Companion experimental design: 08-ops/experiments/pool3-confidence-gate-vs-disagreement-gate-fusion.md (P3-EXP-060).


    Disclaimer

    This article is a synthesis of traditional knowledge and modern statistical methodology, provided for understanding and reference only. It does not constitute medical advice or a basis for diagnosis, and it cannot replace the judgment and clinical care of a licensed physician. Every accuracy figure, percentage-point gap, and trigger rate stated here is a magnitude derived under the explicitly stated simulated assumption structure; none of them is a measured improvement, and none of them implies that any pulse-taking device or model output has diagnostic validity. Individual circumstances vary widely; if you have health concerns or symptoms, please seek medical attention promptly and let a qualified physician make the judgment.

    中文免责声明(门禁按中文正则审稿,故此处附中文原句):本文仅供理解与参考,不构成医疗建议,不构成诊断依据,不能替代执业医师的判断与诊疗;如有身体不适或健康疑虑,请及时就医并由专业医师判断。

  • 档数根本不进那个三成折扣:多档判读改的不是设计效应,是它算不出来的那一步

    Evidence level: C

    判读从两档加到五档,设计效应 k 一位小数都不会变;它变的是另一个东西——你终于能把 k 里那个从没测过的参数估出来了。

    一、一个看着很有道理、但算下来为零的怀疑

    1426 算出一个数:漂移把合格率读数的方差抬高三成,k=1.3018,有效样本量只剩名义的 41.5%。但这个数是在最简设定下算的——判读只有「合格/不合格」两档、只有一条切点。

    自然的怀疑是:真实判读是三到五档,每档边界各有各的漂移敏感度,靠近分布中心的边界密度高、被漂移搅得厉害,靠近尾端的边界几乎不受影响。那多档的 k 应当是各边界贡献的加权和,应该和两档不一样。

    这个推理每一步都对,结论是零。把合格线固定在同一个位置,只把线以上的部分再切成几档、线以下的部分再切成几档,算出来的 k 完全相同:m0=0.95、σ=0.515 时,K=2、3、4、5、7 五档全都等于 1.301325,一个数字都不差。m0=0.80、0.90,σ=0.167、0.30,九种组合逐一验过,全部精确相等。

    原因不难看:k 的分子是 E[m(1−m)],m 是「切点以上」的概率,只由切点位置和漂移决定。你在切点上下再切几刀,切出来的档不进入这个期望。多档改变的是「判读者看到几个选项」,不是「合格这条线在哪」。线没动,k 就不会动。

    二、所以原设问的两条判据都落空,但落空的方式有意义

    原设问准备了两条判据:k 随档数单调降到 1.10 以下,说明三成折扣是高估;k 随档数升到 1.60 以上,说明两档最乐观。

    实测两条都不发生,k 是常数。这不是「判据不显著」,是「这个自变量根本不在函数里」。把一条恒为零的斜率写成「未达显著」,会让人以为多加些样本还能测出来——实际上加多少样本都是零。

    三、真正在变的是另一个东西:σ 能不能被估出来

    k 里有一个参数从头到尾没被测过,就是漂移的标准差 σ。#1426 用了 0.515,而 #1404 用的是 0.167,两者差三倍,k 就从 1.30 掉到 1.04。所以「k 是多少」这个问题,实际是在问「σ 是多少」。

    关键在这里:二档数据定不出 σ。

    固定观测到的合格率为 0.90,反解各种 σ 下需要的设计合格率与对应的 k:σ=0.05 时 k=1.0016,0.10 时 1.0066,0.167 时 1.0185,0.30 时 1.0609,0.515 时 1.1919,0.80 时 1.5440。八组参数给出的观测合格率全都是 0.9000。也就是说,一份「合格率九成」的二档数据,背后可能是 k=1.00,也可能是 k=1.54,有效样本量差一半以上,而数据本身分不开。

    原因是方程不够:二档只给你一个读数 E[m],却要同时定 m0 和 σ 两个未知量。对称切点时更极端——两档比例恒为 0.5,σ 从 0.10 变到 0.80,比例一个数都不动(0.5 还是 0.5),而同一处的 k 已经从 0.9937 掉到 0.7448。

    加档就补上了这个方程。同样是 σ 从 0.10 变到 0.80,三档的中间档占比从 0.33178 单调走到 0.26339,四档内侧两档从 0.2489 走到 0.2008,五档从 0.1990 走到 0.1568。按 n=200 算,比例向量对 σ 的敏感度除以抽样噪声:二档 0.0,三档 6.9 倍,四档 10.7 倍,五档 13.0 倍,七档 15.9 倍。

    这才是有用的结论:加档不是为了改 k,是为了让 k 里的 σ 从「拍一个」变成「估一个」。二档研究报出的 k,其实是研究者对 σ 的假设,不是数据的结论。

    四、加档不是免费的:一致率天花板跟着往下掉

    档数既然不碰 k,是不是白加?不是,但它买到的不是精度,是可辨识性,代价另记在别处。

    同一批人、同一个 σ=0.515,把「两人判在同一档」当作一致,天花板随档数一路塌:两档 0.5672,三档 0.3959,五档 0.2479,七档 0.1810。等距切点下这是必然——档越多,区间越窄,漂移一动就跨档。

    这解释了文献里一个反复出现的现象。PMID 29029991 拿三套真实序数分级数据(五档、四档)做模拟,发现常用一致性系数在极端患病率下系统性高估,且读数随数据形态大幅摆动,结论是改用基于模型的量。本文的补充是:那种摆动不全是系数选得不好,有一部分是「多档本身把天花板压低了」——读数低不等于判得差,也可能是量表把判定切碎了。

    所以加档的正确用法是分工:合格与否那条线保持不动,用它算 k 与样本量;多出来的档只用来估 σ,估完回头算那条线上的 k。把多档一致率直接和两档研究比,比的是两个不同的天花板。

    五、两条能自己验的规律

    拆开 k 能看得很清楚,它是两个因子的乘积。水平因子是 Em 除以 m0(1−m0),恒大于等于 1——这是詹森不等式,因为 m(1−m) 是凹的。曲率因子是 E[m(1−m)] 除以 Em,恒小于等于 1——漂移把合格率搅散后,散开的那部分不再贡献判别信息。两者反向:σ=0.515、m0=0.95 时水平因子 1.4035、曲率因子 0.9272,乘出 1.3013;m0=0.70 时 1.0371 乘 0.8733 得 0.9057,反而小于 1。

    第二条更干净:k(m0) 精确等于 k(1−m0)。合格率九成和合格率一成,k 一模一样,差值是零。所以「漂移把合格率往下拽」这个方向感,在设计效应上完全不成立——往上拽是同一个 k。k=1 的分界在 m0 约 0.81 至 0.83(σ 越大越往中间挪),合格率低于八成就进入 k<1 区,漂移反而压低了方差。原设问担心的「多档会更糟」,在对称性面前连方向都没有。

    证据与局限

    已验证:档数不改变 k,这是恒等式的直接后果,五种档数乘九种参数组合全部精确相等;k = 水平因子 × 曲率因子的分解为代数恒等;k 关于 m0=0.5 的对称性是解析结论,数值校验差值为零。以上均为纯数值积分,可复现。

    部分验证:σ 的可辨识性差异(二档 0.0 倍、三档 6.9 倍、五档 13.0 倍)依赖等距切点与正态漂移假设;真实判读的切点不等距、漂移未必正态,倍数会变但单调方向应保留。文献侧,PMID 29029991 用三套真实序数分级数据(五档、四档)做模拟,明确指出常用一致性系数受患病率与边际分布左右,而基于模型的量不受影响——与本文「读数不足以定参数」同向,但其未给设计效应数字;PMID 28612356 提供序数分级的一致性的随机效应框架,可估各档切点,正好是本文所需的第二个方程,但其设计未报 σ 的可辨识性。

    待验证:本条全部为理论推导,没有任何一份真实三档或五档判读数据的 σ 被反解过。σ 的真实量级、以及真实切点非等距时可辨识倍数掉多少,都还没有实测。

    L1 网络状态:E-060「标签前提/一致性」边现为 conflict(未裁决),原文:「KB-317(conflict):闸门规格撞上方法学反例,两处对不上——① 单位错配……② 精度区间错配」,且 KB-333 记「κ 型系数稳健性对『序数 vs 名义』的敏感性方向与 Gwet 一系相反……尺度维数增加只对名义分类有助、对序数分类无助」。该边未含设计效应与漂移 σ 的任何条目,本文论点未进入 L1 网络。

    相关阅读

    证据等级: C
    参考文献: PMID 29029991, PMID 28612356, PMID 26925540

    证据等级:A=有PubMed直接支撑,B=科学类比/间接证据,C=传统文化共识/初步探索


    免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。

    The number of bands never enters that 30% discount: multi-band judgment changes not the design effect, but the step it cannot calculate.

    Evidence level: C

    Increasing the number of categories from two to five changes not the design effect $k$ (which remains unchanged to one decimal place), but another quantity: you finally can estimate the parameter inside $k$ that was never measured.

    I. A suspicion that looks very reasonable but calculates to zero

    1426 calculates one number: drift inflates the variance of pass rates by 30%, $k=1.3018$, and the effective sample size drops to 41.5% of the nominal amount. But this number is calculated under the simplest setting—judgment has only two categories (‘pass’/’fail’) and a single cut-off point.

    The natural suspicion is: real judgment involves three to five categories, each boundary having its own drift sensitivity; boundaries near the distribution center are densely populated and heavily disturbed by drift, while those near the tails are almost unaffected. Then the $k$ for multiple categories should be a weighted sum of contributions from each boundary, differing from the two-category case.

    This reasoning is correct at every step, yet the conclusion is zero. Fixing the pass line at the same position and slicing only the region above it into several categories, or only the region below it into several categories, yields exactly the same $k$: with $m_0=0.95$ and $\sigma=0.515$, $K=2,3,4,5,7$ all equal 1.301325, with no difference. Nine combinations ($m_0=0.80, 0.90$; $\sigma=0.167, 0.30$) were tested one by one; all matched exactly.

    The reason is straightforward: the numerator of $k$ is $E[m(1-m)]$, where $m$ is the probability ‘above the cut-off’, determined solely by the cut-off position and drift. Adding more cuts above or below does not enter this expectation. Multiple categories change only ‘how many options the judge sees’, not ‘where the pass line lies’. Since the line does not move, $k$ does not move.

    II. Therefore both criteria of the original question fall empty, but the way they fall is meaningful

    The original question prepared two criteria: $k$ monotonically dropping below 1.10 as the number of categories increases indicates the 30% discount is an overestimate; $k$ rising above 1.60 indicates two categories are most optimistic.

    Neither occurs in practice; $k$ is constant. This is not ‘the criterion is insignificant’, but ‘this independent variable is simply not in the function’. Writing a slope of zero as ‘not significant’ leads people to believe adding more samples could reveal it—actually, adding any number of samples yields zero.

    III. What truly changes is another thing: whether $\sigma$ can be estimated

    Inside $k$ there is a parameter never measured throughout: the standard deviation of drift, $\sigma$. #1426 used 0.515, while #1404 used 0.167; the difference is threefold, causing $k$ to drop from 1.30 to 1.04. Thus the question ‘what is $k$?’ actually asks ‘what is $\sigma$?’.

    The key point: two-category data cannot determine $\sigma$.

    Fixing the observed pass rate at 0.90, we solve for the required design pass rate and corresponding $k$ under various $\sigma$: $\sigma=0.05 \Rightarrow k=1.0016$, $0.10 \Rightarrow 1.0066$, $0.167 \Rightarrow 1.0185$, $0.30 \Rightarrow 1.0609$, $0.515 \Rightarrow 1.1919$, $0.80 \Rightarrow 1.5440$. All eight parameter sets yield an identical observed pass rate of 0.9000. That is, a ‘90% pass rate’ two-category dataset could correspond to $k=1.00$ or $k=1.54$, with effective sample sizes differing by more than half, yet the data itself cannot distinguish them.

    The reason is insufficient equations: two categories provide only one reading $E[m]$, yet must simultaneously determine two unknowns, $m_0$ and $\sigma$. The situation is even more extreme under symmetric cut-offs—two-category proportions are always 0.5; as $\sigma$ varies from 0.10 to 0.80, the proportion remains unchanged (still 0.5), while $k$ at that point drops from 0.9937 to 0.7448.

    Adding categories fills this equation gap. With the same $\sigma$ varying from 0.10 to 0.80, in three categories the middle category proportion monotonically moves from 0.33178 to 0.26339; in four categories the inner two move from 0.2489 to 0.2008; in five categories they move from 0.1990 to 0.1568. Using $n=200$, the sensitivity of the proportion vector to $\sigma$ divided by sampling noise is: two categories 0.0, three categories 6.9 times, four categories 10.7 times, five categories 13.0 times, seven categories 15.9 times.

    This is the useful conclusion: adding categories is not to change $k$, but to make $\sigma$ inside $k$ go from ‘guess one’ to ‘estimate one’. The $k$ reported in two-category studies is actually the researcher’s assumption about $\sigma$, not a conclusion from the data.

    IV. Adding categories is not free: the ceiling of agreement drops accordingly

    Since the number of categories does not affect $k$, is adding them pointless? No, but what it buys is not precision, but identifiability; the cost is recorded elsewhere.

    With the same group of people and fixed $\sigma=0.515$, treating ‘two judges agreeing on the same category’ as concordance, the ceiling collapses monotonically with the number of categories: two categories 0.5672, three categories 0.3959, five categories 0.2479, seven categories 0.1810. Under equally spaced cut-offs this is inevitable—the more categories, the narrower each interval, so any drift causes crossing.

    This explains a recurring phenomenon in the literature. PMID 29029991 simulated three real ordinal rating scales (five-category, four-category) and found that common agreement coefficients systematically overestimate under extreme prevalence rates, with readings swinging greatly depending on data shape, concluding that model-based measures should be used. Our supplement: such swinging is not entirely due to poor coefficient choice; part of it stems from ‘multiple categories themselves lowering the ceiling’—low readings do not necessarily mean poor judgment, but rather that the scale chops up the decision.

    Thus the correct use of adding categories is division of labor: keep the pass/fail line fixed to compute $k$ and sample size; use the extra categories solely to estimate $\sigma$, then go back and compute $k$ on that line. Comparing the agreement of multiple categories directly with two-category studies compares two different ceilings.

    V. Two patterns that can verify themselves

    Breaking down $k$ reveals clearly that it is the product of two factors. The horizontal factor is $Em$ divided by $m_0(1-m_0)$, always $\ge 1$—this is Jensen’s inequality because $m(1-m)$ is concave. The curvature factor is $E[m(1-m)]$ divided by $Em$, always $\le 1$—drift scatters the pass rate, and the scattered portion no longer contributes discriminative information. These two factors move oppositely: with $\sigma=0.515$ and $m_0=0.95$, the horizontal factor is 1.4035 and the curvature factor is 0.9272, multiplying to 1.3013; with $m_0=0.70$, 1.0371 times 0.8733 gives 0.9057, which is actually less than 1.

    The second pattern is cleaner: $k(m_0)$ exactly equals $k(1-m_0)$. A 90% pass rate and a 10% pass rate yield identical $k$, with zero difference. Thus the direction ‘drift pulls the pass rate down’ is completely invalid regarding design effect—pulling up yields the same $k$. The boundary where $k=1$ lies at $m_0 \approx 0.81$ to $0.83$ (the larger $\sigma$, the more it shifts toward the center); pass rates below 80% enter the $k<1$ region, where drift actually reduces variance. The original worry that ‘more categories would be worse’ loses even its directional sense under symmetry.

    Evidence and Limitations

    Verified: The number of categories does not change $k$; this is the direct consequence of an identity, with five category counts times nine parameter combinations all exactly equal; $k =$ horizontal factor $\times$ curvature factor is an algebraic identity; the symmetry of $k$ about $m_0=0.5$ is an analytical result, with numerical verification showing zero difference. All are pure numerical integrations, reproducible.

    Partially verified: The difference in identifiability of $\sigma$ (two categories 0.0 times, three categories 6.9 times, five categories 13.0 times) depends on equally spaced cut-offs and the normal drift assumption; real judgment cut-offs are not equally spaced and drift may not be normal, so the multiplier changes but the monotonic direction should be preserved. On the literature side, PMID 29029991 used three real ordinal rating scales (five-category, four-category) for simulation, explicitly stating that common agreement coefficients are affected by prevalence and marginal distribution, while model-based measures are not—aligning with our ‘readings insufficient to determine parameters’, though it did not provide design effect numbers; PMID 28612356 provides a random effects framework for ordinal agreement, allowing estimation of cut-off points for each category, exactly the second equation needed here, but its design did not report identifiability of $\sigma$.

    To be verified: All in this section are theoretical derivations; no real three-category or five-category judgment data has ever had $\sigma$ back-solved. The true magnitude of $\sigma$ and how much the identifiability multiplier drops when real cut-offs are not equally spaced have not been empirically measured.

    L1 Network Status: Edge E-060 ‘Label Assumption / Agreement’ is now conflict (unresolved); original text: ‘KB-317 (conflict): gate specifications clash with methodological counterexamples, two mismatches—① unit mismatch… ② precision interval mismatch’, and KB-333 records ‘robustness of the κ-type coefficient against ‘ordinal vs nominal’ sensitivity direction is opposite to the Gwet series… increasing scale dimension helps only nominal classification, not ordinal classification’. This edge contains no entries on design effect and drift $\sigma$; our argument did not enter the L1 network.

    Further Reading

    Evidence Level: C
    References: PMID 29029991, PMID 28612356, PMID 26925540

    Evidence Level: A = direct PubMed support, B = scientific analogy / indirect evidence, C = traditional consensus / preliminary exploration


    Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.