想从别人的论文里倒推出那个门槛,倒不出来:他们判「合格」用的根本不是这把尺子

Written by

in

Language

Evidence level: C

反推一个参数的前提,是被判定的那个量真的由这个参数决定。而文献里「一致性可接受」这句话,是由另一套指标决定的。

一、这个想法本身很干净

上一轮的缺口是:决定「差多少算不能容忍」的带宽 w 来源不明,0.05 与 0.15 都能自称可接受,宽出三倍。于是有了一条出路——既然 w 是客观存在、只是没人测过的量,就从已发表研究里倒推。每篇报告了双读者边际分布的研究都隐含地提供了一个 w:作者写下「一致性可接受」时,心里必在用某个默认带宽。把这些 w 按结论分组,分离得开则经验值就出来了。这推理无懈可击,但跳过了一句前提。

二、被判定的量必须由那个参数决定

反推要成立,作者判「可接受」时用的判据得真是 w 那条线。这听着像同义反复,实际不是。判读研究报的指标主流为三类:观察一致率、kappa 一类偶然校正系数、组内相关系数,阈值长期由教科书建议给定。系统综述给了直接证据:七十九篇观察者变异研究中五成二报组内相关系数、三成九报 kappa,而作者指出这些变异估计的解读常与报告结论不对应(PMID 37399082)。结论那句话与它前面那个数字之间,本就有一处没人核对的缝。

更麻烦的是两套指标测的不是同一件事。kappa 校正全部不一致,涵盖三档之间所有错配;带宽只问主档占比差了多少,两者没有常数换算。用阈值潜变量模型仿真(三档、双读者、每批三十到五百例、噪声 0.3 至 0.9 个标准差,真判据设为 kappa 不低于 0.60):用主档占比差去分割「可接受」与「不可接受」两组,最优分割点最小误分类率 19.7%——五分之一的研究被放错组;门槛放宽到 kappa 0.40 则升到 42.0%,接近抛硬币。反过来若真判据本就在带宽空间,同样分割的误分类率是 0.0%。

这不是方法钝,而是方向单向:真判据在带宽空间时反推完美,在 kappa 空间时近乎无效。两量相关系数只有负 0.105,几乎正交,而文献里的判据恰恰在后一个空间。

三、第二层:噪声比要测的门槛还大

就算空间摆平,还有更硬的障碍。每篇研究几十到一两百例,两位读者在同一批案子上的主档占比差本身就是随机量。仿真给出这个差的标准差:三十例 0.062、四十七例 0.051、八十八例 0.037、一百二十例 0.031。影像变异研究的病例数中位数正是四十七例(PMID 37399082)。两者一叠就清楚:四十七例时噪声正好是 w 取 0.05 的 1.0 倍、取 0.10 的 0.5 倍。一篇中位规模的研究,报出的差里一半以上是噪声。 拿这堆读数反推 0.05 门槛,等于用比被测物还粗的尺子去量。

加篇数能救一部分。若判据真在带宽空间、真值 0.10,篇数由三十加到六百,反推的九十五分位从 0.070 收敛到 0.074,区间由正负 0.018 收窄到 0.005。但它收敛到 0.074 而非 0.100,偏低 26%——被接受那组截在门槛下,用组内分位数估门槛必然偏小;且这收敛只在最有利世界成立。

四、双峰判不出来,但双峰是真的

原题有一条判据:若反推的 w 明显双峰,说明「可接受」在文献里是两个互不相通的学派。这条数据上可能成立,方法上却不可达。

设两个领域真各用一套门槛,一为 0.05 一为 0.15。各自被接受的研究读数仍带噪声(一百二十例时 0.031),两峰相距 0.10 是噪声 3.2 倍,听上去够分。关键在截断:被接受意味着读数落在门槛以下,两组都是右偏截尾分布,不是两个居中钟形。截尾改掉了峰的形状——0.05 那支被截成贴着零的窄条,0.15 那支是宽得多的斜坡,混起来是一条单调下降曲线,没有第二峰。

实测一致:高斯混合做单峰对双峰比较,双峰世界检出率 100%,单峰截尾世界假阳性率同样 100%。两种世界的读数长得一模一样。 不是模型不够强,是两种生成过程在截尾后信息等价。故「看分布有几个峰」这条判据,无论结论是双峰还是单峰都不作证据。

五、结论

第一,从已发表一致性研究反推 w,这条方法上关闭。不是篇数不够,是被判定的量与待反推的量不在同一空间,用带宽分割 kappa 判定的组,误分类率 19.7% 起。任何声称「从两百篇文献反推出 w」的工作,都得先证明那两百篇的判据真在带宽空间——按现有报告规范,这证明做不出来。

第二,「反推不出统一门槛,所以法则的实证基础不存在」也不成立。原题设了退路:若 w 与结论分离度接近零,说明裁决没用可量化阈值。但分离度接近零有两个原因——裁决真没有门槛,或门槛设在另一个坐标系。本篇给出了后者的量级:在 kappa 世界分离度就是接近零,而那里的门槛明确得不能再明确。把「测不出」读成「不存在」,是这类反推最容易踩的坑。

第三,能救这条路的只有判据写清楚的数据。判据一旦写成「主档占比差不超过百分之几」,反推就精确;反之只报 kappa 与置信区间、不报边际分布的研究,多少篇都不进池子。这压力转回「改报告规范」——既有共识文件早已要求:可靠性研究报告长期不完整,样本选择、设计、统计信息常缺失,使结果解读与合并困难(PMID 21130355)。

证据与局限

已验证: 阈值潜变量模型下,判据位于带宽空间时反推精确(误分类率 0.0%)、位于 kappa 空间时反推失效(19.7% 至 42.0%),两空间相关系数负 0.105。本轮仿真直接结果,可复现。

部分验证: 影像观察者变异研究的方法学现状——七十九篇中病例数中位四十七例、观察者四人、仅 15% 论证样本量、五成二报组内相关系数、三成九报 kappa、且「变异估计的解读常与结论不对应」(PMID 37399082)。为判据空间错配提供现实基础,但该综述未逐篇核对判据口径,只作间接支撑。

待验证: ① 真实文献中判据写成带宽形式的比例,需逐篇抽取,若低于一成则反推彻底无料;② 截尾导致的单峰与双峰信息等价,是否在真实分层数据上同样表现为「怎么分都看不出两峰」;③ 反推估计量系统性偏低 26% 能否用截尾反卷积校正回来。

证据等级: C
参考文献: PMID 37399082, PMID 21130355, PMID 15733050, PMID 2348207, PMID 2189948

相关阅读

L1 网络状态:L1 中最近的一条边为 E-060 标签前提(五脏辨证作分组标签|前提|标签信度,状态 conflict):「KB-292(conflict):中医医师间诊断一致性在开放人群中『低(low)』,作者称为『令人担忧』(PMID 30383398);同系列第二篇指出偶然校正一致性统计常被误用、建议 Gwet AC2(PMID 31730403 / 31634000 / 31730404)」,冲突另一方为 KB-317:「抗灰区最优的是 Gwet AC2 与 Brennan-Prediger κ(PMID 33853549 蒙特卡洛),但所有加权系数在『中低一致性 + 大样本 + 类别数多』时精度显著下降」。该边只登记「标签信度够不够用」,未登记「判定用的指标与待反推的带宽参数是否同一坐标系」。L1 网络中「kappa」「反推」「最小重要差」「判读」「w」命中均为 0,「一致性」6 处与「信度」8 处全部落在 E-060,「可接受」1 处为内感受量表因子结构性质,与判读门槛无关;边表 EDGE_TABLE 概念最高止于生理概念,无一致性度量节点。故本文按规则封顶 C,上述结论应读作「方法学层面的可辨识性论证」,不作机制断言。


免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。

They didn’t use that ruler to judge \”qualified.\

Evidence level: C

A necessary premise for reverse-engineering a parameter is that the measured quantity is truly determined by that parameter. The phrase “acceptable consistency” found in literature is instead governed by a different set of metrics.

I. The Core Idea Is Conceptually Clean

The gap in the previous round was that the bandwidth $w$ defining “what difference is intolerable” had no known source; values of 0.05 and 0.15 both claimed to be acceptable, differing by a factor of three. This created an apparent impasse—until one considers that $w$ is an objective quantity simply never measured.

This opens a path: since published studies reporting the marginal distribution of two readers implicitly provide a $w$, we can reverse-engineer it. When authors state “consistency is acceptable,” they are necessarily using some default bandwidth as their mental criterion. By grouping these $w$ values according to the study’s conclusion, we can separate acceptable from unacceptable cases and derive empirical values. This reasoning is logically sound, yet it skips a critical premise.

II. The Measured Quantity Must Be Determined by That Parameter

For reverse-engineering to hold, the criterion authors use to judge “acceptable” must truly be the $w$ threshold. This sounds tautological, but it is not.

The dominant metrics reported in studies fall into three categories: observed agreement rates, kappa-type chance-corrected coefficients, and intraclass correlation coefficients (ICC). Thresholds for these have long been set by textbook recommendations. Systematic reviews provide direct evidence: among 79 studies of observer variation, 52% reported ICC and 39% reported kappa, yet authors noted that interpretations of these variation estimates often do not align with the reported conclusions (PMID 37399082). There is thus an unverified gap between the numerical result and the stated conclusion.

Complicating matters further, these two metrics measure fundamentally different things. Kappa corrects for all inconsistencies across three categories, capturing every mismatch; bandwidth only asks how much the main category proportion differs. There is no constant conversion between them.

Simulations using threshold latent variable models (three categories, two readers, 30–500 cases per batch, noise 0.3 to 0.9 standard deviations, true criterion set at kappa ≥ 0.60) reveal this disconnect:
– When splitting “acceptable” vs. “unacceptable” based on main category proportion difference, the optimal cutoff yields a minimum misclassification rate of 19.7%—one-fifth of studies are misclassified.
– If the threshold is relaxed to kappa ≥ 0.40, the error rate rises to 42.0%, nearly random.
– Conversely, if the true criterion lies in bandwidth space, the same splitting yields 0.0% misclassification.

This is not a methodological flaw; it reflects directional asymmetry: reverse-engineering works perfectly when the true criterion is in bandwidth space but fails almost entirely when it is in kappa space. The correlation coefficient between these two metrics is only −0.105, nearly orthogonal, yet the literature’s criteria operate precisely in the latter space.

III. Second Layer: Noise Exceeds the Threshold

Even if spatial alignment were resolved, a more fundamental obstacle remains. Each study involves dozens to hundreds of cases; the difference in main category proportions between two readers on the same batch is itself a random variable. Simulations yield standard deviations for this difference:
– 30 cases: 0.062
– 47 cases: 0.051
– 88 cases: 0.037
– 120 cases: 0.031

The median case count in imaging variation studies is exactly 47 (PMID 37399082). Overlaying these reveals that at 47 cases, the noise equals $1.0 \times w$ if $w = 0.05$ and $0.5 \times w$ if $w = 0.10$. In a study of median size, more than half of the reported difference is pure noise.

Increasing sample size helps partially. If the true criterion lies in bandwidth space with $w = 0.10$, expanding from 30 to 600 cases causes the reverse-engineered 95th percentile to converge from 0.070 to 0.074, narrowing the interval from ±0.018 to 0.005. However, it converges to 0.074 rather than 0.100—a 26% underestimate—because the accepted group is truncated below the threshold, and estimating the threshold via within-group quantiles inevitably underestimates it. Moreover, this convergence holds only in the most favorable scenario.

IV. Bimodality Cannot Be Detected, Yet Bimodality Is Real

The original criterion states: if the reverse-engineered $w$ shows clear bimodality, “acceptable” in the literature represents two distinct schools. This may hold empirically but is methodologically unreachable.

Suppose two domains truly use different thresholds: $w = 0.05$ and $w = 0.15$. The accepted studies still carry noise (0.031 at 120 cases), and the two modes are separated by 0.10—3.2 times the noise—which seems sufficient. The critical issue is truncation: acceptance means the reading falls below the threshold, so both groups are right-truncated distributions, not centered bell curves. Truncation distorts the shape of the modes—the $w = 0.05$ mode becomes a narrow strip near zero, while the $w = 0.15$ mode is a much broader slope. Combined, they form a monotonically decreasing curve with no second peak.

Empirical validation confirms this: Gaussian mixture modeling comparing single-mode vs. bimodal distributions yields 100% detection rate in the true bimodal world but also 100% false positive rate in the single-mode truncated world. The readings look identical in both worlds. It is not that the model is insufficient; rather, after truncation, the information in both generative processes becomes equivalent. Thus, “checking how many peaks the distribution has” provides no evidence, regardless of whether the conclusion is bimodal or unimodal.

V. Conclusion

First, reverse-engineering $w$ from published consistency studies is methodologically closed. It is not due to insufficient sample size, but because the measured quantity and the target parameter do not reside in the same space. Using bandwidth to split kappa-defined groups leads to misclassification rates starting at 19.7%. Any claim of deriving $w$ from hundreds of studies must first prove those studies’ criteria truly operate in bandwidth space—a proof impossible under current reporting standards.

Second, “the inability to reverse-engineer a unified threshold implies no empirical basis for the rule” is false. The original setup allowed an escape: if $w$ and the conclusion separate completely, it suggests no quantifiable threshold exists. Yet this separation can arise from two causes: either the rule truly has no threshold, or the threshold lies in another coordinate system. This paper establishes the latter magnitude: separation is near zero in kappa space, where the threshold is unmistakably defined. Misreading “cannot be measured” as “does not exist” is the most common pitfall of such reverse-engineering.

Third, the only way to salvage this approach is through explicitly stated criteria. If the criterion is written as “main category proportion difference not exceeding X percent,” reverse-engineering becomes precise. Conversely, studies reporting only kappa and confidence intervals without marginal distributions cannot contribute, regardless of number. This pressure shifts back to “improving reporting standards”—a consensus document already notes that reliability reports are often incomplete, lacking sample selection, design, and statistical details, making result interpretation and meta-analysis difficult (PMID 21130355).

Evidence and Limitations

Verified: Under threshold latent variable models, reverse-engineering is precise when the criterion lies in bandwidth space (misclassification rate 0.0%) but fails when in kappa space (19.7% to 42.0%). The correlation between the two spaces is −0.105. These simulation results are directly reproducible.

Partially Verified: The current state of methodology in imaging observer variation studies—79 studies with median case count 47, four observers, only 15% arguing for sample size justification, 52% reporting ICC, 39% reporting kappa, and frequent misalignment between variation estimates and conclusions (PMID 37399082)—provides realistic foundation for criterion-space mismatch. However, this review did not verify criteria across all studies, offering only indirect support.

To Be Verified:
1. The proportion of true literature where criteria are expressed in bandwidth form must be extracted study-by-study; if below 10%, reverse-engineering yields nothing.
2. Whether truncation-induced equivalence of single-mode and bimodal information holds in real stratified data, manifesting as “no detectable bimodality regardless of splitting method.”
3. Whether the systematic 26% underestimation in reverse-engineered estimates can be corrected via deconvolution of truncation effects.

Evidence Level: C
References: PMID 37399082, PMID 21130355, PMID 15733050, PMID 2348207, PMID 2189948

Related Reading

L1 Network Status: The most recent edge in L1 is labeled E-060 (TCM diagnosis grouping label | premise | label credibility, status conflict): “KB-292 (conflict): Diagnostic consistency among TCM physicians in open populations is ‘low’ and described as ‘concerning’ (PMID 30383398); the second paper in the series notes that chance-corrected consistency statistics are often misused and recommends Gwet AC2 (PMID 31730403 / 31634000 / 31730404).” The conflicting side is KB-317: “The optimal gray zone is Gwet AC2 and Brennan-Prediger κ (PMID 33853549 Monte Carlo), but all weighted coefficients suffer significant accuracy loss under ‘low-moderate consistency + large sample + many categories’.” This edge only records “whether label credibility suffices,” omitting “whether the criterion metric and the reverse-engineered bandwidth parameter share the same coordinate system.” In the L1 network, “kappa,” “reverse-engineer,” “minimum important difference,” “judgment,” and “$w$” all hit zero; “consistency” appears in 6 places and “reliability” in 8 places, all falling under E-060, while “acceptable” appears once as an introspective scale factor unrelated to judgment thresholds. The EDGE_TABLE concept reaches only physiological concepts, lacking consistency measurement nodes. Therefore, this paper caps at C per rules; the above conclusions should be read as “methodological identifiability arguments,” not mechanistic assertions.


Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *