「错一格」和「错两格」被当成一样重:把档次当无序,等于把最有临床意义的那部分分歧抹成了同一个数

Written by

in

Language

Evidence level: C

只要指标只读两个边际、不读表内结构,那么「轻度判成中度」与「轻度判成重度」对它贡献完全相同。这不是精度损失,是分辨力为零。

一、一个被省掉的差别

两位医生给同一批病人的严重程度打分,用的是「轻、中、重」三档。甲把某位本该「轻」的病人判成「中」,乙把同一位判成「重」。从病人的角度看,这两件事的重量差得很远:前者只是量级偏了一格,后者是方向性误判,可能直接改变处置路径。但在某类一致性指标里,这两个错误被记成同一个数,连小数点后第几位都不差。

不是算错了,而是它读的数据不够。承 #1408 的代数:全部结论只用两个量,一是两读者边际的总变差距离 TVD,二是合并边际的辛普森异质性 H。这套写法换来一个漂亮性质——一致性上限只依赖这两个数,跟表里质量怎么摊开无关,数学上很干净。代价写在同一条定理里:既然上限只依赖边际,那么包括「错得远不远」在内的一切表内信息,都不在指标里。

二、代数上这不是损失,是零

先把这件事说死。设两位读者 A、B 的判读结果构成联合分布表,行和记作 r、列和记作 c,则 TVD 等于两者差额绝对值之和的一半,H 等于一减去合并边际 m(r 与 c 的平均)各项平方之和。两式只含 r 和 c,不含表内任何格子。

逐位验证做过:固定同一对边际,让表内结构在十二种不同支撑集上跑——只在对角线放质量、只在错一格处放、只在跨两格处放、混合放,三到五档各一遍。TVD 小数点后六位一位不变,H 也一位不变。D=TVD 除以 H 这个量对表内结构的敏感度不是「很小」,是精确的零。

这意味着「上限只依赖 D」在数学上没错,但它同时宣判了:只报 D 的报告,在比较两个判读系统时手里其实没有可用信息。因为两系统只要边际一样,D 就一模一样。

三、留白到底有多宽

那么在同一对边际下,一把有序尺子能在这个留白里走多远?可以精确算,不必靠抽样猜。把「行和等于 r、列和等于 c、每格非负」当约束,把加权 kappa 的分子写成权重点乘表内格子,就是一个线性规划:求它的最大最小值,得到的是解析界而非样本极值。

算出来:三档、主档偏中(r 为 0.25、0.40、0.35,c 为 0.20、0.42、0.38)时,线性权 kappa 可达区间为负 0.4496 到正 0.9017,宽 1.3514;二次权为负 0.8182 到正 0.9301,宽 1.7483。四档宽 1.3924 与 1.8693,五档宽 1.3759 与 1.9381。三十万次随机抽样的实测极值与规划解吻合到小数点后三位,说明宽度是结构性的。

一句话:边际相同、D 相同、上限相同,而有序一致性可在宽达一点四到一点九的区间里随意取。比任何一条判读门槛(0.6 或 0.7)都宽得多。

四、最狠的一例:D 等于零

如果两个读者的边际完全一致,TVD 等于零,D 也等于零。按无序读法这是最理想的情况:没有系统性偏移,判读分布严丝合缝。这时有序一致性被锁住了吗?没有。

取主档 0.20、0.60、0.20 这组边际(两人分布相同),线性权 kappa 可达区间为负 0.25 到正 1.00,二次权为负 1.00 到正 1.00。两端都能取到:一端是两人都判对、只在中档微差错开,另一端是一人偏高一人偏低、错位互相抵消——后者在无序指标里完全隐形,在有序指标里是彻底失败。

这是全文最该记住的一点,也比原题预设更强。原题假设「若存在成对的表使排序相反」,实测是凡边际相同的表对,无序指标必然打平,而有序指标的差可以任意大。不是偶发反例,是恒等成立的结构。

五、排序反转接近抛硬币

原题判据:排序相反比例超过一成五,则无序指标禁用于跨系统比较。实测远不止一成五。

一千二百张合成表两两比较,剔除 D 打平的对(这类对无法排序,另行统计),可排序对的排序反转率:三档线性权 52.4%、二次权 52.7%;四档 49.3% 与 49.6%;五档 50.6% 与 50.4%。接近五成,跟抛硬币没区别。 一成五的线被超出三倍以上。

更要紧的是打平对:四档出现一对、五档出现四对 D 完全相同的表,其有序指标差距中位数为 0.4879 与 0.2013,四档那一对百分之百超过 0.10。这正是原题担心的场景——「天花板相同」被读成「两系统同样好」。

反向检验一下:两把有序尺子(线性权与二次权)会不会互相打架?会,但轻得多:方向相反率一成上下(10.5%、11.1%、10.9%),其中两边差距都超过 0.10 的只占 1.4%、0.8%、0.3%。矛盾主要不在「用哪把有序尺」,而在「要不要换有序尺」。

六、判决与报告规范

三条判据全部触发。

第一,无序口径的 D 只可用于同一系统内的样本量规划,不得用于比较两个判读系统——排序反转率五成,比较结果与随机无异。第二,凡出现「两系统一致性上限相同」的表述,必须同时报有序加权指标,并以有序指标作决策依据。第三,若因样本太小只能报无序指标,须显式注明「档位按无序假设处理,跨格权重一律取一」。

一个提醒:A 是错一格多、从不跨两格,B 是错一格少、但一错就跨两格。在 D 完全相同(0.05)的那组里,两者线性权 kappa 为 0.5156 与 0.5164,差不到千分之一;二次权却是 0.6610 与 0.3230,差 0.3380,远超 0.10 门槛。无序指标不是「稍微钝一点」,而是对最危险的那类分歧恰好失明。

证据与局限

已验证:TVD 与 H 仅由两个边际决定、与表内联合分布无关,这一点在固定边际、十二种支撑集、三到五档的设置下逐位成立;同一对边际下加权 kappa 的可达区间,用线性规划求得的精确端点与三十万次抽样实测极值吻合到小数点后三位;可排序对的排序反转率在三到五档、两种权重下均落在 49.3% 至 52.7% 之间。

部分验证:线性权与二次权之间的方向相反率一成上下,其中双边差距均超 0.10 的仅 0.3% 至 1.4%,说明有序指标内部相对自洽;但这一结论依赖均匀 Dirichlet 生成的合成表分布,真实判读数据的边际偏斜与错位形态未必落在同一区域。

待验证:一、加权 kappa 的可达区间能否写成只含边际与档数的解析式,若能则不必每次解规划;二、真实已发表的多读者有序一致性研究,其报告的线性权与二次权之差的实际分布落在可达区间的哪个位置,若普遍贴着一端,说明存在系统性的判读行为约束,可反过来当作设计检查。

L1 网络状态:L1 网络中无「有序度量与无序度量的可辨识性」对应边。最接近的是 E-060 标签前提,其状态为「冲突(未裁决,且新增两处单位错配/精度区间错配)」,原文摘录:「KB-292(conflict):中医医师间诊断一致性在开放人群中『低(low)』……同系列第二篇指出偶然校正一致性统计常被误用、建议 Gwet AC2」;该边只登记「标签信度够不够用」,未登记「指标把档位当无序会丢掉什么」。故本文证据等级封顶 C,上述全部结果为合成表上的代数与仿真结论,不构成对任何真实判读系统临床效力的判断。

相关阅读

证据等级: C
参考文献: PMID 26095449, PMID 25168681, PMID 38283651


免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。

\”Off by one\” and \”off by two\” are treated as equally heavy: treating tiers as unordered effectively erases the most clinically significant part of the divergence into a single n…

Evidence level: C

As long as the indicator reads only two margins and ignores the table’s internal structure, “mild misclassified as moderate” and “mild misclassified as severe” contribute identically to it. This is not a loss of precision; it is zero resolution.

I. A Difference That Was Skipped

Two doctors rate the severity of the same batch of patients using three tiers: “light,” “moderate,” and “heavy.” Doctor A misclassifies a patient who should be “light” as “moderate,” while Doctor B misclassifies the same patient as “heavy.” From the patient’s perspective, the weight of these two errors differs drastically: the former is merely an off-by-one magnitude error, whereas the latter is a directional misjudgment that may directly alter the treatment pathway. Yet in certain consistency indicators, both errors are recorded as the same number, differing not even in the decimal places.

This is not a calculation error; it stems from insufficient data being read. Following the algebra of #1408, all conclusions rely on only two quantities: first, the Total Variation Distance (TVD) of the sum of the two readers’ margins, and second, Simpson’s heterogeneity (H) of the combined margins. This formulation yields a beautiful property—the upper bound of consistency depends solely on these two numbers, irrespective of how quality is distributed within the table, making it mathematically clean. The cost is stated in the same theorem: since the upper bound depends only on the margins, all internal table information—including “how far off” the error is—is excluded from the indicator.

II. Algebraically, This Is Not Loss, It Is Zero

Let us settle this once and for all. Suppose the joint distribution table of ratings by two readers, A and B, has row sums denoted as $r$ and column sums denoted as $c$. Then TVD equals half the sum of absolute differences between corresponding elements, and $H$ equals one minus the sum of squares of the combined margins $m$ (the average of $r$ and $c$). Both formulas depend only on $r$ and $c$, containing no cells from the table itself.

Verification was performed bit by bit: fixing the same pair of margins, the internal table structure was run across twelve different support sets—placing quality only on the diagonal, only at the off-by-one position, only across two tiers, and mixed placements, covering three to five tiers each time. TVD remained invariant up to the sixth decimal place, and $H$ likewise remained constant. The sensitivity of the ratio $D = \text{TVD}/H$ to the internal table structure is not “small”; it is precisely zero.

This means that while “the upper bound depends only on $D$” is mathematically correct, it simultaneously declares that any report reporting only $D$ lacks usable information when comparing two rating systems. Because both systems yield identical margins, $D$ is identical regardless of internal variation.

III. How Wide Is the Blank Space?

So, under the same pair of margins, how far can an ordered ruler traverse within this blank space? This can be calculated precisely without relying on sampling guesses. By treating “row sums equal to $r$, column sums equal to $c$, and every cell non-negative” as constraints, and writing the numerator of weighted kappa as the weighted sum of table cells, we obtain a linear programming problem: finding its maximum and minimum values yields analytical bounds rather than sample extremes.

The results are as follows: with three tiers and the main tier biased toward moderate ($r = 0.25, 0.40, 0.35$; $c = 0.20, 0.42, 0.38$), the linear-weighted kappa ranges from $-0.4496$ to $+0.9017$, a width of $1.3514$; the quadratic-weighted kappa ranges from $-0.8182$ to $+0.9301$, a width of $1.7483$. For four tiers, widths are $1.3924$ and $1.8693$; for five tiers, widths are $1.3759$ and $1.9381$. Real-world extremes from 300,000 random samples matched the planned solutions up to the third decimal place, confirming that the width is structural.

In one sentence: With identical margins and identical $D$, the upper bound is the same, yet ordered consistency can freely take any value within a range as wide as 1.4 to 1.9. This is far wider than any rating threshold (0.6 or 0.7).

IV. The Most Extreme Case: D Equals Zero

If the two readers’ margins are completely identical, TVD equals zero, and $D$ also equals zero. In terms of unordered reading, this is the ideal scenario: no systematic bias, and the rating distribution fits perfectly. Is ordered consistency locked in this case? No.

Taking the main tier values of $0.20, 0.60, 0.20$ (where both readers have identical distributions), the linear-weighted kappa ranges from $-0.25$ to $+1.00$, while the quadratic-weighted kappa ranges from $-1.00$ to $+1.00$. Both extremes are achievable: one end corresponds to both readers agreeing, with only a slight off-by-one error in the middle tier; the other end corresponds to one reader being high and the other low, with errors canceling each other out—this latter case is completely invisible in unordered indicators but represents total failure in ordered indicators.

This is the most important point of the entire text, and it is even stronger than originally assumed. The original hypothesis stated “if there exist pairs of tables with reversed ordering,” but the actual result is that for any pair of tables with identical margins, the unordered indicator must be tied, while the difference in ordered indicators can be arbitrarily large. This is not an occasional counterexample; it is a structural identity.

V. Sorting Reversal Approaches Coin Tossing

The original criterion stated: if the proportion of reversed sorting exceeds 15%, then the unordered indicator is unsuitable for cross-system comparison. In reality, this far exceeds 15%.

Among 1,200 synthetic tables compared pairwise, excluding pairs where $D$ is tied (these cannot be sorted and are counted separately), the sorting reversal rate for sortable pairs is: 52.4% for linear weighting and 52.7% for quadratic weighting in three tiers; 49.3% and 49.6% in four tiers; and 50.6% and 50.4% in five tiers. This approaches 50%, indistinguishable from coin tossing. The 15% threshold is exceeded by more than three times.

Even more critical are the tied pairs: in four tiers, one pair and in five tiers, four pairs had completely identical $D$. Their ordered indicator differences had medians of 0.4879 and 0.2013, with the four-tier pair exceeding 0.10 in all cases. This is precisely the scenario the original paper worried about—”same ceiling” being misread as “both systems equally good.”

A reverse check: do the two ordered rulers (linear and quadratic weights) contradict each other? Yes, but lightly: the rate of opposite direction is around 10% (10.5%, 11.1%, 10.9%), with only 1.4%, 0.8%, and 0.3% having differences exceeding 0.10 on both sides. The main contradiction lies not in “which ordered ruler to use,” but in “whether to switch the ordered ruler.”

VI. Judgment and Reporting Standards

All three criteria are triggered.

First, the $D$ of the unordered definition should be used only for sample size planning within the same system, not for comparing two rating systems—the sorting reversal rate is 50%, making comparison results indistinguishable from random. Second, whenever the statement “both systems have the same consistency upper bound” appears, the ordered weighted indicator must also be reported, and decisions must be based on the ordered indicator. Third, if the sample size is too small to report the ordered indicator, it must explicitly state “tiers are handled under the unordered assumption, and cross-tier weights are all set to one.”

A reminder: A makes off-by-one errors but never crosses two tiers, while B makes fewer off-by-one errors but whenever it errs, it crosses two tiers. In the group where $D$ is completely identical (0.05), their linear-weighted kappas are 0.5156 and 0.5164, differing by less than one-thousandth; however, their quadratic-weighted kappas are 0.6610 and 0.3230, differing by 0.3380—far exceeding the 0.10 threshold. The unordered indicator is not “slightly dull”; it is precisely blind to the most dangerous type of disagreement.

Evidence and Limitations

Verified: TVD and $H$ depend solely on the two margins and are independent of the joint distribution within the table, confirmed under fixed margins, twelve support sets, and three to five tiers; the reachable interval of weighted kappa under the same margin pair, computed via linear programming, matches the extreme values from 300,000 random samples up to the third decimal place; the sorting reversal rate for sortable pairs falls between 49.3% and 52.7% across three to five tiers under both weighting schemes.

Partially Verified: The rate of opposite direction between linear and quadratic weights is around 10%, with only 0.3% to 1.4% having bilateral differences exceeding 0.10, indicating internal self-consistency among ordered indicators; however, this conclusion relies on synthetic table distributions generated by uniform Dirichlet, and real rating data may exhibit margin skewness and misalignment patterns that do not fall within the same region.

Pending Verification: First, whether the reachable interval of weighted kappa can be expressed as an analytical formula containing only margins and tier count, which would eliminate the need to solve linear programming repeatedly; second, for existing published studies on ordered consistency with multiple readers, where the actual distribution of differences between linear and quadratic weights falls within the reachable interval—if most cluster near one end, this suggests systematic constraints in rating behavior, which could serve as a design check.

L1 Network Status: In the L1 network, there is no edge corresponding to “identifiability between ordered and unordered measures.” The closest is the E-060 label prerequisite, whose status is “conflict (unresolved, with two additional unit mismatches/precision interval mismatches).” The original excerpt states: “KB-292 (conflict): diagnostic consistency among TCM physicians in an open population is ‘low’… the second paper in the same series notes that chance-corrected consistency statistics are often misused and recommends Gwet AC2”; this edge only records “whether label reliability is sufficient,” not “what is lost when the indicator treats tiers as unordered.” Therefore, the evidence level of this paper caps at C; all above results are algebraic and simulation conclusions on synthetic tables, constituting no judgment on the clinical efficacy of any real rating system.

Further Reading

Evidence Level: C
References: PMID 26095449, PMID 25168681, PMID 38283651


Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *