三条判据线到底会不会打架:别用 kappa 去判,它自己就带着同一个病

Written by

in

Language

Evidence level: C

要判断三条判据是否一致,最直觉的做法是算一个 Cohen’s kappa;但 kappa 恰恰在分布最不均衡的时候最不可信——而这正是本项目的常态。

一、这个问题是怎么来的

上一轮迭代里,范围损失的判据从「合成加权指数」改成了「逐变量诊断」:标准化均值差小于 0.1 则保留合并;标准化均值差超过 0.25 且交互检验功效高于 80% 则分层报告;交互检验功效低于 30% 则报未知。新判据干净,但从未在同一份数据上与旧的合成指数做过交叉验证。

于是问题变成:同一批参数放到三条判据线下,会不会给出相同的裁定?直觉上的答案是算一个 Cohen’s kappa。但这条路有个坑,而且在本项目的语境下坑特别深。

二、kappa 会把你想要的答案读反

kappa 的定义是:(观察一致率 − 期望一致率)除以(1 − 期望一致率)。分母表示「还剩多少分歧可以被解释」,分子表示「实际一致比瞎猜好多少」。关键因子是「期望一致率」。

当边际分布高度不均衡——绝大多数参数被划进同一档——期望一致率趋近 1,分母趋近 0,kappa 塌向 0,哪怕观察一致率高达 0.9。这就是「高一致、低 kappa」的悖论(PMID 2348207)。同一篇论文还指出第二个悖论:在边际不对称的情形下,kappa 可能比在对称失衡的情形下人为更高,看上去「更好」,而失衡其实更严重。

放到本项目上:三条判据的期望裁定分布恰恰是高度不均衡的——绝大多数参数应当落入「保留合并」,而这正是悖论最严重的区域。直接算 kappa 只会得出「三条判据严重不一致」,而这可能只是「裁定分布太偏」的另一种说法。

三、三分类又多加了一层复杂性

裁定不是二值而是三值:保留、分层、未知。在三个类别下,期望一致率由三个边际比例共同决定,比二值情形更敏感;此外,算加权 kappa 还要选权重(线性或二次),多出一个会改变数值的自由度。方法学比较给出的结论很明确:在 2×2 表的多种边际情形下,kappa 与 alpha 受边际分布影响最大,而 AC1 与 B 统计量更稳健,其中 B 统计量在两种悖论下表现最好(PMID 25168681);偏倚校正与流行率校正的 kappa(PABAK)直接校正了这两重偏倚(PMID 8501467)。这不只是理论:在高特征流行率的病理诊断中,kappa 会显著偏离观察一致率给人的印象(PMID 38283651)。

四、一致不等于正确

三条判据共享一个未经检验的前提:真值空间是被构造出来的。合成指数里的权重、逐变量诊断里的 0.1 与 0.25 阈值、比例度量背后的真值分布,全都是建模者填进去的。它们彼此一致,只说明它们在同一套假设下对齐,并不能证明其中任何一条正确。

所以 kappa 高不是好信号:它意味着三条线没有提供独立信息。真正该担心的是三条一起同意却一起错——这种偏差,一致率永远测不出来。

五、第三条判据根本不是同一个量

前两条判据给出的是逐参数的类别裁定,第三条「结论成立的比例」却是连续量:真值空间中结论仍然成立的比例。要把这个连续量塞进三档裁定,就必须选一个阈值。阈值是自由参数:取 20% 还是 30%,一致率就变了。

因此「三幅画像的 kappa」其实是两个类别量与一个被任意切了阈的连续量之间的一致性。挪一下阈值,结论就变。这与上一轮批评「把判别力改名叫可推广性」是同一个病,只是入口换了一个。

六、一致性是对称的,后果不是

判据 A 判「降级」而 B 判「保留」,与 A 判「保留」而 B 判「降级」,在一致率上完全相同。但后果差别极大:前者多花样本成本,后者导致结论过度外推。应该报告的是不一致的条数与方向,而不是幅度。

kappa 本身也有相当可观的抽样波动。一个旁证:连「算 kappa 需要多少样本」这个问题,文献里也是用观察一致率的比例而非 kappa 值本身来处理的(PMID 25088752)。拿一把自己会晃的尺子去判断另外两把尺子是否一致,得到的精度不会超过那把尺子自己的不稳定性。

七、把问题换掉

四个动作可以替代「算一个 kappa」:

  1. 逐参数分别报告三条原始裁定,不做合并、不做分数求和——这与平衡诊断的提议一致:逐变量报告标准化均值差与方差比,而不是压成单一指标(PMID 30788363)。
  2. 同时报告观察一致率、PABAK 与 B 统计量;任何只依赖 kappa 得出的结果一律作废。
  3. 报告不一致的条数与方向,并计算「会改变设计决策的参数线条数」。这才是与决策相关的量:交互检验功效不足会使调节幅度不可估(PMID 15066682),边际估计与条件估计甚至可能给出相反的决策(PMID 41626969)。
  4. 用自助法给一致率生成置信区间。若区间宽度超过 0.3,每组 60 个样本不足以支撑一致性结论,必须扩样。

四条判据对应四种结论:若观察一致率高于 85% 而 kappa 低于 0.4,则悖论得到确认,kappa 是误报,必须撤回「三幅画像不可互换」这一结论;若校正后 PABAK 仍低于 70%,则确实存在不一致,三幅画像必须分开报告,且禁止用范围损失数值做跨研究比较;若第三条判据的阈值在 20% 到 30% 之间移动时,导致超过 20% 的参数跨档,则它不具备判据资格,只能降级为敏感性分析;若自助法区间宽度超过 0.3,则样本量问题先于一致性问题。

一句话:判据之间是否一致,必须用一把不受分布形状影响的尺子去量;用一把有偏的尺子量,量出来的永远是尺子自己的偏。

证据与局限

已验证:Cohen’s kappa 存在两种结构性悖论——在边际不均衡下,观察一致率高也会被大幅压低;在边际不对称下,kappa 可能人为更高,且这两种悖论都不能靠把 kappa 换成 kappa max 来修掉(PMID 2348207);对 2×2 表中各种一致性系数的系统比较显示,kappa 与 alpha 受边际分布影响最大,AC1 与 B 统计量更稳健,其中 B 统计量在两种悖论下表现最好(PMID 25168681);PABAK 直接校正了这两重偏倚(PMID 8501467);kappa 的样本量规划实践已转向使用观察一致率的比例而非 kappa 值本身(PMID 25088752)。

部分验证:高流行率特征下 kappa 显著偏离观察一致率的现象,已在病理诊断实证数据中被量化(PMID 38283651),但其对象是肺组织学特征、读者是病理医师,与丹田参数的裁定场景不同;平衡诊断应逐变量报告标准化均值差与方差比而非压成单一指标(PMID 30788363),这一提议源自倾向得分匹配领域,移植到本项目构成类比;交互检验功效不足会使调节幅度不可估(PMID 15066682),效应修饰与不可压缩性并存可导致边际与条件估计给出相反决策(PMID 41626969),两者都支持「方向比幅度更重要」,但均基于临床试验与综述语境。

待验证:三条判据在同一份丹田数据上的实际裁定分布从未被测量过——「多数参数落入保留合并」只是预期;若实测分布其实是均衡的,悖论风险显著下降,本文的主论证随之减弱;三档裁定下 kappa 的具体偏离幅度没有模拟数据;PABAK 与 B 统计量在本项目数据上的数值差异没有实测;第三条判据的阈值敏感性曲线(在 20%、25%、30% 三档上跨档的参数比例)从未算过;「一致不等于正确」这一层完全依赖论证,任何一致率指标都测不了它。

相关阅读

证据等级: C
参考文献: PMID 2348207, PMID 25168681, PMID 8501467, PMID 25088752, PMID 38283651, PMID 30788363, PMID 15066682, PMID 41626969
L1 网络状态:该主题尚未进入 L1 对应网络——CORRESPONDENCE_NETWORK.md 节点表按易学/中医侧、物理侧、生理/体验侧三类域分派节点(依概念与量,如五行、五脏、频率、颜色等),三类域中均无「判据一致性 / 裁定类别 / 一致性系数 / 边际分布偏倚」这类测量学与判读层的节点;边表止于 E-050(「六十甲子纳音 | 对应 | 60 个频率」,层级 L1、状态 open,现代证据栏原文「尚无任何可测来源」),不存在以判据、裁定或一致率为源或目标的边。EDGE_TABLE_science-yixue.md 为反抽统计,仅列概念计数(丹田 51 篇 / 气功 48 篇 / 呼吸 189 篇 / 频率 407 篇)与最高证据级,无 L1/L2/L3 层级与 active/open/conflict 状态判定,故本条边状态为缺。


免责说明:本文属方法学与统计口径的讨论,仅供理解与参考,不构成医疗建议,不能替代执业医师的判断与诊疗。文中出现的阈值、百分比与样本量建议均为判读一致性这一方法学层面的推演,不适用于任何个体情形;如有健康疑虑,请及时就医并由专业医师判断。

Will the Three Criteria Lines Clash? Don’t Use Kappa to Judge It, as It Carries the Same Flaw

Evidence level: C

To determine if three criteria are consistent, the most intuitive approach is to calculate Cohen’s kappa; however, kappa is least trustworthy precisely when distribution imbalances are highest—a condition central to this project.

I. How This Question Arises

In the previous iteration, the criterion for range loss shifted from a “synthesized weighted index” to a “per-variable diagnosis”: retain merging if the standardized mean difference is less than 0.1; report stratification if the standardized mean difference exceeds 0.25 and interaction power surpasses 80%; report unknown if interaction power is below 30%. The new criterion is clean, yet it has never been cross-validated against the old synthesized index on the same dataset.

Thus, the question becomes: If the same parameter is subjected to three criteria lines, will they yield identical verdicts? The intuitive answer is to calculate Cohen’s kappa. However, this approach contains a pitfall, and it is particularly deep in the context of this project.

II. Kappa Reads Your Desired Answer Backwards

Kappa is defined as: (Observed Agreement – Expected Agreement) divided by (1 – Expected Agreement). The denominator represents “how much disagreement remains explainable,” while the numerator represents “how much better actual agreement is than guessing.” The critical factor is “Expected Agreement.”

When marginal distributions are highly imbalanced—where the vast majority of parameters are categorized into a single tier—the expected agreement approaches 1, the denominator approaches zero, and kappa collapses toward 0, even if observed agreement reaches 0.9. This is the paradox of high agreement but low kappa (PMID 2348207). The same paper identifies a second paradox: kappa can be artificially higher under marginal asymmetry compared to symmetric imbalance, appearing “better” despite the imbalance.

Applied to this project: The expected verdict distribution across the three criteria is precisely highly imbalanced—most parameters should fall into “retain merging,” which is exactly the region where the paradox is most severe. Calculating kappa directly will yield “severe inconsistency among three criteria,” which may merely be another phrasing for “verdict distribution is too skewed.”

III. Three Categories Add Another Layer of Complexity

Verdicts are not binary but ternary: retain, stratify, or unknown. Under three categories, expected agreement is jointly determined by three marginal proportions, making it more sensitive than in binary cases; furthermore, calculating weighted kappa requires selecting weights (linear or quadratic), introducing another degree of freedom that alters the numerical value. Methodological comparisons conclude clearly: across various marginal scenarios in 2×2 tables, kappa and alpha are most affected by marginal distributions, while AC1 and B statistics are more robust, with the B statistic performing best under both paradoxes (PMID 25168681); the bias-adjusted and prevalence-adjusted kappa (PABAK) directly corrects for these two biases (PMID 8501467). This is not merely theoretical: in pathological diagnoses with high feature prevalence, kappa deviates significantly from the impression given by observed agreement (PMID 38283651).

IV. Agreement Does Not Equal Correctness

The three criteria share an unverified premise: the true value space is constructed. The weights in the synthesized index, the 0.1 and 0.25 thresholds in the per-variable diagnosis, and the true value distribution underlying the proportion metrics are all filled by modelers. Agreement among them only indicates they align under the same set of assumptions; it does not validate any one of them as correct.

Therefore, high kappa is not a positive sign: it implies the three lines provide no independent information. The real concern is when all three agree but are all wrong—a discrepancy that agreement rates can never detect.

V. The Third Criterion Is Not the Same Quantity

The first two criteria yield categorical verdicts per parameter, while the third—”proportion of conclusions holding”—is a continuous quantity: the proportion within the true value space where the conclusion remains valid. To force this continuous quantity into a three-tier verdict, a threshold must be selected. The threshold is a free parameter; setting it at 20% versus 30% alters the agreement rate.

Thus, “three-portrait kappa” is actually consistency between two categorical quantities and a continuous quantity thresholded arbitrarily. Shifting the threshold shifts the conclusion. This is the same disease as the previous criticism of renaming “discriminative ability” to “generalizability,” merely with a different entry point.

VI. Agreement Is Symmetric, But Consequences Are Not

Criterion A judges “downgrade” while B judges “retain,” versus A judging “retain” while B judges “downgrade”—these are identical in terms of agreement rates. Yet the consequences differ vastly: the former incurs extra sample costs, while the latter leads to over-extrapolation of conclusions. What should be reported is the count and direction of inconsistencies, not their magnitude.

Kappa itself suffers from substantial sampling variability. A corroborating example: even the question of “how many samples are needed to calculate kappa” has been addressed in literature using proportions of observed agreement rather than the kappa value itself (PMID 25088752). Using a ruler that shakes itself to judge whether two other rulers are consistent will not yield precision exceeding the ruler’s own instability.

VII. Change the Question

Four actions replace calculating a single kappa:

  1. Report all three original verdicts per parameter separately, without merging or summing scores—this aligns with the balanced diagnosis proposal: report standardized mean differences and variance ratios per variable without compressing them into a single metric (PMID 30788363).
  2. Report observed agreement, PABAK, and the B statistic simultaneously; discard any results based solely on kappa.
  3. Report the count and direction of inconsistencies, and calculate “the number of parameter lines that would alter design decisions.” This is the decision-relevant quantity, whereas insufficient interaction power renders modulation magnitude unestimable (PMID 15066682), and marginal/conditional estimates may even yield opposite decisions (PMID 41626969).
  4. Use bootstrapping to generate confidence intervals for agreement rates. If the interval width exceeds 0.3, a sample size of 60 per group is insufficient to support consistency conclusions; samples must be expanded.

Four criteria correspond to four outcomes: if observed agreement exceeds 85% while kappa is below 0.4, the paradox is confirmed, kappa is a false alarm, and the conclusion that “three portraits are not interchangeable” must be retracted; if PABAK remains below 70% even after correction, there is indeed inconsistency, and the three portraits must be reported separately with range loss values prohibited for cross-study comparison; if shifting the third criterion’s threshold between 20% and 30% causes over 20% of parameters to cross tiers, it lacks criterion qualification and should only serve as a sensitivity analysis; if the bootstrap interval width exceeds 0.3, sample size issues precede consistency issues.

In one sentence: Whether criteria agree must be measured with a ruler unaffected by distribution shape; measuring with a biased ruler always yields the ruler’s own bias.

Evidence and Limitations

Verified: Cohen’s kappa exhibits two structural paradoxes—high observed agreement is drastically suppressed under marginal imbalance, and kappa can be artificially higher under marginal asymmetry, neither of which is fixed by replacing kappa with kappa max (PMID 2348207); systematic comparisons of various agreement coefficients in 2×2 tables show kappa and alpha are most affected by marginal distributions, with AC1 and B statistics being more robust and the B statistic performing best under both paradoxes (PMID 25168681); PABAK provides direct correction for these two biases (PMID 8501467); sample size planning practices for kappa have already shifted to using proportions of observed agreement rather than the kappa value itself (PMID 25088752).

Partially Verified: The phenomenon of kappa deviating significantly from observed agreement under high-prevalence features has been quantified in empirical pathological diagnosis data (PMID 38283651), but the subjects were lung tissue histological features and readers were pathologists, differing from the verdict scenario of Dantian parameters; balanced diagnosis should report per-variable standardized mean differences and variance ratios rather than compressing them into a single metric (PMID 30788363), a proposal originating from propensity score matching fields, which constitutes an analogy when transferred to this project; insufficient interaction test power renders modulation magnitude unestimable (PMID 15066682), and the coexistence of effect modification and incompressibility can cause marginal and conditional estimates to yield opposite decisions (PMID 41626969), both supporting “direction matters more than magnitude,” yet based on clinical trial and review contexts.

To Be Verified: The actual verdict distribution of the three criteria on the same Dantian dataset has never been measured—”most parameters fall into retain merging” is merely an expectation; if the empirical distribution proves balanced, paradox risk drops significantly, weakening this paper’s main argument; no simulation data exists for the specific deviation magnitude of kappa under three-tier verdicts; no empirical data exists for the numerical difference between PABAK and B statistics in this project’s data; the sensitivity curve for the third criterion’s threshold (proportion of parameters crossing tiers at 20%, 25%, and 30%) has never been calculated; the layer “agreement does not equal correctness” relies solely on argumentation and cannot be tested by any agreement rate metric.

Related Reading

Evidence Level: C
References: PMID 2348207, PMID 25168681, PMID 8501467, PMID 25088752, PMID 38283651, PMID 30788363, PMID 15066682, PMID 41626969
L1 Network Status: This topic has not yet entered the L1 corresponding network—the CORRESPONDENCE_NETWORK.md node table assigns nodes by domain (Yijing/TCM side, Physics side, Physiological/Experiential side) based on concepts/quantities (Five Phases, Five Organs, Frequencies, Colors, etc.). Currently, none of the three domains contain measurement or judgment layer nodes related to “Criterion Consistency / Verdict Categories / Agreement Coefficients / Marginal Distribution Bias.” The edge table ends at E-050 (“Six Heavenly Stems and Earthly Branchs Na-Yin | Corresponds | 60 Frequencies,” Level L1, Status open; modern evidence column originally states “No measurable sources at all“). There are no edges originating from or targeting criteria, verdicts, or agreement rates. EDGE_TABLE_science-yixue.md is a reverse statistical extraction listing only concept counts (51 Dantian articles / 48 Qigong articles / 189 Breathing articles / 407 Frequency articles) and highest evidence levels, lacking L1/L2/L3 hierarchies or active/open/conflict status determinations; therefore, the status of this edge is marked as missing.


Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *