Evidence level: C
观察一致率 0.90 这件事,本身不携带任何关于「测得准不准」的信息。真正决定 kappa 的只有一个标量:边际集中度。它不参与计算,它直接决定结果。
正文
前一篇(kappa-paradox-three-tier-decision-margin-calibration)把三档裁决的崩塌曲线算了出来,得出「观察一致率 0.90 而边际 90/5/5 时,未加权 kappa 只有 0.459」。这个数字被当成三档裁决的典型崩塌量,进了本项目的取证清单。
但它只是曲线上一个点的取值。这篇要把它拆开看:那个 0.459 里,有多少是「三档裁决」这件事本身造成的,有多少只是那一个特定的边际分布造成的。
先把代数摆出来。三分类、两判读者、未加权 kappa 的定义和两分类完全一样:
κ = (p₀ − pₑ) / (1 − pₑ),其中 pₑ = Σₖ p₁ₖ · p₂ₖ
p₀ 是观察一致率,pₑ 是机会一致率,p₁ₖ 与 p₂ₖ 是两位判读者在第 k 档上的边际比例。式子只在分母上出现 pₑ,而 pₑ 不含任何关于「谁判对了哪一例」的信息——它只由两张边际分布决定,与对角线怎么摆无关。
这就得出第一个结论:固定 p₀ 之后,κ 是 pₑ 的单值函数,没有第二个自由度。 换句话说,p₀=0.90 这一句话,对 κ 的约束力是零。它把 κ 锁不住,因为 pₑ 可以在 1/3 到 1 之间自由滑动。
把 p₀=0.90 代回去,让边际从均匀(33/33/34,pₑ≈0.333)一路推到极端(95/2.5/2.5,pₑ≈0.903),κ 会这样走:
| 边际分布 | pₑ = Σp² | κ(p₀ 固定 0.90) |
|---|---|---|
| 33/33/34 | 0.333 | 0.850 |
| 40/35/25 | 0.345 | 0.847 |
| 60/30/10 | 0.460 | 0.815 |
| 75/12.5/12.5 | 0.594 | 0.754 |
| 80/10/10 | 0.660 | 0.706 |
| 90/5/5 | 0.815 | 0.459 |
| 92/4/4 | 0.850 | 0.335 |
| 95/2.5/2.5 | 0.904 | −0.039 |
同一句「一致率九成」,κ 从 0.85 一路掉到负值,中间没有任何一处涉及「判得对不对」。这就是 Feinstein 与 Cicchetti 三十多年前说的第一悖论(PMID 2348207):观察一致率会被边际的纵向或横向不平衡严重改写,而用 kappa max 替换 kappa 并不能修复。
第二个结论更硬:κ<0 的充要条件是 Σp² > p₀。p₀=0.90 时,零点卡在边际集中度 0.90。也就是说,只要主档占九成,无论判读者多准,κ 必为负。这句是可解析的,不需要模拟。
现在回到 0.459 是怎么来的。它对应的边际是 90/5/5,即主档占了九成。在真实的气功外气三档裁决里,主档(无变化)占九成是不是常态? 这一条从来没被实测过,而它决定了 0.459 是「普遍现象」还是「极端案例」。
我们把判对率按住,只让主档占比 m 从 0.4 爬到 0.96(设主档判对率 0.95、两侧各 0.75,这是一个偏乐观的设定):
| 主档占比 m | p₀ | pₑ | κ |
|---|---|---|---|
| 0.40 | 0.830 | 0.340 | 0.742 |
| 0.60 | 0.870 | 0.440 | 0.768 |
| 0.75 | 0.900 | 0.594 | 0.754 |
| 0.80 | 0.910 | 0.660 | 0.735 |
| 0.85 | 0.920 | 0.734 | 0.700 |
| 0.92 | 0.934 | 0.850 | 0.561 |
| 0.96 | 0.942 | 0.922 | 0.253 |
κ 随 m 单调下降,但在 m=0.4 到 0.80 之间几乎是平的(0.74–0.77)。真正的膝部在 0.85 之后:m 从 0.85 再涨 0.11 到 0.96,κ 掉了 0.45。
这条曲线的含义很实用:主档占比低于八成时,三档裁决对悖论基本免疫;越过 0.85 才开始塌,越过 0.92 就塌成废值。 所以「三档裁决的 kappa 恒为假警报」这个论断不能成立——它成立与否,取决于一个从未测过的数字:主档占比。
顺便验一下公式的可靠性。前一篇还实测过一个数:A 判读者固定 90/5/5、B 的边际偏到 82/13/5 时,κ 从 0.459 升到 0.605。用 pₑ=Σp₁ₖp₂ₖ 算:0.9×0.82 + 0.05×0.13 + 0.05×0.05 = 0.747,κ = 0.153/0.253 = 0.605。与实测完全对上。0.459 同样对得上(pₑ=0.815,κ=0.085/0.185=0.4595)。两个点都对,说明这套代数可以当作三档裁决的通用换算器用,不必每次重跑模拟。这个对上的过程本身还解释了 Feinstein 的第二悖论:两位判读者分歧越大(边际越不对称),κ 反而越高。因为 pₑ=Σp₁ₖp₂ₖ 是两张边际的点积,把其中一张推离另一张,点积就掉,分母 (1−pₑ) 变大而分子 (p₀−pₑ) 同向变大——κ 被抬高。所以 κ 高不是「更一致」,可能只是「两个人都偏得不一样」。
最后是可操作的落点。既然 p₀ 单独不可解释、Σp² 单独不可解释,两者拼起来才闭合,那么三档裁决的报告规范应该是三个数一起报:(p₀, Σp², κ)。缺任何一项,κ 都读不出意思——只报 κ=0.60,我们无法判断它是「真一致 0.60」还是「一致率 0.93 被 0.90 的主档边际压下来的 0.60」。Byrt 等人(PMID 8501467)与 Shankar、Bangdiwala(PMID 25168681)走的正是这条路:把 κ 拆成偏倚、流行度、观察一致率三个独立指数来报,而不是只给一个合成值;后者还发现 B 统计量、AC1 与 delta 对边际的敏感度明显低于 κ。样本量规划同样该换口径(PMID 25088752)。
L1 网络状态:该主题尚未进入 L1 对应网络。
CORRESPONDENCE_NETWORK.md全表检索「信度 / 重复 / 方差 / 测量」零命中;丹田仅在节点表出现一次(「胃肠蠕动 | 主观+听诊 | 「气沉丹田」的候选物」),无任何承重边连到判读信度。网络中| L2 以上边数 | 有「操作+观测量+阈值」的边 | 0 |——即本文明示的 (p₀, Σp², κ) 阈值三角,在 L1 里尚无对应边可挂。
证据与局限
已验证:κ=(p₀−pₑ)/(1−pₑ) 的代数恒等;p₀ 固定时 κ 对 pₑ 的单值性;κ<0 ⇔ Σp²>p₀(两判读者边际相同时);用该式复现前篇两个实测点(0.459 与 0.605,误差 <0.001)。悖论本身有 1990 年以来的独立文献支撑(PMID 2348207、8501467、25168681)。
部分验证:主档占比 m 的膝部位置(0.85–0.92)依赖「主档判对率 0.95、两侧 0.75」这一组假定;换一组判对率,膝部会平移。所以膝部数值只能当量级读,不能当阈值用。
待验证:① 真实气功外气三档裁决的主档占比从未实测,0.459 是普遍现象还是极端案例仍悬空;② 加权 kappa 下 Σp² 是否仍是唯一自变量(本篇只处理未加权);③ 主档占比与判对率是否独立(若主档易判本身就是「没人敢判变化」的产物,则 p₀ 与 pₑ 会耦合,上述固定判对率的表会失效)。
证据等级: C
参考文献: PMID 2348207, PMID 25168681, PMID 8501467, PMID 25088752
相关阅读
免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。
Kappa has only one independent variable: the observed agreement of 0.90, which can correspond to any value between -0.11 and 0.85.
Evidence level: C
The observation agreement rate of 0.90 carries no information regarding “how accurate the measurement is.” The sole scalar determining kappa is marginal concentration. It does not participate in the calculation; it directly determines the result.
Body
The previous article (kappa-paradox-three-tier-decision-margin-calibration) computed the collapse curve for three-tier verdicts, yielding “an observation agreement rate of 0.90 with margins 90/5/5 results in an unweighted kappa of only 0.459.” This figure was entered into the evidence list for this project as a typical collapse value for three-tier verdicts.
However, it is merely a point on the curve. This article dissects it: how much of that 0.459 is due to the “three-tier verdict” itself, and how much is due solely to that specific marginal distribution.
Let us lay out the algebra. The definition of unweighted kappa for three categories, two raters, and two readers is identical to the binary case:
κ = (p₀ − pₑ) / (1 − pₑ), where pₑ = Σₖ p₁ₖ · p₂ₖ
p₀ is the observation agreement rate, pₑ is the chance agreement rate, and p₁ₖ and p₂ₖ are the marginal proportions of the two raters in category k. The formula involves pₑ only in the denominator, and pₑ contains no information about “who judged which case correctly”—it is determined entirely by the two marginal distributions and is independent of how the diagonal is arranged.
This yields the first conclusion: once p₀ is fixed, κ is a single-valued function of pₑ and has no second degree of freedom. In other words, the statement “p₀ = 0.90” exerts zero constraint on κ because pₑ can freely vary between 1/3 and 1.
Substituting p₀ = 0.90 and varying the margins from uniform (33/33/34, pₑ ≈ 0.333) to extreme (95/2.5/2.5, pₑ ≈ 0.903), κ evolves as follows:
| Marginal Distribution | pₑ = Σp² | κ (with fixed p₀=0.90) |
|---|---|---|
| 33/33/34 | 0.333 | 0.850 |
| 40/35/25 | 0.345 | 0.847 |
| 60/30/10 | 0.460 | 0.815 |
| 75/12.5/12.5 | 0.594 | 0.754 |
| 80/10/10 | 0.660 | 0.706 |
| 90/5/5 | 0.815 | 0.459 |
| 92/4/4 | 0.850 | 0.335 |
| 95/2.5/2.5 | 0.904 | −0.039 |
The same statement “90% agreement rate” sees κ drop from 0.85 to negative values, with no intermediate step involving “judging correctly or not.” This is the first paradox noted by Feinstein and Cicchetti over thirty years ago (PMID 2348207): the observation agreement rate is severely distorted by vertical or horizontal imbalance in the margins, and replacing kappa with kappa max does not repair it.
The second conclusion is even stronger: κ < 0 if and only if Σp² > p₀. With p₀ = 0.90, the zero point occurs at marginal concentration 0.90. That is, as long as the main category occupies 90%, regardless of how accurate the raters are, κ must be negative. This is analytically solvable and does not require simulation.
Now back to how 0.459 arises. It corresponds to margins of 90/5/5, meaning the main category occupies 90%. In real qigong external qi three-tier verdicts, is the main category (no change) occupying 90% the norm? This has never been empirically tested, yet it determines whether 0.459 is a “general phenomenon” or an “extreme case.”
We hold the accuracy rate fixed and let the main category proportion m rise from 0.4 to 0.96 (setting the main category accuracy at 0.95 and both side categories at 0.75, an optimistic assumption):
| Main Category Proportion m | p₀ | pₑ | κ |
|---|---|---|---|
| 0.40 | 0.830 | 0.340 | 0.742 |
| 0.60 | 0.870 | 0.440 | 0.768 |
| 0.75 | 0.900 | 0.594 | 0.754 |
| 0.80 | 0.910 | 0.660 | 0.735 |
| 0.85 | 0.920 | 0.734 | 0.700 |
| 0.92 | 0.934 | 0.850 | 0.561 |
| 0.96 | 0.942 | 0.922 | 0.253 |
κ decreases monotonically with m, but it is nearly flat (between m = 0.4 and 0.80, ranging from 0.74 to 0.77). The true “knee” occurs after m = 0.85: as m rises from 0.85 to 0.96 (an increase of 0.11), κ drops by 0.45.
The meaning of this curve is practical: when the main category proportion is below 80%, three-tier verdicts are largely immune to the paradox; collapse begins only after exceeding 0.85, and becomes worthless once surpassing 0.92. Therefore, the claim “kappa for three-tier verdicts is always a false alarm” cannot hold—it depends on an untested number: the main category proportion.
Incidentally, let us verify the formula’s reliability. The previous article also tested another point: when rater A is fixed at 90/5/5 and rater B’s margins shift to 82/13/5, κ rises from 0.459 to 0.605. Using pₑ = Σp₁ₖp₂ₖ: 0.9×0.82 + 0.05×0.13 + 0.05×0.05 = 0.747, κ = 0.153/0.253 = 0.605. This matches the empirical result exactly. The same holds for 0.459 (pₑ = 0.815, κ = 0.085/0.185 = 0.4595). Since both points align, this algebra can serve as a universal converter for three-tier verdicts without re-running simulations each time. This alignment also explains Feinstein’s second paradox: the greater the disagreement between two raters (i.e., the more asymmetric the margins), the higher κ becomes. Because pₑ = Σp₁ₖp₂ₖ is the dot product of two marginals, pushing one away from the other reduces the dot product, enlarging the denominator (1 − pₑ) while the numerator (p₀ − pₑ) increases in the same direction—thus κ is elevated. So a high κ does not mean “more consistent”; it may simply mean “both raters are biased in different directions.”
Finally, the actionable takeaway. Since p₀ alone is uninterpretable, Σp² alone is uninterpretable, and only their combination closes the loop, the reporting standard for three-tier verdicts should include all three numbers: (p₀, Σp², κ). Omitting any renders κ meaningless—for instance, reporting only κ = 0.60 leaves us unable to determine whether it represents “true agreement of 0.60” or “an observation rate of 0.93 suppressed to 0.60 by the main category margin of 0.90.” Byrt et al. (PMID 8501467) and Shankar, Bangdiwala (PMID 25168681) follow this path: decomposing κ into three independent indices—bias, popularity, and observation agreement—rather than a single composite value; the latter found that B statistics, AC1, and delta are significantly less sensitive to margins than κ. Sample size planning should also shift its metric (PMID 25088752).
L1 Network Status: This topic has not yet entered the corresponding L1 network. A full search of
CORRESPONDENCE_NETWORK.mdfor “reliability / repetition / variance / measurement” yields zero hits; the dantian appears only once in the node table (“intestinal peristalsis | subjective + auscultation | candidate for ‘qi sinking into the dantian’”), with no load-bearing edges connecting to judgment reliability. In the network,| Number of edges above L2 | Edges with 'operation + observation + threshold' | 0 |—that is, the explicitly stated (p₀, Σp², κ) threshold triangle in this text has no corresponding edge to attach to in L1.
Evidence and Limitations
Verified: The algebraic identity κ = (p₀ − pₑ)/(1 − pₑ); the single-valuedness of κ with respect to pₑ when p₀ is fixed; the equivalence κ < 0 ⇔ Σp² > p₀ (for identical marginal distributions of two raters); and the successful reproduction of the two empirical points from the previous article (0.459 and 0.605, with error < 0.001). The paradox itself is supported by independent literature dating back to 1990 (PMIDs 2348207, 8501467, 25168681).
Partially Verified: The “knee” position of the main category proportion m (0.85–0.92) depends on the assumption of “main category accuracy 0.95, both sides 0.75”; changing this accuracy shifts the knee. Thus, the knee value should be read as an order-of-magnitude indicator, not a strict threshold.
To Be Verified: ① the main category proportion in real qigong external qi three-tier verdicts has never been empirically tested, so it remains unknown whether 0.459 is a general phenomenon or an extreme case; ② under weighted kappa, whether Σp² remains the sole independent variable (this article handles only unweighted cases); ③ whether the main category proportion and accuracy rate are independent (if the main category is easy to judge, it may be a product of “nobody daring to judge changes,” causing p₀ and pₑ to couple, rendering the tables with fixed accuracy rates invalid).
Evidence Level: C
References: PMID 2348207, PMID 25168681, PMID 8501467, PMID 25088752
Further Reading
- Observation Agreement Rate at 90%, Yet Kappa Drops to Negative
- If All Three Verdict Criteria Agree, Are They Three Independent Evidences?
- Research Pool
Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.
Leave a Reply