Evidence level: C
固定”十次里有九次同意”,kappa 不是九成、也不是零点几,而是可以走到负数——决定它落在哪里的,只有一个量:边际分布。
正文
一个被反复引用、却从未被量化的问题
气功外气的判读用三档裁决:改善 / 无变化 / 恶化。三位判读者各自给档,再算一致率,最常报的指标是 Cohen’s kappa。
kappa 有个老毛病,叫”高一致率、低 kappa”。Feinstein 与 Cicchetti 把它写成两类悖论:边际严重偏斜时很高的观察一致率 p0 会被大幅压低;而不对称的偏斜反而给出比对称更高的 kappa(PMID 2348207)。他们的结论很干脆:用 kappa_max 替换 kappa 修不好,甚至会让第二个更糟。Lantz 与 Nebenzahl 后来形式化:给定任一 p0,kappa 只有下界、上界、最大值三个特征值(PMID 8621993)。Byrt 等人则把 kappa 拆成偏倚、流行度、观察一致三块,建议别只报一个数(PMID 8501467)。
但这些文献都停在二分类四格表。三档下机会一致率由三个边际共同决定,”偏”可以偏在任意一档。三档的崩塌曲线长什么样、偏到多少才塌,没人算过。
关键一步:kappa 和”对角线怎么摆”无关
先把一个常被搞混的点说清。三分类 kappa = (p0 − pe)/(1 − pe),其中 pe = Σ pᵢqᵢ 只由两位评分者各自的边际 p、q 决定。只要 p、q、p0 定住,kappa 就定住——对角线三个格子内部怎么分配,对 kappa 毫无影响。这让”模拟”变得极轻:不必构造整张 3×3 表,扫边际向量即可。
同时带出一条硬约束:Σ min(pᵢ, qᵢ) ≥ p0。观察一致率不能超过两人在同一档上的最小机会。所以”极端不对称 + 高一致率”在物理上不存在——90/5/5 与 75/20/5 这一对(Σmin = 0.85)在 p0 = 0.90 时根本不可能出现。这一条决定了整个可行域边界。
于是可以闭式反解:κ = (p0 − pe)/(1 − pe) 移项得 pe = (p0 − κ)/(1 − κ)。取 p0 = 0.90:想保住 κ = 0.6,pe 不能超 0.750;κ = 0.5 → 0.800;0.4 → 0.833;0.3 → 0.857*;0.1 → 0.889;0.0 → 0.900。
崩塌曲线:主档 92.4% 是 0.3 的生死线
三档边际设成 a / (1−a)/2 / (1−a)/2,扫 a,p0 固定 0.90:
| 边际(%) | pe | kappa | AC1 | PABAK |
|---|---|---|---|---|
| 33.3 / 33.3 / 33.3 | 0.333 | 0.850 | 0.850 | 0.850 |
| 60 / 20 / 20 | 0.440 | 0.821 | 0.861 | 0.850 |
| 80 / 10 / 10 | 0.660 | 0.706 | 0.880 | 0.850 |
| 90 / 5 / 5 | 0.815 | 0.459 | 0.890 | 0.850 |
| 92.4 / 3.8 / 3.8 | 0.857 | 0.300 | 0.892 | 0.850 |
| 94.8 / 2.6 / 2.6 | 0.900 | −0.001 | 0.895 | 0.850 |
| 96 / 2 / 2 | 0.922 | −0.289 | 0.896 | 0.850 |
| 98 / 1 / 1 | 0.961 | −1.538 | 0.898 | 0.850 |
最刺眼的一列不是 kappa,是 PABAK:整条区间一动不动,恒等于 0.850。因为 PABAK = (3p0 − 1)/2 只依赖 p0,压根没打算看边际。AC1 也几乎不动(0.850 → 0.898)。
所以曲线真正的含义不是”kappa 不可靠”,而是更狠的一句:p0 固定时,三档 kappa 的全部变化都来自边际偏斜,不来自判读质量。主档从 33% 走到 92.4%,判读者一次没变得更差,kappa 掉了 55 个百分点。
三个反直觉推论
一、多分类不比二分类危险,反而更安全。 均衡边际下 p0 = 0.90:k = 2 时 kappa 0.800,k = 3 时 0.850,k = 6 时 0.880。类别越多,机会一致率越低,挤掉的越少。危险不来自档位数,只来自某一档吃掉太大份额。
二、不对称让 kappa 变好看,第二悖论原样成立。 固定 p0 = 0.90、A 判读者 90/5/5,让 B 的边际向第二档倾斜:B = 90/5/5 时 κ = 0.459,88/7/5 时 0.505,85/10/5 时 0.560,82/13/5 时 0.605。两人分歧越大,kappa 越高。任何拿”边际基本对称”替 kappa 辩护的话,在这里直接失效。
三、偏斜形态几乎无关,只有集中度有关。 偏斜摊在两档(3.8/3.8/92.4)时主档门槛合计 7.57%,偏在单档时 3.79/3.79/92.43,两者差不到 0.01 个百分点。只需知道最大那一档占多少,就能判断这条 kappa 有没有意义——可以写成分析模板里的一行判据。
对三档裁决意味着什么
一是先算边际,再读 kappa。某一档占到 90% 以上,p0 = 0.90 时 kappa 已注定低于 0.46;占到 92.4% 就是 0.3。此时报”信度中等”是把结构性问题误读成质量问题。
二是kappa 必须与 PABAK / AC1 / B 并列。AC1 正是为对抗这类悖论设计的,方差估计更稳(PMID 18482474);Wongpakaran 等人在 67 名患者、9 位评分者的实测中,AC1 落在 0.752–1.000 而 kappa 覆盖 0–1.00,且 AC1 对流行度几乎不敏感(PMID 23627889)。这与曲线完全吻合。
三是样本量规划不能拿 kappa 当目标——它的精度同时被 p0 与边际比例撕扯(PMID 25088752)。
这也正是三档判读与易学接口的卡点:判读信度这道前置门不先校准,”气感进退”的任何方向性结论都站在流沙上。
证据与局限
已验证:三分类 kappa 完全由 (p, q, p0) 决定,与对角线内部分配无关——这是定义直接推出的解析结论,可闭式反解 pe* = (p0 − κ)/(1 − κ),本页数值表即由此算得。
部分验证:kappa 悖论的存在与方向有多篇方法学文献支撑(PMID 2348207、8621993、8501467、25168681);AC1 较 kappa 更稳,有模拟与实测两侧证据(PMID 18482474、23627889)。
待验证:① 曲线是在”均等判对率”假设下算的——真实判读里主档往往更容易判对,p0 的构成会偏向主档,形态需重算;② 92.4% 这个门槛只在 p0 = 0.90 下成立,我们的实际 p0 还没测;③ PABAK 在 p0 固定时恒定,意味着它对”判读者真变差”与”边际变偏”完全无分辨力——替代指标不是免费的;④ 有序加权 kappa 在三档下的减压幅度未算,只知四档实测 0.743(PMID 16534051)。
L1 网络状态:该主题尚未进入 L1 对应网络——CORRESPONDENCE_NETWORK.md 边表止于 E-050(纳音↔频率,状态 open,现代证据栏”尚无任何可测来源”),全表无”观察一致率 / kappa / 重测信度 / 判读者间一致性”任何一条边;EDGE_TABLE_science-yixue.md 概念表亦无”信度”节点,故本文证据等级封顶 C。
相关阅读
证据等级: C
参考文献: PMID 2348207, PMID 8621993, PMID 8501467, PMID 25168681, PMID 18482474, PMID 23627889, PMID 25088752, PMID 16534051, PMID 22897413
免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。
Observation Consistency at 90%, Kappa Drops to Negative: The Collapse Curve of Three-Tier Arbitration and Its Sole Independent Variable
Evidence level: C
Fixing “agreement in 9 out of 10 cases” does not yield a kappa of 0.9 or even a small positive value; it can result in a negative number. The sole variable determining this outcome is the marginal distribution.
Body
A Repeatedly Cited but Never Quantified Problem
In the interpretation of Qigong external Qi (Qi), a three-tier judgment scale is employed: improvement / no change / deterioration. Three raters independently assign categories, and agreement is calculated, with Cohen’s kappa most commonly reported as the metric.
Kappa suffers from a well-known flaw termed “high agreement, low kappa.” Feinstein and Cicchetti formalized this into two paradoxes: when the marginal distribution is severely skewed, the observed agreement rate $p_0$ is drastically suppressed; conversely, asymmetric skew yields a higher kappa than symmetric skew (PMID 2348207). Their conclusion is stark: replacing kappa with $\kappa_{max}$ fails to fix the issue and may even worsen the second paradox. Lantz and Nebenzahl later formalized this: given any $p_0$, kappa possesses only three eigenvalues—lower bound, upper bound, and maximum (PMID 8621993). Byrt et al. decomposed kappa into bias, prevalence, and observed agreement, recommending against reporting a single number (PMID 8501467).
However, these studies stop at the two-category fourfold table. Under three categories, the opportunity agreement rate is jointly determined by three marginals, meaning “bias” can skew toward any single category. No one has calculated what the collapse curve looks like or how much skew is required to cause failure.
The Crucial Step: Kappa Depends Only on Margins, Not Diagonal Allocation
First, clarify a frequently confused point. The three-category kappa formula is $\kappa = (p_0 – p_e) / (1 – p_e)$, where $p_e = \sum p_i q_i$ is determined solely by the individual marginal distributions $p$ and $q$ of the two raters. Once $p$, $q$, and $p_0$ are fixed, kappa is determined—the internal allocation of the three diagonal cells has absolutely no effect on kappa. This makes simulation trivial: one does not need to construct the full $3 \times 3$ table; scanning the marginal vectors suffices.
This also establishes a hard constraint: $\sum \min(p_i, q_i) \ge p_0$. The observed agreement rate cannot exceed the minimum opportunity of both raters agreeing on any single category. Therefore, “extreme asymmetry + high agreement” is physically impossible—pairs such as 90/5/5 and 75/20/5 (where $\sum \min = 0.85$) cannot yield $p_0 = 0.90$. This constraint defines the boundary of the feasible region.
Thus, we can derive a closed-form inverse: rearranging $\kappa = (p_0 – p_e) / (1 – p_e)$ yields $p_e^ = (p_0 – \kappa) / (1 – \kappa)$. Setting $p_0 = 0.90$: to preserve $\kappa = 0.6$, $p_e$ must not exceed 0.750; $\kappa = 0.5 \rightarrow 0.800$; $0.4 \rightarrow 0.833$; $0.3 \rightarrow 0.857$*; $0.1 \rightarrow 0.889$; $0.0 \rightarrow 0.900$.
The Collapse Curve: The 92.4% Main Category Threshold is the Line Between Life and Death for $\kappa = 0.3$
Setting the three marginal distributions as $a / (1-a)/2 / (1-a)/2$ and scanning $a$ while fixing $p_0 = 0.90$:
| Margins (%) | $p_e$ | kappa | AC1 | PABAK |
|---|---|---|---|---|
| 33.3 / 33.3 / 33.3 | 0.333 | 0.850 | 0.850 | 0.850 |
| 60 / 20 / 20 | 0.440 | 0.821 | 0.861 | 0.850 |
| 80 / 10 / 10 | 0.660 | 0.706 | 0.880 | 0.850 |
| 90 / 5 / 5 | 0.815 | 0.459 | 0.890 | 0.850 |
| 92.4 / 3.8 / 3.8 | 0.857 | 0.300 | 0.892 | 0.850 |
| 94.8 / 2.6 / 2.6 | 0.900 | −0.001 | 0.895 | 0.850 |
| 96 / 2 / 2 | 0.922 | −0.289 | 0.896 | 0.850 |
| 98 / 1 / 1 | 0.961 | −1.538 | 0.898 | 0.850 |
The most striking row is not kappa, but PABAK: the entire interval remains unchanged, constantly equal to 0.850. Since PABAK = $(3p_0 – 1)/2$ depends only on $p_0$, it ignores marginal distribution entirely. AC1 also remains nearly static (0.850 → 0.898).
Therefore, the true meaning of the curve is not “kappa is unreliable,” but rather a sharper statement: when $p_0$ is fixed, all variation in the three-category kappa stems entirely from marginal skew, not from rater quality. As the main category shifts from 33% to 92.4%, the raters’ performance has not deteriorated at all, yet kappa drops by 55 percentage points.
Three Counterintuitive Implications
First, multi-category assessment is not more dangerous than two-category; it is actually safer. Under balanced marginals with $p_0 = 0.90$: when $k=2$, kappa is 0.800; when $k=3$, it is 0.850; when $k=6$, it is 0.880. Greater numbers of categories reduce the opportunity agreement rate, squeezing out less agreement. Danger does not arise from the number of categories, but from one category consuming too large a share.
Second, asymmetry makes kappa look better; the second paradox holds exactly. Fixing $p_0 = 0.90$ and Rater A at 90/5/5, let us skew Rater B’s margin toward the second category: when B = 90/5/5, $\kappa = 0.459$; at 88/7/5, $\kappa = 0.505$; at 85/10/5, $\kappa = 0.560$; at 82/13/5, $\kappa = 0.605$. The greater the disagreement between raters, the higher the kappa. Any defense relying on “marginal symmetry” fails completely here.
Third, the shape of skew is almost irrelevant; only concentration matters. When skew is distributed across two categories (3.8/3.8/92.4), the main category threshold sums to 7.57%; when concentrated in one category (3.79/3.79/92.43), the difference is less than 0.01 percentage points. Knowing only the proportion of the largest category is sufficient to determine whether this kappa is meaningful—this can be embedded as a single criterion in analysis templates.
What This Means for Three-Category Judgment
First, calculate marginals before interpreting kappa. If one category exceeds 90%, with $p_0 = 0.90$, kappa is inevitably below 0.46; at 92.4%, it drops to 0.3. Reporting “moderate reliability” in such cases misreads structural issues as quality problems.
Second, kappa must be reported alongside PABAK, AC1, and B. AC1 was specifically designed to counter such paradoxes, offering more stable variance estimates (PMID 18482474). In a study of 67 patients and 9 raters, Wongpakaran et al. found AC1 ranging from 0.752–1.000 while kappa spanned 0–1.00, and AC1 showed near-insensitivity to prevalence (PMID 23627889). This aligns perfectly with the curve.
Third, sample size planning cannot target kappa—its precision is simultaneously torn apart by $p_0$ and marginal proportions (PMID 25088752).
This is precisely the bottleneck where three-category judgment interfaces with Yijing Studies: without calibrating this reliability threshold first, any directional conclusion about “Qi sensation advancement or retreat” stands on shifting sand.
Evidence and Limitations
Verified: Three-category kappa is entirely determined by $(p, q, p_0)$ and independent of diagonal internal allocation—this is a direct analytical result from the definition, allowing closed-form inversion $p_e^* = (p_0 – \kappa)/(1 – \kappa)$, from which the numerical table on this page was derived.
Partially Verified: The existence and direction of the kappa paradox are supported by multiple methodological literature (PMID 2348207, 8621993, 8501467, 25168681); AC1 is more stable than kappa, with evidence from both simulation and real-world studies (PMID 18482474, 23627889).
To Be Verified: ① The curve assumes “equal correct judgment rates”—in real assessments, the main category is often easier to judge correctly, so $p_0$ composition skews toward the main category, requiring recalculation of the shape; ② The 92.4% threshold holds only when $p_0 = 0.90$, but our actual $p_0$ has not yet been measured; ③ PABAK remains constant when $p_0$ is fixed, meaning it cannot distinguish between “raters truly deteriorating” and “margins skewing”—alternative metrics are not free; ④ The stress-reduction effect of ordered weighted kappa under three categories has not been calculated, though four-category real-world data shows 0.743 (PMID 16534051).
L1 Network Status: This topic has not yet entered the L1 corresponding network—the CORRESPONDENCE_NETWORK.md edge table stops at E-050 (Na Yin ↔ Frequency, status open, with the “evidence column” stating “no measurable source yet”), and the full table lacks any edge for “observed agreement rate / kappa / retest reliability / inter-rater consistency”; the EDGE_TABLE_science-yixue.md concept table also lacks a “reliability” node, so the evidence level for this article is capped at C.
Related Reading
- Which standard deviation should kappa take: two out of three denominators erase the entire age trend
- Changing the form of the criterion, does the conclusion flip: flip rate and invariance under reparameterization
- Using two criteria together, is it really more than one? How to count independent evidence lines
- Environment-Life Frequency Interface (Research Pool Four)
Evidence Level: C
References: PMID 2348207, PMID 8621993, PMID 8501467, PMID 25168681, PMID 18482474, PMID 23627889, PMID 25088752, PMID 16534051, PMID 22897413
Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.
Leave a Reply