Evidence level: C
一个临界限是”大”还是”小”,不能靠直觉,只能拿实测抖动去比。比完之后,原来那句”小到没人当回事”变成了它自己的反证。
正文
上一篇(#1393)把结论压在一个数上:主档占比 m=0.75 时,两位判读者只要把尺子整体挪 0.167 个标准差,就能在打分标准差之比 κ_s 恒等于 1.000(尺度完全一致、判合格)的前提下,把两人对主档占比的判断差 δ 推到 5 个百分点——正好撞线。上一篇接着说:「这个错位量小到任何校准训练都不会当回事。」
这句话是本篇要审的。0.167 没错,错的是”小”这个形容词——它没有分母。 要让”小”有意义,必须回答:真人重复判读时位置实际会漂多少?库里从没量过。
一、两个”位置”必须拆开
固定错位:甲读者天然比乙严,零点稳定分开一段距离,同一次判读里是常数。自身漂移:同一人隔段时间再判同一批,零点还会移动,是随机量。0.167 SD 说的是两者之差,而可实测的是第二种。
在潜变量阈值模型里(受试者真值 SD 记为 1),一名读者的重测信度与他严厉度漂移标准差有直接关系:σ_drift = √(1−ICC);两人彼此独立时,差的标准差 σ_pair = σ_drift×√2。
二、拿实测 ICC 换算,得到 0.4 左右
判读文献里 ICC 报得最勤。取两组实测:脊柱肌萎缩 RHS 量表、21 名资深物理治疗师、间隔两周看同一批录像,读者内 ICC 0.922–1.0(PMID 36542615);压力痛阈与时间总和、24 人同会话重复三次,ICC 0.80–0.97(PMID 33434233)。
| 实测 ICC | σ_drift | σ_pair | 是 0.167 的几倍 |
|---|---|---|---|
| 0.995 | 0.071 | 0.100 | 0.60 |
| 0.980 | 0.141 | 0.200 | 1.20 |
| 0.950 | 0.224 | 0.316 | 1.89 |
| 0.920 | 0.283 | 0.400 | 2.39 |
| 0.900 | 0.316 | 0.447 | 2.68 |
| 0.850 | 0.387 | 0.548 | 3.28 |
文献区间(ICC 0.80–0.97)对应 σ_pair 0.245–0.632,是 0.167 的 1.5 到 3.8 倍。反解更直白:要 σ_pair 不超过 0.167,需要 ICC ≥ 0.986,而多数判读任务在 0.80–0.95。
另一条独立路径交叉验证:教育测量报告评分者严厉度实测散布(以能力 SD 为单位)为 0.27–0.37,随机配对两人之差即 0.38–0.52,同样落在 2.3–3.1 倍区间。两条路径同量级。
三、”撞线”不是偶发,是常态
δ 的抽样分布近似正态,标准差 = φ(c)×σ_pair×100(m=0.75 时 φ=0.318)。代入:
| σ_pair | δ 的 SE | 超过 5pp 的概率(m=0.75) | 同(m=0.85) |
|---|---|---|---|
| 0.100 | 3.2 pp | 1.4% | 0.3% |
| 0.167 | 5.3 pp | 31.8% | 14.2% |
| 0.245 | 7.8 pp | 49.5% | 26.0% |
| 0.400 | 12.7 pp | 67.6% | 46.6% |
| 0.548 | 17.4 pp | 76.0% | 57.0% |
按文献中心值 σ_pair≈0.44(ICC 约 0.90),m=0.75 时撞线概率约 67%,δ 的期望绝对值约 11 pp。这不是”会不会撞线”,而是”绝大多数时候本来就在线外”。上一篇那句「κ_s 合格在实践中够用」必须撤回:κ_s 只保证尺子一样长,不保证在同一高度;而尺子不在同一高度是常态。
四、这个量还测不准,且加病例救不了
漂移的估计精度只由重复次数决定。个体漂移用 r 次重复估计时自由度是 r−1,相对标准误约 1/√(2(r−1)):3 次 50%、5 次 35%、10 次 24%、20 次 16%。要压到 ±20% 以内,得把多名读者池化(3 次×7 人 或 5 次×4 人)。是靠加读者,不是加病例——这与 #1403 的教训是同一件事的两面:位置参数的自由度来自读者数。
五、最要紧的一层:判据线正好压在噪声底上
线性近似下,δ 到 5pp 所需错位 b≈5/(100·φ)。而 δ̂ 的标准误是 SE=100·φ·σ_pair。把 σ_pair 代成 b,SE 精确等于 5 pp。各 m 验证:m=0.60 时 5.10、0.75 时 5.31、0.85 时 5.72、0.90 时 6.38——都在 5 附近。
判据线恰好落在 δ̂ 的噪声底上,信噪比约 1。 这不是巧合,是一条恒等式:判定线的位置和测量它所需精度的门槛由同一个 φ·b* 决定。凡是”δ 危险到能撞线”的研究,恰恰是”δ 测不准”的同一批。
由此得到与样本量无关的下限。σ_pair=0.167 时(已是”刚好撞线”配置),n 从 60 加到 3840:合计 SE 依次 9.52 / 7.71 / 6.62 / 6.00 / 5.66 / 5.40 pp,n 趋于无穷也停在 5.31 pp。漂移是系统性的、不可平均——N 个受试者共享同一对读者,两个零点偏移产生的效果同向、不随 n 稀释。这是”增加样本量”这条最常用补救在这里完全失效的原因。若要 SE 压到 1.79 pp(80% 功效检出 5pp),需 σ_pair≤0.040,即 ICC≥0.9984,判读研究达不到。
六、改法:报告项从 δ 换成 σ_Δ
既然 δ 的绝对值在现实信度下测不准,判据就不该建在 δ 上。改成直接报漂移:σ_Δ 作为主报告项,δ 只作派生量。成本极低——池化估到 ±20% 只需 3 次×7 名读者,比任何一次一致性研究都小。
| σ_Δ 区间 | δ̂ 的 SE | 该做什么 |
|---|---|---|
| <0.040 | <1.8 pp | δ 可测、5pp 门槛有效 |
| 0.040–0.118 | 1.8–5.3 pp | 门槛可判,须报 CI |
| 0.118–0.167 | 5.3–7.5 pp | 门槛失效,只报漂移 |
| >0.167 | >7.5 pp | κ_s 合格这个结论不承载信息 |
一个设计坑:想用”同一批病例多时点重复”分离漂移与病例难度,理论可行(难度是常数、漂移是时点效应),但一旦难度与时点有交互——判读者记住上次答案、或连续判读疲劳——两者就不分离,退化成 #1406 的记忆效应问题。这条路线必须同时报重复间隔。
证据与局限
已验证:漂移与 ICC 的换算 σ_drift=√(1−ICC)、σ_pair=σ_drift·√2 是潜变量模型的直接推论;文献实测 ICC(0.80–0.97,PMID 33434233;0.922–1.0,PMID 36542615)与严厉度散布(0.31–0.42 logits / 能力 SD 1.14)两条独立路径互校后同量级;漂移估计的自由度只来自重复次数(r−1);SE(σ_pair=b*)≈5pp 的恒等式在 m=0.60–0.90 全部成立(近似误差 +0.10 至 +1.38pp)。
部分验证:文献 ICC 来自量表类判读(物理功能评分、痛阈),与”气功外气变化幅度”这类综合印象判断的相似度只是类比——后者通常信度更低、漂移更大,但无直接数据。另有教育测量场景”115 名评分者仅 1 人跨轮次显著变化”的实测,反解出的漂移/SE 比约 0.65,与 ICC 路径的 0.8 同量级,但属单一场景、n 小,只能作旁证。
待验证:气功外气判读任务的 ICC 从未测过,所有换算都是跨领域外推,须先做一次最小规模实测(≥20 名判读者 × ≥60 段材料 × 各 3 次)才能定标;漂移是否有共同模会显著改变结论(ρ=0.9 时撞线率从 70% 降到 9%,各漂各的则前述全成立),该相关系数无实测;漂移是随时间单调累积还是随机游走,决定”定期重校准”是否有效(随机游走步长 0.05 SD 时约 11 轮漂回 0.167,步长 0.15 SD 时一轮半就回来)。
L1 网络状态:L1 边表中无「判读者一致性 / 位置漂移 / 信度」对应边。检索仅命中 CORRESPONDENCE_NETWORK.md 第 192 行「| 生克一致性 | 相生/相克方向能否复现 | 未测 |」(指五行生克方向,与本主题无关),以及 EDGE_TABLE 中「六十甲子 | 13 篇 | A 级 2 篇」的干支纪日可靠性条目(亦无关)。该主题尚未进入 L1 对应网络。
相关阅读
- 合格区里的尺子本来就够不到那条线:κ_s 拼尽全力只造得出 2.5 个百分点,而它要拦的是 5 个
- 谁的尺子歪了,这个问题数学上没有答案:一个恒亏一秩的模型,加多少判读者都补不上
- 两个自由度的不对称:尺度错有天花板、位置错没有
- 共振意识场:气功外气的可测性边界
证据等级: C
参考文献: PMID 36542615, PMID 33434233, PMID 36284065, PMID 37283589, PMID 8501467, PMID 35670017
证据等级:A=有PubMed直接支撑,B=科学类比/间接证据,C=传统文化共识/初步探索
免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。
How far the reader’s ruler actually floats: the critical threshold of \”0.167 standard deviations hitting the line\” differs from real human measurements by two to three times
Evidence level: C
A borderline threshold is “large” or “small” not by intuition, but only by comparing it to the measured jitter. After comparison, the original claim that “it is so small nobody cares” becomes its own counter-evidence.
Body
The previous post (#1393) pressed the conclusion onto a single number: when the main archive proportion $m=0.75$, two judges can shift their ruler by 0.167 standard deviations and still hit the line, provided the scoring standard deviation ratio $\kappa_s$ remains exactly 1.000 (scales perfectly aligned, deemed qualified). The previous post then stated: “This misalignment is so small that any calibration training would not take it seriously.”
This sentence is what this post must examine. 0.167 is correct; the error lies in the adjective “small”—it lacks a denominator. To make “small” meaningful, one must answer: how much does the position actually drift when real people repeat the judgment? This has never been measured in the literature.
I. Two “positions” must be separated
Fixed offset: Reader A is naturally stricter than Reader B, with a stable zero-point separated by a fixed distance; within the same judgment session this is a constant. Self-drift: The same person, after some time, re-judges the same batch; the zero-point shifts again, representing a random quantity. The 0.167 SD refers to the difference between these two, whereas what can be measured is the second type.
In latent-variable threshold models (where the true subject SD is denoted as 1), a judge’s test-retest reliability is directly related to the standard deviation of their severity drift: $\sigma_{\text{drift}} = \sqrt{1-\text{ICC}}$; when two judges are independent, the standard deviation of their difference is $\sigma_{\text{pair}} = \sigma_{\text{drift}} \times \sqrt{2}$.
II. Converting via measured ICC yields approximately 0.4
Judgment literature reports ICC most frequently. Take two sets of measurements: spinal muscular atrophy RHS scale, 21 senior physical therapists, viewing the same video clips with a two-week interval, yielding intra-judge ICC 0.922–1.0 (PMID 36542615); pressure pain threshold and time sum, 24 people repeating three times in the same session, ICC 0.80–0.97 (PMID 33434233).
| Measured ICC | $\sigma_{\text{drift}}$ | $\sigma_{\text{pair}}$ | Multiple of 0.167 |
|---|---|---|---|
| 0.995 | 0.071 | 0.100 | 0.60 |
| 0.980 | 0.141 | 0.200 | 1.20 |
| 0.950 | 0.224 | 0.316 | 1.89 |
| 0.920 | 0.283 | 0.400 | 2.39 |
| 0.900 | 0.316 | 0.447 | 2.68 |
| 0.850 | 0.387 | 0.548 | 3.28 |
The literature interval (ICC 0.80–0.97) corresponds to $\sigma_{\text{pair}}$ 0.245–0.632, which is 1.5 to 3.8 times 0.167. Inverting this is even clearer: to keep $\sigma_{\text{pair}}$ below 0.167, one needs ICC ≥ 0.986, whereas most judgment tasks fall between 0.80–0.95.
An independent cross-validation path: educational measurement reports the measured spread of rater severity (in units of ability SD) as 0.27–0.37; the difference between randomly paired judges is 0.38–0.52, also falling within the 2.3–3.1 times interval. Both paths are of the same order of magnitude.
III. “Hitting the line” is not accidental; it is the norm
The sampling distribution of $\delta$ approximates normality, with standard error $= \phi(c) \times \sigma_{\text{pair}} \times 100$ (when $m=0.75$, $\phi=0.318$). Substituting:
| $\sigma_{\text{pair}}$ | SE of $\delta$ | Probability exceeding 5pp ($m=0.75$) | Same for $m=0.85$ |
|---|---|---|---|
| 0.100 | 3.2 pp | 1.4% | 0.3% |
| 0.167 | 5.3 pp | 31.8% | 14.2% |
| 0.245 | 7.8 pp | 49.5% | 26.0% |
| 0.400 | 12.7 pp | 67.6% | 46.6% |
| 0.548 | 17.4 pp | 76.0% | 57.0% |
Using the literature central value $\sigma_{\text{pair}} \approx 0.44$ (ICC around 0.90), the probability of hitting the line when $m=0.75$ is approximately 67%, and the expected absolute value of $\delta$ is about 11 pp. This is not “whether it will hit the line,” but rather “in most cases it already lies outside the line.” The previous claim that “$\kappa_s$ being qualified is practically sufficient” must be withdrawn: $\kappa_s$ only may support the rulers are the same length, not that they are at the same height; and rulers not being at the same height is the norm.
IV. This quantity is also hard to measure precisely, and adding cases cannot save it
The precision of drift estimation depends solely on the number of repetitions. When estimating individual drift using $r$ repetitions, the degrees of freedom are $r-1$, and the relative standard error is approximately $1/\sqrt{2(r-1)}$: 3 repetitions yield 50%, 5 repetitions yield 35%, 10 repetitions yield 24%, and 20 repetitions yield 16%. To press this within ±20%, one must pool multiple readers (3 times × 7 people or 5 times × 4 people). It relies on adding readers, not adding cases—this mirrors the lesson of #1403: the degrees of freedom for the location parameter come from the number of readers.
V. The most critical layer: the criterion line sits exactly on the noise floor
Under linear approximation, the misalignment $b^$ required for $\delta$ to reach 5pp is approximately $5/(100 \cdot \phi)$. Meanwhile, the standard error of $\hat{\delta}$ is $SE = 100 \cdot \phi \cdot \sigma_{\text{pair}}$. Substituting $\sigma_{\text{pair}}$ with $b^$, the SE precisely equals 5 pp. Verification across various $m$: $m=0.60$ yields 5.10, $m=0.75$ yields 5.31, $m=0.85$ yields 5.72, $m=0.90$ yields 6.38—all hovering near 5.
The criterion line falls exactly on the noise floor of $\hat{\delta}$, with a signal-to-noise ratio of about 1. This is not a coincidence; it is an identity: the position of the decision line and the precision threshold required to measure it are both determined by the same $\phi \cdot b^*$. Any study where “$\delta$ is dangerous enough to hit the line” is precisely the same set of studies where “$\delta$ cannot be measured accurately.”
This yields a lower bound independent of sample size. When $\sigma_{\text{pair}}=0.167$ (already the “just-hitting-the-line” configuration), increasing $n$ from 60 to 3840 results in SEs of 9.52 / 7.71 / 6.62 / 6.00 / 5.66 / 5.40 pp respectively, stopping at 5.31 pp even as $n$ approaches infinity. Drift is systematic and cannot be averaged out—$N$ subjects share the same pair of judges, so the two zero-point offsets act in the same direction and do not dilute with $N$. This is why “increasing sample size,” the most common remedy, completely fails here. To press SE down to 1.79 pp (80% power to detect 5pp), one needs $\sigma_{\text{pair}} \leq 0.040$, i.e., ICC ≥ 0.9984, which judgment studies cannot achieve.
VI. The fix: replace reporting $\delta$ with reporting $\sigma_\Delta$
Since the absolute value of $\delta$ cannot be accurately measured under realistic reliability conditions, the criterion should not be built on $\delta$. Instead, report drift directly: let $\sigma_\Delta$ be the primary reported quantity, with $\delta$ serving only as a derived measure. The cost is extremely low—pooling estimation to ±20% requires only 3 times × 7 readers, far smaller than any single consistency study.
| $\sigma_\Delta$ interval | SE of $\hat{\delta}$ | What to do |
|---|---|---|
| <0.040 | <1.8 pp | $\delta$ is measurable; the 5pp threshold is valid |
| 0.040–0.118 | 1.8–5.3 pp | Threshold is assessable; confidence intervals must be reported |
| 0.118–0.167 | 5.3–7.5 pp | Threshold fails; report only drift |
| >0.167 | >7.5 pp | The conclusion “$\kappa_s$ is qualified” carries no information |
One design pitfall: theoretically, repeating the same cases at multiple time points could separate drift from case difficulty (since difficulty is constant while drift is a time-point effect). However, once there is interaction between difficulty and time point—judges remembering previous answers or experiencing fatigue during consecutive judgments—the two cannot be separated, degenerating into the memory effect problem of #1406. This route must simultaneously report the repetition interval.
Evidence and Limitations
Verified: The conversion $\sigma_{\text{drift}}=\sqrt{1-\text{ICC}}$, $\sigma_{\text{pair}}=\sigma_{\text{drift}} \cdot \sqrt{2}$ is a direct consequence of latent-variable models; literature-measured ICC (0.80–0.97, PMID 33434233; 0.922–1.0, PMID 36542615) and severity spread (0.31–0.42 logits / ability SD 1.14) from two independent paths cross-validate to the same order; degrees of freedom for drift estimation come solely from repetition count ($r-1$); the identity $SE(\sigma_{\text{pair}}=b^*) \approx 5\text{pp}$ holds for all $m=0.60–0.90$ (approximation error +0.10 to +1.38 pp).
Partially verified: Literature ICC comes from scale-based judgments (physical function scores, pain thresholds), whereas similarity to “Qi Gong external Qi change amplitude” such holistic impression judgments is only an analogy—the latter typically has lower reliability and greater drift, but no direct data. Additionally, the educational measurement scenario “115 raters with only 1 showing significant change across rounds” yields an inverted drift/SE ratio of about 0.65, comparable to the ICC-path’s 0.8, but this is a single scenario with small $n$, serving only as corroborating evidence.
To be verified: ICC for Qi Gong external Qi judgment tasks has never been measured; all conversions are cross-domain extrapolations requiring a minimum-scale real measurement (≥20 judges × ≥60 materials × 3 repetitions each) before calibration; whether drift shares a common mode significantly alters conclusions (when $\rho=0.9$, the hitting-line rate drops from 70% to 9%; if each drifts independently, all prior results hold); this correlation coefficient lacks measurement; whether drift accumulates monotonically over time or performs random walk determines if “periodic re-calibration” is effective (with step size 0.05 SD, about 11 cycles return to 0.167; with step size 0.15 SD, one and a half cycles suffice).
L1 Network Status: No corresponding edge exists in the L1 edge table for “judge consistency / position drift / reliability.” Search only hits CORRESPONDENCE_NETWORK.md line 192 “| Generating/Overcoming Consistency | Whether generating/overcoming directions can be reproduced | Not measured |” (referring to Five Elements generating/overcoming directions, unrelated to this topic), and the EDGE_TABLE entry “Sixty Heavenly Stems | 13 Chapters | A-level 2 chapters” regarding the reliability of Gan-Zhi date marking (also unrelated). This topic has not yet entered the L1 correspondence network.
Related Reading
- The ruler inside the qualified zone never reaches that line: $\kappa_s$ exerts maximum effort yet produces only 2.5 percentage points, while it must block 5
- Whose ruler is tilted; this question has no mathematical answer: a model with one rank deficit, adding more judges cannot compensate
- Asymmetry of two degrees of freedom: scale mismatch has a ceiling, position mismatch does not
- Resonance consciousness field: measurability boundary of Qi Gong external Qi
Evidence Level: C
References: PMID 36542615, PMID 33434233, PMID 36284065, PMID 37283589, PMID 8501467, PMID 35670017
Evidence Level: A = direct PubMed support, B = scientific analogy / indirect evidence, C = traditional consensus / preliminary exploration
Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.