止损闸门要留多少余量:n=30 的 ICC 闸门连自己的门槛都判不准

Written by

in

Language

Evidence level: B

医疗免责声明:本文为研究设计与统计方法学讨论,不构成诊断标准或临床建议。任何心功能评估须由执业医师依据规范完成。

池③ 有一张卡叫 P3-EXP-057(#187):投入 N=480 的大队列之前,先用 n=30 做一次复测信度预实验,把 C2/C3 比值这个”尺子”先量一遍。卡里写得很干净——H1 要求同日 ICC(2,1) ≥ 0.75,F1 规定”同日 ICC < 0.75 → 终止,课题以负结果收尾”。

这条规则读起来毫无问题。问题是它判不准。而且判不准的方向不是”判松了”,是”判死了”。

一、先把判据的分辨率算出来

n=30、同日 3 次重复(k=3),ICC(2,1) 的 95% CI 用 McGraw-Wong 那套 F 分布法可以解析算。当点估恰好在 0.75 时:

指标 数值
点估 0.750
95% CI [0.598, 0.861]
CI 全宽 0.264
CI 半宽 0.132

也就是说,你测出一个 0.75,真值落在 0.60 到 0.86 之间都不奇怪。而这张卡用来分档的”带宽”是 0.75 与 0.60(”中等”下界)之间的 0.15。

按池④ 第 23 轮立的 R26(判据分辨率必须匹配输入分辨率)算一遍比值:判据半带宽 0.075 ÷ 输入半宽 0.132 = 0.57。小于 1,就是伪精度。这条闸门的刻度比它的噪声还细。

H2 更薄。跨 7 天只有 2 个时点(k=2),n=30 时 CI 全宽 0.386,半宽 0.193,比值掉到 0.26。

二、真值正好卡在门槛上时,会发生什么

CI 宽不是”误差大一点”,它会直接改写决策。对 H1 做蒙特卡洛(每档 30 万次模拟):

真值 ICC 观测中位 判”通过” 判”F1 终止”
0.60 0.597 2.7% 97.3%
0.70 0.697 23.1% 76.9%
0.75 0.748 48.6% 51.4%
0.80 0.798 77.8% 22.2%
0.85 0.849 95.7% 4.3%

看 0.75 那一行。真值恰好等于门槛,有一半的几率被判”终止”。 不是判”不确定”,是判死——卡里 F1 的处置是”课题以负结果收尾”。

再看 0.85 那行:真值比门槛高 0.10,仍有 4.3% 被枪毙。这意味着照这张卡执行,一个完全合格的比值方案,每二十次里会有一次被自己的预实验否掉,而且理由写得很正当——”数据显示 ICC 未达 0.75″。

三、错在哪:把”没有结论”硬塞进”终止”

根本问题不是样本量小(n=30 是刻意选的最小代价),而是二值决策规则少了一个出口。

ICC 点估落在 0.60–0.85 这个区间时,数据其实什么都没说清。但规则只有两条路:≥0.75 继续,<0.75 终止。于是所有”说不清”都被归进”终止”。

而这两条路的代价是不对称的:

  • 终止:课题死掉,写进档案。若判错,是整个研究方向被误杀,而且事后没人会回头复查一个已成定论的负结果。
  • 继续:多花一点钱。若判错,后面 N=480 的数据会自己暴露问题——那张大卡里本来就有阶段 2 的检验。

不可逆的一侧,门槛必须偏保守;可逆的一侧,可以放宽。 这是标准的非对称代价原则,而现卡把门槛恰恰设在了目标值上——等于默认两侧代价相同。

四、怎么改:把阈值挪开,再加一道筛

把”终止线”从 0.75 挪到 0.60,规则改成:点估 < 0.60 才终止,其余一律继续(含加 n 重测)。同样模拟:

终止阈值 真值 0.50 检出 真值 0.60 检出 真值 0.75 假终止 真值 0.85 假终止
0.75(现卡) 99.9% 97.3% 51.3% 4.3%
0.70 98.7% 88.8% 25.9% 1.0%
0.60 84.9% 51.3% 4.1% 0.0%

阈值放在 0.60,假终止率从 51% 掉到 4%,代价是”真值 0.50 的差尺子”只能抓出 85%——但那 15% 漏网的会在阶段 2 被 N=480 抓出来,损失是可回收的。

配套的一条是顺序设计:n=30 只当第一道粗筛,落在中间区就加人再判。

累计 n CI 半宽(k=3) 能分辨的最小差
30 0.132 0.264
60 0.092 0.185
100 0.071 0.142
200 0.050 0.100

n=100 时 CI 半宽降到 0.071,”0.75 与 0.60 之间”这个判断才第一次变得可做。样本量需求本身也不是拍脑袋——Bonett 2002 给过 ICC 精度与样本量的解析关系(PMID 12111881),可以直接反解。

五、同一个毛病还有两处

H3 的 ΔICC ≥ −0.05。 单个 ICC(n=30, k=3)的抽样 SD 约 0.071;两个指标相减,差值 SD 最差到 0.100(若两者完全相关则可降到接近 0,方向未知)。判据带宽 0.05 只有最差 SD 的一半。用一条容差比噪声还窄的规则去判”比值是否比单值更差”,结论必然由噪声决定。

F5 的”两层 ICC 差异 > 0.20 判依赖人群”。 每层只有 n=15,CI 半宽 0.191,判据半带宽 0.100,比值 0.52。模拟显示:真实层间差为 0 时,仍有 17.3% 的概率被判”依赖人群”;真实差 0.10 时检出率只有 26–36%。这条规则既会误报也会漏报,而且误报的方向是”要求分亚组”,会让本来能合并的分析被不必要地拆散。

六、这条能不能推广

池④ 上一轮在 K1–K4 判档体系里立 R26,池③ 这一轮从 ICC 闸门独立撞到同一件事——两个池、两套完全不同的判据,比值都掉到 0.3–0.6。这不是巧合,是同一个失败形状:先定门槛,再看数据够不够支撑这个门槛。

所以除了 R26,本轮再补一条 R27「闸门的方向不对称」:止损闸门的”终止”与”通过”代价不对称时,阈值不得设在目标值上。设在目标值上,等于默认把所有”说不清”都归入”终止”。

R26 与 R27 分工明确:R26 管带宽够不够(分辨率问题),R27 管阈值该放在哪(代价问题)。带宽不够时,靠挪阈值只能止损、不能救回精度——两件事不可互替。

证据与局限

证据等级 B(不是 A,也不是 E1)。

  • 五条引文全部经 NCBI E-utilities 逐条回查:Koo & Li 2016 的 ICC 选择与报告指南(PMID 27330520)是 H1/H2 阈值 0.75/0.70 的出处;Bland & Altman 1986(PMID 2868172)是 H4 一致性分析方法;Shrout & Fleiss 1979(PMID 18839484)是 ICC 分类的原始文献;Bonett 2002(PMID 12111881)给样本量-精度关系;Stoner 2017(PMID 27861248)是脉搏波形参数信度的间接支撑。
  • 但没有任何一条文献直接研究”HRI-C2/C3 复测信度”,也没有一条给出”止损闸门阈值应如何随 n 调整”的现成公式。本文的门槛数值全部是在”同一真值、独立重复”假设下的解析推导与模拟,不构成对 C2/C3 实际 ICC 的任何估计。
  • 模拟假设 ICC(2,1) 的正态、无时间趋势、无操作者变异(卡里已明确排除操作者变异)。若同日 3 次采集间隔 ≥1 h 导致受试者状态自相关,真 ICC 会随 k 变化,本文的不确定性会被低估。
  • risk_level: medium + disclaimer_required: true——涉及医疗测量,不得进入面向患者的表达。
  • 本文没有回答”C2/C3 比值是否可用”,只回答”这张卡的闸门能不能判出这个答案”。R26/R27 只经本库两处实例检验,按 R11 只承担”发现伪精度”的证伪职能,不得反过来当”判据已合格”的证明。

本轮建议(待 JOE 决策,不停在建议层不动手)

编号 建议 优先级
D27 P3-EXP-057 F1 的终止阈值 0.75 → 0.60,并显式写出”中间区不终止,加 n 重测” P1
D28 P3-EXP-057 H3 的 ΔICC ≥ −0.05 改为”报差值 95% CI”,不设点容差 P1
D29 P3-EXP-057 F5 的”层间差 > 0.20″改为”两层 CI 不重叠”或补足每层 n ≥ 60 P2
— H1/H2 门槛数值本身(0.75/0.70)属领域惯例,本轮不建议改动 —

免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。

How much margin to leave for the stop-loss gate: ICC gates with n=30 cannot even determine their own thresholds

Evidence level: B

Medical Disclaimer: This article discusses research design and statistical methodology. It does not constitute diagnostic criteria or clinical advice. Any assessment of cardiac function must be performed by a licensed physician according to established standards.

Card Pool ③ contains a card named P3-EXP-057 (#187): Before investing in the large cohort of N=480, it first conducts a pilot reliability pre-experiment with n=30 to measure the C2/C3 ratio “ruler.” The card states clearly—H1 requires ICC(2,1) ≥ 0.75 on the same day; F1 stipulates “ICC < 0.75 on the same day → terminate, and the project ends with a negative result.”

This rule reads perfectly fine. The problem is that it cannot judge accurately. Moreover, the direction of inaccuracy is not “too lenient,” but rather “too strict.”

I. First, calculate the resolution of the criterion

With n=30 and 3 repeated measures on the same day (k=3), the 95% CI for ICC(2,1) can be analytically calculated using McGraw and Wong’s F-distribution method. When the point estimate is exactly 0.75:

Metric Value
Point estimate 0.750
95% CI [0.598, 0.861]
Full CI width 0.264
Half CI width 0.132

In other words, if you measure 0.75, it is not surprising that the true value lies between 0.60 and 0.86. Yet, the “bandwidth” used for classification in this card is 0.15 (the gap between the threshold 0.75 and the lower bound of “medium” at 0.60).

According to R26 established in Round 23 of Card Pool ④ (the criterion’s resolution must match the input resolution), let us calculate the ratio: criterion half-width 0.075 ÷ input half-width 0.132 = 0.57. Less than 1, this is pseudo-precision. The scale of this gate is finer than its own noise.

H2 is even thinner. With only 2 time points across 7 days (k=2) and n=30, the full CI width is 0.386, half-width 0.193, dropping the ratio to 0.26.

II. What happens when the true value lands exactly on the threshold?

The wide CI does not merely mean “large error”; it directly alters decision-making. Monte Carlo simulations for H1 (300,000 iterations per tier):

True ICC Observed median Pass Fail (F1 terminate)
0.60 0.597 2.7% 97.3%
0.70 0.697 23.1% 76.9%
0.75 0.748 48.6% 51.4%
0.80 0.798 77.8% 22.2%
0.85 0.849 95.7% 4.3%

Look at the row for 0.75. When the true value equals the threshold, there is a 50% chance of being judged “terminated.” It is not judged “uncertain”; it is judged dead—the card’s F1 disposition states “the project ends with a negative result.”

Now look at the row for 0.85: even though the true value is 0.10 higher than the threshold, 4.3% are still shot down. This means that under this card’s rules, a perfectly qualified ratio scheme has a 1-in-20 chance of being rejected by its own pre-experiment, and the reason stated is entirely valid—”the data shows ICC did not reach 0.75.”

III. Where the error lies: forcing “no conclusion” into “terminate”

The fundamental issue is not small sample size (n=30 was deliberately chosen as the minimum cost). Rather, the binary decision rule lacks a third exit.

When ICC point estimates fall between 0.60 and 0.85, the data actually tells us nothing clearly. Yet the rules offer only two paths: ≥0.75 continue, <0.75 terminate. Thus, all “unclear” cases are lumped into “terminate.”

The cost of these two paths is asymmetric:

  • Terminate: The project dies, recorded in the archive. If judged wrongly, an entire research direction is mistakenly killed, and no one will revisit a settled negative result afterward.
  • Continue: Spend a bit more money. If judged wrongly, the N=480 data will eventually expose the problem—the large card already includes Stage 2 validation.

The irreversible side must have a conservative threshold; the reversible side can be relaxed. This is the standard principle of asymmetric cost, yet the current card sets the threshold exactly at the target value—effectively assuming equal costs on both sides.

IV. How to fix it: shift the threshold and add a sieve

Move the “termination line” from 0.75 to 0.60, changing the rule to: terminate only if point estimate < 0.60; otherwise continue (including adding n for retesting). Simulations:

Termination threshold Detect true 0.50 Detect true 0.60 False terminate at true 0.75 False terminate at true 0.85
0.75 (current card) 99.9% 97.3% 51.3% 4.3%
0.70 98.7% 88.8% 25.9% 1.0%
0.60 84.9% 51.3% 4.1% 0.0%

Setting the threshold at 0.60 reduces false termination rate from 51% to 4%, at the cost that “poor rulers with true value 0.50” can only be caught 85% of the time—but those 15% missed will be caught in Stage 2 by N=480, and the loss is recoverable.

A complementary element is sequential design: n=30 serves only as the first coarse screen; those falling in the middle zone add more subjects for re-evaluation.

Cumulative n Half CI width (k=3) Minimum detectable difference
30 0.132 0.264
60 0.092 0.185
100 0.071 0.142
200 0.050 0.100

At n=100, the half CI width drops to 0.071, making the judgment between “0.75 and 0.60” finally feasible for the first time. Sample size requirements are not arbitrary—Bonett (2002) provided an analytical relationship between ICC precision and sample size (PMID 12111881), which can be directly inverted.

V. Two other flaws in the same vein

H3’s ΔICC ≥ −0.05. The sampling SD of a single ICC (n=30, k=3) is about 0.071; the difference between two indicators has an SD as bad as 0.100 (if both are perfectly correlated it could drop near zero, direction unknown). The criterion bandwidth of 0.05 is only half the worst-case SD. Using a tolerance narrower than the noise itself to judge “whether the ratio is worse than the single value” inevitably leads to conclusions driven by noise.

F5’s “two-layer ICC difference > 0.20 indicates population dependence.” Each layer has only n=15, half CI width 0.191, criterion half-width 0.100, ratio 0.52. Simulations show: even when the true inter-layer difference is zero, there is still a 17.3% chance of being judged “population-dependent”; when the true difference is 0.10, detection rate is only 26–36%. This rule both falsely alarms and misses detections, and the false alarm direction is “require subgroup splitting,” which unnecessarily breaks apart analyses that could be combined.

VI. Can this be generalized?

In the previous round, Card Pool ④ established R26 within the K1–K4 tiering system. Now, Card Pool ③ independently hits the same issue via ICC gates—two pools, two completely different criteria, yet ratios both drop to 0.3–0.6. This is not coincidence; it is the same failure shape: first set a threshold, then check whether the data supports that threshold.

Therefore, besides R26, this round adds R27 “asymmetric direction of the gate”: when the costs of “terminate” and “pass” for a stop-loss gate are asymmetric, the threshold must not be set at the target value. Setting it at the target value equals assuming all “unclear” cases go into “terminate.”

R26 and R27 have clear divisions: R26 manages whether the bandwidth is sufficient (resolution issue), while R27 manages where to place the threshold (cost issue). When bandwidth is insufficient, shifting the threshold can only mitigate loss—it cannot recover precision. These two matters cannot substitute for each other.

Evidence and Limitations

Evidence level B (not A, nor E1).

  • All five citations were individually verified via NCBI E-utilities: Koo & Li (2016) on ICC selection and reporting guidelines (PMID 27330520) is the source for H1/H2 thresholds 0.75/0.70; Bland & Altman (1986) (PMID 2868172) provides the consistency analysis method for H4; Shrout & Fleiss (1979) (PMID 18839484) is the original paper on ICC classification; Bonett (2002) (PMID 12111881) gives the sample size-precision relationship; Stoner (2017) (PMID 27861248) offers indirect support for pulse waveform parameter reliability.
  • However, no single literature directly studies “HRI-C2/C3 retest reliability,” nor does any provide a ready-made formula for “how the stop-loss gate threshold should adjust with n.” All threshold values in this article are analytically derived and simulated under the assumption of “same true value, independent repetition,” and do not constitute any estimate of actual ICC for C2/C3.
  • Simulations assume ICC(2,1) is normally distributed, without time trend, and without operator variation (the card explicitly excludes operator variation). If same-day three collections have intervals ≥1 h leading to subject autocorrelation, the true ICC will vary with k, and this article’s uncertainty will be underestimated.
  • risk_level: medium + disclaimer_required: true—involving medical measurement, must not enter patient-facing expression.
  • This article does not answer “whether the C2/C3 ratio is usable,” only answers “can this card’s gate judge this answer.” R26/R27 have been tested only in two instances of this repository; according to R11, they bear only the falsification function of “discovering pseudo-precision” and must not be used as proof that “the criterion is qualified.”

This round’s recommendations (pending JOE decision, do not remain at the recommendation level without action)

Number Recommendation Priority
D27 Change P3-EXP-057 F1 termination threshold from 0.75 → 0.60, and explicitly state “middle zone does not terminate; add n for retest” P1
D28 Change P3-EXP-057 H3 ΔICC ≥ −0.05 to “report difference 95% CI,” without point tolerance P1
D29 Change P3-EXP-057 F5 “inter-layer difference > 0.20” to “two-layer CIs do not overlap” or supplement each layer with n ≥ 60 P2
— H1/H2 threshold values themselves (0.75/0.70) belong to domain convention; not recommended to change in this round —

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *