标定出来的信度本身也有误差:折减口径的收益会被侵蚀掉几成

Written by

in

Language

Evidence level: B

折减口径把”标定出的相关系数”当成已知数代入公式省钱,但标定实验自己也是个样本——这个系数本身就带估计误差,误差一旦代入,折减省下来的样本量会被吃掉几成,这条线还能不能走要靠仿真标价。

正文

上一篇文章说过,我们的”有效自由度折减口径”有一张全网格标价表:只要知道判读者之间(或会话之间)的相关系数 ρ,就能按设计效应把例数口径的样本量打一个折,省下的钱就是折减口径的全部卖点。但那张表有一个隐藏假设——ρ 是已知常数。

真实世界里 ρ 从天上掉不下来,它必须由一次”标定实验”估出来:招一批受试者、安排若干个会话、重复读数,算出一个 ρ̂。而 ρ̂ 本身是个统计量,会话数少的时候误差不小。信度方法学里这恰恰是老问题:Walter 与 Bonett 的精度框架给出了”想要把 ρ 的置信区间压到指定宽度需要多少受试者与多少次重复”的闭式算法(PMID 24338600),2025 年的行为科学综述进一步提醒:样本量不足时重测信度的估计会系统性失真(PMID 40119099)。换句话说,标定实验自己也要算样本量,省钱的折减口径在入口处就先花了一笔钱。

更麻烦的是误差的传导路径。群随机试验文献早就踩过这个坑:样本量公式里必须代入一个 ICC,而 ICC 往往只来自小规模预试验,估计的不确定性对最终样本量和功效影响很大——Lewis 与 Julious 的数值方法显示,若不管不确定性,主试验可能严重欠功效;而常用的”取置信上限”做法虽然安全,却经常把试验设计得过度膨胀、低效(PMID 34477455)。2025 年提出的混合保证(assurance)方法干脆把 ICC 的不确定性用先验分布喂进设计,用”平均成功概率”替代单点功效(PMID 39935065);更早的贝叶斯分层模型则把多个历史试验的 ICC 信息合并出后验分布来定样本量(PMID 37036110)。Walter 团队 2001 年的经典结论同样指向这一点:信度研究应该按”ρ 置信区间的预期宽度”来规划,而不是按点估计(PMID 11746313)。2024 年的一篇系统综述把这些方法在单因素方差分析模型下排了个队,指出最常用的 Wald 大样本区间恰恰是统计性质最差的那种(PMID 38320802)。

把这些经验折回我们的问题:折减口径把 ρ̂ 代入设计效应后,SE 的标度不再是对真 ρ 的精确校正——ρ̂ 偏高时折减过猛、区间偏窄、覆盖率掉;ρ̂ 偏低时折减过弱、白白多花钱。折减口径对例数口径的宽度优势,会随标定误差增大而单调被侵蚀。所以核心判据必须量化:在原有仿真网格上让 ρ̂ 从”以 n_s 个标定会话估出的近似正态分布”里抽取(se_ρ 按精度框架闭式给出),对比”ρ 已知 / ρ̂ 带误差”两种折减口径相对例数口径的宽度比。若 n_s≥20 时带误差的折减口径仍保留已知 ρ 情形收益的七成以上,标定实验维持研究性定位、并给出最小标定样本量 n_s≥20;若收益保留不足一半,标定路线整体出局,例数口径成为唯一推荐口径。

侵蚀的形态还可以预判三档,免得仿真跑完才发现问题问错了方向。第一档是不对称侵蚀:ρ̂ 的抽样分布在低 ρ 区是右偏的,标定会话越少,ρ̂ 偶然偏高的机会越大,而设计效应对 ρ 的放大恰好是非线性的——偏高带来的”过度折减”对区间的伤害,比偏低带来的”少省几个样本”更重,所以侵蚀大概率不是对称的均值问题而是尾部问题。第二档是覆盖与宽度的解耦:折减省下的宽度若以牺牲覆盖率为代价,名义上更窄的置信区间实际上不再有 95% 的信誉,这时”宽度优势”本身就不再是收益,判据里必须同时盯宽度和覆盖两条线,只看宽度会奖励偷工减料。第三档是最小标定样本量的台阶效应:精度框架(PMID 24338600)给出的 se_ρ 随会话数衰减是先陡后缓的,n_s 从 5 加到 20 时误差砍掉大半,从 20 加到 40 却所省无几——若仿真实测出这条台阶,最小标定样本量就不该拍脑袋定 20,而应落在台阶拐点附近。这三档形态哪一档主导,直接决定标定路线是”加几场会话就能救”还是”结构性没戏”。

这与易学测量线(气功外气判读、丹田三模态、得气量表)的关系是同一句话:用第一人称与生理读数去标定构型相关系数这条路,成本不只是标定实验本身,还包括误差传导进下游每一个样本量决策。东方修炼传统里”功夫深浅”历来难以量化,现代测量学给出的答案是:量化可以,但每个量化环节的误差都要进账本——省样本量的捷径,要先付标定的学费。

证据与局限

已验证: ICC 估计的抽样误差对样本量/功效决策有实质影响(PMID 34477455、39935065、37036110);信度研究的样本量可按精度闭式规划(PMID 24338600、11746313);小样本重测信度估计会失真(PMID 40119099)。

部分验证: “取置信上限”保守但低效的折中策略在群随机试验中已量化(PMID 34477455);折减口径的侵蚀幅度需要本项目仿真标价,方向可由文献支撑、数值结果待产出。

待验证: n_s≥20 是否是标定实验的最小样本量;侵蚀后收益保留率的具体曲线形态;折减口径与混合保证方法(PMID 39935065)在本项目语境下的优劣对比。

相关阅读

证据等级: B
参考文献: PMID 34477455, PMID 39935065, PMID 38320802, PMID 24338600, PMID 11746313, PMID 40119099
L1 网络状态:该主题尚未进入 L1 对应网络(EDGE_TABLE 与 CORRESPONDENCE_NETWORK 中无「自由度折减/标定误差」对应边;最近邻为 E-060 标签信度,状态 conflict——中医医师间诊断一致性低且一致性统计常被误用,PMID 30383398/31730403)。


免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。

The reliability itself has error: how much of the reduced caliber’s gain will be eroded?

Evidence level: B

The reduction scale substitutes the “calibrated correlation coefficient” as a known constant into the formula to save costs. However, the calibration experiment itself constitutes a sample, and this coefficient inherently carries estimation error. Once this error is introduced, the sample size savings achieved by the reduction scale are eroded by a certain proportion. Whether this path remains viable depends on simulation-based pricing.

Main Text

As mentioned in the previous article, our “effective degrees of freedom reduction scale” includes a full-grid pricing table: knowing the correlation coefficient ρ between raters (or sessions) allows one to discount the sample size based on the design effect, saving money which represents the sole selling point of the reduction scale. However, that table relies on a hidden assumption—ρ is a known constant.

In reality, ρ does not fall from the sky; it must be estimated via a “calibration experiment”: recruiting a batch of subjects, arranging several sessions, repeating readings, and calculating an estimated ρ̂. Since ρ̂ itself is a statistic, the error can be substantial when the number of sessions is small. In reliability methodology, this is precisely a longstanding issue: Walter and Bonett’s precision framework provides a closed-form algorithm for “how many subjects and repetitions are needed to shrink the confidence interval of ρ to a specified width” (PMID 24338600). A 2025 behavioral science review further warns that insufficient sample sizes lead to systematic distortion in the estimation of test-retest reliability (PMID 40119099). In other words, the calibration experiment itself requires sample size calculation, meaning the cost-saving reduction scale incurs an upfront expense at the entrance.

Complicating matters further is the propagation path of error. The literature on cluster randomized trials has long encountered this pitfall: sample size formulas must substitute an ICC, yet the ICC often originates from small-scale pilot studies, and estimation uncertainty significantly impacts the final sample size and power. Lewis and Julious’s numerical methods show that ignoring uncertainty may result in the main trial being severely underpowered; while the common practice of “taking the confidence upper limit” is safe, it frequently leads to an overly inflated and inefficient trial design (PMID 34477455). A hybrid assurance method proposed in 2025 feeds the uncertainty of the ICC into the design via prior distributions, replacing single-point power with “average success probability” (PMID 3993605); earlier Bayesian hierarchical models combine ICC information from multiple historical trials to derive posterior distributions for sample size determination (PMID 37036110). Walter’s team also reached a classic conclusion in 2001: reliability studies should be planned based on the “expected width of the ρ confidence interval” rather than point estimates (PMID 11746313). A systematic review from 2024 ranks these methods under a single-factor ANOVA model, noting that the commonly used Wald large-sample interval is actually the one with the worst statistical properties (PMID 38320802).

Translating these experiences back to our problem: after the reduction scale substitutes ρ̂ into the design effect, the scaling of the standard error (SE) is no longer an exact correction for the true ρ. When ρ̂ is high, the reduction is excessive, leading to narrower intervals and reduced coverage; when ρ̂ is low, the reduction is insufficient, wasting money. The width advantage of the reduction scale over the example scale is monotonically eroded as calibration error increases. Therefore, the core criterion must be quantified: on the original simulation grid, draw ρ̂ from an approximate normal distribution estimated by n_s calibration sessions (where se_ρ is given by the closed-form precision framework), and compare the width ratio of the reduction scale relative to the example scale under “known ρ” versus “ρ̂ with error.” If n_s ≥ 20, the reduction scale with error retains over 70% of the benefit of the known ρ scenario, supporting the research-oriented positioning of the calibration experiment and establishing a minimum calibration sample size of n_s ≥ 20; if the retained benefit falls below half, the calibration route is entirely ruled out, and the example scale becomes the sole recommended option.

Three eroded forms can be anticipated to avoid discovering the wrong direction only after running simulations. The first is asymmetric erosion: the sampling distribution of ρ̂ is right-skewed in low ρ regions; with fewer calibration sessions, the chance of ρ̂ being spuriously high increases, while the design effect’s amplification of ρ is nonlinear—so the “over-reduction” caused by high estimates inflicts greater harm on the interval than the “under-saving” caused by low estimates. Thus, erosion is likely not a symmetric mean issue but a tail problem. The second is decoupling of coverage and width: if the width saved by reduction comes at the cost of reduced coverage, then although the nominal confidence interval appears narrower, it no longer carries 95% credibility; here, “width advantage” itself ceases to be a benefit, and the criterion must monitor both width and coverage simultaneously—focusing only on width rewards cutting corners. The third is the step effect of minimum calibration sample size: the precision framework (PMID 24338600) shows that se_ρ decays with the number of sessions in a pattern that is steep initially then flattens; increasing n_s from 5 to 20 cuts error by half, but going from 20 to 40 saves almost nothing. If simulation reveals this step, the minimum calibration sample size should not be arbitrarily set at 20 but placed near the inflection point of the step. Which of these three forms dominates directly determines whether the calibration route is “saved by adding a few sessions” or “structurally hopeless.”

This connects to the Yijing measurement line (qigong external qi reading, dantian tri-modal, qi-acquisition scale) in one sentence: using first-person physiological readings to calibrate the configuration correlation coefficient incurs costs not only from the calibration experiment itself but also from error propagation into every downstream sample size decision. In the Eastern cultivation tradition, “depth of practice” has long been difficult to quantify; modern metrology offers the answer: quantification is possible, but every quantification step’s error must be recorded in the ledger—the shortcut to saving sample size requires first paying the tuition for calibration.

Evidence and Limitations

Verified: Sampling error in ICC estimation substantially affects sample size/power decisions (PMID 34477455, 3993605, 37036110); reliability study sample sizes can be planned using precision closed-form equations (PMID 24338600, 11746313); small-sample test-retest reliability estimates are prone to distortion (PMID 40119099).

Partially Verified: The conservative yet inefficient compromise of “taking the confidence upper limit” has been quantified in cluster randomized trials (PMID 34477455); the degree of erosion for the reduction scale requires simulation-based pricing in this project, though direction is supported by literature and numerical results pending output.

To Be Verified: Whether n_s ≥ 20 is indeed the minimum sample size for the calibration experiment; the specific curve shape of retained benefit after erosion; and the comparative advantages/disadvantages of the reduction scale versus hybrid assurance methods (PMID 3993605) within this project’s context.

Further Reading

Evidence Level: B
References: PMID 34477455, PMID 3993605, PMID 38320802, PMID 24338600, PMID 11746313, PMID 40119099
L1 Network Status: This topic has not yet entered the L1 corresponding network (no edges for “degrees of freedom reduction/calibration error” in EDGE_TABLE and CORRESPONDENCE_NETWORK; nearest neighbor is E-060 label reliability, status conflict—diagnostic consistency among TCM physicians is low and consistency statistics are often misused, PMID 30383398/31730403).


Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *