Evidence level: C
一条筛选规则的价值,不在它筛掉了谁,而在换一批人重筛时它是否还筛掉同一批。
一、Jaccard 的分母不是「全部参数」,是两次筛选的并集
五个候选参数,第一次入选三个、第二次入选三个,交集两个,Jaccard 就是 2/4 = 0.5。分母 4 不是 5 —— 有并集才有分母。这意味着一个参数从入选掉到落选,分子减一,分母同时可能变,两个方向的误差叠在同一个数上。
更要紧的是,它只数「在不在名单里」,不管离门槛多远。信噪比 0.51 和 1.99 都算落选,1.99 与 2.01 那道细缝被抹平成「一个进、一个不进」。用同一批数据既选又检,描述统计与推断会同时失真,纯噪声也能造出效应(PMID 19396166)。
二、分半差异里,有一半是「人变少了」
随机分成两半,每半样本量减半。标准误按 1/√n 走,n 减半则标准误升约 1.41 倍。信噪比的分母本就是个不确定性量,同一群体、同一台设备,光靠减半就能让一批参数从 ≥2 掉到 <2。
高维选择的重抽样研究给过直接证据:训练样本越小,签名的稳定性越低,不同重抽样各自选出的基因集合交集极小(PMID 29078093)。所以「两个半样本筛出的集合不同」这件事,天然含一个与规则无关的成分。想把它剥出来,需要一个不改样本量的对照:同一批全样本在不同日子、不同操作员下重测两次,看集合差多少。两相对照,才知道 Jaccard 低是规则脆,还是小样本必然。
三、bootstrap 入选频率,在相关参数上没有干净的零假设
入选频率在文献里是被当标准工具用的,但它的估计一致性依赖重抽样方式。Wallisch 等人用模拟比较 subsampling 与 bootstrap,结论很硬:变量入选频率与模型选择频率只在 subsampling 下被一致估计;bootstrap 在偏差与精度上只对相对条件偏倚和均方根差比占优,而这两个量的无偏估计要求协变量彼此独立(PMID 33089538)。De Bin 等人的对比给出同一方向(PMID 26288150)。
呼吸力学参数恰恰最不独立:X5、AX、Fres、R5−R20 共享同一段阻抗谱,彼此强相关。在一组强相关的量上做 bootstrap,打乱的是「谁和谁一起来」,频率就不再度量规则稳定性,而变成相关结构的函数。这时报「某参数入选频率 0.62」并不比不报更好。
四、阈值压在分布密集处时,「入选与否」这个二分本身没有信息
如果五个参数的信噪比恰好挤在 1.8 到 2.2 之间,那么 ≥2 这条线穿过的是整个分布最密的一段。此时名单的构成由小数点后第二位决定,而第二位在重测里本来就会变。
这不是新问题。最优切点一旦由数据选出,I 类错误会严重膨胀、效应量被高估,连置信区间也一并失真,须用收缩加自助法恢复覆盖率(PMID 15160403)。把连续变量二分还会给 meta 分析引入选择偏倚与功效损失,分布法能同时给出连续与二分估计、保持推断一致(PMID 24920271)。门槛本身不是错,把门槛当成「有/无」的判决才是错。
五、稳定入选可以是稳定地错
还有一层容易漏:集合稳定,不等于集合正确。随机森林在过敏基准数据上拿到 0.9999 的准确率,去掉前两个特征重选后掉到 0.8836,且特征排序不稳;逻辑回归准确率 0.9116,排序同样不稳;而特征凝聚、高变基因、Spearman 相关三种方法准确率几乎不变、排序稳定(PMID 40985317)。准确率不等于可复现的重要性,反之亦然。
影像组学的系统综述还指出一个更基础的问题:41 项可重复性研究中只有 7 项完整交代采集、预处理与特征提取的全部方法细节,而「用什么截断值区分稳定特征」的报告质量普遍不合格(PMID 30170872)。分母没定清,稳定性数字就无从比较。
六、反转:该先验的不是 Jaccard,是阈值落在分布的哪个位置
上面五节都在讨论「怎么算稳定性」。但真实世界给了一个更朴素的提醒:连公认的金标准阈值自己都在漂。淀粉样蛋白 PET 的 Centiloid 阈值,个体数据 meta 分析给出数据驱动约 18、目视判读约 27,并明确 11 到 26 这一段应「结合用途谨慎解读」(PMID 42441390);另一项在两个人独立队列里推导的研究,多数指标的切点落在 25–35,唯独脑脊液 Aβ42 给出 12(PMID 30902090);膝关节置换的最小重要变化阈值在术后 3、12、24 个月分别是 0.1、4.2、5.1,随时间上移(PMID 38458544)。
金标准都在漂,说明阈值是「当前人群 + 当前方法 + 当前时点」的产物,不是自然常数。所以正确的验证顺序是反的:先画信噪比分布,看 ≥2 落在第几分位、离众数多远;距离近,则 Jaccard 必然低且这个低说明不了规则好坏,此时应改报连续值与区间;距离远,二分才值得保留,Jaccard 才有解读空间。规则稳不稳,先问它站在分布的哪一段 —— 站在人堆里,谁都会被挤下去。
四格判据:分布图上 ≥2 落在稀疏尾部且 Jaccard ≥0.7 ⇒ 规则可用,可继续做跨中心筛选;落在众数附近且 Jaccard <0.5 ⇒ 二分无效,一律改报连续值加区间,禁止用入选集合推功效;Jaccard 高但每次入选的集合都偏离已知效应方向 ⇒ 稳定地错,须换判据而非调阈值;五项参数的入选频率全落在 0.4–0.6 ⇒ 抽签区,任何「某参数更强」的说法撤回。
证据与局限
已验证: 用同一数据既选又检会使描述统计与推断同时失真,纯噪声也能造出虚假效应(PMID 19396166);训练样本越小,高维筛选出的签名稳定性越低(PMID 29078093);变量入选频率与模型选择频率只在 subsampling 下被一致估计,bootstrap 仅在相对条件偏倚与均方根差比上占优且要求协变量独立(PMID 33089538);subsampling 在模型选择稳定性评估上优于 bootstrap(PMID 26288150);最优切点的选择会使 I 类错误膨胀、效应高估、置信区间失真,须收缩加自助法恢复覆盖率(PMID 15160403);连续变量二分会让 meta 分析出现选择偏倚与功效损失(PMID 24920271);准确率高的特征其重要性排序未必稳定(PMID 40985317);影像组学可重复性研究中「稳定特征的截断值如何定」报告质量普遍不足(PMID 30170872)。
部分验证: 淀粉样蛋白 PET 的 Centiloid 阈值在不同推导路径间存在系统差异(数据驱动 18 vs 目视 27,中间 11–26 段需按用途谨慎解读,PMID 42441390);在两个人独立队列推导时多数切点落在 25–35,但脑脊液 Aβ42 给出 12(PMID 30902090);最小重要变化阈值随术后时间上移(PMID 38458544)—— 三者都说明阈值非自然常数,但都不是针对「筛查规则入选集合」的直接实测。
待验证: ①丹田参数上「信噪比≥2」的分半 Jaccard 从未实测,0.5 这条作废线目前只是预设;②分半造成的样本量减半效应与规则脆弱性各占多少,未做分解;③五项参数相关矩阵下的 bootstrap 入选频率偏差幅度未知;④「阈值距众数多远」与 Jaccard 的定量关系(距离-稳定性曲线)无人标定。
L1 网络状态:该主题尚未进入 L1 对应网络。节点表仅按三类域分派 —— 原文「| 节点 | 各域的概念/量(五行、五脏、频率、颜色…) | 按域分派(见下表) |」,即易学/中医侧、物理侧、生理/体验侧,无「测量学」维度的节点(无信噪比门槛、入选集合、稳定性系数、重抽样);边表止于 E-044(壬癸→水),无任何以测量规则为源或目标的边。
相关阅读
证据等级: C
参考文献: PMID 19396166, PMID 29078093, PMID 33089538, PMID 26288150, PMID 15160403, PMID 24920271, PMID 40985317, PMID 30170872, PMID 42441390, PMID 30902090, PMID 38458544
免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。
Stability of the Threshold for Issuing Entry Tickets to Parameters: The Set Filtered by a Signal-to-Noise Ratio ≥ 2 Changes Halfway Through
Evidence level: C
The value of a screening rule lies not in who it excludes, but in whether it excludes the same cohort when re-screening a new batch.
I. Jaccard’s Denominator Is Not “All Parameters,” But the Union of Two Screenings
Among five candidate parameters, three are selected in the first round and three in the second; the intersection is two. Thus, Jaccard equals 2/4 = 0.5. The denominator is 4, not 5 — there must be a union to form the denominator. This implies that if a parameter moves from selection to exclusion, the numerator decreases by one while the denominator may also change; errors in both directions stack within a single number.
More critically, it only counts “presence or absence,” ignoring the distance from the threshold. An SNR of 0.51 and 1.99 both result in exclusion, yet the narrow gap between 1.99 and 2.01 is flattened into “one in, one out.” Using the same dataset for both selection and validation causes simultaneous distortion in descriptive statistics and inference, where pure noise can generate apparent effects (PMID 19396166).
II. In Split-Half Differences, Half the Variation Is Due to “Fewer People”
Randomly splitting into two halves reduces the sample size per group by half. Standard errors scale with $1/\sqrt{n}$; halving $n$ increases the standard error by approximately 1.41 times. Since the denominator of SNR is inherently a measure of uncertainty, simply halving the sample size can cause a batch of parameters to drop from $\ge 2$ to $< 2$ within the same population and equipment.
High-dimensional selection resampling studies provide direct evidence: smaller training samples yield lower signature stability, with extremely small intersections between gene sets selected by different resampling iterations (PMID 29078093). Therefore, the fact that “sets derived from two half-samples differ” naturally contains a component unrelated to the rule itself. To isolate this, one needs a control that preserves sample size: re-measure the same full sample on different days or under different operators to observe the set difference. Comparing these two scenarios reveals whether low Jaccard values stem from rule fragility or inevitable small-sample effects.
III. Bootstrap Selection Frequencies Lack a Clean Null Hypothesis for Correlated Parameters
Selection frequency is used as a standard tool in literature, but its estimation consistency depends on the resampling method. Wallisch et al., comparing subsampling with bootstrap via simulation, concluded strongly that variable selection frequencies are consistently estimated only under subsampling; bootstrap is superior only in relative conditional bias and root mean square difference ratios, but unbiased estimation of these two quantities requires independent covariates (PMID 33089538). De Bin et al. provide support in the same direction (PMID 26288150).
Respiratory mechanics parameters are particularly non-independent: X5, AX, Fres, and R5−R20 share a segment of the impedance spectrum and exhibit strong correlation. Performing bootstrap on a set of highly correlated variables shuffles “who goes with whom,” causing frequency to no longer measure rule stability but rather become a function of the correlation structure. Reporting “a selection frequency of 0.62 for a specific parameter” offers no advantage over silence in this context.
IV. When Thresholds Press Against Dense Distribution Regions, the Binary Nature of “Inclusion or Exclusion” Itself Yields No Information
If the SNRs of five parameters cluster tightly between 1.8 and 2.2, the line $\ge 2$ traverses the densest segment of the entire distribution. In this case, list composition is determined by the second decimal place, which inherently varies across re-measurements.
This is not a novel issue. Once an optimal cutoff is selected from data, Type I error inflation occurs, effect sizes are overestimated, and confidence intervals become distorted; coverage must be restored via shrinkage and bootstrapping (PMID 15160403). Binning continuous variables also introduces selection bias and power loss in meta-analysis; distribution-based methods can simultaneously provide both continuous and binary estimates while maintaining inferential consistency (PMID 24920271). The threshold itself is not the error; treating it as a binary judgment of “yes/no” is the flaw.
V. Stable Inclusion Can Be Stable Error
Another layer often overlooked: set stability does not equate to set correctness. Random forests achieve 0.9999 accuracy on synthetic allergy benchmark data, yet drop to 0.8836 after removing the top two features for re-selection, with unstable feature rankings; logistic regression accuracy is 0.9116 with similarly unstable rankings. Conversely, feature aggregation, highly variable genes, and Spearman correlation yield nearly unchanged accuracy and stable rankings (PMID 40985317). High accuracy does not equal reproducible importance, nor vice versa.
A systematic review in radiomics highlights a more fundamental issue: among 41 reproducibility studies, only 7 fully detailed acquisition, preprocessing, and feature extraction methods, while reporting quality regarding “what cutoff value distinguishes stable features” is generally inadequate (PMID 30170872). Without a defined denominator, stability metrics cannot be compared.
VI. Reversal: The Prior Should Not Be Jaccard, But the Threshold’s Position Within the Distribution
The preceding five sections discuss “how to calculate stability.” However, the real world offers a simpler reminder: even the consensus gold-standard threshold itself is drifting. The Centiloid threshold for amyloid PET shows data-driven estimates around 18 versus visual reading around 27, explicitly advising “cautious interpretation” for the range 11 to 26 depending on application (PMID 42441390); another study deriving cutpoints across two independent cohorts found most indicators between 25–35, with cerebrospinal fluid Aβ42 at 12 (PMID 30902090); minimum important change thresholds for knee arthroplasty shift upward over postoperative time (3, 12, and 24 months: 0.1, 4.2, and 5.1 respectively) (PMID 38458544).
Since gold standards drift, thresholds are products of “current population + current method + current timepoint,” not natural constants. Therefore, the correct validation sequence is reversed: first plot the SNR distribution to see where $\ge 2$ falls in terms of percentile and distance from the mode. If close to the mode, Jaccard will inevitably be low, and this low value does not indicate rule quality; one should report continuous values and intervals instead. If far from the mode, binary classification warrants retention, providing space for Jaccard interpretation. Whether a rule is stable depends first on which segment of the distribution it stands in — standing in the crowd, everyone gets pushed down.
Four-Grid Criterion:
* Distribution plot shows $\ge 2$ in sparse tails AND Jaccard $\ge 0.7$ ⇒ Rule usable, continue cross-center screening.
* Falls near mode AND Jaccard $< 0.5$ ⇒ Binary invalid, report continuous values with intervals exclusively; prohibit inferring power from inclusion sets.
* High Jaccard but selected sets consistently deviate from known effect directions ⇒ Stable error, change criteria rather than adjust threshold.
* All five parameters’ selection frequencies fall within 0.4–0.6 ⇒ Lottery zone; retract any claim of “one parameter being stronger.”
Evidence and Limitations
Verified: Using the same data for both selection and validation distorts descriptive statistics and inference simultaneously, where pure noise can generate false effects (PMID 19396166); smaller training samples yield lower stability for high-dimensional signatures (PMID 29078093); variable selection frequencies are consistently estimated only under subsampling, while bootstrap is superior only in relative conditional bias and root mean square difference ratios but requires independent covariates for unbiased estimation (PMID 33089538); subsampling outperforms bootstrap for assessing model selection stability (PMID 26288150); selecting optimal cutpoints inflates Type I error, overestimates effect sizes, and distorts confidence intervals, requiring shrinkage and bootstrapping to restore coverage (PMID 15160403); binning continuous variables introduces selection bias and power loss in meta-analysis (PMID 24920271); high-accuracy features do not necessarily have stable importance rankings (PMID 40985317); reporting quality regarding “how to define cutoff values for stable features” is generally inadequate in radiomics reproducibility studies (PMID 30170872).
Partially Verified: Centiloid thresholds for amyloid PET show systematic differences across derivation pathways (data-driven 18 vs. visual 27, with 11–26 requiring cautious interpretation based on use, PMID 42441390); in two independent cohorts, most cutpoints fall between 25–35, but CSF Aβ42 yields 12 (PMID 30902090); minimum important change thresholds shift upward post-operatively (PMID 38458544) — all indicating thresholds are not natural constants, though none directly measure “screening rule inclusion sets.”
To Be Verified: ① The split-half Jaccard for “SNR ≥ 2” on Dantian parameters has never been empirically measured; the 0.5 threshold remains a preset assumption; ② The relative contribution of sample size reduction due to splitting versus rule fragility has not been decomposed; ③ The magnitude of bootstrap selection frequency bias under the correlation matrix of five parameters is unknown; ④ The quantitative relationship between “distance from mode” and Jaccard (distance-stability curve) remains uncalibrated.
L1 Network Status: This topic has not yet entered the L1 corresponding network. Node tables are currently assigned only across three domains — original text “| Node | Concepts/Quantities per domain (Five Phases, Five Organs, Frequency, Color…) | Assigned by Domain (see table below) |”, i.e., Yijing/TCM side, Physics side, Physiological/Experiential side, with no “Measurement Science” dimension nodes (no SNR thresholds, inclusion sets, stability coefficients, resampling); edge tables stop at E-044 (Ren-Gui → Water), with no edges originating from or terminating at measurement rules.
Related Reading
- The Threshold Itself Is Shaking: Reproducibility of the SNR ≥ 2 Screening Rule
- MDD of Dantian Breathing CoV Displacement and SNR Threshold for “Qi Sinking” Effect Size
- Head-to-Head Comparison: Device Bias vs. Effect Size SNR
- Environment-Life Frequency Interface
Evidence Level: C
References: PMID 19396166, PMID 29078093, PMID 33089538, PMID 26288150, PMID 15160403, PMID 24920271, PMID 40985317, PMID 30170872, PMID 42441390, PMID 30902090, PMID 38458544
Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.
Leave a Reply