超额稳定性的 0.6 和 0.3 是借来的尺子:该标定的不是分数,是「分数到选对率」的映射曲线

Written by

in

Language

Evidence level: C

0.6 站得住、0.3 真脆——这两条线从来没对着任何已知真值量过,它们是别处一把尺子上刻的记号。

一、两个数的来历

说法是「超额稳定性 ≥0.6 算稳,<0.3 算脆」。这两个数的出处,是评分者一致性里 kappa 的经验区间:0.6 上下大致对应「尚可」,0.3 以下对应「差」。而 kappa 那套区间本身是口头传统,且在低 base rate 下会过度惩罚一致(PMID 25090041)。把一条从一致性研究借来的尺子插到参数筛选上,中间没有任何一步对着已知真值标过。

二、稳定性与准确性本来就是两个维度

这一点在特征选择文献里是被明确分开写的。FREL 那篇开篇就说,好的特征选择算法由 accuracy 与 stability 两个因素刻画;它的集成版本把稳定性提上去之后,准确率只是「更好或相当」(PMID 25134091)。另一篇做集成选择的,稳定性分数明显改善,分类性能同样只是相当或略好(PMID 35016102)。更直接的一句在第三篇:小样本下不同方法的分类准确率可以彼此相当,但它们选出的特征在精度与稳定性上差得很远(PMID 22403633)。稳定,不等于对。

三、这个领域其实标定过门槛,标的是错误率

stability selection 的正统做法,是把选择概率的阈值与「期望假选个数」挂钩,用一个理论上界来控制。后来有人改用置换来估 FDR 定阈值,结果比 FWER 上界宽松、能捞回更多真信号(PMID 23089816)。再往后有人把它做成自动校准,把期望假阳性压到 20 个以下(PMID 38143734)。三条路指向同一件事:阈值该锚在「会选错几个」上,而不是锚在一个 0 到 1 之间、没有单位的分数上。

四、同一个分数,换个选择器,选对率差几十倍

而「会选错几个」这个锚,随选择器剧烈变动。一项 3300 次模拟的研究里,混合效应模型的 stability selection 把假发现率稳定压在 0.02 以下、真阳性率 0.73 到 0.97;同一个模拟里换成 Lasso,假发现率 0.59 到 0.72,真阳性率却 ≥0.93(PMID 35843027)。两者都在「选出参数」,假阳性比例差三十倍。所以「稳定性 0.6 对应多少选对率」这句话缺个主语:得先说清是哪个选择器。映射曲线不是一条,是一族。

五、门槛里还有「先选后报」的水分

你算的稳定性,是在「已经选出这 5 个」之后算的,而被选中的那几个稳定性天然偏高。Heinze 等的综述明确写着:变量选择会损害最终模型的稳定性、系数无偏性与 p 值有效性,因此建议把整个选择过程重抽样后的量作为常规报告项(PMID 29292533)。先选后报,门槛会显得更容易过。

六、这里没有可算的零分布

还有个硬约束。knockoff 那类方法之所以能给有限样本下的精确 FDR 保证,是因为它构造了一组可交换的对照变量,让真信号与假信号在交换下不可分辨(PMID 36854821)。分半 Jaccard 没有这种构造,它没有原假设下的精确分布,门槛只能靠经验标定或模拟——而模拟里填什么真值,它就答什么。

七、反转:不标定分数,改报选对率

反转不在「门槛调高还是调低」,在「该不该拿一个分数当主判据」。

第一步,把二值化往后挪。empirical Bayes 那条线给的方案是不做 0-1 判决,直接对每个参数报 local fdr 与 local false sign rate,且输入用效应量与标准误两个数,而非一个 p 值或 z 值(PMID 27756721)。这样「选对率」这个概念本身就被拆细了。

第二步,若一定要保留二值判决,就报两条直接估计:选对率(入选集合里真信号所占比例)与漏选率,而不是报一个间接的稳定性分数。配套做法是模拟端设 30 个候选、真信号 5 个,效应量、偏置、信度各三档,跑 1000 次分半与独立抽,拟合「超额稳定性 → 选对率」曲线并反解两个门槛,再用既有多设备丹田数据按文献共识参数做外部核对。

四格判据:超额稳定性 ≥0.6 且对应选对率 ≥70%,则 0.6 可留用;0.6 对应选对率 <70%,则线太松、须上移;0.3 对应选对率已 >50%,则 0.3 太严、应改判「降级为探索性」;若 0.4 到 0.7 区间曲线近乎平坦,则稳定性与原命题脱钩、该指标不能作主判据。

证据与局限

已验证: 稳定性与准确性是可分离的两个维度,集成方法提稳而性能仅相当或略好(PMID 25134091、PMID 35016102);小样本下方法间准确率相当而选出特征的精度与稳定性差异很大(PMID 22403633);stability selection 的阈值可用置换估 FDR 或自动校准到给定假阳性数(PMID 23089816、PMID 38143734);同一数据上不同选择器的假发现率可相差三十倍(PMID 35843027)。

部分验证: 上述均为方法学文献,其「选对率」是在模拟真值已知条件下算的;把同一套映射搬到丹田五参数上,须假定真实信号集已知,而这一点在现有多设备数据里并不成立。

待验证: 0.6 与 0.3 对应的选对率究竟是多少,从未量过;映射曲线是否在 0.4 到 0.7 之间平坦,未知;模拟曲线与实测核对之间的偏差量级未知。

相关阅读

证据等级: C
参考文献: PMID 25134091, PMID 35016102, PMID 22403633, PMID 23089816, PMID 38143734, PMID 35843027, PMID 29292533, PMID 36854821, PMID 27756721, PMID 25090041
L1 网络状态:该主题尚未进入 L1 对应网络——CORRESPONDENCE_NETWORK.md 节点表仅按「易学/中医侧」「物理侧」「生理/体验侧」三类域分派(原文「| 节点 | 各域的概念/量(五行、五脏、频率、颜色…) | 按域分派(见下表) |」),三类域均无「筛选阈值 / 稳定性分数 / 选对率 / 假发现率 / 局部 fdr / 阈值标定」这类测量学维度节点;边表止于 E-050(「六十甲子纳音 | 对应 | 60 个频率」,层级 L1、状态 open,现代证据栏原文「尚无任何可测来源」),无任何以筛选规则、稳定性分数或错误率门槛为源或目标的边。EDGE_TABLE_science-yixue.md 为反抽统计,仅有概念计数(丹田 51 篇 / 气功 48 篇 / 呼吸 189 篇 / 频率 407 篇)与最高证据级,无 L1/L2/L3 层级与 active/open/conflict 状态判定。


免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。

Overstability’s 0.6 and 0.3 are borrowed rulers: what needs calibration is not the score, but the mapping curve from \”score\” to \”accuracy\

Evidence level: C

0.6 holds up; 0.3 is brittle—these two lines have never been calibrated against any known ground truth value. They are merely markings carved onto a ruler from elsewhere.

I. The Origin of the Two Numbers

The claim is that “excess stability ≥0.6 indicates stability” and “<0.3 indicates brittleness.” The origin of these numbers lies in the empirical ranges for kappa within inter-rater reliability: values around 0.6 roughly correspond to “fair,” while values below 0.3 correspond to “poor.” However, the kappa framework itself is rooted in oral tradition and tends to over-penalize agreement under low base rates (PMID 25090041). Inserting a ruler borrowed from consistency research into parameter screening involves no step that has ever been calibrated against known ground truth.

II. Stability and Accuracy Are Inherently Two Distinct Dimensions

This distinction is explicitly separated in the feature selection literature. The FREL paper opens by stating that good feature selection algorithms are characterized by two factors: accuracy and stability; its ensemble version increases stability while accuracy merely becomes “better or equivalent” (PMID 25134091). Another study on ensemble selection shows a marked improvement in stability scores, yet classification performance remains only equivalent or slightly better (PMID 35016102). A more direct statement from a third study notes that under small sample sizes, classification accuracies across different methods can be equivalent, yet the selected features differ significantly in precision and stability (PMID 22403633). Stability does not equal correctness.

III. This Field Has Actually Calibrated Thresholds, But Targets Error Rates

The orthodox approach to stability selection links the selection probability threshold to the “expected number of false selections,” using a theoretical upper bound for control. Later, some researchers adopted permutation methods to estimate FDR and set thresholds, resulting in bounds looser than the FWER upper bound and allowing recovery of more true signals (PMID 23089816). Subsequently, others automated the calibration to cap expected false positives below 20 (PMID 38143734). All three paths point to the same conclusion: thresholds should be anchored on “how many errors will occur,” not anchored on a unitless score between 0 and 1.

IV. The Same Score Yields Selection Accuracy Differing by Orders of Magnitude Across Different Selectors

The anchor of “how many errors will occur” varies drastically depending on the selector. In a simulation study involving 3,300 iterations, stability selection using mixed-effects models kept the false discovery rate (FDR) stable below 0.02 with true positive rates ranging from 0.73 to 0.97; switching to Lasso within the same simulation yielded an FDR of 0.59 to 0.72 but a true positive rate ≥0.93 (PMID 35843027). Both methods “select parameters,” yet the false positive proportions differ by a factor of thirty. Therefore, the statement “stability 0.6 corresponds to what selection accuracy” lacks a subject: one must first specify which selector is being used. The mapping curve is not a single line, but a family of curves.

V. There Is “Post-Selection Reporting” Bias Within the Thresholds

The stability you calculate is performed after “these 5 have already been selected,” meaning the selected variables naturally exhibit higher stability. Heinze et al.’s review explicitly states that variable selection compromises the stability of the final model, the unbiasedness of coefficients, and the validity of p-values; therefore, it is recommended to report quantities derived from resampling the entire selection process as a standard metric (PMID 29292533). Post-selection reporting makes thresholds appear easier to pass.

VI. There Is No Calculable Null Distribution Here

There is also a hard constraint. Methods like knockoff can provide exact FDR may support in finite samples because they construct a set of exchangeable control variables, rendering true signals indistinguishable from false ones under exchangeability (PMID 36854821). Half-split Jaccard lacks this construction; it possesses no exact distribution under the null hypothesis, so thresholds must rely on empirical calibration or simulation—and in simulation, whatever truth values are filled in will dictate the answer.

VII. Reversal: Do Not Calibrate the Score; Report Selection Accuracy Instead

The reversal is not about “raising or lowering the threshold,” but whether “a single score should serve as the primary criterion.”

Step 1: Shift the Binary Decision. The empirical Bayes approach proposes not performing a 0-1 decision, but directly reporting the local false discovery rate (local fdr) and local false sign rate for each parameter, using effect size and standard error as inputs rather than a single p-value or z-score (PMID 27756721). This inherently fragments the concept of “selection accuracy.”

Step 2: If Binary Decisions Must Be Retained, Report Direct Estimates. Instead of reporting an indirect stability score, report selection accuracy (the proportion of true signals within the selected set) and omission rates. Accompanying procedures involve setting up a simulation with 30 candidates, 5 true signals, and three levels each for effect size, bias, and reliability; run 1,000 half-split and independent draws to fit the “excess stability → selection accuracy” curve, solve for two thresholds inversely, and then perform external validation using multi-device Dantian data according to literature-consensus parameters.

Four-Grid Criterion:
* If excess stability ≥0.6 AND corresponding selection accuracy ≥70%, then 0.6 may be retained.
* If excess stability 0.6 corresponds to selection accuracy <70%, then the line is too loose and must be raised.
* If excess stability 0.3 corresponds to selection accuracy already >50%, then 0.3 is too strict and should be reclassified as “exploratory.”
* If the curve between 0.4 and 0.7 is nearly flat, stability is decoupled from the proposition, and this metric cannot serve as a primary criterion.

Evidence and Limitations

Verified: Stability and accuracy are separable dimensions; ensemble methods increase stability while performance remains only equivalent or slightly better (PMID 25134091, PMID 35016102); under small samples, accuracies across methods are equivalent while precision and stability of selected features differ significantly (PMID 22403633); stability selection thresholds can be controlled via permutation estimation of FDR or automated calibration to a specific false positive count (PMID 23089816, PMID 38143734); false discovery rates across different selectors on the same data can differ by a factor of thirty (PMID 35843027).

Partially Verified: The above are methodological literature where “selection accuracy” is calculated under conditions of known simulated truth; applying the same mapping to multi-device Dantian parameters requires assuming the true signal set is known, which does not hold for current multi-device data.

To Be Verified: The exact selection accuracy corresponding to 0.6 and 0.3 has never been measured; whether the mapping curve is flat between 0.4 and 0.7 is unknown; the magnitude of deviation between simulated curves and empirical validation remains unknown.

Related Reading

Evidence Level: C
References: PMID 25134091, PMID 35016102, PMID 22403633, PMID 23089816, PMID 38143734, PMID 35843027, PMID 29292533, PMID 36854821, PMID 27756721, PMID 25090041
L1 Network Status: This topic has not yet entered the L1 Correspondence Network—the CORRESPONDENCE_NETWORK.md node table assigns nodes only by domain categories (“Yijing/TCM side,” “Physics side,” “Physiological/Experiential side”) based on concepts/quantities (Five Phases, Five Organs, Frequencies, Colors, etc.). None of the three domains contain measurement-level nodes related to “screening thresholds,” “stability scores,” “selection accuracy,” “false discovery rate,” “local fdr,” or “threshold calibration.” The edge table ends at E-050 (“Six Heavenly Stems and Earthly Branchs Na Yin | Corresponds | 60 Frequencies,” Level L1, status open; modern evidence column states “No measurable sources yet”). There are no edges originating from or terminating at screening rules, stability scores, or error rate thresholds. EDGE_TABLE_science-yixue.md is a reverse statistical count containing only concept counts (Dantian 51 articles / Qigong 48 articles / Breathing 189 articles / Frequency 407 articles) and highest evidence level, lacking L1/L2/L3 hierarchy and active/open/conflict status determination.


Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *