Jaccard 掉到 0.5,是规则脆还是样本被砍了一半:分半稳定性不能做减法分解

Written by

in

Language

Evidence level: C

分半稳定性少掉的那 0.5,不能拆成两个量相加——它先要减掉一个随机的底,再判有没有超额。

一、两个随机集合本来就能撞出 0.5

若从 30 个候选参数里各随机抽 10 个,两次的 Jaccard 期望不是零:交集约 3.3、并集约 16.7,比值约 0.2。候选池更小、入选数更多,这个底还会抬高。所以「0.5」里先有一块与规则好坏完全无关的东西。

同类账目在评分者一致性里有现成教训:118 名 ARDS 病人用两套标准判,原始一致率 0.73、kappa 只有 0.46、phi 0.63(PMID 11179128)。同一份数据三个数差出 0.27,差的就是机会水平。筛选规则的分半 Jaccard 一样要过这一关,而现行做法没人减。

二、减法分解在数学上不成立

Jaccard 是比值,不是可加量。真分半 J₁、伪分半 J₂、全样本重测 J₃,三者的期望各自是样本量与规则参数的复杂函数。J₁−J₂ 并不等于「互补结构效应」——两个比的差依赖各自基线水平。

看极端就明白:J₃=0.95、J₂=0.80、J₁=0.70,差值 0.15 与 0.10 看着可拆;J₃=0.55、J₂=0.50、J₁=0.45,同样的绝对差在比值尺度上份量完全不同。要拆只能用模型,把 Jaccard 期望写成 n 与规则参数的函数再拟合,不能做减法。

三、样本量效应与规则脆性在观测上恒等

这是最根本的一层,也是原设计绕过去的那一层。观测到 Jaccard=0.5 有两个解释:规则把阈值压在分布稀疏尾部,入选集合本来就在边界抖动;或者 n 太小,每个参数的估计标准误大到让排序整体洗牌。两者给出同一个数。

要分开,必须动一个、锁住另一个:锁规则改 n 大小,或锁 n 改规则严格度。原设计的三重对照都在动 n 的「结构」(互补还是独立、一次还是两次),唯独没动 n 的「大小」——它测不到样本量效应,只能测到「互补带来的额外相关」。真正的 2×2 是 n 大小 × 切分方式。

四、互补确实多出一块,但方向可能反

互补两半共享同一批人的谱相关结构,不是两份独立证据。经典信度理论早写过:测量误差相关会让信度估计膨胀,用系统构造的等价半在两次施测上可控制这种膨胀;若等价半构得不好,失真会走向相反方向,且当量表不满足本质 tau 等价时失真可以很大(PMID 11783569)。

放到筛选上:真分半的两半因共享结构,选出的集合可能比伪分半更「像」。那么 J₁−J₂ 是正的,会被读成「互补有害」,实际是共享结构抬高了相似度。方向判反,整条结论就反。

五、「掉到 0.5」可能只是尺子不准

分半这一步本身在小样本下不可靠:最优切分(Guttman 最大 λ4)在小子样里显著高估总体信度,分位数下界 λ4(0.05) 才是防机会资本化的那一档(PMID 24306557)。「分半 Jaccard」不是无偏观测,它自带偏。

还不止。Cronbach’s alpha 依赖 item 数,把数据切成固定时长段当 item 用会低估信度,换一种切法数值就变(PMID 19055503)。你的分半是按人切还是按次切、切几刀,直接决定 0.5 这个数是多少。

六、加样本能不能补回:有实测,答案分模型类

这是原设计最该问、却用减法回避的问题。已有两条硬结果:

参数模型上,选择偏差可用逆幂律学习曲线(IPL)外推校正,且外推本身就能回答「要不要再招人」(PMID 25086004)。非参数模型上,所有 bootstrap 乐观校正都显著高估,而这个高估不随样本量增大而减少(PMID 41391734)。

后者是关键:「加样本补不回」不是推论,是在特定模型类上被观测到的事实。所以「补不回」要看模型类,不是一个统一门槛。另两条参照:小样本下不同方法的分类准确率可以相当,但选出特征的精度与稳定性差异很大,外部交叉验证环能改善(PMID 22403633);而稳定性提高不等于选对了,ensemble 一类方法只是让重抽样下选出同一批(PMID 25134091、PMID 35016102)。

七、反转:不拆两笔账,改报超额稳定性加曲线斜率

正确顺序是三步,不是一次减法。

第一,减掉机会底。按给定候选池大小与入选数算随机 Jaccard 期望,报超额稳定性 =(J−J_chance)/(1−J_chance),让 0.5 变成可比数。注意减 chance 本身有陷阱:低 base rate 下 kappa 类指标会过度惩罚(PMID 25090041),所以随机底要用解析式算,别套 kappa。第二,入选频率用 subsampling 而非 bootstrap(PMID 26288150),并同时报真分半与独立抽两套。第三,报学习曲线斜率与外推平台,把「n 加到多少」变成可答问题。

四条判据:超额稳定性 ≥0.6 且独立抽口径下仍 ≥0.6,则规则站得住、分半数低是结构伪迹;<0.3 且学习曲线已平台,则规则真脆、0.5 作废线可落地;超额稳定性低但曲线仍陡,则是样本不足,正解是扩样不是改阈值;两套口径符号相反,则该设计没有识别力,任何 Jaccard 结论都不许出。

证据与局限

已验证: 原始一致率与机会校正后在同批数据上可差 0.27(PMID 11179128);最优分半在小样本下高估信度、分位数下界更稳(PMID 24306557);bootstrap 乐观校正在非参数模型上高估且不随样本量收敛(PMID 41391734);学习曲线可外推判断是否需扩样(PMID 25086004)。

部分验证: 「稳定」与「正确」可分离有间接证据(PMID 25134091、PMID 35016102),但缺同一数据集上「稳定性指标 vs 特征真值命中率」的并列对照;减 chance 后的 Jaccard 是否适合做筛选规则的主判据,尚无方法学研究直接验证。

待验证: 三口径(真分半 / 伪分半 / 全样本重测)在同一批丹田多设备数据上的实际数值差;超额稳定性的阈值 0.6 与 0.3 只是从类比推来,需实测标定;按人切与按次切两种分半口径给出的 Jaccard 差量级未知。

相关阅读

证据等级: C
参考文献: PMID 11179128, PMID 24306557, PMID 41391734, PMID 25086004, PMID 22403633, PMID 25134091, PMID 35016102, PMID 11783569, PMID 19055503, PMID 26288150, PMID 25090041
L1 网络状态:该主题尚未进入 L1 对应网络——CORRESPONDENCE_NETWORK.md 节点表仅按易学/中医侧、物理侧、生理/体验侧三类域分派,三类域均无「分半稳定性 / 重抽样口径 / 机会校正 / 学习曲线」类测量学维度节点;边表止于 E-050(「六十甲子纳音 | 对应 | 60 个频率」,状态 open),无任何以筛选规则、样本量或可复现性为源或目标的边。EDGE_TABLE_science-yixue.md 亦仅有概念计数(丹田 51 篇 / 气功 48 篇 / 呼吸 189 篇 / 频率 407 篇),无 L1/L2/L3 层级与 active/open/conflict 状态判定。


免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。

Jaccard Drops to 0.5: Is It Rule Fragility or Half the Samples? Semi-Stability Cannot Be Decomposed by Subtraction

Evidence level: C

The missing 0.5 in split-half stability cannot be decomposed into two additive components—it must first subtract a random baseline before determining if there is an excess.

I. Two Random Sets Naturally Collide at 0.5

If one randomly selects 10 parameters from a pool of 30 candidates twice, the expected Jaccard index is not zero: the intersection is approximately 3.3, the union approximately 16.7, yielding a ratio of roughly 0.2. A smaller candidate pool or a larger selection count would further elevate this baseline. Therefore, a portion of “0.5” is unrelated to the quality of the rule itself.

Precedents in inter-rater reliability offer a clear lesson: among 118 ARDS patients, two sets of criteria yielded an initial agreement rate of 0.73, a kappa of only 0.46, and a phi of 0.63 (PMID 11179128). The same dataset produced three metrics differing by 0.27, representing the chance level. Split-half Jaccard for screening rules must pass this test, yet current practices fail to subtract it.

II. Subtraction Is Mathematically Invalid for Jaccard

Jaccard is a ratio, not an additive quantity. The expectations for true split-half ($J_1$), pseudo-split-half ($J_2$), and full-sample retest ($J_3$) are each complex functions of sample size and rule parameters. The difference $J_1 – J_2$ does not equal a “complementary structural effect”—the difference between two ratios depends on their respective baseline levels.

Extreme cases clarify this: if $J_3=0.95, J_2=0.80, J_1=0.70$, the differences of 0.15 and 0.10 appear decomposable; however, if $J_3=0.55, J_2=0.50, J_1=0.45$, the same absolute difference represents vastly different proportions on a ratio scale. Decomposition requires modeling, expressing Jaccard expectation as a function of $n$ and rule parameters for fitting; subtraction is invalid.

III. Sample Size Effects and Rule Fragility Are Indistinguishable in Observation

This is the fundamental layer that the original design circumvented. Observing Jaccard = 0.5 admits two explanations: the rule suppresses the threshold into a sparse tail of the distribution where selection sets naturally fluctuate at the boundary; or $n$ is too small, causing large standard errors for each parameter that effectively reshuffles the overall ranking. Both yield the same number.

To separate them, one must fix one variable while holding the other constant: lock the rule and vary $n$, or lock $n$ and vary rule strictness. The original design’s three controls all manipulate the “structure” of $n$ (complementary vs. independent, once vs. twice), but never the “magnitude” of $n$. It cannot detect sample size effects, only “extra correlation arising from complementarity.” The true $2 \times 2$ design is $n$ magnitude $\times$ splitting method.

IV. Complementarity Adds a Component, But the Direction May Reverse

The two halves of the complement share the same spectral correlation structure among subjects; they are not two independent pieces of evidence. Classical reliability theory addresses this: measurement error correlation inflates reliability estimates, and using system-constructed equivalent halves across two administrations controls this inflation. If the equivalent halves are constructed poorly, distortion may move in the opposite direction, and when scales fail essential tau-equivalence, the distortion can be substantial (PMID 11783569).

Applied to screening: because the two halves of true split-half share structure, the selected sets may be more “similar” than pseudo-split-half. Thus $J_1 – J_2$ is positive, potentially misread as “complementarity harmful,” whereas it is actually shared structure inflating similarity. Reversing the direction judgment flips the entire conclusion.

V. “Dropping to 0.5” May Simply Reflect an Inaccurate Scale

The split-half step itself is unreliable under small samples: the optimal splitting method (Guttman maximum $\lambda_4$) significantly overestimates total reliability in small samples, while the percentile lower bound $\lambda_4(0.05)$ serves as the guard against capitalizing on chance (PMID 24306557). “Split-half Jaccard” is not an unbiased observation; it carries inherent bias.

Furthermore, Cronbach’s alpha depends on item count; treating fixed-duration data segments as items underestimates reliability, and changing the segmentation alters the value (PMID 19055503). Whether splitting by person or by session, and how many cuts are made, directly determines the magnitude of 0.5.

VI. Can Adding Samples Restore It? Empirical Evidence Varies by Model Class

This is the question the original design should have asked but avoided via subtraction. Two hard results exist:

Under parametric models, selection bias can be corrected via inverse power-law (IPL) learning curve extrapolation, and the extrapolation itself answers “whether to recruit more” (PMID 25086004). Under non-parametric models, all bootstrap optimism corrections significantly overestimate, and this overestimation does not decrease with increased sample size (PMID 41391734).

The latter is critical: “cannot be restored by adding samples” is not an inference but an observed fact within specific model classes. Thus, “cannot be restored” depends on the model class, not a unified threshold. Two additional references: under small samples, different methods can yield comparable classification accuracy, but precision and stability of selected features vary greatly; external cross-validation loops can improve this (PMID 22403633). However, increased stability does not equal correctness; ensemble methods merely ensure the same subset is selected under resampling (PMID 25134091, PMID 35016102).

VII. Reversal: Do Not Split the Books; Report Excess Stability and Curve Slope Instead

The correct sequence is three steps, not a single subtraction.

First, subtract the chance baseline. Calculate the random Jaccard expectation based on candidate pool size and selection count, then report excess stability = $(J – J_{chance}) / (1 – J_{chance})$, making 0.5 a comparable metric. Note the trap in subtracting chance: kappa-class metrics under low base rates excessively penalize (PMID 25090041); therefore, use analytical formulas for the random baseline rather than kappa. Second, use subsampling instead of bootstrap for selection frequency (PMID 26288150), and simultaneously report both true split-half and independent sampling sets. Third, report learning curve slope and extrapolated plateau, turning “how much $n$ is needed” into a solvable question.

Four criteria:
1. If excess stability $\ge 0.6$ and remains $\ge 0.6$ under independent sampling, the rule stands; low split-half is a structural artifact.
2. If $< 0.3$ and the learning curve has plateaued, the rule is truly fragile; the 0.5 cutoff can be grounded.
3. If excess stability is low but the curve remains steep, the issue is insufficient sample size; the correct solution is expanding samples, not adjusting thresholds.
4. If the two sampling methods yield opposite signs, the design lacks discriminatory power; no Jaccard conclusions should be drawn.

Evidence and Limitations

Verified: Raw agreement rates differ by 0.27 after chance correction on the same dataset (PMID 11179128); optimal split-half overestimates reliability in small samples while percentile lower bounds are more stable (PMID 24306557); bootstrap optimism correction overestimates in non-parametric models without converging with sample size (PMID 41391734); learning curves enable extrapolation to judge sample expansion needs (PMID 25086004).

Partially Verified: Indirect evidence exists that “stability” and “correctness” are separable (PMID 25134091, PMID 35016102), but lacks direct comparison of “stability metrics vs. feature true positive hit rates” on the same dataset; whether Jaccard after chance correction is suitable as a primary criterion for screening rules lacks direct methodological validation.

To Be Verified: Actual numerical differences across the three sampling ports (true split-half / pseudo-split-half / full-sample retest) on actual multi-device Dan Tian data; thresholds of 0.6 and 0.3 for excess stability are derived via analogy and require empirical calibration; the magnitude of Jaccard difference between person-based and session-based splitting is unknown.

Related Reading

Evidence Level: C
References: PMID 11179128, PMID 24306557, PMID 41391734, PMID 25086004, PMID 22403633, PMID 25134091, PMID 35016102, PMID 11783569, PMID 19055503, PMID 26288150, PMID 25090041
L1 Network Status: This topic has not yet entered the L1 correspondence network—the CORRESPONDENCE_NETWORK.md node table assigns nodes only across three domains: Yijing/TCM, Physics, and Physiological/Experiential; no measurement dimensions nodes exist for “split-half stability/resampling port/chance correction/learning curve” in any of these three domains; the edge table ends at E-050 (“Six Heavenly Stems and Ten Earthly Branchs Na Yin | Correspondence | 60 Frequencies”, status open), with no edges originating or terminating from/to screening rules, sample size, or reproducibility. EDGE_TABLE_science-yixue.md contains only concept counts (Dan Tian 51 / Qigong 48 / Breathing 189 / Frequency 407), lacking L1/L2/L3 hierarchy and active/open/conflict status determination.


Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *