Evidence level: B
两个模型拿到几乎一样的 Brier 评分,未必是同一回事——把综合准确度拆成”校准”和”判别”两块之后,运气粗筛的价值到底落在哪一块,才第一次说得清。
正文
上一篇提出了一个三模型头对头实验,用来裁决”运气粗筛+DLNM”是否真比”纯DLNM”预测得准。但只比一个 AUC 或者一个总分,会漏掉一个要命的问题:同样的总分,可能是两件完全不同的事。
Brier 评分(预测概率与真实结果的均方误差)是传染病与环境预测里最常用的综合准确度指标,问题在于它是个”复合分数”——它同时装着两样东西:模型能不能把高风险和低风险排对顺序(判别),以及模型给出的概率数字本身准不准(校准)。Rufibach 用一个很小的例子说明过:在有些情况下,一个校准更差的模型反而拿到更低的 Brier 分,因为它的排序能力更强(PMID 20189763)。换句话说,Brier 低不等于校准好,Brier 高也不等于没本事。
这就把”运气粗筛到底有没有用”从一个问题变成了两个问题。
第一,运气粗筛是不是只改了概率水平? 干支是固定的历法标签,如果它只是让模型”整体上把某些年份的概率抬高一点”,那会改变 reliability(校准)那一项,却不会改变模型对”哪个病人、哪一年风险更高”的排序——resolution(判别)项不变。这种价值是装饰性的,极端情况下还有害:把风险概率系统性说高或说低,会误导预警阈值。
第二,运气粗筛是不是真的带来了新的排序信息? 如果”三年化疫”的长滞后(1-3 年)真的携带独立于短期气象(0-3 周)的信息,那么它应该在 resolution 项上留下痕迹——因为长滞后改变的是”哪些年份算高风险年份”这个顺序判断。这才是”真本事”。
所以正确的比较协议不是”谁的 Brier 低”,而是把 Brier 拆成三块:reliability(校准,越小越好)、resolution(判别贡献,越大越好)、uncertainty(由结局基准率决定的、与模型无关的常数项)。只有拆开,才知道差异落在哪一块。近年方法学也在往这个方向走:Yang 等把 Brier 拆成”概率预测误差”与”结局自身方差”两部分,剔掉与模型无关的方差后比较更灵敏(PMID 36031854);Zhu 等进一步把加权 Brier 显式分解为判别成分与校准成分,并把两者与临床效用挂钩(PMID 41103457)。
光有分解还不够,还得配三样纪律。
一是配对检验,不是独立检验。 三个模型是在同一批病例、同一批年份上评估的,它们的 AUC 估计量互相相关。DeLong 1988 年的非参数方法用广义 U 统计量给出协方差矩阵,才是比较相关 ROC 曲线下面积的标准做法(PMID 3203132)。若把两个高度相关的 AUC 当成独立样本丢进两样本 z 检验,会因为忽略正协方差而把差异算得过于保守,甚至给出错误结论。
二是校准要看图,也要看多个数。 校准不能只报一个 Hosmer-Lemeshow 的 p 值。Huang 等的教程明确指出,判别与校准在模型评估里承担不同角色,校准应同时用校准截距、校准斜率、Spiegelhalter z 等多个统计量再加可靠性图(校准曲线)来判断——因为单一摘要统计量会掩盖”某些风险区间系统性高估、另一些区间系统性低估”这类结构性错误(PMID 32106284)。
三是预先登记比较清单。 三模型 × 多滞后窗 × 多领先时间 × 多指标,比得越多越容易撞假阳性,必须事先登记要比的模型和多重校正方案,避免”事后挑最好看的结果”。
把这三条合起来,这个实验能给出三种明确的结果解读,每一种都直接对应运气理论的一个版本:
- 若粗筛的 Brier 优势主要来自 reliability 改善、而 DeLong 检验显示 AUC 无显著判别提升 → 运气改的是”概率水平”,对分级预警有用,但不构成”运气携带新信息”的证据;
- 若主要来自 resolution 改善、且伴随 AUC 显著提升 → 运气提供了独立于气象的排序信息,这是运气粗筛”真本事”的最强证据;
- 若两块都无显著改善 → 粗筛是装饰性的,部署需谨慎。
对五运六气研究而言,这个分解实验的额外价值在于:它把”运气有没有用”这个容易吵架的定性问题,变成一份可核对的分账表——运气粗筛的每一分增量,到底是记在”校准”账上还是”判别”账上,白纸黑字写清楚。 这比一句”提高了预测精度”要诚实得多,也更容易被后续研究检验。
最后提醒一点:Brier 分低也不自动等于”更值得用”。Zhu 等把临床效用引入后指出,模型的价值最终取决于它被用在哪个决策阈值上(PMID 41103457)。对一个只在”极高风险才预警”的阈值下使用的系统,判别好、校准略差的模型可能比反过来更有用。所以三模型头对头实验的终点,应该是三份报告:判别报告、校准报告、效用报告——而不是一个总分排名。
证据与局限
已验证: Brier 评分是可分解的复合指标,其校准与判别成分能被分别估计(PMID 20189763、36031854、41103457);比较同一批样本上多个模型 AUC 的配对非参数方法是成熟的(DeLong,PMID 3203132);校准评估应结合多个统计量与可靠性图,不能只靠单一拟合检验(PMID 32106284)。
部分验证: “运气粗筛+DLNM”在北京结核病回顾中的分层思路初步可行(来源文章已引 PMID 26525546),但样本单中心、指标仅相关系数/准确率,从未做同数据集的 Brier 分解。
待验证: 在同一时间外验证协议下,对”纯DLNM vs 运气粗筛+DLNM vs 运气纯分层”三模型做 Brier 三分量分解(reliability/resolution/uncertainty)+ DeLong 配对 AUC 检验 + 校准曲线,判定运气粗筛的增量落在校准还是判别——全球尚无此实验。
相关阅读
证据等级: B
参考文献: PMID 3203132, PMID 20189763, PMID 32106284, PMID 36031854, PMID 41103457
免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。
Did Luck Coarse Filtering Change Probabilities or Sorting? A Head-to-Head Brier Decomposition Decision Among Three Models
Evidence level: B
Two models may yield nearly identical Brier scores without representing the same underlying mechanism—after decomposing comprehensive accuracy into “calibration” and “discrimination,” it is only now possible to clarify where the value of rough screening of luck actually resides.
Main Text
The previous article proposed a head-to-head experiment involving three models to adjudicate whether “rough screening of luck + DLNM” truly outperforms “pure DLNM” in prediction accuracy. However, comparing solely on a single AUC or a total score overlooks a critical issue: identical total scores may represent two fundamentally different outcomes.
The Brier score (mean squared error between predicted probabilities and actual outcomes) is the most widely used composite accuracy metric in infectious disease and environmental prediction. Its limitation lies in being a “composite score”—it simultaneously encapsulates two distinct aspects: whether the model correctly ranks high-risk versus low-risk cases (discrimination) and whether the probability values it outputs are accurate (calibration). Rufibach illustrated this with a small example: in certain scenarios, a model with poorer calibration can achieve a lower Brier score because its sorting capability is stronger (PMID 20189763). In other words, a low Brier score does not equate to good calibration, nor does a high score imply lack of capability.
This reframes the question of “whether rough screening of luck is useful” from a single inquiry into two distinct ones.
First, does rough screening of luck merely adjust probability levels? The Heavenly Stems and Earthly Branches serve as fixed calendrical labels. If their function is only to shift the model’s overall probability upward for certain years, they would alter the reliability (calibration) component while leaving the resolution (discrimination) component unchanged, as the model’s ranking of “which patient/year carries higher risk” remains consistent. Such value is decorative; in extreme cases, it can be harmful: systematically overestimating or underestimating risk probabilities may mislead warning thresholds.
Second, does rough screening of luck genuinely introduce new sorting information? If the long lag associated with “three-year epidemic cycles” (1–3 years) truly carries information independent of short-term meteorological data (0–3 weeks), it should leave a trace in the resolution component—because long lags alter the sequential judgment of “which years constitute high-risk years.” This constitutes the model’s “true capability.”
Therefore, the correct comparison protocol is not “who has the lower Brier score,” but rather decomposing the Brier score into three parts: reliability (calibration, smaller is better), resolution (discrimination contribution, larger is better), and uncertainty (a constant term determined by the baseline rate of the outcome, independent of the model). Only through decomposition can we identify where the differences lie. Recent methodological advances are moving in this direction: Yang et al. decomposed the Brier score into “probability prediction error” and “outcome variance,” enhancing sensitivity by comparing only after removing variance unrelated to the model (PMID 36031854); Zhu et al. further explicitly decomposed the weighted Brier score into discrimination and calibration components, linking both to clinical utility (PMID 41103457).
Decomposition alone is insufficient; three additional disciplines must accompany it.
First, use paired testing rather than independent testing. The three models were evaluated on the same cohort of cases and years; thus, their AUC estimators are correlated. The non-parametric method proposed by DeLong in 1988, utilizing generalized U-statistics to provide the covariance matrix, remains the standard approach for comparing correlated ROC curve areas (PMID 3203132). Treating two highly correlated AUCs as independent samples in a two-sample z-test would ignore positive covariance, rendering the difference estimate overly conservative and potentially leading to erroneous conclusions.
Second, assess calibration through visual inspection and multiple statistics. Calibration should not rely solely on a Hosmer-Lemeshow p-value. Huang et al.’s tutorial explicitly clarifies that discrimination and calibration serve distinct roles in model evaluation; calibration should be judged using multiple statistics—including calibration intercept, calibration slope, and Spiegelhalter z-values—alongside reliability plots (calibration curves)—because a single summary statistic may obscure structural errors such as systematic overestimation in certain risk intervals and underestimation in others (PMID 32106284).
Third, pre-register comparison lists. With three models × multiple lag windows × multiple lead times × multiple metrics, an excessive number of comparisons increases the risk of false positives. It is essential to pre-register the models to be compared and multiple correction strategies to avoid “selecting the most visually appealing results post-hoc.”
Synthesizing these three points, this experiment can yield three clear interpretations of results, each directly corresponding to a version of the luck theory:
- If the Brier advantage from rough screening stems primarily from reliability improvement, while the DeLong test shows no significant discrimination gain in AUC → luck modifies “probability levels,” useful for tiered warnings but failing to constitute evidence that “luck carries new information”;
- If it stems primarily from resolution improvement, accompanied by a significant AUC increase → luck provides sorting information independent of meteorology, representing the strongest evidence for the “true capability” of rough screening of luck;
- If neither component shows significant improvement → rough screening is decorative, and deployment should proceed with caution.
For research on Five Movements and Six Qi, the additional value of this decomposition experiment lies in transforming the easily contentious qualitative question of “whether luck is useful” into a verifiable ledger: clearly documenting whether every incremental point of value from rough screening is recorded under “calibration” or “discrimination.” This is far more honest than simply stating “improved prediction accuracy” and is easier for subsequent studies to validate.
Finally, a reminder: a low Brier score does not automatically equate to “greater utility.” Zhu et al. noted after introducing clinical utility that a model’s value ultimately depends on the decision threshold at which it is applied (PMID 41103457). For a system used only under a threshold warning for “extreme high risk,” a model with strong discrimination but slightly poorer calibration may be more useful than the reverse. Thus, the endpoint of the head-to-head experiment among three models should be three reports: a discrimination report, a calibration report, and a utility report—not a single total-score ranking.
Evidence and Limitations
Verified: The Brier score is a decomposable composite metric, with calibration and discrimination components estimable separately (PMID 20189763, 36031854, 41103457); paired non-parametric methods for comparing AUCs across multiple models on the same sample are established (DeLong, PMID 3203132); calibration assessment should combine multiple statistics with reliability plots rather than relying on a single fit test (PMID 32106284).
Partially Verified: The stratification approach of “rough screening of luck + DLNM” in the Beijing tuberculosis retrospective study is preliminarily feasible (source article cites PMID 26525546), but the sample was single-center, metrics were limited to correlation coefficients/accuracy, and no Brier decomposition on the same dataset was performed.
To Be Verified: Under a separate-time validation protocol, performing Brier three-component decomposition (reliability/resolution/uncertainty) + DeLong paired AUC testing + calibration curves for the three models (“pure DLNM” vs. “rough screening of luck + DLNM” vs. “luck pure stratification”) to determine whether the incremental value of rough screening falls under calibration or discrimination—no such experiment exists globally.
Related Reading
- Pure DLNM vs. Rough Screening of Luck + DLNM: How to Design a Fair Head-to-Head Comparison Among Three Models?
- Five Movements and Six Qi + Modern Statistical Models: Superposition or Interference?
- Luck Models vs. Modern Epidemiology
- Research Pool: Spatiotemporal Frequency Topology
Evidence Level: B
References: PMID 3203132, PMID 20189763, PMID 32106284, PMID 36031854, PMID 41103457
Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.
Leave a Reply