同一条山脊上四十个点,不算四十个样本:凶地评级模型的过拟合边界

Written by

in

Language

Evidence level: B

上一个模型说”把凶地评级和四个物探变量一起喂给机器学习”。这一篇问的是那台机器最少要喂多少个点才不说谎——答案是:先看你数的是点,还是山。

正文

凶地评级模型的设计图纸已经有了:风水师盲法评级做因变量,断层距离、岩性、土壤氡、地表电导率做自变量,训练一个能预测”凶不凶”的模型。剩下最硬的一关是样本量——用四十个点评级、四个变量,模型能不能站住?直觉会给出一个比例,比如”一个变量至少十个样本”。这个直觉在这一条线上会连错三次。

第一错:横轴画错了。 四十个评级点如果落在同一条山脊、同一个村、同一片花岗岩体上,它们不是四十个独立观测。德国做全国地生氡潜势图时手上有 4448 个实测点,仍然不敢按随机方式切分,而是把数据按 40 公里见方的空间块轮流留出来做交叉验证(PMID 33254926)——因为相邻点的自变量和因变量都高度相关,随机切分等于让模型在考场上遇见复习过的题。四十个点如果聚成三簇,有效样本量可能只有个位数。所以”样本量-特征数比”这条曲线的横轴不该是点数,而该是去除空间相关后的有效点数,这个数通常比表格里的行数小一个量级。

第二错:交叉验证会给你假安慰。 用交叉验证的误差去挑模型参数,再用同一个误差报性能,这是双重使用。Varma 与 Simon 用完全无信号的数据(两类特征分布相同)做过这件事:优化后的分类器交叉验证误差低于 30% 的数据集占 18.5%(收缩质心)和 38%(支持向量机),而它们在独立测试集上的表现与抛硬币无异;只有嵌套交叉验证能给出接近无偏的估计(PMID 16504092)。更麻烦的是,在合并的训练加测试数据上做特征筛选,制造的偏倚比调参更大,而且样本量到 1000 时 K 折交叉验证的偏倚仍然存在(PMID 31697686)。四十个点的规模,正好落在偏倚最严重的那一档。

第三错:变量是风水师圈出来的。 四个物探变量不是预先登记的,而是先看过评级、觉得”这几个大概有关”才纳入的。这就是逐步选择。GUSTO-I 试验(40830 例)的仿真显示:每变量 3 个事件时,8 变量模型里有 7 个变量的回归系数被高估超过 25%;每变量 10 个事件时有 3 个;而完全不做筛选时全部低于 20%(PMID 10513756)。筛选这个动作本身,就会把系数推向”看起来更显著”。

“一个变量十个样本”这条规则本身也站不住。 支持它的三项仿真里只有一项支持(PMID 27881078),另一项大样本仿真直接表明每变量事件数与预测性能(校准、区分度、概率误差)没有强关系,真正决定样本需求的是变量数、总样本量与事件比例三者(PMID 29966490)。Riley 等人给的三准则——收缩因子不低于 0.9、表观与校正后的 R² 差不超过 0.05、总体风险估得够准——同一个方法算出来,一个诊断模型只需每变量 4.8 个事件,一个复发静脉血栓模型要 23 个(PMID 30357870)。差四倍。所以正确做法不是查表,而是先说出你预期这个模型能解释多少方差,再反算样本量。

这就逼出本文最锋利的一刀:天花板已经被别人量过了。 法国做过一万多户室内氡测量,把地生氡潜势、建筑材料、建造年份、基础类型、楼层、生活习惯全部放进去,模型只解释了氡浓度对数值总变异的约 20%(PMID 24056050)。罗马尼亚 100 个案例的研究也说,地质基础能”合理解释”多数记录值——但这是定性表述,不是方差比例(PMID 31897872)。而爱尔兰那篇报”能解释约 75% 变异”的模型,解释的是地生氡潜势这个由土壤氡与渗透率合成的中间量,不是室内氡(PMID 37343860)。这两个数字经常被混着引用,但它们隔着一整层建筑与通风。推论很直接:如果凶地评级模型报出 R² 大于 0.4,它解释的几乎一定不是室内氡,而是它自己泄漏进来的某个中间量。

所以判据不能靠经验比例,得靠零信号基线。 把结局标签随机打乱五十次,每次重跑整条流程,取平均 AUC;凡超过 0.5 的部分全是过拟合。用这个方法实测,要让仅仅 10 个特征的 AUC 膨胀控制在 0.05 以内,样本量至少要 200(PMID 38614825)。影像组学的领域共识也是同一条:数据驱动建模必须预注册分析路径、必须有独立数据验证(PMID 33492473)。凶地评级这条线,应当先跑这个置换基线,再说自己的 R² 有多少是真的。

最后,该报的不是 R²,是增量。 只喂四个物探变量的模型是一个必须存在的基线;凶地评级模型的全部价值在于它比这个基线多解释了多少(ΔR²),以及这个增量的置信区间是否跨零。样本量曲线也该这样画:固定特征数,扫样本量,看 ΔR² 什么时候走平。现成的数字范式可以参考:随机森林在 500 例以下严重过拟合,1000 例配 3 个特征勉强可比,20 个以上特征就再度过拟合,3000 例以上才稳定(PMID 42394051)。四十个点配四个变量,落在哪一格,不用猜。

证据与局限

已验证: 空间块交叉验证可抑制空间自相关造成的性能虚高(PMID 33254926);用同一交叉验证误差既调参又报性能会显著低估真实误差,嵌套交叉验证近无偏(PMID 16504092);小样本下 K 折交叉验证偏倚在 n=1000 仍存在,且特征筛选的偏倚贡献大于调参(PMID 31697686);逐步选择会系统性高估入选变量系数(PMID 10513756);每变量事件数规则缺乏一致依据(PMID 27881078、29966490),最小样本量须按预期 R² 反算且结果可差四倍(PMID 30357870);地质与建筑变量合计只解释室内氡对数变异约 20%(PMID 24056050);结局置换可量化过拟合幅度,200 例方能承载 10 个特征(PMID 38614825)。

部分验证: “凶地评级”作为因变量的可行性尚未有实测数据,其与四个物探变量的关联强度未知;有效样本量的折算方法(如按空间块数计)在本领域尚无统一标准,只有邻域借鉴。

待验证: 四十点/四变量配置下置换基线 AUC 的实际膨胀量;ΔR² 相对纯物探基线的点估计与区间;跨地区(岭南/江南/华北)时样本需求是否随地质复杂度上升;评级点位聚簇程度与有效样本量的定量关系曲线。

相关阅读

证据等级: B(过拟合机制、样本量准则、方差天花板均有 PubMed 直接支撑;”凶地评级作为因变量”本身仍属研究设计提案,无实测)
参考文献: PMID 33254926, PMID 16504092, PMID 31697686, PMID 10513756, PMID 27881078, PMID 29966490, PMID 30357870, PMID 24056050, PMID 37343860, PMID 31897872, PMID 38614825, PMID 33492473, PMID 42394051


免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。

Forty Points on the Same Ridge Do Not Count as Forty Samples: The Overfitting Boundary in凶地 Rating Models

Evidence level: B

The previous model proposed “feeding the machine learning algorithm the adverse land rating alongside four geophysical variables.” This article asks: how many data points must be fed into that machine before it stops lying? The answer depends on whether you are counting points or mountains.

Main Text

The blueprint for the adverse land rating model is already in place: Feng Shui blind ratings serve as the dependent variable, while fault distance, lithology, soil radon, and surface electrical conductivity serve as independent variables. The goal is to train a model capable of predicting “adverse or not.” The most critical remaining hurdle is sample size: can a model stand firm with forty ratings and four variables? Intuition suggests a ratio, such as “at least ten samples per variable.” However, this intuition fails three times along this specific line.

First Error: The X-axis is drawn wrong. If forty rating points fall along the same ridge, in the same village, or within the same granite body, they are not forty independent observations. When Germany produced a national map of terrestrial radon potential, despite having 4,448 measured points, they dared not split the data randomly. Instead, they reserved blocks of 40×40 km for cross-validation (PMID 33254926) because the independent and dependent variables are highly correlated between adjacent points. Random splitting is equivalent to giving the model a review of past exam questions on the same day. If forty points cluster into three groups, the effective sample size may be only single-digit. Therefore, the x-axis of the “sample size-to-feature ratio” curve should not be the raw number of points, but rather the effective number of points after removing spatial correlation, a figure typically an order of magnitude smaller than the row count in a table.

Second Error: Cross-validation gives false comfort. Using cross-validation error to tune model parameters and then reporting performance based on the same error constitutes double dipping. Varma and Simon demonstrated this using completely noise-free data (where feature distributions are identical across classes): optimized classifiers showed cross-validation errors below 30% in 18.5% of datasets (shrinkage centroid) and 38% (support vector machines), yet their performance on independent test sets was no better than coin flipping; only nested cross-validation provided estimates close to unbiased (PMID 16504092). Complicating matters further, performing feature selection on the combined training and testing data creates bias larger than hyperparameter tuning, and even with K-fold cross-validation, bias persists when sample sizes reach 1,000 (PMID 31697686). A scale of forty points falls squarely into the category where bias is most severe.

Third Error: Variables are selected by the Feng Shui circle. The four geophysical variables were not pre-registered; they were included only after reviewing the ratings and concluding “these seem related.” This is stepwise selection. A simulation from the GUSTO-I trial (4,083 cases) showed that with 3 events per variable, 7 of the 8 variables in a model had regression coefficients overestimated by more than 25%; with 10 events per variable, 3 were affected; whereas without any selection, all remained below 20% (PMID 10513756). The act of selection itself pushes coefficients toward “appearing more significant.”

The rule of “ten samples per variable” also stands unsupported. Of the three simulations supporting it, only one does (PMID 27881078); another large-scale simulation directly indicates that the number of events per variable has no strong relationship with predictive performance (calibration, discrimination, probability error). What truly determines sample requirements are the number of variables, total sample size, and event proportion (PMID 29966490). Riley et al. provided three criteria—shrinkage factor not below 0.9, difference between apparent and corrected R² not exceeding 0.05, and accurate overall risk estimation—where the same method yielded a requirement of only 4.8 events per variable for a diagnostic model, but 23 for a recurrent venous thromboembolism model (PMID 30357870). That is a fourfold difference. Therefore, the correct approach is not to consult a table, but to first state how much variance you expect this model to explain, then back-calculate the sample size.

This forces the sharpest blade of this article: the ceiling has already been measured by others. France conducted indoor radon measurements in over 10,000 households, incorporating terrestrial radon potential, building materials, construction year, foundation type, floor level, and lifestyle habits; the model explained only about 20% of the total variance in radon concentration (PMID 24056050). A Romanian study of 100 cases also stated that geological foundations could “reasonably explain” most recorded values—but this is a qualitative statement, not a variance proportion (PMID 31897872). Meanwhile, the Irish study reporting a model that explains “about 75% of the variance” refers to terrestrial radon potential, an intermediate variable synthesized from soil radon and permeability, not indoor radon (PMID 37343860). These two figures are frequently cited together, yet they are separated by an entire layer of building and ventilation. The inference is direct: if the adverse land rating model reports an R² greater than 0.4, it almost certainly does not explain indoor radon, but rather some intermediate variable that leaked into the model.

Therefore, the criterion cannot rely on empirical ratios but must depend on a zero-signal baseline. Randomly shuffle the outcome labels fifty times, re-run the entire pipeline each time, and take the average AUC; any portion exceeding 0.5 indicates overfitting. Applying this method empirically, to keep the AUC inflation of a model with only 10 features within 0.05, the sample size must be at least 200 (PMID 38614825). The field consensus in radiomics is the same: data-driven modeling must have a pre-registered analysis path and independent data validation (PMID 33492473). For the adverse land rating line, one should first run this permutation baseline before claiming how much of the R² is genuine.

Finally, what should be reported is not R², but the increment. A model fed only with four geophysical variables constitutes a necessary baseline; the entire value of the adverse land rating model lies in how much additional variance it explains compared to this baseline (ΔR²) and whether the confidence interval of this increment crosses zero. The sample size curve should also be drawn this way: fix the number of features, sweep the sample size, and observe when ΔR² flattens. Existing numerical paradigms can be referenced: random forests suffer severe overfitting below 500 cases, are barely comparable with 3 features at 1,000 cases, become severely overfit again with more than 20 features, and only stabilize above 3,000 cases (PMID 42394051). With forty points and four variables, one does not need to guess which grid it falls into.

Evidence and Limitations

Verified: Spatial block cross-validation suppresses inflated performance caused by spatial autocorrelation (PMID 33254926); using the same cross-validation error for both tuning and reporting significantly underestimates true error, while nested cross-validation is nearly unbiased (PMID 16504092); K-fold cross-validation bias persists at n=1,000 in small samples, and feature selection contributes greater bias than tuning (PMID 31697686); stepwise selection systematically overestimates coefficients of selected variables (PMID 10513756); the rule of events per variable lacks consistent basis (PMID 27881078, 29966490), and minimum sample size must be back-calculated based on expected R² with results potentially differing by fourfold (PMID 30357870); geological and building variables combined explain only about 20% of the log-variance in indoor radon (PMID 24056050); outcome permutation quantifies overfitting magnitude, requiring 200 cases to support 10 features (PMID 38614825).

Partially Verified: The feasibility of “adverse land rating” as a dependent variable lacks empirical data; the strength of association between it and the four geophysical variables is unknown; methods for calculating effective sample size (e.g., counting spatial blocks) lack a unified standard in this field, relying only on neighborhood borrowing.

To be Verified: The actual inflation of permutation baseline AUC with forty points and four variables; the point estimate and interval for ΔR² relative to a pure geophysical baseline; whether sample requirements rise with geological complexity across regions (Lingnan/Jiangnan/Northern China); the quantitative relationship curve between rating point clustering and effective sample size.

Related Reading

Evidence Level: B (Overfitting mechanisms, sample size criteria, and variance ceilings have direct PubMed support; “adverse land rating as a dependent variable” remains a research design proposal without empirical data)
References: PMID 33254926, PMID 16504092, PMID 31697686, PMID 10513756, PMID 27881078, PMID 29966490, PMID 30357870, PMID 24056050, PMID 37343860, PMID 31897872, PMID 38614825, PMID 33492473, PMID 42394051


Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *