金标准自己就不准:三质分型两模型 ΔAUC 头对头的稀释与污染检验

Written by

in

Language

Evidence level: B

在两把尺子比出胜负之前,得先量清当靶子的那把尺子自己有多大误差——因为低信度金标准会同时做两件相反的事:把真实的 ΔAUC 稀释到看不见,又把医生的族裔判读偏差灌进标签、推向错误方向。

正文

前几轮(#638-#640)我们反复强调:三质分型的”绝对阈值 vs 组内标准化”之争,会被族裔数据质量差异遮蔽。但还有一个更上游的漏洞——用来判定”谁属于哪个体质”的金标准,自己就不准。阿育吠陀 15 名医生独立判读同一批人,脉诊加权 kappa 只有 0.07、舌诊 0.17、体质 0.28(PMID 24191170),按 Landis-Koch 标准属”极差到尚可”。拿这样一把有噪声的尺子当靶子,两模型 ΔAUC 头对头的结论就可能同时被两股方向相反的力量扭曲:稀释(attenuation)与污染(contamination)。把这两股力量分开,本身就是一项可证伪的实验。

一、稀释:低信度金标准把 ΔAUC 压向零

经典流行病学早已证明:当一个二分类变量被随机错分时,用观测值估计的渐近相对效率就等于该变量的 kappa(PMID 7569489),kappa 与优势比衰减的关系亦有专文(PMID 2078611)。2025 年”可靠性悖论”的解析推导把这条线推到效应量层面:观测组间差异 ≈ 真值 × ICC × sin(π/2·κ)(PMID 41169915)。代入 kappa=0.28,sin(π/2×0.28)≈0.43——真实差异被压掉一半以上。

对 ΔAUC 头对头而言,这意味着:即便两模型在真标签下 AUC 相差 0.05,低信度金标准也会把这个差值整体向 0 收缩。诊断准确性研究给出同向证据:把连续参考标准强行二分会产生系统偏倚(PMID 30904568);参考标准不完美时,ROC/AUC 的点估计与检验都需要专门的非参框架(PMID 38994641、PMID 41351166)或潜类别/贝叶斯方法(PMID 19485679、PMID 26415924)。机器学习的标签噪声研究也一致——噪声标签平滑决策边界、压低判别性能(PMID 31837239、PMID 40117144)。稀释让 ΔAUC 变小、方向不变,但可能把”显著优势”缩成”看不见”。

二、污染:差分错分把 ΔAUC 推向错误方向

污染是更危险的对手。稀释源于随机错分(非差分),而医生的判读偏差是差分错分——错分概率依赖族裔。诊断准确性研究明确区分两类:差分验证偏倚会让准确性估计偏离到任意方向(PMID 24238322、PMID 21495059、PMID 29595130)。也就是说,如果金标准对某个族裔系统性误判(例如把该族裔的健康人成批打成”卡法型”),两模型的 ΔAUC 就不再是”谁更公平”的干净答案,而变成”谁恰好与偏差方向一致”。

医生族裔判读偏差有直接实证:PNAS 万人研究显示,黑人疼痛被系统性低估,且部分医生持有”黑人体质不同”的错误信念(PMID 27044069);产科决策中族裔影响剖宫产建议(PMID 36921720);儿科疼痛感知存在族裔偏差(PMID 38823604);临床医生隐性偏差被视为对黑人患者医源性伤害的路径(PMID 35976934)。污染可以反转 ΔAUC 方向,把”更公平的模型”错判成”更差的模型”。

三、怎么把两者分开:一套可执行的模拟设计

稀释是对称收缩,污染是定向平移——数学后果不同,因此可用同一套模拟分离两者:

(1)稀释梯度:取带已知真标签的数据,按 kappa 0.2→0.8 梯度注入随机错分,记录两模型 ΔAUC 的衰减曲线。若 ΔAUC 随 kappa 下降单调收缩、方向不变 → 稀释主导。
(2)污染注入:只对某一族裔标签施加定向误判(误判率 5%→30%),观察 ΔAUC 是否偏离甚至反转。若方向随误判族裔而翻转 → 污染主导。
(3)分离判据:ΔAUC 的”衰减曲线对称性”与”方向翻转点”是两套独立指纹;对真实数据同时拟合两者,可估计稀释占比与污染占比。
(4)最小可检出 ΔAUC:用 DeLong 方法(PMID 3203132)与诊断研究样本量公式(PMID 24582925)计算——在 kappa=0.28 下要检出真实 ΔAUC=0.05 需要多少样本;若所需样本量不可行,则实验本身就不该以 ΔAUC 为终点。

四、与易学的呼应

易学讲”格物致知”,第一件事是校准观测工具本身。三质、中医体质、藏医三因都是把连续状态离散成少数几格,格子边界靠人判——人的判读偏差就是格子的噪声。古人靠多师互参、以证验法压噪声;现代研究则用蒙特卡洛把”尺子的误差如何污染结论”算在明处。本系列 #638-#640 强调”先拉平数据质量再谈尺子”,这一轮把同一原则推进一层:先量清金标准自己的误差结构,再谈 ΔAUC 归因。 没有这一步,两把尺子的胜负只是噪声的胜负。

证据与局限

已验证: 三质医生间判读信度极低(脉诊 kappa=0.07、舌诊 0.17、体质 0.28,PMID 24191170);二分类变量错分使估计效率等于 kappa、并衰减优势比(PMID 7569489、PMID 2078611);低信度经 sin(π/2·κ) 衰减组间差异(PMID 41169915);差分验证偏倚可使准确性估计偏离至任意方向(PMID 24238322、PMID 21495059、PMID 29595130);医生存在族裔判读偏差的实证(PMID 27044069、PMID 36921720、PMID 38823604、PMID 35976934)。

部分验证: 参考标准不完美时需专门非参/潜类别/贝叶斯方法才能无偏估计 AUC(PMID 38994641、PMID 41351166、PMID 19485679、PMID 26415924);连续参考标准强行二分引入偏倚(PMID 30904568);标签噪声压低机器学习判别性能(PMID 31837239、PMID 40117144)。这些方法学结论成熟,但尚未在三质场景实测。

待验证: 三质 ΔAUC 头对头实验中”稀释占比 vs 污染占比”的分离——需多族裔 HRV 数据 + 已知真标签模拟 + kappa 梯度与族裔定向误判双注入 + 衰减曲线对称性与方向翻转点联合拟合;以及 kappa=0.28 下最小可检出 ΔAUC 的样本量可行性。

相关阅读

证据等级: B(信度实测值、kappa 衰减公式、差分偏倚与医生族裔偏差均有 PubMed 直接支撑;”稀释/污染分离方案”属方法学设计与初步探索)
参考文献: PMID 24191170, PMID 7569489, PMID 2078611, PMID 41169915, PMID 30904568, PMID 38994641, PMID 41351166, PMID 24238322, PMID 21495059, PMID 29595130, PMID 27044069, PMID 36921720, PMID 3203132, PMID 24582925


免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。

Gold Standard Is Not Reliable: Dilution and Contamination Tests of Three-Phenotype Classification and Two Models via Head-to-Head ΔAUC

Evidence level: B

Before comparing two rulers to determine a winner, one must first measure the error margin of the target ruler itself. Low-reliability gold standards perform two opposing actions simultaneously: they dilute the true ΔAUC into invisibility while injecting racial reading biases from clinicians into the labels, pushing them in the wrong direction.

Main Text

In previous rounds (#638–#640), we repeatedly emphasized that the debate between “absolute thresholds” and “within-group standardization” in three-trait subtyping is obscured by disparities in ethnic data quality. However, there is a more upstream flaw: the gold standard used to determine “who belongs to which constitution is itself inaccurate. In an Ayurvedic study of 15 physicians independently reading the same cohort, pulse diagnosis had a weighted kappa of 0.07, tongue diagnosis 0.17, and constitution 0.28 (PMID 24191170); by Landis-Koch standards, this ranges from “poor to fair.” Using such a noisy ruler as the target means that head-to-head conclusions on ΔAUC between two models may be simultaneously distorted by two opposing forces: attenuation and contamination. Separating these two forces constitutes a falsifiable experiment in itself.

I. Dilution: Low-Reliability Gold Standards Push ΔAUC Toward Zero

Classical epidemiology has long demonstrated that when a binary variable is misclassified at random, the asymptotic relative efficiency estimated from observed values equals the kappa of that variable (PMID 7569489), and specific literature details the relationship between kappa and odds ratio attenuation (PMID 2078611). The 2025 “reliability paradox” analytical derivation extends this line to effect sizes: observed between-group differences ≈ true value × ICC × sin(π/2·κ) (PMID 41169915). Substituting κ=0.28, sin(π/2×0.28)≈0.43—true differences are compressed by more than half.

For head-to-head ΔAUC, this implies that even if two models differ by 0.05 in AUC under true labels, a low-reliability gold standard will compress this difference toward zero. Diagnostic accuracy studies provide corroborating evidence: forcing continuous reference standards into binary categories introduces systematic bias (PMID 30904568); when reference standards are imperfect, point estimates and testing of ROC/AUC require specialized nonparametric frameworks (PMID 38994641, PMID 41351166) or latent class/Bayesian methods (PMID 19485679, PMID 26415924). Machine learning label noise research also aligns—noisy labels smooth decision boundaries and suppress discriminative performance (PMID 31837239, PMID 40117144). Dilution reduces ΔAUC magnitude without changing direction, but may shrink “statistically significant advantage” into “invisibility.”

II. Contamination: Differential Misclassification Pushes ΔAUC in the Wrong Direction

Contamination is the more dangerous adversary. Dilution stems from random misclassification (non-differential), whereas physician reading biases are differential misclassification—misclassification probabilities depend on ethnicity. Diagnostic accuracy research explicitly distinguishes two types: differential verification bias can push accuracy estimates in arbitrary directions (PMID 24238322, PMID 21495059, PMID 29595130). In other words, if the gold standard systematically misjudges a specific ethnicity (e.g., batching healthy individuals from that group as “Kapha-type”), the ΔAUC between two models is no longer a clean answer of “who is fairer,” but becomes “who coincidentally aligns with the bias direction.”

Direct empirical evidence of physician ethnic reading bias exists: a PNAS study of one million people showed Black pain is systematically underestimated, with some physicians holding erroneous beliefs that “Black constitutions differ” (PMID 27044069); ethnicity influences cesarean section recommendations in obstetric decision-making (PMID 36921720); ethnic bias exists in pediatric pain perception (PMID 38823604); and clinical physician implicit bias is viewed as a pathway to iatrogenic harm against Black patients (PMID 35976934). Contamination can reverse ΔAUC direction, misjudging a “fairer model” as a “worse model.”

III. Separating the Two: An Executable Simulation Design

Dilution is symmetric contraction; contamination is directional translation—mathematical consequences differ, so the same simulation framework can separate them:

  1. Dilution Gradient: Take data with known true labels and inject random misclassification at a kappa gradient of 0.2→0.8, recording the attenuation curves of ΔAUC for both models. If ΔAUC monotonically contracts without direction change as kappa decreases → dilution dominates.
  2. Contamination Injection: Apply directional misclassification to labels for only one ethnicity (misclassification rate 5%→30%) and observe if ΔAUC deviates or reverses. If direction flips with the misclassified ethnicity → contamination dominates.
  3. Separation Criteria: The “symmetry of the attenuation curve” and the “direction flip point” are two independent fingerprints; jointly fitting both to real data can estimate dilution proportion versus contamination proportion.
  4. Minimum Detectable ΔAUC: Calculate using the DeLong method (PMID 3203132) and diagnostic study sample size formulas (PMID 24582925)—how many samples are needed to detect a true ΔAUC=0.05 under κ=0.28; if the required sample size is infeasible, the experiment itself should not use ΔAUC as its endpoint.

IV. Resonance with Yijing Studies

Yijing Studies emphasizes “investigating things to acquire knowledge,” with the first step being calibrating the observation tool itself. Three-trait subtyping, TCM constitutions, and Ayurvedic tridosha all discretize continuous states into a few grids; grid boundaries rely on human judgment—human reading bias is the noise within the grids. Ancients relied on cross-referencing multiple masters and verifying methods with evidence to suppress noise; modern research uses Monte Carlo simulations to calculate explicitly “how the ruler’s error contaminates conclusions.” This series (#638–#640) emphasized “leveling data quality before discussing the ruler”; this round advances the same principle: measure the error structure of the gold standard itself before attributing ΔAUC. Without this step, the victory of two rulers is merely a contest of noise.

Evidence and Limitations

Verified: Extremely low inter-physician reliability in three-trait diagnosis (pulse κ=0.07, tongue κ=0.17, constitution κ=0.28, PMID 24191170); misclassification of binary variables reduces estimation efficiency to kappa and attenuates odds ratios (PMID 7569489, PMID 2078611); low-reliability factors attenuate between-group differences via sin(π/2·κ) (PMID 41169915); differential verification bias can shift accuracy estimates to arbitrary directions (PMID 24238322, PMID 21495059, PMID 29595130); empirical evidence of physician ethnic reading bias exists (PMID 27044069, PMID 36921720, PMID 38823604, PMID 35976934).

Partially Verified: Imperfect reference standards require specialized nonparametric/latent class/Bayesian methods for unbiased AUC estimation (PMID 38994641, PMID 41351166, PMID 19485679, PMID 26415924); forcing continuous reference standards into binary categories introduces bias (PMID 30904568); label noise suppresses machine learning discriminative performance (PMID 31837239, PMID 40117144). These methodological conclusions are mature but have not yet been empirically tested in the three-trait context.

To Be Verified: Separation of “dilution proportion vs. contamination proportion” in head-to-head three-trait ΔAUC experiments—requires multi-ethnic HRV data + known true label simulations + dual injection of kappa gradients and ethnic-specific directional misclassification + joint fitting of attenuation curve symmetry and direction flip points; as well as sample size feasibility for minimum detectable ΔAUC under κ=0.28.

Related Reading

Evidence Level: B (Direct PubMed support for reliability measurements, kappa attenuation formulas, differential bias, and physician ethnic bias; “dilution/contamination separation scheme” is methodological design with preliminary exploration)
References: PMID 24191170, PMID 7569489, PMID 2078611, PMID 41169915, PMID 30904568, PMID 38994641, PMID 41351166, PMID 24238322, PMID 21495059, PMID 29595130, PMID 27044069, PMID 36921720, PMID 3203132, PMID 24582925


Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *