档数越多样本量越爆炸,但发表的论文照样几十例:是真能扛,还是换了系数把欠功效盖住了

Written by

in

Language

Evidence level: C

前一篇算出的样本量表显示四档要 1850 例、五档要 34225 例,而真实发表的等级一致性研究大多只有几十到几百例——要么这些研究集体欠功效,要么它们靠换系数把爆炸部分绕过去了。这一篇要审的就是这条逃生通道。

正文

先把上一条的账摆出来。在”两次评级次档质量均匀摊开”的假设下,测一个 0.90 的中高一致率,若要求估计精度 δ=0.05,四档量表需要约 1850 例,五档需要约 34225 例。这个增长不是线性的,是档数每加一档就翻一个量级的爆炸式增长。可去翻任何一本发表过的等级一致性研究——脉诊辨证、影像分级、症状量表——样本量几乎清一色是几十到几百例。

这产生了一个尖锐的问题:如果样本量表是对的,那现存所有四档以上、报告”一致性中等、可用于临床”的研究,全都在严重欠功效的状态下发表了,其系数区间宽到撑不起任何裁决。但这些论文照样过审、照样发表、照样被引用。两种解释必须选一个,或者都要:第一,它们确实欠功效,结论不可靠,只是审稿人没算这笔账;第二,它们换了某个对档数不敏感的系数,把爆炸部分绕过去了。

第二条通道是真实存在的,文献里甚至写得明明白白。方法学研究的经典结论是:二次加权 κ 随档数增加系统性升高,而线性加权 κ 对档数基本不敏感(PMID 8834562)。换句话说,同一批数据,从二次加权换成线性加权,档数惩罚就大半消失了。蒙特卡洛研究还补了一刀:档数与样本量同时增大时,加权系数的估计精度反而下降——”多招点受试者”并不自动换来精度(PMID 33853549)。更扎眼的是一个中医场景的实例:五位中医师对 42 例做自由边际评级,Fleiss κ 与 Gwet AC2 算出的结论差了整整一个等级(PMID 29851495)。系数选型不是技术细节,它直接决定你论文最后一行写”一致性良好”还是”一致性堪忧”。

有趣的是,这条逃生通道本身在研究内核里就是一处公开冲突。一边的观点是:抗灰区(中低一致率)最优的恰恰是 Gwet AC2 和 Brennan-Prediger κ,但所有加权系数在”中低一致性 + 大样本 + 类别数多”的组合下精度显著下降——也就是档数惩罚躲不掉,AC2 只是相对最好的那个。另一边的模拟结论方向相反:Fleiss κ 和 Conger κ 在序数分类上反而更高,而尺度维数增加只对名义分类有助、对序数分类无助,所以像五脏辨证这种多等级序数场景,Gwet AC2 恰是比 κ 更稳健的选择。两派共同承认的是”换系数改变结论”,分歧在于哪个方向换才是对的——这正是欠功效审计最该吃进去的背景:如果连选哪个系数都未裁决,那”低 n 报出的系数”的参考价值要再打一层折扣。

还有一个常被漏掉的层次:分层设计下 AC1 的样本量单位与齐性检验根本对不上——有的研究算的是评定者数,有的算的是病例数,两者不可直接互校(PMID 40298274)。所以做审计时,”该有多少样本”这个问题本身要先统一单位,否则实际 n 与应有 n 的比值会算出假象。

怎么裁决?做法是文献计量:检索不少于 30 篇报告四档及以上有序一致性系数的临床研究,逐篇提取样本量、档数、系数类型、报告区间宽度,然后按样本量表反推”这个设计应有的最小 n”,计算实际 n 与应有 n 的比值分布。判据分三路:若七成以上研究的实际 n 低于应有值的一半,欠功效立得住,此后任何多档判读设计必须先过功效核算;若比例低于三成,说明存在模型没捕捉到的降维机制(比如名义五档实际只用了三档),样本量因子要补一个”有效档数”修正项;若分布呈双峰——一部分严重不足、一部分充裕——那差异来源就要归因到系数类型或材料难度梯度,顺手产出一份”什么条件下可以省样本”的清单。

这条审计对本项目的意义很直接:脉诊、辨证、穴位判定这些核心研究线,全部依赖”几位评定者对同一批病例分等级”的一致性数据。如果那条逃生通道被证实,那么引用链条里所有”一致性中等”的文献,都要先问一句:这几十例,是真能扛,还是系数换出来的体面。

证据与局限

已验证: 二次加权 κ 随档数升高而线性加权 κ 不敏感(PMID 8834562);灰区场景下系数精度随档数与样本量交互变化(PMID 33853549);中医师评级中 Fleiss κ 与 AC2 结论差一个等级(PMID 29851495)——”换系数改变结论”这条通道文献证据充分。

部分验证: 分层设计中样本量单位错配问题有专门研究但覆盖面窄(PMID 40298274);”换系数绕开档数惩罚”在已发表文献中实际发生的频率,目前没有任何直接统计。

待验证: ≥30 篇多档一致性研究的欠功效比例;实际 n / 应有 n 比值分布的形状;系数类型能否解释双峰。以上全部为提案,未执行。

相关阅读

证据等级: C
参考文献: PMID 8834562, PMID 33853549, PMID 29851495, PMID 40298274

L1 网络状态:E-060 标签信度(KB-317,conflict,未裁决)——「闸门规格撞上方法学反例」:AC1 分层设计样本量 n≥50 与 KB-301 的「120 例」单位错配不可互校;「所有加权系数在中低一致性 + 大样本 + 类别数多时精度显著下降」(PMID 33853549);KB-333(conflict,中)另立第三处错配——「Fleiss κ 与 Conger κ 在序数分类上悖论性更高,尺度维数增加只对名义分类有助、对序数分类无助,Gwet AC2 恰是序数场景更稳健的选择」。本主题(多档一致性研究文献层面的欠功效审计)尚未进入 L1 对应网络,E-060 为最近相邻边。


免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。

The more variables, the more samples explode, yet published papers still have dozens of cases: is it truly robust or did they swap coefficients to cover up low power?

Evidence level: C

The previous calculation showed that a four-tier scale requires 1,850 cases and a five-tier scale requires 34,225 cases to achieve a moderate-to-high consistency rate of 0.90 with an estimation precision of δ=0.05, assuming “uniform allocation across secondary tiers.” However, most published studies on grade consistency—such as pulse diagnosis, imaging grading, or symptom scales—typically involve only dozens to hundreds of cases. Either these studies collectively suffer from underpowering, or they circumvented the exponential growth by swapping coefficients. This paper audits that escape route.

Main Text

Let us first lay out the arithmetic from the previous section. Under the assumption of “uniform allocation across secondary tiers,” achieving a moderate-to-high consistency rate of 0.90 with an estimation precision of δ=0.05 requires approximately 1,850 cases for a four-tier scale and about 34,225 cases for a five-tier scale. This growth is not linear; it is an explosive increase, doubling in magnitude with each additional tier. Yet, any review of published grade consistency studies—pulse diagnosis, imaging grading, symptom scales—reveals that sample sizes are almost exclusively in the range of dozens to hundreds.

This raises a sharp question: if the sample scale is correct, then all existing studies reporting “moderate consistency, clinically useful” with four or more tiers have been published under severe underpowering conditions, with coefficient intervals too wide to support any decision. Nevertheless, these papers were accepted, published, and cited. We must choose between two explanations, or accept both: First, they indeed suffer from underpowering; their conclusions are unreliable, but reviewers failed to account for this cost. Second, they swapped to a coefficient that is insensitive to the number of tiers, thereby circumventing the explosion.

The second route is real; the literature explicitly documents it. A classic methodological conclusion states that quadratic-weighted κ systematically increases with the number of tiers, whereas linear-weighted κ is largely insensitive to tier count (PMID 8834562). In other words, switching from quadratic-weighting to linear-weighting for the same dataset eliminates most of the tier penalty. Monte Carlo simulations further sharpen this point: when both the number of tiers and sample size increase, the estimation precision of weighted coefficients actually declines—”enrolling more subjects” does not automatically improve precision (PMID 33853549). Even more striking is a clinical example from Traditional Chinese Medicine: five practitioners independently graded 42 patients using free margins; Fleiss κ and Gwet AC2 yielded conclusions differing by exactly one grade (PMID 29851495). Coefficient selection is not merely a technical detail; it directly determines whether the last sentence of your paper reads “good consistency” or “poor consistency.”

Interestingly, this escape route itself represents an open conflict within the research core. On one side, the optimal metrics for gray-zone (low-to-moderate consistency) scenarios are Gwet AC2 and Brennan-Prediger κ, yet all weighted coefficients suffer significant precision loss under the combination of “low-to-moderate consistency + large sample size + many categories”—meaning the tier penalty cannot be avoided, and AC2 is only relatively best. On the other side, simulation results point in the opposite direction: Fleiss κ and Conger κ perform better for ordinal classification, whereas increasing scale dimensionality helps only nominal classification and harms ordinal classification; thus, for multi-grade ordinal scenarios like five-phase differentiation, Gwet AC2 is more robust than κ. Both camps agree that “swapping coefficients changes the conclusion,” but they disagree on which direction is correct—precisely the background that underpowering audits must consume: if even the choice of coefficient remains undecided, the value of coefficients reported in low-n studies must be discounted further.

There is also a frequently overlooked layer: under stratified designs, the sample size units for AC1 do not align with homogeneity tests—some studies calculate based on the number of raters, others based on the number of cases, making direct comparison impossible (PMID 40298274). Therefore, during auditing, the question “how many samples are needed” must first unify its units; otherwise, the ratio of actual n to required n will yield spurious results.

How do we resolve this? The approach is bibliometric: retrieve at least 30 clinical studies reporting ordered consistency coefficients for four or more tiers, extract sample size, number of tiers, coefficient type, and reported interval width for each, then reverse-calculate the “minimum required n” based on the sample scale and compute the distribution of actual n / required n ratios. The criteria fall into three categories: if more than 70% of studies have actual n less than half the required value, underpowering is confirmed; any future multi-tier design must undergo power calculation first. If the proportion is less than 30%, it indicates a dimensionality-reduction mechanism not captured by current models (e.g., nominal five tiers effectively using only three), requiring a “effective tier count” correction term in the sample factor. If the distribution is bimodal—some severely underpowered, others adequately powered—the source of difference must be attributed to coefficient type or material difficulty gradient, yielding a checklist of “under what conditions can sample size be reduced.”

The significance of this audit for our project is direct: core research lines such as pulse diagnosis, syndrome differentiation, and acupoint determination all rely on consistency data where “several raters grade the same set of cases.” If this escape route is confirmed, then every “moderate consistency” paper in the citation chain must first be asked: do these dozens of cases truly hold up, or is this respectability manufactured by swapping coefficients?

Evidence and Limitations

Verified: Quadratic-weighted κ increases linearly with tier count while linear-weighted κ remains insensitive (PMID 8834562); coefficient precision in gray-zone scenarios varies with the interaction of tier count and sample size (PMID 33853549); in Traditional Chinese Medicine practitioner grading, Fleiss κ and AC2 differ by one grade (PMID 29851495)—the “swapping coefficients changes conclusions” route is well-supported in the literature.

Partially Verified: The mismatch of sample size units in stratified designs has been studied but covers limited scope (PMID 40298274); the actual frequency of “escaping tier penalties via coefficient swapping” in published literature lacks direct statistical evidence.

Pending Verification: The proportion of underpowered studies among ≥30 multi-tier consistency studies; the shape of the actual n / required n ratio distribution; whether coefficient type explains bimodality. All are proposals, not yet executed.

Further Reading

Evidence Level: C
References: PMID 8834562, PMID 33853549, PMID 29851495, PMID 40298274

L1 Network Status: E-060 label credibility (KB-317, conflict, undecided)—”tier specification collides with methodological counterexample”: AC1 stratified design sample size n≥50 mismatches KB-301’s “120 cases” unit; “all weighted coefficients lose precision significantly under low-to-moderate consistency + large sample size + many categories” (PMID 33853549); KB-333 (conflict, medium) introduces a third mismatch—”Fleiss κ and Conger κ paradoxically perform better for ordinal classification, while increasing scale dimensionality aids only nominal classification and harms ordinal classification, making Gwet AC2 the more robust choice for ordinal scenarios.” This topic (underpowering audit of multi-tier consistency studies at the literature level) has not yet entered the corresponding L1 network; E-060 is the nearest adjacent edge.


Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *