这份 480 例是事前定的还是补出来的:已发表文献里的事后补样,数字本身留不留痕

Written by

in

Language

Evidence level: C

事后补样污染的不是检验的通过率,而是「这是一份 n 例研究」这句话本身——而补样行为可能在样本量的数字痕迹上留下指纹,让已发表文献可以事后回溯筛查。

正文

从一个悬空的前提出发。 前一篇算过一笔账:如果研究者在精确门禁下「先收一批、不够再补」,alpha 的名义通过率其实没有被破坏——精确检验的门禁仍把假阳性压在 5% 附近。真正被污染的是别的东西:等效样本量塌到名义值的四成之一,精度虚高 2.2 倍。换句话说,补样悄悄改写的不是结论对不对,而是「这份数据值多少个独立样本」这个声明。但整个论证有一个悬空前提:真实文献里到底有多少研究补过样,谁也不知道。一项对 82 项一致性研究的描述性研究(PMID 36123642)发现只有 33% 给了样本量依据,而这 33% 里哪些是事前算的、哪些是「先收一批再补」,摘要层面完全看不出。本篇要问的是:这件事能不能不靠作者坦白,而从已发表的数字本身筛出来。

三条痕迹,并行检验。 第一条是整十聚簇。事前按精度算出来的 n 往往是「任意」的数——87、113、152;而事后补出来的总数倾向于落在整齐的阶梯点上:60、120、240、480。如果「最末一次追加后的总数」系统性地偏爱整十整百,末位数字分布的卡方检验会直接显形。当然要小心一个混淆:研究者本来就偏爱整数,聚簇可能只是偏好而非补样——这正是判据要分开处理的情形。第二条是注册 n 与实际 n 的差值分布。有注册号的研究可以直接比:报差值中位数,以及「实际 > 注册」的单向比例。经验上这个方向确实值得怀疑——对比丹麦 1994-95 年批准的试验方案与发表稿,34 项有样本量计算的试验里 18 项存在未声明的样本量不一致(PMID 19056791);一项对 66 项亲密伴侣暴力研究的核查发现 39% 的研究与注册记录在样本量上有大改、44% 是研究完成后才注册的(PMID 29281583)。第三条是精度反查:按论文自报的目标精度,用按精度反解样本量的标准算法(PMID 24338600)算出「应有 n」,与实际 n 比——实际显著偏小者为「可能事后停」,显著偏大者为「可能事后补」。kappa 样本量表(PMID 15733050)这类经典依据被引用的频率,与引用者的实际 n 是否对得上,也从未被核对过。

判据怎么下。 设想三条痕迹里至少两条给出同向结论——整十聚簇超出随机水平、实际>注册占多数、实际 n 系统性偏离反解值——那么「补样」就是可检测的,可以给该领域一个事后追加率的点估计,并写进方法学规范。若只有整十聚簇显著而注册比对不显著,聚簇反映的只是「研究者偏好整数」,此路作废。若三条全部不显著,则已发表文献无法回溯识别补样行为,任何关于「补样普遍性」的断言都不得作出,只能靠前瞻注册数据。而若「实际>注册」的占比超过 40%,那补样本身就是常态——相关的三条规范就必须从「建议」升级为该领域的「强制报告项」。

与前面两篇的分工。 前一篇问的是同一份数据内部两笔误差能不能相加(记账问题);再前一篇算的是补样对检验门禁与等效样本量的代价(后果问题);本篇问的是跨研究之间的账目核对——这份数据的 n 是不是它自称的那个 n。三者互相独立,也互相补位:没有本篇,「补样代价」只对被作者承认的补样生效;有了本篇,代价才有可能落到整片文献上。

可测手段:从 PubMed 抽取一批报告「例数 × 评定者数」的一致性研究,按有无样本量依据、有无注册号分层,三条痕迹检验并行。这是纯文献计量工作,不需要任何新数据采集。

证据与局限

已验证: 样本量的方案-发表不一致真实存在且比例可观(PMID 19056791:18/34 项未声明;PMID 29281583:39% 样本量大改、44% 完成后才注册);样本量依据的报告率低(PMID 36123642:仅 33%)。
部分验证: 按精度反解 n 的算法本身成熟(PMID 24338600),用作「应有 n」基准技术上可行,但反解值对参数假设的敏感度未评估。
待验证: 三条痕迹(整十聚簇 / 注册差值 / 精度反查)作为补样筛查工具的联合判别力完全未实测——本文全部判据均为事前设定,尚无一条被数据检验过。

相关阅读

L1 网络状态:该主题尚未进入 L1 对应网络(CORRESPONDENCE_NETWORK.md 1983 行与 EDGE_TABLE_science-yixue.md 63 行中均无「样本量 / 补样 / 注册」对应边)

证据等级: C
参考文献: PMID 36123642, PMID 19056791, PMID 29281583, PMID 24338600, PMID 15733050

证据等级:A=有PubMed直接支撑,B=科学类比/间接证据,C=传统文化共识/初步探索


免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。

Are these 480 cases pre-specified or post-hoc: does the number itself leave a trace in published literature?

Evidence level: C

Post-hoc sampling contaminates not the pass rate of statistical tests, but the very claim that “this is an n-case study”—and the act of post-hoc sampling may leave fingerprints on sample size numbers, enabling retrospective screening of published literature.

Main Text

Starting from an unresolved premise. The previous article calculated the cost: if researchers operate under strict precision gates using a “collect first, supplement if insufficient” approach, the nominal pass rate for alpha remains intact—strict testing still keeps false positives near 5%. What is truly contaminated is something else: the equivalent sample size collapses to one-quarter of its nominal value, while precision inflates by a factor of 2.2. In other words, post-hoc sampling quietly rewrites not whether the conclusion is right, but the declaration of “how many independent samples this data represents.” However, the entire argument rests on an unresolved premise: we do not know how many studies in the actual literature have been post-hoc sampled. A descriptive study of 82 consistency studies (PMID 36123642) found that only 33% provided sample size justification, and among those 33%, it is impossible to discern from the abstract alone which were pre-calculated versus “collect first, supplement later.” This article asks: can we detect this without relying on author disclosure, by screening the published numbers themselves?

Three traces, tested in parallel. The first is clustering at round tens. Pre-calculated n values based on precision are often “arbitrary” numbers—87, 113, 152; whereas totals from post-hoc sampling tend to fall on neat step points: 60, 120, 240, 480. If “the final total after the last addition” systematically favors round tens and hundreds, a chi-square test on the distribution of trailing digits will reveal this directly. Of course, one must guard against a confounding factor: researchers inherently prefer integers, so clustering might reflect preference rather than post-hoc sampling—this is precisely why the criteria must be separated. The second is the distribution of differences between registered n and actual n. Studies with registration numbers can be compared directly: report the median difference, and the proportion where “actual > registered.” Empirically, this direction is indeed suspicious—comparing Danish trial protocols approved in 1994-95 with published reports, 18 of 34 trials with sample size calculations had undisclosed inconsistencies between registered and published sample sizes (PMID 19056791); a review of 66 intimate partner violence studies found 39% had major changes in sample size compared to registration records, and 44% were registered only after completion (PMID 29281583). The third is reverse-precision checking: using the standard algorithm (PMID 24338600) to solve for “expected n” based on the paper’s stated target precision, then comparing with actual n—significantly smaller actual values suggest “possible stopping,” while significantly larger ones suggest “possible supplementation.” The frequency with which classic tools like the kappa sample scale (PMID 15733050) are cited has never been cross-checked against whether the citing researchers’ actual n values align.

How to set the criteria. Suppose at least two of these three traces yield concordant conclusions—round-tens clustering exceeds random levels, actual > registered is the majority, and actual n systematically deviates from reverse-solved values—then “post-hoc sampling” is detectable, allowing us to produce a point estimate of the post-hoc supplementation rate for this domain and incorporate it into methodological standards. If only round-tens clustering is significant while registration comparison is not, the clustering merely reflects “researchers prefer integers,” and this path is abandoned. If all three traces are insignificant, then published literature cannot be retrospectively identified for post-hoc sampling behavior; any claim about “universality of post-hoc sampling” must be withheld, relying instead on prospective registration data. But if the proportion where “actual > registered” exceeds 40%, then post-hoc sampling itself is the norm—the three related standards must be upgraded from “recommendations” to mandatory reporting items for this domain.

Division of labor with previous articles. The prior article asked whether two errors within the same dataset could be summed (the accounting problem); the earlier one calculated the cost of post-hoc sampling for test gates and equivalent sample size (the consequence problem); this article asks cross-study reconciliation—whether this dataset’s n matches the claimed n. The three are mutually independent yet complementary: without this article, “cost of post-hoc sampling” applies only to cases authors admit; with this article, the cost can extend across the entire literature.

Measurable approach: Extract from PubMed a batch of consistency studies reporting “cases × raters,” stratified by presence/absence of sample size justification and registration numbers, then apply the three-trace test in parallel. This is pure bibliometric work requiring no new data collection.

Evidence and Limitations

Verified: Inconsistency between planned and published sample sizes is real and substantial (PMID 19056791: 18/34 undisclosed; PMID 29281583: 39% major sample size revision, 44% registered after completion); low reporting rate of sample size justification (PMID 36123642: only 33%).
Partially verified: The algorithm to solve for n based on precision is mature (PMID 24338600), making technical use of “expected n” feasible, though sensitivity to parameter assumptions remains unassessed.
To be verified: The combined discriminative power of the three traces (round-tens clustering / registration difference / reverse-precision check) as post-hoc sampling screening tools has never been empirically tested—all criteria in this article are pre-specified, none yet validated by data.

Related Reading

L1 Network Status: This topic has not yet entered the L1 correspondence network (no “sample size / post-hoc sampling / registration” edges present in CORRESPONDENCE_NETWORK.md line 1983 and EDGE_TABLE_science-yixue.md line 63)

Evidence Level: C
References: PMID 36123642, PMID 19056791, PMID 29281583, PMID 24338600, PMID 15733050

Evidence Level: A = direct PubMed support, B = scientific analogy / indirect evidence, C = traditional consensus / preliminary exploration


Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *