判读者不是等负载的:轮换与缺席下,6 人封顶会移到几个人

Written by

in

Language

Evidence level: C

上一轮算出的「判读者人数边际递减」建立在全员评全批的理想设计上;真实世界里判读者有轮换、缺席和分工——递减点会右移多少,以及「换掉最抖的那个人」是不是比「多请一个人」划算,本轮用仿真给出了两个答案:递减点右移 2 人起步,而「淘汰末位」在等预算下永远输给「全保留」。

一、为什么完全交叉是个奢侈品

前一篇(判读者边际递减)的全部数字都假设完全交叉设计:每位判读者评完所有受试者。这在气功外气判读这种小规模场景或许做得到,但只要判读者人数上去,轮换(每人只评一部分例)、缺席(中途退出)、专业分工(某类记录只有某人能判)就会把设计切成不完整交叉——每位判读者只评比例 q 的例,且每例被几个人评过也不相同。

这不只是「数据少了一点」的问题。配对相关这个估计量的信息来源,是「同一位受试者被不同判读者评过」所产生的可比性;不完整交叉会把一部分配对直接拆散,丢的是配对结构,不只是例数。测量学文献对此有正面回应:概化理论框架下的 ICC 选择指南已把不完整观察设计纳入正式处理(PMID 36048052),不平衡、不交叉、完全嵌套设计下的 D 研究也能预测最少判读者数(PMID 20444072),计划缺失设计更被证明能缓解小判读者数下的估计困难(PMID 33818118)。但「递减点右移几人」这种可直接用于守则的数字,文献没有给,只能自己仿真。

二、仿真一:递减点右移起步就是 2 人

设定沿用前篇:每例真值 s_i 加判读者噪声 w_ij,信度目标 ρ=0.4,n=60 例,估计量取全部配对相关平均,3000 次重复。完全交叉作为基线,再让每位判读者随机评 90%、70%、50% 的例(q 档),扫判读者人数 2 到 12 人。

半宽曲线的边际递减点(相对上一个人数的改善首次跌破 5% 处)是这么移动的:完全交叉在 7 人,q=0.9 右移到 8 人,q=0.7 右移到 9 人,q=0.5 右移到 10 人。也就是说,只要轮换掉三成的评阅量,递减点就右移 2 人——踩到了本轮判据「右移大于等于 2 人须单独标定」的线。

换算成「同精度要请几个人」更直观:完全交叉 3 个人的精度,q=0.7 要 6 个人才追平;完全交叉 6 个人的精度,q=0.7 要 12 人。等预算比较更扎心:6 人全评(360 例·人次,半宽 0.059)对比 9 人各评七成(378 例·人次,半宽 0.064)——多花 5% 预算,精度反而差 9%。丢掉的那九个百分点,就是被拆散的配对结构。

反过来,q=0.9 时递减点只右移 1 人,与完全交叉的结论基本可外推。守则由此分层:轮换率在一成以内,按完全交叉标定;轮换率达到三成,必须按轮换设计单独重标递减点。

三、仿真二:淘汰末位是个亏本买卖

第二个问题更反直觉。真实判读者噪声不齐:有的天生稳,有的天生抖。设池子里一半人噪声是标准值的 0.5 倍、一半是 1.5 倍。等预算下(都请 k+1 人、都评全批)比较两条路:全保留(k+1 人都用)对比淘汰末位(按数据估计每人噪声、砍掉最抖的、留 k 人)。

结果是淘汰末位全面落败:留 4 人时半宽差 16.3%,留 6 人差 13.7%,留 8 人差 10.7%。更要命的是,这不是认错人的问题——按两向残差估计噪声,砍中的确实几乎都是抖的那档(准确率 97% 到 100%)。问题在结构:砍掉一个人就丢掉他参与的全部配对,即使那人噪声是别人的五倍,等预算下淘汰仍差 25%。已经付过钱的评阅,留在加权平均里的价值,几乎总是高于把它扔掉。

那正确动作是什么?按估计噪声反比加权——把抖的人降权而不是清零——在 8 人档反而比全保留多赚 5%;小 k 档因权重本身估计不准而暂时不赚。另外别忘了池子整体质量这个大头:同样预算,全好手池的半宽 0.057 对混合池的 0.107,差了将近一倍——这一项任何事后统计技巧都补不回来。

四、守则改写

本轮判据二的前提被推翻:「淘汰末位多降 15% 以上」不成立,方向整个反了——等预算下淘汰末位只会更差。质量管理的正确位置在请人之前(筛选、试评、校准轮),不在评完之后(淘汰);评完之后可用的杠杆是降权,不是清零。结合前篇与配比篇,判读者侧的三条守则现在齐了:人数按递减点封顶、预算按等精度线配比、异方差靠加权不靠裁员。

证据与局限

已验证:五个 PMID 的方法学框架真实可查;仿真在 ρ=0.4、n=60、3000 次重复下数字稳定。

部分验证:结论绑定在「配对相关平均」这一估计量与两档噪声(0.5 倍与 1.5 倍)设定上;更极端的噪声分布、其他估计量(如概化系数、多层面 Rasch 严重度校准)下淘汰末位是否仍然亏本,未测。

待验证:一是锚题与螺旋式链接设计能否用更低预算补回被拆散的配对结构——大型测评实践里这类设计正是为省评阅量而生,值得下一轮仿真;二是反比加权在小 k 下的权重估计样本量要求:几个人、几例才够把权重估稳,决定加权是工程可执行还是纸上最优。

相关阅读

证据等级: C
参考文献: PMID 36048052, PMID 20444072, PMID 33818118, PMID 16143966

证据等级:A=有PubMed直接支撑,B=科学类比/间接证据,C=传统文化共识/初步探索

L1 网络状态:该主题(不完整交叉判读设计 × 边际递减点右移 × 噪声倒序淘汰)尚未进入 L1 对应网络(边表 0 命中);最近邻为 E-060 标签信度边,当前状态为「冲突(未裁决)」——原文摘录:「KB-292(conflict):中医医师间诊断一致性在开放人群中『低(low)』,作者称为『令人担忧』(PMID 30383398);KB-317(conflict):样本量单位错配 + 精度区间错配」。本文按规则封顶 C 级,缺口已登记至 science-yixue-l1-gaps.md。


免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。

Judging readers are not equal loads: under rotation and absence, how many people will the 6-person cap shift to

Evidence level: C

判读者不是等负载的:轮换与缺席下,6 人封顶会移到几个人

The previous round’s calculated “diminishing marginal returns of readers” was built upon an ideal design where all readers evaluate the entire batch; in reality, readers undergo rotation, absenteeism, and division of labor—by how much does the diminishing point shift to the right, and is “replacing the most unstable reader” more cost-effective than “adding another person”? This round uses simulation to provide two answers: the diminishing point shifts right by at least 2 people, and under equal budget constraints, “eliminating the last-place reader” always loses to “keeping everyone.”

I. Why Complete Crossing is a Luxury

The numerical results of the preceding section (on diminishing marginal utility of judges) all assume a completely crossed design: each judge evaluates every subject. In small-scale settings such as Qi Gong external energy reading, this might be feasible. However, as the number of judges increases, rotation (each judge evaluates only a subset), attrition (subjects dropping out mid-study), and specialization (only certain judges can interpret specific records) will fragment the design into an incomplete crossing—where each judge evaluates only a proportion $q$ of cases, and the number of judges per case varies.

This is not merely an issue of “having fewer data points.” The information content of the paired correlation estimator relies on comparability generated by “the same subject being evaluated by different judges.” Incomplete crossing directly breaks many such pairs, losing the pairing structure itself, not just sample size. Measurement literature has addressed this positively: ICC selection guidelines under the generalized theory framework formally incorporate incomplete observation designs (PMID 36048052); D-studies under unbalanced, non-crossed, and fully nested designs can predict the minimum number of judges required (PMID 20444052); and missing-data designs have been shown to mitigate estimation difficulties under small numbers of judges (PMID 33818118). However, directly actionable figures such as “how many additional judges shift the diminishing point to the right” are absent from the literature and must be derived via simulation.

II. Simulation 1: Rightward Shift of the Diminishing Point Begins at 2 Persons

Adopting parameters from the preceding study: each true value $s_i$ is perturbed by assessor noise $w_{ij}$, with target reliability $\rho=0.4$, $n=60$ cases, and the estimator defined as the average of all pairwise correlations, repeated 3,000 times. The baseline employs complete crossing, against which we vary the proportion of cases each assessor reviews (q-levels: 90%, 70%, 50%) while sweeping the number of assessors from 2 to 12.

The marginal diminishing point (defined as the point where improvement relative to the previous number of assessors first drops below 5%) shifts as follows: complete crossing occurs at 7 assessors; for $q=0.9$ it shifts rightward to 8 assessors; for $q=0.7$ to 9 assessors; and for $q=0.5$ to 10 assessors. In other words, reducing the review volume by one-third shifts the diminishing point rightward by 2 assessors—exactly meeting the criterion for this round: “a rightward shift of $\ge$ 2 assessors requires separate calibration.”

Expressed more intuitively as “how many people are needed for the same precision”: achieving the precision of 3 assessors under complete crossing requires 6 assessors when $q=0.7$; achieving the precision of 6 assessors under complete crossing requires 12 assessors when $q=0.7$. Comparing budgets reveals a starker trade-off: 6 assessors reviewing all cases (360 case·assessor units, half-width 0.059) versus 9 assessors each reviewing 70% (378 case·assessor units, half-width 0.064)—spending 5% more budget yields 9% worse precision. That lost 9% stems from the disrupted pairing structure.

Conversely, when $q=0.9$, the diminishing point shifts rightward by only 1 assessor, allowing conclusions under complete crossing to be reasonably extrapolated. Thus, the protocol is stratified: if the rotation rate remains within 10%, calibration follows the complete-crossing model; if the rotation rate reaches 30%, the diminishing point must be recalibrated separately using the rotation design.

III. Simulation II: Eliminating the Lowest Performer is a Loss-Making Strategy

The second question is more counterintuitive. Real readers exhibit heterogeneous noise: some are inherently stable, others inherently jittery. Consider a pool where half the readers have noise equal to 0.5 times the standard value and half have noise equal to 1.5 times the standard value. Under a fixed budget (all invite k+1 readers, all evaluate the full batch), compare two approaches: retaining everyone versus eliminating the lowest performer (estimating individual noise from data, removing the most jittery, keeping k readers).

The result: eliminating the lowest performer consistently underperforms. Retaining 4 readers yields a half-width difference of 16.3%; retaining 6 readers yields 13.7%; retaining 8 readers yields 10.7%. More critically, this is not a problem of misidentifying the noisy readers—using two-way residuals to estimate noise correctly identifies the jittery group with 97% to 100% accuracy. The issue lies in the structure: removing one reader discards all pairings they participated in. Even if that reader’s noise is five times higher than others, under a fixed budget, elimination still incurs a 25% loss. The value of already-paid evaluations retained in the weighted average almost always exceeds discarding them.

What is the correct action? Weight inversely by estimated noise—downweighting jittery readers rather than zeroing them out. In the 8-reader group, this approach actually gains 5% compared to retaining everyone; smaller k groups do not yet gain because the weights themselves are poorly estimated. Also remember the overall pool quality: under the same budget, a high-quality pool achieves a half-width of 0.057 versus 0.107 for a mixed pool—a difference of nearly double—which no post-hoc statistical technique can recover.

IV. Revised Guidelines

The premise underlying Criterion II of this round has been overturned: the assertion that “eliminating the bottom tier reduces performance by more than 15%” does not hold; in fact, the direction is entirely reversed—under equal budget constraints, eliminating the bottom tier results in even worse outcomes. The correct placement of quality management lies prior to evaluation (in the stages of screening, trial assessment, and calibration rounds), not after evaluation (in the elimination phase); once evaluation is complete, the available lever for adjustment is down-weighting, not zeroing out. Integrating this with the preceding article and the section on proportional allocation, the three guidelines for the reader side are now complete: cap the number of participants according to the diminishing returns point, allocate the budget along lines of equal precision, and address heteroscedasticity through weighting rather than culling.

Evidence and Limitations

Verified: Five methodological frameworks with associated PMID numbers are traceable; simulations yield stable numerical results under ρ=0.4, n=60, and 3,000 repetitions.

Partially Verified: Conclusions are contingent upon the estimator of “paired correlation averages” and the two-tier noise settings (0.5× and 1.5×). It remains untested whether the elimination of lower-ranked items still incurs a deficit under more extreme noise distributions or alternative estimators, such as generalized coefficients or multi-level Rasch severity calibration.

Pending Verification: First, whether anchor items and spiral-link designs can recover the fragmented pairing structure at reduced cost—such designs are employed in large-scale assessments to minimize grading workload and merit further simulation. Second, the sample size requirements for weight estimation under inverse weighting with small k: how many individuals and cases are sufficient to stabilize the weights, thereby determining whether weighting is practically feasible or merely theoretically optimal.

Related Reading

Evidence level: C
References: PMID 36048052, PMID 20444072, PMID 33818118, PMID 16143966

Evidence level: A = direct PubMed support; B = scientific analogy or indirect evidence; C = traditional consensus or preliminary exploration.

L1 network status: This topic (incomplete cross-judgment design, rightward shift of the diminishing-returns point, and noise elimination in reverse order) has not yet entered the corresponding L1 network (zero hits on the edge table). Its nearest neighbor is the credibility edge labeled E-060; the current status is “conflict (unresolved)”—excerpted from the original text: “KB-292 (conflict): diagnostic consistency among TCM practitioners is ‘low’ in open populations, described by the authors as ‘concerning’ (PMID 30383398); KB-317 (conflict): mismatch of sample-size units and mismatch of precision intervals.” This entry is capped at level C per the rules, and the gap has been logged to science-yixue-l1-gaps.md.


Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *