档数根本不进那个三成折扣:多档判读改的不是设计效应,是它算不出来的那一步

Written by

in

Language

Evidence level: C

判读从两档加到五档,设计效应 k 一位小数都不会变;它变的是另一个东西——你终于能把 k 里那个从没测过的参数估出来了。

一、一个看着很有道理、但算下来为零的怀疑

1426 算出一个数:漂移把合格率读数的方差抬高三成,k=1.3018,有效样本量只剩名义的 41.5%。但这个数是在最简设定下算的——判读只有「合格/不合格」两档、只有一条切点。

自然的怀疑是:真实判读是三到五档,每档边界各有各的漂移敏感度,靠近分布中心的边界密度高、被漂移搅得厉害,靠近尾端的边界几乎不受影响。那多档的 k 应当是各边界贡献的加权和,应该和两档不一样。

这个推理每一步都对,结论是零。把合格线固定在同一个位置,只把线以上的部分再切成几档、线以下的部分再切成几档,算出来的 k 完全相同:m0=0.95、σ=0.515 时,K=2、3、4、5、7 五档全都等于 1.301325,一个数字都不差。m0=0.80、0.90,σ=0.167、0.30,九种组合逐一验过,全部精确相等。

原因不难看:k 的分子是 E[m(1−m)],m 是「切点以上」的概率,只由切点位置和漂移决定。你在切点上下再切几刀,切出来的档不进入这个期望。多档改变的是「判读者看到几个选项」,不是「合格这条线在哪」。线没动,k 就不会动。

二、所以原设问的两条判据都落空,但落空的方式有意义

原设问准备了两条判据:k 随档数单调降到 1.10 以下,说明三成折扣是高估;k 随档数升到 1.60 以上,说明两档最乐观。

实测两条都不发生,k 是常数。这不是「判据不显著」,是「这个自变量根本不在函数里」。把一条恒为零的斜率写成「未达显著」,会让人以为多加些样本还能测出来——实际上加多少样本都是零。

三、真正在变的是另一个东西:σ 能不能被估出来

k 里有一个参数从头到尾没被测过,就是漂移的标准差 σ。#1426 用了 0.515,而 #1404 用的是 0.167,两者差三倍,k 就从 1.30 掉到 1.04。所以「k 是多少」这个问题,实际是在问「σ 是多少」。

关键在这里:二档数据定不出 σ。

固定观测到的合格率为 0.90,反解各种 σ 下需要的设计合格率与对应的 k:σ=0.05 时 k=1.0016,0.10 时 1.0066,0.167 时 1.0185,0.30 时 1.0609,0.515 时 1.1919,0.80 时 1.5440。八组参数给出的观测合格率全都是 0.9000。也就是说,一份「合格率九成」的二档数据,背后可能是 k=1.00,也可能是 k=1.54,有效样本量差一半以上,而数据本身分不开。

原因是方程不够:二档只给你一个读数 E[m],却要同时定 m0 和 σ 两个未知量。对称切点时更极端——两档比例恒为 0.5,σ 从 0.10 变到 0.80,比例一个数都不动(0.5 还是 0.5),而同一处的 k 已经从 0.9937 掉到 0.7448。

加档就补上了这个方程。同样是 σ 从 0.10 变到 0.80,三档的中间档占比从 0.33178 单调走到 0.26339,四档内侧两档从 0.2489 走到 0.2008,五档从 0.1990 走到 0.1568。按 n=200 算,比例向量对 σ 的敏感度除以抽样噪声:二档 0.0,三档 6.9 倍,四档 10.7 倍,五档 13.0 倍,七档 15.9 倍。

这才是有用的结论:加档不是为了改 k,是为了让 k 里的 σ 从「拍一个」变成「估一个」。二档研究报出的 k,其实是研究者对 σ 的假设,不是数据的结论。

四、加档不是免费的:一致率天花板跟着往下掉

档数既然不碰 k,是不是白加?不是,但它买到的不是精度,是可辨识性,代价另记在别处。

同一批人、同一个 σ=0.515,把「两人判在同一档」当作一致,天花板随档数一路塌:两档 0.5672,三档 0.3959,五档 0.2479,七档 0.1810。等距切点下这是必然——档越多,区间越窄,漂移一动就跨档。

这解释了文献里一个反复出现的现象。PMID 29029991 拿三套真实序数分级数据(五档、四档)做模拟,发现常用一致性系数在极端患病率下系统性高估,且读数随数据形态大幅摆动,结论是改用基于模型的量。本文的补充是:那种摆动不全是系数选得不好,有一部分是「多档本身把天花板压低了」——读数低不等于判得差,也可能是量表把判定切碎了。

所以加档的正确用法是分工:合格与否那条线保持不动,用它算 k 与样本量;多出来的档只用来估 σ,估完回头算那条线上的 k。把多档一致率直接和两档研究比,比的是两个不同的天花板。

五、两条能自己验的规律

拆开 k 能看得很清楚,它是两个因子的乘积。水平因子是 Em 除以 m0(1−m0),恒大于等于 1——这是詹森不等式,因为 m(1−m) 是凹的。曲率因子是 E[m(1−m)] 除以 Em,恒小于等于 1——漂移把合格率搅散后,散开的那部分不再贡献判别信息。两者反向:σ=0.515、m0=0.95 时水平因子 1.4035、曲率因子 0.9272,乘出 1.3013;m0=0.70 时 1.0371 乘 0.8733 得 0.9057,反而小于 1。

第二条更干净:k(m0) 精确等于 k(1−m0)。合格率九成和合格率一成,k 一模一样,差值是零。所以「漂移把合格率往下拽」这个方向感,在设计效应上完全不成立——往上拽是同一个 k。k=1 的分界在 m0 约 0.81 至 0.83(σ 越大越往中间挪),合格率低于八成就进入 k<1 区,漂移反而压低了方差。原设问担心的「多档会更糟」,在对称性面前连方向都没有。

证据与局限

已验证:档数不改变 k,这是恒等式的直接后果,五种档数乘九种参数组合全部精确相等;k = 水平因子 × 曲率因子的分解为代数恒等;k 关于 m0=0.5 的对称性是解析结论,数值校验差值为零。以上均为纯数值积分,可复现。

部分验证:σ 的可辨识性差异(二档 0.0 倍、三档 6.9 倍、五档 13.0 倍)依赖等距切点与正态漂移假设;真实判读的切点不等距、漂移未必正态,倍数会变但单调方向应保留。文献侧,PMID 29029991 用三套真实序数分级数据(五档、四档)做模拟,明确指出常用一致性系数受患病率与边际分布左右,而基于模型的量不受影响——与本文「读数不足以定参数」同向,但其未给设计效应数字;PMID 28612356 提供序数分级的一致性的随机效应框架,可估各档切点,正好是本文所需的第二个方程,但其设计未报 σ 的可辨识性。

待验证:本条全部为理论推导,没有任何一份真实三档或五档判读数据的 σ 被反解过。σ 的真实量级、以及真实切点非等距时可辨识倍数掉多少,都还没有实测。

L1 网络状态:E-060「标签前提/一致性」边现为 conflict(未裁决),原文:「KB-317(conflict):闸门规格撞上方法学反例,两处对不上——① 单位错配……② 精度区间错配」,且 KB-333 记「κ 型系数稳健性对『序数 vs 名义』的敏感性方向与 Gwet 一系相反……尺度维数增加只对名义分类有助、对序数分类无助」。该边未含设计效应与漂移 σ 的任何条目,本文论点未进入 L1 网络。

相关阅读

证据等级: C
参考文献: PMID 29029991, PMID 28612356, PMID 26925540

证据等级:A=有PubMed直接支撑,B=科学类比/间接证据,C=传统文化共识/初步探索


免责说明:本文属传统知识与科学文献的整理、对照与研究设计讨论,仅供理解与参考,不构成医疗建议,不能替代专业诊断与治疗。文中涉及的效应量、频率区间与暴露结论多为研究报告层面的推断,个体情况差异很大;如有身体不适或健康疑虑,请就医并由专业医师判断。

The number of bands never enters that 30% discount: multi-band judgment changes not the design effect, but the step it cannot calculate.

Evidence level: C

Increasing the number of categories from two to five changes not the design effect $k$ (which remains unchanged to one decimal place), but another quantity: you finally can estimate the parameter inside $k$ that was never measured.

I. A suspicion that looks very reasonable but calculates to zero

1426 calculates one number: drift inflates the variance of pass rates by 30%, $k=1.3018$, and the effective sample size drops to 41.5% of the nominal amount. But this number is calculated under the simplest setting—judgment has only two categories (‘pass’/’fail’) and a single cut-off point.

The natural suspicion is: real judgment involves three to five categories, each boundary having its own drift sensitivity; boundaries near the distribution center are densely populated and heavily disturbed by drift, while those near the tails are almost unaffected. Then the $k$ for multiple categories should be a weighted sum of contributions from each boundary, differing from the two-category case.

This reasoning is correct at every step, yet the conclusion is zero. Fixing the pass line at the same position and slicing only the region above it into several categories, or only the region below it into several categories, yields exactly the same $k$: with $m_0=0.95$ and $\sigma=0.515$, $K=2,3,4,5,7$ all equal 1.301325, with no difference. Nine combinations ($m_0=0.80, 0.90$; $\sigma=0.167, 0.30$) were tested one by one; all matched exactly.

The reason is straightforward: the numerator of $k$ is $E[m(1-m)]$, where $m$ is the probability ‘above the cut-off’, determined solely by the cut-off position and drift. Adding more cuts above or below does not enter this expectation. Multiple categories change only ‘how many options the judge sees’, not ‘where the pass line lies’. Since the line does not move, $k$ does not move.

II. Therefore both criteria of the original question fall empty, but the way they fall is meaningful

The original question prepared two criteria: $k$ monotonically dropping below 1.10 as the number of categories increases indicates the 30% discount is an overestimate; $k$ rising above 1.60 indicates two categories are most optimistic.

Neither occurs in practice; $k$ is constant. This is not ‘the criterion is insignificant’, but ‘this independent variable is simply not in the function’. Writing a slope of zero as ‘not significant’ leads people to believe adding more samples could reveal it—actually, adding any number of samples yields zero.

III. What truly changes is another thing: whether $\sigma$ can be estimated

Inside $k$ there is a parameter never measured throughout: the standard deviation of drift, $\sigma$. #1426 used 0.515, while #1404 used 0.167; the difference is threefold, causing $k$ to drop from 1.30 to 1.04. Thus the question ‘what is $k$?’ actually asks ‘what is $\sigma$?’.

The key point: two-category data cannot determine $\sigma$.

Fixing the observed pass rate at 0.90, we solve for the required design pass rate and corresponding $k$ under various $\sigma$: $\sigma=0.05 \Rightarrow k=1.0016$, $0.10 \Rightarrow 1.0066$, $0.167 \Rightarrow 1.0185$, $0.30 \Rightarrow 1.0609$, $0.515 \Rightarrow 1.1919$, $0.80 \Rightarrow 1.5440$. All eight parameter sets yield an identical observed pass rate of 0.9000. That is, a ‘90% pass rate’ two-category dataset could correspond to $k=1.00$ or $k=1.54$, with effective sample sizes differing by more than half, yet the data itself cannot distinguish them.

The reason is insufficient equations: two categories provide only one reading $E[m]$, yet must simultaneously determine two unknowns, $m_0$ and $\sigma$. The situation is even more extreme under symmetric cut-offs—two-category proportions are always 0.5; as $\sigma$ varies from 0.10 to 0.80, the proportion remains unchanged (still 0.5), while $k$ at that point drops from 0.9937 to 0.7448.

Adding categories fills this equation gap. With the same $\sigma$ varying from 0.10 to 0.80, in three categories the middle category proportion monotonically moves from 0.33178 to 0.26339; in four categories the inner two move from 0.2489 to 0.2008; in five categories they move from 0.1990 to 0.1568. Using $n=200$, the sensitivity of the proportion vector to $\sigma$ divided by sampling noise is: two categories 0.0, three categories 6.9 times, four categories 10.7 times, five categories 13.0 times, seven categories 15.9 times.

This is the useful conclusion: adding categories is not to change $k$, but to make $\sigma$ inside $k$ go from ‘guess one’ to ‘estimate one’. The $k$ reported in two-category studies is actually the researcher’s assumption about $\sigma$, not a conclusion from the data.

IV. Adding categories is not free: the ceiling of agreement drops accordingly

Since the number of categories does not affect $k$, is adding them pointless? No, but what it buys is not precision, but identifiability; the cost is recorded elsewhere.

With the same group of people and fixed $\sigma=0.515$, treating ‘two judges agreeing on the same category’ as concordance, the ceiling collapses monotonically with the number of categories: two categories 0.5672, three categories 0.3959, five categories 0.2479, seven categories 0.1810. Under equally spaced cut-offs this is inevitable—the more categories, the narrower each interval, so any drift causes crossing.

This explains a recurring phenomenon in the literature. PMID 29029991 simulated three real ordinal rating scales (five-category, four-category) and found that common agreement coefficients systematically overestimate under extreme prevalence rates, with readings swinging greatly depending on data shape, concluding that model-based measures should be used. Our supplement: such swinging is not entirely due to poor coefficient choice; part of it stems from ‘multiple categories themselves lowering the ceiling’—low readings do not necessarily mean poor judgment, but rather that the scale chops up the decision.

Thus the correct use of adding categories is division of labor: keep the pass/fail line fixed to compute $k$ and sample size; use the extra categories solely to estimate $\sigma$, then go back and compute $k$ on that line. Comparing the agreement of multiple categories directly with two-category studies compares two different ceilings.

V. Two patterns that can verify themselves

Breaking down $k$ reveals clearly that it is the product of two factors. The horizontal factor is $Em$ divided by $m_0(1-m_0)$, always $\ge 1$—this is Jensen’s inequality because $m(1-m)$ is concave. The curvature factor is $E[m(1-m)]$ divided by $Em$, always $\le 1$—drift scatters the pass rate, and the scattered portion no longer contributes discriminative information. These two factors move oppositely: with $\sigma=0.515$ and $m_0=0.95$, the horizontal factor is 1.4035 and the curvature factor is 0.9272, multiplying to 1.3013; with $m_0=0.70$, 1.0371 times 0.8733 gives 0.9057, which is actually less than 1.

The second pattern is cleaner: $k(m_0)$ exactly equals $k(1-m_0)$. A 90% pass rate and a 10% pass rate yield identical $k$, with zero difference. Thus the direction ‘drift pulls the pass rate down’ is completely invalid regarding design effect—pulling up yields the same $k$. The boundary where $k=1$ lies at $m_0 \approx 0.81$ to $0.83$ (the larger $\sigma$, the more it shifts toward the center); pass rates below 80% enter the $k<1$ region, where drift actually reduces variance. The original worry that ‘more categories would be worse’ loses even its directional sense under symmetry.

Evidence and Limitations

Verified: The number of categories does not change $k$; this is the direct consequence of an identity, with five category counts times nine parameter combinations all exactly equal; $k =$ horizontal factor $\times$ curvature factor is an algebraic identity; the symmetry of $k$ about $m_0=0.5$ is an analytical result, with numerical verification showing zero difference. All are pure numerical integrations, reproducible.

Partially verified: The difference in identifiability of $\sigma$ (two categories 0.0 times, three categories 6.9 times, five categories 13.0 times) depends on equally spaced cut-offs and the normal drift assumption; real judgment cut-offs are not equally spaced and drift may not be normal, so the multiplier changes but the monotonic direction should be preserved. On the literature side, PMID 29029991 used three real ordinal rating scales (five-category, four-category) for simulation, explicitly stating that common agreement coefficients are affected by prevalence and marginal distribution, while model-based measures are not—aligning with our ‘readings insufficient to determine parameters’, though it did not provide design effect numbers; PMID 28612356 provides a random effects framework for ordinal agreement, allowing estimation of cut-off points for each category, exactly the second equation needed here, but its design did not report identifiability of $\sigma$.

To be verified: All in this section are theoretical derivations; no real three-category or five-category judgment data has ever had $\sigma$ back-solved. The true magnitude of $\sigma$ and how much the identifiability multiplier drops when real cut-offs are not equally spaced have not been empirically measured.

L1 Network Status: Edge E-060 ‘Label Assumption / Agreement’ is now conflict (unresolved); original text: ‘KB-317 (conflict): gate specifications clash with methodological counterexamples, two mismatches—① unit mismatch… ② precision interval mismatch’, and KB-333 records ‘robustness of the κ-type coefficient against ‘ordinal vs nominal’ sensitivity direction is opposite to the Gwet series… increasing scale dimension helps only nominal classification, not ordinal classification’. This edge contains no entries on design effect and drift $\sigma$; our argument did not enter the L1 network.

Further Reading

Evidence Level: C
References: PMID 29029991, PMID 28612356, PMID 26925540

Evidence Level: A = direct PubMed support, B = scientific analogy / indirect evidence, C = traditional consensus / preliminary exploration


Disclaimer: This article compiles traditional knowledge and scientific literature for comparison and research-design discussion only. It is not medical advice and cannot replace professional diagnosis or treatment. Effect sizes, frequency ranges and exposure conclusions cited here are largely inferences at the research-report level, and individual circumstances vary widely. If you have any discomfort or health concerns, please consult a physician.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *