2.3 Measuring exposure: validity and reliability

A measurement is valid if it measures what it is meant to, on average, and reliable (repeatable) if it gives the same answer when repeated. Validity is judged against a better reference (a gold standard): self-reported smoking against cotinine in saliva, recalled diet against weighed food records. Reliability can be judged without one, by repeating the measurement or having two observers make it independently. A measurement can be reliable and invalid (a scale that always reads two kilograms light); it cannot be valid for individuals unless it is reasonably reliable.

Agreement between two classifications: kappa

When two observers classify the same \(n\) subjects into two categories, the proportion of subjects on whom they agree, \(p_o\), is a poor measure of reliability by itself, because two observers who classify at random agree on many subjects by chance. If the observers’ marginal proportions of “positive” are \(r_1\) and \(c_1\), independent random classification would agree with probability \(p_e = r_1 c_1 + (1 - r_1)(1 - c_1)\).

So \(\kappa = 1\) for perfect agreement, \(\kappa = 0\) for agreement no better than chance, and \(\kappa < 0\) for less. Values above about \(0.8\) are usually read as very good and below \(0.4\) as poor, although the scale is a convention.

Continuous exposures measured with error

If a continuous exposure \(X\) is measured as \(W = X + U\), with \(U\) independent of \(X\), of mean zero and variance \(\sigma ^2_U\), then a regression of an outcome on \(W\) instead of \(X\) estimates the true slope multiplied by the reliability ratio \[ \lambda = \frac {\sigma ^2_X}{\sigma ^2_X + \sigma ^2_U} , \] the proportion of the variance of \(W\) that is true variation between people. For if \(Y = \alpha + \beta X + \varepsilon \) with \(\varepsilon \) independent of \(X\) and \(U\), the slope of \(Y\) on \(W\) is \(\Cov (Y, W)/\Var (W) = \beta \sigma ^2_X/(\sigma ^2_X + \sigma ^2_U) = \lambda \beta \). This attenuation is called regression dilution. Its epidemiological reading is that an exposure measured on a single occasion, such as one reading of blood pressure, understates the strength of its relation with disease. If repeated measurements on a sample of subjects give \(\lambda = 0.8\), the observed slope is \(80\%\) of the true one, and dividing by \(\lambda \) corrects it.