17.1 Binary choice
Let \(y_i \in \{0, 1\}\) and \(p_i = P(y_i = 1 \mid x_i)\). The linear probability model (LPM) is the regression \(y_i = x_i'\beta + u_i\), so that \(p_i = x_i'\beta \). It is easy to estimate and interpret, but it has two defects. Fitted probabilities can fall outside \([0, 1]\); and the disturbance is heteroscedastic, since \(\Var (u_i \mid x_i) = p_i(1 - p_i)\) depends on \(x_i\), so robust standard errors are needed. More fundamentally, a constant marginal effect is implausible: an extra year of education cannot raise the probability of employment by the same amount for someone already almost certain to work as for someone with a probability near one half.
Definition 17.1.1. In the latent variable formulation, \(y_i^* = x_i'\beta + \varepsilon _i\), and we observe \(y_i = 1\) if \(y_i^* > 0\) and \(y_i = 0\) otherwise. If \(\varepsilon _i\) has a symmetric distribution function \(F\), then \[ p_i = P(\varepsilon _i > -x_i'\beta ) = F(x_i'\beta ) . \] The probit model takes \(F = \Phi \), the standard normal distribution function; the logit model takes the logistic \(F(c) = \Lambda (c) = e^c/(1 + e^c)\).
The scale of \(\varepsilon _i\) is not identified, since multiplying \(y^*\) by any positive constant leaves \(y\) unchanged; the variance is therefore normalised, to \(1\) for probit and to \(\pi ^2/3\) (standard deviation \(1.81\)) for logit. Logit coefficients are accordingly about \(1.6\) to \(1.8\) times probit coefficients, and the two models, whose distribution functions are very similar after rescaling, usually give almost identical fitted probabilities.
Theorem 17.1.2. With independent observations, the log-likelihood of the binary model is \[ \ell (\beta ) = \sum _{i=1}^n\bigl [y_i\log F(x_i'\beta ) + (1 - y_i)\log \bigl (1 - F(x_i'\beta )\bigr )\bigr ] . \] For the logit, \[ \frac {\partial \ell }{\partial \beta } = \sum _{i=1}^n(y_i - \Lambda _i)x_i, \qquad \frac {\partial ^2\ell }{\partial \beta \partial \beta '} = -\sum _{i=1}^n\Lambda _i(1 - \Lambda _i)x_ix_i' , \] where \(\Lambda _i = \Lambda (x_i'\beta )\). The Hessian is negative definite whenever \(X\) has full column rank, so \(\ell \) is strictly concave and has at most one maximum, which Newton–Raphson iteration finds from any starting point.
Proof. \(P(y_i = 1) = F_i\) and \(P(y_i = 0) = 1 - F_i\), and the likelihood of an observation is \(F_i^{y_i}(1 - F_i)^{1 - y_i}\). For the logit, \(\Lambda '(c) = \Lambda (c)(1 - \Lambda (c))\), so \(\partial \log \Lambda _i/\partial \beta = (1 - \Lambda _i)x_i\) and \(\partial \log (1 - \Lambda _i)/\partial \beta = -\Lambda _ix_i\), which give the score \(\sum [y_i(1 - \Lambda _i) - (1 - y_i)\Lambda _i]x_i = \sum (y_i - \Lambda _i)x_i\). Differentiating again gives the Hessian. For any \(c \neq 0\), \(c'\bigl (\sum \Lambda _i(1 - \Lambda _i)x_ix_i'\bigr )c = \sum \Lambda _i(1 - \Lambda _i)(x_i'c)^2 > 0\), because each weight is positive and \(Xc \neq 0\). A strictly concave function has at most one stationary point, which is its maximum. □
The logit score equations say that \(\sum (y_i - \hat \Lambda _i)x_i = 0\): with an intercept, the average fitted probability equals the sample proportion, just as least squares residuals average zero. The probit log-likelihood is also globally concave, though the proof is longer. A maximum may fail to exist if some combination of regressors predicts \(y\) perfectly (separation): the likelihood then increases without bound as a coefficient goes to infinity.
Interpreting the coefficients
The coefficients are not marginal effects. For a continuous regressor, \[ \frac {\partial p_i}{\partial x_{ij}} = f(x_i'\beta )\beta _j , \] where \(f = F'\), so the effect depends on where the individual is on the curve, and is largest near \(p = \frac 12\). Two summaries are common: the effect at the mean of the regressors, \(f(\bar x'\beta )\beta _j\), and the average marginal effect \(\frac 1n\sum _if(x_i'\hat \beta )\hat \beta _j\), which is usually preferred. For a dummy regressor the effect is the difference in probabilities with the dummy at one and at zero. The logit has a further interpretation: \(\log \bigl (p/(1 - p)\bigr ) = x'\beta \), so \(e^{\beta _j}\) is the odds ratio, the factor by which a unit increase in \(x_j\) multiplies the odds of the outcome. Measures of fit include McFadden’s pseudo-\(R^2\), \(1 - \ell (\hat \beta )/\ell _0\), where \(\ell _0\) is the log-likelihood with only an intercept, and the likelihood ratio test of all the slopes, \(2(\ell (\hat \beta ) - \ell _0) \sim \chi ^2_{k-1}\).
With more than two alternatives, the multinomial logit models the choice among unordered categories (such as mode of transport), and the ordered probit models ordered categories (such as a credit rating) by a single latent variable and several thresholds.
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.