3.8 Applications of Orthogonal Polynomials
Theorem 3.8.1 (Least-Squares Approximations with Orthogonal Polynomials). A continuous function \(f(x)\) and \([a,b]\) can be approximated by a sum \[\sum _{i = 0}^nC_i\, P_i(x),\] where \(C_0, \,C_1, \, \cdots \, C_n\) are constants to be determined \(\{P_i(x)\}^n_{i = 0}\) are orthogonal with respect to the weight \(w(x)\) over \([a,b]\), in particular \[C_i = \frac {\langle f,P_i\rangle _w}{\lVert P_i\rVert ^2}.\]
Proof. Let \[\gamma \left (C_0, \, C_1, \, \cdots \, , \, C_n\right ) = \int _a^b\left [\sum _{i = 0}^nC_iP_i(x) - f(x)\right ]^2\, w(x)\, dx = \Bigg \|\sum _{i = 0}^nC_iP_i(x) - f(x)\Bigg \|^2_w.\] Differentiating \(\gamma \) with respect to \(C_0, \, C_1, \, \cdots \, , \, C_n\) and equate the partial derivatives to zero, get \[\frac {\partial \gamma }{\partial C_i} = 2\int _a^b\left [\sum _{j=0}^nC_jP_j(x) - f(x)\right ]\,P_i(x)\, w(x)\, dx = 0; \hspace {0.3cm} i = 0, \, 1, \, \cdots \, , \, n\] Hence, \begin {equation} \sum _{j = 0}^n\left [\int _a^bP_i(x)\, P_j(x) \, w(x)\, dx\right ]\, C_j = \int _a^bf(x)\, P_i(x)w(x)\, dx, \hspace {0.3cm} i = 0, \, 1, \, \cdots \, , \, n \end {equation} We can write equation (4.12) in vector form as \[\overline {S}\,\,\overline {C} = \overline {u},\] where \(\overline {C} = \left (C_0, \, C_1, \, \cdots \, , \, C_n\right )^T\), \(\, \overline {u} = \left (u_0, \, u_1, \, \cdots \, ,\, u_n\right )^T\), with \[u_i = \int _a^bf(x)P_i(x)w(x)\, dx,\] and \(\overline {S}\) is an \((n+1)\times (n+1)\) matrix whose \((i,j)^{\text {th}}\) element \(S_{ij}\) is given by \[S_{ij} = \int _a^bP_i(x)P_j(x)w(x)\, dx, \hspace {0.3cm} i,j = 0, \, 1, \, \cdots \, , \, n.\] Since \(P_0(x), \, P_1(x), \, \cdots \, ,\, P_n(x)\) are orthogonal, \(\overline {S}\) must be a diagonal matrix with diagonal elements given by \[S_{ij} = \int _a^bP^2_i(x) w(x)\, dx = \lVert P_i\rVert ^2_w\] from \(\overline {S}\, \overline {C} = \overline {u}\), we obtain \(\overline {C} = \overline {S}^{-1}\, \overline {u}\), \[C_i = \frac {u_i}{S_{ii}} = \frac {\langle f, P_i\rangle _w}{\lVert P_i\rVert ^2_w}\] since \(\overline {S}\) is positive definite, \(\gamma \) has an absolute minimum we refer to \[P^*_n(x) = \sum _{i = 0}^nC_i P_i(x) = \sum _{i=0}^n\frac {\langle f,P_i\rangle _w}{\lVert P_i\rVert _w^2}\, P_i(x)\] as the least-squares approximation of \(f(x)\) with respect to \(P_0(x), \, P_1(x), \, \cdots \, , \, P_n(x)\). □
Theorem 3.8.2. If \(f: \, [a,b]\, \longrightarrow \, \mathbb {R}\) is continuous, then \[\int _a^b\left [f(x) - P^*_n(x)\right ]^2\, w(x)\, dx\, \longrightarrow \, 0\] as \(n\longrightarrow \infty \), where \[P^*_n(x) = \sum _{i=0}^n\frac {\langle f,P_i\rangle }{\lVert P_i\rVert ^2}\, P_i(x).\]
Proof. By Weierstrass theorem, there exists a polynomial \(b_n(x)\) of degree \(n\) that converges uniformly to \(f(x)\) on \([a,b]\), that is \[\sup _{a\leq x\leq b}\left |f(x) - b_n(x)\right | \longrightarrow 0\] as \(n \longrightarrow \infty .\) Now \[\int _a^b\left |f(x) - b_n(x)\right |^2 w(x)\, dx \leq \sup _{a\leq x\leq b}\left |f(x) - b_n(x)\right |^2\int _a^bw(x)\, dx \longrightarrow 0\] as \(n\longrightarrow \infty \), furthermore, \[\int _a^b\left |f(x) - P_n^*(x)\right |^2\, w(x)\, dx \leq \int _a^b\left |f(x)-b_n(x)\right |^2\, w(x)\, dx\] since, by definition, \(P^*_n(x)\) is the least-squares approximation of \(f(x)\). Hence, \(\lVert f - P^*_n\rVert _w \longrightarrow 0\) as \(n\longrightarrow \infty \). □
3.8.1 Applications in Statistics
Orthogonal polynomials are important in approximating distribution functions of certain random variables. In particular Hermite polynomials provide a convenient too for approximating density functions and quantiles of distributions using series.
Approximation of a Normal Integral
A convergence series representing the
\[\Psi (x) = \frac {1}{\sqrt {2\pi }}\int _0^{x}e^{-\frac {x^2}{2}}\, dt\]
has been constructed as follows:
Let \(f(t)\) have a Taylor expansion in the neighborhood of \(\frac {x}{2}\), given by
\[f(t) = \sum _{n=0}^{\infty }\frac {1}{n!}\left (t - \frac {x}{2}\right )^nf^{(n)}\left (\frac {x}{2}\right ).\]
Integrating this with respect to \(t\) from \(t = 0\) to \(t = x\), and noting that the even terms vanish, we have
\begin {align*} \int _0^xf(t)\, dt & = \int _0^x\sum ^{\infty }_{n=0} \frac {1}{n!}\left (t - \frac {x}{2}\right )^nf^{(n)}\left (\frac {x}{2}\right )\, dt\\\\ & = \sum _{n=0}^{\infty }\frac {1}{n!}f^{(n)}\left (\frac {x}{2}\right )\int _0^x\left (t - \frac {x}{2}\right )^n\, dt \end {align*}
let \(\, u = t - \frac {x}{2}\) \begin {align*} \int _0^tf(t)\, dt & = \sum _{n=0}^{\infty } \frac {1}{n!}f^{(n)}\left (\frac {x}{2}\right )\int ^{\frac {x}{2}}_{-\frac {x}{2}}u^n\, du\\\\ & = 2\sum _{n=0}^{\infty }\frac {\left (\frac {x}{2}\right )^{2n+1}}{(2n+1)!}f^{(2n)}\left (\frac {x}{2}\right ). \end {align*}
Taking \(f(t) = e^{-\frac {t^2}{2}}\), we get \[\int _0^xe^{-\frac {t^2}{2}}\, dt = 2\sum _{n=0}^{\infty } \frac {\left (\frac {x}{2}\right )^{2n + 1}}{(2n + 1)!}\, \frac {d^{2n}}{dx^{2n}}\left ( e^{-\frac {t^2}{2}}\right )\Bigg |_{t = \frac {x}{2}}\] using Rodrigues formula for Hermite polynomials, get \[\frac {d^n}{dx^n}\left (e^{-\frac {x^2}{2}}\right ) = (-1)^n\, e^{-\frac {x^2}{2}}\, H_n(x)\, , \hspace {0.3cm} n = 0, \, 1, \, 2, \, \cdots \] we thus have \begin {align*} \int _0^xe^{-\frac {t^2}{2}}\, dt & = 2\sum ^{\infty }_{n=0} \frac {\left (\frac {x}{2}\right )^{2n+1}}{(2n+1)!}\, e^{-\frac {x^2}{2}}\, H_{2n}\left (\frac {x}{2}\right )\\\\ & = 2 e^{-\frac {x^2}{2}}\sum ^{\infty }_{n=0} \frac {\left (\frac {x}{2}\right )^{2n +1}}{(2n + 1)!} H_{2n}\left (\frac {x}{2}\right ). \end {align*}
If we set \(\Phi _n(x) = \frac {x^n\, H_n(x)}{n!}\), we get \[\int _0^xe^{-\frac {t^2}{2}}\, dt = xe^{-\frac {x^2}{2}}\sum _{n=0}^{\infty }\frac {\Phi _{2n}\left (\frac {x}{2}\right )}{2n + 1}\] Hence \[\Psi (x) = \frac {1}{\sqrt {2\pi }}\,x\,e^{-\frac {x^2}{2}}\sum _{n=0}^{\infty }\frac {\Phi _{2n}\left (\frac {x}{2}\right )}{2n + 1}.\]
Approximation of Density Function and Quantiles of Distributions
Let \(\Phi (x)\) denote the density function of the standard normal distribution, that is \[\Phi (x) = \frac {1}{\sqrt {2\pi }}e^{-\frac {x^2}{2}}\, , \hspace {0.3cm} -\infty < x < \infty .\] Recall from section 4.6 that the sequence \(\{H_n(x)\}^{\infty }_{n=0}\) of Hermite polynomials of orthogonal with respect to the weight \(w(x) = e^{-\frac {x^2}{2}}\) and that \[H_n(x) = \sum _{r=0}^{\left [\frac {n}{2}\right ]}\frac {n!\, (-1)^r\, x^{n-2r}}{2^r\, (n - 2r)!\, r!}.\] Suppose now that \(g(x)\) is some density function for some distribution. We can represent \(g(x)\) as a series \[g(x) = \sum _{n=0}^{\infty } b_n\, H_n(x)\, \Phi (x),\] where as in theorem 4.6.6 \[b_n = \frac {1}{n!}\int _{-\infty }^{\infty } g(x)\, H_n(x)\, dx\] substituting the series of \(H_n(x)\), we have \begin {align*} b_n & = \frac {1}{n!}\int _{-\infty }^{\infty } g(x)\, \sum _{r = 0}^{\left [\frac {n}{2}\right ]}\frac {n!\, (-1)^r\, x^{n-2r}}{2^r\, (n - 2r)!\, r!}\\\\ & = \int _{-\infty }^{\infty } g(x)\, \sum _{r=0}^{\left [\frac {n}{2}\right ]} \frac {(-1)^r\, x^{n-2r}}{2^r\, (n-2r)!\, r!}. \end {align*}
The \(b_n\) can then be expressed in terms of Central moments, \(\mu _0, \, \mu _1, \, \cdots \, , \, \mu _n\) of the distribution function \(g(x)\), where we
define
\[\mu _n = \int _{-\infty }^{\infty } (x - \mu )^n\, g(x)\, dx, \hspace {0.3cm} n = 0, \, 1,\, 2, \, \cdots \]
where \(\mu \) is the mean of the distribution.
\(\mu _0 = 1, \, \mu _1 = 0\) and \(\mu _2 = \sigma ^2\), the variance of the distribution.
In particular, if \(\mu = 0\), then \(b_0 = 1,\, b_1 = 0, \, b_2 = \frac {1}{2}(\mu _2 - 1), \, b_3 = \frac {1}{6}\mu _3, \, b_4 = \frac {1}{24}(\mu _4-6\mu _2 + 3), \, b_5 = \frac {1}{120}(\mu _5 - 10\mu _3)\,\) e.t.c
The expression for \(g(x)\) can then be written as \[g(x) = \Phi (x)\left [1 + \frac {1}{2}(\mu _2-1)H_2(x) + \frac {1}{6}\mu _3H_3(x) + \frac {1}{24}(\mu _4-6\mu _2 + 3)H_4(x) + \cdots \right ],\] known as the Gram-Charliér series of type A. Thus the Gram-Charliér series provides an expression of the density function in terms of its central moments, the standard normal and the Hermite polynomial. Next we have
Definition 3.8.3. The \(\alpha \)-quantile, \(x_{\alpha }\), of the distribution with density function \(g(x)\) is defined as \[\int _{-\infty }^{x_{\alpha }}g(x)\, dx = 1 - \alpha .\]
We have that \[g(x) = \Phi (x) + \sum _{n=2}^{\infty }b_n\, H_n(x)\, \Phi (x)\] so that \[\int _{-\infty }^{x_{\alpha }}g(x)\, dx = \int _{-\infty }^{x_{\alpha }}\Phi (x)\, dx + \sum _{n = 2}^{\infty } b_n \, \int _{-\infty }^{x_{\alpha }}H_n(x)\, \Phi (x)\, dx.\]
But then \begin {align*} \int _{-\infty }^{x_{\alpha }}H_n(x)\, \Phi (x)\, dx & = (-1)^n\int _{-\infty }^{x_{\alpha }}\left (\frac {d}{dx}\right )^n\Phi (x)\, dx\\\\ &= (-1)^n\left (\frac {d}{dx}\right )^{n-1}\Phi (x_{\alpha }) \end {align*}
where \(\left (\frac {d}{dx}\right )^{n-1}\Phi (x_{\alpha })\) is the \((n-1)^{\text {st}}\) derivative of \(\Phi (x)\) at \(x_{\alpha }\). \begin {align*} \int _{-\infty }^{x_{\alpha }}H_n(x)\, \Phi (x)\, dx & = (-1)^n(-1)^{n-1}\, H_{n-1}(x_{\alpha })\, \Phi (x_{\alpha })\\ & = -H_n(x_{\alpha })\, \Phi (x_{\alpha }), \end {align*}
where we have used \[H_n(x) = (-1)^n\, e^{\frac {x^2}{2}}\, \frac {d^n}{dx^n}\left (e^{-\frac {x^2}{2}}\right )\] and \[\Phi (x) = \frac {1}{\sqrt {2\pi }}e^{-\frac {x^2}{2}}.\] We have then that \[\int _{-\infty }^{x_{\alpha }} g(x)\, dx = \int _{-\infty }^{x_{\alpha }}\Phi (x)\, dx - \sum _{n=2}^{\infty } b_n\, H_{n-1}(x_{\alpha })\, \Phi (x_{\alpha }).\] Suppose that \(z_{\alpha }\) is the upper \(\alpha \)-quantile of the standard normal distribution. Then \[\int _{-\infty }^{x_{\alpha }}g(x)\, dx = 1 - \alpha = \int _{-\infty }^{z_{\alpha }}\Phi (x)\, dx\] we have that \begin {equation} \int _{-\infty }^{x_{\alpha }}\Phi (x)\, dx - \sum _{n=2}^{\infty }b_n\, H_{n-1}(x_{\alpha })\, \Phi (x_{\alpha }) = \int _{-\infty }^{z_{\alpha }}\Phi (x)dx \end {equation} we can expand \[\int _{-\infty }^{z_{\alpha }}\Phi (x)\, dx\] as a Taylor series about \(x_{\alpha }\) as follows \[\int _{-\infty }^{z_{\alpha }}\Phi (x)\, dx = \int _{-\infty }^{x_{\alpha }} \Phi (x)\, dx + \sum _{j=1}^{\infty } \frac {(z_{\alpha } - x_{\alpha })^j}{j!}\, \left (\frac {d}{dx}\right )^{j-1}\, \Phi (x_{\alpha }).\] \begin {align*} f(z_{\alpha }) & = f(x_{\alpha }) + \sum _{j = 1}^{\infty } \left (\frac {z_{\alpha } - x_{\alpha }}{j!}\right )^j\, f^{(j)}(x_{\alpha })\\\\ & = \int _{-\infty }^{x_{\alpha }} \Phi (x)\, dx + \sum _{j = 1}^{\infty } \frac {(z_{\alpha } - x_{\alpha })^j}{j!}(-1)^{j-1}H_{j-1}(x_{\alpha }) \Phi (x_{\alpha }). \end {align*}
Hence \begin {equation} f(z_{\alpha }) = \int _{-\infty }^{x_{\alpha }}\Phi (x)\, dx - \sum _{j=1}^{\infty }(x_{\alpha }-z_{\alpha })^j\, H_{j-1}(x_{\alpha })\, \Phi (x_{\alpha }) \end {equation}
from (4.13) and (4.14) get \[\sum _{j=2}^{\infty }b_n\, H_{n-1}(x_{\alpha })\, \Phi (x_{\alpha }) = \sum _{j=1}^{\infty } \frac {(x_{\alpha } - z_{\alpha })^j}{j!}\, H_{j-1}(x_{\alpha })\, \Phi (x_{\alpha })\] so that \[\sum _{n=2}^{\infty }b_n \, H_{n-1}(x_{\alpha }) = \sum _{j=1}^{\infty }\frac {(x_{\alpha }-z_{\alpha })^j}{j!}\, H_{j-1}(x_{\alpha }).\]
This last equation provides a relationship between \(x_{\alpha }\), the \(\alpha \)-quantile of the distribution with density
function \(g(x)\), and \(z_{\alpha }\), the corresponding quantile for the standard normal.
Since the \(b_n\)’s are functions of the moments associated to with \(g(x)\), it is possible to express \(x_{\alpha }\) in terms of \(z_{\alpha }\)
and the moments of \(g(x)\).
Questions on this section
Stuck on something here? Ask below and it stays attached to this topic.