5 Inner Product Spaces

A norm gives length but not angle. An inner product gives both, and with it the geometry of \(\mathbb {R}^n\) — perpendicularity, projection, Pythagoras — transfers to spaces of sequences and functions.

The section proves the Cauchy–Schwarz inequality, which is what makes the angle between two vectors well defined, and then develops orthogonality: orthonormal sets, Bessel’s inequality, and Parseval’s relation. A Hilbert space is a complete inner product space, and completeness is what allows an infinite orthonormal expansion to converge to something.

The projection theorem is the central result. It says every vector splits uniquely into a part lying in a closed subspace and a part perpendicular to it — and the first part is the closest point of that subspace, which is why the same theorem underlies both Fourier series and least squares approximation.

Definition 5.1. An inner product space is a vector space \(V\) over a field \(\mathbb {F}\) with a function \((\cdot ,\cdot ):V\times V\rightarrow \mathbb {F}\) that satisfies the following four conditions for all \(x,y,z\in V\) and \(\alpha \in \mathbb {F}\):

1.
\((x,x)\geq 0\) and \((x,x)=0\) iff \((x=0)\).
2.
\((x+y,z)=(x,z)+(y,z)\)
3.
\((\alpha x,y)=\alpha (x,y)\)
4.
\((x,y)=\overline {(y,x)}\) where bar denotes complex conjugation.

The function \((\cdot ,\cdot )\) is called an inner product on \(V\).

Example 5.2. For \(x=(x_1,x_2,\dots ,x_n)\) and \(y=(y_1,y_2,\dots ,y_n)\) in \(\mathbb {C}^n\) define \(\displaystyle {(x,y)=\sum ^{n}_{i=1}x_i\overline {y_i}}.\) Then \((\mathbb {C}^n,(\cdot ,\cdot ))\) is an inner product space.

Solution. Write \((x,y)=\sum _{i=1}^{n}x_i\overline {y_i}\) and check the axioms.

Linearity in the first argument

For scalars \(\alpha ,\beta \), \[(\alpha x+\beta y,z)=\sum _{i=1}^{n}(\alpha x_i+\beta y_i)\overline {z_i} =\alpha \sum _{i=1}^{n}x_i\overline {z_i}+\beta \sum _{i=1}^{n}y_i\overline {z_i} =\alpha (x,z)+\beta (y,z).\]

Conjugate symmetry

\[\overline {(y,x)}=\overline {\sum _{i=1}^{n}y_i\overline {x_i}} =\sum _{i=1}^{n}\overline {y_i}x_i=(x,y).\]

Positive definiteness

\[(x,x)=\sum _{i=1}^{n}x_i\overline {x_i}=\sum _{i=1}^{n}|x_i|^2\geq 0 ,\] and this vanishes only if every \(|x_i|=0\), that is \(x=0\). Note that \((x,x)\) is automatically real, as conjugate symmetry requires.

Conjugate symmetry together with linearity in the first slot gives conjugate linearity in the second, so the form is sesquilinear rather than bilinear — the price of working over \(\mathbb {C}\) instead of \(\mathbb {R}\).

Example 5.3. Let \(x=\{x_n\}_{n=1}^{\infty }\) and \(y=\{y_n\}^{\infty }_{n=1}\in l^2\), define \(\displaystyle {(x,y)=\sum ^{\infty }_{n=1}x_n\overline {y_n}}.\) Then \(\big (l^2,(\cdot ,\cdot )\big )\) is an inner product space.

Solution.

The series converges

This needs checking before anything else. For each \(N\), the Cauchy–Schwarz inequality for finite sums gives \[\sum _{n=1}^{N}|x_n\overline {y_n}| \leq \left (\sum _{n=1}^{N}|x_n|^2\right )^{\frac 12} \left (\sum _{n=1}^{N}|y_n|^2\right )^{\frac 12}\leq ||x||\,||y|| .\] The partial sums of \(\sum |x_n\overline {y_n}|\) are therefore bounded, so the series converges absolutely and \((x,y)\) is a well-defined complex number.

The axioms

Each is inherited from the finite case by letting \(N\rightarrow \infty \), the limits being legitimate because all the series converge. Linearity in the first argument and conjugate symmetry follow termwise as in the previous example, and \[(x,x)=\sum _{n=1}^{\infty }|x_n|^2=||x||^2\geq 0 ,\] which vanishes only when every \(x_n=0\).

The restriction to \(p=2\) is essential: for \(p\neq 2\) the series need not even converge, and the norm of \(l_p\) fails the parallelogram law, so no inner product can induce it.

Questions on this section

Stuck on something here? Ask below and it stays attached to this topic.