Modules › Inner product spaces
Inner product spaces
Dot product and inner products, norm, angle, orthogonality, projections, Gram–Schmidt, orthogonal complements, least squares.
Lecture 5 · Exercise Sheet 5 · Inner product spaces
Textbook sections: Elementary Linear Algebra ch. 5 §5.1–5.4 · Strang ch. 4 §4.1–4.4
Textbook sections: Elementary Linear Algebra ch. 5 §5.1–5.4 · Strang ch. 4 §4.1–4.4
Inner product spaces
Linear Algebra · Module 5
After this module you can
- Compute norms, distances, angles and projections for the dot product and weighted inner products.
- Apply the Gram–Schmidt process and find an orthogonal complement and a projection onto a subspace.
- Fit a least-squares line.
Inner products, norm, angle
- An inner product is symmetric, linear in the first argument and positive definite. $\langle u,v\rangle=u_1v_1-2u_2v_2$ is not one (it can be negative).
- $\|u\|=\sqrt{\langle u,u\rangle}$, $d(u,v)=\|u-v\|$, $\cos\theta=\frac{\langle u,v\rangle}{\|u\|\|v\|}$, Cauchy–Schwarz $|\langle u,v\rangle|\le\|u\|\|v\|$.
- Weighted product $u_1v_1+2u_2v_2+u_3v_3$: the weight multiplies the second coordinate. In $C[-1,1]$: $\langle f,g\rangle=\int_{-1}^1fg\,dx$.
- Orthogonal $\iff\langle u,v\rangle=0$; $\|u+v\|^2=\|u\|^2+\|v\|^2$ for orthogonal $u,v$.
Projections and Gram–Schmidt
$$\operatorname{proj}_vu=\frac{\langle u,v\rangle}{\langle v,v\rangle}v,\qquad u_k=v_k-\sum_{j<k}\frac{\langle v_k,u_j\rangle}{\langle u_j,u_j\rangle}u_j,\quad e_k=\frac{u_k}{\|u_k\|}$$
- Projection onto a subspace $W$ with an orthogonal basis: add the projections onto each basis vector.
- Orthogonalise first, normalise last (keeps fractions simple). Take the vectors in the order given.
- Coordinates in an orthonormal basis: $[x]_i=\langle x,e_i\rangle$.
Orthogonal complement
- $W^\perp=\{x:\langle x,w\rangle=0\ \forall w\in W\}$; $\dim W+\dim W^\perp=n$.
- If $W=\operatorname{span}\{u_1,u_2\}$ then $W^\perp=\ker\begin{bmatrix}u_1\\u_2\end{bmatrix}$ (rows!). For the column space of $A$: $\ker A^T$.
- $\mathbf u=\operatorname{proj}_W\mathbf u+\operatorname{proj}_{W^\perp}\mathbf u$, so $\operatorname{proj}_{W^\perp}u=u-\operatorname{proj}_Wu$.
- Final exams ask: basis of $U^\perp$, orthonormalise it, project $u$ onto $U^\perp$ (or onto $U$).
Least squares
- Fit $y=a+bx$ to points: $X=\begin{bmatrix}1&x_1\\\vdots&\vdots\\1&x_n\end{bmatrix}$, solve the normal equations $X^TX\,c=X^T\mathbf y$.
- Points $(-2,2),(-1,1),(0,1),(1,3)$: $X^TX=\begin{bmatrix}4&-2\\-2&6\end{bmatrix}$, $X^T\mathbf y=\begin{bmatrix}7\\-2\end{bmatrix}$, so $y=\frac{19}{10}+\frac3{10}x$.
- The residual $\mathbf y-X c$ is orthogonal to the columns of $X$.
- If $u$ depends on a parameter $\alpha$, minimise $\|u-\operatorname{proj}_Uu\|^2$ as a function of $\alpha$ (Final 2024–25, Shift 2: the minimum is at $\alpha=1$).
Typical mistakes
Projection formulaDivide by $\langle v,v\rangle$, not $\|v\|$.
ComplementUse the vectors as ROWS to find $U^\perp$.
Gram–SchmidtProjections use $u_j$ (the new orthogonal vectors), not $v_j$.
Practice next
- Chapter quizzes Ch. 5 (3 sets)
- Practice bank, filtered to this chapter, then written mode
- Lab: Gram–Schmidt, projections, orthogonal complement, least squares
- Then the mock exams (Exams page)
Worked examples from the course problems
Worked example 1 · Function spaces (Exercise Sheet 5, Ex. 3(c))
In $C[-1,1]$ with $\langle f,g\rangle=\int_{-1}^{1}fg\,dx$, $\|x^2+1\|=$
Show the answer and the solution
Answer: $\frac{2 \sqrt{210}}{15}$
$\|g\|^2=\int_{-1}^{1}(x^2+1)^2dx=\tfrac25+\tfrac43+2=\frac{56}{15}$, so $\|g\|=\frac{2 \sqrt{210}}{15}$.
Worked example 2 · Inner products (Exercise Sheet 5, Ex. 4(b))
On $\mathbb{R}^2$, $\langle u,v\rangle=u_1v_1-2u_2v_2$ is NOT an inner product. Which axiom fails?
Show the answer and the solution
Answer: It is not positive definite
Test the axioms: the others hold, but it is not positive definite (e.g. positivity needs $\langle u,u\rangle>0$ for all $u\ne0$).
Worked example 3 · Gram–Schmidt (Exercise Sheet 5, Ex. 8)
Apply Gram–Schmidt (in the given order) to $B=\{(1,1),(0,1)\}$. The last vector $e_2$ of the orthonormal basis is
Show the answer and the solution
Answer: $(- \frac{\sqrt{2}}{2}, \frac{\sqrt{2}}{2})$
Orthogonalise: $u_2=v_2-\sum_{j<2}\operatorname{proj}_{u_j}v_2=(- \frac{1}{2}, \frac{1}{2})$, then divide by its length $\|u_2\|=\frac{\sqrt{2}}{2}$: $e_2=(- \frac{\sqrt{2}}{2}, \frac{\sqrt{2}}{2})$.
Worked example 4 · Orthogonal complements (Exercise Sheet 5, Ex. 13)
Let $W$ be the span of the columns of $A=\begin{bmatrix}1 & 2 \\ 2 & 1 \\ 0 & -1\end{bmatrix}$. Which vector spans $W^\perp$?
Show the answer and the solution
Answer: $(2, -1, 3)$
$W^\perp=\ker A^T$. Solve $A^Tx=0$: $x=t(2, -1, 3)$.
Practice
Quiz set A Quiz set B Quiz set C Practice bank (this chapter) Written mode Lab: Gram–Schmidt, projections, orthogonal complement, least squares