MATH347DS

MATH347DS L08: Linear system solution

Overview

Orthogonal projectors
Gaussian elimination
Row echelon reduction
Matrix rank from row echelon reduction
$L U$ -factorization

Orthogonal projectors and linear systems

Consider the linear system $𝑨 𝒙 = 𝒃$ with $𝑨 \in ℝ^{m \times n}$ , $𝒙 \in ℝ^{n}$ , $𝒃 \in ℝ^{m}$ . Orthogonal projectors and knowledge of the four fundamental matrix subspaces allows us to succintly express whether there exist no solutions, a single solution of an infinite number of solutions:
- Consider the factorization $𝑸 𝑹 = 𝑨$ , the orthogonal projector $𝑷 = 𝑸 𝑸^{T}$ , and the complementary orthogonal projector $𝑰 - 𝑷$
- If $|| (𝑰 - 𝑷) 𝒃 || \neq 0$ , then $𝒃$ has a component outside the column space of $𝑨$ , and $𝑨 𝒙 = 𝒃$ has no solution
- If $|| (𝑰 - 𝑷) 𝒃 || = 0$ , then $𝒃 \in C (𝑸) = C (𝑨)$ and the system has at least one solution
- If $N (𝑨) = {𝟎}$ (null space only contains the zero vector, i.e., null space of dimension 0) the system has a unique solution
- If $\dim N (𝑨) = n - r > 0$ , then a vector $𝒚 \in N (𝑨)$ in the null space is written as
  $𝒚 = c_{1} 𝒛_{1} + \dots + c_{n - r} 𝒛_{n - r}$
  and if $𝒙$ is a solution of $𝑨 𝒙 = 𝒃$ , so is $𝒙 + 𝒚$ , since
  $𝑨 (𝒙 + 𝒚) = 𝑨 𝒙 + c_{1} 𝑨 𝒛_{1} + \dots + c_{n - r} 𝑨 𝒛_{n - r} = 𝒃 + 𝟎 + \dots + 𝟎 = 𝒃$
  The linear system has an $(n - r)$ -parameter family of solutions

Gaussian elimination: an alternative approach that does not require orthogonality

Idea: make one fewer unknown appear in each equation. Use first equation to eliminate $x_{1}$ in equations 2,3
${\begin{array}{ccc} x_{1} + 2 x_{2} - x_{3} & = & 2 \\ 2 x_{1} - x_{2} + x_{3} & = & 2 \\ 3 x_{1} - x_{2} - x_{3} & = & 1 \end{array} . \Rightarrow {\begin{array}{ccc} x_{1} + 2 x_{2} - x_{3} & = & 2 \\ - 5 x_{2} + 3 x_{3} & = & - 2 \\ - 7 x_{2} + 2 x_{3} & = & - 5 \end{array} .$
Use second equation to eliminate $x_{2}$ in equation 3
${\begin{array}{ccc} x_{1} + 2 x_{2} - x_{3} & = & 2 \\ - 5 x_{2} + 3 x_{3} & = & - 2 \\ - 7 x_{2} + 2 x_{3} & = & - 5 \end{array} . \Rightarrow {\begin{array}{ccc} x_{1} + 2 x_{2} - x_{3} & = & 2 \\ - 5 x_{2} + 3 x_{3} & = & - 2 \\ - \frac{11}{5} x_{3} & = & - \frac{11}{5} \end{array} .$
Start finding components from last to first to obtain $x_{3} = 1$ , $x_{2} = 1$ , $x_{1} = 1$

Matrix similarity transformations: reduce to triangular form

Explicitly writing the unknowns $x_{1}, x_{2}, x_{3}$ is not necessary. Intoduce the “bordered” matrix
$[\begin{array}{cccc} 1 & 2 & - 1 & 2 \\ 2 & - 1 & 1 & 2 \\ 3 & - 1 & - 1 & 1 \end{array}]$
Define allowed operations:
- multiply a row by a non-zero scalar
- add a row to another
Bordered matrices obtained by the allowed operations are said to be similar, in that the solution of the linear system stays the same
$[\begin{array}{cccc} 1 & 2 & - 1 & 2 \\ 2 & - 1 & 1 & 2 \\ 3 & - 1 & - 1 & 1 \end{array}] \sim [\begin{array}{cccc} 1 & 2 & - 1 & 2 \\ 0 & - 5 & 3 & - 2 \\ 0 & - 7 & 2 & - 5 \end{array}] \sim [\begin{array}{cccc} 1 & 2 & - 1 & 2 \\ 0 & - 5 & 3 & - 2 \\ 0 & 0 & - \frac{11}{5} & - \frac{11}{5} \end{array}]$

Matrix similarity transformations: Back substitute

To find solution, use allowed operations to make an identity matrix appear
$[\begin{array}{cccc} 1 & 2 & - 1 & 2 \\ 0 & - 5 & 3 & - 2 \\ 0 & 0 & - \frac{11}{5} & - \frac{11}{5} \end{array}] \sim [\begin{array}{cccc} 1 & 2 & - 1 & 2 \\ 0 & - 5 & 3 & - 2 \\ 0 & 0 & 1 & 1 \end{array}] \sim [\begin{array}{cccc} 1 & 0 & 0 & 1 \\ 0 & 1 & 0 & 1 \\ 0 & 0 & 1 & 1 \end{array}]$

The above constitute “Gaussian elimination”

[\begin{array}{cccc} 1 & 2 & - 1 & 2 \\ 2 & - 1 & 1 & 2 \\ 3 & - 1 & - 1 & 1 \end{array}] \sim [\begin{array}{cccc} 1 & 2 & - 1 & 2 \\ 0 & - 5 & 3 & - 2 \\ 0 & - 7 & 2 & - 5 \end{array}]

∴	A=[1. 2 -1 2; 2 -1 1 2; 3 -1 -1 1]; A[2,:]=A[2,:]-2A[1,:]; A[3,:]=A[3,:]-3A[1,:];

∴

$[\begin{array}{cccc} 1.0 & 2.0 & - 1.0 & 2.0 \\ 0.0 & - 5.0 & 3.0 & - 2.0 \\ 0.0 & - 7.0 & 2.0 & - 5.0 \end{array}]$ (1)

∴	A[3,:]=A[3,:]-(7/5)*A[2,:]; A

$[\begin{array}{cccc} 1.0 & 2.0 & - 1.0 & 2.0 \\ 0.0 & - 5.0 & 3.0 & - 2.0 \\ 0.0 & 0.0 & - 2.1999999999999993 & - 2.2 \end{array}]$ (2)

Two linear systems: same system matrix, different right hand sides

𝑨 = [\begin{array}{lll} 1 & 2 & 3 \\ 0 & 1 & 1 \\ 1 & 2 & 3 \end{array}] \in ℝ^{3 \times 3}, 𝒃 = [\begin{array}{l} 3 \\ 1 \\ 3 \end{array}] \in ℝ^{3}, 𝒄 = [\begin{array}{l} 3 \\ 1 \\ 4 \end{array}] \in ℝ^{3} .

[\begin{array}{ll} 𝑨 & 𝒃 \end{array}] = [\begin{array}{llll} 1 & 2 & 3 & 3 \\ 0 & 1 & 1 & 1 \\ 1 & 2 & 3 & 3 \end{array}] \sim [\begin{array}{ll} 𝑨_{1} & 𝒃_{1} \end{array}] = [\begin{array}{llll} 1 & 2 & 3 & 3 \\ 0 & 1 & 1 & 1 \\ 0 & 0 & 0 & 0 \end{array}] \Leftrightarrow {\begin{cases} x_{1} + 2 x_{2} + 3 x_{3} & = 3 \\ \begin{array}{llll}  \end{array} x_{2} + x_{3} & = 1 \\ \begin{array}{llllllll}  \end{array} 0 & = 0 \end{cases},

[\begin{array}{ll} 𝑨 & 𝒄 \end{array}] = [\begin{array}{llll} 1 & 2 & 3 & 3 \\ 0 & 1 & 1 & 1 \\ 1 & 2 & 3 & 4 \end{array}] \sim [\begin{array}{ll} 𝑨_{1} & 𝒄_{1} \end{array}] = [\begin{array}{llll} 1 & 2 & 3 & 3 \\ 0 & 1 & 1 & 1 \\ 0 & 0 & 0 & 1 \end{array}] \Leftrightarrow {\begin{cases} x_{1} + 2 x_{2} + 3 x_{3} & = 3 \\ \begin{array}{llll}  \end{array} x_{2} + x_{3} & = 1 \\ \begin{array}{llllllll}  \end{array} 0 & = 1 \end{cases} .

How to determine number of solutions?

Use similarity transformations to reduced row echelon form:
- All zero rows are below non-zero rows
- First non-zero entry on a row is called the leading entry
- In each non-zero row, the leading entry is to the left of lower leading entries
- Each leading entry equals 1 and is the only non-zero entry in its column
Row echelon form:
- Allow additional non-zero elements in a column, above the leading entry
After carrying out rref on bordered matrix $[\begin{array}{lll} 𝑨 & | . & 𝒃 \end{array}]$ , if:
- there is a row with $[\begin{array}{llllll} 0 & 0 & \dots & 0 & | . & 1 \end{array}] \Rightarrow$ No solutions
- the result is of form $[\begin{array}{lll} 𝑰 & | . & 𝒄 \end{array}] \Rightarrow$ Unique solution
- there is no row of form $[\begin{array}{llllll} 0 & 0 & \dots & 0 & | . & 1 \end{array}]$ , and there is a row of all zeros $[\begin{array}{llllll} 0 & 0 & \dots & 0 & | . & 0 \end{array}] \Rightarrow$ Infinitely many solutions
Examples
$[\begin{array}{llllll} 1 & - 1 & 0 & 1 & | . & 2 \\ 0 & 0 & 1 & - 1 & | . & 1 \\ 0 & 0 & 0 & 0 & | . & 0 \end{array}] \Rightarrow Infinitely many solutions$ $[\begin{array}{lllll} 1 & 2 & - 4 & | . & - 4 \\ 0 & 3 & - 1 & | . & 2 \\ 0 & 0 & 8 & | . & 8 \end{array}] \sim [\begin{array}{lllll} 1 & 0 & 0 & | . & - 2 \\ 0 & 1 & 0 & | . & 1 \\ 0 & 0 & 1 & | . & 1 \end{array}] \Rightarrow Unique solution$

Elementary matrices: Row combinations

Recall the basic operation in row echelon reduction: constructing a linear combination of rows to form zeros beneath the main diagonal, e.g.
$𝑨 = (\begin{array}{cccc} a_{11} & a_{12} & \dots & a_{1 m} \\ a_{21} & a_{22} & \dots & a_{2 m} \\ a_{31} & a_{32} & \dots & a_{3 m} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ a_{m 1} & a_{m 2} & \dots & a_{m m} \end{array}) \sim (\begin{array}{cccc} a_{11} & a_{12} & \dots & a_{1 m} \\ 0 & a_{22} - \frac{a_{21}}{a_{11}} a_{12} & \dots & a_{2 m} - \frac{a_{21}}{a_{11}} a_{1 m} \\ 0 & a_{32} - \frac{a_{31}}{a_{11}} a_{12} & \dots & a_{3 m} - \frac{a_{31}}{a_{11}} a_{1 m} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ 0 & a_{m 2} - \frac{a_{m 1}}{a_{11}} a_{12} & \dots & a_{m m} - \frac{a_{m 1}}{a_{11}} a_{1 m} \end{array})$
This can be stated as a matrix multiplication operation, with $l_{i 1} = a_{i 1} / a_{11}$
$(\begin{array}{ccccc} 1 & 0 & 0 & \dots & 0 \\ - l_{21} & 1 & 0 & \dots & 0 \\ - l_{31} & 0 & 1 & \dots & 0 \\ ⋮ & ⋮ & ⋮ & ⋱ & ⋮ \\ - l_{m 1} & 0 & 0 & \dots & 1 \end{array}) (\begin{array}{cccc} a_{11} & a_{12} & \dots & a_{1 m} \\ a_{21} & a_{22} & \dots & a_{2 m} \\ a_{31} & a_{32} & \dots & a_{3 m} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ a_{m 1} & a_{m 2} & \dots & a_{m m} \end{array}) = (\begin{array}{cccc} a_{11} & a_{12} & \dots & a_{1 m} \\ 0 & a_{22} - l_{21} a_{12} & \dots & a_{2 m} - l_{21} a_{1 m} \\ 0 & a_{32} - l_{31} a_{12} & \dots & a_{3 m} - l_{31} a_{1 m} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ 0 & a_{m 2} - l_{m 1} a_{12} & \dots & a_{m m} - l_{m 1} a_{1 m} \end{array})$

Elementary matrices: Permutation

Denote a permutation by
$σ = (\begin{array}{cccc} 1 & 2 & \dots & m \\ i_{1} & i_{2} & \dots & i_{m} \end{array})$
with $i_{1}, \dots, i_{m} \in {1, \dots, m}$ , $i_{j} \neq i_{k}$ for $j \neq k$
The sign of a permutation, $ν (σ)$ is the number of pair swaps needed to obtain the permutation starting from the identity permutation
$(\begin{array}{cccc} 1 & 2 & \dots & m \\ 1 & 2 & \dots & m \end{array})$
A permutation can be specified by a permutation matrix $𝑷$ obtained from $𝑰$ by swapping rows and columns $k \leftrightarrow i_{k}$

Gaussian multiplier matrix

Definition. The matrix

$𝑳_{k} = (\begin{array}{ccccc} 1 & \dots & 0 & \dots & 1 \\ 0 & ⋱ & 0 & \dots & 0 \\ 0 & \dots & 1 & \dots & 0 \\ 0 & \dots & - l_{k + 1, k} & \dots & 0 \\ 0 & \dots & - l_{k + 2, k} & \dots & 0 \\ ⋮ & \dots & ⋮ & ⋱ & ⋮ \\ 0 & \dots & - l_{m, k} & \dots & 1 \end{array})$

with $l_{i, k} = a_{i, k}^{(k)} / a_{k, k}^{(k)}$ , and $𝑨^{(k)} = (a_{i, j}^{(k)})$ the matrix obtained after step $k$ of row echelon reduction (or, equivalently, Gaussian elimination) is called a Gaussian multiplier matrix.

Permutation and Gaussian multiplier matrices are elementary matrices.

Gaussian multiplier inverse

The Gaussian multiplier matrix ...
$𝑳_{k} = (\begin{array}{ccccc} 1 & \dots & 0 & \dots & 0 \\ 0 & ⋱ & 0 & \dots & 0 \\ 0 & \dots & 1 & \dots & 0 \\ 0 & \dots & - l_{k + 1, k} & \dots & 0 \\ ⋮ & \dots & ⋮ & ⋱ & ⋮ \\ 0 & \dots & - l_{m, k} & \dots & 1 \end{array})$
... has inverse (matrix that “undoes” the linear transformation)
$𝑳_{k}^{- 1} = (\begin{array}{ccccc} 1 & \dots & 0 & \dots & 0 \\ 0 & ⋱ & 0 & \dots & 0 \\ 0 & \dots & 1 & \dots & 0 \\ 0 & \dots & l_{k + 1, k} & \dots & 0 \\ ⋮ & \dots & ⋮ & ⋱ & ⋮ \\ 0 & \dots & l_{m, k} & \dots & 1 \end{array})$

Inverse matrix

Consider elementary matrices
$𝑬_{1} = [\begin{array}{lll} 1 & 0 & 0 \\ 0 & 1 & 0 \\ - 3 & 0 & 1 \end{array}], 𝑬_{2} = [\begin{array}{lll} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 3 & 0 & 1 \end{array}], 𝑬_{1} 𝑬_{2} = 𝑬_{2} 𝑬_{1} = 𝑰,$
stating that $𝑬_{1}$ undoes the effect of $𝑬_{2}$ .
$𝑨 \in ℝ^{m \times m}$ is invertible if there exists $𝑿 \in ℝ^{m \times m}$ such that
$𝑨 𝑿 = 𝑿 𝑨 = 𝑰$
Notation $𝑿 = 𝑨^{- 1}$ , is the inverse of $𝑨$ .

Gauss-Jordan algorithm

What about general square matrices $𝑨 \in ℝ^{m \times m}$ ? How to find inverse
$𝑿$ is inverse if $𝑨 𝑿 = 𝑰$ or
$𝑨 [\begin{array}{llll} 𝒙_{1} & 𝒙_{2} & \dots & 𝒙_{m} \end{array}] = [\begin{array}{llll} 𝑨 𝒙_{1} & 𝑨 𝒙_{2} & \dots & 𝑨 𝒙_{m} \end{array}] = [\begin{array}{llll} 𝒆_{1} & 𝒆_{2} & \dots & 𝒆_{m} \end{array}]$
Find the inverse is equivalent to solving systems $𝑨 𝒙_{1} = 𝒆_{1}$ , ..., $𝑨 𝒙_{m} = 𝒆_{m}$
Gauss Jordan algoritm generalizes Gaussian elimination that solves a single linear system to solving $m$ systems simultaneously by forming the bordered matrix $[\begin{array}{lll} 𝑨 & | . & 𝑰 \end{array}]$
$[\begin{array}{lll} 𝑨 & | . & 𝑰 \end{array}] \sim [\begin{array}{lll} 𝑰 & | . & 𝑿 \end{array}]$

Existence of inverse

When does a matrix inverse exist? $𝑨 \in ℝ^{m \times m}$
1. $𝑨$ invertible
2. $𝑨 𝒙 = 𝒃$ has a unique solution for all $𝒃 \in ℝ^{m}$
3. $𝑨 𝒙 = 𝟎$ has a unique solution
4. The reduced row echelon form of $𝑨$ is $𝑰$
5. $𝑨$ can be written as product of elementary matrices
$a \Rightarrow b \Rightarrow c \Rightarrow d \Rightarrow e \Rightarrow a$

$a \Rightarrow b$

$𝑨$ invertible $\Rightarrow$ $𝑨^{- 1}$ exists, and $𝒙 = 𝑨^{- 1} 𝒃$ is a solution $𝑨 (𝑨^{- 1} 𝒃) = (𝑨 𝑨^{- 1}) 𝒃 = 𝒃$ . If there were two solutions $𝒙, 𝒚$ , then
$𝒙 - 𝒚 = (𝑨^{- 1} 𝑨) (𝒙 - 𝒚) = 𝑨^{- 1} (𝑨 𝒙 - 𝑨 𝒚) = 𝑨^{- 1} (𝒃 - 𝒃) = 𝑨^{- 1} 𝟎 = 𝟎 .$

$b \Rightarrow c$

Choose $𝒃 = 𝟎$

$c \Rightarrow d$

$[\begin{array}{lll} 𝑨 & | . & 𝟎 \end{array}] \sim [\begin{array}{lll} 𝑼 & | . & 𝟎 \end{array}]$ . If $𝑼 \neq 𝑰$ there is a row of zeros, and solution is not unique. If solution is unique then $𝑼 = 𝑰$

$d \Rightarrow e$

$[\begin{array}{lll} 𝑨 & | . & 𝟎 \end{array}] \sim [\begin{array}{lll} 𝑰 & | . & 𝟎 \end{array}]$ implies $𝑬_{k} \dots 𝑬_{1} 𝑨 = 𝑰 \Rightarrow 𝑨 = 𝑬_{1}^{- 1} \dots 𝑬_{k}^{- 1}$

$e \Rightarrow a$

$𝑨 =$ $𝑬_{1}^{- 1} \dots 𝑬_{k}^{- 1} \Rightarrow 𝑨^{- 1} = 𝑬_{k} \dots 𝑬_{1}$ .

Operations with matrix inverses

The inverse of a product ${(𝑨 𝑩)}^{- 1} = 𝑩^{- 1} 𝑨^{- 1}$
$(𝑨 𝑩) 𝑩^{- 1} 𝑨^{- 1} = 𝑨 (𝑩 𝑩^{- 1}) 𝑨^{- 1} = 𝑨 𝑰 𝑨^{- 1} = 𝑨 𝑨^{- 1} = 𝑰$ $𝑩^{- 1} 𝑨^{- 1} (𝑨 𝑩) = 𝑩^{- 1} (𝑨^{- 1} 𝑨) 𝑩 = 𝑩^{- 1} 𝑰 𝑩 = 𝑩^{- 1} 𝑩 = 𝑰$
If $𝑨 \in ℝ^{m \times m}$ invertible so are: $c 𝑨$ , $𝑨^{T}$ , $𝑨^{k}$
${(𝑨^{T})}^{- 1} = {(𝑨^{- 1})}^{T}$
Verify
$𝑨^{T} {(𝑨^{- 1})}^{T} = {(𝑨^{- 1} 𝑨)}^{T} = 𝑰$ ${(𝑨^{- 1})}^{T} 𝑨^{T} = {(𝑨^{} 𝑨^{- 1})}^{T} = 𝑰$