Overview

• Linear dependence and independence

• Orthogonal, orthonormal vector sets

• Orthogonal matrices

• Basis, dimension

• Realistic application of vector operations framework: ECG representation and compression

  • Sampling

  • Recursive definition of $𝑰$

  • Hadamard-Walsh matrices

  • Compression by truncation of linear combinations.

• Let $𝑨\in {ℝ}^{m\times n}$ be a matrix with $n$ column vectors, each with $m$ components

  ${\displaystyle 𝑨=\left[\begin{array}{llll}{𝒂}_1& {𝒂}_2& \dots & {𝒂}_n\end{array}\right],{𝒂}_1,{𝒂}_2,\dots ,{𝒂}_n\in {ℝ}^m}$

• $𝑨$ can be thought of as representing a linear mapping $𝒇$ from ${ℝ}^n$ to ${ℝ}^m$, ${ℝ}^n\stackrel{𝑨}{\to }{ℝ}^m$

  ${\displaystyle 𝒇:{ℝ}^n\to {ℝ}^m,𝑨=\left[\begin{array}{llll}𝒇\left({𝒆}_1\right)& 𝒇\left({𝒆}_2\right)& \dots & 𝒇\left({𝒆}_n\right)\end{array}\right],{𝑰}_n\in {ℝ}^{n\times n},{𝑰}_n=\left[\begin{array}{llll}{𝒆}_1& {𝒆}_2& \dots & {𝒆}_n\end{array}\right]}$

• Column space, $C\left(𝑨\right)=\{𝒃\in {ℝ}^m|\exists 𝒙\in {ℝ}^n\mathrm{such}\mathrm{that}𝒃=𝑨𝒙\}\subseteq {ℝ}^m$, the part of ${ℝ}^m$ reachable by linear combination of columns of $𝑨$

• Left null space, $N\left({𝑨}^T\right)=\{𝒚\in {ℝ}^m|{𝑨}^T𝒚=0\}\subseteq {ℝ}^m$, the part of ${ℝ}^m$ not reachable by linear combination of columns of $𝑨$

• Row space, $R\left(𝑨\right)=C\left({𝑨}^T\right)=\{𝒄\in {ℝ}^n|\exists 𝒚\in {ℝ}^m\mathrm{such}\mathrm{that}𝒄={𝑨}^T𝒚\}\subseteq {ℝ}^n$, the part of ${ℝ}^n$ reachable by linear combination of rows of $𝑨$

• Null space, $N\left(𝑨\right)=\{𝒙\in {ℝ}^n|𝑨𝒙=0\}\subseteq {ℝ}^n$, the part of ${ℝ}^n$ not reachable by linear combination of rows of $𝑨$

• Zero product property of scalar multiplication: $ax=0\Rightarrow a=0$ or $x=0$

• Matrix-vector counterexamples of zero product property

  • $𝑨=\left[\begin{array}{ll}{𝒂}_1& {𝒂}_2\end{array}\right]=\left[\begin{array}{ll}1& 1\\ 2& 2\end{array}\right],𝑨𝒙=\left[\begin{array}{ll}1& 1\\ 2& 2\end{array}\right]\left[\begin{array}{l}1\\ -1\end{array}\right]=\left[\begin{array}{l}0\\ 0\end{array}\right]=𝟎$

  • $𝑩=\left[\begin{array}{lll}{𝒃}_1& {𝒃}_2& {𝒃}_3\end{array}\right]=\left[\begin{array}{lll}1& -1& 1\\ 2& 0& 4\\ 3& 1& 7\end{array}\right],𝑩𝒙=\left[\begin{array}{lll}1& -1& 1\\ 2& 0& 4\\ 3& 1& 7\end{array}\right]\left[\begin{array}{l}2\\ 1\\ -1\end{array}\right]=\left[\begin{array}{l}0\\ 0\\ 0\end{array}\right]=𝟎$

• Matrix-vector example satisfying the zero product property $𝑰𝒙=0\Rightarrow 𝒙=𝟎$

• Question: how to distinguish between above examples?

• Note:

  • ${𝒂}_1={𝒂}_2$

  • ${𝒃}_3=2{𝒃}_1+{𝒃}_2$

Definition. The vectors ${𝒂}_1,{𝒂}_2,\dots ,{𝒂}_n\in 𝒱,$are linearly dependent if there exist $n$ scalars, $x_1,\dots ,x_n\in 𝒮$, at least one of which is different from zero such that

${\displaystyle x_1{𝒂}_1+\dots x_n{𝒂}_n=𝟎}$

Note that $\left\{𝟎\right\}$, with $𝟎\in 𝒱$ is a linearly dependent set of vectors since $1\cdot 𝟎=𝟎$.

The converse of linear dependence is linear independence, a member of the set cannot be expressed as a non-trivial linear combination of the other vectors

Definition. The vectors ${𝒂}_1,{𝒂}_2,\dots ,{𝒂}_n\in 𝒱,$are linearly independent if the only $n$ scalars, $x_1,\dots ,x_n\in 𝒮$, that satisfy

${\displaystyle x_1{𝒂}_1+\dots x_n{𝒂}_n=𝟎,}$

(1)

are $x_1=0$, $x_2=0$,…,$x_n=0$.

The choice $𝒙={\left(\begin{array}{lll}{x}_1& \dots & x_n\end{array}\right)}^T=𝟎$ that always satisfies (1) is called a trivial solution. We can restate linear independence as (1) being satisfied only by the trivial solution.

Recall:

Definition. The null space of a matrix $𝑨\in {ℝ}^{m\times n}$ is the set

• If $N\left(𝑨\right)=\left\{𝟎\right\}$ then the column vectors of $𝑨$ are linearly independent, since the only way to satisfy (1) is by the trivial solution $𝒙=𝟎$

Definition. The left null space of a matrix $𝑨\in {ℝ}^{m\times n}$ is the set

• If $N\left({𝑨}^T\right)=\left\{𝟎\right\}$ then the row vectors of $𝑨$ are linearly independent, since the only way to satisfy (1) is by the trivial solution $𝒙=𝟎$

Definition. The column vectors ${𝒖}_1,{𝒖}_2,\dots ,{𝒖}_n\in {ℝ}^m$ of matrix $𝑼\in {ℝ}^{m\times n}$ are orthogonal if

${\displaystyle {𝑼}^T𝑼=\mathrm{diag}({||{𝒖}_1||}^2,\dots ,{||{𝒖}_n||}^2).}$

Definition. The column vectors ${𝒒}_1,{𝒒}_2,\dots ,{𝒒}_n\in {ℝ}^m$ of matrix $𝑸\in {ℝ}^{m\times n}$ are orthonormal if

${\displaystyle {𝑸}^T𝑸=𝑰.}$

Definition. The matrix $𝑸\in {ℝ}^{m\times m}$ is orthogonal if

${\displaystyle {𝑸}^T𝑸=𝑸{𝑸}^T=𝑰.}$

Example. The reflection matrix across direction $𝒒$, $||𝒒||=1$ in ${ℝ}^m$,${𝑹}_{𝒒}=2𝒒{𝒒}^T-𝑰,$is orthogonal

${\displaystyle {𝑹}_{𝒒}{𝑹}_{𝒒}^T=(2𝒒{𝒒}^T-𝑰){(2𝒒{𝒒}^T-𝑰)}^T=(2𝒒{𝒒}^T-𝑰)(2𝒒{𝒒}^T-𝑰)=4𝒒{𝒒}^T𝒒{𝒒}^T-4𝒒{𝒒}^T-I=𝑰}$

since $𝒒{𝒒}^T𝒒{𝒒}^T=𝒒\left({𝒒}^T𝒒\right){𝒒}^T=𝒒\left(1\right){𝒒}^T=𝒒{𝒒}^T$.

Suppose in $𝒱=(V,ℝ,+,\cdot )$ the set $ℬ=\{{𝒂}_1,{𝒂}_2,\dots \}$ spans $V$, $V=\mathrm{span}ℬ$. Adding another vector does not change the span $\mathrm{span}ℬ=\mathrm{span}(ℬ\cup \left\{𝒃\right\})$. Intuitively $ℬ\cup \left\{𝒃\right\}$ contains a redundant vector, it is not a minimal spanning set. Avoid redundancy by defining minimal spanning sets.

Definition. A set of vectors ${𝒖}_1,\dots ,{𝒖}_n\in V$ is a basis for vector space $𝒱$ if:

Adding another vector $𝒃\in$leads to a linearly dependent set $\{{𝒖}_1,\dots ,{𝒖}_n,𝒃\}$.

Definition. The number of vectors ${𝒖}_1,\dots ,{𝒖}_n\in V$ within a basis is the dimension of the vector space $𝒱$.

• $C\left(𝑨\right)$ the column space of $𝑨$, $C\left(𝑨\right)\le {ℝ}^m$

• $C\left({𝑨}^T\right)$ the row space of $𝑨$, $C\left({𝑨}^T\right)\le {ℝ}^n$

• $N\left(𝑨\right)$ the null space of $𝑨$, $N\left(𝑨\right)\le {ℝ}^n$

• $N\left({𝑨}^T\right)$ the left null space of $𝑨$, or null space of ${𝑨}^T$, $N\left({𝑨}^T\right)\le {ℝ}^m$.

The dimensions of these subspaces arise so often in applications to warrant formal definition.

Definition. The rank of a matrix $𝑨\in {ℝ}^{m\times n}$ is the dimension of its column space.

Definition. The nullity of a matrix $𝑨\in {ℝ}^{m\times n}$ is the dimension of its null space.

Dimension of column space equals dimension of row space

${\displaystyle 𝒃=𝑨𝒙=x_1{𝒂}_1+\cdots +x_n{𝒂}_n\iff {𝒃}^T={(𝑨𝒙)}^T={𝒙}^T{𝑨}^T=x_1{𝒂}_1^T+\cdots +x_n{𝒂}_n^T.}$

∴

using MAT

∴	DataFileName = homedir()*"/courses/MATH347DS/data/ecg/ECGData.mat";

∴	DataFile = matopen(DataFileName,"r");

∴	dict = read(DataFile,"ECGData");

∴	data = dict["Data"]';

∴	size(data)

${\displaystyle \left[\begin{array}{c}65536\\ 162\end{array}\right]}$

(4)

∴	q=12; m=2^q; k=15; b=data[1:m,k];

∴	figure(1); clf(); plot(b); title("Electrocardiogram");

∴	cd(homedir()*"/courses/MATH347DS/images"); savefig("S04Fig01.eps");

∴

• Consider $m=2^q$, denote ${𝑰}_q$ the identity matrix of size $m\times m=2^q\times 2^q$

  ${\displaystyle {𝑰}_0=\left[1\right],{𝑰}_1=\left[\begin{array}{ll}1& 0\\ 0& 1\end{array}\right].}$ ${\displaystyle {𝑰}_2=\left[\begin{array}{llll}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]=\left[\begin{array}{ll}{𝑰}_1& 𝟎\\ 𝟎& {𝑰}_1\end{array}\right]=\left[\begin{array}{ll}1\cdot {𝑰}_1& 0\cdot {𝑰}_1\\ 0\cdot {𝑰}_1& 1\cdot {𝑰}_1\end{array}\right],{𝑰}_3=\left[\begin{array}{ll}1\cdot {𝑰}_2& 0\cdot {𝑰}_2\\ 0\cdot {𝑰}_2& 1\cdot {𝑰}_2\end{array}\right].}$

Definition. The exterior product of matrices $𝑨=\left[a_{ij}\right]\in {ℝ}^{m\times n}$ and $𝑩\in {ℝ}^{p\times q}$ is the matrix $𝑪\in {ℝ}^{(mp)\times (nq)}$

${\displaystyle 𝑪=𝑨\otimes 𝑩=\left[\begin{array}{llll}{a}_{11}𝑩& a_{12}𝑩& \dots & a_{1n}𝑩\\ a_{21}𝑩& a_{21}𝑩& \dots & a_{2n}𝑩\\ ⋮& ⋮& \ddots & ⋮\\ a_{m1}𝑩& a_{m2}𝑩& \dots & a_{mn}𝑩\end{array}\right].}$

• ${𝑰}_2={𝑰}_1\otimes {𝑰}_1,{𝑰}_3={𝑰}_1\otimes {𝑰}_2$

• Choose a differnt set of starting matrices and obtain another sequence ${𝑯}_q\in {ℝ}^{m\times m}$, $m=2^q$

  ${\displaystyle {𝑯}_0=\left[1\right],{𝑯}_1=\left[\begin{array}{ll}1& 1\\ 1& -1\end{array}\right],{𝑯}_q={𝑯}_1\otimes {𝑯}_{q-1}.}$

  ∴	using Hadamard

  ∴	H2=hadamard(2^2)

  ${\displaystyle \left[\begin{array}{cccc}1& 1& 1& 1\\ 1& -1& 1& -1\\ 1& 1& -1& -1\\ 1& -1& -1& 1\end{array}\right]}$

  (5)

  ∴	H2=hadamard(2^3)

  ${\displaystyle \left[\begin{array}{cccccccc}1& 1& 1& 1& 1& 1& 1& 1\\ 1& -1& 1& -1& 1& -1& 1& -1\\ 1& 1& -1& -1& 1& 1& -1& -1\\ 1& -1& -1& 1& 1& -1& -1& 1\\ 1& 1& 1& 1& -1& -1& -1& -1\\ 1& -1& 1& -1& -1& 1& -1& 1\\ 1& 1& -1& -1& -1& -1& 1& 1\\ 1& -1& -1& 1& -1& 1& 1& -1\end{array}\right]}$

  (6)

• The pattern of components in ${𝑯}_q$ less discernable than that in ${𝑰}_q$

• Visualize the non-zero elements in matrices

  ∴	q=5; m=2^q; Iq=Matrix(1.0I,m,m); Hq=hadamard(m);

  ∴	clf(); subplot(1,2,1); spy(Iq); subplot(1,2,2); spy(Hq.+1);

  ∴	cd(homedir()*"/courses/MATH347DS/images"); savefig("S03Fig02.eps");

  ∴

Vectors of $𝑰$ sample one moment in time, vectors of $𝑯$ sample multiple moments

• Idea: drop some of the terms in the linear combinations. Instead of

  ${\displaystyle 𝒃=𝑰𝒃=b_1{𝒆}_1+\cdots +b_m{𝒆}_m=c_1{𝒉}_1+\cdots +c_m{𝒉}_m=𝑯𝒄}$

  define

  ${\displaystyle 𝒖=b_1{𝒆}_1+\cdots +b_n{𝒆}_n,𝒗=c_1{𝒉}_1+\cdots +c_n{𝒉}_n}$

  ∴	q=12; m=2^q; k=15; b=data[1:m,k];

  ∴	Iq=Matrix(1.0I,m,m); Hq=hadamard(m); c=(1/m)*transpose(Hq)*b;

  ∴	n=2^10; u=Iq[:,1:n]*b[1:n]; v=Hq[:,1:n]*c[1:n];

  ∴	figure(2); clf(); subplot(3,1,1); plot(b);

  ∴	subplot(3,1,2); plot(u);

  ∴	subplot(3,1,3); plot(v);

  ∴	cd(homedir()*"/courses/MATH347DS/images"); savefig("S04Fig03.eps");

  ∴