MATH347DS

MATH347DS L01: Vectors and Matrices

Overview

Quantitites
Vectors
Vector operations
Linear combinations
Matrix vector multiplication
Matrices
Matrix-matrix multiplication

Quantities

Numbers in mathematics

$ℕ$: The set of natural numbers, $ℕ = {0, 1, 2, 3, \dots}$ , infinite and countable, $ℕ_{+} = {1, 2, 3, \dots}$ ;
$ℤ$: The set of integers, $ℤ = {0, \pm 1, \pm 2, \pm 3, \dots}$ , infinite and countable;
$ℚ$: The set of rational numbers $ℚ = {p / q, p \in ℤ, q \in ℕ_{+}}$ , infinite and countable;
$ℝ$: The set of real numbers, infinite, not countable, can be ordered;
$ℂ$: The set of complex numbers, $ℂ = {x + i y, x, y \in ℝ}$ , infinite, not countable, cannot be ordered.

Numbers on a computer

Subsets of $ℕ$: The number types uint8, uint16, uint32, uint64 represent subsets of the natural numbers (unsigned integers) using 8, 16, 32, 64 bits respectively.
Subsets of $ℤ$: The number types int8, int16, int32, int64 represent subsets of the integers. One bit is used to store the sign of the number.
Subsets of $ℚ, ℝ, ℂ$: Computers approximate the real numbers through the set $𝔽$ of floating point numbers. Floating point numbers that use $b = 32$ bits are known as single precision, while those that use $b = 64$ are double precision.

Vectors

Some quantities arising in applications can be expressed as single numbers, called “scalars”
- Speed of a car on a highway $v = 35$ mph
- A person's height $H = 183$ cm
Many other quantitites require more than one number:
- Position in a city: “Intersection of 86^{$t h$} St and 3^{$r d$} Av”
- Position in 3D space: $(x, y, z)$
- Velocity in 3D space: $(u, v, w)$

Vector definition

Definition. A vector is a grouping of $m$ scalars

$𝒗 = [\begin{array}{c} v_{1} \\ v_{2} \\ ⋮ \\ v_{m} \end{array}] \in S^{m}, v_{i} \in S$

The scalars usually are naturals ( $S = N$ ), integers ( $S = Z$ ), rationals ( $S = Q$ ), reals ( $S = R$ ), or complex numbers ( $S = C$ )
We often denote the dimension and set of scalars as $𝒗 \in S^{m}$ , e.g. $𝒗 \in R^{m}$

Sets of vectors are denoted as

𝒱 = {𝒗 = [\begin{array}{c} v_{1} \\ v_{2} \\ ⋮ \\ v_{m} \end{array}], v_{i} \in S}

(1)

A vector can also be interpreted as a function from a subset of $ℕ$ to $S$
$v : {1, 2, . ., m} \to S$

Vector operations

Vector addition. Consider two vectors $𝒖, 𝒗 \in 𝒱$ . We define the sum of the two vectors as the vector containing the sum of the components

𝒘 = 𝒖 + 𝒗 = [\begin{array}{c} u_{1} \\ u_{2} \\ ⋮ \\ u_{m} \end{array}] + [\begin{array}{c} v_{1} \\ v_{2} \\ ⋮ \\ v_{m} \end{array}] = [\begin{array}{c} u_{1} + v_{1} \\ u_{2} + v_{2} \\ ⋮ \\ u_{m} + v_{m} \end{array}] = [\begin{array}{c} w_{1} \\ w_{2} \\ ⋮ \\ w_{m} \end{array}]

∴	u=[1 2 3]; v=[-2 1 2]; u+v

$[\begin{array}{ccc} - 1 & 3 & 5 \end{array}]$ (2)

Scalar multiplication. Consider $α \in S$ , $𝒖 \in 𝒱$ . We define the multiplication of vector $𝒖$ by scalar $α$ as the vector containing the product of each component of $𝒖$ with the scalar $α$
$𝒘 = α 𝒖 = [\begin{array}{c} α u_{1} \\ α u_{2} \\ ⋮ \\ α u_{m} \end{array}] = [\begin{array}{c} w_{1} \\ w_{2} \\ ⋮ \\ w_{m} \end{array}]$

Linear combinations

Linear combination. Let $α, β \in S$ , $𝒖, 𝒗 \in 𝒱$ . Define a linear combination of two vectors by
$𝒘 = α 𝒖 + β 𝒗 = [\begin{array}{c} α u_{1} \\ α u_{2} \\ ⋮ \\ α u_{m} \end{array}] + [\begin{array}{c} β v_{1} \\ β v_{2} \\ ⋮ \\ b β v_{m} \end{array}] = [\begin{array}{c} α u_{1} + β v_{1} \\ α u_{2} + β v_{2} \\ ⋮ \\ α u_{m} + β v_{m} \end{array}] = [\begin{array}{c} w_{1} \\ w_{2} \\ ⋮ \\ w_{m} \end{array}]$
Linear combination of $n$ vectors
$𝒃 = x_{1} 𝒂_{1} + x_{2} 𝒂_{2} + \dots + x_{n} 𝒂_{n} = [\begin{array}{l} x_{1} a_{11} + x_{2} a_{12} + \dots + x_{n} a_{1 n} \\ x_{1} a_{21} + x_{2} a_{22} + \dots + x_{n} a_{2 n} \\ ⋮ \\ x_{1} a_{m 1} + x_{2} a_{m 2} + \dots + x_{n} a_{m n} \end{array}]$

Uses of linear combinations

"Start at the center of town. Go east 3 blocks and north 2 blocks. What is your final position?"

𝒔 = [\begin{array}{c} 0 \\ 0 \end{array}], 𝒆_{E} = [\begin{array}{c} 1 \\ 0 \end{array}], 𝒆_{N} = [\begin{array}{c} 0 \\ 1 \end{array}]

α_{E} = 3, α_{N} = 2

𝒇 = 𝒔 + α_{E} 𝒆_{E} + α_{N} 𝒆_{N} = [\begin{array}{c} 0 \\ 0 \end{array}] + 3 [\begin{array}{c} 1 \\ 0 \end{array}] + 2 [\begin{array}{c} 0 \\ 1 \end{array}] = [\begin{array}{c} 3 \\ 2 \end{array}]

Linear combinations allow us to express a position in space using a standard set of directions. Questions:

How many standard directions are needed?
Can any position be specified as a linear combination?
How to find the scalars needed to express a position as a linear combination?

Matrices

Seek a more compact notation for the linear combination

a [\begin{array}{c} 1 \\ 2 \\ 3 \end{array}] + b [\begin{array}{c} - 1 \\ 0 \\ 1 \end{array}] + c [\begin{array}{c} 2 \\ 0 \\ 1 \end{array}] = [\begin{array}{c} a - b + 2 c \\ 2 a \\ 3 a + b + c \end{array}]

Group the vectors together to form a “matrix”
$𝑨 = [\begin{array}{ccc} 1 & - 1 & 2 \\ 2 & 0 & 0 \\ 3 & 1 & 1 \end{array}]$
Group the scalars together to form a vector
$𝒖 = [\begin{array}{c} a \\ b \\ c \end{array}]$

Matrix-vector multiplication = linear combination

Define matrix-vector multiplication
$𝑨 𝒖 = [\begin{array}{ccc} 1 & - 1 & 2 \\ 2 & 0 & 0 \\ 3 & 1 & 1 \end{array}] [\begin{array}{c} a \\ b \\ c \end{array}] = [\begin{array}{c} a - b + 2 c \\ 2 a \\ 3 a + b + c \end{array}]$
In general
$𝑨 = [\begin{array}{llll} a_{11} & a_{12} & \dots & a_{1 n} \\ a_{21} & a_{22} & \dots & a_{2 n} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ a_{m 1} & a_{m 2} & \dots & a_{m n} \end{array}] = [\begin{array}{llll} 𝒂_{1} & 𝒂_{2} & \dots & 𝒂_{n} \end{array}], 𝒙 = [\begin{array}{l} x_{1} \\ x_{2} \\ ⋮ \\ x_{n} \end{array}]$ $𝒃 = 𝑨 𝒙 = x_{1} 𝒂_{1} + x_{2} 𝒂_{2} + \dots + x_{n} 𝒂_{n} = [\begin{array}{l} x_{1} a_{11} + x_{2} a_{12} + \dots + x_{n} a_{1 n} \\ x_{1} a_{21} + x_{2} a_{22} + \dots + x_{n} a_{2 n} \\ ⋮ \\ x_{1} a_{m 1} + x_{2} a_{m 2} + \dots + x_{n} a_{m n} \end{array}]$

Examples: linear combination

Construct linear combination of vectors $𝒖 = [\begin{array}{lll} 1 & - 1 & 2 \end{array}]$ , $𝒗 = [\begin{array}{lll} 2 & 1 & - 1 \end{array}]$ scaled by $α = 2$ and $β = 3$ , respectively

∴	u=[1 -1 2]; v=[2 1 -1]; alpha=2; beta=3;

∴	alphau+betav

$[\begin{array}{ccc} 8 & 1 & 1 \end{array}]$ (3)

Construct linear combination of vectors $𝒖 = [\begin{array}{l} 1 \\ - 1 \\ 2 \end{array}]$ , $𝒗 = [\begin{array}{l} 2 \\ 1 \\ - 1 \end{array}]$ scaled by $α = 2$ and $β = 3$ , respectively

∴	u=[1; -1; 2]; v=[2; 1; -1]; alphau+betav

$[\begin{array}{c} 8 \\ 1 \\ 1 \end{array}]$ (4)

Examples: linear combination by matrix-vector combination

Construct linear combination of vectors $𝒖 = [\begin{array}{l} 1 \\ - 1 \\ 2 \end{array}]$ , $𝒗 = [\begin{array}{l} 2 \\ 1 \\ - 1 \end{array}]$ scaled by $α = 2$ and $β = 3$ , respectively

∴	u=[1; -1; 2]; v=[2; 1; -1]; alpha=2; beta=3;

∴	A=[u v]; x=[alpha; beta]; A*x

$[\begin{array}{c} 8 \\ 1 \\ 1 \end{array}]$ (5)

Matrices

Definition. An $m$ by $n$ matrix is a grouping of $n$ vectors,

$𝑨 = [\begin{array}{llll} 𝒂_{1} & 𝒂_{2} & \dots & 𝒂_{n} \end{array}] \in S^{m \times n},$

where each vector has $m$ scalar components $𝒂_{1}, 𝒂_{2}, \dots, 𝒂_{n} \in S^{m}$ .

Notation conventions:
- scalars: normal face, Latin or Greek letters, $a, b, α, β, u_{1}, a_{11}$ , $A_{11}, I_{12}$
- vectors: bold face, lower case Latin letters, $𝒖, 𝒗, 𝒂_{1}$
- matrices: bold face, upper case Latin letters, $𝑨, 𝑩, 𝑳_{1}$

Matrices

Matrix components
$𝒂_{1} = [\begin{array}{l} a_{11} \\ a_{21} \\ ⋮ \\ a_{m 1} \end{array}], 𝒂_{2} = [\begin{array}{l} a_{12} \\ a_{22} \\ ⋮ \\ a_{m 2} \end{array}], \dots, 𝒂_{n} = [\begin{array}{l} a_{1 n} \\ a_{2 n} \\ ⋮ \\ a_{m n} \end{array}] \Rightarrow$ $𝑨 = [\begin{array}{llll} a_{11} & a_{12} & \dots & a_{1 n} \\ a_{21} & a_{22} & \dots & a_{2 n} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ a_{m 1} & a_{m 2} & \dots & a_{m n} \end{array}]$

A real-valued matrix with $m$ lines and $n$ : $𝑨 \in ℝ^{m \times n}$

∴	A=[3 1 2; -1 0 1; 3 4 1]

$[\begin{array}{ccc} 3 & 1 & 2 \\ - 1 & 0 & 1 \\ 3 & 4 & 1 \end{array}]$ (6)

Matrix components

Instead of explicitly writing out components, it is often convenient to specify a matrix by a rule to construct each component
$𝑨 \in ℝ^{m \times n},, 𝑨 = [a_{i j}]$
with indices taking values $i \in {1, \dots, m}, j \in {1, \dots, n}$ .

Example: A Hilbert matrix $𝑯_{m} \in ℝ^{m \times m}$ is defined as $𝑯_{m} = [\frac{1}{i + j - 1}]$

∴	function hilb(m) H=ones(m,m) for i=1:m for j=1:m H[i,j]=1.0/(i+j-1.0) end end return H end;

∴

hilb(3)

$[\begin{array}{ccc} 1.0 & 0.5 & 0.3333333333333333 \\ 0.5 & 0.3333333333333333 & 0.25 \\ 0.3333333333333333 & 0.25 & 0.2 \end{array}]$ (7)

Note that a vector is a matrix with a single column. The notation $𝒗 \in ℝ^{m},$ is a customary shorter form of $𝒗 \in ℝ^{m \times 1}$ .

Example: Column vectors of a matrix

𝑨 = [\begin{array}{lll} 3 & 1 & 2 \\ - 1 & 0 & 1 \\ 3 & 4 & 1 \end{array}] = [\begin{array}{lll} 𝒂_{1} & 𝒂_{2} & 𝒂_{3} \end{array}]

∴	A=[3 1 2; -1 0 1; 3 4 1]

$[\begin{array}{ccc} 3 & 1 & 2 \\ - 1 & 0 & 1 \\ 3 & 4 & 1 \end{array}]$ (8)

∴

A[:,2]

$[\begin{array}{c} 1 \\ 0 \\ 4 \end{array}]$ (9)

∴

Example: Row vectors of a matrix

𝑨 = [\begin{array}{lll} 3 & 1 & 2 \\ - 1 & 0 & 1 \\ 3 & 4 & 1 \end{array}] = [\begin{array}{lll} 𝒂_{1} & 𝒂_{2} & 𝒂_{3} \end{array}]

∴	A=[3 1 2; -1 0 1; 3 4 1]

$[\begin{array}{ccc} 3 & 1 & 2 \\ - 1 & 0 & 1 \\ 3 & 4 & 1 \end{array}]$ (10)

∴

A[2,:]'

$[\begin{array}{ccc} - 1 & 0 & 1 \end{array}]$ (11)

∴

Example: Block within a matrix

𝑨 = [\begin{array}{lll} 3 & 1 & 2 \\ - 1 & 0 & 1 \\ 3 & 4 & 1 \end{array}] = [\begin{array}{lll} 𝒂_{1} & 𝒂_{2} & 𝒂_{3} \end{array}]

∴	A=[3 1 2; -1 0 1; 3 4 1]

$[\begin{array}{ccc} 3 & 1 & 2 \\ - 1 & 0 & 1 \\ 3 & 4 & 1 \end{array}]$ (12)

∴

A[:,2:3]

$[\begin{array}{cc} 1 & 2 \\ 0 & 1 \\ 4 & 1 \end{array}]$ (13)

∴

Simple vectors, matrices, matrix addition and scaling

Single component vector is a scalar $𝒗 = [v_{1}] \equiv v_{1}$
Single column vector matrix is a vector $𝑨 = [𝒂_{1}] \equiv 𝒂_{1}$
Vector addition carries over to matrices $𝑨, 𝑩 \in ℝ^{m \times n}$
$𝑨 + 𝑩 = [\begin{array}{llll} 𝒂_{1} & 𝒂_{2} & \dots & 𝒂_{n} \end{array}] + [\begin{array}{llll} 𝒃_{1} & 𝒃_{2} & \dots & 𝒃_{n} \end{array}]$
Vector scaling carries over to matrices
$α 𝑨 = α [\begin{array}{llll} 𝒂_{1} & 𝒂_{2} & \dots & 𝒂_{n} \end{array}] = [\begin{array}{llll} α 𝒂_{1} & α 𝒂_{2} & \dots & α 𝒂_{n} \end{array}]$
Identity matrix $𝑰 = [\begin{array}{llll} 𝒆_{1} & 𝒆_{2} & \dots & 𝒆_{m} \end{array}] \in ℝ^{m \times m}$
$𝒙 = [\begin{array}{l} x_{1} \\ x_{2} \\ ⋮ \\ x_{m} \end{array}] = x_{1} 𝒆_{1} + x_{2} 𝒆_{2} + \dots + x_{m} 𝒆_{m} = 𝑰 𝒙, 𝒆_{1} = [\begin{array}{l} 1 \\ 0 \\ ⋮ \\ 0 \\ 0 \end{array}], 𝒆_{2} = [\begin{array}{l} 0 \\ 1 \\ ⋮ \\ 0 \\ 0 \end{array}], \dots, 𝒆_{m} = [\begin{array}{l} 0 \\ 0 \\ ⋮ \\ 0 \\ 1 \end{array}] .$

Swapping representations: from columns to rows through transposition

Matrix transpose in terms of column vectors
$𝑨 = [\begin{array}{llll} 𝒂_{1} & 𝒂_{2} & \dots & 𝒂_{n} \end{array}] \in ℝ^{m \times n}, 𝑨^{T} = [\begin{array}{l} 𝒂_{1}^{T} \\ 𝒂_{2}^{T} \\ ⋮ \\ 𝒂_{n}^{T} \end{array}] \in ℝ^{n \times m} .$
Matrix transpose in terms of components
$𝑨 = [\begin{array}{llll} a_{1, 1} & a_{1, 2} & \dots & a_{1, n} \\ a_{2, 1} & a_{2, 2} & \dots & a_{2, n} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ a_{m, 1} & a_{m, 2} & a_{m, n} \end{array}] \in ℝ^{m \times n}, 𝑨^{T} = [\begin{array}{llll} a_{1, 1} & a_{2, 1} & \dots & a_{m, 1} \\ a_{1, 2} & a_{2, 2} & \dots & a_{m, 2} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ a_{1, n} & a_{2, n} & a_{m, n} \end{array}] \in ℝ^{n \times m}$

Matrix-matrix multiplication

Definition. Consider matrices $𝑨 = [\begin{array}{lll} 𝒂_{1} & \dots & 𝒂_{n} \end{array}] \in ℝ^{m \times n}$ , and $𝑿 = [\begin{array}{lll} 𝒙_{1} & \dots & 𝒙_{p} \end{array}] \in ℝ^{n \times p} .$ The matrix product $𝑩 = 𝑨 𝑿$ is a matrix $𝑩 = [\begin{array}{lll} 𝒃_{1} & \dots & 𝒃_{p} \end{array}] \in ℝ^{m \times p}$ with column vectors given by the matrix vector products

$𝒃_{k} = 𝑨 𝒙_{k}, for k = 1, 2 \dots, p .$

A matrix-matrix product is simply a set of matrix-vector products, and hence expresses multiple linear combinations in a concise way.
The dimensions of the matrices must be compatible, the number of rows of $𝑿$ must equal the number of columns of $𝑨$ .
A matrix-vector product is a special case of a matrix-matrix product when $p = 1$ .
We often write $𝑩 = 𝑨 𝑿$ in terms of columns as
$[\begin{array}{lll} 𝒃_{1} & \dots & 𝒃_{p} \end{array}] = 𝑨 [\begin{array}{lll} 𝒙_{1} & \dots & 𝒙_{p} \end{array}] = [\begin{array}{lll} 𝑨 𝒙_{1} & \dots & 𝑨 𝒙_{p} \end{array}]$

Matrix multiplication example

∴	A=[1 0 3; 2 1 4; -1 0 3]

$[\begin{array}{ccc} 1 & 0 & 3 \\ 2 & 1 & 4 \\ - 1 & 0 & 3 \end{array}]$ (14)

∴	X=[1 -1 0; 1 1 1; 0 1 0]

$[\begin{array}{ccc} 1 & - 1 & 0 \\ 1 & 1 & 1 \\ 0 & 1 & 0 \end{array}]$ (15)

∴

A*X

$[\begin{array}{ccc} 1 & 2 & 0 \\ 3 & 3 & 1 \\ - 1 & 4 & 0 \end{array}]$ (16)

∴	[AX[:,1] AX[:,2] A*X[:,3]]

$[\begin{array}{ccc} 1 & 2 & 0 \\ 3 & 3 & 1 \\ - 1 & 4 & 0 \end{array}]$ (17)

∴

Matrix multiplication: componentwise

Definition. Consider matrices $𝑨 = [\begin{array}{l} a_{i, j} \end{array}] \in ℝ^{m \times n}$ , and $𝑿 = [x_{i, j}] \in ℝ^{n \times p} .$ The matrix product $𝑩 = 𝑨 𝑿 = [b_{i, j}]$ is a matrix $𝑩 \in ℝ^{m \times p}$ with components

$b_{i, j} = a_{i, 1} x_{1, j} + a_{i, 2} x_{2, j} + \dots + a_{i, n} x_{n, j} = \sum_{k = 1}^{n} a_{i, k} x_{k, j}$

$𝑩 = [\begin{array}{lll} 𝒃_{1} & \dots & 𝒃_{p} \end{array}], 𝒃_{1} = [\begin{array}{l} b_{1, 1} \\ b_{2, 1} \\ ⋮ \\ b_{m, 1} \end{array}] = x_{1, 1} 𝒂_{1} + x_{2, 1} 𝒂_{2} + \dots + x_{n, 1} 𝒂_{n}$

$𝑩 = [\begin{array}{lll} b_{1, 1} & \dots & b_{1, p} \\ b_{2, 1} & \dots & b_{2, p} \\ ⋮ & ⋱ & ⋮ \\ b_{m, 1} & b_{m, p} \end{array}] = 𝑨 𝑿 = [\begin{array}{llll} a_{1, 1} & a_{1, 2} & \dots & a_{1, n} \\ a_{2, 1} & a_{2, 2} & \dots & a_{2, n} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ a_{m, 1} & a_{m, 2} & a_{m, n} \end{array}] [\begin{array}{llll} x_{1, 1} & x_{1, 2} & \dots & x_{1, p} \\ x_{2, 1} & x_{2, 2} & \dots & x_{2, p} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ x_{n, 1} & x_{n, 2} & x_{n, p} \end{array}]$

$b_{2, 1} = a_{2, 1} x_{1, 1} + a_{2, 2} x_{2, 1} + \dots + a_{2, n} x_{n, 1}$