1.1.Relations

The previous chapter focused on mathematical expression of the concept of quantification, the act of associating human observation with measurements, as a first step of scientific inquiry. Consideration of different types of quantities led to various types of numbers, vectors as groupings of numbers, and matrices as groupings of vectors. Symbols were introduced for these quantities along with some intial rules for manipulating such objects, laying the foundation for an algebra of vectors and matrices. Science seeks to not only observe, but to also explain, which now leads to additional operations for working with vectors and matrices that will define the framework of linear algebra.

Explanations within scientific inquiry are formulated as hypotheses, from which predictions are derived and tested. A widely applied mathematical transcription of this process is to organize hypotheses and predictions as two sets $X$ and $Y$, and then construct another set $R$ of all of the instances in which an element of $X$ is associated with an element in $Y$. The set of all possible instances of $x\in X$ and $y\in Y$, is the Cartesian product of $X$ with $Y$, denoted as $X\times Y=\{(x,y)|.x\in X,y\in Y\}$, a construct already encountered in the definition of the real 2-space ${ℛ}_2=({ℝ}^2,ℝ,+,\cdot )$ where ${ℝ}^2=ℝ\times ℝ$. Typically, not all possible tuples $(x,y)\in X\times Y$ are relevant leading to the following definition.

Definition. (Relation) . A relation $R$ between two sets $X,Y$ is a subset of the Cartesian product $X\times Y$, $R\subseteq X\times Y$.

The key concept is that of associating an input $x\in X$ with an output $y\in Y$. Inverting the approach and associating an output to an input is also useful, leading to the definition of an inverse relation as $R^{-1}\subseteq Y\times X$, $R^{-1}=\{(y,x)|(x,y)\in R.\}$. Note that an inverse exists for any relation, and the inverse of an inverse is the original relation, ${\left(R^{-1}\right)}^{-1}=R$. From the above, a relation is a triplet (a tuple with three elements), $(X,Y,R)$, that will often be referred to by just its last member $R$.

Homogeneous relations.

1.2.Functions

Functions between sets $X$ and $Y$ are a specific type of relationship that often arise in science. For a given input $x\in X$, theories that predict a single possible output $y\in Y$ are of particular scientific interest.

Definition. (Function) . A function from set $X$ to set $Y$ is a relation $F\subseteq X\times Y$, that associates to $x\in X$ a single $y\in Y$.

The above intuitive definition can be transcribed in precise mathematical terms as $F\subseteq X\times Y$ is a function if $(x,y)\in F$ and $(x,z)\in F$ implies $y=z$. Since it's a particular kind of relation, a function is a triplet of sets $(X,Y,F)$, but with a special, common notation to denote the triplet by $f:X\to Y$, with $F=\left\{(x,f\left(x\right))|x\in X,f\left(x\right)\in Y.\right\}$ and the property that $(x,y)\in F\Rightarrow y=f\left(x\right)$. The set $X$ is the domain and the set $Y$ is the codomain of the function $f$. The value from the domain $x\in X$ is the argument of the function associated with the function value $y=f\left(x\right)$. The function value $y$ is said to be returned by evaluation $y=f\left(x\right)$.

Whereas all relations can be inverted, and inversion of a function defines a new relation, but which might not itself be a function. For example the relation $S^{-1}=\{(\alpha ,a),(\beta ,a),(\gamma ,a)\}$ is a function, but its inverse ${\left(S^{-1}\right)}^{-1}=S$ is not.

Simple functions such as sin, cos, exp, log, are predefined in Julia, and can be applied to each component of a vector argument by broadcasting, denoted by a period in front of the paranteses enclosing the argument.

∴ 

θ=π; [sin(θ) cos(θ) exp(θ) log(θ)]

∴ 

θ=0:π/6:π; short(x)=round(x,digits=6); short.(sin.(θ))'

∴ 

short.(log2.(1:8))'

∴ 

A construct that will be often used is to interpret a vector within $E_m$ as a function, since $𝒗\in {ℝ}^m$ with components $𝒗={\left[\begin{array}{llll}{v}_1& v_2& \dots & v_m\end{array}\right]}^T$ also defines a function $v:\{1,2,\dots ,m\}\to ℝ$, with values $v\left(i\right)=v_i$. As the number of components grows the function $v$ can provide better approximations of some continuous function $f\in {𝒞}^0\left(ℝ\right)$ through the function values $v_i=v\left(i\right)=f\left(x_i\right)$ at distinct sample points $x_1,x_2,\dots ,x_m$.

The above function examples are all defined on a domain of scalars or naturals and returned scalar values. Within linear algebra the particular interest is on functions defined on sets of vectors from some vector space $𝒱=(V,S,+,\cdot )$ that return either scalars $f:V\to S$, or vectors from some other vector space $𝒲=(W,S,+,\cdot )$, $𝒈:V\to W$. The codomain of a vector-valued function might be the same set of vectors as its domain, $𝒉:V\to V$. The fundamental operation within linear algebra is the linear combination $a𝒖+b𝒗$ with $a,b\in S$, $𝒖,𝒗\in V$. A key aspect is to characterize how a function behaves when given a linear combination as its argument, for instance $f(a𝒖+b𝒗)$ or $𝒈(a𝒖+b𝒗).$

1.3.Linear functionals

Consider first the case of a function defined on a set of vectors that returns a scalar value. These can be interpreted as labels attached to a vector, and are very often encountered in applications from natural phenomena or data analysis.

Definition. (Functional) . A functional on vector space $𝒱=(V,S,+,\cdot )$ is a function from the set of vectors $V$ to the set of scalars $S$ of the vector space $𝒱$.

Definition. (Linear Functional) . The functional $f:V\to S$ on vector space $𝒱=(V,S,+,\cdot )$ is a linear functional if for any two vectors $𝒖,𝒗\in V$ and any two scalars $a,b$

1.4.Linear mappings

Consider now functions $𝒇:V\to W$ from vector space $𝒱=(V,S,+,\cdot )$ to another vector space $𝒲=(W,T,+,\cdot )$. As before, the action of such functions on linear combinations is of special interest.

Definition. (Linear Mapping) . A function $𝒇:V\to W$, from vector space $𝒱=(V,S,+,\cdot )$ to vector space $𝒲=(W,S,+,\cdot )$ is called a linear mapping if for any two vectors $𝒖,𝒗\in V$ and any two scalars $a,b\in S$

The image of a linear combination $a𝒖+b𝒗$ through a linear mapping is another linear combination $a𝒇\left(𝒖\right)+b𝒇\left(𝒗\right)$, and linear mappings are said to preserve the structure of a vector space.

Note that $f:ℝ\to ℝ$ defined as $y=f\left(x\right)=ax+b$ represents a line in the $(x,y)$-plane, but is not a linear mapping for $b\ne 0$ since

Matrix-vector multiplication has been introduced as a concise way to specify a linear combination

with ${𝒂}_1,\dots ,{𝒂}_n$ the columns of the matrix, $𝑨=\left[\begin{array}{llll}{𝒂}_1& {𝒂}_2& \dots & {𝒂}_n\end{array}\right]$. This is a linear mapping between the real spaces ${ℛ}_m$, ${ℛ}_n$, $𝒇:{ℝ}^m\to {ℝ}^n$, and indeed any linear mapping between real spaces can be given as a matrix-vector product. Consider some $𝒙\in {ℝ}^m$

Applying the linear mapping $𝒇$ to $𝒙$ leads to

The matrix $𝑨$ with columns ${𝒂}_1=𝒇\left({𝒆}_1\right),\dots ,{𝒂}_m=𝒇\left({𝒆}_m\right)$ now allows finding

through a matrix-vector multiplication for any input vector $𝒙$. The matrix $𝑨$ thus defined is a representation of the linear mapping $𝒇$. As will be shown later, it is not the only possible representation.

2.Measurements

Vectors within the real space ${ℛ}_m$ can be completely specified by $m$ real numbers, and $m$ is large in many realistic applications. The task of describing the elements of a vector space $𝒱=(V,S,+,\cdot )$ by simpler means arises. Within data science this leads to classification problems in accordance with some relevant criteria, and one of the simplest classifications is to attach a scalar label to a vector. Commonly encountered labels include the magnitude of a vector or its orientation with respect to another vector.

2.1.Norms

The above observations lead to the mathematical concept of a norm as a tool to evaluate vector magnitude. Recall that a vector space is specified by two sets and two operations, $𝒱=(V,S,+,\cdot )$, and the behavior of a norm with respect to each of these components must be defined. The desired behavior includes the following properties and formal definition.

Unique value

The magnitude of a vector $𝒗\in V$ should be a unique scalar, requiring the definition of a function. The scalar could have irrational values and should allow ordering of vectors by size, so the function should be from $V$ to $ℝ$, $f:V\to ℝ$. On the real line the point at coordinate $x$ is at distance $\left|x\right|$ from the origin, and to mimic this usage the norm of $𝒗\in V$ is denoted as $||𝒗||$, leading to the definition of a function $||||:V\to {ℝ}_{+}$, ${ℝ}_{+}=\left\{a|a\in ℝ,a⩾0.\right\}$.

Null vector case

Provision must be made for the only distinguished element of $V$, the null vector $𝟎$. It is natural to associate the null vector with the null scalar element, $||𝟎||=0$. A crucial additional property is also imposed namely that the null vector is the only vector whose norm is zero, $||𝒗||=0\Rightarrow 𝒗=𝟎$. From knowledge of a single scalar value, an entire vector can be determined. This property arises at key junctures in linear algebra, notably in providing a link to mathematical analysis, and is needed to establish the fundamental theorem of linear algbera or the singular value decomposition encountered later.

Scaling

Transfer of the scaling operation $𝒗=a𝒖$ property leads to imposing $||𝒗||=\left|a\right|||𝒖||$. This property ensures commensurability of vectors, meaning that the magnitude of vector $𝒗$ can be expressed as a multiple of some standard vector magnitude $||𝒖||$.

Vector addition

Position vectors from the origin to coordinates $x,y>0$ on the real line can be added and $|x+y|=\left|x\right|+\left|y\right|$. If however the position vectors point in different directions, $x>0$, $y<0$, then $|x+y|<\left|x\right|+\left|y\right|$. For a general vector space the analogous property is known as the triangle inequality, $||𝒖+𝒗||⩽||𝒖||+||𝒗||$ for $𝒖,𝒗\in V$.

Definition. (Norm) . A norm on the vector space $𝒱=(V,S,+,\cdot )$ is a function $||||:V\to {ℝ}_{+}$ that for $𝒖,𝒗\in V$, $a\in S$ satisfies:

A commonly encountered norm of $𝒗\in {ℝ}^m$ is the Euclidean norm

useful in many physics applications. The form of the above norm, square root of sum of squares of components, can be generalized to obtain other useful norms.

Definition. ($p$-Norm in ${ℛ}_m$) . The $p$-norm on the real vector space ${ℛ}_m=({ℝ}^m,ℝ,+,\cdot )$ for $p⩾1$ is the function ${||||}_p:V\to {ℝ}_{+}$ with values ${||𝒙||}_p={({\left|x_1\right|}^p+{\left|x_2\right|}^p+\cdots +{\left|x_m\right|}^p)}^{1/p}$, or

Note that the Euclidean norm corresponds to $p=2$, and is often called the $2$-norm. Denote by $x_i$ the largest component in absolute value of $𝒙\in {ℝ}^m$. As $p$ increases, ${\left|x_i\right|}^p$ becomes dominant with respect to all other terms in the sum suggesting the definition of an inf-norm by

This also works for vectors with equal components, since the fact that the number of components is finite while $p\to \infty$ can be used as exemplified for $𝒙={\left[\begin{array}{cccc}a& a& \dots & a\end{array}\right]}^T$, by ${||𝒙||}_p={(m{\left|a\right|}^p)}^{1/p}=m^{1/p}\left|a\right|$, with $m^{1/p}\to 1$.

Figure 1. Regions within ${ℝ}^2$ for which ${||𝒙||}_p⩽1$, for $p=1,2,3,\infty$.

Vector norms arise very often in applications, especially in data science since they can be used to classify data, and are implemented in software systems such as Julia in which the norm function with a single argument computes the most commonly encountered norm, the $2$-norm. If a second argument $p$ is specified the $p$-norm is computed.

∴ 

x=[1; 1; 1]; [norm(x) sqrt(3)]

∴ 

m=9; x=ones(m,1); [norm(x) sqrt(m)]

∴ 

m=4; x=ones(m,1); [norm(x,1) m]

∴ 

2.2.Inner product

Norms are functionals that define what is meant by the size of a vector, but are not linear. Even in the simplest case of the real line, the linearity relation $|x+y|=\left|x\right|+\left|y\right|$ is not verified for $x>0$, $y<0$. Nor do norms characterize the familiar geometric concept of orientation of a vector. A particularly important orientation from Euclidean geometry is orthogonality between two vectors. Another function is required, one that would take two vector arguments to enable characterizing their relative orientation. It would return a scalar, hence $s:V\times V\to S$, with $S$ often chosen as the set of real numbers.

Definition. (Inner Product) . A real inner product in the vector space $𝒱=(V,ℝ,+,\cdot )$ is a function $s:V\times V\to ℝ$ with properties

A commonly encountered inner product is the dot product of two vectors $𝒂,𝒙\in {ℝ}^m$

Using the convention of representing $𝒂,𝒙$ as column vectors, the dot product is also expressed as

and is therefore a matrix multiplication between ${𝒂}^T\in {ℝ}^{1\times m}$ and $𝒙\in {ℝ}^{m\times 1}$ resulting in a scalar, also referred to as a scalar product. Inner products also provide a procedure to evaluate geometrical quantities and relationships.

Vector norm

The square of the 2-norm of $𝒙\in {ℝ}^m$ is given as

${\displaystyle s(𝒙,𝒙)={𝒙}^T𝒙={||𝒙||}_2^2.}$

In general, the square root of $s(𝒙,𝒙)$ satisfies the properties of a norm, and is called the norm induced by an inner product

${\displaystyle ||𝒙||=s{(𝒙,𝒙)}^{1/2}.}$

A real space together with the scalar product $s(𝒙,𝒚)={𝒙}^T𝒚$ and induced norm $||𝒙||=s{(𝒙,𝒙)}^{1/2}$ defines an Euclidean vector space ${ℰ}_m$.

Orientation

In ${ℰ}_2$ the point specified by polar coordinates $(r,\theta )$ has the Cartesian coordinates $x_1=r\cos \theta$, $x_2=r\sin \theta$, and position vector $𝒙={\left[\begin{array}{cc}{x}_1& x_2\end{array}\right]}^T$. The inner product

${\displaystyle {𝒆}_1^T𝒙=\left[\begin{array}{cc}1& 0\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right]=1\cdot x_1+0\cdot x_2=r\cos \theta ,}$

is seen to contain information on the relative orientation of $𝒙$ with respect to ${𝒆}_1$. In general, the angle $\theta$ between two vectors $𝒙,𝒚$ with any vector space with a scalar product can be defined by

${\displaystyle \cos \theta =\frac{s(𝒙,𝒚)}{{[s(𝒙,𝒙)s(𝒚,𝒚)]}^{1/2}}=\frac{s(𝒙,𝒚)}{||𝒙||||𝒚||},}$

which becomes

${\displaystyle \cos \theta =\frac{{𝒙}^T𝒚}{||𝒙||||𝒚||},}$

in a Euclidean space, $𝒙,𝒚\in {ℝ}^m$.

Orthogonality

In ${ℰ}_2$ two vectors are orthogonal if the angle between them is such that $\cos \theta =0$, and this can be extended to an arbitrary vector space $𝒱=(V,ℝ,+,\cdot )$ with a scalar product by stating that $𝒙,𝒚\in V$ are orthogonal if $s(𝒙,𝒚)=0$. In ${ℰ}_m$ vectors $𝒙,𝒚\in {ℝ}^m$ are orthogonal if ${𝒙}^T𝒚=0$.

3.Linear mapping matrices

3.1.Common geometric transformations

Several geometric transformations are linear mappings and are widely used in applications.

Stretching.

The matrix associated with stretching is

and has a remarkably simple form known as a diagonal matrix.

Projection.

The resulting vector is $𝒘=p_q\left(𝒗\right)\in {ℝ}^m$ has the same number of components as $𝒗$, and is of length $||𝒗||\cos \theta$ in the direction of $𝒒$, stated as

The matrix associated with projection is

The $j^{\mathrm{th}}$ unit vector ${𝒆}_j$ is at angle ${\theta }_j$ with respect to $𝒒$,

and the $$projection of ${𝒆}_j$ along $𝒒$ is

Gathering projections of all the unit vectors within the identity matrix gives

Note that ${𝑷}_{𝒒}$ contains $m$ column vectors that are all scalings of the $𝒒$ vector, by coefficients $q_1,q_2,\dots ,q_m$ that are the components of $𝒒$ itself. Since scaling is a linear combination, the above $m$ linear combinations can be expressed as a matrix-matrix product

leading to the remarkably simple expression

Figure 2. Projection ($p_{𝒒}$) and reflection ($r_{𝒒}$) operations in two dimensions

Reflection.

that can be interpreted as stating that the projection $𝒘$ is obtained from $𝒗$ by addition of the vector $𝒚$. The reflection of $𝒘$ across $𝒒$ is obtained by starting at $𝒗$, and adding $2𝒚$,

The reflection matrix results

Rotation in ${ℝ}^2$.

Figure 3. Rotation $r_{\theta }\left(𝒗\right)={𝑹}_{\theta }𝒗$ in two dimensions.

Rotation in ${ℝ}^3$.

3.2.Matrix-matrix product

From two functions $f:A\to B$ and $g:B\to C$, a composite function, $h=g\circ f$, $h:A\to C$ is defined by

Consider linear mappings between Euclidean spaces $𝒇:{ℝ}^n\to {ℝ}^m$, $𝒈:{ℝ}^m\to {ℝ}^p$. Recall that linear mappings are expressed as matrix vector multiplications

The composite function $𝒉=𝒈\circ 𝒇$ is $𝒉:{ℝ}^n\to {ℝ}^p$, defined by

Note that the intemediate vector $𝒖=𝑨𝒙$ is subsequently multiplied by the matrix $𝑩$. The composite function $𝒉$ is itself a linear mapping

so it also can be expressed a matrix-vector multiplication

Using the above, $𝑪$ is defined as the product of matrix $𝑩$ with matrix $𝑨$

The columns of $𝑪$ can be determined from those of $𝑨$ by considering the action of $𝒉$ on the the column vectors of the identity matrix $𝑰=\left[\begin{array}{cccc}{𝒆}_1& {𝒆}_2& \dots & {𝒆}_n\end{array}\right]\in {ℝ}^{n\times n}$. First, note that

The above can be repeated for the matrix $𝑪=\left[\begin{array}{cccc}{𝒄}_1& {𝒄}_2& \dots & {𝒄}_n\end{array}\right]$ giving

Combining the above equations leads to ${𝒄}_j=𝑩{𝒂}_j$, or

From the above the matrix-matrix product $𝑪=𝑩𝑨$ is seen to simply be a grouping of all the products of $𝑩$ with the column vectors of $𝑨$,

The above results can readily be verified computationally.

∴ 

a1=[1; 2]; a2=[3; 4]; A=[a1 a2]

∴ 

b1=[-1; 1; 3]; b2=[2; -2; 3]; B=[b1 b2]

∴ 

C=B*A

∴ 

c1=B*a1; c2=B*a2; [c1 c2]

∴ 

3.3.Properties of the matrix-matrix product