Abstract 5

1.Linear Combinations 11

Vectors and Matrices 11

1.Numbers 11

1.1.Number sets 11

1.2.Quantities described by a single number 11

1.3.Quantities described by multiple numbers 12

2.Vectors 12

2.1.Vector spaces 12

2.2.Real vector space ${ℛ}_m$ 12

Column vectors. 12

Row vectors. 13

Compatible vectors. 14

2.3.Working with vectors 14

Ranges. 14

Visualization. 15

3.Matrices 15

3.1.Forming matrices 15

3.2.Identity matrix 16

4.Linear combinations 17

4.1.Linear combination as a matrix-vector product 17

Linear combination. 17

Matrix-vector product. 17

4.2.Linear algebra problem examples 17

Linear combinations in $E_2$. 17

5.Vectors and matrice in data science 18

Linear Mappings 19

1.Functions 19

1.1.Relations 19

Homogeneous relations. 21

1.2.Functions 21

1.3.Linear functionals 22

1.4.Linear mappings 23

2.Measurements 24

2.1.Equivalence classes 24

2.2.Norms 24

2.3.Inner product 27

3.Linear mapping composition 28

3.1.Matrix-matrix product 28

2.Vector Spaces 31

Formal Rules 31

1.Algebraic structures 31

1.1.Typical structures 31

Groups. 31

Rings. 31

Fields. 31

1.2.Vector subspaces 32

2.Vector subspaces of a linear mapping 34

Data Redundancy 37

1.Linear dependence 37

2.Basis and dimension 38

3.Dimension of matrix spaces 38

3.Fundamental Theorem of Linear Algebra 41

Data Information 41

1.Partition of linear mapping domain and codomain 41

Data Partitioning 43

1.Mappings as data 43

1.1.Vector spaces of mappings and matrix representations 43

1.2.Measurement of mappings 44

2.The Singular Value Decomposition (SVD) 46

2.1.Orthogonal matrices 46

2.2.Intrinsic basis of a linear mapping 47

2.3.SVD solution of linear algebra problems 50

Change of coordinates. 50

Best 2-norm approximation. 50

The pseudo-inverse. 50

4.Least Squares 51

Data Compression 51

1.Projection 51

2.Gram-Schmidt 53

3.$QR$ solution of linear algebra problems 55

3.1.Transformation of coordinates 55

3.2.General orthogonal bases 56

3.3.Least squares 58

Model Reduction 61

1.Projection of mappings 61

2.Reduced bases 62

2.1.Correlation matrices 62

Correlation coefficient. 62

Patterns in data. 63

5.Change of Basis 65

Data Transformation 65

1.Gaussian elimination and row echelon reduction 65

2.$LU$-factorization 66

2.1.Example for $m=3$ 66

2.2.General $m$ case 67

3.Inverse matrix 70

3.1.Gauss-Jordan algorithm 70

4.Determinants 71

4.1.Cross product 75

Data Efficiency 75

1.Krylov bases 75

2.Greedy approximation 76

6.Eigenproblems 77

Data Stability 77

1.The eigenvalue problem 77

2.Computation of the SVD 82

Data Resonance 82

1.Bases induced by eigenmodes 82

Several types of numbers have been introduced in mathematics to express different types of quantities, and the following will be used throughout this text:

The set of natural numbers, , infinite and countable, ;

The set of integers, , infinite and countable;

The set of rational numbers , infinite and countable;

The set of real numbers, infinite, not countable, can be ordered;

The set of complex numbers, , infinite, not countable, cannot be ordered.

These sets of numbers form a hierarchy, with . The size of a set of numbers is an important aspect of its utility in describing natural phenomena. The set has three elements, and its size is defined by the cardinal number, . The sets have an infinite number of elements, but the relation

${\displaystyle z=\{\begin{array}{ll}-n/2& \mathrm{for}n\mathrm{even}\\ (n+1)/2& \mathrm{for}n\mathrm{odd}\end{array}.}$

defines a one-to-one correspondence between and , so these sets are of the same size denoted by the transfinite number (aleph-zero). The rationals can also be placed into a one-to-one correspondence with , hence

${\displaystyle \left|\mathbb{N}\right|=\left|\mathbb{Z}\right|=\left|\mathbb{Q}\right|={\aleph }_0.}$

In contrast there is no one-to-one mapping of the reals to the naturals, and the cardinality of the reals is (Fraktur-script c). Intuitively, there are exponentially more reals than naturals, formally stated in set theory as .

The above numbers and their computer approximations are sufficient to describe many quantities encountered in applications. Typical examples include:

• the position of a point on the unit line segment , approximated by the floating point number , to within machine epsilon precision, ;

• the measure of resistance to change of the rate of motion known as mass, , ;

• the population of a large community expressed as a float , even though for a community of individuals the population is a natural number, as in “the population of the United States is , i.e., 328.2 million”.

In most disciplines, there is a particular interest in comparison of two quantities, and to facilitate such comparison a common reference is used known as a standard unit. For measurement of a length , the meter is a standard unit, as in the statement , that states that is obtained by taking the standard unit ten times, . The rules for carrying out such comparisons are part of the definition of real and rational numbers.

Other quantities require more than a single number. The distribution of population in the year 2000 among the alphabetically-ordered South American countries (Argentina, Bolivia,..,Venezuela) requires 12 numbers. These are placed together in a list known in mathematics as a tuple, in this case a 12-tuple , with the population of Argentina, that of Bolivia, and so on. An analogous 12-tuple can be formed from the South American populations in the year 2020, say . Note that it is difficult to ascribe meaning to apparently plausible expressions such as since, for instance, some people in the 2000 population are also in the 2020 population, and would be counted twice.

In contrast to the population 12-tuple example above, combining multiple numbers is well defined in operations such as specifying a position within a three-dimensional Cartesian grid, or determining the resultant of two forces in space. Both of these lead to the consideration of 3-tuples or triples such as the force . When combined with another force the resultant is . If the force is amplified by the scalar and the force is similarly scaled by , the resultant becomes

${\displaystyle \alpha (f_1,f_2,f_3)+\beta (g_1,g_2,g_3)=(\alpha f_1,\alpha f_2,\alpha f_3)+(\beta g_1,\beta g_2,\beta g_3)=(\alpha f_1+\beta g_1,\alpha f_2+\beta g_2,\alpha f_3+\beta g_3).}$

It is useful to distinguish tuples for which scaling and addition is well defined from simple lists of numbers. In fact, since the essential difference is the behavior with respect to scaling and addition, the focus should be on these operations rather than the elements of the tuple.

The above observations underlie the definition of a vector space by a set whose elements satisfy scaling and addition properties, denoted all together by the 4-tuple . The first element of the 4-tuple is a set whose elements are called vectors. The second element is a set of scalars, and the third is the vector addition operation. The last is the scaling operation, seen as multiplication of a vector by a scalar. Vector addition and scaling operations must satisfy rules suggested by positions or forces in three-dimensional space, which are listed in Table . In particular, a vector space requires definition of two distinguished elements: the zero vector , and the identity scalar element .

The definition of a vector space reflects everyday experience with vectors in Euclidean geometry, and it is common to refer to such vectors by descriptions in a Cartesian coordinate system. For example, a position vector within the plane can be referred through the pair of coordinates . This intuitive understanding can be made precise through the definition of a vector space , called the real 2-space. Vectors within are elements of , meaning that a vector is specified through two real numbers, . Addition of two vectors, , is defined by addition of coordinates . Scaling by scalar is defined by . Similarly, consideration of position vectors in three-dimensional space leads to the definition of the , or more generally a real -space , , .

Since the real spaces play such an important role in themselves and as a guide to other vector spaces, familiarity with vector operations in is necessary to fully appreciate the utility of linear algebra to a wide range of applications. Following the usage in geometry and physics, the real numbers that specify a vector are called the components of . The one-to-one correspondence between a vector and its components , is by convention taken to define an equality relationship,

with the components arranged vertically and enclosed in square brackets. Given two vectors , and a scalar , vector addition and scaling are defined in by real number addition and multiplication of components

The vector space is defined using the real numbers as the set of scalars, and constructing vectors by grouping together scalars, but this approach can be extended to any set of scalars , leading to the definition of the vector spaces . These will often be referred to as -vector space of scalars, signifying that the set of vectors is .

To aid in visual recognition of vectors, the following notation conventions are introduced:

vectors are denoted by lower-case bold Latin letters: ;

scalars are denoted by normal face Latin or Greek letters: ;

the components of a vector are denoted by the corresponding normal face with subscripts as in equation ();

related sets of vectors are denoted by indexed bold Latin letters: .

In Julia, successive components placed vertically are separated by a semicolon.

The equal sign in mathematics signifies a particular equivalence relationship. In computer systems such as Julia the equal sign has the different meaning of assignment, that is defining the label on the left side of the equal sign to be the expression on the right side. Subsequent invocation of the label returns the assigned object. Components of a vector are obtained by enclosing the index in parantheses.

Instead of the vertical placement or components into one column, the components of could have been placed horizontally in one row , that contains the same data, differently organized. By convention vertical placement of vector components is the preferred organization, and shall denote a column vector henceforth. A transpose operation denoted by a superscript is introduced to relate the two representations

${\displaystyle {𝒖}^T=\left[\begin{array}{lll}{u}_1& \dots & u_m\end{array}\right],}$

and is the notation used to denote a row vector. In Julia, horizontal placement of successive components in a row is denoted by a space.

Addition of real vectors defines another vector . The components of are the sums of the corresponding components of and , , for . Addition of vectors with different number of components is not defined, and attempting to add such vectors produces an error. Such vectors with different number of components are called incompatible, while vectors with the same number of components are said to be compatible. Scaling of by defines a vector , whose components are , for . Vector addition and scaling in Julia are defined using the and operators.

The vectors used in applications usually have a large number of components, , and it is important to become proficient in their manipulation. Previous examples defined vectors by explicit listing of their components. This is impractical for large , and support is provided for automated generation for often-encountered situations. First, observe that Table mentions one distinguished vector, the zero element that is a member of any vector space . The zero vector of a real vector space is a column vector with components, all of which are zero, and a mathematical convention for specifying this vector is . This notation specifies that transpose of the zero vector is the row vector with zero components, also written through explicit indexing of each component as , for . Keep in mind that the zero vector and the zero scalar are different mathematical objects. The ellipsis symbol in the mathematical notation is transcribed in Julia by the notion of a range, with 1:m denoting all the integers starting from to , organized as a row vector. The notation is extended to allow for strides different from one, and the mathematical ellipsis is denoted as m:-1:1. In general r:s:t denotes the set of numbers with , and real numbers and a natural number, , . If there is no natural number such that , an empty vector with no components is returned.

A component-by-component display of a vector becomes increasingly unwieldy as the number of components becomes large. For example, the numbers below seem an inefficient way to describe the sine function.

Indeed, such a piece-by-piece approach is not the way humans organize large amounts of information, preferring to conceptualize the data as some other entity: an image, a sound excerpt, a smell, a taste, a touch, a sense of balance, or relative position. All seven of these human senses will be shown to allow representation by linear algebra concepts, including representation by vectors.

The real numbers themselves form the vector space , as does any field of scalars, . Juxtaposition of real numbers has been seen to define the new vector space . This process of juxtaposition can be continued to form additional mathematical objects. A matrix is defined as a juxtaposition of compatible vectors. As an example, consider vectors within some vector space . Form a matrix by placing the vectors into a row,

To aid in visual recognition of a matrix, upper-case bold Latin letters will be used to denote matrices. The columns of a matrix will be denoted by the corresponding lower-case bold letter with a subscripted index as in equation (). Note that the number of columns in a matrix can be different from the number of components in each column, as would be the case for matrix from equation () when choosing vectors from, say, the real space , .

Vectors were seen to be useful juxtapositions of scalars that could describe quantities a single scalar could not: a position in space, a force in physics, or a sampled function graph. The crucial utility of matrices is their central role in providing a means to obtain new vectors from their column vectors, as suggested by experience with Euclidean spaces.

Consider first , the vector space of real numbers. A position vector on the real axis is specified by a single scalar component, , . Read this to mean that the position is obtained by traveling units from the origin at position vector . Look closely at what is meant by “unit” in this context. Since is a scalar, the mathematical expression has no meaning, as addition of a vector to a scalar has not been defined. Recall that scalars were introduced to capture the concept of scaling of a vector, so in the context of vector spaces they always appear as multiplying some vector. The correct mathematical description is , where is the unit vector . Taking the components leads to , where are the first (and in this case only) components of the vectors. Since , , , one obtains the identity .

Now consider , the vector space of positions in the plane. Repeating the above train of thought leads to the identification of two direction vectors and

${\displaystyle 𝒓=\left[\begin{array}{l}x\\ y\end{array}\right]=x\left[\begin{array}{l}1\\ 0\end{array}\right]+y\left[\begin{array}{l}0\\ 1\end{array}\right]=x{𝒆}_1+y{𝒆}_2,{𝒆}_1=\left[\begin{array}{l}1\\ 0\end{array}\right],{𝒆}_2=\left[\begin{array}{l}0\\ 1\end{array}\right].}$

Continuing the procees to gives

${\displaystyle 𝒙=\left[\begin{array}{l}{x}_1\\ x_2\\ ⋮\\ x_m\end{array}\right]=x_1{𝒆}_1+x_2{𝒆}_2+\cdots +x_m{𝒆}_m,{𝒆}_1=\left[\begin{array}{l}1\\ 0\\ ⋮\\ 0\\ 0\end{array}\right],{𝒆}_2=\left[\begin{array}{l}0\\ 1\\ ⋮\\ 0\\ 0\end{array}\right],\dots ,{𝒆}_m=\left[\begin{array}{l}0\\ 0\\ ⋮\\ 0\\ 1\end{array}\right].}$

For arbitrary , the components are now rather than the alphabetically ordered letters common for or . It is then consistent with the adopted notation convention to use to denote the position vector whose components are . The basic idea is the same as in the previous cases: to obtain a position vector scale direction by , by ,, by , and add the resulting vectors.

Juxtaposition of the vectors leads to the formation of a matrix of special utility known as the identity matrix

${\displaystyle 𝑰=\left[\begin{array}{llll}{𝒆}_1& {𝒆}_2& \dots & {𝒆}_m\end{array}\right].}$

The identity matrix is an example of a matrix in which the number of column vectors is equal to the number of components in each column vector . Such matrices with equal number of columns and rows are said to be square. Due to entrenched practice an exception to the notation convention is made and the identity matrix is denoted by , but its columns are denoted the indexed bold-face of a different lower-case letter, . If it becomes necessary to explicitly state the number of columns in , the notation is used to denote the identity matrix with columns, each with components.

The previous chapter focused on mathematical expression of the concept of quantification, the act of associating human observation with measurements, as a first step of scientific inquiry. Consideration of different types of quantities led to various types of numbers, vectors as groupings of numbers, and matrices as groupings of vectors. Symbols were introduced for these quantities along with some intial rules for manipulating such objects, laying the foundation for an algebra of vectors and matrices. Science seeks to not only observe, but to also explain, which now leads to additional operations for working with vectors and matrices that will define the framework of linear algebra.

Explanations within scientific inquiry are formulated as hypotheses, from which predictions are derived and tested. A widely applied mathematical transcription of this process is to organize hypotheses and predictions as two sets and , and then construct another set of all of the instances in which an element of is associated with an element in . The set of all possible instances of and , is the Cartesian product of with , denoted as , a construct already encountered in the definition of the real 2-space where . Typically, not all possible tuples are relevant leading to the following definition.

The key concept is that of associating an input with an output . Inverting the approach and associating an output to an input is also useful, leading to the definition of an inverse relation as , . Note that an inverse exists for any relation, and the inverse of an inverse is the original relation, . From the above, a relation is a triplet (a tuple with three elements), , that will often be referred to by just its last member .

Many types of relations are defined in mathematics and encountered in linear algebra, and establishing properties of specific relations is an important task within data science. A commonly encountered type of relationship is from a set onto itself, known as a homogeneous relation. For homogeneous relations , it is common to replace the set membership notation to state that is in relationship with , with a binary operator notation . Familiar examples include the equality and less than relationships between reals, , in which is replaced by , and is replaced by . The equality relationship is its own inverse, and the inverse of the less than relationship is the greater than relation , , .

Functions between sets and are a specific type of relationship that often arise in science. For a given input , theories that predict a single possible output are of particular scientific interest.

The above intuitive definition can be transcribed in precise mathematical terms as is a function if and implies . Since it's a particular kind of relation, a function is a triplet of sets , but with a special, common notation to denote the triplet by , with and the property that . The set is the domain and the set is the codomain of the function . The value from the domain is the argument of the function associated with the function value . The function value is said to be returned by evaluation . The previously defined relations are functions but is not. All relations can be inverted, and inversion of a function defines a new relation, but which might not itself be a function. For example the relation is a function, but its inverse is not.

Simple functions such as sin, cos, exp, log, are predefined in Octave, and when given a vector argument return the function applied to each vector component.

As seen previously, a Euclidean space can be used to suggest properties of more complex spaces such as the vector space of continuous functions . A construct that will be often used is to interpret a vector within as a function, since with components also defines a function , with values . As the number of components grows the function can provide better approximations of some continuous function through the function values at distinct sample points .

The above function examples are all defined on a domain of scalars or naturals and returned scalar values. Within linear algebra the particular interest is on functions defined on sets of vectors from some vector space that return either scalars , or vectors from some other vector space , . The codomain of a vector-valued function might be the same set of vectors as its domain, . The fundamental operation within linear algebra is the linear combination with , . A key aspect is to characterize how a function behaves when given a linear combination as its argument, for instance or

Consider first the case of a function defined on a set of vectors that returns a scalar value. These can be interpreted as labels attached to a vector, and are very often encountered in applications from natural phenomena or data analysis.

Many different functionals may be defined on a vector space , and an insightful alternative description is provided by considering the set of all linear functionals, that will be denoted as . These can be organized into another vector space with vector addition of linear functionals and scaling by defined by

As is often the case, the above abstract definition can better be understood by reference to the familiar case of Euclidean space. Consider , the set of vectors in the plane with the position vector from the origin to point in the plane with coordinates . One functional from the dual space is , i.e., taking the second coordinate of the position vector. The linearity property is readily verified. For , ,

${\displaystyle f_2(a𝒙+b𝒚)=a{x}_2+b{y}_2=a{f}_2\left(𝒙\right)+b{f}_2\left(𝒚\right).}$

Given some constant value , the curves within the plane defined by are called the contour lines or level sets of . Several contour lines and position vectors are shown in Figure . The utility of functionals and dual spaces can be shown by considering a simple example from physics. Assume that is the height above ground level and a vector is the displacement of a body of mass in a gravitational field. The mechanical work done to lift the body from ground level to height is with the gravitational acceleration. The mechanical work is the same for all displacements that satisfy the equation . The work expressed in units can be interpreted as the number of contour lines intersected by the displacement vector . This concept of duality between vectors and scalar-valued functionals arises throughout mathematics, the physical and social sciences and in data science. The term “duality” itself comes from geometry. A point in with coordinates can be defined either as the end-point of the position vector , or as the intersection of the contour lines of two funtionals and . Either geometric description works equally well in specifying the position of , so it might seem redundant to have two such procedures. It turns out though that many quantities of interest in applications can be defined through use of both descriptions, as shown in the computation of mechanical work in a gravitational field.

Consider now functions from vector space to another vector space . As before, the action of such functions on linear combinations is of special interest.

The image of a linear combination through a linear mapping is another linear combination , and linear mappings are said to preserve the structure of a vector space, and called homomorphisms in mathematics. The codomain of a linear mapping might be the same as the domain in which case the mapping is said to be an endomorphism.

Matrix-vector multiplication has been introduced as a concise way to specify a linear combination

${\displaystyle 𝒇\left(𝒙\right)=𝑨𝒙=x_1{𝒂}_1+\cdots +x_n{𝒂}_n,}$

with the columns of the matrix, . This is a linear mapping between the real spaces , , , and indeed any linear mapping between real spaces can be given as a matrix-vector product.

Vectors within the real space can be completely specified by real numbers, even though is large in many realistic applications. A vector within , i.e., a continuous function defined on the reals, cannot be so specified since it would require an infinite, non-countable listing of function values. In either case, the task of describing the elements of a vector space by simpler means arises. Within data science this leads to classification problems in accordance with some relevant criteria.

Many classification criteria are scalars, defined as a scalar-valued function on a vector space, . The most common criteria are inspired by experience with Euclidean space. In a Euclidean-Cartesian model of the geometry of a plane , a point is arbitrarily chosen to correspond to the zero vector , along with two preferred vectors grouped together into the identity matrix . The position of a point with respect to is given by the linear combination

${\displaystyle 𝒙=𝑰𝒙+𝟎=\left[\begin{array}{ll}{𝒆}_1& {𝒆}_2\end{array}\right]\left[\begin{array}{l}{x}_1\\ x_2\end{array}\right]=x_1{𝒆}_1+x_2{𝒆}_2.}$

Several possible classifications of points in the plane are depicted in Figure : lines, squares, circles. Intuitively, each choice separates the plane into subsets, and a given point in the plane belongs to just one in the chosen family of subsets. A more precise characterization is given by the concept of a partition of a set.

In precise mathematical terms, a partition of set is such that , for which . Since there is only one set ( signifies “exists and is unique”) to which some given belongs, the subsets of the partition are disjoint, . The subsets are labeled by within some index set . The index set might be a subset of the naturals, in which case the partition is countable, possibly finite. The partitions of the plane suggested by Figure are however indexed by a real-valued label, with .

A technique which is often used to generate a partition of a vector space is to define an equivalence relation between vectors, . For some element , the equivalence class of is defined as all vectors that are equivalent to , . The set of equivalence classes of is called the quotient set and denoted as , and the quotient set is a partition of . Figure depicts four different partitions of the plane. These can be interpreted geometrically, such as parallel lines or distance from the origin. With wider implications for linear algebra, the partitions can also be given in terms of classification criteria specified by functions.

The partition of by circles from Figure is familiar; the equivalence classes are sets of points whose position vector has the same size, , or is at the same distance from the origin. Note that familiarity with Euclidean geometry should not obscure the fact that some other concept of distance might be induced by the data. A simple example is statement of walking distance in terms of city blocks, in which the distance from a starting point to an address blocks east and blocks north is city blocks, not the Euclidean distance since one cannot walk through the buildings occupying a city block.

The above observations lead to the mathematical concept of a norm as a tool to evaluate vector magnitude. Recall that a vector space is specified by two sets and two operations, , and the behavior of a norm with respect to each of these components must be defined. The desired behavior includes the following properties and formal definition.

Unique value

The magnitude of a vector should be a unique scalar, requiring the definition of a function. The scalar could have irrational values and should allow ordering of vectors by size, so the function should be from to , . On the real line the point at coordinate is at distance from the origin, and to mimic this usage the norm of is denoted as , leading to the definition of a function , .

Null vector case

Provision must be made for the only distinguished element of , the null vector . It is natural to associate the null vector with the null scalar element, . A crucial additional property is also imposed namely that the null vector is the only vector whose norm is zero, . From knowledge of a single scalar value, an entire vector can be determined. This property arises at key junctures in linear algebra, notably in providing a link to another branch of mathematics known as analysis, and is needed to establish the fundamental theorem of linear algbera or the singular value decomposition encountered later.

Scaling

Transfer of the scaling operation property leads to imposing . This property ensures commensurability of vectors, meaning that the magnitude of vector can be expressed as a multiple of some standard vector magnitude .

Vector addition

Position vectors from the origin to coordinates on the real line can be added and . If however the position vectors point in different directions, , , then . For a general vector space the analogous property is known as the triangle inequality, for .

Note that the norm is a functional, but the triangle inequality implies that it is not generally a linear functional. Returning to Figure , consider the functions defined for through values

${\displaystyle f_1\left(𝒙\right)=\left|x_1\right|,f_2\left(𝒙\right)=\left|x_2\right|,f_3\left(𝒙\right)=\left|x_1\right|+\left|x_2\right|,f_4\left(𝒙\right)={({\left|x_1\right|}^2+{\left|x_2\right|}^2)}^{1/2}.}$

Sets of constant value of the above functions are also equivalence classes induced by the equivalence relations for .

1. , ;

2. , ;

3. , ;

4. , .

These equivalence classes correspond to the vertical lines, horizontal lines, squares, and circles of Figure . Not all of the functions are norms since is zero for the non-null vector , and is zero for the non-null vector . The functions and are indeed norms, and specific cases of the following general norm.

Denote by the largest component in absolute value of . As increases, becomes dominant with respect to all other terms in the sum suggesting the definition of an inf-norm by

${\displaystyle {||𝒙||}_{\infty }={max}_{1⩽i⩽m}\left|x_i\right|.}$

This also works for vectors with equal components, since the fact that the number of components is finite while can be used as exemplified for , by , with .

Note that the Euclidean norm corresponds to , and is often called the -norm. The analogy between vectors and functions can be exploited to also define a -norm for , the vector space of continuous functions defined on .

The integration operation can be intuitively interpreted as the value of the sum from equation () for very large and very closely spaced evaluation points of the function , for instance . An inf-norm can also be define for continuous functions by

${\displaystyle {||f||}_{\infty }={sup}_{x\in [a,b]}\left|f\left(x\right)\right|,}$

where sup, the supremum operation can be intuitively understood as the generalization of the max operation over the countable set to the uncountable set .

Vector norms arise very often in applications, especially in data science since they can be used to classify data, and are implemented in software systems such as Octave in which the norm function with a single argument computes the most commonly encountered norm, the -norm. If a second argument is specified the -norm is computed.

Norms are functionals that define what is meant by the size of a vector, but are not linear. Even in the simplest case of the real line, the linearity relation is not verified for , . Nor do norms characterize the familiar geometric concept of orientation of a vector. A particularly important orientation from Euclidean geometry is orthogonality between two vectors. Another function is required, but before a formal definition some intuitive understanding is sought by considering vectors and functionals in the plane, as depicted in Figure . Consider a position vector and the previously-encountered linear functionals

${\displaystyle f_1,f_2:{ℝ}^2\to ℝ,f_1\left(𝒙\right)=x_1,f_2\left(𝒙\right)=x_2.}$

The component of the vector can be thought of as the number of level sets of times it crosses; similarly for the component. A convenient labeling of level sets is by their normal vectors. The level sets of have normal , and those of have normal vector . Both of these can be thought of as matrices with two columns, each containing a single component. The products of these matrices with the vector gives the value of the functionals

${\displaystyle {𝒆}_1^T𝒙=\left[\begin{array}{ll}1& 0\end{array}\right]\left[\begin{array}{l}{x}_1\\ x_2\end{array}\right]=1\cdot x_1+0\cdot x_2=x_1=f_1\left(𝒙\right),}$ ${\displaystyle {𝒆}_2^T𝒙=\left[\begin{array}{ll}0& 1\end{array}\right]\left[\begin{array}{l}{x}_1\\ x_2\end{array}\right]=0\cdot x_1+1\cdot x_2=x_1=f_2\left(𝒙\right).}$

In general, any linear functional defined on the real space can be labeled by a vector

${\displaystyle {𝒂}^T=\left[\begin{array}{llll}{a}_1& a_2& \dots & a_m\end{array}\right],}$

and evaluated through the matrix-vector product . This suggests the definition of another function ,

${\displaystyle s(𝒂,𝒙)={𝒂}^T𝒙.}$

The function is called an inner product, has two vector arguments from which a matrix-vector product is formed and returns a scalar value, hence is also called a scalar product. The definition from an Euclidean space can be extended to general vector spaces. For now, consider the field of scalars to be the reals .

The inner product returns the number of level sets of the functional labeled by crossed by the vector , and this interpretation underlies many applications in the sciences as in the gravitational field example above. Inner products also provide a procedure to evaluate geometrical quantities and relationships.

Vector norm

In the number of level sets of the functional labeled by crossed by itself is identical to the square of the 2-norm

${\displaystyle s(𝒙,𝒙)={𝒙}^T𝒙={||𝒙||}_2^2.}$

In general, the square root of satisfies the properties of a norm, and is called the norm induced by an inner product

${\displaystyle ||𝒙||=s{(𝒙,𝒙)}^{1/2}.}$

A real space together with the scalar product and induced norm defines an Euclidean vector space .

Orientation

In the point specified by polar coordinates has the Cartesian coordinates , , and position vector . The inner product

${\displaystyle {𝒆}_1^T𝒙=\left[\begin{array}{ll}1& 0\end{array}\right]\left[\begin{array}{l}{x}_1\\ x_2\end{array}\right]=1\cdot x_1+0\cdot x_2=r\cos \theta ,}$

is seen to contain information on the relative orientation of with respect to . In general, the angle between two vectors with any vector space with a scalar product can be defined by

${\displaystyle \cos \theta =\frac{s(𝒙,𝒚)}{{[s(𝒙,𝒙)s(𝒚,𝒚)]}^{1/2}}=\frac{s(𝒙,𝒚)}{||𝒙||||𝒚||},}$

which becomes

${\displaystyle \cos \theta =\frac{{𝒙}^T𝒚}{||𝒙||||𝒚||},}$

in a Euclidean space, .

Orthogonality

In two vectors are orthogonal if the angle between them is such that , and this can be extended to an arbitrary vector space with a scalar product by stating that are orthogonal if . In vectors are orthogonal if .

From two functions and , a composite function, , is defined by

${\displaystyle h\left(x\right)=g\left(f\left(x\right)\right).}$

Consider linear mappings between Euclidean spaces , . Recall that linear mappings between Euclidean spaces are expressed as matrix vector multiplication

${\displaystyle 𝒇\left(𝒙\right)=𝑨𝒙,𝒈\left(𝒚\right)=𝑩𝒚,𝑨\in {ℝ}^{m\times n},𝑩\in {ℝ}^{p\times m}.}$

The composite function is , defined by

${\displaystyle 𝒉\left(𝒙\right)=𝒈\left(𝒇\left(𝒙\right)\right)=𝒈(𝑨𝒙)=𝑩𝑨𝒙.}$

Note that the intemediate vector is subsequently multiplied by the matrix . The composite function is itself a linear mapping

${\displaystyle 𝒉(a𝒙+b𝒚)=𝑩𝑨(a𝒙+b𝒚)=𝑩(a𝑨𝒙+b𝑨𝒚)=𝑩(a𝒖+b𝒗)=a𝑩𝒖+b𝑩𝒗=a𝑩𝑨𝒙+b𝑩𝑨𝒚=a𝒉\left(𝒙\right)+b𝒉\left(𝒚\right),}$

so it also can be expressed a matrix-vector multiplication

Using the above, is defined as the product of matrix with matrix

${\displaystyle 𝑪=𝑩𝑨.}$

The columns of can be determined from those of by considering the action of on the the column vectors of the identity matrix . First, note that

The above can be repeated for the matrix giving

Combining the above equations leads to , or

${\displaystyle 𝑪=\left[\begin{array}{llll}{𝒄}_1& {𝒄}_2& \dots & {𝒄}_n\end{array}\right]=𝑩\left[\begin{array}{llll}{𝒂}_1& {𝒂}_2& \dots & {𝒂}_n\end{array}\right].}$

From the above the matrix-matrix product is seen to simply be a grouping of all the products of with the column vectors of ,

${\displaystyle 𝑪=\left[\begin{array}{llll}{𝒄}_1& {𝒄}_2& \dots & {𝒄}_n\end{array}\right]=\left[𝑩\begin{array}{llll}{𝒂}_1& 𝑩{𝒂}_2& \dots & 𝑩{𝒂}_n\end{array}\right]}$

Matrix-vector and matrix-matrix products are implemented in Octave, the above results can readily be verified.

A vector space has been introduced as a 4-tuple with specific behavior of the vector addition and scaling operations. Arithmetic operations between scalars were implicitly assumed to be similar to those of the real numbers, but also must be specified to obtain a complete definition of a vector space. Algebra defines various structures that specify the behavior operations with objects. Knowledge of these structures is useful not only in linear algebra, but also in other mathematical approaches to data analysis such as topology or geometry.

A group is a 2-tuple containing a set and an operation with properties from Table . If , , the group is said to be commutative. Besides the familiar example of integers under addition , symmetry groups that specify spatial or functional relations are of particular interest. The rotations by or vertices of a square form a group.

A ring is a 3-tuple containing a set and two operations with properties from Table . As is often the case, a ring is more complex structure built up from simpler algebraic structures. With respect to addition a ring has the properties of a commutative group. Only associativity and existence of an identity element is imposed for multiplication. Matrix addition and multiplication has the structure of ring

A ring is a 3-tuple containing a set and two operations , each with properties of a commutative group, but with special behavior for the inverse of the null element. The multiplicative inverse is denoted as . Scalars in the definition of a vector space must satisfy the properties of a field. Since the operations are often understood from context a field might be referred to as the full , or, more concisely just through the set of elements as in the definition of a vector space.

Using the above definitions, a vector space can be described as a commutative group combined with a field that satisfies the scaling properties , , , , , for , .

For the simple scalar mapping , , the condition implies either that or . Note that can be understood as defining a zero mapping . Linear mappings between vector spaces, , can exhibit different behavior, and the condtion , might be satisfied for both , and . Analogous to the scalar case, can be understood as defining a zero mapping, .

In vector space , vectors related by a scaling operation, , , are said to be colinear, and are considered to contain redundant data. This can be restated as , from which it results that . Colinearity can be expressed only in terms of vector scaling, but other types of redundancy arise when also considering vector addition as expressed by the span of a vector set. Assuming that , then the strict inclusion relation holds. This strict inclusion expressed in terms of set concepts can be transcribed into an algebraic condition.

Introducing a matrix representation of the vectors

${\displaystyle 𝑨=\left[\begin{array}{llll}{𝒂}_1& {𝒂}_2& \dots & {𝒂}_n\end{array}\right];𝒙=\left[\begin{array}{l}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right]}$

allows restating linear dependence as the existence of a non-zero vector, , such that . Linear dependence can also be written as , or that one cannot deduce from the fact that the linear mapping attains a zero value that the argument itself is zero. The converse of this statement would be that the only way to ensure is for , or , leading to the concept of linear independence.

A partition of a set has been introduced as a collection of subsets such that any given element belongs to only one set in the partition. This is modified when applied to subspaces of a vector space, and a partition of a set of vectors is understood as a collection of subsets such that any vector except belongs to only one member of the partition.

Linear mappings between vector spaces can be represented by matrices with columns that are images of the columns of a basis of

${\displaystyle 𝑨=\left[\begin{array}{lll}𝒇\left({𝒖}_1\right)& 𝒇\left({𝒖}_2\right)& \dots \end{array}\right].}$

Consider the case of real finite-dimensional domain and co-domain, , in which case ,

${\displaystyle 𝑨=\left[\begin{array}{llll}𝒇\left({𝒆}_1\right)& 𝒇\left({𝒆}_2\right)& \dots & 𝒇\left({𝒆}_n\right)\end{array}\right]=\left[\begin{array}{llll}{𝒂}_1& {𝒂}_2& \dots & {𝒂}_n\end{array}\right].}$

The column space of is a vector subspace of the codomain, , but according to the definition of dimension if there remain non-zero vectors within the codomain that are outside the range of ,

${\displaystyle n<m\Rightarrow \exists 𝒗\in {ℝ}^m,𝒗\ne 𝟎,𝒗\notin C\left(𝑨\right).}$

All of the non-zero vectors in , namely the set of vectors orthogonal to all columns in fall into this category. The above considerations can be stated as

${\displaystyle \begin{array}{llll}C\left(𝑨\right)\le {ℝ}^m,& N\left({𝑨}^T\right)\le {ℝ}^m,& C\left(𝑨\right)\perp N\left({𝑨}^T\right)& C\left(𝑨\right)+N\left({𝑨}^T\right)\le {ℝ}^m\end{array}.}$

The question that arises is whether there remain any non-zero vectors in the codomain that are not part of or . The fundamental theorem of linear algebra states that there no such vectors, that is the orthogonal complement of , and their direct sum covers the entire codomain .

In the vector space the subspaces are said to be orthogonal complements is , and . When , the orthogonal complement of is denoted as , .