Most scientific disciplines introduce an idea of the amount of some entity or property of interest. Furthermore, the amount is usually combined with the concept of a number, an abstraction of the observation that the two sets {Mary, Jane, Tom} and {apple, plum, cherry} seem quite different, but we can match one distinct person to one distinct fruit as in {Maryplum, Janeapple, Tomcherry}. In contrast we cannot do the same matching of distinct persons to a distinct color from the set {red, green}, and one of the colors must be shared between two persons. Formal definition of the concept of a number from the above observations is surprisingly difficult since it would be self-referential due to the apperance of the numbers “one” and “two”. Leaving this aside, the key concept is that of quantity of some property of interest that is expressed through a number. Several types of numbers have been introduced in mathematics to express different types of quantities, and the following will be used throughout this text:

The set of natural numbers, , infinite and countable, ;

The set of integers, , infinite and countable;

The set of rational numbers , infinite and countable;

The set of real numbers, infinite, not countable, can be ordered;

The set of complex numbers, , infinite, not countable, cannot be ordered.

A computer has a finite amount of memory, hence cannot represent all numbers, but rather subsets of the above sets. Furthermore, computers internally use binary numbers composed of binary digits, or bits. Many computer number types are defined for specific purposes, and are often encountered in applications such as image representation or digital data acquisition. Here are the main types.

Subsets of

The number types uint8, uint16, uint32, uint64 represent subsets of the natural numbers (unsigned integers) using 8, 16, 32, 64 bits respectively. An unsigned integer with bits can store a natural number in the range from to . Two arbitrary natural numbers, written as can be added and will give another natural number, . In contrast, addition of computer unsigned integers is only defined within the specific range to .

Subsets of

The number types int8, int16, int32, int64 represent subsets of the integers. One bit is used to store the sign of the number, so the subset of that can be represented is from to

Subsets of

Computers approximate the real numbers through the set of floating point numbers. Floating point numbers that use bits are known as single precision, while those that use are double precision. A floating point number is stored internally as where , are bits within the mantissa of length , and , are bits within the exponent, along with signs for each. The default number type is usually double precision, more concisely referred to double. Common constants such as , are predefined as double, can be truncated to single, and the number of displayed decimal digits is controlled by format. The function disp(x) displays its argument x.

The approximation of the reals by the floats is characterized by: realmax, the largest float, realmin the smallest positive float, and eps known as machine epsilon. Machine epsilon highlights the differences between floating point and real numbers since it is defined as the largest number that satisfies . If of course implies , but floating points exhibit “granularity”, in the sense that over a unit interval there are small steps that are indistinguishable from zero due to the finite number of bits available for a float. Machine epsilon is small, and floating point errors can usually be kept under control. Keep in mind that perfect accuracy is a mathematical abstraction, not encountered in nature. In fields as sociology or psychology 3 digits of accuracy are excellent, in mechanical engineering this might increase to 6 digits, or in electronic engineering to 8 digits. The most precisely known physical constant is the Rydberg constant known to 12 digits. The granularity of double precision expressed by machine epsilon is sufficient to represent natural phenomena.

Within the reals certain operations are undefined such as . Special float constants are defined to handle such situations: Inf is a float meant to represent infinity, and NaN (“not a number”) is meant to represent an undefinable result of an arithmetic operation.

Complex numbers are specified by two reals, in Cartesian form as , or in polar form as , , . The computer type complex is similarly defined from two floats and the additional constant I is defined to represent . Functions are available to obtain the real and imaginary parts within the Cartesian form, or the absolute value and argument of the polar form.

Care should be exercised about the cummulative effect of many floating point errors. For instance, in an “irrational” numerical investigation of Zeno's paradox, one might want to compare the distance traversed by step sizes that are scaled by starting from one to , traversed by step sizes scaled by starting from

${\displaystyle S_N=1+\frac{1}{\pi }+\frac{1}{{\pi }^2}+\cdots +\frac{1}{{\pi }^N},T_N=\frac{1}{{\pi }^N}+\frac{1}{{\pi }^{N-1}}+\cdots +1.}$

In the reals the above two expressions are equal, , but this is not verfied for all when using floating point numbers. Lists of the values , for the two orderings , and , can be generated and summed.

In the above numerical experiment a==b expresses an equality relationship which might evaluate as true denoted by 1, or false denoted by 0.

The above was called an “irrational” investigation since in Zeno's original paradox the scaling factor was 2 rather than , and due to the binary representation used by floats equality always holds.

The above numbers and their computer approximations are sufficient to describe many quantities encountered in applications. Typical examples include:

• the position of a point on the unit line segment , approximated by the floating point number , to within machine epsilon precision, ;

• the measure of resistance to change of the rate of motion known as mass, , ;

• the population of a large community expressed as a float , even though for a community of individuals the population is a natural number, as in “the population of the United States is , i.e., 328.2 million”.

In most disciplines, there is a particular interest in comparison of two quantities, and to facilitate such comparison a common reference is used known as a standard unit. For measurement of a length , the meter is a standard unit, as in the statement , that states that is obtained by taking the standard unit ten times, . The rules for carrying out such comparisons are part of the definition of real and rational numbers. These rules are formalized in the mathematical definition of a field presented in the next chapter. Quantities that obey such rules, i.e., belong to a field, can be used in changes of scale and are called scalars. Not all numbers are scalars in this sense. For instance, the integers would not allow a scaling of 1:2 (halving the scale) even though 1,2 are integers.

Other quantities require more than a single number. The distribution of population in the year 2000 among the alphabetically-ordered South American countries (Argentina, Bolivia,..,Venezuela) requires 12 numbers. These are placed together in a list known in mathematics as a tuple, in this case a 12-tuple , with the population of Argentina, that of Bolivia, and so on. An analogous 12-tuple can be formed from the South American populations in the year 2020, say . Note that it is difficult to ascribe meaning to apparently plausible expressions such as since, for instance, some people in the 2000 population are also in the 2020 population, and would be counted twice.

In contrast to the population 12-tuple example above, combining multiple numbers is well defined in operations such as specifying a position within a three-dimensional Cartesian grid, or determining the resultant of two forces in space. Both of these lead to the consideration of 3-tuples or triples such as the force . When combined with another force the resultant is . If the force is amplified by the scalar and the force is similarly scaled by , the resultant becomes

${\displaystyle \alpha (f_1,f_2,f_3)+\beta (g_1,g_2,g_3)=(\alpha f_1,\alpha f_2,\alpha f_3)+(\beta g_1,\beta g_2,\beta g_3)=(\alpha f_1+\beta g_1,\alpha f_2+\beta g_2,\alpha f_3+\beta g_3).}$

It is useful to distinguish tuples for which scaling and addition is well defined from simple lists of numbers. In fact, since the essential difference is the behavior with respect to scaling and addition, the focus should be on these operations rather than the elements of the tuple.

The above observations underlie the definition of a vector space by a set whose elements satisfy certain scaling and addition properties, denoted all together by the 4-tuple . The first element of the 4-tuple is a set whose elements are called vectors. The second element is a set of scalars, and the third is the vector addition operation. The last is the scaling operation, seen as multiplication of a vector by a scalar. The vector addition and scaling operations must satisfy rules suggested by positions or forces in three-dimensional space, which are listed in Table . In particular, a vector space requires definition of two distinguished elements: the zero vector , and the identity scalar element .

The definition of a vector space reflects everyday experience with vectors in Euclidean geometry, and it is common to refer to such vectors by descriptions in a Cartesian coordinate system. For example, a position vector within the plane can be referred through the pair of coordinates . This intuitive understanding can be made precise through the definition of a vector space , called the real 2-space. Vectors within are elements of , meaning that a vector is specified through two real numbers, . Addition of two vectors, , is defined by addition of coordinates . Scaling by scalar is defined by . Similarly, consideration of position vectors in three-dimensional space leads to the definition of the , or more generally a real -space , , .

Note however that there is no mention of coordinates in the definition of a vector space as can be seen from the list of properties in Table . The intent of such a definition is to highlight that besides position vectors, many other mathematical objects follow the same rules. As an example, consider the set of all continuous functions , with function addition defined by the sum at each argument , and scaling by defined as . Read this as: “given two continuous functions and , the function is defined by stating that its value for argument is the sum of the two real numbers and ”. Similarly: “given a continuous function , the function is defined by stating that its value for argument is the product of the real numbers and ”. Under such definitions is a vector space, but quite different from . Nonetheless, the fact that both and are vector spaces can be used to obtain insight into the behavior of continuous functions from Euclidean vectors, and vice versa.

Since the real spaces play such an important role in themselves and as a guide to other vector spaces, familiarity with vector operations in is necessary to fully appreciate the utility of linear algebra to a wide range of applications. Following the usage in geometry and physics, the real numbers that specify a vector are called the components of . The one-to-one correspondence between a vector and its components , is by convention taken to define an equality relationship,

with the components arranged vertically and enclosed in square brackets. Given two vectors , and a scalar , vector addition and scaling are defined in by real number addition and multiplication of components

The vector space is defined using the real numbers as the set of scalars, and constructing vectors by grouping together scalars, but this approach can be extended to any set of scalars , leading to the definition of the vector spaces . These will often be referred to as -vector space of scalars, signifying that the set of vectors is .

To aid in visual recognition of vectors, the following notation conventions are introduced:

vectors are denoted by lower-case bold Latin letters: ;

scalars are denoted by normal face Latin or Greek letters: ;

the components of a vector are denoted by the corresponding normal face with subscripts as in equation ();

related sets of vectors are denoted by indexed bold Latin letters: .

In Octave, successive components placed vertically are separated by a semicolon.

The equal sign in mathematics signifies a particular equivalence relationship. In computer systems such as Octave the equal sign has the different meaning of assignment, that is defining the label on the left side of the equal sign to be the expression on the right side. Subsequent invocation of the label returns the assigned object. Components of a vector are obtained by enclosing the index in parantheses.

Instead of the vertical placement or components into one column, the components of could have been placed horizontally in one row , that contains the same data, differently organized. By convention vertical placement of vector components is the preferred organization, and shall denote a column vector henceforth. A transpose operation denoted by a superscript is introduced to relate the two representations

${\displaystyle {𝒖}^T=\left[\begin{array}{lll}{u}_1& \dots & u_m\end{array}\right],}$

and is the notation used to denote a row vector. In Octave, horizontal placement of successive components in a row is denoted by a space.

Addition of real vectors defines another vector . The components of are the sums of the corresponding components of and , , for . Addition of vectors with different number of components is not defined, and attempting to add such vectors produces an error. Such vectors with different number of components are called incompatible, while vectors with the same number of components are said to be compatible. Scaling of by defines a vector , whose components are , for . Vector addition and scaling in Octave are defined using the and operators.

The vectors used in applications usually have a large number of components, , and it is important to become proficient in their manipulation. Previous examples defined vectors by explicit listing of their components. This is impractical for large , and support is provided for automated generation for often-encountered situations. First, observe that Table mentions one distinguished vector, the zero element that is a member of any vector space . The zero vector of a real vector space is a column vector with components, all of which are zero, and a mathematical convention for specifying this vector is . This notation specifies that transpose of the zero vector is the row vector with zero components, also written through explicit indexing of each component as , for . Keep in mind that the zero vector and the zero scalar are different mathematical objects. The ellipsis symbol in the mathematical notation is transcribed in Octave by the notion of a range, with 1:m denoting all the integers starting from to , organized as a row vector. The notation is extended to allow for strides different from one, and the mathematical ellipsis is denoted as m:-1:1. In general r:s:t denotes the set of numbers with , and real numbers and a natural number, , . If there is no natural number such that , an empty vector with no components is returned.

An efficient, expressive feature of many software systems including Octave is to use ranges as indices to a vector, as shown below for the definition of . Note that the index range i is organized as a row, and a transpose operation must be applied to obtain z as a column vector.

A component-by-component display of a vector becomes increasingly unwieldy as the number of components becomes large. For example, the numbers below seem an inefficient way to describe the sine function.

Indeed, such a piece-by-piece approach is not the way humans organize large amounts of information, preferring to conceptualize the data as some other entity: an image, a sound excerpt, a smell, a taste, a touch, a sense of balance, or relative position. All seven of these human senses will be shown to allow representation by linear algebra concepts, including representation by vectors.

As a first example consider visualization, the process of transforming data into a sight perception. A familiar example is constructing a plot of the graph of a function. Recall that in mathematics the graph of a function relating elements of the domain to those in codomain is a set of ordered pairs . For a commonly encountered function such as , the graph contains an uncountably infinite number of elements, and obviously cannot be explicitly listed. The sine function is continuous, meaning that no matter how small an open interval within the function codomain one considers, there exists an interval in the function domain whose image by the sine function is contained in . In mathematical “” notation this is stated as: . This mathematical notation is concise and precise, but perceptive mainly to the professional mathematician. A more intuitive visualization of continuity is obtained by approximating the graph of a function by a finite set of samples, . Strictly speaking, the sampled graph would indicate jumps interpretable as discontinuities, but when plotting the points human sight perception conveys a sense of continuity for large sample sizes, . For the sine function example, consider sampling the domain with a step size , . To obtain a visual representation of the sampled sine function the Octave plot function can be used to produce a figure that will appear in another window, interactively investigated, and subsequently closed. For large one cannot visually distinguish the points in the graph sample, though this is apparent for smaller sample sizes. This is shown below by displaying a subrange of the sampled points with stride . This example also shows the procedure to save a permanent copy of the displayed figure through the Octave print -deps command that places the currently displayed plot into an Encapsulated Postscript file. The generated figure file can be linked to a document as shown here in Figure , in which both plots render samples of the graph of the sine function, but the one with large is perceived as being continuous.

The real numbers themselves form the vector space , as does any field of scalars, . Juxtaposition of real numbers has been seen to define the new vector space . This process of juxtaposition can be continued to form additional mathematical objects. A matrix is defined as a juxtaposition of compatible vectors. As an example, consider vectors within some vector space . Form a matrix by placing the vectors into a row,

To aid in visual recognition of a matrix, upper-case bold Latin letters will be used to denote matrices. The columns of a matrix will be denoted by the corresponding lower-case bold letter with a subscripted index as in equation (). Note that the number of columns in a matrix can be different from the number of components in each column, as would be the case for matrix from equation () when choosing vectors from, say, the real space , .

Vectors were seen to be useful juxtapositions of scalars that could describe quantities a single scalar could not: a position in space, a force in physics, or a sampled function graph. The crucial utility of matrices is their central role in providing a description of new vectors other then their column vectors, and is suggested by experience with Euclidean spaces.

Consider first , the vector space of real numbers. A position vector on the real axis is specified by a single scalar component, , . Read this to mean that the position is obtained by traveling units from the origin at position vector . Look closely at what is meant by “unit” in this context. Since is a scalar, the mathematical expression has no meaning, as addition of a vector to a scalar has not been defined. Recall that scalars were introduced to capture the concept of scaling of a vector, so in the context of vector spaces they always appear as multiplying some vector. The correct mathematical description is , where is the unit vector . Taking the components leads to , where are the first (and in this case only) components of the vectors. Since , , , one obtains the identity .

Now consider , the vector space of positions in the plane. Repeating the above train of thought leads to the identification of two direction vectors and

${\displaystyle 𝒓=\left[\begin{array}{l}x\\ y\end{array}\right]=x\left[\begin{array}{l}1\\ 0\end{array}\right]+y\left[\begin{array}{l}0\\ 1\end{array}\right]=x{𝒆}_1+y{𝒆}_2,{𝒆}_1=\left[\begin{array}{l}1\\ 0\end{array}\right],{𝒆}_2=\left[\begin{array}{l}0\\ 1\end{array}\right].}$

Continuing the procees to gives

${\displaystyle 𝒙=\left[\begin{array}{l}{x}_1\\ x_2\\ ⋮\\ x_m\end{array}\right]=x_1{𝒆}_1+x_2{𝒆}_2+\cdots +x_m{𝒆}_m,{𝒆}_1=\left[\begin{array}{l}1\\ 0\\ ⋮\\ 0\\ 0\end{array}\right],{𝒆}_2=\left[\begin{array}{l}0\\ 1\\ ⋮\\ 0\\ 0\end{array}\right],\dots ,{𝒆}_m=\left[\begin{array}{l}0\\ 0\\ ⋮\\ 0\\ 1\end{array}\right].}$

For arbitrary , the components are now rather than the alphabetically ordered letters common for or . It is then consistent with the adopted notation convention to use to denote the position vector whose components are . The basic idea is the same as in the previous cases: to obtain a position vector scale direction by , by ,, by , and add the resulting vectors.

Juxtaposition of the vectors leads to the formation of a matrix of special utility known as the identity matrix

${\displaystyle 𝑰=\left[\begin{array}{llll}{𝒆}_1& {𝒆}_2& \dots & {𝒆}_m\end{array}\right].}$

The identity matrix is an example of a matrix in which the number of column vectors is equal to the number of components in each column vector . Such matrices with equal number of columns and rows are said to be square. Due to entrenched practice an exception to the notation convention is made and the identity matrix is denoted by , but its columns are denoted the indexed bold-face of a different lower-case letter, . If it becomes necessary to explicitly state the number of columns in , the notation is used to denote the identity matrix with columns, each with components.

The previous chapter focused on mathematical expression of the concept of quantification, the act of associating human observation with measurements, as a first step of scientific inquiry. Consideration of different types of quantities led to various types of numbers, vectors as groupings of numbers, and matrices as groupings of vectors. Symbols were introduced for these quantities along with some intial rules for manipulating such objects, laying the foundation for an algebra of vectors and matrices. Science seeks to not only observe, but to also explain, which now leads to additional operations for working with vectors and matrices that will define the framework of linear algebra.

Explanations within scientific inquiry are formulated as hypotheses, from which predictions are derived and tested. A widely applied mathematical transcription of this process is to organize hypotheses and predictions as two sets and , and then construct another set of all of the instances in which an element of is associated with an element in . The set of all possible instances of and , is the Cartesian product of with , denoted as , a construct already encountered in the definition of the real 2-space where . Typically, not all possible tuples are relevant leading to the following definition.

Similar to the difficulties encountered in attempting rigorous definition of a natural number, careful parsing of the above definition also would reveal self-references since the member of symbol , and the subset of symbol are both themselves examples of relations. As before, this is set aside to concetrate on the key concept of associating an input with an output . Associating an output to an input is also useful, leading to the definition of an inverse relation as , . Note that an inverse exists for any relation, and the inverse of an inverse is the original relation, . From the above, a relation is a triplet (a tuple with three elements), , that will often be referred to by just its last member .

Computers can be programmed to work not only with numbers as Octave does, but also with general symbols as exemplified by another freely available software package called Maxima. Most data science applications involve numerical computation, but some knowledge of symbolic computation is also useful, as when working with sets that often arises in data classification. The colon symbol denotes assignment in Maxima, and sets can be defined using curly braces with automatic elimination of repeated elements. All common set manipulations are provided, such as the Cartesian product , and element of . operations.

Associate the first three Latin and Greek letters by defining . This is a relation between and since it is a subset of , which can be checked by defining a function that checks whether some is also an element of . Maxima functions are defined using the :=operator, and map applies a function to all elements of a set.

Suppose that the Greek alphabet ordering is not known, and might conceivably be associated to any of . This defines another relation . Finally consider possible reorderings of the Greek alphabet, formulated as relationships between and itself, with and two such possible reorderings. The relations defined here will be used to exemplify various properties below.

Many types of relations are defined in mathematics and encountered in linear algebra, and establishing properties of specific relations is an important task within data science. A commonly encountered type of relationship is from a set onto itself, known as a homogeneous relation. Among the above-defined relations are homogeneous, while are not. For homogeneous relations , it is common to replace the set membership notation to state that is in relationship with , with a binary operator notation . Familiar examples include the equality and less than relationships between reals, , in which is replaced by , and is replaced by . The equality relationship is its own inverse, and the inverse of the less than relationship is the greater than relation , , . Homogeneous relations are classified according to the following criteria.

Reflection

Relation is reflexive if for any . The equality relation is reflexive, , the less than relation is not, .

Symmetry

Relation is symmetric if implies that , . The equality relation is symmetric, , the less than relation is not, .

Anti-symmetry

Relation is anti-symmetric if for , then . The less than relation is antisymmetric, .

Transitivity

Relation is transitive if and implies . for any . The equality relation is transitive, , as is the less than relation , .

Certain combinations of properties often arise. A homogeneous relation that is reflexive, symmetric, and transitive is said to be an equivalence relation. Equivalence relations include equality among the reals, or congruence among triangles. A homogeneous relation that is reflexive, anti-symmetric and transitive is a partial order relation, such as the less than or equal relation between reals. Finally, a homogeneous relation that is anti-symmetric and transitive is an order relation, such as the less than relation between reals.

Functions between sets and are a specific type of relationship that often arise in science. For a given input , theories that predict a single possible output are of particular scientific interest.

The above intuitive definition can be transcribed in precise mathematical terms as is a function if and implies . Since it's a particular kind of relation, a function is a triplet of sets , but with a special, common notation to denote the triplet by , with and the property that . The set is the domain and the set is the codomain of the function . The value from the domain is the argument of the function associated with the function value . The function value is said to be returned by evaluation . The previously defined relations are functions but is not. All relations can be inverted, and inversion of a function defines a new relation, but which might not itself be a function. For example the relation is a function, but its inverse is not.

Simple functions such as sin, cos, exp, log, are predefined in Octave, and when given a vector argument return the function applied to each vector component.

As seen previously, a Euclidean space can be used to suggest properties of more complex spaces such as the vector space of continuous functions . A construct that will be often used is to interpret a vector within as a function, since with components also defines a function , with values . As the number of components grows the function can provide better approximations of some continuous function through the function values at distinct sample points .

The above function examples are all defined on a domain of scalars or naturals and returned scalar values. Within linear algebra the particular interest is on functions defined on sets of vectors from some vector space that return either scalars , or vectors from some other vector space , . The codomain of a vector-valued function might be the same set of vectors as its domain, . The fundamental operation within linear algebra is the linear combination with , . A key aspect is to characterize how a function behaves when given a linear combination as its argument, for instance or

Consider first the case of a function defined on a set of vectors that returns a scalar value. These can be interpreted as labels attached to a vector, and are very often encountered in applications from natural phenomena or data analysis.

Many different functionals may be defined on a vector space , and an insightful alternative description is provided by considering the set of all linear functionals, that will be denoted as . These can be organized into another vector space with vector addition of linear functionals and scaling by defined by

As is often the case, the above abstract definition can better be understood by reference to the familiar case of Euclidean space. Consider , the set of vectors in the plane with the position vector from the origin to point in the plane with coordinates . One functional from the dual space is , i.e., taking the second coordinate of the position vector. The linearity property is readily verified. For , ,

${\displaystyle f_2(a𝒙+b𝒚)=a{x}_2+b{y}_2=a{f}_2\left(𝒙\right)+b{f}_2\left(𝒚\right).}$

Given some constant value , the curves within the plane defined by are called the contour lines or level sets of . Several contour lines and position vectors are shown in Figure . The utility of functionals and dual spaces can be shown by considering a simple example from physics. Assume that is the height above ground level and a vector is the displacement of a body of mass in a gravitational field. The mechanical work done to lift the body from ground level to height is with the gravitational acceleration. The mechanical work is the same for all displacements that satisfy the equation . The work expressed in units can be interpreted as the number of contour lines intersected by the displacement vector . This concept of duality between vectors and scalar-valued functionals arises throughout mathematics, the physical and social sciences and in data science. The term “duality” itself comes from geometry. A point in with coordinates can be defined either as the end-point of the position vector , or as the intersection of the contour lines of two funtionals and . Either geometric description works equally well in specifying the position of , so it might seem redundant to have two such procedures. It turns out though that many quantities of interest in applications can be defined through use of both descriptions, as shown in the computation of mechanical work in a gravitational field.

Consider now functions from vector space to another vector space . As before, the action of such functions on linear combinations is of special interest.

The image of a linear combination through a linear mapping is another linear combination , and linear mappings are said to preserve the structure of a vector space, and called homomorphisms in mathematics. The codomain of a linear mapping might be the same as the domain in which case the mapping is said to be an endomorphism.

Matrix-vector multiplication has been introduced as a concise way to specify a linear combination

${\displaystyle 𝒇\left(𝒙\right)=𝑨𝒙=x_1{𝒂}_1+\cdots +x_n{𝒂}_n,}$

with the columns of the matrix, . This is a linear mapping between the real spaces , , , and indeed any linear mapping between real spaces can be given as a matrix-vector product.

Vectors within the real space can be completely specified by real numbers, even though is large in many realistic applications. A vector within , i.e., a continuous function defined on the reals, cannot be so specified since it would require an infinite, non-countable listing of function values. In either case, the task of describing the elements of a vector space by simpler means arises. Within data science this leads to classification problems in accordance with some relevant criteria.

Many classification criteria are scalars, defined as a scalar-valued function on a vector space, . The most common criteria are inspired by experience with Euclidean space. In a Euclidean-Cartesian model of the geometry of a plane , a point is arbitrarily chosen to correspond to the zero vector , along with two preferred vectors grouped together into the identity matrix . The position of a point with respect to is given by the linear combination

${\displaystyle 𝒙=𝑰𝒙+𝟎=\left[\begin{array}{ll}{𝒆}_1& {𝒆}_2\end{array}\right]\left[\begin{array}{l}{x}_1\\ x_2\end{array}\right]=x_1{𝒆}_1+x_2{𝒆}_2.}$

Several possible classifications of points in the plane are depicted in Figure : lines, squares, circles. Intuitively, each choice separates the plane into subsets, and a given point in the plane belongs to just one in the chosen family of subsets. A more precise characterization is given by the concept of a partition of a set.

In precise mathematical terms, a partition of set is such that , for which . Since there is only one set ( signifies “exists and is unique”) to which some given belongs, the subsets of the partition are disjoint, . The subsets are labeled by within some index set . The index set might be a subset of the naturals, in which case the partition is countable, possibly finite. The partitions of the plane suggested by Figure are however indexed by a real-valued label, with .

A technique which is often used to generate a partition of a vector space is to define an equivalence relation between vectors, . For some element , the equivalence class of is defined as all vectors that are equivalent to , . The set of equivalence classes of is called the quotient set and denoted as , and the quotient set is a partition of . Figure depicts four different partitions of the plane. These can be interpreted geometrically, such as parallel lines or distance from the origin. With wider implications for linear algebra, the partitions can also be given in terms of classification criteria specified by functions.

The partition of by circles from Figure is familiar; the equivalence classes are sets of points whose position vector has the same size, , or is at the same distance from the origin. Note that familiarity with Euclidean geometry should not obscure the fact that some other concept of distance might be induced by the data. A simple example is statement of walking distance in terms of city blocks, in which the distance from a starting point to an address blocks east and blocks north is city blocks, not the Euclidean distance since one cannot walk through the buildings occupying a city block.

The above observations lead to the mathematical concept of a norm as a tool to evaluate vector magnitude. Recall that a vector space is specified by two sets and two operations, , and the behavior of a norm with respect to each of these components must be defined. The desired behavior includes the following properties and formal definition.

Unique value

The magnitude of a vector should be a unique scalar, requiring the definition of a function. The scalar could have irrational values and should allow ordering of vectors by size, so the function should be from to , . On the real line the point at coordinate is at distance from the origin, and to mimic this usage the norm of is denoted as , leading to the definition of a function , .

Null vector case

Provision must be made for the only distinguished element of , the null vector . It is natural to associate the null vector with the null scalar element, . A crucial additional property is also imposed namely that the null vector is the only vector whose norm is zero, . From knowledge of a single scalar value, an entire vector can be determined. This property arises at key junctures in linear algebra, notably in providing a link to another branch of mathematics known as analysis, and is needed to establish the fundamental theorem of linear algbera or the singular value decomposition encountered later.

Scaling

Transfer of the scaling operation property leads to imposing . This property ensures commensurability of vectors, meaning that the magnitude of vector can be expressed as a multiple of some standard vector magnitude .

Vector addition

Position vectors from the origin to coordinates on the real line can be added and . If however the position vectors point in different directions, , , then . For a general vector space the analogous property is known as the triangle inequality, for .

Note that the norm is a functional, but the triangle inequality implies that it is not generally a linear functional. Returning to Figure , consider the functions defined for through values

${\displaystyle f_1\left(𝒙\right)=\left|x_1\right|,f_2\left(𝒙\right)=\left|x_2\right|,f_3\left(𝒙\right)=\left|x_1\right|+\left|x_2\right|,f_4\left(𝒙\right)={({\left|x_1\right|}^2+{\left|x_2\right|}^2)}^{1/2}.}$

Sets of constant value of the above functions are also equivalence classes induced by the equivalence relations for .

1. , ;

2. , ;

3. , ;

4. , .

These equivalence classes correspond to the vertical lines, horizontal lines, squares, and circles of Figure . Not all of the functions are norms since is zero for the non-null vector , and is zero for the non-null vector . The functions and are indeed norms, and specific cases of the following general norm.

Denote by the largest component in absolute value of . As increases, becomes dominant with respect to all other terms in the sum suggesting the definition of an inf-norm by

${\displaystyle {||𝒙||}_{\infty }={max}_{1⩽i⩽m}\left|x_i\right|.}$

This also works for vectors with equal components, since the fact that the number of components is finite while can be used as exemplified for , by , with .

Note that the Euclidean norm corresponds to , and is often called the -norm. The analogy between vectors and functions can be exploited to also define a -norm for , the vector space of continuous functions defined on .

The integration operation can be intuitively interpreted as the value of the sum from equation () for very large and very closely spaced evaluation points of the function , for instance . An inf-norm can also be define for continuous functions by

${\displaystyle {||f||}_{\infty }={sup}_{x\in [a,b]}\left|f\left(x\right)\right|,}$

where sup, the supremum operation can be intuitively understood as the generalization of the max operation over the countable set to the uncountable set .

Vector norms arise very often in applications, especially in data science since they can be used to classify data, and are implemented in software systems such as Octave in which the norm function with a single argument computes the most commonly encountered norm, the -norm. If a second argument is specified the -norm is computed.

Norms are functionals that define what is meant by the size of a vector, but are not linear. Even in the simplest case of the real line, the linearity relation is not verified for , . Nor do norms characterize the familiar geometric concept of orientation of a vector. A particularly important orientation from Euclidean geometry is orthogonality between two vectors. Another function is required, but before a formal definition some intuitive understanding is sought by considering vectors and functionals in the plane, as depicted in Figure . Consider a position vector and the previously-encountered linear functionals

${\displaystyle f_1,f_2:{ℝ}^2\to ℝ,f_1\left(𝒙\right)=x_1,f_2\left(𝒙\right)=x_2.}$

The component of the vector can be thought of as the number of level sets of times it crosses; similarly for the component. A convenient labeling of level sets is by their normal vectors. The level sets of have normal , and those of have normal vector . Both of these can be thought of as matrices with two columns, each containing a single component. The products of these matrices with the vector gives the value of the functionals

${\displaystyle {𝒆}_1^T𝒙=\left[\begin{array}{ll}1& 0\end{array}\right]\left[\begin{array}{l}{x}_1\\ x_2\end{array}\right]=1\cdot x_1+0\cdot x_2=x_1=f_1\left(𝒙\right),}$ ${\displaystyle {𝒆}_2^T𝒙=\left[\begin{array}{ll}0& 1\end{array}\right]\left[\begin{array}{l}{x}_1\\ x_2\end{array}\right]=0\cdot x_1+1\cdot x_2=x_1=f_2\left(𝒙\right).}$

In general, any linear functional defined on the real space can be labeled by a vector

${\displaystyle {𝒂}^T=\left[\begin{array}{llll}{a}_1& a_2& \dots & a_m\end{array}\right],}$

and evaluated through the matrix-vector product . This suggests the definition of another function ,

${\displaystyle s(𝒂,𝒙)={𝒂}^T𝒙.}$

The function is called an inner product, has two vector arguments from which a matrix-vector product is formed and returns a scalar value, hence is also called a scalar product. The definition from an Euclidean space can be extended to general vector spaces. For now, consider the field of scalars to be the reals .

The inner product returns the number of level sets of the functional labeled by crossed by the vector , and this interpretation underlies many applications in the sciences as in the gravitational field example above. Inner products also provide a procedure to evaluate geometrical quantities and relationships.

Vector norm

In the number of level sets of the functional labeled by crossed by itself is identical to the square of the 2-norm

${\displaystyle s(𝒙,𝒙)={𝒙}^T𝒙={||𝒙||}_2^2.}$

In general, the square root of satisfies the properties of a norm, and is called the norm induced by an inner product

${\displaystyle ||𝒙||=s{(𝒙,𝒙)}^{1/2}.}$

A real space together with the scalar product and induced norm defines an Euclidean vector space .

Orientation

In the point specified by polar coordinates has the Cartesian coordinates , , and position vector . The inner product

${\displaystyle {𝒆}_1^T𝒙=\left[\begin{array}{ll}1& 0\end{array}\right]\left[\begin{array}{l}{x}_1\\ x_2\end{array}\right]=1\cdot x_1+0\cdot x_2=r\cos \theta ,}$

is seen to contain information on the relative orientation of with respect to . In general, the angle between two vectors with any vector space with a scalar product can be defined by

${\displaystyle \cos \theta =\frac{s(𝒙,𝒚)}{{[s(𝒙,𝒙)s(𝒚,𝒚)]}^{1/2}}=\frac{s(𝒙,𝒚)}{||𝒙||||𝒚||},}$

which becomes

${\displaystyle \cos \theta =\frac{{𝒙}^T𝒚}{||𝒙||||𝒚||},}$

in a Euclidean space, .

Orthogonality

In two vectors are orthogonal if the angle between them is such that , and this can be extended to an arbitrary vector space with a scalar product by stating that are orthogonal if . In vectors are orthogonal if .

From two functions and , a composite function, , is defined by

${\displaystyle h\left(x\right)=g\left(f\left(x\right)\right).}$

Consider linear mappings between Euclidean spaces , . Recall that linear mappings between Euclidean spaces are expressed as matrix vector multiplication

${\displaystyle 𝒇\left(𝒙\right)=𝑨𝒙,𝒈\left(𝒚\right)=𝑩𝒚,𝑨\in {ℝ}^{m\times n},𝑩\in {ℝ}^{p\times m}.}$

The composite function is , defined by

${\displaystyle 𝒉\left(𝒙\right)=𝒈\left(𝒇\left(𝒙\right)\right)=𝒈(𝑨𝒙)=𝑩𝑨𝒙.}$

Note that the intemediate vector is subsequently multiplied by the matrix . The composite function is itself a linear mapping

${\displaystyle 𝒉(a𝒙+b𝒚)=𝑩𝑨(a𝒙+b𝒚)=𝑩(a𝑨𝒙+b𝑨𝒚)=𝑩(a𝒖+b𝒗)=a𝑩𝒖+b𝑩𝒗=a𝑩𝑨𝒙+b𝑩𝑨𝒚=a𝒉\left(𝒙\right)+b𝒉\left(𝒚\right),}$

so it also can be expressed a matrix-vector multiplication

Using the above, is defined as the product of matrix with matrix

${\displaystyle 𝑪=𝑩𝑨.}$

The columns of can be determined from those of by considering the action of on the the column vectors of the identity matrix . First, note that

The above can be repeated for the matrix giving

Combining the above equations leads to , or

${\displaystyle 𝑪=\left[\begin{array}{llll}{𝒄}_1& {𝒄}_2& \dots & {𝒄}_n\end{array}\right]=𝑩\left[\begin{array}{llll}{𝒂}_1& {𝒂}_2& \dots & {𝒂}_n\end{array}\right].}$

From the above the matrix-matrix product is seen to simply be a grouping of all the products of with the column vectors of ,

${\displaystyle 𝑪=\left[\begin{array}{llll}{𝒄}_1& {𝒄}_2& \dots & {𝒄}_n\end{array}\right]=\left[𝑩\begin{array}{llll}{𝒂}_1& 𝑩{𝒂}_2& \dots & 𝑩{𝒂}_n\end{array}\right]}$

Matrix-vector and matrix-matrix products are implemented in Octave, the above results can readily be verified.

A vector space has been introduced as a 4-tuple with specific behavior of the vector addition and scaling operations. Arithmetic operations between scalars were implicitly assumed to be similar to those of the real numbers, but also must be specified to obtain a complete definition of a vector space. Algebra defines various structures that specify the behavior operations with objects. Knowledge of these structures is useful not only in linear algebra, but also in other mathematical approaches to data analysis such as topology or geometry.

A group is a 2-tuple containing a set and an operation with properties from Table . If , , the group is said to be commutative. Besides the familiar example of integers under addition , symmetry groups that specify spatial or functional relations are of particular interest. The rotations by or vertices of a square form a group.

A ring is a 3-tuple containing a set and two operations with properties from Table . As is often the case, a ring is more complex structure built up from simpler algebraic structures. With respect to addition a ring has the properties of a commutative group. Only associativity and existence of an identity element is imposed for multiplication. Matrix addition and multiplication has the structure of ring

A ring is a 3-tuple containing a set and two operations , each with properties of a commutative group, but with special behavior for the inverse of the null element. The multiplicative inverse is denoted as . Scalars in the definition of a vector space must satisfy the properties of a field. Since the operations are often understood from context a field might be referred to as the full , or, more concisely just through the set of elements as in the definition of a vector space.

Using the above definitions, a vector space can be described as a commutative group combined with a field that satisfies the scaling properties , , , , , for , .

For the simple scalar mapping , , the condition implies either that or . Note that can be understood as defining a zero mapping . Linear mappings between vector spaces, , can exhibit different behavior, and the condtion , might be satisfied for both , and . Analogous to the scalar case, can be understood as defining a zero mapping, .

In vector space , vectors related by a scaling operation, , , are said to be colinear, and are considered to contain redundant data. This can be restated as , from which it results that . Colinearity can be expressed only in terms of vector scaling, but other types of redundancy arise when also considering vector addition as expressed by the span of a vector set. Assuming that , then the strict inclusion relation holds. This strict inclusion expressed in terms of set concepts can be transcribed into an algebraic condition.

Introducing a matrix representation of the vectors

${\displaystyle 𝑨=\left[\begin{array}{llll}{𝒂}_1& {𝒂}_2& \dots & {𝒂}_n\end{array}\right];𝒙=\left[\begin{array}{l}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right]}$

allows restating linear dependence as the existence of a non-zero vector, , such that . Linear dependence can also be written as , or that one cannot deduce from the fact that the linear mapping attains a zero value that the argument itself is zero. The converse of this statement would be that the only way to ensure is for , or , leading to the concept of linear independence.

A partition of a set has been introduced as a collection of subsets such that any given element belongs to only one set in the partition. This is modified when applied to subspaces of a vector space, and a partition of a set of vectors is understood as a collection of subsets such that any vector except belongs to only one member of the partition.

Linear mappings between vector spaces can be represented by matrices with columns that are images of the columns of a basis of

${\displaystyle 𝑨=\left[\begin{array}{lll}𝒇\left({𝒖}_1\right)& 𝒇\left({𝒖}_2\right)& \dots \end{array}\right].}$

Consider the case of real finite-dimensional domain and co-domain, , in which case ,

${\displaystyle 𝑨=\left[\begin{array}{llll}𝒇\left({𝒆}_1\right)& 𝒇\left({𝒆}_2\right)& \dots & 𝒇\left({𝒆}_n\right)\end{array}\right]=\left[\begin{array}{llll}{𝒂}_1& {𝒂}_2& \dots & {𝒂}_n\end{array}\right].}$

The column space of is a vector subspace of the codomain, , but according to the definition of dimension if there remain non-zero vectors within the codomain that are outside the range of ,

${\displaystyle n<m\Rightarrow \exists 𝒗\in {ℝ}^m,𝒗\ne 𝟎,𝒗\notin C\left(𝑨\right).}$

All of the non-zero vectors in , namely the set of vectors orthogonal to all columns in fall into this category. The above considerations can be stated as

${\displaystyle \begin{array}{llll}C\left(𝑨\right)\le {ℝ}^m,& N\left({𝑨}^T\right)\le {ℝ}^m,& C\left(𝑨\right)\perp N\left({𝑨}^T\right)& C\left(𝑨\right)+N\left({𝑨}^T\right)\le {ℝ}^m\end{array}.}$

The question that arises is whether there remain any non-zero vectors in the codomain that are not part of or . The fundamental theorem of linear algebra states that there no such vectors, that is the orthogonal complement of , and their direct sum covers the entire codomain .

In the vector space the subspaces are said to be orthogonal complements is , and . When , the orthogonal complement of is denoted as , .