# Chapter 6 Linear Algebra

## 6.1 Linear Algebra Concepts

*Linear transformations* and *change of basis* are widely used in statistics, for this
reason I briefly describe the definition of these concepts and how they are related.

### 6.1.1 Linear Transformation

Letting \(V\) and \(W\) be vector spaces, a function \(f: V \rightarrow W\) is a linear transformation if the additivity and scalar multiplication properties are hold for any two vectors \(\ve{u}, \ve{v} \in V\) and a constant \(c\): \[f(\ve{u}+\ve{v}) = f(\ve{u}) + f(\ve{v})\] \[f(c\ve{v}) = cf(\ve{v}).\]

This concept is more common to use when working with matrices. Considering the
vector spaces \(V \in \real^n\) and \(W \in \real^m\), a matrix \(\m{A}_{m \times n}\) and
the vector \(\ve{x} \in V\); then the function
\[f(\ve{x}) = \m{A}\ve{x}\]
is a linear
transformation \(V \in \real^n\) to \(W \in \real^m\) because it holds the properties
mentioned above. In this definition, although not mentioned, we are assuming that both
\(V\) and \(W\) are defined using the *standard basis* for \(\real^n\) and \(\real^m\)
respectively.

### 6.1.2 Change of Basis

Consider a vector \(\ve{u} \in \real^n\), it is implicitly defined using the
*standard basis* \(\{\ve{e}_1,\dots,\ve{e}_n\}\) for \(\real^n\), such as
\(\ve{u}=\sum_{i=1}^n u_i \ve{e}_i\). In a similar manner, this vector \(\ve{u}\) can
also be represented in vector spaces with different *basis*, this is called
*change of basis*. For example, consider
the vector space \(V \in \real^n\) with *basis* \(\{\ve{v}_1,\dots,\ve{v}_n\}\). Then, in
order to make the change of basis, it is required to find
\(\ve{u}_v=(u_{v_1},\dots,u_{v_n})^\tr\) such as
\[\ve{u} = \sum_{i=1}^n u_{v_i} \ve{v}_i = \m{V}\ve{u}_v,\]
where the \(n\times n\) matrix \(\m{V}=(\ve{v}_1,\dots,\ve{v}_n)\), hence the change from
the *standard basis* to the vector space \(V\) is
\[\ve{u}_v = \m{V}^{-1}\ve{u},\] while the change from the vector space \(V\) to the
*standard basis* is
\[\ve{u} = \m{V}\ve{u}_v.\]

Now, consider another vector space \(W \in \real^n\) with *basis*
\(\{\ve{w}_1,\dots,\ve{w}_n\}\), the vector \(\ve{u}_v\) defined on the space \(V\) can
also be defined on the space \(W\) as
\[\ve{u}_w = \m{W}^{-1}\m{V}\ve{u}_v\]
where the \(n\times n\) matrix \(\m{W}=(\ve{w}_1,\dots,\ve{w}_n)\);
similarly, the vector \(\ve{u}_w \in W\) can be defined on the space \(V\) as
\[\ve{u}_v = \m{V}^{-1}\m{W}\ve{u}_w.\]
It can be seen that in both cases, the
original vector is first transformed to the space vector with *standard basis*
(left-multiplying the basis matrix) and
then transformed to the desired vector space (left-multiplying the basis matrix
inverse ).

### 6.1.3 Change of Basis for Linear Transformations

Previously, we have presented a linear transformation
\(f(\ve{x})=\m{A}\ve{x}:\real^n\rightarrow\real^m\)
using *standard basis*. This transformation can also be represented from a vector space
\(V\) with basis \(\{\ve{v}_1,\dots,\ve{v}_n\}\) to a vector space \(W\) with basis
\(\{\ve{w}_1,\dots,\ve{w}_n\}\), then \(f': V \rightarrow W\) is defined as
\[f'(\ve{x}_v) = \m{W}^{-1}\m{A}\m{V}\ve{x}_v,\]
where the matrices \(\m{W}\) and \(\m{V}\) are the basis matrix of the vector spaces \(W\)
and \(V\) respectively. The matrix multiplication \(\m{W}^{-1}\m{A}\m{V}\) implies a
change of basis from to standard basis, the linear transformation using the
standard basis, and the change from the standard basis to the space \(W\). In cases
that \(V=W\), then the linear transformation is defined as
\[f'(\ve{x}_v) = \m{V}^{-1}\m{A}\m{V}\ve{x}_v.\]

### 6.1.4 Eigenvalues and Eigenvectors

*Eigenvalues* and *eigenvectors* are used in several concepts of statistical inference
and modelling. It can be useful for dimension reduction, decomposition of
variance-covariance matrices, so on. For this reason, we provide basic details about
eigenvectors and eigenvalues and their close relationship with linear
transformations.

#### 6.1.4.1 Definition

The eigenvector of a linear transformation \(\m{A}_{n\times n}\) is a non-zero vector \(\ve{v}\) such as the linear transformation of this vector is proportional to itself: \[\m{A}\ve{v} = \lambda \ve{v} \iff (\m{A}-\lambda\m{I})\ve{v} = \ve{0},\] where \(\lambda\) is the eigenvalue associated to the eigenvector \(\ve{v}\). The equation above has non-zero solution if and only if \[\det(\m{A}-\lambda\m{I}) = 0.\] Then, all the eigenvalues \(\lambda\) of \(\m{A}\) hold the condition above.

There is an equivalence between the linear transformation \(f(\ve{x}) = \m{A}\ve{x}\), and the eigenvalues \(\lambda_1, \lambda_2, \dots, \lambda_n\) and eigenvectors \(\ve{v}_1, \ve{v}_2, \dots, \ve{v}_n\) of itself. This relationship provide more useful interpretation of the eigenvalues and eigenvectors, we will use the change of basis concept to describe it.

#### 6.1.4.2 Eigendecomposition and geometric interpretation

Considering a vector space \(V\) with basis \(\{\ve{v}_1, \ve{v}_2, \dots, \ve{v}_n\}\),
any vector \(\ve{x} \in \real^n\) can be represented as \(\m{V}\ve{x}_v\),
where \(\ve{x}_v\) is the representation of \(\ve{x}\) using the matrix of basis
\(\m{V}=(\ve{v}_1, \dots, \ve{v}_n)\) of the vector space \(V\). Then, the linear
transformation can be expressed as
\[f(\ve{x}) = \m{A}\ve{x} = \m{A}\m{V}\ve{x}_v = \m{V}\m{D}\ve{x}_v,\]
where the diagonal matrix \(D=\diag{\lambda_1, \dots, \lambda_n}\) and the last
equivalence hold because \(\m{A}\ve{v}_i=\ve{v}_i\lambda_i\). Finally, expressing
\(\ve{x}_v\) in terms of the vector \(\ve{x}\) defined on the standard basis, we
obtain that
\[f(\ve{x}) = \m{V}\m{D}\m{V}^{-1}\ve{x},\]
the equality \(\m{A}=\m{V}\m{D}\m{V}^{-1}\) is called *eigendecomposition*.
Hence, the linear transformation is equivalent to the following:
change the basis of \(\ve{x}\) to the vector space \(V\)
, apply the diagonal linear transformation \(D\) and return to the space with
standard basis. Geometrically, you can think of
\(\{\ve{v}_1, \ve{v}_2, \dots, \ve{v}_n\}\) as the basis of vectorial space \(V\) where
the transformation \(\m{A}\) becomes only an scaling transformation \(\m{D}\) and the
eigenvalues \(\lambda_1, \lambda_2, \dots, \lambda_n\) are the scaling factor in
direction of the corresponding eigenvector \(\ve{v}_1, \ve{v}_2, \dots, \ve{v}_n\).

#### 6.1.4.3 Basis properties

There are certain properties the are useful for statistical modelling such as:

- Trace of \(\m{A}\) is equals to the sum of the eigenvalues.
- Determinant of \(\m{A}\) is equals to the sum of the eigenvalues.
- If \(\m{A}\) is symmetric, then all eigenvalues are real.
- If \(\m{A}\) is positive definite, then all eigenvalues are positive.

Note that, some of these properties can be explained using the *eigendecomposition*
\(\m{A} = \m{V}\m{D}\m{V}^{-1}\).

### 6.1.5 Cauchy–Schwartz inequality

\(|<u, v>| <= ||u|| * ||v||\)