Chapter 6 Linear Algebra

6.1 Linear Algebra Concepts

Linear transformations and change of basis are widely used in statistics, for this reason I briefly describe the definition of these concepts and how they are related.

6.1.1 Linear Transformation

Letting $V$ and $W$ be vector spaces, a function $f: V \rightarrow W$ is a linear transformation if the additivity and scalar multiplication properties are hold for any two vectors $\ve{u}, \ve{v} \in V$ and a constant $c$: $$f(\ve{u}+\ve{v}) = f(\ve{u}) + f(\ve{v})$$ $$f(c\ve{v}) = cf(\ve{v}).$$

This concept is more common to use when working with matrices. Considering the vector spaces $V \in \real^n$ and $W \in \real^m$, a matrix $\m{A}_{m \times n}$ and the vector $\ve{x} \in V$; then the function $$f(\ve{x}) = \m{A}\ve{x}$$ is a linear transformation $V \in \real^n$ to $W \in \real^m$ because it holds the properties mentioned above. In this definition, although not mentioned, we are assuming that both $V$ and $W$ are defined using the standard basis for $\real^n$ and $\real^m$ respectively.

6.1.2 Change of Basis

Consider a vector $\ve{u} \in \real^n$, it is implicitly defined using the standard basis $\{\ve{e}_1,\dots,\ve{e}_n\}$ for $\real^n$, such as $\ve{u}=\sum_{i=1}^n u_i \ve{e}_i$. In a similar manner, this vector $\ve{u}$ can also be represented in vector spaces with different basis, this is called change of basis. For example, consider the vector space $V \in \real^n$ with basis $\{\ve{v}_1,\dots,\ve{v}_n\}$. Then, in order to make the change of basis, it is required to find $\ve{u}_v=(u_{v_1},\dots,u_{v_n})^\tr$ such as $$\ve{u} = \sum_{i=1}^n u_{v_i} \ve{v}_i = \m{V}\ve{u}_v,$$ where the $n\times n$ matrix $\m{V}=(\ve{v}_1,\dots,\ve{v}_n)$, hence the change from the standard basis to the vector space $V$ is $$\ve{u}_v = \m{V}^{-1}\ve{u},$$ while the change from the vector space $V$ to the standard basis is $$\ve{u} = \m{V}\ve{u}_v.$$

Now, consider another vector space $W \in \real^n$ with basis $\{\ve{w}_1,\dots,\ve{w}_n\}$, the vector $\ve{u}_v$ defined on the space $V$ can also be defined on the space $W$ as $$\ve{u}_w = \m{W}^{-1}\m{V}\ve{u}_v$$ where the $n\times n$ matrix $\m{W}=(\ve{w}_1,\dots,\ve{w}_n)$; similarly, the vector $\ve{u}_w \in W$ can be defined on the space $V$ as $$\ve{u}_v = \m{V}^{-1}\m{W}\ve{u}_w.$$ It can be seen that in both cases, the original vector is first transformed to the space vector with standard basis (left-multiplying the basis matrix) and then transformed to the desired vector space (left-multiplying the basis matrix inverse ).

6.1.3 Change of Basis for Linear Transformations

Previously, we have presented a linear transformation $f(\ve{x})=\m{A}\ve{x}:\real^n\rightarrow\real^m$ using standard basis. This transformation can also be represented from a vector space $V$ with basis $\{\ve{v}_1,\dots,\ve{v}_n\}$ to a vector space $W$ with basis $\{\ve{w}_1,\dots,\ve{w}_n\}$, then $f': V \rightarrow W$ is defined as $$f'(\ve{x}_v) = \m{W}^{-1}\m{A}\m{V}\ve{x}_v,$$ where the matrices $\m{W}$ and $\m{V}$ are the basis matrix of the vector spaces $W$ and $V$ respectively. The matrix multiplication $\m{W}^{-1}\m{A}\m{V}$ implies a change of basis from to standard basis, the linear transformation using the standard basis, and the change from the standard basis to the space $W$. In cases that $V=W$, then the linear transformation is defined as $$f'(\ve{x}_v) = \m{V}^{-1}\m{A}\m{V}\ve{x}_v.$$

6.1.4 Eigenvalues and Eigenvectors

Eigenvalues and eigenvectors are used in several concepts of statistical inference and modelling. It can be useful for dimension reduction, decomposition of variance-covariance matrices, so on. For this reason, we provide basic details about eigenvectors and eigenvalues and their close relationship with linear transformations.

6.1.4.1 Definition

The eigenvector of a linear transformation $\m{A}_{n\times n}$ is a non-zero vector $\ve{v}$ such as the linear transformation of this vector is proportional to itself: $$\m{A}\ve{v} = \lambda \ve{v} \iff (\m{A}-\lambda\m{I})\ve{v} = \ve{0},$$ where $\lambda$ is the eigenvalue associated to the eigenvector $\ve{v}$. The equation above has non-zero solution if and only if $$\det(\m{A}-\lambda\m{I}) = 0.$$ Then, all the eigenvalues $\lambda$ of $\m{A}$ hold the condition above.

There is an equivalence between the linear transformation $f(\ve{x}) = \m{A}\ve{x}$, and the eigenvalues $\lambda_1, \lambda_2, \dots, \lambda_n$ and eigenvectors $\ve{v}_1, \ve{v}_2, \dots, \ve{v}_n$ of itself. This relationship provide more useful interpretation of the eigenvalues and eigenvectors, we will use the change of basis concept to describe it.

6.1.4.2 Eigendecomposition and geometric interpretation

Considering a vector space $V$ with basis $\{\ve{v}_1, \ve{v}_2, \dots, \ve{v}_n\}$, any vector $\ve{x} \in \real^n$ can be represented as $\m{V}\ve{x}_v$, where $\ve{x}_v$ is the representation of $\ve{x}$ using the matrix of basis $\m{V}=(\ve{v}_1, \dots, \ve{v}_n)$ of the vector space $V$. Then, the linear transformation can be expressed as $$f(\ve{x}) = \m{A}\ve{x} = \m{A}\m{V}\ve{x}_v = \m{V}\m{D}\ve{x}_v,$$ where the diagonal matrix $D=\diag{\lambda_1, \dots, \lambda_n}$ and the last equivalence hold because $\m{A}\ve{v}_i=\ve{v}_i\lambda_i$. Finally, expressing $\ve{x}_v$ in terms of the vector $\ve{x}$ defined on the standard basis, we obtain that $$f(\ve{x}) = \m{V}\m{D}\m{V}^{-1}\ve{x},$$ the equality $\m{A}=\m{V}\m{D}\m{V}^{-1}$ is called eigendecomposition. Hence, the linear transformation is equivalent to the following: change the basis of $\ve{x}$ to the vector space $V$ , apply the diagonal linear transformation $D$ and return to the space with standard basis. Geometrically, you can think of $\{\ve{v}_1, \ve{v}_2, \dots, \ve{v}_n\}$ as the basis of vectorial space $V$ where the transformation $\m{A}$ becomes only an scaling transformation $\m{D}$ and the eigenvalues $\lambda_1, \lambda_2, \dots, \lambda_n$ are the scaling factor in direction of the corresponding eigenvector $\ve{v}_1, \ve{v}_2, \dots, \ve{v}_n$.

6.1.4.3 Basis properties

There are certain properties the are useful for statistical modelling such as:

• Trace of $\m{A}$ is equals to the sum of the eigenvalues.
• Determinant of $\m{A}$ is equals to the sum of the eigenvalues.
• If $\m{A}$ is symmetric, then all eigenvalues are real.
• If $\m{A}$ is positive definite, then all eigenvalues are positive.

Note that, some of these properties can be explained using the eigendecomposition $\m{A} = \m{V}\m{D}\m{V}^{-1}$.

6.1.5 Cauchy–Schwartz inequality

$|<u, v>| <= ||u|| * ||v||$