Determinants

The Determinant is a special scalar value associated with every square matrix ($\mathbf{A}$). It provides crucial information about the matrix, particularly whether it can be inverted and its effect on geometric transformations.

1. Geometric Meaning

Conceptually, a matrix represents a linear transformation in space. The determinant of that matrix measures the scaling factor of the area (in 2D), volume (in 3D), or hyper-volume (in higher dimensions) caused by that transformation.

• If the determinant is $2$, the transformation doubles the area/volume.
• If the determinant is $0.5$, the transformation halves the area/volume.
• If the determinant is negative, the orientation of the space is flipped (like mirroring).

2. Calculation of the Determinant

The determinant of a matrix $\mathbf{A}$ is denoted as $\det(\mathbf{A})$ or $|\mathbf{A}|$.

A. $2 \times 2$ Matrix

For a $2 \times 2$ matrix, the determinant is calculated simply:

$$\mathbf{A} = \begin{bmatrix} a & b \\ c & d \end{bmatrix}$$ $$\det(\mathbf{A}) = |\mathbf{A}| = ad - bc$$

Example: 2x2

Let $\mathbf{A} = \begin{bmatrix} 4 & 1 \\ 2 & 3 \end{bmatrix}$.

$$\det(\mathbf{A}) = (4)(3) - (1)(2) = 12 - 2 = 10$$

B. $3 \times 3$ Matrix (Method of Cofactors)

For larger matrices, the calculation is more complex and typically involves the cofactor expansion method. This process recursively breaks down the determinant into determinants of smaller sub-matrices.

For a $3 \times 3$ matrix $\mathbf{A}$:

$$\mathbf{A} = \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \end{bmatrix}$$

Expanding along the first row:

$$\det(\mathbf{A}) = a_{11} \begin{vmatrix} a_{22} & a_{23} \\ a_{32} & a_{33} \end{vmatrix} - a_{12} \begin{vmatrix} a_{21} & a_{23} \\ a_{31} & a_{33} \end{vmatrix} + a_{13} \begin{vmatrix} a_{21} & a_{22} \\ a_{31} & a_{32} \end{vmatrix}$$

3. Determinants in Machine Learning: Invertibility

The most crucial role of the determinant in ML, particularly in classical linear models, is determining if a matrix is invertible.

A. Non-Singular (Invertible) Matrix

If $\det(\mathbf{A}) \ne 0$:

• The matrix $\mathbf{A}$ has an inverse ($\mathbf{A}^{-1}$).
• The system of linear equations represented by $\mathbf{A}\mathbf{x} = \mathbf{b}$ has a unique solution.

B. Singular (Non-Invertible) Matrix

If $\det(\mathbf{A}) = 0$:

• The matrix $\mathbf{A}$ is singular (or degenerate) and does not have an inverse.
• The transformation compresses the space into a lower dimension (e.g., a 3D volume collapses onto a 2D plane, hence the volume scaling factor is zero).
• The columns (or rows) of $\mathbf{A}$ are linearly dependent (redundant).

ML Application: Normal Equation

In Linear Regression, the closed-form solution (Normal Equation) is given by:

$$\mathbf{w} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$$

This equation requires the matrix $(\mathbf{X}^T\mathbf{X})$ to be invertible. If $\det(\mathbf{X}^T\mathbf{X}) = 0$, the inverse does not exist, and the Normal Equation cannot be solved directly. This often happens if features are perfectly correlated (multicollinearity). Regularization techniques (like Ridge Regression) are used precisely to make this matrix invertible by ensuring $\det(\mathbf{X}^T\mathbf{X}) \ne 0$.

4. Properties of the Determinant

1. Identity Matrix: $\det(\mathbf{I}) = 1$.
2. Transpose: The determinant of a matrix is equal to the determinant of its transpose: $\det(\mathbf{A}) = \det(\mathbf{A}^T)$.
3. Product: The determinant of a product of matrices is the product of their determinants: $\det(\mathbf{A}\mathbf{B}) = \det(\mathbf{A})\det(\mathbf{B})$.

---

The concept of invertibility, driven by the determinant, leads directly to the next crucial topic: finding the matrix inverse itself.