Linear Discriminant Analysis (LDA) is a dimensionality reduction technique commonly used for supervised classification problems. The goal of LDA is to project the dataset onto a lower-dimensional space while maximizing the class separability.
LDA is very similar to Principal Component Analysis (PCA), but there are some important differences. PCA is an unsupervised algorithm, meaning it doesn't need class labels 
. PCA's goal is to find the principal components that maximize the variance in a dataset. LDA, on the other hand, is a supervised algorithm, which uses both the input data 
and the class labels to find linear discriminants that maximize the separation between multiple classes.
LDA can be performed in 5 steps:
- Compute the mean vectors for the different classes from the dataset.
- Compute the scatter matrices (in-between-class and within-class scatter matrices).
- Compute the eigenvectors and corresponding eigenvalues for the scatter matrices.
- Sort the eigenvectors by decreasing eigenvalues and choose k eigenvectors with the largest eigenvalues.
- Use this eigenvector matrix to transform the samples onto the new subspace.
First, calculate the mean vectors for all classes inside the dataset.
After calculating the mean vectors, the within-class and between-class scatter matrices can be calculated.
where 
is the scatter matrix for a specific class
and 
is the mean vector for that class
Alternativeley the class-covariance matrices can be used by adding the scaling factor 
to the within-class scatter matrix.
where 
is the overall mean, is the mean of the respective class, and 
is the sample size of that class.
Next, LDA solves the generalized eigenvalue problem for the matrix 
to obtain the linear discriminants.
After calculating the eigenvectors and eigenvalues, we sort the eigenvectors from highest to lowest depending on their corresponding eigenvalue and then choose the top 
eigenvectors, where is the number of dimensions we want to keep.
After selecting the eigenvectors, we can use the resulting 
-dimensional eigenvector matrix 
to transform data onto the new subspace via the following equation:
