Feature Engineering
Last updated: 1/9/2025
Feature engineering is the process of transforming raw data into features that are suitable for [[machine-learning]] models. In other words, it is the process of selecting, extracting, and transforming the most relevant features from the available data to build more accurate and efficient [[machine-learning]] models.
We engineer features to improve the performance of the [[machine-learning|ML]] model by giving it relevant input data.
Feature engineering and decreasing the amount of features also helps with improve the interpretability of the model.
Feature engineering has 5 main steps:
Feature Creation
Creating new features by leveraging domain specific knowledge or by observing patterns in the data.
Feature Transformation
Transforming the features into a more suitable representation for the [[machine-learning|ML]] model. Examples:
- Normalization (from 0 to 1, -1 to 1)
- Scaling (making it so that features have a similar scale, ex. have a standard deviation of 1)
- Encoding (one-hot encoding or label encoding).
Feature Extraction
This is creating new features from existing ones. Examples:
- Dimensionality reduction: reduce number of features by transforming them into a lower-dimensional space ([[pca]] and [[t-SNE]])
- Feature Combination: Combining two or more existing features to create a new one. For example, the interaction between two features.
- Feature Aggregation: Aggregating features to create a new one. For example, calculating the mean, sum, or count of a set of features.
- Feature Transformation: Transforming existing features into a new representation. For example, log transformation of a feature with a skewed distribution.
Feature Selection
Selecting the most relevant features to the dataset.
Examples:
- Filter Method: Use statistical methods to measure the relationship between the feature and the label. (Examples: correlation coefficient, chi-square test, and information gain)
- Wrapper Method: Evaluate many different models with different subsets of features and select the most important features. This is much better than the filter method but more computationally expensive.
- Embedded Method: Have the feature selection be a part of the training process where the model dynamically selects and decides the most important features for it.
Feature Scalling
Scaling the data so that each feature has an equal affect on the model.
Examples:
- Min-Max Scaling: Scale so that each feature has the same effect
- Standard Scaling: Rescale the data so that the mean is zero and standard deviation is 1. This makes the data unit-less and removes outliers
- Robust Scaling: Use the IQR's to calculate the median and set the median to zero. This makes it so that outliers don't have an effect on the scaling (such as for Min-Max scaling), but they are still present in the data.