Bias-Variance Tradeoff
In the world of machine learning when discussing model prediction, it’s important to understand prediction errors (bias and variance). The prediction error for any machine learning algorithm can be broken down into three parts:
- Bias Error
- Variance Error
- Irreducible Error
Bias
Bias is the difference between the average prediction of a model and the correct true value which is being predicted. A model with high bias does not perform well on both the training and testing data due to oversimplification.
Variance
Variance is the variability of a model’s prediction for a given data point or value which signifies the spread of the data. A model with high variance will learn and get accustomed to the training data, but does not generalize well on unseen data. As a result, such models perform very well on the training data but has high error rates on unseen test data.
Irreducible Error
The irreducible error cannot be minimized regardless of which machine learning algorithm is used. It is the random “noise” error introduced from the chosen framing of the problem and may be caused by factors such as unkown variables which influence the mapping of the input variables to the output variable.
The Golden Ratio
For a model to get good predictions, you’ll need to find a balance between Bias and Variance which minimizes the total error.
Conclusion
The goal of any supervised machine learning algorithm is to achieve low bias and low variance. Which should result in a model that consistently achieves good prediction performance.
If only the optimal strategy was so easily achieved then there wouldn’t be a tradeoff!
There is no espacing the relationship between bias and variance in machine learning. It’s inevitable that:
Increasing the bias will decrease the variance
Increasing the variance will decrease the bias
Finding the balance (golden ratio) is key. In practice, we cannot calculate the real bias and variance error terms because we do not know the actual underlying target function. Nevertheless, as a framework, bias and variance provide the tools to understand the behavior of machine learning algorithms in the prusuit of predictive performance.
