Kdtyjb

I have made some research about overfitting and underfitting, and I have understood what they exactly are, but I cannot find the reasons.

What are the main reasons for overfitting and underfitting?

Why do we face these two problems in training a model?

I'll try to answer in the simplest way. Each of those problems has its own main origin:

Overfitting: Data is noisy, meaning that there are some deviations from reality (because of measurement errors, random influentially factors, non-observed variables and rubbish correlations) that makes us harder to see their true relationship with our explaining factors. Also, it is usually not complete (we don't have examples of everything).

As an example, let's say I am trying to classify boys and girls based on their height, just because that's the only information I have about them. We all know that even though boys are taller on average than girls, there is a huge overlap region so it's impossible to perfectly separate them just with that bit of information. Depending on the density of the data, a sufficiently complex model might be able to achieve a better success rate on this task than is theoretically possible on the training dataset because it could draw boundaries that allow some points to stand alone by themselves. So, if we only have a person who is 2.04 meters tall and she's a woman, then the model could draw a little circle around that area meaning that a random person who is 2.04 meters tall is most likely to be a woman.

The underlying reason for it all is trusting too much in training data (and in the example, the model says that as there is no man with 2.04 height, then it is only possible for women).

Underfitting is the opposite problem, in which the model fails to recognize the real complexities in our data (i.e. the non-random changes in our data). The model assumes that noise is greater than it really is and thus uses a too simplistic shape. So, if the dataset has much more girls than boys for whatever reason, then the model could just classify them all like girls.

In this case, the model didn't trust enough in data and it just assumed that deviations are all noise (and in the example, the model assumes that boys simply do not exist).

Bottom line is that we face these problems because:

• We don't have complete information.


• We don't know how noisy the data is (we don't know how much should we trust it).


• We don't know in advance the underlying function that generated our data, and thus the optimal model complexity.

I hope this helps.

Overfitting is when a model estimates the variable you are modeling really well on the original data, but it does not estimate well on new data set (hold out, cross validation, forecasting, etc.). You have too many variables or estimators in your model (dummy variables, etc.) and these cause your model to become too sensitive to the noise in your original data. As a result of overfitting on the noise in your original data, the model predicts poorly.

Underfitting is when a model does not estimate the variable well in either the original data or new data. Your model is missing some variables that are necessary to better estimate and predict the behavior of your dependent variable.

The balancing act between over and underfitting is challenging and sometimes without a clear finish line. In modeling econometrics time series, this issue is resolved pretty well with regularization models (LASSO, Ridge Regression, Elastic-Net) that are catered specifically to reducing overfitting by respectively reducing the number of variables in your model, reducing the sensitivity of the coefficients to your data, or a combination of both.

Perhaps during your research you came across the following equation:

Error = IrreducibleError + Bias² + Variance.

Why do we face these two problems in training a model ?

The learning problem itself is basically a trade-off between bias and variance.

What are the main reasons for overfitting and underfitting ?

Short: Noise.

Long: The irreducible error: Measurement errors/fluctuations in the data as well as the part of the target function that cannot be represented by the model. Remeasuring the target variable or changing the hypothesis space (i.e. selecting a different model) changes this component.

Edit (to link to the other answers): Model performance as complexity is varied:

where errorD is the error over the entire distribution D (in practice estimated with test sets).

What are the main reasons for overfitting and underfitting ?

For overfitting, the model is too complex to fit the training data well. For underfitting, the model is too simple.

Why do we face these two problems in training a model ?

It is hard to pick the "just right" model and parameters for the data.

Almost all statistical problems can be stated in the following form:

1. Given the data $(y, x)$ find $hat{f}$ which produces $hat{y}=hat{f}(x)$.




2. Make this $hat{f}$ as close as possible to "true" $f$, where $f$ is defined as

$$y = f(x) + varepsilon$$

The temptation is always to make $hat{f}$ produce $hat{y}$ which are very close to the data $y$. But when new data point arrives, or we use data which was not used to construct $hat{f}$ the prediction may be way off. This happens because we are trying to explain $varepsilon$ instead of $f$. When we do this we stray from "true" $f$ and hence when new observation comes in we get a bad prediction. This when overfitting happens.

On the other hand when we find $hat{f}$ the question is always maybe we can get a better $tilde{f}$ which produces better fit and is close to "true" $f$? If we can then we underfitted in the first case.

If you look at the statistical problem this way, fitting the model is always a balance between underfitting and overfitting and any solution is always a compromise. We face this problem because our data is random and noisy.