Overfitting occurs when a model learns the training data too well, including its noise and specific patterns, at the expense of its ability to generalize to new, unseen data. This phenomenon is identified by monitoring the model's performance on both the training and validation datasets during training. The classic sign of overfitting is when the training loss continues to decrease, indicating the model is still improving its fit to the training data, while the validation loss, after an initial decrease, begins to increase. This divergence shows that the model is losing its generalization capability.

A. Constant losses for both training and validation indicate that the model is not learning effectively, which is a sign of underfitting, not overfitting.
B. An increasing training loss is contrary to the goal of training and suggests a problem with the training process itself, not a standard overfitting scenario.
D. A constant training loss while validation loss decreases is a highly improbable scenario, as it implies the model is generalizing better without learning from the training data.

1. Microsoft Learn

"Prevent overfitting and underfitting": In the learning path for DP-100

this document states

"A classic sign of overfitting is when the validation metric starts to get worse while the training metric continues to improve." This directly corresponds to a decreasing training loss and an increasing validation loss.

Source: Microsoft

"Build and operate machine learning solutions with Azure Machine learning

" Module: "Train a model with a script

" Unit: "Prevent overfitting and underfitting."

2. Goodfellow

I.

Bengio

Y.

& Courville

A. (2016). Deep Learning. MIT Press.

Section 5.2.2

Capacity

Overfitting and Underfitting: "As training progresses

training error tends to decrease... By contrast

the generalization error... begins to increase again. This phenomenon is known as overfitting." (Note: Generalization error is measured using a validation or test set).

3. Stanford University

CS231n: Convolutional Neural Networks for Visual Recognition.

Course Notes

"Training Neural Networks

Part 1"

Section: "Track your learning process": The notes illustrate overfitting with a graph showing the training accuracy continually increasing while the validation accuracy reaches a peak and then starts to decrease. The inverse is true for loss: training loss decreases while validation loss begins to increase. This gap between training and validation performance is the indicator of overfitting.