The described symptoms—high, oscillating training and validation loss—are characteristic of a learning rate that is too high for the given model and data. In stochastic gradient descent (SGD), a high learning rate causes the optimizer to take excessively large steps, overshooting the minimum of the loss function. This leads to the loss value bouncing around the minimum without converging, creating the observed oscillating pattern. Decreasing the learning rate will reduce the step size, allowing the optimizer to descend more smoothly and stably towards a minimum, thereby improving the training process.

A. Introduce early stopping: Early stopping is a regularization technique to prevent overfitting (when validation loss increases while training loss decreases), not to fix a fundamental convergence problem where both losses are high and oscillating.

B. Increase the size of the test set: The test set is used for final model evaluation after training. Its size has no impact on the training dynamics or the convergence of the loss function.

C. Increase the learning rate: The symptoms indicate the learning rate is already too high. Increasing it further would exacerbate the oscillations and likely cause the loss to diverge completely.

1. Goodfellow

I.

Bengio

Y.

& Courville

A. (2016). Deep Learning. MIT Press.

Section 8.3.1

"Learning Rate": The text explains

"If the learning rate is too large

gradient descent can inadvertently increase the value of the objective function... This happens when a step is so large that it steps over the minimum and lands in a region with a higher cost than the starting point." This directly explains the oscillating and increasing loss pattern.

2. Karpathy

A. (n.d.). CS231n: Convolutional Neural Networks for Visual Recognition. Stanford University.

Module 1: Neural Networks Part 3

"Learning and Evaluation" section: The course notes illustrate how a learning rate that is too high causes the loss to "jump" around or diverge. A visual diagram shows the loss curve oscillating wildly when the learning rate is too high

matching the scenario in the question.

3. Amazon Web Services (AWS). (2023). Tune a Model with the SageMaker API. AWS Documentation.

Section: "Learning rate": The documentation identifies the learning rate as a critical hyperparameter that "controls how much to change the model in response to the estimated error each time the model weights are updated." Unstable training

as described

is a primary reason to adjust this hyperparameter

typically by decreasing it.