Ringing in the loss curve while training Pytorch Neural network

Question

Why is there ringing towards the end of training of my pytorch neural network

I am using a basic sequential neural network with Adam optimizer and loss value of 0.0001

Here is the model:

class ANN(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(in_features=956, out_features=500)
        self.fc2 = nn.Linear(in_features=500, out_features=500)
        self.fc3 = nn.Linear(in_features=500, out_features=100)
        self.output = nn.Linear(in_features=100, out_features=2)
 
    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = self.output(x)
        return x

and the criterion, loss and training call

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.0001)

epochs = 400
loss_arr = []

for i in range(epochs):
    y_hat = model.forward(X_train)
    loss = criterion(y_hat, y_train)
    loss_arr.append(loss)
    
 
    if i % 10 == 0:
        print(f'Epoch: {i} Loss: {loss}')
 
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

I expected the loss curve to decrease normally over time or increase abruptly if the model overfits

I donot have a lot of data to break it into validation set to monitor overfitting so I just train the model again and again to prevent overfitting

Answer 1

According to the loss you loss curve you posted, it seems to me that the problem could be that you don't change the learning rate during training. As the training is approaching the optimum, it is more difficult to find the step with correct direction. It could even happen that the training will oscilate near the optimum back and forth due to the large learning rate; in such a case the loss would not improve anymore. Try to set learning rate scheduler, see the How to adjust learning rate in PyTorch docs.

Furthermore, you wrote:

I expected the loss curve to decrease normally over time or increase abruptly if the model overfits

The training loss curve will not increase abrupty when the model overfits. Overfitting happens when training loss is very small, but validation loss increases abruptly. You cannot detect overfitting by measuring training loss only. If you don't have enough data, you could consider cross-validation.