Can I change the size of input data during training?

Question

I am trying to do active learning and bump into the error:

RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. 

I think the reason of this error is because that I add labeled data during training. Is it possible to add data to dataset during training?

I have tried using just CPU. But it's too slow.

Answer 1

Yes , it is possible to add data during training time , it dynamically updates during training ! Check the inputs providing to training model, if encoded well or mapped to your model performance ? CUDA Error - occurring due to exceeding the dataset range or accessing elements beyond dataset size.

Answer 2

The error is likely coming from an out of bounds index. Since you're adding a label, my guess is somewhere in the code the class index of the new label is trying to index into something sized for the old number of labels.