I trained some tabular forecasting model and executed batch prediction jobs with Vertex AI, then sometimes the following error occurred 30-60 minutes after the start of the jobs.
Internal error occurred. Please retry in a few minutes. If you still experience errors, contact Vertex AI.
As the message mentioned, I retried the next day with same data and then I got the prediction job succeed. However, I have seen this error from time to time with Vertex AI. I cannot identify the reproduction conditions. I cannot find the logs of the job execution.
What causes this error in Vertex AI? How can I avoid this errror?
Resource name: projects/832409671062/locations/us-central1/batchPredictionJobs/3813678620929425408