Databricks create new job maintaining history

Question

Everytime I created a new job, instead of overwriting the existing one it created a new one with the same name. First thing that came to my mind was to delete all the jobs with a certain name and create the job afterwards. The problem with this is that I cant keep the deleted job history (logs etc) associated with the new job. Is there a way to overwrite the job mantaining all the info associated with the previous job using databricks cli?

databricks jobs list --output json | jq '.jobs[] | select(.settings.name == "test_job") | .job_id' | xargs -n 1 databricks jobs delete --job-id
databricks jobs create --json-file ./jobs_jsons/test_job.json

Answer 1

You need to use databricks jobs resets for that - it will overwrite jobs definition, without creating a new job (see docs):

databricks jobs reset --job-id 246 --json-file reset-job.json