How can I sync the complete `local_dir` (`~/ray_results`) with the head node?

Question

I am creating custom result files in my jobs and want to sync them from the worker nodes to the head nodes (To rsync them down to my local computer later on). I tried to write them all into the local_dir e.g. ~./ray_results but unfortunately it seems that ray tune is only synching the individual trial folders in the local_dir. Is this correct?

Answer 1

Yes; try writing them to the self.logdir for each trainable.