Mount Bucket on Google Storage on login node in google slurm cluster

Question

I'm using the Google Slurm cluster. All the nodes, including the login nodes in the cluster, have mounts to the home directory on the controller. I can my Google bucket with the gcsfuse command to /home/my_data/ on the login node. I could access the data from the login node. However, on the other nodes, the directory /home/my_data/ is empty and not visible, even though /home/ is shared between all nodes. I want to be able to mount the bucket from the login node and make it accessible on all the other nodes. Thanks

Answer 1

Are you using login_network_storage, as per this document:

login_network_storage: Network storage to mount on login and controller nodes. NFS, CIFS, Lustre, and GCSFuse will be installed automatically. Can be repeated for additional mounts.

So it will only be mounted on the login node, because of that you are getting an empty directory for other nodes.

Instead of using login_network_storage use network_storage where:

network_storage: Network storage to mount on all compute nodes in the partition. Fields will be added directly to fstab. Can be repeated for additional mounts.

It will be mounted to login, controller and compute nodes.

If you are still facing the issue, in the mount options try allow_other which will allow others to access the bucket mount point. You can check it in this document.

For more information about network storage mounts follow this git link. Check the similar issue being discussed in this group bucket available on all compute nodes, which might help to resolve your issue.