Uploading files to google cloud datalab VM instance

Question

I have an open google cloud datalab notebook, and I'm looking at folder contents. There is an upload button, and I can use it to upload files. It works fine for any files under about 500kb, but the moment I try to upload anything larger, it simply hangs forever. I'm not trying to upload massive files via a web interface, just 10mb or less worth of data, but it still won't go through.

Does datalab have a maximum file size that you can use the web uploader for?

Answer 1

There's an outstanding issue on the Github repo for Datalab that discusses this, it's something that Datalab inherits from Jupyter 4, on which it's based, and there's unfortunately currently no way around it.

Answer 2

Since Google Cloud Datalab Notebook is running as a docker container within the provisioned Google Compute Engine (GCE) instance, we will need to copy the file into the container itself.

1/ Copying the large file from local to GCS bucket:

[Local machine prompt] gsutil cp <my_file> gs://<my_bucket>

2/ SSH into the Datalab GCE instance from Cloud Shell:

[Cloud Shell prompt] gcloud compute ssh <instance name>

3/ Find the id of the Cloud Datalab container:

[GCE Instance prompt] docker ps -a
CONTAINER ID        IMAGE                                       COMMAND
0aa745853b54        gcr.io/cloud-datalab/datalab:latest         "/datalab/run.sh"

4/ Exec into the container with the id just found:

[GCE instance prompt] docker exec -it 0aa745853b54 /bin/sh

5/ In the container's new prompt, copy the file into the notebook location:

[Container prompt] gsutil cp gs://<my_bucket>/<my_file> /content/datalab

6/ Confirm the file is copied:

[Container prompt] ls -l /content/datalab

7/ Exit the container by holding Ctrl + p + q

The my_file should also be visible from the Web GUI of Notebook.