Kubernetes - Jupyter PySpark notebook - OSError: [Errno 30] Read-only file system: '/home/jovyan/.jupyter/migrated'

Question

I'm trying to deploy jupyter-pyspark-notebook on AKS(Azure Kubernetes Services) and I use k8s configmap to have custom "jupyter_server_config.py" in order to apply Self signed cerificates. Also, I'm mounting a PVC on "/jupyter-storage"

Here are the yaml files used:

sts.yaml:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: jupyter-pyspark-notebook
  namespace: monitor-azure
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jupyter-pyspark-notebook
  template:
    metadata:
      labels:
        app: jupyter-pyspark-notebook
    spec:
      containers:
      - args:
        - bash
        - -c
        - |
          cd /jupyter-storage;
          for i in $(seq 1 $NUMBER_OF_NOTEBOOKS); do
            jupyter-lab --no-browser --port=$((8889 + i)) --ip=$HOSTNAME --allow-root &
          done
          while true; do
            sleep 10
          done      
        env:
        - name: NUMBER_OF_NOTEBOOKS
          value: "10"
        image: jupyter/pyspark-notebook
        imagePullPolicy: Always
        name: jupyter-container
        ports:
        - containerPort: 8890
          name: port1
          protocol: TCP
        - containerPort: 8891
          name: port2
          protocol: TCP
        - containerPort: 8892
          name: port3
          protocol: TCP
        - containerPort: 8893
          name: port4
          protocol: TCP
        - containerPort: 8894
          name: port5
          protocol: TCP
        - containerPort: 8895
          name: port6
          protocol: TCP
        - containerPort: 8896
          name: port7
          protocol: TCP
        - containerPort: 8897
          name: port8
          protocol: TCP
        - containerPort: 8898
          name: port9
          protocol: TCP
        - containerPort: 8899
          name: port10
          protocol: TCP
        resources:
          limits:
            cpu: "4"
            memory: 2Gi
          requests:
            cpu: "1"
            memory: 1Gi
        securityContext:
          runAsUser: 0
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /jupyter-storage
          name: jupyter-storage
        - name: jupyter-config-volume
          mountPath: /home/jovyan/.jupyter
        - name: jupyter-tls-secret-volume
          readOnly: true
          mountPath: /etc/jupyter             
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      securityContext: {}
      terminationGracePeriodSeconds: 30
      tolerations:
        - key: monitor-azure
          operator: Equal
          value: dedicated
          effect: NoSchedule      
      volumes:
      - name: jupyter-storage
        persistentVolumeClaim:
          claimName: jupyter-storage
      - name: jupyter-config-volume
        configMap:
          name: jupyter-config
      - name: jupyter-tls-secret-volume
        secret:
          secretName: jupyter-tls 
  updateStrategy:
    rollingUpdate:
      partition: 0
type: RollingUpdate

configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: jupyter-config
  namespace: monitor-azure
data:
  jupyter_server_config.py: |
    c.ServerApp.certfile = '/etc/jupyter/tls.crt'
    c.ServerApp.keyfile = '/etc/jupyter/tls.key'
    c.ServerApp.ip = '0.0.0.0'
    c.ServerApp.open_browser = False

secret description

C:\Users\test-user>kubectl -n monitor-azure describe secret jupyter-tls
Name:         jupyter-tls
Namespace:    monitor-azure
Labels:       <none>
Annotations:  <none>

Type:  kubernetes.io/tls

Data
====
tls.key:  1704 bytes
tls.crt:  1407 bytes

Above secret is used to apply ssl certificates so that the jupyter notebook can be accessed using https.

When I issue the command "jupyter server list" from the pod, I get the below error

Error:

(base) root@jupyter-pyspark-notebook-0:~# jupyter server list Traceback (most recent call last): File "/opt/conda/bin/jupyter-server", line 10, in sys.exit(main()) ^^^^^^ File "/opt/conda/lib/python3.11/site-packages/jupyter_core/application.py", line 280, in launch_instance super().launch_instance(argv=argv, **kwargs) File "/opt/conda/lib/python3.11/site-packages/traitlets/config/application.py", line 1052, in launch_instance app.initialize(argv) File "/opt/conda/lib/python3.11/site-packages/traitlets/config/application.py", line 117, in inner return method(app, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/jupyter_server/serverapp.py", line 2574, in initialize super().initialize(argv=argv) File "/opt/conda/lib/python3.11/site-packages/traitlets/config/application.py", line 117, in inner return method(app, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/jupyter_core/application.py", line 251, in initialize self.parse_command_line(argv) File "/opt/conda/lib/python3.11/site-packages/jupyter_server/serverapp.py", line 1862, in parse_command_line super().parse_command_line(argv) File "/opt/conda/lib/python3.11/site-packages/traitlets/config/application.py", line 117, in inner return method(app, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/traitlets/config/application.py", line 857, in parse_command_line return self.initialize_subcommand(subc, subargv) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/traitlets/config/application.py", line 117, in inner return method(app, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/traitlets/config/application.py", line 721, in initialize_subcommand self.subapp.initialize(argv) # type:ignore[union-attr] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/traitlets/config/application.py", line 117, in inner return method(app, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/jupyter_core/application.py", line 255, in initialize self.migrate_config() File "/opt/conda/lib/python3.11/site-packages/jupyter_core/application.py", line 184, in migrate_config migrate() File "/opt/conda/lib/python3.11/site-packages/jupyter_core/migrate.py", line 246, in migrate with open(os.path.join(env["jupyter_config"], "migrated"), "w", encoding="utf-8") as f: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ OSError: [Errno 30] Read-only file system: '/home/jovyan/.jupyter/migrated'

Also, below error in the pod log:

[W 2023-10-25 19:03:51.609 ServerApp] 500 PUT /lab/api/workspaces/default?1698260631247 (10.244.4.1): [Errno 30] Read-only file system: '/home/jovyan/.jupyter/lab' [W 2023-10-25 19:03:51.609 LabApp] wrote error: "[Errno 30] Read-only file system: '/home/jovyan/.jupyter/lab'" Traceback (most recent call last): File "/opt/conda/lib/python3.11/pathlib.py", line 1116, in mkdir os.mkdir(self, mode) FileNotFoundError: [Errno 2] No such file or directory: '/home/jovyan/.jupyter/lab/workspaces'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/site-packages/jupyterlab_server/workspaces_handler.py",

line 216, in put self.manager.save(space_name, raw) File "/opt/conda/lib/python3.11/site-packages/jupyterlab_server/workspaces_handler.py", line 136, in save self.workspaces_dir.mkdir(parents=True) File "/opt/conda/lib/python3.11/pathlib.py", line 1120, in mkdir self.parent.mkdir(parents=True, exist_ok=True) File "/opt/conda/lib/python3.11/pathlib.py", line 1116, in mkdir os.mkdir(self, mode) OSError: [Errno 30] Read-only file system: '/home/jovyan/.jupyter/lab'

Answer 1

I was mounting the k8s configmap(which has the cert & key file location) on "/home/jovyan/.jupyter". And, the notebook was trying to write on the same location, thus getting error wrote error: "[Errno 30] Read-only file system"

Solution:

I have added the cert & key details in the args section instead of mounting the configmap on "/home/jovyan/.jupyter" and no more errors now.

--ServerApp.certfile='/etc/jupyter/tls.crt' --ServerApp.keyfile='/etc/jupyter/tls.key'

---

  - args:
    - bash
    - -c
    - "cd /jupyter-file-storage;\nfor i in $(seq 1 $NUMBER_OF_NOTEBOOKS); do\n
      \ jupyter-lab --ServerApp.certfile='/etc/jupyter/tls.crt' --ServerApp.keyfile='/etc/jupyter/tls.key'
      --no-browser --port=$((8888 + i)) --ip=$HOSTNAME --allow-root &\ndone\nwhile
      true; do\n  sleep 10\ndone      \n"