FileNotFoundError: [Errno 2] No such file or directory: '/python/ml/mnist27_train.csv' on Kubeflow pipeline

29 Views Asked by At

I have this issue while trying to import my csv file located on my local machine : FileNotFoundError: [Errno 2] No such file or directory: '/python/ml/mnist27_train.csv', I find it in the logs of kubeflow pipeline. It's seen that i need a volume to be mounted on kubernetes configuration. Please do you have any idea to fix this issue ?

Here is my code to running my kubeflow pipeline :

import pandas as pd
from kfp import dsl
from kfp.dsl import component
import os

# Define a simple component to read a CSV file and print the first few rows
@component
def read_csv(file_path: str = '/python/ml/mnist27_train.csv'):
    
    try:
        import pandas as pd
    except ImportError:
        print("Installing pandas...")
        # Install pandas using pip within the component
        import subprocess
        subprocess.run(["pip", "install", "pandas"])
        import pandas as pd
        
    df = pd.read_csv(file_path)
    print(df.head())

# Define a pipeline using the component
@dsl.pipeline(
    name='Read CSV Pipeline',
    description='A pipeline to read a CSV file'
)
def read_csv_pipeline():
    file_path = '/python/ml/mnist27_train.csv'
    if os.path.exists(file_path):
        print(f"File exists at {file_path}")
    else:
        print(f"File not found at {file_path}")
    read_csv(file_path=file_path)  # Pass file_path as a keyword argument

# Compile and run the pipeline
if __name__ == '__main__':
    import kfp.compiler as compiler
    compiler.Compiler().compile(read_csv_pipeline, 'read_csv_pipeline.yaml')
    run = kfp.Client().create_run_from_pipeline_func(read_csv_pipeline, arguments={})

I try without success to create a persistent volume call 'persvol.yaml' with this content :

    apiVersion: v1
kind: PersistentVolume
metadata:
  name: data-pv-volume
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: "/python/ml/mnist27_train.csv"

and make a :

kubectl apply -f persvol.yaml

any help will be very appreciated

0

There are 0 best solutions below