Pyspark renaming file in HDFS

Question

I'm using spark 2.3.1 and want to rename HDFS file in pyspark program. I'm following this answer[https://stackoverflow.com/a/58172249/12927963] (Pyspark)

For this line

fs.rename(Path(hdfs_dir+''+file_name),Path(hdfs_dir+''+new_filename))

I'm getting following Error.

NameError: name 'Path' is not defined

If I remove Path keyword from line

I'm getting following error.

Py4JError: An error occurred while calling o562.rename. Trace: py4j.Py4JException: Method rename([class java.lang.String, class java.lang.String]) does not exist

So How to solve this issue?

Answer 1

Define path as

Path = sc._gateway.jvm.org.apache.hadoop.fs.Path