Databricks -Can we variablize the file name for while loading(mounting a file name)

809 Views Asked by At

can we variablize as below if not how do we do it enter image description here

2

There are 2 best solutions below

0
On

as I see that should be the python, then just use: .load("..../{}".format(filename)). If it's Scala, then you can use .load(s".../$filename") to substitute file name...

0
On

You may checkout the steps to configure the variables filename while loading the mount file name:

Step1: Declaring the variables:

mountname = 'test'
csvname = 'original.csv'
path = "dbfs:/mnt/{0}/{1}".format(mountname,csvname)

Step2: Mounting the storage account

dbutils.fs.mount(
  source = "wasbs://[email protected]/",
  mount_point = "/mnt/{0}".format(mountname),
  extra_configs = {"fs.azure.sas.test.chepra.blob.core.windows.net":"gv7nXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXlOiA=="})
print("=> Succeeded") 

Step3: Creating the Spark Dataframe

df = spark.read.format("csv").option("sep", ",").options(header= "true", inferschema='true').option('escape','"').load("{0}".format(path))

enter image description here