How can append a dataframe data to an existing file in adls2 using fastparquet

Question

I have a file in adls2.By using below statement I am unable to append the data to the existing file.

filepath = "abfs://shopifyparquet/test/parquet/LIVE/filename"

adls2_data_df.to_parquet(path=filepath,engine='fastparquet',storage_options={'account_name': 'test', 'account_key': 'mykey'},append=True)

I tried with local file and able to append the data to an existing file using fastpaquet. I tried the same using adls2 but getting the following exception

File mode not supported
Exception ignored in: <function AzureBlobFile.__del__ at 0x000001CBA3BC6950>
    self.close()
  File "C:\Users\Sivasankar.Muthuraju\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\adlfs\spec.py", line 1851, in close
    super().close()
  File "C:\Users\Sivasankar.Muthuraju\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\fsspec\spec.py", line 1740, in close
    if not self.forced:
AttributeError: 'AzureBlobFile' object has no attribute 'forced'

Answer 1

You can try, this can use to append the new data into an existing file:

dbutils.fs.cp("data_new.csv", "abfss://<Container_name>@<Storage_account_name>.dfs.core.windows.net/", True)