how to build delta-rs python for hdfs support

74 Views Asked by At

i would build delta-rs v0.10.1 with HDFS support. for this purpose i have costomized Cargo.toml file and add hdfs feature then building and installing it in environment using make install command.

features = ["azure", "gcs", "python", "datafusion", "unity-experimental", "hdfs"]

at the end it will add libjvm.so shared library in deltalake.libs directory but i am stuck on need of libzip.so file in environment site packages.

Error occurred during initialization of VM
Unable to load ZIP library: /path/delta/python/venv/lib/python3.10/site-packages/libzip.so
/arrow/cpp/src/arrow/filesystem/s3fs.cc:2598

I have tried to copy system libzip.so.4 files to site packages but i get Corrupted shared library.

Error occurred during initialization of VM
Corrupted ZIP library: /path/delta/python/venv/lib/python3.10/site-packages/libzip.so
/arrow/cpp/src/arrow/filesystem/s3fs.cc:2598:  arrow::fs::FinalizeS3 was not called even though
S3 was initialized.  This could lead to a segmentation fault at exit
0

There are 0 best solutions below