I have installed the PDAL library ( https://pdal.io/en/latest/ ) on databricks cluster by using a databricks cluster based on container image. The issue is that the pdal library is not working properly, i am not able to translate to some of the data formats. For example, for las, i am not able to convert from obj to las, but conversion from obj to pcd or text is working.
I have installed pdal from conda in container using runtime9.x.
This is the docker file:
FROM databricksruntime/standard:9.x
# Install miniconda
ENV CONDA_DIR /databricks/conda
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-py38_23.1.0-1-Linux-x86_64.sh -O ~/miniconda.sh && \
/bin/bash ~/miniconda.sh -b -p /databricks/conda
COPY env.yml /databricks/.conda-env-def/env.yml
RUN /databricks/conda/bin/conda env create --file /databricks/.conda-env-def/env.yml \
&& ln -s /databricks/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh
RUN /databricks/conda/bin/conda config --system --set channel_priority strict \
&& /databricks/conda/bin/conda config --system --set always_yes True
ENV DEFAULT_DATABRICKS_ROOT_CONDA_ENV=dcs-minimal
ENV DATABRICKS_ROOT_CONDA_ENV=dcs-minimal
the env.yml file:
name: dcs-minimal
channels:
- conda-forge
dependencies:
- python=3.8.8
- pip=20.2.4
- six=1.15.0
- ipython=7.19.0
- nomkl=3.0
- numpy=1.19.2
- pandas=1.1.5
- traitlets=5.0.5
- wheel=0.35.1
- pdal
- python-pdal
- pip:
- pyarrow==1.0.1
the conda list from the databricks cluster:
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_kmp_llvm conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backports 1.0 pyhd8ed1ab_3 conda-forge
backports.functools_lru_cache 1.6.5 pyhd8ed1ab_0 conda-forge
blas 2.117 openblas conda-forge
blas-devel 3.9.0 17_linux64_openblas conda-forge
blosc 1.21.4 h0f2a231_0 conda-forge
boost-cpp 1.78.0 h5adbc97_2 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.19.1 hd590300_0 conda-forge
ca-certificates 2023.7.22 hbcca054_0 conda-forge
cairo 1.16.0 hb05425b_5
cfitsio 4.2.0 hd9d235c_0 conda-forge
curl 7.87.0 h6312ad2_0 conda-forge
decorator 5.1.1 pyhd8ed1ab_0 conda-forge
draco 1.5.6 hf52228f_0 conda-forge
eigen 3.4.0 h00ab1b0_0 conda-forge
expat 2.5.0 hcb278e6_1 conda-forge
fontconfig 2.14.2 h14ed4e7_0 conda-forge
freetype 2.12.1 hca18f0e_1 conda-forge
freexl 1.0.6 h166bdaf_1 conda-forge
geos 3.11.1 h27087fc_0 conda-forge
geotiff 1.7.1 h7a142b4_6 conda-forge
giflib 5.2.1 h0b41bf4_3 conda-forge
glib 2.69.1 h4ff587b_1
hdf4 4.2.15 h9772cbc_5 conda-forge
hdf5 1.12.2 nompi_h2386368_101 conda-forge
icu 70.1 h27087fc_0 conda-forge
ipython 7.19.0 py38h81c977d_2 conda-forge
ipython_genutils 0.2.0 py_1 conda-forge
jedi 0.17.2 py38h578d9bd_2 conda-forge
jpeg 9e h0b41bf4_3 conda-forge
json-c 0.16 hc379101_0 conda-forge
kealib 1.5.0 ha7026e8_0 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
krb5 1.20.1 hf9c8cef_0 conda-forge
lcms2 2.15 hfd0df8a_0 conda-forge
ld_impl_linux-64 2.40 h41732ed_0 conda-forge
lerc 4.0.0 h27087fc_0 conda-forge
libaec 1.0.6 hcb278e6_1 conda-forge
libblas 3.9.0 17_linux64_openblas conda-forge
libcblas 3.9.0 17_linux64_openblas conda-forge
libcurl 7.87.0 h6312ad2_0 conda-forge
libdeflate 1.17 h0b41bf4_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libexpat 2.5.0 hcb278e6_1 conda-forge
libffi 3.3 h58526e2_2 conda-forge
libgcc-ng 13.1.0 he5830b7_0 conda-forge
libgdal 3.6.2 hae1867e_4 conda-forge
libgfortran-ng 13.1.0 h69a702a_0 conda-forge
libgfortran5 13.1.0 h15d22d2_0 conda-forge
libiconv 1.17 h166bdaf_0 conda-forge
libkml 1.3.0 h37653c0_1015 conda-forge
liblapack 3.9.0 17_linux64_openblas conda-forge
liblapacke 3.9.0 17_linux64_openblas conda-forge
libnetcdf 4.8.1 nompi_h261ec11_106 conda-forge
libnghttp2 1.51.0 hdcd2b5c_0 conda-forge
libnsl 2.0.0 h7f98852_0 conda-forge
libopenblas 0.3.23 pthreads_h80387f5_0 conda-forge
libpng 1.6.39 h753d276_0 conda-forge
libpq 15.1 h2baec63_3 conda-forge
librttopo 1.1.0 ha49c73b_12 conda-forge
libspatialite 5.0.1 h221c8f1_23 conda-forge
libsqlite 3.42.0 h2797004_0 conda-forge
libssh2 1.10.0 haa6b8db_3 conda-forge
libstdcxx-ng 13.1.0 hfd8a6a1_0 conda-forge
libtiff 4.5.0 h6adf6a1_2 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libwebp-base 1.3.1 hd590300_0 conda-forge
libxcb 1.15 h0b41bf4_0 conda-forge
libxml2 2.10.3 hca2bb57_4 conda-forge
libzip 1.9.2 hc869a4a_1 conda-forge
libzlib 1.2.13 hd590300_5 conda-forge
llvm-openmp 16.0.6 h4dfa4b3_0 conda-forge
lz4-c 1.9.4 hcb278e6_0 conda-forge
ncurses 6.4 hcb278e6_0 conda-forge
nitro 2.7.dev8 h59595ed_0 conda-forge
nomkl 3.0 0
nspr 4.35 h27087fc_0 conda-forge
nss 3.89 he45b914_0 conda-forge
numpy 1.19.2 py38hf89b668_1 conda-forge
openblas 0.3.23 pthreads_h855a84d_0 conda-forge
openjpeg 2.5.0 hfec8fc6_2 conda-forge
openssl 1.1.1u hd590300_0 conda-forge
pandas 1.1.5 py38h51da96c_0 conda-forge
parso 0.7.1 pyh9f0ad1d_0 conda-forge
pcre 8.45 h9c3ff4c_0 conda-forge
pcre2 10.40 hc3806b6_0 conda-forge
pdal 2.4.3 hdfc1b4a_5 conda-forge
pexpect 4.8.0 pyh1a96a4e_2 conda-forge
pickleshare 0.7.5 py_1003 conda-forge
pip 20.2.4 py_0 conda-forge
pixman 0.40.0 h36c2ea0_0 conda-forge
poppler 22.12.0 h381b16e_0
poppler-data 0.4.12 hd8ed1ab_0 conda-forge
postgresql 15.1 h84e8d4a_3 conda-forge
proj 9.1.1 h8ffa02c_2 conda-forge
prompt-toolkit 3.0.39 pyha770c72_0 conda-forge
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge
pyarrow 1.0.1 pypi_0 pypi
pygments 2.15.1 pyhd8ed1ab_0 conda-forge
python 3.8.8 hffdb5ce_0_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-pdal 3.1.2 py38h514daf8_0 conda-forge
python_abi 3.8 3_cp38 conda-forge
pytz 2023.3 pyhd8ed1ab_0 conda-forge
readline 8.2 h8228510_1 conda-forge
setuptools 59.8.0 py38h578d9bd_1 conda-forge
six 1.15.0 pyh9f0ad1d_0 conda-forge
snappy 1.1.10 h9fff704_0 conda-forge
sqlite 3.42.0 h2c6b66d_0 conda-forge
tiledb 2.13.2 hc2ae436_0 conda-forge
tk 8.6.12 h27826a3_0 conda-forge
traitlets 5.0.5 py_0 conda-forge
tzcode 2023c h0b41bf4_0 conda-forge
tzdata 2023c h71feb2d_0 conda-forge
wcwidth 0.2.6 pyhd8ed1ab_0 conda-forge
wheel 0.35.1 pyh9f0ad1d_0 conda-forge
xerces-c 3.2.4 h55805fa_1 conda-forge
xorg-libxau 1.0.11 hd590300_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
zlib 1.2.13 hd590300_5 conda-forge
zstd 1.5.2 hfc55251_7 conda-forge
The command:
result = subprocess.run(['pdal','translate','/dbfs/mnt/demo.obj','/dbfs/mnt/demo.las'],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT )
returns:
CompletedProcess(args=['pdal', 'translate', '/dbfs/mnt/demo.obj', '/dbfs/mnt/demo.las', returncode=0, stdout=b'')
But the demo.las file is not created.
Here is an example init script that has been used back on DBR 9.x to install and configure PDAL. This might be of use for your needs either as-is or you can incorporate into your docker build. Also, we offer GDAL (directly on DBR, not docker) which can be foundational for you to add PDAL [see docs].