I am running an analysis on a large number of rasters with Dask in a Jupyter Lab notebook on a Windows computer. When I run my analysis using Coiled (cloud cluster), the analysis runs fine. However, when I run the analysis on a local Dask cluster with from dask.distributed import Client, LocalCluster , local_cluster = LocalCluster(), I get an error that ends with the below when I run the code. It seems like the workers in the local cluster aren't able to import rasterio or gdal for some reason.
File ~\AppData\Local\anaconda3\envs\carbon_notebook\Lib\site-packages\rasterio\__init__.py:28
24 os.add_dll_directory(os.path.abspath(p))
27 from rasterio._show_versions import show_versions
---> 28 from rasterio._version import gdal_version, get_geos_version, get_proj_version
29 from rasterio.crs import CRS
30 from rasterio.drivers import driver_from_extension, is_blacklisted
ImportError: DLL load failed while importing _version: The specified procedure could not be found.
In order to import rasterio into my Jupyter Lab notebook, I'm importing it with the following, which doesn't cause any errors and allows for successful usage in Coiled.
from osgeo import gdal
import rasterio
The lines below are where rasterio is introduced:
with rasterio.open(uri) as ds:
return ds.read(1, window=rasterio.windows.from_bounds(*bounds, ds.transform))
When I run dask.compute on a function that includes rasterio.open , the local cluster returns the above error at the end of a lengthy Jupyter Lab stack trace. I'd assumed that I'd be able to run my analysis in a local cluster the same as in a Coiled cluster but that's not the case.
The packages in my notebook are as follows. To be clear, I can import rasterio to my Jupyter notebook just fine. The problem is with using it in a local Dask cluster. Any idea why I can't use rasterio in a local Dask cluster? Thanks very much.
And please let me know if I should provide other background information.
# Name Version Build Channel
affine 2.4.0 pyhd8ed1ab_0 conda-forge
aiohttp 3.8.5 py311ha68e1ae_0 conda-forge
aiosignal 1.3.1 pyhd8ed1ab_0 conda-forge
anyio 3.7.1 pyhd8ed1ab_0 conda-forge
argon2-cffi 21.3.0 pyhd8ed1ab_0 conda-forge
argon2-cffi-bindings 21.2.0 py311ha68e1ae_3 conda-forge
arrow 1.2.3 pyhd8ed1ab_0 conda-forge
asttokens 2.2.1 pyhd8ed1ab_0 conda-forge
async-lru 2.0.4 pyhd8ed1ab_0 conda-forge
async-timeout 4.0.3 pyhd8ed1ab_0 conda-forge
attrs 23.1.0 pyh71513ae_1 conda-forge
aws-c-auth 0.7.0 h15a3724_3 conda-forge
aws-c-cal 0.6.0 h6ba3258_0 conda-forge
aws-c-common 0.8.23 hcfcfb64_0 conda-forge
aws-c-compression 0.2.17 h420beca_1 conda-forge
aws-c-event-stream 0.3.1 had47b81_1 conda-forge
aws-c-http 0.7.11 h72ba615_0 conda-forge
aws-c-io 0.13.28 ha35c040_0 conda-forge
aws-c-mqtt 0.9.0 h4941efa_0 conda-forge
aws-c-s3 0.3.13 he04eaa7_2 conda-forge
aws-c-sdkutils 0.1.12 h420beca_0 conda-forge
aws-checksums 0.1.16 h420beca_1 conda-forge
aws-crt-cpp 0.21.0 h4a8302c_2 conda-forge
aws-sdk-cpp 1.10.57 hc0abff2_18 conda-forge
babel 2.12.1 pyhd8ed1ab_1 conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backoff 2.2.1 pyhd8ed1ab_0 conda-forge
backports 1.0 pyhd8ed1ab_3 conda-forge
backports.functools_lru_cache 1.6.5 pyhd8ed1ab_0 conda-forge
beautifulsoup4 4.12.2 pyha770c72_0 conda-forge
bleach 6.0.0 pyhd8ed1ab_0 conda-forge
blosc 1.21.4 hdccc3a2_0 conda-forge
bokeh 3.2.1 pyhd8ed1ab_0 conda-forge
boost-cpp 1.78.0 h9f4b32c_3 conda-forge
boto3 1.28.24 pyhd8ed1ab_0 conda-forge
botocore 1.31.24 pyhd8ed1ab_0 conda-forge
branca 0.6.0 pyhd8ed1ab_0 conda-forge
brotli 1.0.9 hcfcfb64_9 conda-forge
brotli-bin 1.0.9 hcfcfb64_9 conda-forge
brotlipy 0.7.0 py311ha68e1ae_1005 conda-forge
bzip2 1.0.8 he774522_0
c-ares 1.19.1 hcfcfb64_0 conda-forge
ca-certificates 2023.7.22 h56e8100_0 conda-forge
cached-property 1.5.2 hd8ed1ab_1 conda-forge
cached_property 1.5.2 pyha770c72_1 conda-forge
cairo 1.16.0 hdecc03f_1016 conda-forge
certifi 2023.7.22 pyhd8ed1ab_0 conda-forge
cffi 1.15.1 py311h7d9ee11_3 conda-forge
cfitsio 4.2.0 h9ebe7e4_0 conda-forge
cftime 1.6.3 pypi_0 pypi
charset-normalizer 3.2.0 pyhd8ed1ab_0 conda-forge
click 8.1.6 win_pyh7428d3b_0 conda-forge
click-plugins 1.1.1 py_0 conda-forge
cligj 0.7.2 pyhd8ed1ab_1 conda-forge
cloudpickle 2.2.1 pyhd8ed1ab_0 conda-forge
coiled 0.9.9 pyhd8ed1ab_0 conda-forge
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
colorcet 3.0.1 pyhd8ed1ab_0 conda-forge
comm 0.1.4 pyhd8ed1ab_0 conda-forge
contourpy 1.1.0 py311h005e61a_0 conda-forge
cryptography 41.0.3 py311h28e9c30_0 conda-forge
curl 8.2.1 hd5e4a3a_0 conda-forge
cycler 0.11.0 pyhd8ed1ab_0 conda-forge
cytoolz 0.12.2 py311ha68e1ae_0 conda-forge
dask 2023.8.0 pyhd8ed1ab_0 conda-forge
dask-core 2023.8.0 pyhd8ed1ab_0 conda-forge
dask-geopandas 0.3.1 pyhd8ed1ab_0 conda-forge
datashader 0.15.1 pyhd8ed1ab_0 conda-forge
datashape 0.5.4 py_1 conda-forge
debugpy 1.6.8 py311h12c1d0e_0 conda-forge
decorator 5.1.1 pyhd8ed1ab_0 conda-forge
defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge
distributed 2023.8.0 pyhd8ed1ab_0 conda-forge
entrypoints 0.4 pyhd8ed1ab_0 conda-forge
exceptiongroup 1.1.2 pyhd8ed1ab_0 conda-forge
executing 1.2.0 pyhd8ed1ab_0 conda-forge
expat 2.5.0 h63175ca_1 conda-forge
filelock 3.12.2 pyhd8ed1ab_0 conda-forge
fiona 1.9.4 py311h4e4dc46_0 conda-forge
flit-core 3.9.0 pyhd8ed1ab_0 conda-forge
folium 0.14.0 pyhd8ed1ab_0 conda-forge
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 3.000 h77eed37_0 conda-forge
font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge
font-ttf-ubuntu 0.83 hab24e00_0 conda-forge
fontconfig 2.14.2 hbde0cde_0 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
fonttools 4.42.0 py311ha68e1ae_0 conda-forge
fqdn 1.5.1 pyhd8ed1ab_0 conda-forge
freetype 2.12.1 h546665d_1 conda-forge
freexl 1.0.6 h67ca5e6_1 conda-forge
frozenlist 1.4.0 py311ha68e1ae_0 conda-forge
fsspec 2023.6.0 pyh1a96a4e_0 conda-forge
gdal 3.7.0 py311h68c9619_3 conda-forge
geopandas 0.13.2 pyhd8ed1ab_1 conda-forge
geopandas-base 0.13.2 pyha770c72_1 conda-forge
geos 3.11.2 h1537add_0 conda-forge
geotiff 1.7.1 h4e61e90_11 conda-forge
gettext 0.21.1 h5728263_0 conda-forge
gilknocker 0.4.1 py311h9a9e57f_2 conda-forge
hdf4 4.2.15 h1334946_6 conda-forge
hdf5 1.14.1 nompi_h73e8ff5_100 conda-forge
icu 72.1 h63175ca_0 conda-forge
idna 3.4 pyhd8ed1ab_0 conda-forge
importlib-metadata 6.8.0 pyha770c72_0 conda-forge
importlib_metadata 6.8.0 hd8ed1ab_0 conda-forge
importlib_resources 6.0.1 pyhd8ed1ab_0 conda-forge
intel-openmp 2023.2.0 h57928b3_49496 conda-forge
ipykernel 6.25.1 pyh6817e22_0 conda-forge
ipython 8.14.0 pyh08f2357_0 conda-forge
ipywidgets 8.1.0 pyhd8ed1ab_0 conda-forge
isoduration 20.11.0 pyhd8ed1ab_0 conda-forge
jedi 0.19.0 pyhd8ed1ab_0 conda-forge
jinja2 3.1.2 pyhd8ed1ab_1 conda-forge
jmespath 1.0.1 pyhd8ed1ab_0 conda-forge
joblib 1.3.2 pyhd8ed1ab_0 conda-forge
json5 0.9.14 pyhd8ed1ab_0 conda-forge
jsondiff 2.0.0 pyhd8ed1ab_0 conda-forge
jsonpointer 2.0 py_0 conda-forge
jsonschema 4.19.0 pyhd8ed1ab_1 conda-forge
jsonschema-specifications 2023.7.1 pyhd8ed1ab_0 conda-forge
jsonschema-with-format-nongpl 4.19.0 pyhd8ed1ab_1 conda-forge
jupyter-lsp 2.2.0 pyhd8ed1ab_0 conda-forge
jupyter_client 8.3.0 pyhd8ed1ab_0 conda-forge
jupyter_core 5.3.1 py311h1ea47a8_0 conda-forge
jupyter_events 0.7.0 pyhd8ed1ab_2 conda-forge
jupyter_server 2.7.0 pyhd8ed1ab_0 conda-forge
jupyter_server_terminals 0.4.4 pyhd8ed1ab_1 conda-forge
jupyterlab 4.0.4 pyhd8ed1ab_0 conda-forge
jupyterlab_pygments 0.2.2 pyhd8ed1ab_0 conda-forge
jupyterlab_server 2.24.0 pyhd8ed1ab_0 conda-forge
jupyterlab_widgets 3.0.8 pyhd8ed1ab_0 conda-forge
kealib 1.5.1 h96def13_4 conda-forge
kiwisolver 1.4.4 py311h005e61a_1 conda-forge
krb5 1.21.1 heb0366b_0 conda-forge
lcms2 2.15 h3e3b177_1 conda-forge
lerc 4.0.0 h63175ca_0 conda-forge
libabseil 20230125.3 cxx17_h63175ca_0 conda-forge
libaec 1.0.6 h63175ca_1 conda-forge
libarchive 3.6.2 h6f8411a_1 conda-forge
libarrow 12.0.1 ha06dc20_8_cpu conda-forge
libblas 3.9.0 17_win64_mkl conda-forge
libbrotlicommon 1.0.9 hcfcfb64_9 conda-forge
libbrotlidec 1.0.9 hcfcfb64_9 conda-forge
libbrotlienc 1.0.9 hcfcfb64_9 conda-forge
libcblas 3.9.0 17_win64_mkl conda-forge
libcrc32c 1.1.2 h0e60522_0 conda-forge
libcurl 8.2.1 hd5e4a3a_0 conda-forge
libdeflate 1.18 hcfcfb64_0 conda-forge
libevent 2.1.12 h3671451_1 conda-forge
libexpat 2.5.0 h63175ca_1 conda-forge
libffi 3.4.4 hd77b12b_0
libgdal 3.7.0 h123a834_3 conda-forge
libglib 2.76.4 he8f3873_0 conda-forge
libgoogle-cloud 2.12.0 hbc1b25b_1 conda-forge
libgrpc 1.56.2 hea2d5f7_0 conda-forge
libhwloc 2.9.2 nocuda_h15da153_1008 conda-forge
libiconv 1.17 h8ffe710_0 conda-forge
libjpeg-turbo 2.1.5.1 hcfcfb64_0 conda-forge
libkml 1.3.0 hf2ab4e4_1015 conda-forge
liblapack 3.9.0 17_win64_mkl conda-forge
libnetcdf 4.9.2 nompi_h5902ca5_107 conda-forge
libpng 1.6.39 h19919ed_0 conda-forge
libpq 15.4 h43585b0_0 conda-forge
libprotobuf 4.23.3 h1975477_0 conda-forge
librttopo 1.1.0 he1da8c1_13 conda-forge
libsodium 1.0.18 h8d14728_1 conda-forge
libspatialindex 1.9.3 h39d44d4_4 conda-forge
libspatialite 5.0.1 hf8d749d_27 conda-forge
libsqlite 3.42.0 hcfcfb64_0 conda-forge
libssh2 1.11.0 h7dfc565_0 conda-forge
libthrift 0.18.1 h06f6336_2 conda-forge
libtiff 4.5.1 h6c8260b_0 conda-forge
libutf8proc 2.8.0 h82a8f57_0 conda-forge
libwebp-base 1.3.1 hcfcfb64_0 conda-forge
libxcb 1.15 hcd874cb_0 conda-forge
libxml2 2.11.5 hc3477c8_0 conda-forge
libzip 1.9.2 h519de47_1 conda-forge
libzlib 1.2.13 hcfcfb64_5 conda-forge
line-profiler 4.1.1 pypi_0 pypi
llvmlite 0.40.1 py311h5bc0dda_0 conda-forge
locket 1.0.0 pyhd8ed1ab_0 conda-forge
lz4 4.3.2 py311haddf500_0 conda-forge
lz4-c 1.9.4 hcfcfb64_0 conda-forge
lzo 2.10 he774522_1000 conda-forge
m2w64-gcc-libgfortran 5.3.0 6 conda-forge
m2w64-gcc-libs 5.3.0 7 conda-forge
m2w64-gcc-libs-core 5.3.0 7 conda-forge
m2w64-gmp 6.1.0 2 conda-forge
m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge
mapclassify 2.5.0 pyhd8ed1ab_1 conda-forge
markdown-it-py 3.0.0 pyhd8ed1ab_0 conda-forge
markupsafe 2.1.3 py311ha68e1ae_0 conda-forge
matplotlib-base 3.7.1 py311h6e989c2_0 conda-forge
matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge
mdurl 0.1.0 pyhd8ed1ab_0 conda-forge
mistune 3.0.0 pyhd8ed1ab_0 conda-forge
mkl 2022.1.0 h6a75c08_874 conda-forge
msgpack-python 1.0.5 py311h005e61a_0 conda-forge
msys2-conda-epoch 20160418 1 conda-forge
multidict 6.0.4 py311ha68e1ae_0 conda-forge
multipledispatch 0.6.0 py_0 conda-forge
munch 4.0.0 pyhd8ed1ab_0 conda-forge
munkres 1.1.4 pyh9f0ad1d_0 conda-forge
nbclient 0.8.0 pyhd8ed1ab_0 conda-forge
nbconvert-core 7.7.3 pyhd8ed1ab_0 conda-forge
nbformat 5.9.2 pyhd8ed1ab_0 conda-forge
nest-asyncio 1.5.6 pyhd8ed1ab_0 conda-forge
networkx 3.1 pyhd8ed1ab_0 conda-forge
noise 1.2.2 py311ha68e1ae_1004 conda-forge
notebook-shim 0.2.3 pyhd8ed1ab_0 conda-forge
numba 0.57.1 py311h2c0921f_0 conda-forge
numpy 1.24.4 py311h0b4df5a_0 conda-forge
openjpeg 2.5.0 ha2aaf27_2 conda-forge
openssl 3.1.2 hcfcfb64_0 conda-forge
orc 1.9.0 hf2b8f0d_1 conda-forge
overrides 7.4.0 pyhd8ed1ab_0 conda-forge
packaging 23.1 pyhd8ed1ab_0 conda-forge
pandas 2.0.3 py311hf63dbb6_1 conda-forge
pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge
param 1.13.0 pyh1a96a4e_0 conda-forge
parso 0.8.3 pyhd8ed1ab_0 conda-forge
partd 1.4.0 pyhd8ed1ab_0 conda-forge
pcre2 10.40 h17e33f8_0 conda-forge
pickleshare 0.7.5 py_1003 conda-forge
pillow 10.0.0 py311hde623f7_0 conda-forge
pip 23.2.1 py311haa95532_0
pixman 0.40.0 h8ffe710_0 conda-forge
pkgutil-resolve-name 1.3.10 pyhd8ed1ab_0 conda-forge
platformdirs 3.10.0 pyhd8ed1ab_0 conda-forge
pooch 1.7.0 pyha770c72_3 conda-forge
poppler 23.05.0 h45d20d0_1 conda-forge
poppler-data 0.4.12 hd8ed1ab_0 conda-forge
postgresql 15.4 hc80876b_0 conda-forge
proj 9.2.1 h660b3b0_0 conda-forge
prometheus_client 0.17.1 pyhd8ed1ab_0 conda-forge
prompt-toolkit 3.0.39 pyha770c72_0 conda-forge
prompt_toolkit 3.0.39 hd8ed1ab_0 conda-forge
psutil 5.9.5 py311ha68e1ae_0 conda-forge
pthread-stubs 0.4 hcd874cb_1001 conda-forge
pthreads-win32 2.9.1 hfa6e2cd_3 conda-forge
pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge
pyarrow 12.0.1 py311h6a6099b_8_cpu conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pyct 0.4.6 py_0 conda-forge
pyct-core 0.4.6 py_0 conda-forge
pygeos 0.14 py311h343093d_1 conda-forge
pygments 2.16.1 pyhd8ed1ab_0 conda-forge
pyopenssl 23.2.0 pyhd8ed1ab_1 conda-forge
pyparsing 3.1.1 pyhd8ed1ab_0 conda-forge
pyproj 3.6.0 py311hcff2a09_1 conda-forge
pysocks 1.7.1 pyh0701188_6 conda-forge
pystac 1.8.3 pyhd8ed1ab_0 conda-forge
python 3.11.4 he1021f5_0
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-fastjsonschema 2.18.0 pyhd8ed1ab_0 conda-forge
python-json-logger 2.0.7 pyhd8ed1ab_0 conda-forge
python-tzdata 2023.3 pyhd8ed1ab_0 conda-forge
python_abi 3.11 2_cp311 conda-forge
pytz 2023.3 pyhd8ed1ab_0 conda-forge
pywin32 304 py311h12c1d0e_2 conda-forge
pywinpty 2.0.11 py311h12c1d0e_0 conda-forge
pyyaml 6.0 py311ha68e1ae_5 conda-forge
pyzmq 25.1.1 py311h7b3f143_0 conda-forge
rasterio 1.3.8 py311h58bba51_0 conda-forge
re2 2023.03.02 hd4eee63_0 conda-forge
referencing 0.30.2 pyhd8ed1ab_0 conda-forge
requests 2.31.0 pyhd8ed1ab_0 conda-forge
rfc3339-validator 0.1.4 pyhd8ed1ab_0 conda-forge
rfc3986-validator 0.1.1 pyh9f0ad1d_0 conda-forge
rich 13.5.1 pyhd8ed1ab_0 conda-forge
rioxarray 0.14.1 pyhd8ed1ab_0 conda-forge
rpds-py 0.9.2 py311hc37eb10_0 conda-forge
rtree 1.0.1 py311hcacb13a_2 conda-forge
s3transfer 0.6.1 pyhd8ed1ab_0 conda-forge
scikit-learn 1.3.0 py311h142b183_0 conda-forge
scipy 1.11.1 py311h37ff6ca_0 conda-forge
send2trash 1.8.2 pyh08f2357_0 conda-forge
setuptools 68.0.0 py311haa95532_0
shapely 2.0.1 py311h343093d_1 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
snappy 1.1.10 hfb803bf_0 conda-forge
sniffio 1.3.0 pyhd8ed1ab_0 conda-forge
snuggs 1.4.7 py_0 conda-forge
sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge
soupsieve 2.3.2.post1 pyhd8ed1ab_0 conda-forge
sqlite 3.41.2 h2bbff1b_0
stack_data 0.6.2 pyhd8ed1ab_0 conda-forge
tbb 2021.10.0 h91493d7_0 conda-forge
tblib 1.7.0 pyhd8ed1ab_0 conda-forge
terminado 0.17.0 pyh08f2357_0 conda-forge
threadpoolctl 3.2.0 pyha21a80b_0 conda-forge
tiledb 2.13.2 h3132609_0 conda-forge
tinycss2 1.2.1 pyhd8ed1ab_0 conda-forge
tk 8.6.12 h2bbff1b_0
tomli 2.0.1 pyhd8ed1ab_0 conda-forge
toolz 0.12.0 pyhd8ed1ab_0 conda-forge
tornado 6.3.2 py311ha68e1ae_0 conda-forge
traitlets 5.9.0 pyhd8ed1ab_0 conda-forge
typing-extensions 4.7.1 hd8ed1ab_0 conda-forge
typing_extensions 4.7.1 pyha770c72_0 conda-forge
typing_utils 0.1.0 pyhd8ed1ab_0 conda-forge
tzdata 2023c h04d1e81_0
ucrt 10.0.22621.0 h57928b3_0 conda-forge
uri-template 1.3.0 pyhd8ed1ab_0 conda-forge
urllib3 1.26.15 pyhd8ed1ab_0 conda-forge
vc 14.2 h21ff451_1
vc14_runtime 14.36.32532 hfdfe4a8_17 conda-forge
vs2015_runtime 14.36.32532 h05e6639_17 conda-forge
wcwidth 0.2.6 pyhd8ed1ab_0 conda-forge
webcolors 1.13 pyhd8ed1ab_0 conda-forge
webencodings 0.5.1 py_1 conda-forge
websocket-client 1.6.1 pyhd8ed1ab_0 conda-forge
wheel 0.38.4 py311haa95532_0
widgetsnbextension 4.0.8 pyhd8ed1ab_0 conda-forge
win_inet_pton 1.1.0 pyhd8ed1ab_6 conda-forge
winpty 0.4.3 4 conda-forge
xarray 2023.7.0 pyhd8ed1ab_0 conda-forge
xarray-spatial 0.3.5 pyhd8ed1ab_0 conda-forge
xerces-c 3.2.4 h63175ca_2 conda-forge
xorg-libxau 1.0.11 hcd874cb_0 conda-forge
xorg-libxdmcp 1.1.3 hcd874cb_0 conda-forge
xyzservices 2023.7.0 pyhd8ed1ab_0 conda-forge
xz 5.4.2 h8cc25b3_0
yaml 0.2.5 h8ffe710_2 conda-forge
yarl 1.9.2 py311ha68e1ae_0 conda-forge
zeromq 4.3.4 h0e60522_1 conda-forge
zict 3.0.0 pyhd8ed1ab_0 conda-forge
zipp 3.16.2 pyhd8ed1ab_0 conda-forge
zlib 1.2.13 hcfcfb64_5 conda-forge
zstd 1.5.2 h12be248_7 conda-forge