numpy.core._exceptions._ArrayMemoryError error when writing a large numpy memmap array with Rasterio

222 Views Asked by At

I have a very large numpy memmap array, with the size of (4, 57743, 547583). I want to write this array into a GeoTiff file by using rasterio. I am using the following code:

    with rasterio.open(
            world_image_geotiff_path,
            'w',
            driver='GTiff',
            height=self.worldImageShape[0],
            width=self.worldImageShape[1],
            count=4,
            dtype='uint8',
            crs="EPSG:3857",
            transform=world_image_transformation,
            compress="JPEG",
            blockxsize=256,
            blockysize=256,
            photometric=None,
            tiled=True,
            BIGTIFF="YES",
            ALPHA="YES",
            JPEG_QUALITY=jpeg_quality) as georeferenced_image:
        georeferenced_image.colorinterp = [ColorInterp.red, ColorInterp.green, ColorInterp.blue, ColorInterp.alpha]
        world_image = np.transpose(world_image, axes=(2, 0, 1))
        georeferenced_image.write(arr=world_image, indexes=[3, 2, 1, 4])
        georeferenced_image.build_overviews([2, 4, 8, 16, 32, 64, 128], Resampling.average)
        georeferenced_image.update_tags(ns='rio_overview', resampling='average')

I get the following error:

numpy.core._exceptions._ArrayMemoryError: Unable to allocate 118. GiB for an array with shape (4, 57743, 547583) and data type uint8

When it tries to execute georeferenced_image.write(arr=world_image, indexes=[3, 2, 1, 4]). It obviously tries to create new a Numpy array of the proper size in order to write into from the source array. However Rasterio does seem to create this huge array in memory and seems to fail naturally (at least this is my guess). What would be a work around for this? Can I write into a Rasterio database with blocks, or is there any way to tell it that it should also work with a memmap array like the given source array? The documentation about write() function does not mention about that, so I got stuck in this point.

0

There are 0 best solutions below