Opening a file in pyfilesystem's s3 filesystem fails

Question

I'm trying to simply open a file in pyfilesystem's s3 filesystem as such:

>>> s3fs.listdir('dfparks/test')
['test.txt']
>>> s3fs.open('dfparks/test/test.txt')

I get the error:

fs.errors.ResourceNotFound: resource 'dfparks/test/test.txt' not found

I've tried every variant of the path / s3 url I can think of but I always get the not found error.

Answer 1

fs.s3fs by default works in strict mode, checking for existence of directory markers (keys ending with "/"). You can turn off strict mode by passing strict=False to S3FS constructor or append "strict=0" to url passed to fs.open_fs

But, unfortunately, S3FS still does these checks in few places even in strict=False mode. There is my PR waiting for review fixing that: https://github.com/PyFilesystem/s3fs/pull/60. In the meantime you can install fixed version using fs-s3fs-ng package:

pip install fs-s3fs-ng

Additionally it enables non-strict mode by default and turns on S3 streaming (other PR waiting for acceptance)

Please let me know if this works for you.