HTTP Code 400 getting objectfrom AWS S3 bucket with Boto3 in python

Question

I'm working on a script that is supposed to download a file from my s3 bucket and then make some changes on it. The problem is that my script runs well when executed as root but does not when executed with the dedicated user.

Here is the piece of code we are interested in :

s3 = boto3.client('s3', aws_access_key_id='xxx', aws_secret_access_key='xxx')

s3.download_file('my-bucket', 'my_file.csv', '/my/path/to/file.csv')

When I run the script as root, no problem, the file is downloaded and the script continues, but executed with my dedicated user :

logstash:x:501:501::/home/logstash:/sbin/nologin, it doesn't work.

I tried running Python interpreter as root and as logstash and executed the script command by command, it works as root but not as logstash. Here is the error :

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/site-packages/boto3/s3/inject.py", line 126, in download_file
extra_args=ExtraArgs, callback=Callback)
File "/usr/local/lib/python2.7/site-packages/boto3/s3/transfer.py", line 299, in download_file
future.result()
File "/usr/local/lib/python2.7/site-packages/s3transfer/futures.py", line 73, in result
return self._coordinator.result()
File "/usr/local/lib/python2.7/site-packages/s3transfer/futures.py", line 233, in result
raise self._exception
botocore.exceptions.ClientError: An error occurred (400) when calling the HeadObject operation: Bad Request

I searched a lot on google but there is no topic about this special 'user switch' case.

Answer 1

I'm first :)

Are the permissions for the user logstash set correctly for the directory you are downloading into?

Did you by any chance miss the -R in chown -R logstash?

Answer 2

Just in case someone else runs into this error for the same reason I did: For me the fix is to supply a region when initializing the s3-client like for example:

    s3 = boto3.client("s3", region_name="eu-central-1")

Where "eu-central-1" needs to be replaced by the region where the bucket really is. It looks like it is this bug: https://github.com/boto/botocore/issues/1413, though it sounds like it should be fixed. Just for reference the two errors I encountered due to this region not being set are

botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid

And the one already stated in the question:

botocore.exceptions.ClientError: An error occurred (400) when calling the HeadObject operation: Bad Request

Interestingly some calls did work in "eu-central-1" without that, while the same requests failed in China for "cn-north-1".

Answer 3

Had the same error. The problem was in the incorrect path to the file