Pyathena in Jupyter Notebook Can't find aws config file

Question

I'm using a jupyter notebook, and I'm trying to query AWS athena using PyAthena module, via the notebook.

My first line in the project is:

athena = pyathena.connect(profile_name="NAME")

which works in any other platform on my computer (iPython, Pycharm..) but doesn't work on the Jupyter Notebook, and returns:

ProfileNotFound: The config profile (core) could not be found

I tried changing the env variable like so: !set AWS_CONFIG_FILE="~/.aws/config" but it didn't work.

Any ideas?

Answer 1

Most likely you are confusing the role of an AWS config and credential file (see the docs here: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html)

• You should attempt first to authenticate using the AWS CLI in bash with a very simple command to confirm your named profile is setup correctly.

For reference, you can use the ! (exclamation point) inside of a Jupyter Notebook cell to run bash commands. For reference see Jupyter Magics here: https://ipython.readthedocs.io/en/stable/interactive/magics.html#line-magics

!aws sts get-caller-identity --profile <YOUR_NAMED_PROFILE_HERE>

Assuming that returns a successful response, you should be able to use PyAthena like the following:

from pyathena import connect

conn = connect(profile_name='<YOUR_NAMED_PROFILE_HERE>',
               work_group='<YOUR_WORKGROUP_HERE>', 
               region_name='<YOUR_REGION_HERE>')

optionally you can do either work_group or the S3 bucket output location as s3_staging_dir

If that still fails, it's likely because you did not setup your named profile correctly in ~/.aws/credentials