Access aws elastic search role based using python elastic search package

Question

I am using deepset/haystack and communicating with elastic search. Using OpenDistroElasticsearchDocumentStore method works fine with username,pasword access to aws elastic search. Doesnt seem to work with role based access when deployed in ec2. Please suggest me a solution to access aws elastic search using python elastic search package given a role access

Answer 1

Do you mean IAM based access on AWS like this? We just recently merged a feature that might help you here (#965). Please install the latest Haystack version from the master branch and try something along those lines:

import boto3

from requests_aws4auth import AWS4Auth
from haystack.document_store.elasticsearch import ElasticsearchDocumentStore
from elasticsearch import RequestsHttpConnection

host = '<vpc_host>'
port = 443
region = 'eu-central-1'
service = 'es'
 
credentials = boto3.Session().get_credentials()
aws4auth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)
 
document_store = OpenDistroElasticsearchDocumentStore(host=host,
                                            port=port,
                                            aws4auth=aws4auth,
                                            # can't be used with default es client version used in e.g. aws sagemaker
                                            embedding_field=None,
                                            index="document")

Answer 2

from requests_aws4auth import AWS4Auth
from botocore.session import Session
credentials = Session().get_credentials()
auth = AWS4Auth(region='eu-west-1', service='es', refreshable_credentials=credentials)

This example shows how to construct an AWS4Auth instance with automatically refreshing credentials, suitable for long-running applications using AWS IAM assume-role. The RefreshableCredentials instance is used to generate valid static credentials per-request, eliminating the need to recreate the AWS4Auth instance when temporary credentials expire.

Source: https://github.com/tedder/requests-aws4auth#dynamic-sts-credentials-using-botocore-refreshablecredentials

This was merged into AWS4Auth in May 2021.