How to Access VPC enabled Elasticsearch Using AWS Lambda function

Question

I am trying to access Elastic search cluster using lambda function. The problem I am facing is that it gives an error. The response is:

{
  "errorMessage": "Failed to parse: https://[https://vpc-myendpoint-us-east-1.es.amazonaws.com/]:443/_cluster/health",
  "errorType": "InvalidURL",
  "stackTrace": [
    "  File \"/var/task/lambda_function.py\", line 23, in lambda_handler\n    print(es.cluster.health())\n",
    "  File \"/var/task/elasticsearch/client/utils.py\", line 153, in _wrapped\n    return func(*args, params=params, headers=headers, **kwargs)\n",
    "  File \"/var/task/elasticsearch/client/cluster.py\", line 66, in health\n    return self.transport.perform_request(\n",
    "  File \"/var/task/elasticsearch/transport.py\", line 381, in perform_request\n    status, headers_response, data = connection.perform_request(\n",
    "  File \"/var/task/elasticsearch/connection/http_requests.py\", line 159, in perform_request\n    prepared_request = self.session.prepare_request(request)\n",
    "  File \"/var/task/requests/sessions.py\", line 456, in prepare_request\n    p.prepare(\n",
    "  File \"/var/task/requests/models.py\", line 316, in prepare\n    self.prepare_url(url, params)\n",
    "  File \"/var/task/requests/models.py\", line 384, in prepare_url\n    raise InvalidURL(*e.args)\n"
  ]
}

My code is in python and it follows as:

from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth
import boto3


def lambda_handler(event, context):
    region = 'us-east-1'
    host = 'https://vpc-myendpoint.us-east-1.es.amazonaws.com/'
    service = 'es'
    
    credentials = boto3.Session().get_credentials()
    awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service,
                       session_token=credentials.token)

    es = Elasticsearch(
        hosts=[{'host': host, 'port': 443}],
        http_auth= awsauth,
        use_ssl=True,
        verify_certs=True,
        connection_class=RequestsHttpConnection
    )

    print(es.cluster.health())

How can I solve this error. My guess is that VPC enabled endpoints use a different technique to be accessed. If so, how to access them?

Thanks in advance

Answer 1

Look at the error message:

"Failed to parse: https://[https://vpc-myendpoint-us-east-1.es.amazonaws.com/]:443/_cluster/health",

https://[https://vpc-myendpoint-us-east-1.es.amazonaws.com isn't a valid URL.

You are specifying a URL in a variable that expects a hostname.

Change this:

host = 'https://vpc-myendpoint.us-east-1.es.amazonaws.com/'

To this: host = 'vpc-myendpoint.us-east-1.es.amazonaws.com'