How to connect OpenSearch Serverless via Bastion

Question

I'm trying to understand how to connect to my OpenSearch Serverless instance using Bastion EC2 instance. I have a VPC Endpoint connected to my OpenSearch instance, and I want to connect to that instance from my local machine using a Bastion host.

This is the network policies:

Thanks!

Answer 1

I couldn't reach the solution, but I can give you the steps I followed to see if you can get the issue solved.

First, I created a connection to the Bastion host using paramiko:

from paramiko import SSHClient

proxy_client = SSHClient()
proxy_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())

proxy_client.connect(
   proxy_hostname,
   port=proxy_port,
   username=proxy_username,
   key_filename='bastion-keypair.pem'
)

Once you're able to connect with your bastion what I did is to create a tunnel from my local machine to the actual OpenSearch host.

bastion_transport = proxy_client.get_transport()
source_address = ('localhost', 10000)
destination_address = (opensearch_host, opensearch_port)
    
bastion_channel = bastion_transport.open_channel("direct-tcpip", destination_address, source_address)

Now it's supposed that I've enabled this tunnel between my computer and opensearch, and if I call to my localhost:10000 then I'll be accessing to my OpenSearch.

from opensearchpy import OpenSearch as OpenSearchClass
from opensearchpy import RequestsHttpConnection, AWSV4SignerAuth

opensearch_client = OpenSearchClass(
   hosts=[{
       "host": "localhost",
       "port": 10000
   }],
   http_auth=AWSV4SignerAuth(
       credentials=aws_session.get_credentials(),
       region=region_name
   ),
   use_ssl=True,
   verify_certs=True,
   connection_class = RequestsHttpConnection
)

Then finally you can do a ping to your opensearch_client to see if it's working

opensearch_client.ping() #This should return True

Hope this solution works fine for you :D or at least gives you a thought.

Answer 2

I want to also share another way of connecting to AWS OpenSearch through AWS Bastion Host, but in this case from the CLI.

First, you've to open a CLI and start a connection to your AWS Bastion host, for reach that, you should execute the following command.

ssh -i $secret_name.pem $ec2_username@$ec2_public_ip -L $local_port:$opensearch_host:$opensearch_port

This will forward OpenSearch to the local port you want to specify, just notice that by default opensearch uses port 443 and ec2 username is by default ec2-user, but this can be different if you've configured any other values.

Finally, you've to open another CLI and get logged in AWS, and then you''re able to execute queries inside your OpenSearch instance from your CLI, using this following structure as base command.

curl -u $AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY https://localhost:$local_port/$query --header "x-amz-security-token: ${AWS_SESSION_TOKEN}" --aws-sigv4 aws:amz:eu-west-1:es

Something important is that $AWS_ACCESS_KEY_ID, $AWS_SECRET_ACCESS_KEY and $AWS_SESSION_TOKEN aren't user specified parameters, they are taken from the logged in AWS session you should previously did.

Finally, I'm going to let some useful queries that I'd built but also, you can check the OpenSearch documentation to see the different queries you can send to OpenSearch, please take a look: OpenSearch Official Site

List Indices

curl -u $AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY https://localhost:$local_port/_cat/indices --header "x-amz-security-token: ${AWS_SESSION_TOKEN}" --aws-sigv4 aws:amz:eu-west-1:es

Export Index Mapping

curl -u $AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY https://localhost:$local_port/${index_name} --header "x-amz-security-token: ${AWS_SESSION_TOKEN}" --aws-sigv4 aws:amz:eu-west-1:es | jq '.${index_name}.mappings.properties' | jq '{mappings: {properties: (to_entries | map(select(.value.type=="date")) | from_entries)}}' > ${index_name}_mapping.json

Note: I'm selecting only date columns and the result is formatted for recreate the same index again

Export Data

curl -u $AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY https://localhost:$local_port/${index_name}/_search --header "x-amz-security-token: ${AWS_SESSION_TOKEN}" --aws-sigv4 aws:amz:eu-west-1:es | jq '.hits.hits' | jq '.[]' | jq -c '{index: {_id: ._id}}, ._source' > ${index_name}.json

Note: The format of the exported data is for use with _bulk method

Create Index (Using previous exported index mapping)

curl -X PUT -u $AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY https://localhost:$local_port/${new_index_name} --header "x-amz-security-token: ${AWS_SESSION_TOKEN}" --aws-sigv4 aws:amz:eu-west-1:es -H 'Content-Type: application/json' -d @${index_name}_mapping.json

Import Data (Using previous exported index data)

curl -X POST-u $AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY https://localhost:$local_port/${index_name}/_bulk --header "x-amz-security-token: ${AWS_SESSION_TOKEN}" --aws-sigv4 aws:amz:eu-west-1:es -H 'Content-Type: application/json' --data-binary @${index_name}.json

Delete Index

curl -X DELETE -u $AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY https://localhost:$local_port/${index_name} --header "x-amz-security-token: ${AWS_SESSION_TOKEN}" --aws-sigv4 aws:amz:eu-west-1:es

Hope you can find this information useful :)