AWS Elasticsearch returning 403 forbidden error using ES client

Question

I'm trying to create an index via an indexing job written in Go. I have all the access to ES cluster on AWS and using my access key and secret key.

I can easily create indices using Kibana but when I try to use Go client, it does not work and returns a 403 forbidden error.

AWS Elasticsearch Policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:<region>:111111111111:domain/prod-elasticsearch/*"
    }
  ]
}

indexing.go

package main

import (
    "flag"
    "fmt"
    "log"

    "github.com/aws/aws-sdk-go/aws/credentials"
    "github.com/olivere/elastic/v7"
    "github.com/spf13/viper"

    aws "github.com/olivere/elastic/v7/aws/v4"
)

func main() {
    var (
        accessKey = viper.GetString("aws.access_key")
        secretKey = viper.GetString("aws.secret_key")
        url       = viper.GetString("elasticsearch.host")
        sniff     = flag.Bool("sniff", false, "Enable or disable sniffing")
        region    = flag.String("region", "ap-southeast-1", "AWS Region name")
    )

    if url == "" {
        log.Fatal("please specify a URL with -url")
    }
    if accessKey == "" {
        log.Fatal("missing -access-key or AWS_ACCESS_KEY environment variable")
    }
    if secretKey == "" {
        log.Fatal("missing -secret-key or AWS_SECRET_KEY environment variable")
    }
    if *region == "" {
        log.Fatal("please specify an AWS region with -region")
    }

    creds := credentials.NewStaticCredentials(accessKey, secretKey, "")
    _, err := creds.Get()

    if err != nil {
        log.Fatal("Wrong credentials: ", err)
    }

    signingClient := aws.NewV4SigningClient(creds, *region)

    // Create an Elasticsearch client
    client, err := elastic.NewClient(
        elastic.SetURL(url),
        elastic.SetSniff(*sniff),
        elastic.SetHealthcheck(*sniff),
        elastic.SetHttpClient(signingClient),
    )

    if err != nil {
        log.Fatal(err)
    }

    // This part gives 403 forbidden error
    indices, err := client.IndexNames()

    if err != nil {
        log.Fatal(err)
    }

    // Just a status message
    fmt.Println("Connection succeeded")
}

config.toml

[elasticsearch]
host = "https://vpc-prod-elasticsearch-111111.ap-southeast-1.es.amazonaws.com"

[aws]
access_key = <ACCESS_KEY>
secret_key = <SECRET_KEY>

Answer 1

The Go code looks good. The 403 error forbidden was coming up due to some issues in the AWS secret key.

To debug such an issue:

1. Check if the VPC subnet has been removed in which AWS Elasticsearch cluster is operational. If the VPC subnet has been removed, then it will always throw a 403 error.
2. If subnets configuration is fine, then it's most probably an issue with AWS secrets. You can create a new AWS application user for the AWS Elasticsearch and retry. This should work fine.

I solved the above issue following the above 2 points.