i/o timeout issue when receiving the message on SQS and memory leak https usage

Question

AWS SDK Version: v1.38.19

Go Version: go1.15.7

alpine 3.7

I am using the standard queue I am initializing the SQS connection at once in my application like this;

// Connection connection to the SQS
var Connection *sqs.SQS

// InitSQS initialize the AWS SQS connection
func InitSQS() {
    sess := session.Must(session.NewSessionWithOptions(session.Options{
        SharedConfigState: session.SharedConfigEnable,
    }))

    Connection = sqs.New(sess, &aws.Config{
        Region:     aws.String("eu-west-1"),
        DisableSSL: aws.Bool(true),
    })
}

I am disabling the SSL because; I am having memory and CPU leaks when I am going with the SSL in my application (my application isn't open for the rest of the world btw it's an internal service for my other applications).

Here is the config I use to read a message from SQS:

func ConsumeUpdateMessage(db *database.MySQLWrap, sqsApi queue.SQSAPI) error {
    result, err := sqsApi.ReceiveMessage(&sqs.ReceiveMessageInput{
        AttributeNames: []*string{
            aws.String(sqs.MessageSystemAttributeNameSentTimestamp),
        },
        MessageAttributeNames: []*string{
            aws.String(sqs.QueueAttributeNameAll),
        },
        QueueUrl:            &qURL,
        MaxNumberOfMessages: aws.Int64(10),
        WaitTimeSeconds:     aws.Int64(20),
    })

    if err != nil {
        return fmt.Errorf("error on receiving the message from queue: %s", err.Error())
    }

    for _, msg := range result.Messages {
            // business logic
        }

    return err
}

this is how I am calling the ConsumeUpdateMessage method;

// InitializeUpdateMessage ..
func InitializeUpdateMessage(db *database.MySQLWrap, sqsApi queue.SQSAPI) {
    go func() {
        for {
            time.Sleep(500 * time.Millisecond)

            err := ConsumeUpdateMessage(db, sqsApi)

            if err != nil {
                log.Error(err)

                continue
            }
        }
    }()
}

but sometimes my subscriber return error like this;

*awserr.baseError: RequestError: send request failed
caused by: Post "http://sqs.eu-west-1.amazonaws.com/": dial tcp xx.x.xx.xxx:80: i/o timeout

(Note: I put xx instead of sharing the IP Address)

I made my search over the forums and other places but I can't find a solution for these two-issue,

1. Memory leak when I using the connection with SSL (idea: but since my application is an internal service I think I don't have to use with SSL)
2. i/o timeout

Answer 1

This is a very late answer, but I ran into a similar issue and I think I discovered what the problem was: You set a WaitTimeSeconds of 20 seconds for long polling (which is a good idea), but the HTTP client times out earlier than that.

The documentation for WaitTimeSeconds says:

// To avoid HTTP errors, ensure that the HTTP response timeout for ReceiveMessage
// requests is longer than the WaitTimeSeconds parameter. For example, with
// the Java SDK, you can set HTTP transport settings using the NettyNioAsyncHttpClient
// (https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/http/nio/netty/NettyNioAsyncHttpClient.html)
// for asynchronous clients, or the ApacheHttpClient (https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/http/apache/ApacheHttpClient.html)
// for synchronous clients.

At least, as of v1.44.39 of aws-sdk-go.

The following page describes how to configure a custom HTTP client: https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/custom-http.html

I'll copy all of the code from that page into here, for posterity:

type HTTPClientSettings struct {
    Connect          time.Duration
    ConnKeepAlive    time.Duration
    ExpectContinue   time.Duration
    IdleConn         time.Duration
    MaxAllIdleConns  int
    MaxHostIdleConns int
    ResponseHeader   time.Duration
    TLSHandshake     time.Duration
}

func NewHTTPClientWithSettings(httpSettings HTTPClientSettings) (*http.Client, error) {
    var client http.Client
    tr := &http.Transport{
        ResponseHeaderTimeout: httpSettings.ResponseHeader,
        Proxy:                 http.ProxyFromEnvironment,
        DialContext: (&net.Dialer{
            KeepAlive: httpSettings.ConnKeepAlive,
            DualStack: true,
            Timeout:   httpSettings.Connect,
        }).DialContext,
        MaxIdleConns:          httpSettings.MaxAllIdleConns,
        IdleConnTimeout:       httpSettings.IdleConn,
        TLSHandshakeTimeout:   httpSettings.TLSHandshake,
        MaxIdleConnsPerHost:   httpSettings.MaxHostIdleConns,
        ExpectContinueTimeout: httpSettings.ExpectContinue,
    }

    // So client makes HTTP/2 requests
    err := http2.ConfigureTransport(tr)
    if err != nil {
        return &client, err
    }

    return &http.Client{
        Transport: tr,
    }, nil
}

httpClient, err := NewHTTPClientWithSettings(HTTPClientSettings{
    Connect:          5 * time.Second,
    ExpectContinue:   1 * time.Second,
    IdleConn:         90 * time.Second,
    ConnKeepAlive:    30 * time.Second,
    MaxAllIdleConns:  100,
    MaxHostIdleConns: 10,
    ResponseHeader:   5 * time.Second,
    TLSHandshake:     5 * time.Second,
})
if err != nil {
    fmt.Println("Got an error creating custom HTTP client:")
    fmt.Println(err)
    return
}

sess := session.Must(session.NewSession(&aws.Config{
    HTTPClient: httpClient,
}))

sqsClient := sqs.New(sess)

You will want to configure the timeout settings. In particular, I set Connect, TLSHandshake, and ResponseHeader to 25 seconds and no longer get any errors.