Running AWS S3 client functions seems to have issues on my end and is taking up so much memory (generating "Allowed memory size of 3670016000 bytes exhausted"). I have the following code snippet below
Things that I have tried:
- Have the version set to "2006-03-01"
- Placing a false endpoint (this leads to a host error output)
- A different Region (Region error output)
I would assume the client instance works well given those responses when the config is changed
I currently have 2 spaces (1 is empty and 1 has about 60gb worth of files - both are in the same region and account)
public $config = [
's3-access' => [
'key' => 'some key',
'secret' => 'some secret',
'region' => 'ap-southeast-1',
'version' => 'latest',
'endpoint' => 'sgp1.digitaloceanspaces.com',
]
];
public function createInstance(){
$client = new S3Client([
'credentials' => [
'key' => $this->config['s3-access']['key'],
'secret' => $this->config['s3-access']['secret']
],
'version' => $this->config['s3-access']['version'],
'region' => $this->config['s3-access']['region'],
'endpoint' => $this->config['s3-access']['endpoint']
]);
//Blank array result
$spaces = $client->listBuckets();
foreach ($spaces['Buckets'] as $space){
echo $space['Name']."\n";
}
//Memory size error exception with peak memory usage to 2813mb, allocated 3500mb
$client->createBucket([
'Bucket' => 'test-space-102201',
]);
//list objects also generated the memory error
}
Edit: After turing on debug mode on the AWS S3 config i got the following response
-> Entering step init, name 'idempotency_auto_fill' --------------------------------------------------- command was set to array(3) { ["instance"]=> string(32) "000000007fb4cf920000000012b9d6de" ["name"]=> string(11) "ListBuckets" ["params"]=> array(2) { ["@http"]=> array(1) { ["debug"]=> resource(307) of type (stream) } ["@context"]=> array(0) { } } } request was set to array(0) { } -> Entering step init, name 's3.ssec' ------------------------------------- no changes -> Entering step init, name 's3.source_file' -------------------------------------------- no changes -> Entering step init, name 's3.save_as' ---------------------------------------- no changes -> Entering step init, name 's3.location' ----------------------------------------- no changes -> Entering step init, name 's3.auto_encode' -------------------------------------------- no changes -> Entering step init, name 's3.head_object' -------------------------------------------- no changes -> Entering step validate, name 'validation' -------------------------------------------- no changes -> Entering step validate, name 'input_validation_middleware' ------------------------------------------------------------- no changes -> Entering step build, name 'builder' -------------------------------------- request.instance was set to 000000007fb4cf300000000012b9d6de request.method was set to GET request.headers was set to array(1) { ["X-Amz-Security-Token"]=> string(7) "[TOKEN]" } request.path was set to sgp1.digitaloceanspaces.com -> Entering step build, name 'ApiCallMonitoringMiddleware' ---------------------------------------------------------- no changes -> Entering step build, name '' ------------------------------- request.instance changed from 000000007fb4cf300000000012b9d6de to 000000007fb4cf2c0000000012b9d6de request.headers.User-Agent was set to array(1) { [0]=> string(56) "aws-sdk-php/3.178.6 OS/Linux/3.10.0-1127.19.1.el7.x86_64" } -> Entering step build, name 'endpoint_parameter' ------------------------------------------------- no changes -> Entering step build, name 's3.checksum' ------------------------------------------ no changes -> Entering step build, name 's3.content_type' ---------------------------------------------- no changes -> Entering step build, name 's3.endpoint_middleware' ----------------------------------------------------- no changes -> Entering step build, name 's3.bucket_endpoint_arn' ----------------------------------------------------- no changes -> Entering step sign, name 'StreamRequestPayloadMiddleware' ------------------------------------------------------------ no changes -> Entering step sign, name 'invocation-id' ------------------------------------------- request.instance changed from 000000007fb4cf2c0000000012b9d6de to 000000007fb4cf280000000012b9d6de request.headers.aws-sdk-invocation-id was set to array(1) { [0]=> string(32) "03e4030ef1f4fd18f7d38b8265b34b7f" } -> Entering step sign, name 'retry' ----------------------------------- request.instance changed from 000000007fb4cf280000000012b9d6de to 000000007fb4cf270000000012b9d6de request.headers.aws-sdk-retry was set to array(1) { [0]=> string(3) "0/0" } -> Entering step sign, name 'signer' ------------------------------------ request.instance changed from 000000007fb4cf270000000012b9d6de to 000000007fb4cf340000000012b9d6de request.headers.x-amz-content-sha256 was set to array(1) { [0]=> string(64) "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" } request.headers.X-Amz-Date was set to array(1) { [0]=> string(16) "20210423T110304Z" } request.headers.Authorization was set to array(1) { [0]=> string(211) "AWS4-HMAC-SHA256 Credential=[KEY]/20210423/ap-southeast-1/s3/aws4_request, SignedHeaders=x-amz-content-sha256;x-amz-date, Signature=[SIGNATURE] } -> Entering step sign, name 's3.put_object_url' ----------------------------------------------- no changes -> Entering step sign, name 's3.permanent_redirect' --------------------------------------------------- no changes -> Entering step attempt, name 'ApiCallAttemptMonitoringMiddleware' ------------------------------------------------------------------- no changes * Trying 103.253.144.208:80... * Connected to sgp1.digitaloceanspaces.com (103.253.144.208) port 80 (#0) > GET / HTTP/1.1 Host: sgp1.digitaloceanspaces.com aws-sdk-invocation-id: 03e4030ef1f4fd18f7d38b8265b34b7f aws-sdk-retry: 0/0 x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 X-Amz-Date: 20210423T110304Z Authorization: AWS4-HMAC-SHA256 Credential=[KEY]/20210423/ap-southeast-1/s3/aws4_request, SignedHeaders=x-amz-content-sha256;x-amz-date, Signature=[SIGNATURE] User-Agent: aws-sdk-php/3.178.6 OS/Linux/3.10.0-1127.19.1.el7.x86_64 GuzzleHttp/7 * Mark bundle as not supporting multiuse < HTTP/1.1 302 Found < content-length: 0 < location: https://sgp1.digitaloceanspaces.com/ < cache-control: no-cache < * Connection #0 to host sgp1.digitaloceanspaces.com left intact * Trying 103.253.144.208:443... * Connected to sgp1.digitaloceanspaces.com (103.253.144.208) port 443 (#1) * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/pki/tls/certs/ca-bundle.crt * CApath: none * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 * ALPN, server did not agree to a protocol * Server certificate: * subject: C=US; ST=New York; L=New York; O=DigitalOcean, LLC; CN=*.sgp1.digitaloceanspaces.com * start date: Nov 6 00:00:00 2020 GMT * expire date: Dec 7 23:59:59 2021 GMT * subjectAltName: host "sgp1.digitaloceanspaces.com" matched cert's "sgp1.digitaloceanspaces.com" * issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1 * SSL certificate verify ok. > GET / HTTP/1.1 Host: sgp1.digitaloceanspaces.com aws-sdk-invocation-id: 03e4030ef1f4fd18f7d38b8265b34b7f aws-sdk-retry: 0/0 x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 X-Amz-Date: 20210423T110304Z User-Agent: aws-sdk-php/3.178.6 OS/Linux/3.10.0-1127.19.1.el7.x86_64 GuzzleHttp/7 * old SSL session ID is stale, removing * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < date: Fri, 23 Apr 2021 11:03:07 GMT < content-length: 215 < content-type: text/xml; charset=utf-8 < strict-transport-security: max-age=15552000; includeSubDomains; preload < * Connection #1 to host sgp1.digitaloceanspaces.com left intact <- Leaving step attempt, name 'ApiCallAttemptMonitoringMiddleware' ------------------------------------------------------------------ result was set to array(2) { ["instance"]=> string(32) "000000007fb4cf360000000012b9d6de" ["data"]=> array(3) { ["Buckets"]=> array(0) { } ["Owner"]=> array(2) { ["DisplayName"]=> string(0) "" ["ID"]=> string(9) "anonymous" } ["@metadata"]=> array(4) { ["statusCode"]=> int(200) ["effectiveUri"]=> string(27) "sgp1.digitaloceanspaces.com" ["headers"]=> array(4) { ["date"]=> string(29) "Fri, 23 Apr 2021 11:03:07 GMT" ["content-length"]=> string(3) "215" ["content-type"]=> string(23) "text/xml; charset=utf-8" ["strict-transport-security"]=> string(44) "max-age=15552000; includeSubDomains; preload" } ["transferStats"]=> array(0) { } } } } Inclusive step time: 0.024339914321899 <- Leaving step sign, name 's3.permanent_redirect' -------------------------------------------------- no changes Inclusive step time: 0.024394989013672 <- Leaving step sign, name 's3.put_object_url' ---------------------------------------------- no changes Inclusive step time: 0.02444314956665 <- Leaving step sign, name 'signer' ----------------------------------- no changes Inclusive step time: 0.024537086486816 <- Leaving step sign, name 'retry' ---------------------------------- no changes Inclusive step time: 0.026021003723145 <- Leaving step sign, name 'invocation-id' ------------------------------------------ [email protected] was set to array(1) { [0]=> array(0) { } } Inclusive step time: 0.026099920272827 <- Leaving step sign, name 'StreamRequestPayloadMiddleware' ----------------------------------------------------------- no changes Inclusive step time: 0.026165008544922 <- Leaving step build, name 's3.bucket_endpoint_arn' ---------------------------------------------------- no changes Inclusive step time: 0.026195049285889 <- Leaving step build, name 's3.endpoint_middleware' ---------------------------------------------------- no changes Inclusive step time: 0.026221990585327 <- Leaving step build, name 's3.content_type' --------------------------------------------- no changes Inclusive step time: 0.026258945465088 <- Leaving step build, name 's3.checksum' ----------------------------------------- no changes Inclusive step time: 0.026288986206055 <- Leaving step build, name 'endpoint_parameter' ------------------------------------------------ no changes Inclusive step time: 0.026321887969971 <- Leaving step build, name '' ------------------------------ no changes Inclusive step time: 0.026376008987427 <- Leaving step build, name 'ApiCallMonitoringMiddleware' --------------------------------------------------------- no changes Inclusive step time: 0.026448011398315 <- Leaving step build, name 'builder' ------------------------------------- no changes Inclusive step time: 0.027688980102539 <- Leaving step validate, name 'input_validation_middleware' ------------------------------------------------------------ no changes Inclusive step time: 0.028922080993652 <- Leaving step validate, name 'validation' ------------------------------------------- no changes Inclusive step time: 0.028947114944458 <- Leaving step init, name 's3.head_object' ------------------------------------------- no changes Inclusive step time: 0.028982877731323 <- Leaving step init, name 's3.auto_encode' ------------------------------------------- no changes Inclusive step time: 0.029003858566284 <- Leaving step init, name 's3.location' ---------------------------------------- no changes Inclusive step time: 0.02902889251709 <- Leaving step init, name 's3.save_as' --------------------------------------- no changes Inclusive step time: 0.029052972793579 <- Leaving step init, name 's3.source_file' ------------------------------------------- no changes Inclusive step time: 0.029078006744385 <- Leaving step init, name 's3.ssec' ------------------------------------ no changes Inclusive step time: 0.029115915298462 <- Leaving step init, name 'idempotency_auto_fill' -------------------------------------------------- no changes Inclusive step time: 0.029443979263306