We are trying to create an app in GCP using postgresql as backend and nodejs/angular for application side . As part of our requirement we need to be able to upload files of around 1 million records atleast in the development environment .
We werent able to pass threshold of 20k records initially howver after updating the value of POSTGRES_POOL_ACQUIRE env variable to 30000 and then to eventually 2million we are now able to upload 90 to 95k records.Not exactly sure how that helps but it has atleast allowed us to increase our threshold failure. However increase the value of the env variable post that (2million) doesnt provide any additional benefit.
We have also increased the timeout to 10 mins
POSTGRES_DB=XXXX
POSTGRES_USER=XXXXX
POSTGRES_PASSWORD=XXXXX
POSTGRES_HOST=localhost
POSTGRES_PORT=XXXX
POSTGRES_DIALECT=postgres
POSTGRES_POOL_MAX=5
POSTGRES_POOL_MIN=0
POSTGRES_POOL_ACQUIRE=30000
POSTGRES_POOL_IDLE=10000
POSTGRES_OPERATOR_ALIASES=false
BCRYPT_NB = 5
What possible steps can we take further to be able to hit our upload target of 1 million records ? Do let us know if any additional information is required