CloudRun cold start issue and 429 Rate exceeded error

Question

We have a nodejs monorepo project with >179 packages where each package may have >30 files. It also contains proxy with routing and several forked processes (as usual). So, when we packed all of this stuff into Docker image and moved it into CloudRun (with min instance = 1, max instance >10, concurrency=1000), users randomly start see the error '429 Rate exceeded'. (As far as we understood from the documentation, it happens when 'max instances' limit is reached by CloudRun and it can not scale our application anymore. Indirect reason of that maybe too long cold start (which is also limited in CloudRun = 10s). We measured our cold start - it was ~20sec.

To identify the issue of cold start we used this package - https://www.npmjs.com/package/require-so-slow
It showed us that each our small module requires ~5ms for 'import' or 'require' modules. So, average calculation may show why do we have so long cold start: 170 packages * 30 files * 5ms > 25s

For monorepo we use pnmpm and each package builds with tsc.

So, the question is how to improve cold start in CloudRun?

Note: Locally, on dev environment on laptops, we do not have issue with cold start, only in CloudRun. So, looks like, this issue is a platform specific issue.

UPDATED: the best approach at this moment, for our case, is bundling all project (or partially) into single js file (with help of webpack or esbuild, like here) + Cloud Run 'cpu boost' option

Answer 1

...this is continuation for comments above...

3.2) Starting point for nodejs bundling can be this webpack.config.js:

const path = require('path');
const nodeExternals = require('webpack-node-externals');

module.exports =  (env) => {
  return {
  target: 'node',
  externals: [nodeExternals()],
    node: {
    global: false,
    __filename: false,
    __dirname: false,
  },
  entry: './src/index.ts',
  optimization: {
        minimize: false
    },
  module: {
    rules: [
      {
        test: /\.tsx?$/,
        loader: 'ts-loader',
        exclude: /node_modules/,
        options: {
          compilerOptions: {
              outDir: env[1].output
          }
        }
      },
    ],
  },
  resolve: {
    extensions: ['.tsx', '.ts', '.js'],
  },
  output: {
    filename: 'index.js',
    pathinfo: false,
    libraryTarget: 'commonjs2'
  },
}};

package.json

    "devDependencies": {
        "typescript": "4.5.4",
        "ts-loader": "8.0.3",
        "webpack": "4.33.0",
        "webpack-node-externals": "3.0.0"
    }

command to start bundling:

node_modules/.bin/webpack \
            --config=./webpack.config.js \
            --output-path="./dist" \
            --env=development \
            --env.output="./dist"

env.output - uses for emitting *.d.ts files

Answer 2

To improve cold starts in CloudRun. Please try to implement the points below .
when it comes to applications written in dynamic languages like node.js,etc. You should be aware of some modules that run initialization code upon importing.
Minimize the number and size of dependencies if you’re using a dynamic language.
Instead of computing things upon startup, compute them lazily. The initialization of global variables always occurs during startup, which increases cold start time. Use lazy initialization for infrequently used objects to defer the time cost and decrease cold start times. For the details you can check the documentation
For detailed description you can check Article and Document

Answer 3

1. Concurrency of 1,000 sounds optimistic. The default is 80, and I sometimes find that I need to lower that number for best performance. I would try setting concurrency back to its default, and remove the max-instances limit.

2. If your container takes a long time to start you may want to check out the new Container health check feature.