webpack 5 maxing out on RAM when bundling for karma tests

927 Views Asked by At

I recently updated webpack to version 5, and now when compiling a bundle for karma tests my system is maxing out of RAM and I get the JavaScript heap out of memory error. This happens both on my local machine that has 8gb RAM, and also on a gitlab runner hosted on digital ocean which has 4gb RAM.

Locally, if I kill all other apps, and set the memory allocation options node --max-old-space-size=4096 testscript.js I can sometimes get past the bundling, so I think it currently requires around 4gb+ RAM just to bundle, and this worries me. On the gitlab runner, linux terminates the docker container and I get the "killed" error

Before updating to weback 5 and therefore also before the memory issue my webpack configuration looked a bit differently, because webpack 5 doesn't include node libs by default, I have had to add some polyfills.

These are new in the configuration

crypto: false,
fs: false,
http: false,
path: require.resolve('path-browserify'),
stream: require.resolve('stream-browserify')

and these:

new webpack.ProvidePlugin({ Buffer: ['buffer', 'Buffer'] }),
new webpack.ProvidePlugin({ process: 'process/browser' }),

This is what the final webpack configuration for the karma bundling looks like:

const path = require('path');
const webpack = require('webpack');
const { excludeNodeModulesExcept } = require('../src/webpack/config');
const moduleListToTranspileForIE11 = ['debug', 'loader-utils', 'query-string', 'split-on-first', 'strict-uri-encode'];
const devModuleListToTranspileForIE11 = ['fetch-mock', 'proxy-polyfill', 'sinon', 'tr46', 'webidl-conversions', 'whatwg-url'];

module.exports = {
  context: path.resolve(__dirname, '..'),
  devtool: 'source-map',
  entry: [
    './test/karma-webpack.main.js'
  ],
  externals: {
    'react/addons': true,
    'react/lib/ExecutionEnvironment': true,
    'react/lib/ReactContext': true
  },
  module: {
    rules: [
      {
        test: /\.css$/,
        use: [ 'css-loader', 'postcss-loader']
      },
      {
        test: /\.js$/,
        exclude: excludeNodeModulesExcept([].concat(devModuleListToTranspileForIE11, moduleListToTranspileForIE11)),
        use: {
          loader: 'babel-loader',
          options: {
            plugins: [
              ['@babel/plugin-transform-for-of', { assumeArray: true }]
            ],
            presets: [
              ['@babel/preset-env', {}]
            ],
            cacheDirectory: true // Note: for faster rebuilds
          }
        }
      },
      {
        test: /\.svg$/,
        loader: 'svg-react-loader'
      }
    ]
  },
  mode: process.env.NODE_ENV || 'development',
  resolveLoader: {
    modules: [
      'node_modules',
      path.resolve(__dirname, '../src/webpack/loader')
    ]
  },
  output: {
    path: path.resolve(__dirname, '../output/test'),
    publicPath: '/',
    filename: 'karma-webpack.js'
  },
  resolve: {
    fallback: {
      crypto: false,
      fs: false,
      http: false,
      path: require.resolve('path-browserify'),
      stream: require.resolve('stream-browserify')
    }
  },
  performance: {
    hints: false
  },
  plugins: [
    new webpack.ProvidePlugin({ Buffer: ['buffer', 'Buffer'] }),
    new webpack.ProvidePlugin({ process: 'process/browser' }),
    new webpack.IgnorePlugin(/style\/demoHack\/\w+\.css/),
    new webpack.IgnorePlugin(/app\.css/),
    new webpack.IgnorePlugin(/config\/output\/build\/clientConfig\.json/),
    new webpack.IgnorePlugin(/output\/local\/\w+\.json/)
  ]
};

The entry point ./test/karma-webpack.main.js looks like this:

require('./mocha.main');

var appReq = require.context('../src', true, /\.js$/);
appReq.keys().filter(function (key) {
  return ['./react.main.js', './src/helper/polyfill.js'].indexOf(key) === -1 && !key.startsWith('./tool') && !key.startsWith('./src/test');
}).map(appReq);

// Only start mocha in browser.
if (!window.__karma__) {
  window.mocha.run();
}

and is based on: https://github.com/FormidableLabs/full-stack-testing/blob/master/test/client/main.js

This seems to be including all mocha (file.test.js) files in the bundling, which makes it very slow, as those files also includes a lot of devDependencies. However, this was not a problem prior to the update. I have tried filtering out all files that are .test.js - then the bundling is fast, and I don't see any memory issues, however, karma is not running any tests then, the test suite is simply empty.

I am looking for advice with anything that could look suspicious about this configuration, and why it cannot bundle with less than 4gb memory available.

0

There are 0 best solutions below