Can I programmatically generate a list of local pure ESM node_modules for Webpack/babel-loader?

Question

Is there an easy way to get a list of your local node_modules that are pure ESM (i.e. only support ES6 import/export syntax)?

While trying to create a CommonJS bundle for a Serverless Framework project I needed to use npm packages that had recently gone pure ESM. This meant manually listing them in my Webpack config.

The solution to using pure ESM modules (details in this blog post) that I came up with was to first tweak the babel-loader exclude definition so that any modules I know to be pure ESM (and its pure ESM dependencies) are still transpiled then also make sure that webpack-node-externals passed these transpiled versions into the bundle by defining them in the allowlist.

Now I'd like to automate getting that list of third-party pure ESM modules in my project (along with their dependencies), so I don't have to keep maintaining it by hand.

Maybe loop through everything in package-lock.json as a starting point?

Here's what that currently looks like (sample repo). pureESMModules is the list I'd like to programmatically generate:

const path = require('path');

const babelLoaderExcludeNodeModulesExcept = require('babel-loader-exclude-node-modules-except');
const nodeExternals = require('webpack-node-externals');
const slsw = require('serverless-webpack');

const { isLocal } = slsw.lib.webpack;

const pureESMModules = ['pretty-ms', 'parse-ms'];

module.exports = {
  target: 'node',
  stats: 'normal',
  entry: slsw.lib.entries,
  externals: [nodeExternals({ allowlist: pureESMModules })],
  mode: isLocal ? 'development' : 'production',
  optimization: { concatenateModules: false },
  resolve: { extensions: ['.js'] },
  module: {
    rules: [
      {
        test: /\.js$/,
        exclude: babelLoaderExcludeNodeModulesExcept(pureESMModules),
        use: {
          loader: 'babel-loader',
          options: {
            cacheDirectory: true,
          },
        },
      },
    ],
  },
  output: {
    libraryTarget: 'commonjs2',
    filename: '[name].js',
    path: path.resolve(__dirname, '.webpack'),
  },
};

I use serverless-webpack and set includeModules to true to specifically tell it to bundle modules, which then applies to the transpiled versions that babel-loader generated:

custom:
  webpack:
    includeModules: true

Have I actually over-engineered this? It does feel like something Webpack should have already been doing for me. Or are there valid reasons it (or webpack-node-externals) can't programmatically identify which pure ESM modules to handle differently?

I know that excluding most modules in babel-loader is just there for performance reasons. If I skip that exclude option entirely, it would transpile everything and just run a little slower. I still need pureESMModules for my allowlist definition though.

Edit: Turns out webpack-node-module-types does get you a list of ESM modules, so in my example above, I use this now:

const { determineModuleTypes } =
  require('webpack-node-module-types/sync');

// ...

const pureESMModules = determineModuleTypes()?.esm || [];

You might need to do something about filtering out the ESM modules that are only used by devDependencies, in case Webpack can't tree-kshake those, or you will end up with unnecessary code in your bundle.

Answer 1

I ended up using webpack-node-module-types to get a list of ESM modules, and then also matching those up against just my dependencies, so I don't needlessly bundle up devDependencies:

const path = require('path');

const babelLoaderExcludeNodeModulesExcept = require('babel-loader-exclude-node-modules-except');
const nodeExternals = require('webpack-node-externals');
const slsw = require('serverless-webpack');
const getPureESMDependencies = require('./util/pureESMDependencies.js');

const pureESMDependencies = getPureESMDependencies();

module.exports = {
  target: 'node',
  stats: 'normal',
  entry: slsw.lib.entries,
  externals: [nodeExternals({ allowlist: pureESMDependencies.map((dep) => RegExp(`^${dep}(/.*)?$`)) })],
  mode: slsw.lib.webpack.isLocal ? 'development' : 'production',
  optimization: { concatenateModules: false },
  resolve: { extensions: ['.js'] },
  module: {
    rules: [
      {
        test: /\.js$/,
        exclude: babelLoaderExcludeNodeModulesExcept(pureESMDependencies),
        use: {
          loader: 'babel-loader',
          options: {
            cacheDirectory: true,
          },
        },
      },
    ],
  },
  output: {
    libraryTarget: 'commonjs2',
    filename: '[name].js',
    path: path.resolve(__dirname, '.webpack'),
  },
};

Note how for the allowlist, I also use an array of regexps, which also catches some packages that imported stuff like formdata-polyfill/esm.min.js.

pureESMDependencies.js then looks like this:

const jsonfile = require('jsonfile');
const { determineModuleTypes } = require('webpack-node-module-types/sync');

// Recursively get all dependencies (not dev, peer) for the current project
function getDependencies(packageJsonLock, packages, knownDependencies = new Set()) {
  if (!packages?.length) return [];

  packages.forEach((packageName) => {
    const dependencies = Object.keys(packageJsonLock.packages[`node_modules/${packageName}`].dependencies || {});

    knownDependencies.add(packageName);
    if (dependencies.length) {
      dependencies.forEach((dependency) => knownDependencies.add(dependency));

      getDependencies(packageJsonLock, dependencies, knownDependencies);
    }
  });

  return [...knownDependencies];
}

// Get a list of packages that are pure ESM and in the 'dependencies' list in package.json
module.exports = function getPureESMDependencies() {
  const packageJsonLock = jsonfile.readFileSync('./package-lock.json');
  const packageJson = jsonfile.readFileSync('./package.json');

  const esmModules = determineModuleTypes()?.esm;
  if (!esmModules?.length) return [];

  const allDependencies = getDependencies(packageJsonLock, Object.keys(packageJson.dependencies));
  console.log({ allDependencies });

  return esmModules.filter((packageName) => allDependencies.includes(packageName));
};