Sourcemap transformation issue with Browserify > Exorcist > Sorcery and a local NPM Dependency

Question

I'm developing a Chrome extension (typescript, browser target) with a dependency to a local npm module (node target).

The problem: my build pipeline is broken, most likely the sourcemap transformations (webpack > browserify > exorcist > sorcery) through various dependencies (my app > core library > node depencendies).

Project Setup & Build Pipeline

Multi module project setup

• chrome extension "quickbrain.chrome"
  • this is what I want to build
  • browser target
• core library "quickbrain.core"
  • used by "quickbrain.chrome" and others
  • node target / dependencies to nodejs modules (e.g. https://github.com/NaturalNode/natural)
• some other apps that use the "quickbrain.core" module (node express)

Builld pipeline for 'quickbrain.chrome'

• Typescript (Input)

• Webpack

• Browserify, https://github.com/browserify/browserify

  (make quickbrain.core (node) run within quickbrain.chrome (browser))

• Exorcist, https://github.com/thlorenz/exorcist

  (externalize sourcemaps, since Chrome doesn't understand inline souremaps for extensions, but browserify only generates inline)

• Sorcery, https://github.com/Rich-Harris/sorcery (resolve chain of sourcemap uris through this horrible project setup)

• JS + Sourcemaps (Output)

Error Message

When building I get the following artifacts and an error at the Sorcery stage.

• Webpack -> contentscript.js, contentscript.js.map
• Browserify -> Exorcist -> contentscript.exorcist.js, contentscript.exorcist.js.map
• Sorcery -> contentscript.final.js, concentscript.final.js.map

(node:30724) UnhandledPromiseRejectionWarning: Error: ENOENT: no such file or directory,
open 'C:\QB\com.quickbrain\chrome\dist.development\core\node_modules\underscore\underscore-node-f.cjs.map'

Underscore / underscore-node is a dependency from a node module 'natural', used within 'quikbrain.core' and I didn't have that problem in the past.

Hints

1. I can see that the root path in the error message is completely wrong:

   • Error Message: com.quickbrain\____chrome\dist.development____\core\node_modules\underscore\underscore-node-f.cjs.map (note the wrong absolute path highlighted by ____)
   • Actual File: com.quickbrain\core\node_modules\underscore\underscore-node-f.cjs.map

2. When looking into the sourcemap itself I can see the same relative path for other sourcemaps as well, e.g. webpack:///../core/node_modules/axios/index.js vs webpack:///../core/node_modules/underscore/underscore-node-f.cjs, but only the latter causes the error message in sorcery with the wrong absolute path.

3. Builds in the past didn't include the underscore-node sourcemap. There is already a difference after running webpack, but I can't figure out why it is now included.

4. Please note that quickbrain.chrome installs quickbrain.core as a local dependency "quickbrain.core": "file:../core"

Reference Files

> Browserified parts of contentscript.js and contentscript.js.map

const _ = __webpack_require__(/*! underscore */ "../core/node_modules/underscore/underscore-node.cjs")

    "webpack:///../core/node_modules/underscore/underscore-node-f.cjs","webpack:///../core/node_modules/underscore/underscore-node.cjs",

> quickbrain.chrome > package.json (only relevant parts)

{
  "name": "quickbrain.chrome",
  "version": "0.3.0",
  "description": "",
  "main": "index.js",
  "watch": {
    "webpack-dev": {
      "patterns": [
        "src"
      ],
      "extensions": "ts,html"
    }
  },
  "scripts": {
    "webpack-dev": "webpack --env NODE_ENV=development --config webpack.config.js && npm run browserify-contentscript-dev && npx node sorcery_finalize_sourcemaps.js",
    "browserify-contentscript-dev": "browserify dist.development/contentscript.js --debug --ignore-missing | exorcist --error-on-missing --root ../ dist.development/contentscript.exorcist.js.map >  dist.development/contentscript.exorcist.js",
  },
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "@types/chrome": "0.0.127",
    "@types/node": "^14.14.28",
    "awesome-typescript-loader": "^5.2.1",
    "browserify": "^17.0.0",
    "clean-webpack-plugin": "^3.0.0",
    "convert-source-map": "^1.7.0",
    "copy-webpack-plugin": "^7.0.0",
    "exorcist": "^1.0.1",
    "node-sass": "^5.0.0",
    "npm": "^6.14.10",
    "npm-run-all": "^4.1.5",
    "npm-watch": "^0.7.0",
    "sass-loader": "^10.1.0",
    "sorcery": "^0.10.0",
    "sourceify": "^1.0.0",
    "ts-node": "^9.1.1",
    "typescript": "^4.1.3",
    "webpack": "^5.10.3",
    "webpack-cli": "^4.2.0"
  },
  "dependencies": {
    "@types/ejs": "^3.0.5",
    "@types/jquery": "^3.5.5",
    "axios": "^0.21.0",
    "quickbrain.core": "file:../core",
    "ejs": "^3.1.5",
    "html-loader": "^1.3.2",
    "jquery": "^3.5.1",
    "nodejs-base64-converter": "^1.0.5"
  }
}

> quickbrain.chrome > tsconfig.json

{
  "compileOnSave": false,
  "compilerOptions": {
    "target": "es5",
    "module": "commonjs",
    "sourceMap": true,
    "inlineSourceMap": false,
    "inlineSources": false,
    "types": ["@types/chrome", "node", "jquery"],
    "typeRoots": ["node_modules/@types"],
    "moduleResolution": "node",
    "esModuleInterop": true,
    "lib": ["es2017", "dom"]
  },
  "include": ["src/background", "src/contentscript-chrome"]
}

> quickbrain.chrome > sorcery_finalize_sourcemaps.js (only relevant parts)

var sorcery = require("sorcery");
let i = 0;

function doMagic(src, dst) {

  console.log(`loading ` + src);
  sorcery.load(src).then(function(chain) {
    // generate a flattened sourcemap
    var map = chain.apply(); // { version: 3, file: 'code.min.js', ... }
    // get a JSON representation of the sourcemap
    map.toString(); // '{"version":3,"file":"code.min.js",...}'
    // get a data URI representation
    map.toUrl(); // 'data:application/json;charset=utf-8;base64,eyJ2ZXJ...'
    // write to a new file, but append the flattened sourcemap as a data URI
    chain.write(dst, { inline: true });
  });
}

try {
  doMagic(
    "dist.development/contentscript.exorcist.js",
    "dist.development/contentscript.final.js"
  );
  console.log("\n\n_,.~*`( sorcery done )`*~.,_\n");
} catch (e) {
  console.error("Computer says no: " + e + " :(");
}

(When I just use npx sorcery --input dist.development/contentscript.exorcist.js --output dist.development/contentscript.sorcery.js I get the same error message)

> quickbrain.core > package.json

{
  "name": "quickbrain.core",
  "version": "0.2.0",
  "description": "",
  "main": "index.js",
  "types": "index.d.ts",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "build": "npx tsc"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@types/axios": "^0.14.0",
    "@types/mongoose": "^5.10.3",
    "@types/natural": "^2.1.1",
    "@types/wink-tokenizer": "^4.0.2",
    "atlassian-jwt": "^1.0.3",
    "axios": "^0.21.1",
    "compute-cosine-similarity": "^1.0.0",
    "express": "^4.17.1",
    "fuzzball": "^1.3.1",
    "jwt-decode": "^3.1.2",
    "moment": "^2.29.1",
    "mongoose": "^5.11.17",
    "natural": "^4.0.3",
    "stopwords-iso": "^1.1.0",
    "string-strip-html": "^8.2.2",
    "ts-lib": "0.0.5",
    "tslib": "^2.1.0",
    "wink-tokenizer": "^5.2.3"
  },
  "devDependencies": {
    "mongodb-memory-server": "^6.9.6",
    "typescript": "^4.2.3"
  }
}

> quickbrain.core > tsconfig.json

{
  "compilerOptions": {
    "target": "es5",                          /* Specify ECMAScript target version: 'ES3' (default), 'ES5', 'ES2015', 'ES2016', 'ES2017', 'ES2018', 'ES2019', 'ES2020', or 'ESNEXT'. */
    "module": "commonjs",                     /* Specify module code generation: 'none', 'commonjs', 'amd', 'system', 'umd', 'es2015', 'es2020', or 'ESNext'. */
    "declaration": true,                   /* Generates corresponding '.d.ts' file. */
    "outDir": "./dist",                        /* Redirect output structure to the directory. */
    "strict": true,                           /* Enable all strict type-checking options. */
    "esModuleInterop": true,                  /* Enables emit interoperability between CommonJS and ES Modules via creation of namespace objects for all imports. Implies 'allowSyntheticDefaultImports'. */
    "skipLibCheck": true,                     /* Skip type checking of declaration files. */
    "forceConsistentCasingInFileNames": true  /* Disallow inconsistently-cased references to the same file. */
  }
}

Answer 1

Edit: Unfortunately not solved. The pipeline works, but I am missing most of the sourcemaps

I was able to fix this using https://stackoverflow.com/a/55356078/5244937 (and https://github.com/browserify/browserify/issues/772) about Browserify and modules with precompiled sourcemaps (like underscore that is causing the trouble).

According to the post, Browserify will ignore 3rd party sourcemaps when tsconfig.json is set to inline-sourcemaps.

This means I have changed the tsconfig.json of quickbrain.chrome to

    "sourceMap": false,
    "inlineSourceMap": true,
    "inlineSources": true,

.. and Browserify (2st step of the pipeline) drops the sourcemaps of natural, underscore, etc avoiding the problem at the end of the pipeline altogether.

---

Fun fact: "inlineSourceMap": true is not even used by my pipeline, since I compile using webpack with external sourcemaps. The only effect is influencing Browserfify to drop other sourcemaps.