How to package sqlite3 in node.js executable packages?

Question

I want to make use of a simple database in a compiled node.js app. Is this possible without installing the database separately? i.e I want the database to be included in the .exe file so that I can just copy and execute that file.

I'm using pkg to create .exe files and it works well, but when I use the sqlite3 npm module the .exe errors when I try to execute with the following warning:

pkg/prelude/bootstrap.js:1155
      throw error;
      ^

Error: Cannot find module 'C:\snapshot\sqlite\node_modules\sqlite3\lib\binding\node-v51-win32-x64\node_sqlite3.node'
1) If you want to compile the package/file into executable, please pay attention to compilation warnings and specify a literal in 'require' call. 2) If you don't want to compile the package/file into executable and want to 'require' it from filesystem (likely plugin), specify an absolute path in 'require' call using process.cwd() or process.execPath.
    at Function.Module._resolveFilename (module.js:470:15)
    at Function.Module._resolveFilename (pkg/prelude/bootstrap.js:1252:46)
    at Function.Module._load (module.js:418:25)
    at Module.require (module.js:498:17)
    at Module.require (pkg/prelude/bootstrap.js:1136:31)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (C:\snapshot\sqlite\node_modules\sqlite3\lib\sqlite3.js:4:15)
    at Module._compile (pkg/prelude/bootstrap.js:1226:22)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:488:32)

Which looks similar to a sqlite3 bug that was fixed here: https://github.com/zeit/pkg/issues/183 (since the bug was fixed I assume this is a user issue)

Looking at the error message, it looks like the ...../bide_sqlit3.node file can't be found. And looking at node_modules/ in my development env (where the module works) I can't find that file. So I assume that the file is NOT being included in the executable and that I need to do something that:

1. Makes pkg include the file in the binary
2. Alters the path to the file to be the path to the file in the binary

How do I do this with zeit/pkg? or, if this is more correct: How do I force npm to to install binaries to node_modules and then reference those binaries?

Answer 1

You should put the built node-sqlite3.node in the same directory as the binary that you built with pkg (as stated in this issue). This file can be found in your node_modules/sqlite3/lib/binding/node-vxx-xxxxx-xxx/node_sqlite3.node.

You also need to make sure that you build your project with pkg with the same node version that you built sqlite3 with.

Answer 2

Just add this to your package.json:

"pkg": {
    "assets": ["node_modules/sqlite3/lib/bindings"]
}

Answer 3

Here is my full gitlab ci solution for Linux environments:

My package.json have this section:

  "bin": "src/app.js",
  "pkg": {
    "targets": [
      "node16-linux-x64"
    ],
    "assets": [
      "node_modules/sqlite3/lib/binding/napi-v6-linux-glibc-x64/node_sqlite3.node"
    ],
    "outputPath": "./"
  }

My gitlab ci pipeline:

variables:
  GIT_DEPTH: 1
  APT_CACHE_DIR: .aptcache/
  YARN_CACHE_DIR: .yarn-cache/
  PKG_CACHE_DIR: .pkg-cache/

build-deploy:
  image: node:16.13-slim
  cache:
    key: backend
    paths:
      - node_modules
      - $APT_CACHE_DIR
      - $YARN_CACHE_DIR
      - $PKG_CACHE_DIR
  before_script:
    - rm -f /etc/apt/apt.conf.d/docker-clean
    - mkdir -pv $APT_CACHE_DIR $YARN_CACHE_DIR $PKG_CACHE_DIR && mkdir /var/cache/apt/archives && mount --bind $APT_CACHE_DIR /var/cache/apt/archives/
    - apt update -y && apt upgrade -y
    - apt install rsync openssh-client sqlite3 -y
    - node -v && yarn -v
    - yarn install --cache-folder $YARN_CACHE_DIR
  script:
    - PKG_CACHE_PATH=$PKG_CACHE_DIR node_modules/.bin/pkg package.json
    # deploy via rsync

Please note:

• you need the sqlite3 package in your build system, otherwise you will not have necessary binding under node_modules/sqlite3/lib/binding/. Here you can see I am installing sqlite3 package in my debian system.
• if you will build on alpine linux and then deploy to debian\ubuntu\gentoo release environment it will not work since alpine use musl and most other linux systems use glibc. You must build on same system that you will use in release