Workerpool says my workers are terminated when I try to execute a function

16 Views Asked by At

I'm trying to add workers to my Node.JS application so that I can run jobs after an endpoint gets hit and eventually process files.

I am trying to use the npm package workerpool but when I call my workerpool to execute a function I get the following error: Error running analysis: Error: Worker is terminated

I setup my workerpool in worker.ts

import workerpool from "workerpool"

export const setupWorkers = async () => {
  try {
    const filepath = __dirname + "/analysisWorker.ts"

    const pool = workerpool.pool(filepath, {
      minWorkers: 2,
      maxWorkers: 4,
      maxQueueSize: 20,
      workerType: "thread",
      emitStdStreams: true,
    })

    console.log(`Total workers: ${pool.stats().totalWorkers}`)
    return pool
  } catch (err) {
    console.error("Error when setting up workers")
  }
}

the filepath points to my analysisWorker.ts file which only has this in it:

import workerpool from "workerpool"

function runAnalysis() {
  console.log("job started")
  return 1000
}

// NOTE: I have tried removing this aswel
workerpool.worker({
  runAnalysis: runAnalysis,
})

I then setup my workers in my app.ts file like so:

...
app.use(cookieParser("test_123"))
app.use(express.json())
app.use(cors())

const startApp = async () => {
  // Setup workers
  const workerpool = await setupWorkers()

  // Attach workerpool to app
  app.set("workerpool", workerpool)

  // Connect to database
  await connect()

  // Setup routes
  routes(app)

  // Error handling
  app.use(errorHandler)

  app.listen(PORT, async () => {
    console.log(
      `Server started on port ${PORT} in ${process.env.NODE_ENV} mode`
    )
  })
}

startApp()

When I start the app I get the expected console log of "Total workers: 2" in my terminal. Note I am also using app.set to set the workerpool so I can use it in the rest of my project.

Finally, in my controller file productController.ts, I have a product class with the following function which is ran when I hit localhost:3000/products with a POST.

  public async create(req: Request, res: Response, next: NextFunction) {
    const product = await productService.addProduct(req.body.name)
    const workerpool = req.app.get("workerpool")
    workerpool
      .exec("runAnalysis")
      .then((result: number) => {
        console.log("result: ", result)
      })
      .catch((err: Error) => {
        console.error("Error running analysis: ", err)
      })
    console.log(workerpool.stats().totalWorkers)

    res.send({ status: "OK", data: product, count: 1 })
  }

Notice the catch block is what is producing the error above, however the console.log directly after it prints 2 to the terminal, which means I still have active workers.

Can anyone see what I am doing wrong, I have tried looking at the docs and can't figure out why my workers are terminated but still active.

0

There are 0 best solutions below