Spawning of nested background/parallel processes fails

188 Views Asked by At

I'm trying to build a dockerized R microservice that runs on AWS ECS and involves packages {future}, {future.callr} and {furrr} (and thus implicitly {callr} and {processx})

When I do local testing with a docker container that is limited to 1 CPU and 650 MB memory (matches the stage setup of a client of mine; production has 1500 MB and 4 CPUs), everything works fine.

However, on the actual AWS instance I get the following errors:

Error 1:

Error in get_result(out, private$options) : 
  callr subprocess failed: could not start R, exited with non-zero status, has crashed or was killed

Error 2:

Error in rethrow_call(c_processx_exec, command, c(command, args), stdin,  : 
  Cannot fork when running '/usr/local/lib/R/bin/R' (system error -12, Unknown error -12) @unix/processx.c:499 (processx_exec)

I'm completely out of my depth here, would anyone be able to give me a hint to what's going on and/or how I can fix this?

It's kind of hard to provide a reprex on this issue, but here's the main gist of the general setup:

Ensuring background processes

ensure_background_processes <- function(workers_level_1, workers_level_2){
  future::plan(
    list(
      future::tweak(future.callr::callr, workers = workers_level_1),
      future::tweak(future.callr::callr, workers = workers_level_2)
    )
  )
}

Wrapper function that handles two operations to happen "concurrently" where each operation needs to be parallelized over 3 entities (so x would e.g. be of length 3)

foo <- function(x) {
  future::future({
    x %>%
      furrr::future_map(
        ~.x %>% bar()
      )
  })

  future::future({
    x %>%
      furrr::future_map(
        ~.x %>% baz()
      )
  })
}

Actual functions to be executed in each parallelization cycle

bar <- function(x) {
  # Do something useful in a while loop
}

baz <- function(x) {
  # Do something useful in a while loop
}
0

There are 0 best solutions below