I'm trying to build a dockerized R microservice that runs on AWS ECS and involves packages {future}, {future.callr} and {furrr} (and thus implicitly {callr} and {processx})
When I do local testing with a docker container that is limited to 1 CPU and 650 MB memory (matches the stage setup of a client of mine; production has 1500 MB and 4 CPUs), everything works fine.
However, on the actual AWS instance I get the following errors:
Error 1:
Error in get_result(out, private$options) :
callr subprocess failed: could not start R, exited with non-zero status, has crashed or was killed
Error 2:
Error in rethrow_call(c_processx_exec, command, c(command, args), stdin, :
Cannot fork when running '/usr/local/lib/R/bin/R' (system error -12, Unknown error -12) @unix/processx.c:499 (processx_exec)
I'm completely out of my depth here, would anyone be able to give me a hint to what's going on and/or how I can fix this?
It's kind of hard to provide a reprex on this issue, but here's the main gist of the general setup:
Ensuring background processes
ensure_background_processes <- function(workers_level_1, workers_level_2){
future::plan(
list(
future::tweak(future.callr::callr, workers = workers_level_1),
future::tweak(future.callr::callr, workers = workers_level_2)
)
)
}
Wrapper function that handles two operations to happen "concurrently" where each operation needs to be parallelized over 3 entities (so x
would e.g. be of length 3)
foo <- function(x) {
future::future({
x %>%
furrr::future_map(
~.x %>% bar()
)
})
future::future({
x %>%
furrr::future_map(
~.x %>% baz()
)
})
}
Actual functions to be executed in each parallelization cycle
bar <- function(x) {
# Do something useful in a while loop
}
baz <- function(x) {
# Do something useful in a while loop
}