Issue with library(RcppParallel) with {renv} in R

Question

Problem: In R, I can run library(RcppParallel) in the usual way without issues but in an RStudio Project that uses {renv} for local dependency management, after running renv::install("RcppParallel"), library(RcppParallel) fails with an error. I am using an M2 MacBook.

Details:

In R, I can install & load {RcppParallel} in the usual way without issues:

install.packages("RcppParallel")
library(RcppParallel)

In an RStudio Project using {renv}, the package installs:

project <- file.path(tempdir(), "project")
dir.create(project)
setwd(project)
renv::init(settings = list(use.cache = FALSE), restart = FALSE)
writeLines("library(RcppParallel)", con = "dependencies.R")
renv::install("RcppParallel", type = "binary", prompt = FALSE)

which seems to work:

# Downloading packages -------------------------------------------------------
- Downloading RcppParallel from CRAN ...        OK [file is up to date]
Successfully downloaded 1 package in 0.64 seconds.

The following package(s) will be installed:
- RcppParallel [5.1.7]
These packages will be installed into "/private/var/folders/nl/ygb3g6tj24z06jddbqqhj6hw0000gn/T/RtmpEO6paC/project/renv/library/R-4.3/aarch64-apple-darwin20".

# Installing packages --------------------------------------------------------
- Installing RcppParallel ...                   OK [installed binary in 0.44s]
Successfully installed 1 package in 0.45 seconds.

But running library(RcppParallel) fails:

source("dependencies.R")

Error: package or namespace load failed for ‘RcppParallel’:
 .onLoad failed in loadNamespace() for 'RcppParallel', details:
  call: dyn.load(file, DLLpath = DLLpath, ...)
  error: unable to load shared object '/private/var/folders/nl/ygb3g6tj24z06jddbqqhj6hw0000gn/T/RtmpEO6paC/project/renv/library/R-4.3/aarch64-apple-darwin20/RcppParallel/libs/RcppParallel.so':
  dlopen(/private/var/folders/nl/ygb3g6tj24z06jddbqqhj6hw0000gn/T/RtmpEO6paC/project/renv/library/R-4.3/aarch64-apple-darwin20/RcppParallel/libs/RcppParallel.so, 0x0006): symbol not found in flat namespace '__ZN3tbb10interface78internal15task_arena_base24internal_max_concurrencyEPKNS0_10task_arenaE'

I thought installing {RcppParallel} from source might be a solution, but I can't get this to work, even outside of {renv}:

setwd(tempdir())
install.packages("RcppParallel", type = "source")

trying URL 'https://cran.rstudio.com/src/contrib/RcppParallel_5.1.7.tar.gz'
Content type 'application/x-gzip' length 1629119 bytes (1.6 MB)
==================================================
downloaded 1.6 MB

* installing *source* package ‘RcppParallel’ ...
** package ‘RcppParallel’ successfully unpacked and MD5 sums checked
** using staged installation
** preparing to configure package 'RcppParallel' ...
*** configured file: 'R/tbb-autodetected.R.in' => 'R/tbb-autodetected.R'
*** configured file: 'src/Makevars.in' => 'src/Makevars'
*** configured file: 'src/install.libs.R.in' => 'src/install.libs.R'
** finished configure for package 'RcppParallel'
** libs
using C++ compiler: ‘g++ (GCC) 12.2.0’
using SDK: ‘MacOSX14.0.sdk’
(tbb) Building TBB using bundled sources ...
OS: macos
arch=arm64
compiler=clang
runtime=cc15.0.0_os13.6.1
tbb_build_prefix=macos_arm64_clang_cc15.0.0_os13.6.1
work_dir=/private/var/folders/nl/ygb3g6tj24z06jddbqqhj6hw0000gn/T/RtmpJxF6Aa/R.INSTALL286e431dd45c/RcppParallel/src/build/macos_arm64_clang_cc15.0.0_os13.6.1_release
Created ../build/lib_release directory
make -C "../build/lib_release"  -r -f ../../build/Makefile.tbb cfg=release
../../build/Makefile.tbb:28: CONFIG: cfg=release arch=arm64 compiler=clang target=macos runtime=cc15.0.0_os13.6.1
/usr/local/gfortran/bin/g++ -fopenmp  -I../inst/include -I/usr/local/gfortran/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -o concurrent_hash_map.o -c -MMD -O2 -DUSE_PTHREAD  -stdlib=libc++ -arch arm64 -mmacosx-version-min=10.11  -fPIC  -D__TBB_BUILD=1 -Wall  -falign-functions=64 -Wall -g -O2 -DTBB_NO_LEGACY=1   -I../../src -I../../src/rml/include -I../../include ../../src/tbb/concurrent_hash_map.cpp
In file included from ../../include/tbb/tbb_stddef.h:91,
                 from ../../include/tbb/concurrent_hash_map.h:20,
                 from ../../src/tbb/concurrent_hash_map.cpp:17:
../../include/tbb/tbb_config.h:36:14: fatal error: cstddef: No such file or directory
   36 |     #include <cstddef>
      |              ^~~~~~~~~
compilation terminated.
make[2]: *** [concurrent_hash_map.o] Error 1
make[1]: *** [tbb_release] Error 2
/usr/local/gfortran/bin/g++ -fopenmp -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include  -I/usr/local/gfortran/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include   -std=gnu++11 -DRCPP_PARALLEL_USE_TBB=1 -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -fPIC  -g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe -c init.cpp -o init.o
/usr/local/gfortran/bin/g++ -fopenmp -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include  -I/usr/local/gfortran/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include   -std=gnu++11 -DRCPP_PARALLEL_USE_TBB=1 -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -fPIC  -g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe -c options.cpp -o options.o
In file included from ../inst/include/tthread/tinythread.h:80,
                 from ../inst/include/RcppParallel/TinyThread.h:9,
                 from ../inst/include/RcppParallel.h:6,
                 from options.cpp:2:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/pthread.h:337:49: warning: this use of "defined" may not be portable [-Wexpansion-to-defined]
  337 | #if !_PTHREAD_SWIFT_IMPORTER_NULLABILITY_COMPAT()
      |                                                 ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/pthread.h:337:49: warning: this use of "defined" may not be portable [-Wexpansion-to-defined]
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/pthread.h:563:49: warning: this use of "defined" may not be portable [-Wexpansion-to-defined]
  563 | #if !_PTHREAD_SWIFT_IMPORTER_NULLABILITY_COMPAT()
      |                                                 ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/pthread.h:563:49: warning: this use of "defined" may not be portable [-Wexpansion-to-defined]
/usr/local/gfortran/bin/g++ -fopenmp -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/gfortran/lib -Wl,-rpath,/usr/local/gfortran/lib -o RcppParallel.so init.o options.o -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
-macosx_version_min has been renamed to -macos_version_min
ld: warning: -multiply_defined is obsolete
ld: warning: -single_module is obsolete
ld: warning: duplicate -rpath '/usr/local/gfortran/lib' ignored
ld: warning: ignoring duplicate libraries: '-lgcc'
0  0x102cf3648  __assert_rtn + 72
1  0x102c27fac  ld::AtomPlacement::findAtom(unsigned char, unsigned long long, ld::AtomPlacement::AtomLoc const*&, long long&) const + 1204
2  0x102c3d924  ld::InputFiles::SliceParser::parseObjectFile(mach_o::Header const*) const + 15164
3  0x102c4ae30  ld::InputFiles::parseAllFiles(void (ld::AtomFile const*) block_pointer)::$_7::operator()(unsigned long, ld::FileInfo const&) const + 420
4  0x1964cc440  _dispatch_client_callout2 + 20
5  0x1964e1544  _dispatch_apply_invoke_and_wait + 224
6  0x1964e084c  _dispatch_apply_with_attr_f + 1180
7  0x1964e0a38  dispatch_apply + 96
8  0x102cc53b8  ld::AtomFileConsolidator::parseFiles(bool) + 292
9  0x102c66170  main + 9048
ld: Assertion failed: (resultIndex < sectData.atoms.size()), function findAtom, file Relocations.cpp, line 1336.
collect2: error: ld returned 1 exit status
make: *** [RcppParallel.so] Error 1
ERROR: compilation failed for package ‘RcppParallel’
* removing ‘/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppParallel’
* restoring previous ‘/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppParallel’

If it helps, the contents of /Users/{username}/.R/Makevars is:

F77 = /usr/local/bin/gfortran
FC = /usr/local/bin/gfortran
FLIBS = -L/usr/local/bin/gfortran

LOC = /usr/local/gfortran
CC=$(LOC)/bin/gcc -fopenmp
CXX=$(LOC)/bin/g++ -fopenmp
CXX11 = $(LOC)/bin/g++ -fopenmp # for fst package

CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe
LDFLAGS=-L$(LOC)/lib -Wl,-rpath,$(LOC)/lib
CPPFLAGS=-I$(LOC)/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include

And here is my sessionInfo():

> sessionInfo()
R version 4.3.1 (2023-06-16)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Ventura 13.6.1

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib 
LAPACK: /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.11.0

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

time zone: Europe/Zurich
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_4.3.1 tools_4.3.1    renv_1.0.3 

Any suggestions? Thanks!

Answer 1

Fighting a similar issue. Best thread I've found on the topic is at: https://community.rstudio.com/t/cant-install-packages-with-renv/96696/10

I was able to get my renv::install() - renv::snapshot() to behave correctly after switching to a computer with full-admin. Pulling the code and running renv::restore() on my work computer still giving me fits.

Answer 2

This error:

../../include/tbb/tbb_config.h:36:14: fatal error: cstddef: No such file or directory
   36 |     #include <cstddef>
      |              ^~~~~~~~~

indicates that the compiler wasn't able to find one of the standard C++ library headers. Given that you're using gcc and g++ rather than clang on macOS, I think you need to add extra include paths so that C++ library headers can be found. On my macOS machine, I see this here:

/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1

So you might need to add that somewhere -- perhaps CXXFLAGS, or similar. You may want to use -isystem when including this, since it's a system header. It might also suffice to just set -isysroot to the root of your compiler tools directory. See:

https://gcc.gnu.org/onlinedocs/gcc/Directory-Options.html https://gcc.gnu.org/onlinedocs/cpp/Invocation.html

for more details.

Answer 3

In the end, I solved the {RcppParallel} installation from source by modifying the Makevars file to use Apple's clang rather than gcc & g++. My complete Makevars file is as follows:

#### Set Fortan compilers
# Define compilers for Fortan 77 & Fortran (gfortran)
F77=/usr/local/bin/gfortran
FC=/usr/local/bin/gfortran
# Define Fortran library paths (-L)
FLIBS=-L/usr/local/bin/gfortran

#### Set C/C++ compilers (clang/clang++)
# Set compilers
CC=/usr/bin/clang
CXX=/usr/bin/clang++
# Set flags to enable OpenMP
CPPFLAGS += -Xclang -fopenmp
LDFLAGS += -lomp

The relevant revisions are in the C/C++ section. With {RcppParallel} inside an RStudio Project using {renv}, installation from binary succeeds (apparently) but library(RcppParallel) still fails:

project <- file.path(tempdir(), "project")
dir.create(project)
setwd(project)
renv::init(settings = list(use.cache = FALSE), restart = FALSE)
writeLines("library(RcppParallel)", con = "dependencies.R")
renv::install("RcppParallel", type = "binary", prompt = FALSE)
source("dependencies.R")

with

Error: package or namespace load failed for ‘RcppParallel’:
  .onLoad failed in loadNamespace() for 'RcppParallel', details:
  call: dyn.load(file, DLLpath = DLLpath, ...)
error: unable to load shared object '/private/var/folders/nl/ygb3g6tj24z06jddbqqhj6hw0000gn/T/RtmpEzGAXd/project/renv/library/R-4.3/aarch64-apple-darwin20/RcppParallel/libs/RcppParallel.so':
  dlopen(/private/var/folders/nl/ygb3g6tj24z06jddbqqhj6hw0000gn/T/RtmpEzGAXd/project/renv/library/R-4.3/aarch64-apple-darwin20/RcppParallel/libs/RcppParallel.so, 0x0006): symbol not found in flat namespace '__ZN3tbb10interface78internal15task_arena_base24internal_max_concurrencyEPKNS0_10task_arenaE'

But, after restarting R (this step is essential), repeating the same code but installing from source works:

project <- file.path(tempdir(), "project")
dir.create(project)
setwd(project)
renv::init(settings = list(use.cache = FALSE), restart = FALSE)
writeLines("library(RcppParallel)", con = "dependencies.R")
renv::install("RcppParallel", type = "source", prompt = FALSE)
source("dependencies.R")

And seems to enable {RcppParallel} to be used within a project using {renv}.