I am using quanteda to build document feature matrices from different data sources. While building dfm with parliamentary speech data and Facebook data takes just a few minutes, it takes more than 7 hours to compile a dfm based on a Twitter dataset. The three datasets are approximately equal in size (60mb).
R is updated (R version 3.5.3), RStudio is updated (Version 1.3.923) and quanteda is updated (Version 2.0.1) and I am using a MacBook Pro 2018 (OS X version 10.14.5).
Running the exact same code on another machine with an older version of quanteda (version 1.5.2) takes just a few minutes instead of several hours.
Unfortunately, I cannot provide a reproducible example since the data cannot be shared.
Do you have any ideas what the problem might be and how I can circumvent it?
Here are the sessionInfo() and code plus output from the problematic machine that needs more than 7 hours for creating the dfm:
> sessionInfo()
R version 3.5.3 (2019-03-11)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Mojave 10.14.5
Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.5/Resources/lib/libRlapack.dylib
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] quanteda_2.0.1 forcats_0.5.0 stringr_1.4.0 dplyr_0.8.5 purrr_0.3.3 readr_1.3.1 tidyr_1.0.2
[8] tibble_3.0.0 ggplot2_3.3.0 tidyverse_1.3.0
loaded via a namespace (and not attached):
[1] tinytex_0.20 tidyselect_1.0.0 xfun_0.12 haven_2.2.0 lattice_0.20-40 colorspace_1.4-1
[7] vctrs_0.2.4 generics_0.0.2 yaml_2.2.1 rlang_0.4.5 pillar_1.4.3 glue_1.3.2
[13] withr_2.1.2 DBI_1.1.0 dbplyr_1.4.2 modelr_0.1.6 readxl_1.3.1 lifecycle_0.2.0
[19] munsell_0.5.0 gtable_0.3.0 cellranger_1.1.0 rvest_0.3.5 fansi_0.4.1 broom_0.5.5
[25] Rcpp_1.0.4 scales_1.1.0 backports_1.1.5 RcppParallel_5.0.0 jsonlite_1.6.1 fs_1.3.2
[31] fastmatch_1.1-0 stopwords_1.0 hms_0.5.3 stringi_1.4.6 grid_3.5.3 cli_2.0.2
[37] tools_3.5.3 magrittr_1.5 crayon_1.3.4 pkgconfig_2.0.3 ellipsis_0.3.0 Matrix_1.2-18
[43] data.table_1.12.8 xml2_1.3.0 reprex_0.3.0 lubridate_1.7.4 assertthat_0.2.1 httr_1.4.1
[49] rstudioapi_0.11 R6_2.4.1 nlme_3.1-145 compiler_3.5.3
> dtmTW <- dfm(corpTW, groups = "user.id",
+ remove = stopwords("de"),
+ tolower = TRUE,
+ remove_punct = TRUE,
+ remove_numbers = TRUE,
+ remove_twitter = TRUE,
+ remove_url = TRUE,
+ dictionary = myDict,
+ verbose = TRUE)
Creating a dfm from a corpus input...
...lowercasing
...found 886,166 documents, 543,035 features
...grouping texts
...applying a dictionary consisting of 1 key
...removed 0 features
...complete, elapsed time: 25338 seconds.
Finished constructing a 408 x 1 sparse dfm.
Warning message:
'remove_twitter' is deprecated; for FALSE, use 'what = "word"' instead.
Here are the sessionInfo() and code plus output from the machine that creates the same dfm in less than a minute:
R version 3.6.1 (2019-07-05)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS High Sierra 10.13.6
Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib
Random number generation:
RNG: Mersenne-Twister
Normal: Inversion
Sample: Rounding
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] quanteda_1.5.2 forcats_0.4.0 stringr_1.4.0 dplyr_0.8.4 purrr_0.3.3
[6] readr_1.3.1 tidyr_1.0.0 tibble_2.1.3 ggplot2_3.2.1 tidyverse_1.3.0
> dtmTW <- dfm(corpTW, groups = "user.id",
+ remove = stopwords("de"),
+ tolower = TRUE,
+ remove_punct = TRUE,
+ remove_numbers = TRUE,
+ remove_twitter = TRUE,
+ remove_url = TRUE,
+ dictionary = myDict,
+ verbose = TRUE)
Creating a dfm from a corpus input...
... lowercasing
... found 886,166 documents, 471,981 features
... grouping texts
... applying a dictionary consisting of 1 key
... removed 0 features
... created a 408 x 1 sparse dfm
... complete.
Elapsed time: 108 seconds.