I created an R environment on Conda 23.1 and below is my sessionInfo:
R version 4.2.2 (2022-10-31)
Platform: x86_64-conda-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux Server 7.9 (Maipo)
Matrix products: default
BLAS/LAPACK: /data/conda/envs/$USER/r_mamba_dev/lib/libopenblasp-r0.3.21.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] abfunctions_1.0 arrow_9.0.0 sparklyr_1.8.2 glue_1.6.2
[5] dbplyr_2.3.3 lubridate_1.9.2 forcats_1.0.0 stringr_1.5.0
[9] dplyr_1.1.2 purrr_1.0.1 readr_2.1.4 tidyr_1.3.0
[13] tibble_3.2.0 ggplot2_3.4.2 tidyverse_2.0.0 repr_1.1.4
loaded via a namespace (and not attached):
[1] pbdZMQ_0.3-8 tidyselect_1.2.0 colorspace_2.0-3 vctrs_0.6.0
[5] generics_0.1.3 htmltools_0.5.3 yaml_2.3.6 base64enc_0.1-3
[9] utf8_1.2.2 rlang_1.1.0 pillar_1.9.0 withr_2.5.0
[13] DBI_1.1.3 bit64_4.0.5 uuid_1.1-0 lifecycle_1.0.3
[17] munsell_0.5.0 gtable_0.3.1 evaluate_0.18 tzdb_0.3.0
[21] fastmap_1.1.0 parallel_4.2.2 fansi_1.0.3 IRdisplay_1.1
[25] openssl_2.0.6 scales_1.2.1 IRkernel_1.3.1 jsonlite_1.8.4
[29] config_0.3.1 bit_4.0.5 askpass_1.1 hms_1.1.2
[33] digest_0.6.30 stringi_1.7.8 grid_4.2.2 cli_3.6.1
[37] tools_4.2.2 magrittr_2.0.3 crayon_1.5.2 pkgconfig_2.0.3
[41] ellipsis_0.3.2 timechange_0.2.0 assertthat_0.2.1 httr_1.4.4
[45] rstudioapi_0.14 R6_2.5.1 compiler_4.2.2
As an example problem here is a simple query. I'm querying a hadoop table called events through spark with the following code:
tokens<- events %>%
compute("tokens")
I would expect this to create a table in spark called "tokens". However when I run it i get the following error:
Error in `db_save_query.DBIConnection()`:
! Can't save query to "tokens".
Caused by error:
! org.apache.spark.sql.catalyst.parser.ParseException:
no viable alternative at input 'CREATE OR REPLACE TEMPORARY VIEW \n"tokens"'(line 2, pos 0)
== SQL ==
CREATE OR REPLACE TEMPORARY VIEW
"tokens" AS SELECT *
^^^
FROM `db`.`events `
As an added note this error only seems to occur when I use the compute or pivot_wider functions. If I were to run a simple example like this i would not get the error:
tokens<- events
I have tried updating sparklyr, tidyverse, dplyr, dbplyr, and no updates have resolved it. I've also tried creating entirely new code blocks thinking I put the '\n' in there somehow but that didn't fix it. I also tried putting the entire code on one line and that didn't fix it either.
For some reason it appears as though a new line is being entered and that's causing an issue in the SQL that's being translated. I'm not sure how to solve that problem so I can run the R script as normal.