I am trying to write 80GB csv file onto the MonetDB using MonetDBLite using the following code.
mtcars[1:12,] %>% write_csv("mtcars1_12.csv")
mtcars[13:32,] %>% write_csv("mtcars13_32.csv")
setwd("path") # please use your folder path
dbdir <- "path" # provide path where you want to create the MonetDB
con <- dbConnect(MonetDBLite::MonetDBLite(), dbdir)
f <- function(x, pos) dbWriteTable(conn=con, name="mtcars1", value= x, header = T, append = T)
read_csv_chunked("mtcars1_12.csv", DataFrameCallback$new(f), chunk_size = 10)
read_csv_chunked("mtcars13_32.csv", DataFrameCallback$new(f), chunk_size = 10)
Now, this mtcars data works perfectly fine, but my actual data with exactly the same code throws the following error.
Evaluation error: Failed to insert data: MALException:embedded:Could not convert column 2 var_abc to type 5.
Unfortunately, I can't reproduce the above error using mtcars as I don't know what's causing it!
What is "convert column to type 5" issue and how can I fix this?