Please can anyone help on this?
The same code runs fine in my Google Colab but throws an error in Databricks.
It seems like the vector assembler is not accepting the data type 'double' in most columns. However, when I check the datatypes in the dataframe prior to applying the pipeline, it does not have any 'double'. Does that mean that the datatype changed when applying StringIndexer or OHE as these are the only steps in the pipeline prior to the VectorAssembler?