When I scan the data from S3 using a Glue crawler I get this schema:
{id: integer, value: String}
This is because Spark writes data back in String type and not varchar type. Although there is a VarcharType in spark. (See: VarcharType — PySpark master documentation)
The problem is I need the data in Redshift as a Spectrum query in VarcharType only as my BI tool is not capable of reading string type data.
If I use Create External Table I get all null values in the table. Only If I change it to an internal table the issue solves. But, I really need it as a Spectrum query. Any suggestions to convert the string type to varchartype?