to_json in pyspark is exluding null values, but I need the null values as blank

Question

I am converting a struct column in dataframe to json column using to_json in pyspark, but null values in few of the struct fields are ignored in json, I dont want the null values to be ignored.

Answer 1

When reading the official documentation for to_json, it says :

options dict, optional
options to control converting. accepts the same options as the JSON datasource. See Data Source Option for the version you use. Additionally the function supports the pretty option which enables pretty JSON generation.

Following the link, there is an option ignoreNullFields that you can try setting to False:

pyspark.sql.functions.to_json(col, options={"ignoreNullFields":False})

Test is on your side because you did not provide any Minimal, Reproducible Example.