Exploding a StructType column in PySpark

Question

Is there a way to explode a Struct column in a Spark DataFrame like you would explode an Array column? Meaning to take each element of the Struct (a key-value pair) value and create a separate row for each.

I tried flattening it and then stacking it up in a for loop but it is quite computationally demanding.

Update: a more specific context for this question

I have a dataframe with two columns, surveyId and questions. SurveyId is simply a unique identifier of a survey blueprint and questions is a StructType column containing the questions involved in that particular survey, along with subfields containing information about those questions. example of dataframe

The main problem is that each question’s value has a different structure/schema. If I use “.*” on the “questions” column I will get a dataframe where the question code is the column name and the values is the struct with question information. However what I want to achieve in the end is a dataframe where these questions are rows not columns, more specifically: one column with the question code and a different column for each subfield in the values. Hope that makes sense.

Answer 1

StructType assumes that you know the schema of it, and to my knowledge there's no way to generically get all attributes. In pyspark you can read the schema of a struct (fields) and cross join your dataframe with the list of fields.

Alternatively, you can convert the struct into a map and then just explode it - in this question there some thoughts on how to convert struct to map.