How to Convert Map of Struct type to Json in Spark2

Question

I have a map field in dataset with below schema

     |-- party: map (nullable = true)
     |    |-- key: string
     |    |-- value: struct (valueContainsNull = true)
     |    |    |-- partyName: string (nullable = true)
     |    |    |-- cdrId: string (nullable = true)
     |    |    |-- legalEntityId: string (nullable = true)
     |    |    |-- customPartyId: string (nullable = true)
     |    |    |-- partyIdScheme: string (nullable = true)
     |    |    |-- customPartyIdScheme: string (nullable = true)
     |    |    |-- bdrId: string (nullable = true)

Need to convert it to JSON type. Please suggest how to do it. Thanks in advance

Answer 1

Spark provides to_json function available for DataFrame operations:

import org.apache.spark.sql.functions._
import spark.implicits._
val df =
  List(
    ("key1", "party01", "cdrId01"),
    ("key2", "party02", "cdrId02"),
  )
    .toDF("key", "partyName", "cdrId")
    .select(struct($"key", struct($"partyName", $"cdrId")).as("col1"))
    .agg(map_from_entries(collect_set($"col1")).as("map_col"))
    .select($"map_col", to_json($"map_col").as("json_col"))