I'new for GCP Tools and Scala. I have this issue. I create the job to upload json file to GCS bucket via DataflowRunner. But, when I check the data inside file. it shows like this. ���.��� ( Actually it should show Thai Language). But, when i dev in local and run with command DirectRunner. It show data correctly. I don't sure what happen. As my understand, it may have some incompatible Unicode between Dataflow and GCS. Can anyone suggest me. Thank you so much.
Appreciated So much :)
val query = sc.bigQuerySelect(sql).map{
row =>
val user_name: String = row.getString("user_name")
val decryptJson = parse(user_name).getOrElse(Json.Null)
val name = decryptJson.hcursor.downField("a").downField("name").focus.get.asString.getOrElse("")
TableRow(
("name" -> name),
)
}
val res = query.filter(_ != null)
print(nonEmptyQueryResult)
res.saveAsTableRowJsonFile(outputPath, numShards=50)
sc.close()
sbt "runMain dev.test --project=test --runner=DataflowRunner --workerMachineType=n1-standard-1