I save the data in json file and upload to gcs but got this unicode-context. "���.���"

70 Views Asked by At

I'new for GCP Tools and Scala. I have this issue. I create the job to upload json file to GCS bucket via DataflowRunner. But, when I check the data inside file. it shows like this. ���.��� ( Actually it should show Thai Language). But, when i dev in local and run with command DirectRunner. It show data correctly. I don't sure what happen. As my understand, it may have some incompatible Unicode between Dataflow and GCS. Can anyone suggest me. Thank you so much.

Appreciated So much :)

        val query = sc.bigQuerySelect(sql).map{
            row => 
                val user_name: String = row.getString("user_name")     
                val decryptJson = parse(user_name).getOrElse(Json.Null)
                val name = decryptJson.hcursor.downField("a").downField("name").focus.get.asString.getOrElse("")  
                    TableRow(
                        ("name" -> name),
                       
                    )
        }
        val res = query.filter(_ != null)
        print(nonEmptyQueryResult)
        res.saveAsTableRowJsonFile(outputPath, numShards=50)
        sc.close()

sbt "runMain dev.test --project=test --runner=DataflowRunner --workerMachineType=n1-standard-1

0

There are 0 best solutions below