I have created a emr serverless application to run hudi spark job but neither of driver and executor logs are having logs related to hudi.
I tried setting applicationProperties of emr serverless app to enable logs for hudi but nothing worked.
{
"runtimeConfiguration": [
{
"classification": "spark-driver-log4j2",
"configurations": null,
"properties": {
"rootLogger.level": "debug",
"logger.HoodieMultiTableStreamer.level": "debug",
"logger.HoodieMultiTableStreamer.name": "org.apache.hudi.utilities.streamer",
"logger.hudi.level": "debug",
"logger.hudi.name": "org.apache.hudi"
}
},
{
"classification": "spark-executor-log4j2",
"configurations": null,
"properties": {
"rootLogger.level": "debug",
"logger.HoodieMultiTableStreamer.level": "debug",
"logger.HoodieMultiTableStreamer.name": "org.apache.hudi.utilities.streamer",
"logger.hudi.level": "debug",
"logger.hudi.name": "org.apache.hudi"
}
}
]
}
Can someone tell whats missing? TIA