How to see my application's exception in hdinsight

Question

How to see my application's exception in hdinsight?

I created an hadoop stream job, when I run my job, it fails with

Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 255
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)
    at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
    at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

As far as I know, this is because my code has some bug and throws an exception out and then crash, how can I get the exception infomation? Is there a application log or something for hdinsight?

Answer 1

RDP to the head node (you will have to enable remote to the hdinsight cluster) and click on the yarn UI shortcut on the desktop. This will show the task logs.