Apache Twill HelloWorld example fails

Question

When I run Apache twill HelloWorld example as stated in http://twill.incubator.apache.org/GettingStarted.html I get this log:

10:44:47.888 [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController - 
    Yarn application status for HelloWorldRunnable
    application_1443786884805_0185: ACCEPTED  
10:44:48.383 [ STARTING-SendThread(hadice.dev:2181)] DEBUG org.apache.zookeeper.ClientCnxn - 
    Got ping response for sessionid: 0x15028da0ff0009d after 0ms  
10:44:48.889 [IPC Parameter Sending Thread #0] DEBUG org.apache.hadoop.ipc.Client - 
    IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root sending #45  
10:44:48.894 [IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root] 
    DEBUG org.apache.hadoop.ipc.Client - 
    IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root got value #45  
10:44:48.894 [ STARTING] DEBUG o.a.hadoop.ipc.ProtobufRpcEngine - Call: getApplicationReport took 6ms  
10:44:48.895 [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController - 
    Yarn application status for HelloWorldRunnable application_1443786884805_0185: 
    ACCEPTED  
10:44:49.711 [Kafka-Consumer-log-0] DEBUG o.a.t.i.k.client.SimpleKafkaConsumer - 
    No leader for topic partition TopicPartition{topic=log, partition=0}.  
10:44:49.895 [IPC Parameter Sending Thread #0] DEBUG org.apache.hadoop.ipc.Client - 
    IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root sending #46  
10:44:49.902 [IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root] DEBUG org.apache.hadoop.ipc.Client -
    IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root got value #46  
10:44:49.902 [ STARTING] DEBUG o.a.hadoop.ipc.ProtobufRpcEngine - Call: getApplicationReport took 7ms  
10:44:49.902 [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController - 
    Yarn application status for HelloWorldRunnable application_1443786884805_0185: 
    FAILED  
10:44:50.902 [ STARTING] INFO  o.a.twill.yarn.YarnTwillController - 
    Yarn application HelloWorldRunnable application_1443786884805_0185 is in state 
    FAILED  
10:44:50.903 [ STARTING] INFO  o.a.twill.yarn.YarnTwillController - 
    Yarn application HelloWorldRunnable application_1443786884805_0185 is not in running state. Shutting down controller.  
10:44:50.907 [IPC Parameter Sending Thread #0] DEBUG org.apache.hadoop.ipc.Client - 
    IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root sending #47  
10:44:50.908 [ STARTING-SendThread(hadice.dev:2181)] DEBUG org.apache.zookeeper.ClientCnxn - 
    Reading reply sessionid:0x15028da0ff0009d, packet:: 
    clientPath:/HelloWorldRunnable/instances/5e72cb8c-cf94-4718-a44b-ec983304efa0 
    serverPath:/HelloWorldRunnable/instances/5e72cb8c-cf94-4718-a44b-ec983304efa0 
    finished:false header:: 10,3  replyHeader:: 10,1797,-101  request:: '/HelloWorldRunnable/instances/5e72cb8c-cf94-4718-a44b-ec983304efa0,T  response::  
10:44:50.913 [IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root] DEBUG org.apache.hadoop.ipc.Client - 
    IPC Client (431687661) connection to gin1.dev/10.0.22.129:8032 from root got value #47  
10:44:50.913 [ STOPPING] DEBUG o.a.hadoop.ipc.ProtobufRpcEngine - Call: getApplicationReport took 6ms  
10:44:50.916 [ STOPPING] DEBUG o.a.twill.yarn.YarnTwillController - 
    Yarn application HelloWorldRunnable application_1443786884805_0185 completed with status 
    FAILED

The Application gets ACCEPTed but then transitions to the "FAILED" state.

The YARN Web UI shows this as the error (very unspecific):

Application application_1443786884805_0185 failed 2 times due to AM Container for appattempt_1443786884805_0185_000002 exited with exitCode: 1
For more detailed output, check application tracking page:http://gin1.dev:8088/proxy/application_1443786884805_0185/Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1443786884805_0185_02_000001
Exit code: 1 
Stack trace: ExitCodeException exitCode=1:  
  at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
  at org.apache.hadoop.util.Shell.run(Shell.java:455)
  at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
  at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
  at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
  at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
Failing this attempt. Failing the application. 

And the node log shows:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
  at java.lang.Class.getDeclaredMethods0(Native Method)
  at java.lang.Class.privateGetDeclaredMethods(Class.java:2693)
  at java.lang.Class.privateGetMethodRecursive(Class.java:3040)
  at java.lang.Class.getMethod0(Class.java:3010)
  at java.lang.Class.getMethod(Class.java:1776)
  at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:85)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
  at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  ... 6 more

What can be wrong? I supposed that hadoop classes should be on classpath in yarn application. How to fix that?

Answer 1

I assume you are launching the application with the result of hadoop classpath in the classpath (as shown in the example). You need to make sure the result of running hadoop classpath on launcher box points to local paths that contain hadoop jars. The other thing you might want to check is the stdout file in the container log directory. It prints out classpath that it uses to launch the application and see if you find hadoop jars there.