I have a YARN Cluster with 2 nodes. I have validated the YARN cluster is running fine. I am trying to get XD deployed.
I followed the steps in latest XD guide a Configured Zk, Postgres, Redis and modified the servers.yml accordingly. No error messages after executing start commands. But I don't see any applications running in YARN after executing start admin and start container. Also didn't see anything in yarn applicationDir I configured.
java version "1.7.0_55"
OS version Linux 2.6.32-431.20.3.el6.x86_64 x86_64 GNU/Linux, RHEL 6.5
Also where can I see log messages?
Couple of general questions on install
Where should I copy YARN binaries? In any of the YARN cluster nodes? Or all of them? Or can I just copy in my local and just yarn-push will take care of the actual install?
Should the applicationDir be created on all YARN nodes or just on the Resource Manager?
If the XD guide can call out these explicitly, it will be great.
[jmupfnd@awsupfxdqa1nodejs1 bin]$ cd ..
[jmupfnd@awsupfxdqa1nodejs1 spring-xd-1.0.1.RELEASE-yarn]$ bin/xd-yarn push
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v1.1.7.RELEASE)
2014-12-01 09:07:54,799 INFO [ConfiguringBeanFactoryPostProcessor] - No bean named 'taskScheduler' has been explicitly defined. Therefore, a default ThreadPoolTaskScheduler will be created.
2014-12-01 09:07:54,808 INFO [ConfiguringBeanFactoryPostProcessor] - No bean named 'taskExecutor' has been explicitly defined. Therefore, a default SyncTaskExecutor will be created.
2014-12-01 09:07:55,856 INFO [SpringYarnConfiguration] - Enabling CLIENT for Yarn
2014-12-01 09:07:55,875 INFO [SpringYarnConfiguration] - We couldn't figure out if we could use existing configuration
2014-12-01 09:07:55,875 INFO [SpringYarnConfiguration] - Building configuration for bean 'yarnConfiguration'
2014-12-01 09:07:55,907 INFO [SpringYarnConfigBuilder] - Existing yarnConfiguration: null
2014-12-01 09:07:56,084 INFO [ConfigurationFactoryBean] - Overwriting fsUri=[file:///] with fsUri=[hdfs://10.94.70.3:8020]
2014-12-01 09:07:56,085 INFO [ConfigurationFactoryBean] - Overwriting rmAddress=[0.0.0.0:8032] with rmAddress=[10.94.70.3:8032]
2014-12-01 09:07:56,091 INFO [SpringYarnConfigBuilder] - Setting configuration for SpringYarnConfigs: fs.defaultFS=hdfs://10.94.70.3:8020 yarn.resourcemanager.address=10.94.70.3:8032 Configuration: core-default.xml, core-site.xml, yarn-default.xml, yarn-site.xml
2014-12-01 09:07:56,097 INFO [EnvironmentFactoryBean] - Adding CLASSPATH=./*
2014-12-01 09:07:56,431 WARN [NativeCodeLoader] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-12-01 09:07:57,146 INFO [AbstractResourceLocalizer] - About to copy localized files
[jmupfnd@awsupfxdqa1nodejs1 spring-xd-1.0.1.RELEASE-yarn]$ ls -l data
total 0
[jmupfnd@awsupfxdqa1nodejs1 spring-xd-1.0.1.RELEASE-yarn]$ ls -l
total 124564
drwxr-xr-x 2 jmupfnd gjmupfnd 4096 Dec 1 08:57 bin
drwxr-xr-x 2 jmupfnd gjmupfnd 4096 Dec 1 09:05 config
-rw-r--r-- 1 jmupfnd gjmupfnd 417 Oct 2 11:36 custom-modules.zip
drwxr-x--- 2 jmupfnd gjmupfnd 4096 Dec 1 08:56 data
drwxr-xr-x 2 jmupfnd gjmupfnd 4096 Oct 2 11:36 lib
-rw-r--r-- 1 jmupfnd gjmupfnd 49114 Oct 2 11:07 LICENSE
-rw-r----- 1 jmupfnd gjmupfnd 7338 Dec 1 09:07 manifest.yml
-rw-r--r-- 1 jmupfnd gjmupfnd 1456 Oct 2 11:07 README
-rw-r--r-- 1 jmupfnd gjmupfnd 127340310 Oct 2 11:36 spring-xd-yarn-1.0.1.RELEASE.zip
[jmupfnd@awsupfxdqa1nodejs1 spring-xd-1.0.1.RELEASE-yarn]$ ./bin/xd-yarn start admin
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v1.1.7.RELEASE)
2014-12-01 09:08:43,382 INFO [ConfiguringBeanFactoryPostProcessor] - No bean named 'taskScheduler' has been explicitly defined. Therefore, a default ThreadPoolTaskScheduler will be created.
2014-12-01 09:08:43,403 INFO [ConfiguringBeanFactoryPostProcessor] - No bean named 'taskExecutor' has been explicitly defined. Therefore, a default SyncTaskExecutor will be created.
2014-12-01 09:08:44,516 INFO [SpringYarnConfiguration] - Enabling CLIENT for Yarn
2014-12-01 09:08:44,530 INFO [SpringYarnConfiguration] - We couldn't figure out if we could use existing configuration
2014-12-01 09:08:44,531 INFO [SpringYarnConfiguration] - Building configuration for bean 'yarnConfiguration'
2014-12-01 09:08:44,556 INFO [SpringYarnConfigBuilder] - Existing yarnConfiguration: null
2014-12-01 09:08:44,734 INFO [ConfigurationFactoryBean] - Overwriting fsUri=[file:///] with fsUri=[hdfs://10.94.70.3:8020]
2014-12-01 09:08:44,735 INFO [ConfigurationFactoryBean] - Overwriting rmAddress=[0.0.0.0:8032] with rmAddress=[10.94.70.3:8032]
2014-12-01 09:08:44,741 INFO [SpringYarnConfigBuilder] - Setting configuration for SpringYarnConfigs: fs.defaultFS=hdfs://10.94.70.3:8020 yarn.resourcemanager.address=10.94.70.3:8032 Configuration: core-default.xml, core-site.xml, yarn-default.xml, yarn-site.xml
2014-12-01 09:08:44,748 INFO [EnvironmentFactoryBean] - Adding CLASSPATH=./*
2014-12-01 09:08:45,156 WARN [NativeCodeLoader] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-12-01 09:08:46,043 INFO [AbstractYarnClient] - submitApplication, got applicationId=[application_1417028324995_0004]
2014-12-01 09:08:46,044 INFO [AbstractResourceLocalizer] - Setting stagingId=application_1417028324995_0004
2014-12-01 09:08:46,044 INFO [AbstractResourceLocalizer] - Marking distributed state false
2014-12-01 09:08:46,044 INFO [AbstractResourceLocalizer] - About to resolve localized files
[jmupfnd@awsupfxdqa1nodejs1 spring-xd-1.0.1.RELEASE-yarn]$ yarn application -list
14/12/01 09:09:03 INFO client.RMProxy: Connecting to ResourceManager at /10.94.70.3:8032
14/12/01 09:09:03 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Total number of applications (application-types: [] and states: [SUBMITTED, ACCEPTED, RUNNING]):0
Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL
[jmupfnd@awsupfxdqa1nodejs1 spring-xd-1.0.1.RELEASE-yarn]$ ./bin/xd-yarn start container
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v1.1.7.RELEASE)
2014-12-01 09:09:44,199 INFO [ConfiguringBeanFactoryPostProcessor] - No bean named 'taskScheduler' has been explicitly defined. Therefore, a default ThreadPoolTaskScheduler will be created.
2014-12-01 09:09:44,210 INFO [ConfiguringBeanFactoryPostProcessor] - No bean named 'taskExecutor' has been explicitly defined. Therefore, a default SyncTaskExecutor will be created.
2014-12-01 09:09:45,307 INFO [SpringYarnConfiguration] - Enabling CLIENT for Yarn
2014-12-01 09:09:45,326 INFO [SpringYarnConfiguration] - We couldn't figure out if we could use existing configuration
2014-12-01 09:09:45,326 INFO [SpringYarnConfiguration] - Building configuration for bean 'yarnConfiguration'
2014-12-01 09:09:45,353 INFO [SpringYarnConfigBuilder] - Existing yarnConfiguration: null
2014-12-01 09:09:45,532 INFO [ConfigurationFactoryBean] - Overwriting fsUri=[file:///] with fsUri=[hdfs://10.94.70.3:8020]
2014-12-01 09:09:45,533 INFO [ConfigurationFactoryBean] - Overwriting rmAddress=[0.0.0.0:8032] with rmAddress=[10.94.70.3:8032]
2014-12-01 09:09:45,539 INFO [SpringYarnConfigBuilder] - Setting configuration for SpringYarnConfigs: fs.defaultFS=hdfs://10.94.70.3:8020 yarn.resourcemanager.address=10.94.70.3:8032 Configuration: core-default.xml, core-site.xml, yarn-default.xml, yarn-site.xml
2014-12-01 09:09:45,547 INFO [EnvironmentFactoryBean] - Adding CLASSPATH=./*
2014-12-01 09:09:45,940 WARN [NativeCodeLoader] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-12-01 09:09:46,861 INFO [AbstractYarnClient] - submitApplication, got applicationId=[application_1417028324995_0005]
2014-12-01 09:09:46,861 INFO [AbstractResourceLocalizer] - Setting stagingId=application_1417028324995_0005
2014-12-01 09:09:46,862 INFO [AbstractResourceLocalizer] - Marking distributed state false
2014-12-01 09:09:46,862 INFO [AbstractResourceLocalizer] - About to resolve localized files
[jmupfnd@awsupfxdqa1nodejs1 spring-xd-1.0.1.RELEASE-yarn]$ yarn application -list
14/12/01 09:09:52 INFO client.RMProxy: Connecting to ResourceManager at /10.94.70.3:8032
14/12/01 09:09:52 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Total number of applications (application-types: [] and states: [SUBMITTED, ACCEPTED, RUNNING]):0
Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL
[jmupfnd@awsupfxdqa1nodejs1 spring-xd-1.0.1.RELEASE-yarn]$