With the release of AMI 3.3.0, AWS supports Hue as an installable "app" in EMR, like Hive/Pig. Using the EMR web UI, creating a cluster with Hue works fine for me, however when adding a Hue installation bootstrap action via Boto, I am getting a non-deterministic error (it periodically crashes). I have tested 4 times with identical configuration and the crash rate is 50%.
In Boto, I add an additional bootstrap action, as is done automatically when creating a cluster from the web UI when Hue is enabled:
BootstrapAction('Install Hue', 's3://elasticmapreduce/libs/hue/install-hue', [])
The cluster then terminates with a:
Terminated with errors: On the master instance (i-c6b7582a),
bootstrap action 2 returned a non-zero return code
And in the bootstrap action logs:
Existing lock /var/run/yum.pid: another copy is running as pid 2007.
Another app is currently holding the yum lock; waiting for it to exit...
The other application is: yum
Memory : 22 M RSS (305 MB VSZ)
Started: Tue Nov 11 21:00:12 2014 - 00:19 ago
State : Sleeping, pid: 2007
Another app is currently holding the yum lock; waiting for it to exit...
Tons of those, and finally a large stacktrace:
Trying other mirror.
http://packages.ap-southeast-2.amazonaws.com/2014.09/main/20140901f63e/x86_64/repodata/repomd.xml?instance_id=i-c6b7582a®ion=us-east-1: [Errno 12] Timeout on http://packages.ap-southeast-2.amazonaws.com/2014.09/main/20140901f63e/x86_64/repodata/repomd.xml?instance_id=i-c6b7582a®ion=us-east-1: (28, 'Connection timed out after 10000 milliseconds')
Trying other mirror.
Traceback (most recent call last):
File "/usr/bin/yum", line 29, in <module>
yummain.user_main(sys.argv[1:], exit_code=True)
File "/usr/share/yum-cli/yummain.py", line 355, in user_main
errcode = main(args)
File "/usr/share/yum-cli/yummain.py", line 174, in main
result, resultmsgs = base.doCommands()
File "/usr/share/yum-cli/cli.py", line 572, in doCommands
return self.yum_cli_commands[self.basecmd].doCommand(self, self.basecmd, self.extcmds)
File "/usr/share/yum-cli/yumcommands.py", line 432, in doCommand
return base.installPkgs(extcmds, basecmd=basecmd)
File "/usr/share/yum-cli/cli.py", line 968, in installPkgs
txmbrs = self.install(pattern=arg)
File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 4721, in install
mypkgs = self.pkgSack.returnPackages(patterns=pats,
File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 1069, in <lambda>
pkgSack = property(fget=lambda self: self._getSacks(),
File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 774, in _getSacks
self.repos.populateSack(which=repos)
File "/usr/lib/python2.6/site-packages/yum/repos.py", line 383, in populateSack
sack.populate(repo, mdtype, callback, cacheonly)
File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 250, in populate
if self._check_db_version(repo, mydbtype):
File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 342, in _check_db_version
return repo._check_db_version(mdtype)
File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1520, in _check_db_version
repoXML = self.repoXML
File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1706, in <lambda>
repoXML = property(fget=lambda self: self._getRepoXML(),
File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1702, in _getRepoXML
self._loadRepoXML(text=self.ui_id)
File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1693, in _loadRepoXML
return self._groupLoadRepoXML(text, self._mdpolicy2mdtypes())
File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1667, in _groupLoadRepoXML
if self._commonLoadRepoXML(text):
File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1495, in _commonLoadRepoXML
self._revertOldRepoXML()
File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1345, in _revertOldRepoXML
os.rename(old_data['old_local'], old_data['local'])
OSError: [Errno 2] No such file or directory
In contrast, the bootstrap log shows a single line on success:
Warning: RPMDB altered outside of yum.
Example to install and run Hue in EMR AMI 3.3