Cloudera Manager: Where do I put Java ClassPath for MapReduce jobs?

Question

I've got Hadoop-Lzo working happily on my local pseudo-cluster but the second I try the same jar file in production, I get:

java.lang.RuntimeException: native-lzo library not available

The libraries are verified to be on the DataNodes, so my question is:

In what screen / setting do I specify the location of the native-lzo library?

Answer 1

For MapReduce you need to add the entries to the MapReduce Client Environment Safety valve. You can find MapReduce Client Safety by going to View and Edit tab under Configuration. Then add these lines over there :

• HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/opt/cloudera/parcels/HADOOP_LZO/lib/hadoop/lib/*
• JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:/opt/cloudera/parcels/HADOOP_LZO/lib/hadoop/lib/native

Also add the LZO codecs to the io.compression.codecs property under the MapReduce Service. To do that go to io.compression under View and Edit tab under Configuration and these lines :

• com.hadoop.compression.lzo.LzoCodec
• com.hadoop.compression.lzo.LzopCodec

Do not forget to restart your MR daemons after making the changes. Once restarted redeploy your MR client configuration.

For a detailed help on how to use LZO you can visit this link : http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/latest/Cloudera-Manager-Installation-Guide/cmig_install_LZO_Compression.html

HTH

Answer 2

try sudo apt-get install lzop in your TaskTracker nodes.