On Mac OS X, when I use compression codec is hadoop I get following error:
java.lang.IllegalArgumentException: SequenceFile doesn't work with GzipCodec without native-hadoop code!
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:386)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:354)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:476)
at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:61)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.(ReduceTask.java:569)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:638)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:260)
But I have configured native libraries. (I've build hadoop locally and copy native libs to hadoop dir.
hadoop checknative -a
16/06/06 13:16:54 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
16/06/06 13:16:54 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop: true /opt/hadoop-2.5.0-cdh5.3.3/lib/native/libhadoop.dylib
zlib: true /usr/lib/libz.1.dylib
snappy: true /opt/hadoop-2.5.0-cdh5.3.3/lib/native/libsnappy.1.dylib
lz4: true revision:99
bzip2: false
openssl: false org.apache.hadoop.crypto.OpensslCipher.initIDs()V
16/06/06 13:16:54 INFO util.ExitUtil: Exiting with status 1
So it looks like native libs are OK.
PS: when I use B2ZIPCodec everything works fine.
PSS: mayby my .profile helps to understand what is going on?
export JAVA_LIBRARY_PATH=/opt/hadoop-2.5.0-cdh5.3.3/lib/native
export LD_LIBRARY_PATH=/opt/hadoop-2.5.0-cdh5.3.3/lib/native
export HADOOP_HOME=/opt/hadoop-2.5.0-cdh5.3.3
export M2_HOME="/opt/maven/"
export MAVEN_OPTS="-Xms1024m -Xmx3000m -XX:MaxPermSize=228m -Dmaven.surefire.debug=-Xmx3000m"
export JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.7.0_72.jdk/Contents/Home"
export PATH=${JAVA_HOME}/bin:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:${HCAT_HOME}/bin:${M2_HOME}/bin:${PATH}
UPDATE This configuration worked fine half a year ago. After some software update it was broken.