What are the JARs required for Hadoop project at startup?

Question

This must seem like an idiotic question but I am new to Hadoop and was trying to run a simple example. However, I keep running into problems with JAR files and dont seem to have the correct imports. I am using cloudera CDH 4 and it has a hole lot of JAR files to choose from with several versions of each. Since I am just starting off am not being able to differentiate. Presently i am getting this exception :

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:38)
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:36)
at org.apache.hadoop.security.UserGroupInformation$UgiMetrics.create(UserGroupInformation.java:120)
at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:236)
at org.apache.hadoop.mapreduce.task.JobContextImpl.<init>(JobContextImpl.java:72)
at org.apache.hadoop.mapreduce.Job.<init>(Job.java:144)
at org.apache.hadoop.mapreduce.Job.<init>(Job.java:131)
at org.apache.hadoop.mapreduce.Job.<init>(Job.java:139)
at com.ayon.hadoop.firstimpl.Driver1.main(Driver1.java:23)Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.Configuration
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 9 more

It is probably because of a wrong JAR file import. So my question is this:

Can anyone tell me what are the basic JAR files needed to start up a simple Hadoop job? Also any pointers to the above exception will be much appreciated.

Answer 1

If you're using Eclipse, follow below steps to add JARs to your project

1.Right click on the project
2.Go to Properties
3.Click on "Java Build Path"
4.Libraries Tab
5.Add External JARs
6.Select the path "/usr/local/hadoop"
7.Select all "JAR files" & add all JARs
8.Again "Add EXternal JARs"
9.Go to path "/usr/local/hadoop/lib"
10.ADD all JARs in there
11.Click "Ok"

You're set. Hope this helps.. again it depends on the Eclipse version you're using.

Answer 2

To include all the necessary jar files, compile your code using the following command:

# javac -cp `$HADOOP_HOME\bin\hadoop classpath` YourClassFile.java 

Please note the quotes