Distributed Cache Jar in classpath but getting ClassNotFoundException

Question

Using HBaseTestingUtility I would start a minicluster.

testUtil = new HBaseTestingUtility(configuration);
testUtil.startMiniCluster();

and provide my jar to place in DistributedCache's classpath

final String aJarResourcePath = Thread.currentThread().getContextClassLoader()
            .getResource("abc/a-1.0.jar").getPath();
    final FileSystem fs = FileSystem.get(configuration);
    final Path pathToArtifacts = new Path("/Runtime/a-artifacts");
        fs.mkdirs(pathToArtifacts);
    fs.copyFromLocalFile(new Path(aJarResourcePath), pathToArtifacts));

DistributedCache.addFileToClassPath(disqualified, configuration);

On my MapFn I am using the jar a-1.0.jar. I am able to see the jar in the MapFn

final Path[] fus = DistributedCache.getFileClassPaths(config);

So I am thinking the jar is in classpath and points to HDFS location but I get classnotfoundexception trying to create an object. I am not sure whats going on please help.

Answer 1

I don't know if this the cause of your problem, but the 2 argument overload of addFileToClasspath is deprecated in Hadoop 1.2.1. (Indeed in Hadoop 2.7.2, it appears that the DistributedCache class is deprecated in its entirety!) The javadocs don't say why the method has been deprecated, so it is hard to know if this could be the problem.

UPDATE

A bit more research suggests that your problem may actually be that the JAR file is not being expanded. Try using addArchiveToClasspath instead.