no libopencv_core.so in java.library.path hadoop

Question

I was working in hadoop and suddenly once i created runnable jar for my image processing program this error occured it has relation with opencv native library path

while using eclipse the path i can set using :

// System.loadLibrary(Core.NATIVE_LIBRARY_NAME);

But while executing runnable jar using hadoop it gives error. Anyone who can rectify ?

    hduser@master:/home/mnh/Desktop$ hadoop jar  opencv19.jar /usr/local/hadoop/input/cars.mp4 /usr/local/hadoop/cars89
17/06/07 16:15:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/06/07 16:15:39 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.137.52:8050
17/06/07 16:15:40 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
17/06/07 16:16:08 INFO input.FileInputFormat: Total input paths to process : 1
17/06/07 16:16:08 INFO mapreduce.JobSubmitter: number of splits:1
17/06/07 16:16:09 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1496831815466_0009
17/06/07 16:16:09 INFO impl.YarnClientImpl: Submitted application application_1496831815466_0009
17/06/07 16:16:09 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1496831815466_0009/
17/06/07 16:16:09 INFO mapreduce.Job: Running job: job_1496831815466_0009
17/06/07 16:16:20 INFO mapreduce.Job: Job job_1496831815466_0009 running in uber mode : false
17/06/07 16:16:20 INFO mapreduce.Job:  map 0% reduce 0%
17/06/07 16:16:29 INFO mapreduce.Job: Task Id : attempt_1496831815466_0009_m_000000_0, Status : FAILED
Error: no libopencv_core.so in java.library.path
17/06/07 16:16:37 INFO mapreduce.Job: Task Id : attempt_1496831815466_0009_m_000000_1, Status : FAILED
Error: no libopencv_core.so in java.library.path
17/06/07 16:16:45 INFO mapreduce.Job: Task Id : attempt_1496831815466_0009_m_000000_2, Status : FAILED
Error: no libopencv_core.so in java.library.path
17/06/07 16:16:54 INFO mapreduce.Job:  map 100% reduce 100%
17/06/07 16:16:55 INFO mapreduce.Job: Job job_1496831815466_0009 failed with state FAILED due to: Task failed task_1496831815466_0009_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

17/06/07 16:16:56 INFO mapreduce.Job: Counters: 12
    Job Counters 
        Failed map tasks=4
        Launched map tasks=4
        Other local map tasks=3
        Data-local map tasks=1
        Total time spent by all maps in occupied slots (ms)=26582
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=26582
        Total vcore-seconds taken by all map tasks=26582
        Total megabyte-seconds taken by all map tasks=27219968
    Map-Reduce Framework
        CPU time spent (ms)=0
        Physical memory (bytes) snapshot=0
        Virtual memory (bytes) snapshot=0

Answer 1

The error message you are getting is related with the workers not being able to use OpenCV, because it is not installed on the respective machines. So the solution would be installing OpenCV on the cluster machines which have the Nodemanager component installed (those which will be executing the tasks). To be able to load the library using System.loadLibrary() method, you would need to add the location of the so file to the PATH system variable.