Usage of arpack-ng/openblas/javacpp in the smile java library

Question

I'm trying to set up the Smile machine learning library in Java but I'm having some issues getting some dependencies to work. Currently I am on a Fedora machine, but I'd like this to work on any machine, just like a normal jar.

Here is some example code where an error occurs:

double threshold = 1;
DistanceMeasure measure = new EuclideanDistance();
double[][] data = embedding.getVectors();
AdjacencyMatrix adj = new AdjacencyMatrix(embedding.getNumberOfPoints());
  for(int i = 0; i < embedding.getNumberOfPoints(); i++) {
    for(int j = 0; j < embedding.getNumberOfPoints(); j++ ) {
        if(i == j) {
            adj.addEdge(i, j, threshold);
            continue;
        }
        double weight = measure.compute(data[i], data[j]);
        if(weight <= threshold) {
            adj.addEdge(i, j, weight);
        }
    }
}
SpectralClustering clusters = SpectralClustering.fit(adj.toMatrix(), 3); // <-- Error here
double[][] reduced = PCA.fit(data).setProjection(3).project(data);
ScatterPlot.of(reduced, clusters.y, '*').canvas().window();

Now the SpectralClustering class makes use of some extra dependencies. From the documentation I could find (scroll down to the part that mentions 'Some algorithms rely on BLAS and LAPACK', I gather that one is supposed to add these dependencies to your pom.xml:

    <dependency>
        <groupId>org.bytedeco</groupId>
        <artifactId>javacpp</artifactId>
        <version>1.5.5</version>
    </dependency>
    <dependency>
        <groupId>org.bytedeco</groupId>
        <artifactId>openblas</artifactId>
        <version>0.3.13-1.5.5</version>
    </dependency>
    <dependency>
        <groupId>org.bytedeco</groupId>
        <artifactId>arpack-ng</artifactId>
        <version>3.8.0-1.5.5</version>
    </dependency>

This, however, results in these 3 errors:

1. no jnijavacpp in java.library.path
2. no jniopenblas_nolapack in java.library.path
3. no openblas_nolapack in java.library.path

So I do some more research and I come across these different dependencies. (I am not sure how '-platform' differs.) When I replace the above with these 3 new dependencies:

    <dependency>
        <groupId>org.bytedeco</groupId>
        <artifactId>javacpp-platform</artifactId>
        <version>1.5.5</version>
    </dependency>
    <dependency>
        <groupId>org.bytedeco</groupId>
        <artifactId>openblas-platform</artifactId>
        <version>0.3.13-1.5.5</version>
    </dependency>
    <dependency>
        <groupId>org.bytedeco</groupId>
        <artifactId>arpack-ng-platform</artifactId>
        <version>3.8.0-1.5.5</version>
    </dependency>

This time we make some progress (I hope) and get a new error:

Exception in thread "main" java.lang.NoSuchMethodError: void org.bytedeco.arpackng.global.arpack.dseupd_c(boolean, byte[], int[], double[], double[], int, double, byte[], int, byte[], int, double, double[], int, double[], int, int[], int[], double[], double[], int, int[])'
at smile.math.matrix.ARPACK.syev(ARPACK.java:180)

Can anyone help me with the last part? Do I need to add another dependency or do I need to install software externally with 'dnf install ...'? Perhaps the Smile class needs a specific version of the dseupd_c method?

Cheers

Answer 1

Wow, it seems I was near to finding the answer myself. It turns out you need to use the '-platform' version and these specific versions in your pom.xml:

    <dependency>
        <groupId>org.bytedeco</groupId>
        <artifactId>javacpp-platform</artifactId>
        <version>1.5.3</version>
    </dependency>
    <dependency>
        <groupId>org.bytedeco</groupId>
        <artifactId>openblas-platform</artifactId>
        <version>0.3.9-1.5.3</version>
    </dependency>
    <dependency>
        <groupId>org.bytedeco</groupId>
        <artifactId>arpack-ng-platform</artifactId>
        <version>3.7.0-1.5.3</version>
    </dependency>

This made my code run without error. I hope this will be of use to someone else some day.

Cheers!

Answer 2

Very helpful thread, thanks! Here's my contribution:

The versions change over time, so just match them up to what you find here:

https://github.com/haifengl/smile/blob/master/core/build.sbt

which at this time is:

    <dependency>
        <groupId>org.bytedeco</groupId>
        <artifactId>javacpp-platform</artifactId>
        <version>1.5.8</version>
    </dependency>
    <dependency>
        <groupId>org.bytedeco</groupId>
        <artifactId>openblas-platform</artifactId>
        <version>0.3.21-1.5.8</version>
    </dependency>
    <dependency>
        <groupId>org.bytedeco</groupId>
        <artifactId>arpack-ng-platform</artifactId>
        <version>3.8.0-1.5.8</version>
    </dependency>