DL4J: binarizing middle layer of autoencoder for semantic hashing

Question

I'm trying to implement semantic hashing for the MNISTAutoencoder example, using DL4J. How would I binarize the middle layer activations? In the ideal case, I'm looking for some change to my network setup which gives (almost) binary activations for the middle layer out-of-the-box. Alternatively, I'm happy with some "receipt" to binarize the current RELU activations. Which of both approaches is favorable in terms of generalization capabilities?

My current network setup is:

MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()
    .seed(12345)
    .weightInit(WeightInit.XAVIER)
    .updater(new AdaGrad(0.05))
    .activation(Activation.RELU)
    .l2(0.0001)
    .list()
    .layer(new DenseLayer.Builder().nIn(784).nOut(250)
               .build())
    .layer(new DenseLayer.Builder().nIn(250).nOut(10)
               .build())
    .layer(new DenseLayer.Builder().nIn(10).nOut(250)
               .build())
    .layer(new OutputLayer.Builder().nIn(250).nOut(784)
               .activation(Activation.LEAKYRELU)
               .lossFunction(LossFunctions.LossFunction.MSE)
               .build())
    .build();

After 30 epochs, typical middle layer activations look like:

[[   11.3044,   12.3678,    7.3547,    1.6518,    1.0068,         0,    5.4340,    2.1388,    2.0708,    2.5764]]
[[    9.9051,   12.5345,   11.1941,    4.7900,    1.2935,         0,    7.9786,    4.1915,    3.1802,    7.5659]]
[[    6.4629,   11.1013,   10.8903,    5.4528,    0.8009,         0,    9.4881,    3.6684,    6.4524,    7.2334]]
[[    2.3953,    0.2429,    3.7125,    4.1561,    0.8607,         0,   11.2486,    7.0178,    2.8771,    2.1996]]
[[         0,    1.6378,    0.8993,    0.3347,    0.7708,         0,    3.7053,         0,    1.6704,    2.1380]]
[[         0,    1.5158,    0.7937,         0,    0.8190,         0,    4.7548,    0.0655,    1.4635,    1.8173]]
[[    6.8344,    5.9989,   10.1286,    2.8528,    1.1178,         0,    9.1865,   10.3677,    5.3564,    4.3420]]
[[    7.0942,    7.0364,    4.8538,    0.5096,    0.0442,         0,    8.4336,    8.2783,    5.6474,    3.8944]]
[[    3.6895,   14.9696,    6.5351,    8.0446,         0,         0,   12.7816,   12.7445,    7.8495,    3.8600]]

Answer 1

This can be established by assigning a custom IActivation function to the middle layer. For example:

public static class ActivationBinary extends BaseActivationFunction {
    public INDArray getActivation(INDArray in, boolean training) {
        in.replaceWhere(Nd4j.ones(in.length()).muli(-1), new LessThan(0));
        in.replaceWhere(Nd4j.ones(in.length()), new GreaterThanOrEqual(0));
        return in;
    }

    public org.nd4j.common.primitives.Pair<INDArray, INDArray> backprop(INDArray in, INDArray epsilon) {
        this.assertShape(in, epsilon);
        Nd4j.getExecutioner().execAndReturn(new TanhDerivative(in, epsilon, in)); // tanh's gradient is a reasonable approximation
        return new org.nd4j.common.primitives.Pair(in, (Object)null);
    }

    public int hashCode() {
        return 1;
    }

    public boolean equals(Object obj) {
        return obj instanceof ActivationBinary;
    }

    public String toString() {
        return "Binary";
    }
}