Unit Test Cases for Hive Custom UDFs by mocking Java Reflection calls

Question

I have a requirement where I need to develop Hive Custom UDFs that uses Java reflection API to make calls to external classes.

Since I'm new to Java Reflection, I have dedicated some time learning it and was able to do a basic implementation.

But I'm facing issues while writing unit test cases for this implementation as I'm facing some challenges in mocking Reflection APIs.

Below is the example for Hive Custom UDF.

ReverseString.java

public class ReverseString extends GenericUDF {

    private StringObjectInspector input;
    Class<?> c = null;
    Object ob = null;


    @Override
    public ObjectInspector initialize(ObjectInspector[] arg0) throws UDFArgumentException {

        // create an ObjectInspector for the input
        ObjectInspector input = arg0[0];

        // check to make sure the input is a string
        if (!(input instanceof StringObjectInspector)) {
            throw new UDFArgumentException("input must be a string");
        }


        this.input = (StringObjectInspector) input;
        System.out.println("Success. Input formatted correctly");

        return init(arg0);
    }

    public ObjectInspector init(ObjectInspector[] arg0) throws UDFArgumentException {

        try {
            c = Class.forName("com.hive.inherit.DummyUdf");
            ob = c.getConstructor().newInstance();
            Method method = c.getMethod("print",String.class);
            String res = (String) method.invoke(ob,"TEst");
            System.out.println("RES: "+res);

        } catch (NoSuchMethodException | ClassNotFoundException | IllegalAccessException | InvocationTargetException | InstantiationException e) {
            e.printStackTrace();
        }

        return PrimitiveObjectInspectorFactory.javaStringObjectInspector;
    }


    @Override
    public Object evaluate(DeferredObject[] arg0) throws HiveException {

        if (input == null || arg0.length != 1 || arg0[0].get() == null) {
            return null;
        }
        String forwards = input.getPrimitiveJavaObject(arg0[0].get());
        System.out.println("forwards:"+forwards);

        return reverse(forwards);
    }


    public  Object reverse(String in) {
        Object res = null ;
        try {
            if (this.c != null && this.ob != null) {
                Method method = this.c.getMethod("reverse", String.class);
                res = (String) method.invoke(this.ob, in);
            }else{
                c = Class.forName("com.hive.inherit.DummyUdf");
                ob = c.getConstructor().newInstance();
                Method method = c.getMethod("reverse", String.class);
                res = method.invoke(ob, in);
            }
        } catch (NoSuchMethodException | IllegalAccessException | InvocationTargetException | ClassNotFoundException | InstantiationException e) {
            e.printStackTrace();
        }
        return res;
    }

    @Override
    public String getDisplayString(String[] strings) {
        return null;
    }
}

Below is the class that is being called by Reflection.

DummyUdf.java

package com.hive.inherit;

public class DummyUdf {
    

    public DummyUdf(){
        System.out.println("DummyUdf");
    }

    public String print(String str){
        System.out.println("DummyUdf-str:"+str);
        return str;
    }

    public String reverse(String in) {

        int l = in.length();
        StringBuilder sb = new StringBuilder();

        for (int i = 0; i < l; i++) {
            sb.append(in.charAt(l - i - 1));
        }
        return sb.toString();
    }
}

The unit test cases that I'm trying to implement,

ReverseStringTest.class

@RunWith(MockitoJUnitRunner.class)
public class ReverseStringTest {

    @Test
    public void testSimpleString() throws HiveException {

        //ReverseString r = Mockito.spy(new ReverseString());
        ReverseString r = mock(ReverseString.class);
        ObjectInspector input = PrimitiveObjectInspectorFactory.javaStringObjectInspector;
        when(r.init(Mockito.any())).thenReturn(input);
        JavaStringObjectInspector resultInspector = (JavaStringObjectInspector) r.initialize(
                new ObjectInspector[] { input });
        Text forwards = new Text("hello");
        when(r.reverse(Mockito.any())).thenReturn("olleh");
        Object result = r.evaluate(new GenericUDF.DeferredObject[] { new GenericUDF.DeferredJavaObject(forwards) });
        System.out.println(result);
        assertEquals("olleh", resultInspector.getPrimitiveJavaObject(result));
    }
}

The test case is failing with NullPointerException.

java.lang.NullPointerException at com.hive.udf.ReverseStringTest.testSimpleString(ReverseStringTest.java:34) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)

Can someone please suggest on how to mock this appropriately?

Thanks in advance.

Answer 1

After some research, I found a working solution to the above question.

Below is the working solution:

public class ReverseStringTest {

    @Test
    public void testSimpleString() throws Exception {

        ReverseString reverseString = org.mockito.Mockito.spy(ReverseString.class);

        ObjectInspector[] arg0 = new ObjectInspector[] {PrimitiveObjectInspectorFactory.javaStringObjectInspector,
                PrimitiveObjectInspectorFactory.javaStringObjectInspector} ;

        when(reverseString.init(arg0)).thenReturn(PrimitiveObjectInspectorFactory.javaStringObjectInspector);
        reverseString.initialize(arg0);

        Text text = new Text("Ans");
        GenericUDF.DeferredObject[] deferredObjects = { new GenericUDF.DeferredJavaObject(text) };
        when(reverseString.reverse("Ans")).thenReturn(testcase());

        Object result = reverseString.evaluate(deferredObjects);
        System.out.println(result);
        Assertions.assertEquals(new Text("snA"),result);

    }

    public Object testcase() {
        return new Text("snA") ;

    }

Instead of 'mock', we can use 'spy' where part of the object will be mocked and part will use real method invocation (This helps to improve your code coverage as well if you want to consider that).

You can refer below link to understand more about Mockito 'mock' and 'spy'.

Mockito - @Spy vs @Mock

Since we have extended GenericUDF, and Hive implementation calls initialize method first, it is important to mock the arguments passed correctly before making a call to evaluate.