MRunit: how to test custom sortcomparator?

459 Views Asked by At

I'm trying to use MRunit to test my sortComparatorClass. It seems MRunit should be able to do this with the setKeyOrderComparator method, but when I run the mapReduceDriver it is not calling the compare() method of SortComparator class.
Pretty sure I'm doing something wrong with MRunit API.

Here's my code for the unit test:

public class UnitTests {

private static transient Log log = LogFactory.getLog(UnitTests.class);

MapReduceDriver<Text, Text, Text, Text, Text, Text> mapReduceDriver;
MapDriver<Text, Text, Text, Text> mapDriver;
ReduceDriver<Text, Text, Text, Text> reduceDriver;

@Before
public void setUp() throws InterruptedException, IOException {
    mapDriver = new MapDriver<Text, Text, Text, Text>();
    mapDriver.setMapper(new TestMapper());
    reduceDriver = new ReduceDriver<Text, Text, Text, Text>();
    reduceDriver.setReducer(new TestReducer());
    mapReduceDriver = new MapReduceDriver(new TestMapper(), new TestReducer());
    mapReduceDriver.setKeyOrderComparator(new TestSortCompartor());
}

@Test
public void testSort() throws IOException {

    Text inputKey1 = new Text("def");
    Text inputKey2 = new Text("abc");
    Text inputValue = new Text("BlahBlahBlah");
    mapReduceDriver.addInput(inputKey1, inputValue);
    mapReduceDriver.addInput(inputKey2, inputValue);

    List<Pair<Text, Text>> output = mapReduceDriver.run();
    log.info("Got output of size "+output.size()+" with first pair = "+output.get(0).toString());
}
}

And here's my test sortComparator:

public class TestSortCompartor extends WritableComparator{
private static transient Log log = LogFactory.getLog(TestSortCompartor.class);

public TestSortCompartor() {
    super(Text.class, true);
}

@SuppressWarnings("rawtypes")
@Override
public int compare(WritableComparable w1, WritableComparable w2) {
    log.info("calling compare with key1 = "+w1.toString()+" and key2 "+w2.toString());
    return w1.compareTo(w2) * -1;
}
}

When I run the test I get this output:

INFO 2014-01-13 09:34:27,362 [main] (com.gradientx.gxetl.testMR.UnitTests:53) - 
Got output of size 2 with first pair = (abc, BlahBlahBlah)

But there's no output from the sortComparator class - and it's not ordering the keys in reverse, so I know my comparator() method is not being called.

Can anyone advise what am I doing wrong? Is it possible to use MRunit to test my own comparator class? Is there a better way to make a unit test for a custom comparator class?

FYI, here are the relevant dependencies from my Pom:

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-core</artifactId>
    <version>2.0.0-mr1-cdh4.4.0</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>2.0.0-mr1-cdh4.4.0</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.mrunit</groupId>
    <artifactId>mrunit</artifactId>
    <version>1.0.0</version>
    <classifier>hadoop2</classifier>
</dependency>
0

There are 0 best solutions below