TableNotFoundException when upgrading hbase client from 0.98 to 1.1

Question

I've just upgraded to HortonWorks HBase client 1.1.2.2.4.2.0-258 from version 0.98. All works fine locally, but under production load after a couple of minutes I'm starting to have TableNotFoundException:

Caused by: org.apache.hadoop.hbase.TableNotFoundException
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1264)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1162)
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:152)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)

The behaviour seems random and unpredictable, e.g. repeating the same request just works (i.e. no exception is thrown and data are successfully retrieved)

I was trying to understand what was changed from 0.98 to 1.1 and the only significant thing I found was https://issues.apache.org/jira/browse/HBASE-16973

Playing around with those values didn't help.

Any thing else I should take into account? Any pointers are highly appreciated!

Thanks!

Answer 1

The problem was in the way how we used HConnection object, there were several instances of them (which, I agree, is not the right way to use it, IIUC). In the new version of the hbase client, there are also several "hconnection" threads for meta lookup per HConnection. All of this accumulated leading to the extremely high number of threads that JVM were not able to handle. After using a single HConnection instance, the problem went away and everything works as expected.