HBase Failed to Start with "FileNotFoundException" in RegionServer Logs

Symptom

Cause Analysis

1. Check the RegionServer log (/var/log/Bigdata/hbase/rs/hbase-omm-XXX.out). It is found that the following information is printed:

    | ERROR | RS_OPEN_REGION-ab-dn01:21302-2 | ABORTING region server ab-dn01,21302,1487663269375: The coprocessor org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService threw java.io.FileNotFoundException: File does not exist: hdfs://hacluster/kylin/kylin_metadata/coprocessor/kylin-coprocessor-1.6.0-SNAPSHOT-0.jar | org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2123)
   java.io.FileNotFoundException: File does not exist: hdfs://hacluster/kylin/kylin_metadata/coprocessor/kylin-coprocessor-1.6.0-SNAPSHOT-0.jar
   at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1399)
   at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1391)
   at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
   at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1391)
   at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
   at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
   at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2038)
   at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2007)
   at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1983)
   at org.apache.hadoop.hbase.util.CoprocessorClassLoader.init(CoprocessorClassLoader.java:168)
   at org.apache.hadoop.hbase.util.CoprocessorClassLoader.getClassLoader(CoprocessorClassLoader.java:250)
   at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:224)
   at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:365)
   at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:227)
   at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:783)
   at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:689)
   at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source)
   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
   at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6312)
   at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6622)
   at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6594)
   at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6550)
   at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6501)
   at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:363)
   at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)
   at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   at java.lang.Thread.run(Thread.java:745)
   
   

2. Run the hdfs command on the client. It is found that the following file does not exist:

   hdfs://hacluster/kylin/kylin_metadata/coprocessor/kylin-coprocessor-1.6.0-SNAPSHOT-0.jar

3. When configuring the coprocessor for HBase, make sure that the path of the corresponding JAR package is correct. Otherwise, HBase cannot be started.

Solution