HBase Failed to Start with "FileNotFoundException" in RegionServer Logs
Symptom
HBase fails to start, and the RegionServer stays in the Restoring state.
Cause Analysis
- Check the RegionServer log (/var/log/Bigdata/hbase/rs/hbase-omm-XXX.out). It is found that the following information is printed:| ERROR | RS_OPEN_REGION-ab-dn01:21302-2 | ABORTING region server ab-dn01,21302,1487663269375: The coprocessor org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService threw java.io.FileNotFoundException: File does not exist: hdfs://hacluster/kylin/kylin_metadata/coprocessor/kylin-coprocessor-1.6.0-SNAPSHOT-0.jar | org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2123)java.io.FileNotFoundException: File does not exist: hdfs://hacluster/kylin/kylin_metadata/coprocessor/kylin-coprocessor-1.6.0-SNAPSHOT-0.jarat org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1399)at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1391)at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1391)at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2038)at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2007)at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1983)at org.apache.hadoop.hbase.util.CoprocessorClassLoader.init(CoprocessorClassLoader.java:168)at org.apache.hadoop.hbase.util.CoprocessorClassLoader.getClassLoader(CoprocessorClassLoader.java:250)at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:224)at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:365)at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:227)at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:783)at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:689)at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6312)at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6622)at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6594)at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6550)at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6501)at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:363)at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)
- Run the hdfs command on the client. It is found that the following file does not exist:
hdfs://hacluster/kylin/kylin_metadata/coprocessor/kylin-coprocessor-1.6.0-SNAPSHOT-0.jar
- When configuring the coprocessor for HBase, make sure that the path of the corresponding JAR package is correct. Otherwise, HBase cannot be started.
Solution
Use the Apache Kylin engine to interconnect with MRS and make sure that the JAR file of the Kylin engine exists.
Parent topic: Using HBase
- Symptom
- Cause Analysis
- Solution