0%

BigData,small problem

大数据开发学习中遇到的问题、原因及解决方法。


Cloudera Manager

Zookeeper

问题:Zookeeper 启动失败

现象

  1. 两台服务器上的 Zookeeper 启动成功,一台服务器上的 Zookeeper 启动失败。
  2. 查看完整日志,显示 “Unable to load database on disk”:

Zookeeper_load_database

1
2
3
4
5
6
7
8
9
下午450:27.458分	ERROR	QuorumPeer	
Unable to load database on disk
java.io.IOException: Found in /var/lib/zookeeper/version-2/acceptedEpoch
at org.apache.zookeeper.server.quorum.QuorumPeer.readLongFromFile(QuorumPeer.java:1205)
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:563)
at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:521)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:169)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:118)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)

解决

  1. 备份 Zookeeper 的 version-2 目录:
1
mv /var/lib/zookeeper/version-2 /data/version-2
  1. 创建新的 version-2 目录:
1
mkdir /var/lib/zookeeper/version-2
  1. 变更 Owner:
1
chown zookeeper:zookeeper /var/lib/zookeeper/*

HBase

问题:HBase 无法使用

现象

  1. HBase shell 提示 HBase 服务未运行。
1
2
3
4
hbase(main):001:0> list
TABLE

ERROR: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
  1. HBase 监控页面 中 Region Servers 列表显示为空。

  2. HDFS 监控页面 显示安全模式开启,并且提示资源不足。

1
2
3
Safe mode is ON. Resources are low on NN. 
Please add or free up more resources then turn off safe mode manually.
NOTE: If you turn off safe mode before adding resources, the NN will immediately return to safe mode. Use "hdfs dfsadmin -safemode leave" to turn safe mode off.

原因
虚拟机磁盘空间不足,HDFS 一直处于安全模式无法退出,而 HBase 强依赖于 HDFS,因此无法使用

解决
删除文件留出磁盘空间后输入 hdfs dfsadmin -safemode leave 离开安全模式,HBase 启动成功

查看本机文件系统

1
2
3
4
5
6
# df -lh
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_master-lv_root
18G 17G 27M 100% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
/dev/sda1 477M 40M 412M 9% /boot

查找当前目录下大于100M的文件

1
2
3
# find . -type f -size +100M -print0 | xargs -0 du -h
257M ./servers/hadoop-2.6.0-cdh5.14.0/logs/hadoop-root-namenode-master.log.1
257M ./servers/hadoop-2.6.0-cdh5.14.0/logs/hadoop-root-namenode-master.log.2

Hive

问题:Hive 无法使用

现象
Hive Shell 提示 SessionHiveMetaStoreClient 未实例化:

1
2
3
4
hive (default)> show databases;
FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException:
Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

Hive_FAILED

原因
Hive 的 Metastore Server 服务进程未正常启动。

解决
启动 Hive 的 Metastore Server 服务进程:

/export/servers/hive-1.1.0-cdh5.14.0
1
nohup bin/hive --service metastore &

问题:Beeline JDBC 连接被拒绝

现象

  1. 使用 10000 端口连接被拒绝:
1
2
3
4
5
6
beeline> !connect jdbc:hive2://node03:10000
Connecting to jdbc:hive2://node03:10000
Enter username for jdbc:hive2://node03:10000: root
Enter password for jdbc:hive2://node03:10000: ******
Could not open connection to the HS2 server. Please check the server URI and if the URI is correct, then ask the administrator to check the server status.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://node03:10000: java.net.ConnectException: 拒绝连接 (Connection refused) (state=08S01,code=0)
  1. 但使用默认地址连接成功:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
beeline> !connect jdbc:hive2://
scan complete in 2ms
Connecting to jdbc:hive2://
Enter username for jdbc:hive2://: root
Enter password for jdbc:hive2://: ******
Connected to: Apache Hive (version 1.1.0-cdh5.14.0)
Driver: Hive JDBC (version 1.1.0-cdh5.14.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://> show databases;
OK
+----------------+--+
| database_name |
+----------------+--+
| default |
| myhive |
+----------------+--+
2 rows selected (1.432 seconds)

原因
Hive 的 HiveServer2 服务进程未正常启动。

解决
启动 Hive 的 HiveServer2 服务进程:

/export/servers/hive-1.1.0-cdh5.14.0
1
nohup bin/hive --service hiveserver2 &