3
动态添加/删除 Hadoop DataNode
source link: https://fann.im/blog/2019/07/31/add-and-remove-hadoop-datanode/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
动态添加/删除 Hadoop DataNode
Jul 31, 2019
- NameNode 添加节点
etc/hadoop/slaves - 同步
etc/hadoop配置 - 在新节点
./sbin/hadoop-daemon.sh start datanode - 在 NameNode 刷新
hdfs dfsadmin -refreshNodes
- 在
etc/hadoop/excludes写入要删掉的节点地址 - 修改
etc/hadoop/hdfs-site.xml:
<property>
<name>dfs.hosts.exclude</name>
<value>/home/web/hadoop/etc/hadoop/excludes</value>
</property>
- 修改
etc/hadoop/mapred-site.xml, 这个是下线 nodemanager
<property>
<name>mapred.hosts.exclude</name>
<value>/home/web/hadoop/etc/hadoop/excludes</value>
<final>true</final>
</property>
- 修改
etc/hadoop/slaves，去掉要删除的节点 - 同步
etc/hadoop/excludes和etc/hadoop/slaves到所有 NameNode - 在 NameNode 执行
hdfs dfsadmin -refreshNodes hdfs dfsadmin -report查看要删除的节点状态变化Normal -> Decommission in progress -> Decommissioned- 在要删除的节点
./sbin/hadoop-daemon.sh stop datanode，等待 Admin State 变更为 Dead
hdfs fsck / 检查文件系统信息，正常是 Status: HEALTHY，如果 Status: CORRUPT 说明 blocks 有损坏，其中 Missing blocks 表示有丢失，但有备份，Missing blocks (with replication factor 1) 表示 block 损坏丢失也没有备份，不可恢复。
可以用 hdfs fsck / -delete 来检查并删除有损坏的 blocks.
调整 JournalNode
- 修改
etc/hadoop/hdfs-site.xml:
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://hadoop01:8485;hadoop02:8485;hadoop03:8485/cluster</value>
</property>
- 同步到所有节点
- 如果是新增节点，要同步
dfs.journalnode.edits.dir下 edits 文件 - 在调整的 journalnode 节点启动/关停:
./sbin/hadoop-daemon.sh start journalnode - 重启 standby NameNode:
sbin/hadoop-daemon.sh stop|start namenode - 切换节点为 active:
hdfs haadmin -failover nn1 nn2，重启其他 namenode - 检查 NN 状态
hdfs haadmin -getServiceState nn1
Was this page helpful?
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK