6
spark+zookeeper高可用配置
source link: https://jimolonely.github.io/2019/10/07/spark/010-spark-zookeeper-HA/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
spark+zookeeper高可用配置
本文讲解spark基于zookeeper的高可用配置。
上一篇：spark历史服务配置
spark-env配置
spark-env.sh
# 注释掉单击版
#SPARK_MASTER_HOST=localhost
#SPARK_MASTER_PORT=7077
# 修改为zookeeper管理master
SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=node1,node2,node3 -Dspark.deploy.zookeeper.dir=/spark"
启动zookeeper集群
启动spark
$ sbin/start-all.sh
指定某些节点为master
# node1
$ sbin/start-master.sh
# node2
$ sbin/start-master.sh
那么会有一个为standby状态，一旦有其他挂了，会被顶替。
比如spark-shell使用：
$ bin/spark-shell --master spark://node1:7077,node2:7077
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK