55

大数据之一：Hadoop安装-wx5d66921cb1eb1的博客

 4 years ago
source link: https://blog.51cto.com/14522074/2434368
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.

一、安装Hadoop

1.先解压hadoop到相关的文件夹中

1566901830564

2.进入到解压好的文件夹以后，对相关文件进行修改

1566902099154
1566893588624

3.配置hadoop-env.sh

vi hadoop-env.sh:
export= JAVA_HOME=/opt/inst/jdk181
1566893705161

4.配置core-site.xml

vi core-site.xml
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://bigdata:9000</value>
        </property>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/opt/hadoopdata</value>
        </property>
        <property>
                <name>hadoop.proxyuser.root.users</name>
                <value>*</value>
        </property>
        <property>
                <name>hadoop.proxyuser.root.groups</name>
                <value>*</value>
        </property>
1566893927457
1566894986655

5.配置hdfs-site.xml

vi hdfs-site.xml
        <property>
                <name>dfs.replication</name>
                <value>1</value>
        </property>
1566895205777

6.配置cp mapred-site.xml.template mapred-site.xml

cp mapred-site.xml.template mapred-site.xml

1566895334652

7.vi mapred-site.xml

        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
1566896513659

8.vi yarn-site.xml

vi
1566896452284

9.vi /etc/profile

export HADOOP_HOME=/opt/bigdata/hadoop260
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_INSTALL=$HADOOP_HOME
1566896579450

10.更新并格式化

source  /etc/profile
hdfs namenode -format

1566897164905

11.启动

start-all.sh

1566897200256

jps
#查看进程
1566897333891
1566897884766

12.把linux导入到hadoop中

hdfs dfs -put /opt/a.txt  /cm/
hdfs dfs -ls /cm
1566898227114

Hadoop理解

1.HDFS: Hadoop Distribute File System 分布式文件系统

​ NameNode:主节点，目录

​ DataNode：从节点，数据

​ SecondaryNameNode：主节点的备份

2.YARN：Yet Another Resource Negotiator 资源管理调度系统

​ 调度的是内存的资源和CPU的算力

​ 通过ResourceManager（只有一个） 来调度

​ ResourceManager主要作用：

​ 1.处理客户端请求

​ 2.监控NodeManager

​ 3.启动或监控ApplicationMaster（）

​ 4.资源的分配或调度

​ NodeManager（多个）

​ NodeManager主要作用：

​ 1.管理单个节点上的资源

​ 2.处理来自ResourceManager的命令

​ 3.处理来自ApplicationMaster的命令

3.MapReduce：分布式运算框架

1566913676798

report

Recommend

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK