ubuntu18.04.2 hadoop3.1.2+zookeeper3.5.5高可用完全分布式集群搭建
集群规划：
hostname
NameNode
DataNode
JournalNode
ResourceManager
Zookeeper
node01
√
√
√
node02
√
√
node03
√
√
√
√
node04
√
√
√
node05
√
√
√
准备工作:
首先克隆5台ubuntu虚拟机
vim /etc/netplan/01-network-manager-all.yaml修改网络配置
我的5台网络配置如下： (ps: 由于这次是家里台式, 所以网关和之前笔记本搭的那次不一样)
1# Let NetworkManager manage all devices on this system 2# node01 3network: 4 version: 2 5 renderer: NetworkManager 6 ethernets: 7 ens33: 8 dhcp4: no 9 dhcp6: no 10 addresses: [192.168.180.130/24] 11 gateway4: 192.168.180.2 12 nameservers: 13 addresses: [114.114.114.114, 8.8.8.8] 14 15# Let NetworkManager manage all devices on this system 16# node02 17network: 18 version: 2 19 renderer: NetworkManager 20 ethernets: 21 ens33: 22 dhcp4: no 23 dhcp6: no 24 addresses: [192.168.180.131/24] 25 gateway4: 192.168.180.2 26 nameservers: 27 addresses: [114.114.114.114, 8.8.8.8] 28 29# Let NetworkManager manage all devices on this system 30# node03 31network: 32 version: 2 33 renderer: NetworkManager 34 ethernets: 35 ens33: 36 dhcp4: no 37 dhcp6: no 38 addresses: [192.168.180.132/24] 39 gateway4: 192.168.180.2 40 nameservers: 41 addresses: [114.114.114.114, 8.8.8.8] 42# Let NetworkManager manage all devices on this system 43network: 44 version: 2 45 renderer: NetworkManager 46 47 ethernets: 48 ens33: 49 dhcp4: no 50 dhcp6: no 51 addresses: [192.168.180.133/24] 52 gateway4: 192.168.180.2 53 nameservers: 54 addresses: [114.114.114.114, 8.8.8.8] 55# Let NetworkManager manage all devices on this system 56network: 57 version: 2 58 renderer: NetworkManager 59 60 ethernets: 61 ens33: 62 dhcp4: no 63 dhcp6: no 64 addresses: [192.168.180.134/24] 65 gateway4: 192.168.180.2 66 nameservers: 67 addresses: [114.114.114.114, 8.8.8.8] 68
修改完毕后netplan apply 应用该网络配置, ping一下百度有连接网络配置就ok了。
修改主机名
vim /etc/hostname 将对应主机名分别修改为node01, niode02, node03, node04, node05
修改hosts文件
vim /etc/hosts 修改每台机器的hosts文件为如下:
1127.0.0.1 localhost 2192.168.180.130 node01 3192.168.180.131 node02 4192.168.180.132 node03 5192.168.180.133 node04 6192.168.180.134 node05 7 8# The following lines are desirable for IPv6 capable hosts 9::1 ip6-localhost ip6-loopback 10fe00::0 ip6-localnet 11ff00::0 ip6-mcastprefix 12ff02::1 ip6-allnodes 13ff02::2 ip6-allrouters 14
JDK配置
如果没有配置jdk请参考https://www.cnblogs.com/ronnieyuan/p/11461377.html
如果之前有下过别的版本的jdk请将jdk的tar包解压到/usr/lib/jvm, 比如我的jvm目录为如下：
1drwxr-xr-x 4 root root 4096 9月 13 08:57 ./ 2drwxr-xr-x 133 root root 4096 9月 13 08:57 ../ 3lrwxrwxrwx 1 root root 25 4月 8 2018 default-java -> java-1.11.0-openjdk-amd64/ 4lrwxrwxrwx 1 root root 21 3月 27 04:57 java-1.11.0-openjdk-amd64 -> java-11-openjdk-amd64/ 5-rw-r--r-- 1 root root 1994 3月 27 04:57 .java-1.11.0-openjdk-amd64.jinfo 6drwxr-xr-x 9 root root 4096 4月 25 20:43 java-11-openjdk-amd64/ 7drwxr-xr-x 7 uucp 143 4096 12月 16 2018 jdk1.8/
然后再修改配置文件并选择jdk(指令在参考中已写)
jdk版本信息:
1root@node01:~# java -version 2java version "1.8.0_202" 3Java(TM) SE Runtime Environment (build 1.8.0_202-b08) 4Java HotSpot(TM) 64-Bit Server VM (build 25.202-b08, mixed mode)
免密登录
在每台机器上执行ssh-keygen -t rsa -P "" (记得enter)
示例
1root@node01:~# ssh-keygen -t rsa -P "" 2Generating public/private rsa key pair. 3Enter file in which to save the key (/root/.ssh/id_rsa): 4Created directory '/root/.ssh'. 5Your identification has been saved in /root/.ssh/id_rsa. 6Your public key has been saved in /root/.ssh/id_rsa.pub. 7The key fingerprint is: 8SHA256:el2s+e9UXxWjfGY1LS6RYD1CcHLmlXY+zJCopqRnuf0 root@node01 9The key's randomart image is: 10+---[RSA 2048]----+ 11| ooB+.+ +o| 12| Bo.@ + *| 13| ..o % =.| 14| . o .. X .| 15| o +S o. .o| 16| . =. . + .o| 17| o.o. + . .| 18| ... . . | 19| .E .oo | 20+----[SHA256]-----+ 21
在~目录下vim .ssh/authorized_keys:
将5台虚拟机公钥都存入该文件中, 每台的authorized_keys都一致
1ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBw4yPomSFt009LQ3gvxv9vnAF4tSXrJvVBMkpoi78mLMspgxYW6q3vLCWFEHT6HOLrLAQ/+UjclXjuVEEUGVOyn+dgvX7fK+XCOuTVdTyJZ3nIGbHUZ5zB+KHcJN3tiGjFQ3vGEuUeVkQ4jkN5RXI33nSx1eUM/sOuXtQ7DdhJjAuBko7RNw/jjTXW8znv8l8n5hb4fu4B+2CLkIkO+1+mTu8hljE2B+pu4o6cIiY/RTb0hNRLSs6w7K7BJFa+3ZkeMtiLtI8MUaIQzo4/nv4FKa8/GSvxLyyBZGoaunAYsUn7qmlNxNjEXY7wojAnVkRMiyCsEXQU6cEsR//Zocz root@node01 2 3ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg2AsPvf9TjjIVUlZutDqxFH579THtl6e7/SxYxHJ/def/T4dY5glzwW3AJ30Gcsw+k9E8PKiZIAiaQ7kU4/EmFK9LFhAuQx+glZS5GS88lXv7qSYOLmZtJPp0l4tgrIgk9u+PtZToCdlWpGLO2Xi3Dfggt//Lsl4Dqhl3dtrpZSjMGY7zkAd4fu696ri4rjv3kDciUdFNlKBFBkGA4RNFKylkPTlxLZfpqNU2pkZtBySHsGbEHMvnMQ1KOXRoW7pVvZ4QveR/eiQVXqq+v53oZ5KUmC5jpp6Abe3PVa7tG6s2ZOSP9ikOuFKrwXWArjp5H4oaYZIF/UenhhIdjxh3 root@node02 4 5ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDwtqNeAwYLWqY5otArMcKd4iMBCpZ5cd+RyECunVnmeefuN1U53fR+h2UcS6/Jr84ZKlDmJ5+r9jgcBPIftbkGi9RE4aHEqo14sC3P4t6DODxNCI+enytx5/kw3gpKmxOdanrtojSWLdL+5v/h4qPt5e8AFfxqJ9HfZ5darXgRLWbkYcBADH51XvisY9Gf+DJKPjcD+3E8gMbHHdeYWt0crOkxbRVgnjmZVuWsYBRFH5x6ueR5SOHUC3WPzfeEdBvIeRddl4y1DvtvZZuVOxs1rQF59KdDSKSKt4s1lScZS1Kc57yXY2s+L6HrFqxfOO0u1pisfiDwDKvZDwKeMd3n root@node03 6 7ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+QfivNznStFt8xCZ1Qav6jKdErir0VbNRN0nqJaXUe+KL8YmYygofKEZRGQHCpYY2/rM7Cla6Pl9HLoatbvi89OYVy7V3hnu7SJwrqbkAGOqxCzW+OGdV9GRvhi3LTwJMAKxSrXB73tKK9ZqJd7WrP7o7ibyYMAbUiJTc0qa4gSXxXTunUuF2hOG7D88/93bxXXqSI9AydWrXBVxzmrP7CipXFOBqVC/mA/8SEdbVxSK0oGwa9KAAm690onoVevOVtTXWcvKSE/57WM94KJMbIKM/ypxKtUqKrgKuMfBsgs31Zu1j3SDkFC3Vm8uGj4yKnpxsaVJOwuMoRYiW90tT root@node04 8 9ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF/mqbRAwPusxpz5FA9FtIa97QJSuXjaRP+/37S7JvtCAh2FvgPBLIQeAdp7hvc/RFJ8WqDlQWj2UVpBsu2sn3Kg2VZ30qEghMLkMcCTtKknNX+U7SvBWCRoGojxl9lmi/Y1kkVNQUTRPQ8QeNGN2SvUi5A4Q+X1H6MEy16sLuamMlXqiIeqttY33odXj6oXI6OFqoE98FrNbTBrPwJFCk4Uhgnplbb0YE+4dbs9mVdR/iHpGm84WfvITe6Rn9Ry4K+Wo4C+Bms4dGfcO8eh8lrwSCff2IUIc877Zzc6ImYrdvZu7rvrCPyfNdoCJzA5wtExPoAfUbuN5T77ieLgWH root@node05 10
依次测试免密登录是否成功
成功案例：
1root@node01:~# ssh root@192.168.180.131 2The authenticity of host '192.168.180.131 (192.168.180.131)' can't be established. 3ECDSA key fingerprint is SHA256:++PMZ5boD2CgToi43EdaCSLtNGdVFt0xxCBoAIkggqk. 4Are you sure you want to continue connecting (yes/no)? yes 5Warning: Permanently added '192.168.180.131' (ECDSA) to the list of known hosts. 6Welcome to Ubuntu 18.04.2 LTS (GNU/Linux 4.18.0-17-generic x86_64) 7 8 * Documentation: https://help.ubuntu.com 9 * Management: https://landscape.canonical.com 10 * Support: https://ubuntu.com/advantage 11 12 13 * Canonical Livepatch is available for installation. 14 - Reduce system reboots and improve kernel security. Activate at: 15 https://ubuntu.com/livepatch 16 17254 packages can be updated. 18253 updates are security updates. 19 20Your Hardware Enablement Stack (HWE) is supported until April 2023. 21Last login: Sat Sep 14 08:44:54 2019 from 192.168.180.1 22root@node02:~#
Zookeeper3.5.5安装
注：zookeeper只安装在node03, node04 和node05上
上传zookeeper3.5.5到/home/ronnie/soft目录下
1root@node03:/home/ronnie/soft# ll 2total 524532 3drwxr-xr-x 2 root root 4096 9月 14 09:51 ./ 4drwxr-xr-x 32 ronnie ronnie 4096 9月 14 08:39 ../ 5-rw-r--r-- 1 root root 10622522 9月 13 11:35 apache-zookeeper-3.5.5-bin.tar.gz 6-rw-r--r-- 1 root root 332433589 9月 13 09:18 hadoop-3.1.2.tar.gz 7-rw-r--r-- 1 root root 194042837 1月 18 2019 jdk-8u202-linux-x64.tar.gz 8
tar -zxvf apache-zookeeper-3.5.5-bin.tar.gz -C /opt/ronnie/ 将其解压至/opt/ronnie目录
修改zookeeper目录名：
1cd /opt/ronnie 2mv apache-zookeeper-3.5.5-bin/ zookeeper
创建并修改zookeeper配置文件
首先进入配置文件目录:
1root@node03:/opt/ronnie# cd zookeeper/conf/ 2root@node03:/opt/ronnie/zookeeper/conf# ll 3total 20 4drwxr-xr-x 2 2002 2002 4096 4月 2 21:05 ./ 5drwxr-xr-x 6 root root 4096 9月 14 09:54 ../ 6-rw-r--r-- 1 2002 2002 535 2月 15 2019 configuration.xsl 7-rw-r--r-- 1 2002 2002 2712 4月 2 21:05 log4j.properties 8-rw-r--r-- 1 2002 2002 922 2月 15 2019 zoo_sample.cfg 9
拷贝一份zoo_sample.cfg 为 zoo.cfg:
1cp zoo_sample.cfg zoo.cfg 2vim zoo.cfg 3 4 5# The number of milliseconds of each tick 6tickTime=2000 7# The number of ticks that the initial 8# synchronization phase can take 9initLimit=10 10# The number of ticks that can pass between 11# sending a request and getting an acknowledgement 12syncLimit=5 13# the directory where the snapshot is stored. 14# do not use /tmp for storage, /tmp here is just 15# example sakes. 16dataDir=/var/ronnie/zookeeper 17# the port at which the clients will connect 18clientPort=2181 19# the maximum number of client connections. 20# increase this if you need to handle more clients 21#maxClientCnxns=60 22# 23# Be sure to read the maintenance section of the 24# administrator guide before turning on autopurge. 25# 26# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance 27# 28# The number of snapshots to retain in dataDir 29#autopurge.snapRetainCount=3 30# Purge task interval in hours 31# Set to "0" to disable auto purge feature 32#autopurge.purgeInterval=1 33server.1=node03:2888:3888 34server.2=node04:2888:3888 35server.3=node05:2888:3888 36
将配置好的zookeeper目录传送给其他2台机子
1scp -r /opt/ronnie/zookeeper/ root@192.168.180.133:/opt/ronnie/ 2scp -r /opt/ronnie/zookeeper/ root@192.168.180.134:/opt/ronnie/
在Datadir下创建myid， 并分别指定1, 2, 3
node03上的操作(其他同理), 若没有该目录需要先创建mkdir -p /var/ronnie/zookeeper/
1cd /var/ronnie/zookeeper/ 2touch myid 3echo 1 > myid
启动Zookeeper
1/opt/ronnie/zookeeper/bin/zkServer.sh start 2# 若启动成功 3ZooKeeper JMX enabled by default 4Using config: /opt/ronnie/zookeeper/bin/../conf/zoo.cfg 5Starting zookeeper ... STARTED 6 7# 检测状态 8/opt/ronnie/zookeeper/bin/zkServer.sh status 9 10# 这是一个从节点 11ZooKeeper JMX enabled by default 12Using config: /opt/ronnie/zookeeper/bin/../conf/zoo.cfg 13Client port found: 2181. Client address: localhost. 14Mode: follower 15 16# 这是一个主节点 17ZooKeeper JMX enabled by default 18Using config: /opt/ronnie/zookeeper/bin/../conf/zoo.cfg 19Client port found: 2181. Client address: localhost. 20Mode: leader 21 22
关闭zookeeper:
/opt/ronnie/zookeeper/bin/zkServer.sh stop
那么zookeeper到这里就安装完毕了
Hadoop配置
vim ~/.bashrc 添加hadoop路径
1#HADOOP VARIABLES 2export HADOOP_HOME=/opt/ronnie/hadoop-3.1.2 3export PATH=$PATH:$HADOOP_HOME/bin 4export PATH=$PATH:$HADOOP_HOME/sbin 5export HADOOP_MAPRED_HOME=$HADOOP_HOME 6export HADOOP_COMMON_HOME=$HADOOP_HOME 7export HADOOP_HDFS_HOME=$HADOOP_HOME 8export YARN_HOME=$HADOOP_HOME 9export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native 10export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
source ~/.bashrc 使新配置生效(记得每台都要改)
hadoop version查看版本， 显示如下则hadoop路径配置成功:
1Hadoop 3.1.2 2Source code repository https://github.com/apache/hadoop.git -r 1019dde65bcf12e05ef48ac71e84550d589e5d9a 3Compiled by sunilg on 2019-01-29T01:39Z 4Compiled with protoc 2.5.0 5From source with checksum 64b8bdd4ca6e77cce75a93eb09ab2a9 6This command was run using /opt/ronnie/hadoop-3.1.2/share/hadoop/common/hadoop-common-3.1.2.jar 7
修改hadoop-env.sh, mapred-env.sh, yarn-env.sh中的JAVA_HOME路径, 没有就在底下添加
vim /opt/ronnie/hadoop-3.1.2/etc/hadoop/hadoop-env.sh
1 53 # variable is REQUIRED on ALL platforms except OS X! 2 54 export JAVA_HOME=/usr/lib/jvm/jdk1.8
vim /opt/ronnie/hadoop-3.1.2/etc/hadoop/mapred-env.sh
1 47 # JDK 2 48 export JAVA_HOME=/usr/lib/jvm/jdk1.8
vim /opt/ronnie/hadoop-3.1.2/etc/hadoop/yarn-env.sh
1171 # JDK 2172 export JAVA_HOME=/usr/lib/jvm/jdk1.8
vim /opt/ronnie/hadoop-3.1.2/etc/hadoop/core-site.xml配置core-site.xml文件
1 1 <?xml version="1.0" encoding="UTF-8"?> 2 4 Licensed under the Apache License, Version 2.0 (the "License"); 3 7 4 8 http://www.apache.org/licenses/LICENSE-2.0 5 9 6 12 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 7 13 See the License for the specific language governing permissions and 8 14 limitations under the License. See accompanying LICENSE file. 9 15 --> 10 16 11 17 <!-- Put site-specific property overrides in this file. --> 12 18 13 19 <configuration> 14 20 <!-- 指定hdfs的nameservice名称 --> 15 21 <property> 16 22 <name>fs.defaultFS</name> 17 23 <value>hdfs://ns</value> 18 24 </property> 19 25 <!-- 指定临时目录 --> 20 26 <property> 21 27 <name>hadoop.tmp.dir</name> 22 28 <value>/var/ronnie/hadoop/tmp</value> 23 29 </property> 24 30 <!-- 指定zookeeper --> 25 31 <property> 26 32 <name>ha.zookeeper.quorum</name> 27 33 <value>node03:2181,node04:2181,node05:2181</value> 28 34 </property> 29 35 <!-- Namenode向JournalNode发起的ipc连接请求重试最大次数 --> 30 36 <property> 31 37 <name>ipc.client.connect.max.retries</name> 32 38 <value>100</value> 33 39 <description>Indicates the number of retries a client will make to establish a server c onnection. 34 40 </description> 35 41 </property> 36 42 <!-- Namenode向JournalNode发起的ipc连接请求的重试间隔时间 --> 37 43 <property> 38 44 <name>ipc.client.connect.retry.interval</name> 39 45 <value>10000</value> 40 46 <description>Indicates the number of milliseconds a client will wait for before retryin g to establish. 41 47 </description> 42 48 </property> 43 49 <!-- 开启回收功能, 并设置垃圾删除间隔(min) --> 44 50 <property> 45 51 <name>fs.trash.interval</name> 46 52 <value>360</value> 47 53 <description> 48 54 Trash deletion interval in minutes. If zero, the trash feature is diabled. 49 55 </description> 50 56 </property> 51 57 <!-- 设置垃圾检查点介个(min), 不设置的话默认和fs.trash.interval一样 --> 52 58 <property> 53 59 <name>fs.trash.checkpoint.interval</name> 54 60 <value>60</value> 55 61 <description> 56 62 Trash checkpoint interval in minutes. If zero, the deletion interval is used. 57 63 </description> 58 64 </property> 59 65 <!-- 配置oozie时使用以下参数 --> 60 66 <property> 61 67 <name>hadoop.proxyuser.deplab.groups</name> 62 68 <value>*</value> 63 69 </property> 64 70 <property> 65 71 <name>hadoop.proxyuser.deplab.hosts</name> 66 72 <value>*</value> 67 73 </property> 68 74 </configuration> 69
vim /opt/ronnie/hadoop-3.1.2/etc/hadoop/hdfs-site.xml修改hdfs-site.xml
1 1 <?xml version="1.0" encoding="UTF-8"?> 2 2 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> 3 3 <!-- 4 4 Licensed under the Apache License, Version 2.0 (the "License"); 5 5 you may not use this file except in compliance with the License. 6 6 You may obtain a copy of the License at 7 7 8 8 http://www.apache.org/licenses/LICENSE-2.0 9 9 10 10 Unless required by applicable law or agreed to in writing, software 11 11 distributed under the License is distributed on an "AS IS" BASIS, 12 12 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 13 See the License for the specific language governing permissions and 14 14 limitations under the License. See accompanying LICENSE file. 15 15 --> 16 16 17 17 <!-- Put site-specific property overrides in this file. --> 18 18 19 19 <configuration> 20 20 <!-- 指定hdfs的nameservice为ns, 需要和core-site.xml中的保持一致 --> 21 21 <property> 22 22 <name>dfs.nameservices</name> 23 23 <value>ns</value> 24 24 </property> 25 25 <!-- ns下有nn1, nn2 两个NameNode --> 26 28 <value>nn1,nn2</value> 27 29 </property> 28 30 <!-- nn1的RPC通信地址 --> 29 33 <value>node01:9000</value> 30 34 </property> 31 35 <!-- nn1 的http通信地址 --> 32 36 <property> 33 37 <name>dfs.namenode.http-address.ns.nn1</name> 34 38 <value>node01:50070</value> 35 39 </property> 36 40 <!-- nn2的RPC通信地址 --> 37 41 <property> 38 42 <name>dfs.namenode.rpc-address.ns.nn2</name> 39 43 <value>node02:9000</value> 40 44 </property> 41 45 <!-- nn2 的http通信地址 --> 42 46 <property> 43 47 <name>dfs.namenode.http-address.ns.nn2</name> 44 48 <value>node02:50070</value> 45 49 </property> 46 50 <!-- 指定NameNode的edits元数据在JournalNode上的存放位置 --> 47 51 <property> 48 52 <name>dfs.namenode.shared.edits.dir</name> 49 53 <value>qjournal://node03:8485;node04:8485;node05:8485/ns</value> 50 54 </property> 51 55 <!-- 指定JournalNode在本地磁盘存放数据的尾椎 --> 52 56 <property> 53 57 <name>dfs.journalnode.edits.dir</name> 54 58 <value>/var/ronnie/hadoop/jdata</value> 55 59 </property> 56 60 <!-- 开启NameNode失败自动切换 --> 57 61 <property> 58 62 <name>dfs.ha.automatic-failover.enabled</name> 59 63 <value>true</value> 60 64 </property> 61 65 <!-- 配置失败自动切换实现方式 --> 62 66 <property> 63 67 <name>dfs.client.failover.proxy.provider.ns</name> 64 68 <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value > 65 69 </property> 66 70 <!-- 配置隔离机制方法, 多个机制用换行分割, 即每个机制暂用一行 --> 67 71 <property> 68 72 <name>dfs.ha.fencing.methods</name> 69 73 <value> 70 74 sshfence 71 75 shell(/bin/true) 72 76 </value> 73 77 </property> 74 78 <!-- 使用sshfence隔离机制时需要ssh免密登录 --> 75 79 <property> 76 80 <name>dfs.ha.fencing.ssh.private-key-files</name> 77 81 <value>/root/.ssh/id_rsa</value> 78 82 </property> 79 83 <!-- 配置sshfence隔离机制超时时间 --> 80 84 <property> 81 85 <name>dfs.ha.fencing.ssh.connect-timeout</name> 82 86 <value>30000</value> 83 87 </property> 84 88 </configuration> 85
vim /opt/ronnie/hadoop-3.1.2/etc/hadoop/mapred-site.xml修改mapred-site.xml
1 1 <?xml version="1.0"?> 2 2 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> 3 3 <!-- 4 4 Licensed under the Apache License, Version 2.0 (the "License"); 5 5 you may not use this file except in compliance with the License. 6 6 You may obtain a copy of the License at 7 7 8 8 http://www.apache.org/licenses/LICENSE-2.0 9 9 10 10 Unless required by applicable law or agreed to in writing, software 11 11 distributed under the License is distributed on an "AS IS" BASIS, 12 12 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 13 See the License for the specific language governing permissions and 14 14 limitations under the License. See accompanying LICENSE file. 15 15 --> 16 16 17 17 <!-- Put site-specific property overrides in this file. --> 18 18 19 19 <configuration> 20 20 <!-- 指定mr框架为yarn方式 --> 21 21 <property> 22 22 <name>mapreduce.framework.name</name> 23 23 <value>yarn</value> 24 24 </property> 25 25 </configuration> 26
vim /opt/ronnie/hadoop-3.1.2/etc/hadoop/yarn-site.xml修改yarn-site.xml
1 1 <?xml version="1.0"?> 2 2 <!-- 3 3 Licensed under the Apache License, Version 2.0 (the "License"); 4 4 you may not use this file except in compliance with the License. 5 5 You may obtain a copy of the License at 6 6 7 7 http://www.apache.org/licenses/LICENSE-2.0 8 8 9 9 Unless required by applicable law or agreed to in writing, software 10 10 distributed under the License is distributed on an "AS IS" BASIS, 11 11 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 12 See the License for the specific language governing permissions and 13 13 limitations under the License. See accompanying LICENSE file. 14 14 --> 15 15 <configuration> 16 16 <!-- 开启RM高可用 --> 17 17 <property> 18 18 <name>yarn.resourcemanager.ha.enabled</name> 19 19 <value>true</value> 20 20 </property> 21 21 <!-- 指定RM的集群id --> 22 22 <property> 23 23 <name>yarn.resourcemanager.cluster-id</name> 24 24 <value>yrc</value> 25 25 </property> 26 26 <!-- 指定RM的名称 --> 27 27 <property> 28 28 <name>yarn.resourcemanager.ha.rm-ids</name> 29 29 <value>rm1,rm2</value> 30 30 </property> 31 31 <!-- 指定rm1, rm2的地址 --> 32 32 <property> 33 33 <name>yarn.resourcemanager.hostname.rm1</name> 34 34 <value>node01</value> 35 35 </property> 36 36 <property> 37 37 <name>yarn.resourcemanager.hostname.rm2</name> 38 38 <value>node02</value> 39 39 </property> 40 40 <!-- 指定zookeeper集群地址 --> 41 41 <property> 42 42 <name>yarn.resourcemanager.zk-address</name> 43 43 <value>node03:2181,node04:2181,node05:2181</value> 44 44 </property> 45 45 <!-- 设定洗牌 --> 46 46 <property> 47 47 <name>yarn.nodemanager.aux-services</name> 48 48 <value>mapreduce_shuffle</value> 49 49 </property> 50 50 </configuration>
vim /opt/ronnie/hadoop-3.1.2/etc/hadoop/workers修改工作组
1node01 2node02 3node03 4node04 5node05 6
vim /opt/ronnie/hadoop-3.1.2/sbin/start-dfs.sh
vim /opt/ronnie/hadoop-3.1.2/sbin/stop-dfs.sh
在文件顶部头文件之后添加:
1#!/usr/bin/env bash 2 3HDFS_DATANODE_USER=root 4 5HADOOP_SECURE_DN_USER=hdfs 6 7HDFS_NAMENODE_USER=root 8 9HDFS_SECONDARYNAMENODE_USER=root 10 11HDFS_JOURNALNODE_USER=root 12 13HDFS_ZKFC_USER=root 14
vim /opt/ronnie/hadoop-3.1.2/sbin/start-yarn.sh
vim /opt/ronnie/hadoop-3.1.2/sbin/stop-yarn.sh
在文件顶部头文件之后添加:
1#!/usr/bin/env bash 2 3YARN_RESOURCEMANAGER_USER=root 4 5HADOOP_SECURE_DN_USER=yarn 6 7YARN_NODEMANAGER_USER=root 8
将配置好的hadoop目录传送给其他节点
1scp -r /opt/ronnie/hadoop-3.1.2/ root@192.168.180.131:/opt/ronnie/ 2scp -r /opt/ronnie/hadoop-3.1.2/ root@192.168.180.132:/opt/ronnie/ 3scp -r /opt/ronnie/hadoop-3.1.2/ root@192.168.180.133:/opt/ronnie/ 4scp -r /opt/ronnie/hadoop-3.1.2/ root@192.168.180.134:/opt/ronnie/
启动集群
先启动zookeeper集群：
1root@node03:/var/ronnie/zookeeper# /opt/ronnie/zookeeper/bin/zkServer.sh start 2root@node04:/var/ronnie/zookeeper# /opt/ronnie/zookeeper/bin/zkServer.sh start 3root@node05:/var/ronnie/zookeeper# /opt/ronnie/zookeeper/bin/zkServer.sh start
分别在node03, node04, node05上启动journalnode
1root@node03:~# /opt/ronnie/hadoop-3.1.2/sbin/hadoop-daemon.sh start journalnode 2root@node04:~# /opt/ronnie/hadoop-3.1.2/sbin/hadoop-daemon.sh start journalnode 3root@node05:~# /opt/ronnie/hadoop-3.1.2/sbin/hadoop-daemon.sh start journalnode
jps查看进程
1root@node03:~# jps 26770 Jps 36724 JournalNode 46616 QuorumPeerMain
在两台NameNode中选一台进行格式化(这里选node01)
root@node01:~# hdfs namenode -format
如有报错会写明哪里配置文件写错错了, 再回去改。
开启NameNode:
hdfs --daemon start namenode
jps查看是否开启
1root@node01:~# jps 25622 Jps 35549 NameNode
在另一台NameNode上同步格式化后的相关信息
root@node02:~# hdfs namenode -bootstrapStandby
在node01上格式化ZKFC(只需要执行一次)
root@node01:~# hdfs zkfc -formatZK
启动hdfs集群
root@node01:~# start-dfs.sh
打开node01的50070端口
启动yarn集群
root@node01:~# start-yarn.sh
打开node01的8088端口
到此配置成功 如果上传文件遇到permission denied，则在hdfs-site.xml文件末尾添加:
1 <property> 2 <name>dfs.permissions</name> 3 <value>false</value> 4 </property>