一、Hadoop集群配置好后，执行start-dfs.sh后报错，一堆permission denied
1zf sbin $ ./start-dfs.sh 2Starting namenodes on [master] 3master: chown: changing ownership of '/home/zf/hadoop/hadoop-2.9.1/logs': Operation not permitted 4master: starting namenode, logging to /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-namenode-master.out 5master: /home/zf/hadoop/hadoop-2.9.1/sbin/hadoop-daemon.sh: line 159: /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-namenode-master.out: Permission denied 6master: head: cannot open '/home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-namenode-master.out' for reading: No such file or directory 7master: /home/zf/hadoop/hadoop-2.9.1/sbin/hadoop-daemon.sh: line 177: /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-namenode-master.out: Permission denied 8master: /home/zf/hadoop/hadoop-2.9.1/sbin/hadoop-daemon.sh: line 178: /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-namenode-master.out: Permission denied 9slave-1: chown: changing ownership of '/home/zf/hadoop/hadoop-2.9.1/logs': Operation not permitted 10slave-1: starting datanode, logging to /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-datanode-slave-1.out 11slave-1: /home/zf/hadoop/hadoop-2.9.1/sbin/hadoop-daemon.sh: line 159: /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-datanode-slave-1.out: Permission denied 12slave-2: chown: changing ownership of '/home/zf/hadoop/hadoop-2.9.1/logs': Operation not permitted 13slave-2: starting datanode, logging to /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-datanode-slave-2.out 14slave-2: /home/zf/hadoop/hadoop-2.9.1/sbin/hadoop-daemon.sh: line 159: /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-datanode-slave-2.out: Permission denied 15slave-1: head: cannot open '/home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-datanode-slave-1.out' for reading: No such file or directory 16slave-1: /home/zf/hadoop/hadoop-2.9.1/sbin/hadoop-daemon.sh: line 177: /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-datanode-slave-1.out: Permission denied 17slave-1: /home/zf/hadoop/hadoop-2.9.1/sbin/hadoop-daemon.sh: line 178: /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-datanode-slave-1.out: Permission denied 18slave-2: head: cannot open '/home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-datanode-slave-2.out' for reading: No such file or directory 19slave-2: /home/zf/hadoop/hadoop-2.9.1/sbin/hadoop-daemon.sh: line 177: /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-datanode-slave-2.out: Permission denied 20slave-2: /home/zf/hadoop/hadoop-2.9.1/sbin/hadoop-daemon.sh: line 178: /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-datanode-slave-2.out: Permission denied 21Starting secondary namenodes [slave-1] 22slave-1: chown: changing ownership of '/home/zf/hadoop/hadoop-2.9.1/logs': Operation not permitted 23slave-1: starting secondarynamenode, logging to /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-secondarynamenode-slave-1.out 24slave-1: /home/zf/hadoop/hadoop-2.9.1/sbin/hadoop-daemon.sh: line 159: /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-secondarynamenode-slave-1.out: Permission denied 25slave-1: head: cannot open '/home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-secondarynamenode-slave-1.out' for reading: No such file or directory 26slave-1: /home/zf/hadoop/hadoop-2.9.1/sbin/hadoop-daemon.sh: line 177: /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-secondarynamenode-slave-1.out: Permission denied 27slave-1: /home/zf/hadoop/hadoop-2.9.1/sbin/hadoop-daemon.sh: line 178: /home/zf/hadoop/hadoop-2.9.1/logs/hadoop-zf-secondarynamenode-slave-1.out: Permission denied
解决方案：对hadoop安装目录执行命令：sudo chmod a+w *
对文件敞开权限
二、执行./start-dfs.sh 和 ./start-yarn.sh 后master主机上jps 无法启动NameNode和ResourceManager，但slave-1 和slave-2 DataNode启动正常。
查看logs，错误如下：
1**************namenode.log**************************** 2 32018-08-01 04:01:28,091 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode. 4java.net.BindException: Port in use: master:50070 5 at org.apache.hadoop.http.HttpServer2.constructBindException(HttpServer2.java:1014) 6 at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1037) 7 at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:1094) 8 at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:951) 9 at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:179) 10 at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:889) 11 at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:725) 12 at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:953) 13 at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:932) 14 at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1673) 15 at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1741) 16Caused by: java.net.BindException: Cannot assign requested address 17 at sun.nio.ch.Net.bind0(Native Method) 18 at sun.nio.ch.Net.bind(Net.java:433) 19 at sun.nio.ch.Net.bind(Net.java:425) 20 at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) 21 at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) 22 at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216) 23 at org.apache.hadoop.http.HttpServer2.bindListener(HttpServer2.java:1002) 24 at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1033) 25 ... 9 more 262018-08-01 04:01:28,097 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1: java.net.BindException: Port in use: master:50070 272018-08-01 04:01:28,099 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: 28/************************************************************ 29SHUTDOWN_MSG: Shutting down NameNode at master/182.61.39.233 30************************************************************/ 31 32*************secondnamemode.log**************************** 332018-08-01 02:29:33,947 INFO org.apache.hadoop.http.HttpServer2: HttpServer.start() threw a non Bind IOException 34java.net.BindException: Port in use: master:9001 35 at org.apache.hadoop.http.HttpServer2.constructBindException(HttpServer2.java:1014) 36 at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1037) 37 at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:1094) 38 at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:951) 39 at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.startInfoServer(SecondaryNameNode.java:498) 40 at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:701) 41Caused by: java.net.BindException: Cannot assign requested address 42 at sun.nio.ch.Net.bind0(Native Method) 43 at sun.nio.ch.Net.bind(Net.java:433) 44 at sun.nio.ch.Net.bind(Net.java:425) 45 at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) 46 at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) 47 at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216) 48 at org.apache.hadoop.http.HttpServer2.bindListener(HttpServer2.java:1002) 49 at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1033) 50 ... 4 more 512018-08-01 02:29:33,948 FATAL org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Failed to start secondary namenode 52java.net.BindException: Port in use: master:9001 53 at org.apache.hadoop.http.HttpServer2.constructBindException(HttpServer2.java:1014) 54 at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1037) 55 at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:1094) 56 at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:951) 57 at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.startInfoServer(SecondaryNameNode.java:498) 58 at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:701) 59Caused by: java.net.BindException: Cannot assign requested address 60 at sun.nio.ch.Net.bind0(Native Method) 61 at sun.nio.ch.Net.bind(Net.java:433) 62 at sun.nio.ch.Net.bind(Net.java:425) 63 at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) 64 at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) 65 at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216) 66 at org.apache.hadoop.http.HttpServer2.bindListener(HttpServer2.java:1002) 67 at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1033) 68 ... 4 more
可见主要原因就是：
1、 java.net.BindException: Port in use: master:50070/9001
2、Caused by: java.net.BindException: Cannot assign requested address
端口被占用是直接原因，但起因是不能分配所需的地址，跟地址有关的就联想到 /etc/hosts文件
原来我的是：
1127.0.0.1 localhost localhost.localdomain 2182.xx.xx.33 master (外网IP， 主机名） 3106.xx.xx.72 slave-2 4106.xx.xx.73 slave-1
后来把外网IP全部换成内网IP后就可以了
1127.0.0.1 localhost localhost.localdomain 2172.xx.x.2 master 3172.xx.x.5 slave-1 4172.xx.x.6 slave-2
原因我也不明，我三台主机都是使用百度云服务器。
三、Hadoop 配置好后，发现DataNode没有启动，很大原因是因为NameNode 格式化时与上次的残留有冲突，所以要先删除上次的残留。或者每次使用完后及时Hadoop关闭，重新开启后不要重新格式化NameNode。