最近开始研究大数据这块，现在从最基础的Hadoop开始，后续将逐渐学习Hadoop整个生态圈的各个部分组件。
Hadoop安装分为本地安装、伪分布式、完全分布式和高可用分布式，这里为个人学习用（实际情况是本人没有那么多机器，装虚拟机的话，内存可能也不够，T_T），仅涉及到本地安装和伪分布式安装。
环境准备
1# 操作系统信息 2$ cat /etc/centos-release 3CentOS Linux release 7.4.1708 (Core) 4 5# 系统内核信息 6$ uname -r 73.10.0-693.11.6.el7.x86_64 8 9# hostname信息 ( 配置 ) 10$ hostnamectl set-hostname v108.zlikun.com 11$ hostnamectl status 12 Static hostname: v108.zlikun.com 13 Icon name: computer-vm 14 Chassis: vm 15 Machine ID: da1dac0e4969496a8906d711f95f2a7f 16 Boot ID: 8ffc47fb1b7148ab992d8bf6f3f32ac1 17 Virtualization: vmware 18 Operating System: CentOS Linux 7 (Core) 19 CPE OS Name: cpe:/o:centos:centos:7 20 Kernel: Linux 3.10.0-693.11.6.el7.x86_64 21 Architecture: x86-64 22 23# 在`/etc/hosts`文件中配置主机名 ( 不要在意为什么是v108这个细节，只是我电脑上的第8个虚拟机而已，^_^ ) 24192.168.1.108 v108.zlikun.com v108
安装JAVA
1# 解压 `jdk-8u151-linux-x64.tar.gz` 包，并移动到 `/usr/local` 目录下 2/usr/local/jdk1.8.0_151 3 4# 在 `/etc/profile` 中配置环境变量 5export JAVA_HOME=/usr/local/jdk1.8.0_151 6export PATH=$PATH:$JAVA_HOME/bin 7 8# 查看JDK版本 9$ java -version 10java version "1.8.0_151" 11Java(TM) SE Runtime Environment (build 1.8.0_151-b12) 12Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
本地安装Hadoop
1# Hadoop使用2.7.5版，文档地址：http://hadoop.apache.org/docs/r2.7.5/ ，下面是参考安装文档： 2# http://hadoop.apache.org/docs/r2.7.5/hadoop-project-dist/hadoop-common/SingleCluster.html 3 4# 安装过程：本地安装只需要将 `hadoop-2.7.5.tar.gz` 安装包解压到指定目录即可 5$ tar zxvf hadoop-2.7.5.tar.gz 6$ mv hadoop-2.7.5 /opt/hadoop 7# 这里删除全部 *.cmd 格式文件 ( 这些文件仅限Windows下使用，非必须删除，取决于个人习惯 ) 8$ rm -rf /opt/hadoop/*/*.cmd 9# 配置环境变量 HADOOP_HOME 10$ echo 'export HADOOP_HOME=/opt/hadoop' >> /etc/profile 11 12# 下面是Hadoop的目录结构 13/opt/hadoop/ 14├── bin 15│ ├── container-executor 16│ ├── hadoop 17│ ├── hdfs 18│ ├── mapred 19│ ├── rcc 20│ ├── test-container-executor 21│ └── yarn 22├── etc 23│ └── hadoop 24├── include 25│ ├── hdfs.h 26│ ├── Pipes.hh 27│ ├── SerialUtils.hh 28│ ├── StringUtils.hh 29│ └── TemplateFactory.hh 30├── lib 31│ └── native 32├── libexec 33│ ├── hadoop-config.sh 34│ ├── hdfs-config.sh 35│ ├── httpfs-config.sh 36│ ├── kms-config.sh 37│ ├── mapred-config.sh 38│ └── yarn-config.sh 39├── LICENSE.txt 40├── NOTICE.txt 41├── README.txt 42├── sbin 43│ ├── distribute-exclude.sh 44│ ├── hadoop-daemon.sh 45│ ├── hadoop-daemons.sh 46│ ├── hdfs-config.sh 47│ ├── httpfs.sh 48│ ├── kms.sh 49│ ├── mr-jobhistory-daemon.sh 50│ ├── refresh-namenodes.sh 51│ ├── slaves.sh 52│ ├── start-all.sh 53│ ├── start-balancer.sh 54│ ├── start-dfs.sh 55│ ├── start-secure-dns.sh 56│ ├── start-yarn.sh 57│ ├── stop-all.sh 58│ ├── stop-balancer.sh 59│ ├── stop-dfs.sh 60│ ├── stop-secure-dns.sh 61│ ├── stop-yarn.sh 62│ ├── yarn-daemon.sh 63│ └── yarn-daemons.sh 64└── share 65 ├── doc 66 └── hadoop 67 68# 执行 `bin/hadoop` 命令，可以查看帮助文档 69$ cd /opt/hadoop 70$ bin/hadoop 71Usage: hadoop [--config confdir] [COMMAND | CLASSNAME] 72 CLASSNAME run the class named CLASSNAME 73 or 74 where COMMAND is one of: 75 fs run a generic filesystem user client 76 version print the version 77 jar <jar> run a jar file 78 note: please use "yarn jar" to launch 79 YARN applications, not this command. 80 checknative [-a|-h] check native hadoop and compression libraries availability 81 distcp <srcurl> <desturl> copy file or directories recursively 82 archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive 83 classpath prints the class path needed to get the 84 credential interact with credential providers 85 Hadoop jar and the required libraries 86 daemonlog get/set the log level for each daemon 87 trace view and modify Hadoop tracing settings 88 89Most commands print help when invoked w/o parameters. 90 91# 通常安装Hadoop后第一件事就是配置它的JAVA_HOME参数 92$ vi $HADOOP_HOME/etc/hadoop/hadoop-env.sh 93export JAVA_HOME=/usr/local/jdk1.8.0_151
词频统计示例
1# 准备一个本地文件，后面将对其进行词频统计 2$ mkdir input 3$ echo 'java golang ruby rust erlang java javascript lua rust java' > input/lang.txt 4 5# 运行官方自带 `MapReduce` 程序，进行词频统计 6$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.5.jar wordcount input output 718/01/30 08:42:34 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 818/01/30 08:42:34 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 918/01/30 08:42:34 INFO input.FileInputFormat: Total input paths to process : 1 1018/01/30 08:42:34 INFO mapreduce.JobSubmitter: number of splits:1 1118/01/30 08:42:34 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local935371141_0001 1218/01/30 08:42:34 INFO mapreduce.Job: The url to track the job: http://localhost:8080/ 1318/01/30 08:42:34 INFO mapreduce.Job: Running job: job_local935371141_0001 1418/01/30 08:42:34 INFO mapred.LocalJobRunner: OutputCommitter set in config null 1518/01/30 08:42:34 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1 1618/01/30 08:42:34 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 1718/01/30 08:42:34 INFO mapred.LocalJobRunner: Waiting for map tasks 1818/01/30 08:42:34 INFO mapred.LocalJobRunner: Starting task: attempt_local935371141_0001_m_000000_0 1918/01/30 08:42:34 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1 2018/01/30 08:42:34 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2118/01/30 08:42:34 INFO mapred.MapTask: Processing split: file:/opt/hadoop/input/lang.txt:0+59 2218/01/30 08:42:34 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2318/01/30 08:42:34 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100 2418/01/30 08:42:34 INFO mapred.MapTask: soft limit at 83886080 2518/01/30 08:42:34 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600 2618/01/30 08:42:34 INFO mapred.MapTask: kvstart = 26214396; length = 6553600 2718/01/30 08:42:34 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2818/01/30 08:42:34 INFO mapred.LocalJobRunner: 2918/01/30 08:42:34 INFO mapred.MapTask: Starting flush of map output 3018/01/30 08:42:34 INFO mapred.MapTask: Spilling map output 3118/01/30 08:42:34 INFO mapred.MapTask: bufstart = 0; bufend = 99; bufvoid = 104857600 3218/01/30 08:42:34 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214360(104857440); length = 37/6553600 3318/01/30 08:42:34 INFO mapred.MapTask: Finished spill 0 3418/01/30 08:42:34 INFO mapred.Task: Task:attempt_local935371141_0001_m_000000_0 is done. And is in the process of committing 3518/01/30 08:42:35 INFO mapred.LocalJobRunner: map 3618/01/30 08:42:35 INFO mapred.Task: Task 'attempt_local935371141_0001_m_000000_0' done. 3718/01/30 08:42:35 INFO mapred.Task: Final Counters for attempt_local935371141_0001_m_000000_0: Counters: 18 38 File System Counters 39 FILE: Number of bytes read=296042 40 FILE: Number of bytes written=585271 41 FILE: Number of read operations=0 42 FILE: Number of large read operations=0 43 FILE: Number of write operations=0 44 Map-Reduce Framework 45 Map input records=1 46 Map output records=10 47 Map output bytes=99 48 Map output materialized bytes=92 49 Input split bytes=96 50 Combine input records=10 51 Combine output records=7 52 Spilled Records=7 53 Failed Shuffles=0 54 Merged Map outputs=0 55 GC time elapsed (ms)=16 56 Total committed heap usage (bytes)=165744640 57 File Input Format Counters 58 Bytes Read=59 5918/01/30 08:42:35 INFO mapred.LocalJobRunner: Finishing task: attempt_local935371141_0001_m_000000_0 6018/01/30 08:42:35 INFO mapred.LocalJobRunner: map task executor complete. 6118/01/30 08:42:35 INFO mapred.LocalJobRunner: Waiting for reduce tasks 6218/01/30 08:42:35 INFO mapred.LocalJobRunner: Starting task: attempt_local935371141_0001_r_000000_0 6318/01/30 08:42:35 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1 6418/01/30 08:42:35 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ] 6518/01/30 08:42:35 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@475ebd65 6618/01/30 08:42:35 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=363285696, maxSingleShuffleLimit=90821424, mergeThreshold=239768576, ioSortFactor=10, memToMemMergeOutputsThreshold=10 6718/01/30 08:42:35 INFO reduce.EventFetcher: attempt_local935371141_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events 6818/01/30 08:42:35 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local935371141_0001_m_000000_0 decomp: 88 len: 92 to MEMORY 6918/01/30 08:42:35 INFO reduce.InMemoryMapOutput: Read 88 bytes from map-output for attempt_local935371141_0001_m_000000_0 7018/01/30 08:42:35 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 88, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->88 7118/01/30 08:42:35 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning 7218/01/30 08:42:35 INFO mapred.LocalJobRunner: 1 / 1 copied. 7318/01/30 08:42:35 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs 7418/01/30 08:42:35 WARN io.ReadaheadPool: Failed readahead on ifile 75EBADF: Bad file descriptor 76 at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method) 77 at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267) 78 at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146) 79 at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206) 80 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 81 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 82 at java.lang.Thread.run(Thread.java:748) 8318/01/30 08:42:35 INFO mapred.Merger: Merging 1 sorted segments 8418/01/30 08:42:35 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 79 bytes 8518/01/30 08:42:35 INFO reduce.MergeManagerImpl: Merged 1 segments, 88 bytes to disk to satisfy reduce memory limit 8618/01/30 08:42:35 INFO reduce.MergeManagerImpl: Merging 1 files, 92 bytes from disk 8718/01/30 08:42:35 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce 8818/01/30 08:42:35 INFO mapred.Merger: Merging 1 sorted segments 8918/01/30 08:42:35 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 79 bytes 9018/01/30 08:42:35 INFO mapred.LocalJobRunner: 1 / 1 copied. 9118/01/30 08:42:35 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords 9218/01/30 08:42:35 INFO mapred.Task: Task:attempt_local935371141_0001_r_000000_0 is done. And is in the process of committing 9318/01/30 08:42:35 INFO mapred.LocalJobRunner: 1 / 1 copied. 9418/01/30 08:42:35 INFO mapred.Task: Task attempt_local935371141_0001_r_000000_0 is allowed to commit now 9518/01/30 08:42:35 INFO output.FileOutputCommitter: Saved output of task 'attempt_local935371141_0001_r_000000_0' to file:/opt/hadoop/output/_temporary/0/task_local935371141_0001_r_000000 9618/01/30 08:42:35 INFO mapred.LocalJobRunner: reduce > reduce 9718/01/30 08:42:35 INFO mapred.Task: Task 'attempt_local935371141_0001_r_000000_0' done. 9818/01/30 08:42:35 INFO mapred.Task: Final Counters for attempt_local935371141_0001_r_000000_0: Counters: 24 99 File System Counters 100 FILE: Number of bytes read=296258 101 FILE: Number of bytes written=585433 102 FILE: Number of read operations=0 103 FILE: Number of large read operations=0 104 FILE: Number of write operations=0 105 Map-Reduce Framework 106 Combine input records=0 107 Combine output records=0 108 Reduce input groups=7 109 Reduce shuffle bytes=92 110 Reduce input records=7 111 Reduce output records=7 112 Spilled Records=7 113 Shuffled Maps =1 114 Failed Shuffles=0 115 Merged Map outputs=1 116 GC time elapsed (ms)=2 117 Total committed heap usage (bytes)=165744640 118 Shuffle Errors 119 BAD_ID=0 120 CONNECTION=0 121 IO_ERROR=0 122 WRONG_LENGTH=0 123 WRONG_MAP=0 124 WRONG_REDUCE=0 125 File Output Format Counters 126 Bytes Written=70 12718/01/30 08:42:35 INFO mapred.LocalJobRunner: Finishing task: attempt_local935371141_0001_r_000000_0 12818/01/30 08:42:35 INFO mapred.LocalJobRunner: reduce task executor complete. 12918/01/30 08:42:35 INFO mapreduce.Job: Job job_local935371141_0001 running in uber mode : false 13018/01/30 08:42:35 INFO mapreduce.Job: map 100% reduce 100% 13118/01/30 08:42:35 INFO mapreduce.Job: Job job_local935371141_0001 completed successfully 13218/01/30 08:42:35 INFO mapreduce.Job: Counters: 30 133 File System Counters 134 FILE: Number of bytes read=592300 135 FILE: Number of bytes written=1170704 136 FILE: Number of read operations=0 137 FILE: Number of large read operations=0 138 FILE: Number of write operations=0 139 Map-Reduce Framework 140 Map input records=1 141 Map output records=10 142 Map output bytes=99 143 Map output materialized bytes=92 144 Input split bytes=96 145 Combine input records=10 146 Combine output records=7 147 Reduce input groups=7 148 Reduce shuffle bytes=92 149 Reduce input records=7 150 Reduce output records=7 151 Spilled Records=14 152 Shuffled Maps =1 153 Failed Shuffles=0 154 Merged Map outputs=1 155 GC time elapsed (ms)=18 156 Total committed heap usage (bytes)=331489280 157 Shuffle Errors 158 BAD_ID=0 159 CONNECTION=0 160 IO_ERROR=0 161 WRONG_LENGTH=0 162 WRONG_MAP=0 163 WRONG_REDUCE=0 164 File Input Format Counters 165 Bytes Read=59 166 File Output Format Counters 167 Bytes Written=70 168 169# 查看统计结果 170$ cat output/* 171erlang 1 172golang 1 173java 3 174javascript 1 175lua 1 176ruby 1 177rust 2
至此，本地Hadoop就安装完成了，可以进行一些简单的测试，这里可以参考官方演示示例：http://hadoop.apache.org/docs/r2.7.5/hadoop-project-dist/hadoop-common/SingleCluster.html#Standalone_Operation