Flume实战案例运维篇
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
一.Flume概述
1>.什么是Flume
1 Flume是一个分布式、可靠、高可用的海量日志聚合系统，支持在系统中定制各类数据发送方，用于收集数据；同时，Flume提供对数据进行简单处理，并写到各种数据接收方。 2 官方地址：http://flume.apache.org/。
2>.Flume特性
1（1）高可靠性 2 Flume提供了end to end的数据可靠性机制 3（2）易于扩展 4 Agent为分布式架构，可水平扩展 5（3）易于恢复 6 Channel中保存了与数据源有关的事件，用于失败时的恢复 7（4）功能丰富 8 Flume内置了多种组件，包括不同数据源和不同存储方式
3>.Flume常用组件
1（1）Source： 2 数据源，简单的说就是agent获取数据的入口。 3 4（2）Channel： 5 管道，数据流通和存储的通道。一个source必须至少和一个channel关联。 6 7（3）Sink： 8 用来接收channel传输的数据并将之传送到指定的地方，成功后从channel中删除。
4>.Flume架构
二.部署Flume环境
1>.下载flume组件
1[root@node101.yinzhengjie.org.cn ~]# yum -y install wget 2Loaded plugins: fastestmirror 3Determining fastest mirrors 4 * base: mirrors.tuna.tsinghua.edu.cn 5 * extras: mirrors.aliyun.com 6 * updates: mirror.bit.edu.cn 7base | 3.6 kB 00:00:00 8extras | 3.4 kB 00:00:00 9updates | 3.4 kB 00:00:00 10updates/7/x86_64/primary_db | 6.5 MB 00:00:02 11Resolving Dependencies 12--> Running transaction check 13---> Package wget.x86_64 0:1.14-18.el7_6.1 will be installed 14--> Finished Dependency Resolution 15 16Dependencies Resolved 17 18============================================================================================================================================================================================================================================================================== 19 Package Arch Version Repository Size 20============================================================================================================================================================================================================================================================================== 21Installing: 22 wget x86_64 1.14-18.el7_6.1 updates 547 k 23 24Transaction Summary 25============================================================================================================================================================================================================================================================================== 26Install 1 Package 27 28Total download size: 547 k 29Installed size: 2.0 M 30Downloading packages: 31wget-1.14-18.el7_6.1.x86_64.rpm | 547 kB 00:00:00 32Running transaction check 33Running transaction test 34Transaction test succeeded 35Running transaction 36 Installing : wget-1.14-18.el7_6.1.x86_64 1/1 37 Verifying : wget-1.14-18.el7_6.1.x86_64 1/1 38 39Installed: 40 wget.x86_64 0:1.14-18.el7_6.1 41 42Complete! 43[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# yum -y install wget
1[root@node101.yinzhengjie.org.cn ~]# wget http://mirrors.tuna.tsinghua.edu.cn/apache/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz 2--2019-07-19 14:29:35-- http://mirrors.tuna.tsinghua.edu.cn/apache/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz 3Resolving mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)... 101.6.8.193, 2402:f000:1:408:8100::1 4Connecting to mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)|101.6.8.193|:80... connected. 5HTTP request sent, awaiting response... 302 Found 6Location: http://103.238.48.8/mirrors.tuna.tsinghua.edu.cn/apache/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz [following] 7--2019-07-19 14:29:35-- http://103.238.48.8/mirrors.tuna.tsinghua.edu.cn/apache/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz 8Connecting to 103.238.48.8:80... connected. 9HTTP request sent, awaiting response... 200 OK 10Length: 67938106 (65M) [application/x-gzip] 11Saving to: ‘apache-flume-1.9.0-bin.tar.gz’ 12 13100%[====================================================================================================================================================================================================================================>] 67,938,106 2.87MB/s in 22s 14 152019-07-19 14:29:57 (2.95 MB/s) - ‘apache-flume-1.9.0-bin.tar.gz’ saved [67938106/67938106] 16 17[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# wget http://mirrors.tuna.tsinghua.edu.cn/apache/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz
2>.解压flume
1[root@node105.yinzhengjie.org.cn ~]# ll 2total 66348 3-rw-r--r-- 1 root root 67938106 Jan 2 2019 apache-flume-1.9.0-bin.tar.gz 4[root@node105.yinzhengjie.org.cn ~]# 5[root@node105.yinzhengjie.org.cn ~]# 6[root@node105.yinzhengjie.org.cn ~]# tar -zxf apache-flume-1.9.0-bin.tar.gz -C /home/softwares/ 7[root@node105.yinzhengjie.org.cn ~]# 8[root@node105.yinzhengjie.org.cn ~]# ll /home/softwares/apache-flume-1.9.0-bin/ 9total 168 10drwxr-xr-x 2 mysql mysql 62 Jul 19 14:31 bin 11-rw-rw-r-- 1 mysql mysql 85602 Nov 29 2018 CHANGELOG 12drwxr-xr-x 2 mysql mysql 127 Jul 19 14:31 conf 13-rw-r--r-- 1 mysql mysql 5681 Nov 16 2017 DEVNOTES 14-rw-r--r-- 1 mysql mysql 2873 Nov 16 2017 doap_Flume.rdf 15drwxrwxr-x 12 mysql mysql 4096 Dec 18 2018 docs 16drwxr-xr-x 2 root root 8192 Jul 19 14:31 lib 17-rw-rw-r-- 1 mysql mysql 43405 Dec 10 2018 LICENSE 18-rw-r--r-- 1 mysql mysql 249 Nov 29 2018 NOTICE 19-rw-r--r-- 1 mysql mysql 2483 Nov 16 2017 README.md 20-rw-rw-r-- 1 mysql mysql 1958 Dec 10 2018 RELEASE-NOTES 21drwxr-xr-x 2 root root 68 Jul 19 14:31 tools 22[root@node105.yinzhengjie.org.cn ~]#
3>.配置flume的环境变量
1[root@node105.yinzhengjie.org.cn ~]# vi /etc/profile 2[root@node105.yinzhengjie.org.cn ~]# 3[root@node105.yinzhengjie.org.cn ~]# tail -3 /etc/profile 4#Add by yinzhengjie 5FLUME_HOME=/home/softwares/apache-flume-1.9.0-bin 6PATH=$PATH:$FLUME_HOME/bin 7[root@node105.yinzhengjie.org.cn ~]# 8[root@node105.yinzhengjie.org.cn ~]# source /etc/profile 9[root@node105.yinzhengjie.org.cn ~]# 10[root@node105.yinzhengjie.org.cn ~]# flume-ng version 11Flume 1.9.0 12Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git 13Revision: d4fcab4f501d41597bc616921329a4339f73585e 14Compiled by fszabo on Mon Dec 17 20:45:25 CET 2018 15From source with checksum 35db629a3bda49d23e9b3690c80737f9 16[root@node105.yinzhengjie.org.cn ~]#
4>.自定义flume的配置文件存放目录
1[root@node105.yinzhengjie.org.cn ~]# mkdir -pv /home/data/flume/{log,job,shell} 2mkdir: created directory ‘/home/data’ 3mkdir: created directory ‘/home/data/flume’ 4mkdir: created directory ‘/home/data/flume/log’ 5mkdir: created directory ‘/home/data/flume/job’ 6mkdir: created directory ‘/home/data/flume/shell’ 7[root@node105.yinzhengjie.org.cn ~]# 8[root@node105.yinzhengjie.org.cn ~]# 9[root@node105.yinzhengjie.org.cn ~]# ll /home/data/flume/ 10total 0 11drwxr-xr-x 2 root root 6 Jul 19 14:42 job #用于存放flume启动的agent端的配置文件 12drwxr-xr-x 2 root root 6 Jul 19 14:42 log #用于存放日志文件 13drwxr-xr-x 2 root root 6 Jul 19 14:42 shell #用于存放启动脚本 14[root@node105.yinzhengjie.org.cn ~]# 15[root@node105.yinzhengjie.org.cn ~]#
三.Flume案例
1>.监控端口数据（netcat source-memory channel-logger sink）
1[root@node105.yinzhengjie.org.cn ~]# yum -y install telnet net-tools 2Loaded plugins: fastestmirror 3Determining fastest mirrors 4 * base: mirror.bit.edu.cn 5 * extras: mirrors.aliyun.com 6 * updates: mirrors.aliyun.com 7base | 3.6 kB 00:00:00 8extras | 3.4 kB 00:00:00 9updates | 3.4 kB 00:00:00 10updates/7/x86_64/primary_db | 6.5 MB 00:00:02 11Package net-tools-2.0-0.24.20131004git.el7.x86_64 already installed and latest version 12Resolving Dependencies 13--> Running transaction check 14---> Package telnet.x86_64 1:0.17-64.el7 will be installed 15--> Finished Dependency Resolution 16 17Dependencies Resolved 18 19============================================================================================================================================================================================================================================================================== 20 Package Arch Version Repository Size 21============================================================================================================================================================================================================================================================================== 22Installing: 23 telnet x86_64 1:0.17-64.el7 base 64 k 24 25Transaction Summary 26============================================================================================================================================================================================================================================================================== 27Install 1 Package 28 29Total download size: 64 k 30Installed size: 113 k 31Downloading packages: 32telnet-0.17-64.el7.x86_64.rpm | 64 kB 00:00:04 33Running transaction check 34Running transaction test 35Transaction test succeeded 36Running transaction 37 Installing : 1:telnet-0.17-64.el7.x86_64 1/1 38 Verifying : 1:telnet-0.17-64.el7.x86_64 1/1 39 40Installed: 41 telnet.x86_64 1:0.17-64.el7 42 43Complete! 44[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# yum -y install telnet net-tools
1[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/job/flume-netcat.conf 2# 这里的“yinzhengjie”是agent的名称，它是我们自定义的。我们分别给“yinzhengjie”的sources，sinks，channels的别名分别为r1,k1和c1 3yinzhengjie.sources = r1 4yinzhengjie.sinks = k1 5yinzhengjie.channels = c1 6 7yinzhengjie.sources.r1.type = netcat 8yinzhengjie.sources.r1.bind = node105.yinzhengjie.org.cn 9yinzhengjie.sources.r1.port = 8888 10 11# 指定sink的类型，我们这里指定的为logger，即控制台输出。 12yinzhengjie.sinks.k1.type = logger 13 14# 指定channel的类型为memory，指定channel的容量是1000，每次传输的容量是100 15yinzhengjie.channels.c1.type = memory 16yinzhengjie.channels.c1.capacity = 1000 17yinzhengjie.channels.c1.transactionCapacity = 100 18 19# 绑定source和sink 20yinzhengjie.sources.r1.channels = c1 21yinzhengjie.sinks.k1.channel = c1 22[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/job/flume-netcat.conf #编写flume的agent配置文件
1[root@node105.yinzhengjie.org.cn ~]# flume-ng agent --conf /home/softwares/apache-flume-1.9.0-bin/conf --name yinzhengjie --conf-file /home/data/flume/job/flume-netcat.conf -Dflume.monitoring.type=http -Dflume.monitoring.port=10501 -Dflume.root.logger==INFO,console 2Warning: JAVA_HOME is not set! 3Info: Including Hive libraries found via () for Hive access 4+ exec /home/softwares/jdk1.8.0_201/bin/java -Xmx20m -Dflume.monitoring.type=http -Dflume.monitoring.port=10501 -Dflume.root.logger==INFO,console -cp '/home/softwares/apache-flume-1.9.0-bin/conf:/home/softwares/apache-flume-1.9.0-bin/lib/*:/lib/*' -Djava.library.path= o 5rg.apache.flume.node.Application --name yinzhengjie --conf-file /home/data/flume/job/flume-netcat.conf2019-07-19 15:07:00,130 (main) [DEBUG - org.apache.flume.util.SSLUtil.initSysPropFromEnvVar(SSLUtil.java:95)] No global SSL keystore path specified. 62019-07-19 15:07:00,136 (main) [DEBUG - org.apache.flume.util.SSLUtil.initSysPropFromEnvVar(SSLUtil.java:95)] No global SSL keystore password specified. 72019-07-19 15:07:00,136 (main) [DEBUG - org.apache.flume.util.SSLUtil.initSysPropFromEnvVar(SSLUtil.java:95)] No global SSL keystore type specified. 82019-07-19 15:07:00,136 (main) [DEBUG - org.apache.flume.util.SSLUtil.initSysPropFromEnvVar(SSLUtil.java:95)] No global SSL truststore path specified. 92019-07-19 15:07:00,154 (main) [DEBUG - org.apache.flume.util.SSLUtil.initSysPropFromEnvVar(SSLUtil.java:95)] No global SSL truststore password specified. 102019-07-19 15:07:00,154 (main) [DEBUG - org.apache.flume.util.SSLUtil.initSysPropFromEnvVar(SSLUtil.java:95)] No global SSL truststore type specified. 112019-07-19 15:07:00,154 (main) [DEBUG - org.apache.flume.util.SSLUtil.initSysPropFromEnvVar(SSLUtil.java:95)] No global SSL include protocols specified. 122019-07-19 15:07:00,154 (main) [DEBUG - org.apache.flume.util.SSLUtil.initSysPropFromEnvVar(SSLUtil.java:95)] No global SSL exclude protocols specified. 132019-07-19 15:07:00,154 (main) [DEBUG - org.apache.flume.util.SSLUtil.initSysPropFromEnvVar(SSLUtil.java:95)] No global SSL include cipher suites specified. 142019-07-19 15:07:00,154 (main) [DEBUG - org.apache.flume.util.SSLUtil.initSysPropFromEnvVar(SSLUtil.java:95)] No global SSL exclude cipher suites specified. 152019-07-19 15:07:00,265 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:62)] Configuration provider starting 162019-07-19 15:07:00,272 (lifecycleSupervisor-1-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:79)] Configuration provider started 172019-07-19 15:07:00,274 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:131)] Checking file:/home/data/flume/job/flume-netcat.conf for changes 182019-07-19 15:07:00,274 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:138)] Reloading configuration file:/home/data/flume/job/flume-netcat.conf 192019-07-19 15:07:00,278 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addComponentConfig(FlumeConfiguration.java:1203)] Processing:r1 202019-07-19 15:07:00,279 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addComponentConfig(FlumeConfiguration.java:1207)] Created context for r1: type 212019-07-19 15:07:00,282 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addComponentConfig(FlumeConfiguration.java:1203)] Processing:c1 222019-07-19 15:07:00,283 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addComponentConfig(FlumeConfiguration.java:1207)] Created context for c1: type 232019-07-19 15:07:00,283 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1117)] Added sinks: k1 Agent: yinzhengjie 242019-07-19 15:07:00,283 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addComponentConfig(FlumeConfiguration.java:1203)] Processing:c1 252019-07-19 15:07:00,283 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addComponentConfig(FlumeConfiguration.java:1203)] Processing:r1 262019-07-19 15:07:00,283 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addComponentConfig(FlumeConfiguration.java:1203)] Processing:k1 272019-07-19 15:07:00,293 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addComponentConfig(FlumeConfiguration.java:1207)] Created context for k1: channel 282019-07-19 15:07:00,294 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addComponentConfig(FlumeConfiguration.java:1203)] Processing:r1 292019-07-19 15:07:00,294 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addComponentConfig(FlumeConfiguration.java:1203)] Processing:r1 302019-07-19 15:07:00,294 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addComponentConfig(FlumeConfiguration.java:1203)] Processing:k1 312019-07-19 15:07:00,294 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addComponentConfig(FlumeConfiguration.java:1203)] Processing:c1 322019-07-19 15:07:00,294 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:350)] Starting validation of configuration for agent: yinzhengjie 332019-07-19 15:07:00,295 (conf-file-poller-0) [INFO - org.apache.flume.conf.LogPrivacyUtil.<clinit>(LogPrivacyUtil.java:51)] Logging of configuration details is disabled. To see configuration details in the log run the agent with -Dorg.apache.flume.log.printconfig=true J 34VM argument. Please note that this is not recommended in production systems as it may leak private information to the logfile.2019-07-19 15:07:00,295 (conf-file-poller-0) [WARN - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateConfigFilterSet(FlumeConfiguration.java:623)] Agent configuration for 'yinzhengjie' has no configfilters. 352019-07-19 15:07:00,310 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:583)] Created channel c1 362019-07-19 15:07:00,314 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:861)] Creating sink: k1 using LOGGER 372019-07-19 15:07:00,315 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:158)] Channels:c1 38 392019-07-19 15:07:00,315 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:159)] Sinks k1 40 412019-07-19 15:07:00,315 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:160)] Sources r1 42 432019-07-19 15:07:00,316 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:163)] Post-validation flume configuration contains configuration for agents: [yinzhengjie] 442019-07-19 15:07:00,316 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:151)] Creating channels 452019-07-19 15:07:00,340 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:42)] Creating instance of channel c1 type memory 462019-07-19 15:07:00,343 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:205)] Created channel c1 472019-07-19 15:07:00,344 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:41)] Creating instance of source r1, type netcat 482019-07-19 15:07:00,354 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)] Creating instance of sink: k1, type: logger 492019-07-19 15:07:00,358 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:120)] Channel c1 connected to [r1, k1] 502019-07-19 15:07:00,379 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:162)] Starting new configuration:{ sourceRunners:{r1=EventDrivenSourceRunner: { source:org.apache.flume.source.NetcatSource{name:r1,state:IDLE} }} 51sinkRunners:{k1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@3344b1b counterGroup:{ name:null counters:{} } }} channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1}} }2019-07-19 15:07:00,383 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:169)] Starting Channel c1 522019-07-19 15:07:00,386 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:184)] Waiting for channel: c1 to start. Sleeping for 500 ms 532019-07-19 15:07:00,470 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: CHANNEL, name: c1: Successfully registered new MBean. 542019-07-19 15:07:00,471 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: CHANNEL, name: c1 started 552019-07-19 15:07:00,887 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:196)] Starting Sink k1 562019-07-19 15:07:00,890 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:207)] Starting Source r1 572019-07-19 15:07:00,893 (lifecycleSupervisor-1-4) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:155)] Source starting 582019-07-19 15:07:00,958 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.log.Log.initialized(Log.java:180)] Logging to org.slf4j.impl.Log4jLoggerAdapter(org.eclipse.jetty.util.log) via org.eclipse.jetty.util.log.Slf4jLog 592019-07-19 15:07:00,959 (conf-file-poller-0) [INFO - org.eclipse.jetty.util.log.Log.initialized(Log.java:192)] Logging initialized @1169ms to org.eclipse.jetty.util.log.Slf4jLog 602019-07-19 15:07:00,964 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:322)] org.eclipse.jetty.server.Server@346a3eed added {qtp1818551798{STOPPED,8<=0<=200,i=0,q=0},AUTO} 612019-07-19 15:07:00,967 (SinkRunner-PollingRunner-DefaultSinkProcessor) [DEBUG - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:141)] Polling sink runner starting 622019-07-19 15:07:00,982 (lifecycleSupervisor-1-4) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:166)] Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/172.30.1.105:8888] 632019-07-19 15:07:00,985 (lifecycleSupervisor-1-4) [DEBUG - org.apache.flume.source.NetcatSource.start(NetcatSource.java:191)] Source started 642019-07-19 15:07:00,986 (Thread-2) [DEBUG - org.apache.flume.source.NetcatSource$AcceptHandler.run(NetcatSource.java:271)] Starting accept handler 652019-07-19 15:07:01,025 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:322)] HttpConnectionFactory@5ac8a68a[HTTP/1.1] added {HttpConfiguration@7454628a{32768/8192,8192/8192,https://:0,[]},POJO} 662019-07-19 15:07:01,028 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:322)] ServerConnector@65639034{null,[]}{0.0.0.0:0} added {org.eclipse.jetty.server.Server@346a3eed,UNMANAGED} 672019-07-19 15:07:01,029 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:322)] ServerConnector@65639034{null,[]}{0.0.0.0:0} added {qtp1818551798{STOPPED,8<=0<=200,i=0,q=0},AUTO} 682019-07-19 15:07:01,030 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:322)] ServerConnector@65639034{null,[]}{0.0.0.0:0} added {org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@1da3d1e8,AUTO} 692019-07-19 15:07:01,030 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:322)] ServerConnector@65639034{null,[]}{0.0.0.0:0} added {org.eclipse.jetty.io.ArrayByteBufferPool@4fa9a485,POJO} 702019-07-19 15:07:01,031 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:322)] ServerConnector@65639034{null,[http/1.1]}{0.0.0.0:0} added {HttpConnectionFactory@5ac8a68a[HTTP/1.1],AUTO} 712019-07-19 15:07:01,032 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.server.AbstractConnector.addConnectionFactory(AbstractConnector.java:406)] ServerConnector@65639034{HTTP/1.1,[http/1.1]}{0.0.0.0:0} added HttpConnectionFactory@5ac8a68a[HTTP/1.1] 722019-07-19 15:07:01,033 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:322)] ServerConnector@65639034{HTTP/1.1,[http/1.1]}{0.0.0.0:0} added {org.eclipse.jetty.server.ServerConnector$ServerConnectorManage 73r@4fc180ce,MANAGED}2019-07-19 15:07:01,046 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:322)] org.eclipse.jetty.server.Server@346a3eed added {ServerConnector@65639034{HTTP/1.1,[http/1.1]}{0.0.0.0:10501},AUTO} 742019-07-19 15:07:01,101 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:322)] org.eclipse.jetty.server.Server@346a3eed added {org.apache.flume.instrumentation.http.HTTPMetricsServer$HTTPMetricsHandler@52e 75879b7,MANAGED}2019-07-19 15:07:01,101 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.AbstractLifeCycle.setStarting(AbstractLifeCycle.java:185)] starting org.eclipse.jetty.server.Server@346a3eed 762019-07-19 15:07:01,102 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:322)] org.eclipse.jetty.server.Server@346a3eed added {org.eclipse.jetty.server.handler.ErrorHandler@4ed57293,AUTO} 772019-07-19 15:07:01,103 (conf-file-poller-0) [INFO - org.eclipse.jetty.server.Server.doStart(Server.java:372)] jetty-9.4.6.v20170531 782019-07-19 15:07:01,142 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:110)] starting org.eclipse.jetty.server.Server@346a3eed 792019-07-19 15:07:01,143 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.AbstractLifeCycle.setStarting(AbstractLifeCycle.java:185)] starting qtp1818551798{STOPPED,8<=0<=200,i=0,q=0} 802019-07-19 15:07:01,170 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:177)] STARTED @1380ms qtp1818551798{STARTED,8<=8<=200,i=7,q=0} 812019-07-19 15:07:01,171 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.AbstractLifeCycle.setStarting(AbstractLifeCycle.java:185)] starting org.apache.flume.instrumentation.http.HTTPMetricsServer$HTTPMetricsHandler@52e879b7 822019-07-19 15:07:01,171 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:110)] starting org.apache.flume.instrumentation.http.HTTPMetricsServer$HTTPMetricsHandler@52e879b7 832019-07-19 15:07:01,171 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:177)] STARTED @1381ms org.apache.flume.instrumentation.http.HTTPMetricsServer$HTTPMetricsHandler@52e879b7 842019-07-19 15:07:01,171 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.AbstractLifeCycle.setStarting(AbstractLifeCycle.java:185)] starting org.eclipse.jetty.server.handler.ErrorHandler@4ed57293 852019-07-19 15:07:01,172 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:110)] starting org.eclipse.jetty.server.handler.ErrorHandler@4ed57293 862019-07-19 15:07:01,172 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:177)] STARTED @1382ms org.eclipse.jetty.server.handler.ErrorHandler@4ed57293 872019-07-19 15:07:01,172 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.AbstractLifeCycle.setStarting(AbstractLifeCycle.java:185)] starting ServerConnector@65639034{HTTP/1.1,[http/1.1]}{0.0.0.0:10501} 882019-07-19 15:07:01,173 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:322)] ServerConnector@65639034{HTTP/1.1,[http/1.1]}{0.0.0.0:10501} added {sun.nio.ch.ServerSocketChannelImpl[/0.0.0.0:10501],POJO} 892019-07-19 15:07:01,173 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.AbstractLifeCycle.setStarting(AbstractLifeCycle.java:185)] starting org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@1da3d1e8 902019-07-19 15:07:01,174 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:177)] STARTED @1384ms org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@1da3d1e8 912019-07-19 15:07:01,174 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.AbstractLifeCycle.setStarting(AbstractLifeCycle.java:185)] starting HttpConnectionFactory@5ac8a68a[HTTP/1.1] 922019-07-19 15:07:01,174 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:177)] STARTED @1384ms HttpConnectionFactory@5ac8a68a[HTTP/1.1] 932019-07-19 15:07:01,175 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.AbstractLifeCycle.setStarting(AbstractLifeCycle.java:185)] starting org.eclipse.jetty.server.ServerConnector$ServerConnectorManager@4fc180ce 942019-07-19 15:07:01,178 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:322)] org.eclipse.jetty.io.ManagedSelector@1fe6c0aa id=0 keys=-1 selected=-1 added {EatWhatYouKill@41905dc0/org.eclipse.jetty.io.Man 95agedSelector$SelectorProducer@7dcfdb71/IDLE/0/1,AUTO}2019-07-19 15:07:01,178 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:322)] org.eclipse.jetty.server.ServerConnector$ServerConnectorManager@4fc180ce added {org.eclipse.jetty.io.ManagedSelector@1fe6c0aa 96id=0 keys=-1 selected=-1,AUTO}2019-07-19 15:07:01,179 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.AbstractLifeCycle.setStarting(AbstractLifeCycle.java:185)] starting org.eclipse.jetty.io.ManagedSelector@1fe6c0aa id=0 keys=-1 selected=-1 972019-07-19 15:07:01,179 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.AbstractLifeCycle.setStarting(AbstractLifeCycle.java:185)] starting EatWhatYouKill@41905dc0/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@7dcfdb71/IDLE/0/1 982019-07-19 15:07:01,179 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:177)] STARTED @1389ms EatWhatYouKill@41905dc0/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@7dcfdb71/IDLE/0/1 992019-07-19 15:07:01,266 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:381)] queue org.eclipse.jetty.io.ManagedSelector$$Lambda$1/1484530269@6d147c43 1002019-07-19 15:07:01,267 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:177)] STARTED @1477ms org.eclipse.jetty.io.ManagedSelector@1fe6c0aa id=0 keys=0 selected=0 1012019-07-19 15:07:01,267 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:177)] STARTED @1477ms org.eclipse.jetty.server.ServerConnector$ServerConnectorManager@4fc180ce 1022019-07-19 15:07:01,268 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:322)] ServerConnector@65639034{HTTP/1.1,[http/1.1]}{0.0.0.0:10501} added {acceptor-0@2a11399b,POJO} 1032019-07-19 15:07:01,268 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:381)] queue acceptor-0@2a11399b 1042019-07-19 15:07:01,268 (conf-file-poller-0) [INFO - org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:280)] Started ServerConnector@65639034{HTTP/1.1,[http/1.1]}{0.0.0.0:10501} 1052019-07-19 15:07:01,269 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:177)] STARTED @1479ms ServerConnector@65639034{HTTP/1.1,[http/1.1]}{0.0.0.0:10501} 1062019-07-19 15:07:01,269 (conf-file-poller-0) [INFO - org.eclipse.jetty.server.Server.doStart(Server.java:444)] Started @1479ms 1072019-07-19 15:07:01,269 (conf-file-poller-0) [DEBUG - org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:177)] STARTED @1479ms org.eclipse.jetty.server.Server@346a3eed 1082019-07-19 15:07:01,269 (qtp1818551798-22) [DEBUG - org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)] run org.eclipse.jetty.io.ManagedSelector$$Lambda$1/1484530269@6d147c43 1092019-07-19 15:07:01,270 (qtp1818551798-22) [DEBUG - org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:123)] EatWhatYouKill@41905dc0/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@7dcfdb71/PRODUCING/0/1 execute true 1102019-07-19 15:07:01,270 (qtp1818551798-22) [DEBUG - org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:206)] EatWhatYouKill@41905dc0/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@7dcfdb71/PRODUCING/0/1 produce non-blocking 1112019-07-19 15:07:01,270 (qtp1818551798-22) [DEBUG - org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233)] Selector loop waiting on select 1122019-07-19 15:07:01,272 (qtp1818551798-23) [DEBUG - org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)] run acceptor-0@2a11399b 1132019-07-19 15:07:31,271 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:131)] Checking file:/home/data/flume/job/flume-netcat.conf for changes 1142019-07-19 15:07:47,034 (netcat-handler-0) [DEBUG - org.apache.flume.source.NetcatSource$NetcatSocketHandler.run(NetcatSource.java:316)] Starting connection handler 1152019-07-19 15:07:58,158 (netcat-handler-0) [DEBUG - org.apache.flume.source.NetcatSource$NetcatSocketHandler.run(NetcatSource.java:328)] Chars read = 10 1162019-07-19 15:07:58,174 (netcat-handler-0) [DEBUG - org.apache.flume.source.NetcatSource$NetcatSocketHandler.run(NetcatSource.java:332)] Events processed = 1 1172019-07-19 15:08:01,273 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:131)] Checking file:/home/data/flume/job/flume-netcat.conf for changes 1182019-07-19 15:08:02,988 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: E5 B0 B9 E6 AD A3 E6 9D B0 E5 88 B0 E6 AD A4 E4 ................ } 1192019-07-19 15:08:31,274 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:131)] Checking file:/home/data/flume/job/flume-netcat.conf for changes 1202019-07-19 15:08:33,975 (netcat-handler-0) [DEBUG - org.apache.flume.source.NetcatSource$NetcatSocketHandler.run(NetcatSource.java:328)] Chars read = 29 1212019-07-19 15:08:33,976 (netcat-handler-0) [DEBUG - org.apache.flume.source.NetcatSource$NetcatSocketHandler.run(NetcatSource.java:332)] Events processed = 1 1222019-07-19 15:08:33,976 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 79 69 6E 7A 68 65 6E 67 6A 69 65 20 64 61 6F 20 yinzhengjie dao } 1232019-07-19 15:08:51,938 (netcat-handler-0) [DEBUG - org.apache.flume.source.NetcatSource$NetcatSocketHandler.run(NetcatSource.java:328)] Chars read = 8 1242019-07-19 15:08:51,938 (netcat-handler-0) [DEBUG - org.apache.flume.source.NetcatSource$NetcatSocketHandler.run(NetcatSource.java:332)] Events processed = 1 1252019-07-19 15:08:51,938 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 67 6F 6C 61 6E 67 0D golang. } 1262019-07-19 15:08:54,481 (netcat-handler-0) [DEBUG - org.apache.flume.source.NetcatSource$NetcatSocketHandler.run(NetcatSource.java:328)] Chars read = 8 1272019-07-19 15:08:54,481 (netcat-handler-0) [DEBUG - org.apache.flume.source.NetcatSource$NetcatSocketHandler.run(NetcatSource.java:332)] Events processed = 1 1282019-07-19 15:08:54,481 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 70 79 74 68 6F 6E 0D python. } 1292019-07-19 15:08:56,285 (netcat-handler-0) [DEBUG - org.apache.flume.source.NetcatSource$NetcatSocketHandler.run(NetcatSource.java:328)] Chars read = 6 1302019-07-19 15:08:56,285 (netcat-handler-0) [DEBUG - org.apache.flume.source.NetcatSource$NetcatSocketHandler.run(NetcatSource.java:332)] Events processed = 1 1312019-07-19 15:08:56,285 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 6A 61 76 61 0D java. } 1322019-07-19 15:09:01,277 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:131)] Checking file:/home/data/flume/job/flume-netcat.conf for changes
[root@node105.yinzhengjie.org.cn ~]# flume-ng agent --conf /home/softwares/apache-flume-1.9.0-bin/conf --name #启动flume
1[root@node105.yinzhengjie.org.cn ~]# ss -ntl 2State Recv-Q Send-Q Local Address:Port Peer Address:Port 3LISTEN 0 128 *:22 *:* 4LISTEN 0 50 172.30.1.105:8888 *:* 5LISTEN 0 50 *:10501 *:* 6LISTEN 0 128 :::22 :::* 7[root@node105.yinzhengjie.org.cn ~]# 8[root@node105.yinzhengjie.org.cn ~]# telnet node105.yinzhengjie.org.cn 8888 9Trying 172.30.1.105... 10Connected to node105.yinzhengjie.org.cn. 11Escape character is '^]'. 12尹正杰到此一游! 13OK 14yinzhengjie dao ci yi you ! 15OK 16golang 17OK 18python 19OK 20java 21OK
[root@node105.yinzhengjie.org.cn ~]# telnet node105.yinzhengjie.org.cn 8888 #测试连接flume
1[root@node105.yinzhengjie.org.cn ~]# vi /home/data/flume/shell/start-netcat.sh 2[root@node105.yinzhengjie.org.cn ~]# 3[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/shell/start-netcat.sh 4#!/bin/bash 5#@author :yinzhengjie 6#blog:http://www.cnblogs.com/yinzhengjie 7#EMAIL:y1053419035@qq.com 8#Data:Thu Oct 18 11:26:06 CST 2018 9 10#将监控数据发送给ganglia，需要指定ganglia服务器地址，使用请确认是否部署好ganglia服务！ 11#nohup flume-ng agent -c /home/data/flume/job/ --conf-file=/home/data/flume/job/flume-netcat.conf --name yinzhengjie -Dflume.monitoring.type=ganglia -Dflume.monitoring.hosts=node105.yinzhengjie.org.cn:8649 -Dflume.root.logger=INFO,console >> /home/data/flume/log/flume-g 12anglia-flume-netcat.log 2>&1 & 13 14#启动flume自身的监控参数，默认执行以下脚本 15nohup flume-ng agent -c /home/softwares/apache-flume-1.9.0-bin/conf --conf-file=/home/data/flume/job/flume-netcat.conf --name yinzhengjie -Dflume.monitoring.type=http -Dflume.monitoring.port=10501 -Dflume.root.logger=INFO,console >> /home/data/flume/log/flume-netcat.l 16og 2>&1 &[root@node105.yinzhengjie.org.cn ~]# 17[root@node105.yinzhengjie.org.cn ~]# chmod +x /home/data/flume/shell/start-netcat.sh 18[root@node105.yinzhengjie.org.cn ~]# 19[root@node105.yinzhengjie.org.cn ~]# ll /home/data/flume/shell/start-netcat.sh 20-rwxr-xr-x 1 root root 902 Jul 19 15:15 /home/data/flume/shell/start-netcat.sh 21[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/shell/start-netcat.sh #编写flume的启动脚本，生产环境推荐大家使用该方式！
1[root@node105.yinzhengjie.org.cn ~]# ss -ntl 2State Recv-Q Send-Q Local Address:Port Peer Address:Port 3LISTEN 0 128 *:22 *:* 4LISTEN 0 128 :::22 :::* 5[root@node105.yinzhengjie.org.cn ~]# 6[root@node105.yinzhengjie.org.cn ~]# 7[root@node105.yinzhengjie.org.cn ~]# /home/data/flume/shell/start-netcat.sh 8[root@node105.yinzhengjie.org.cn ~]# 9[root@node105.yinzhengjie.org.cn ~]# ss -ntl 10State Recv-Q Send-Q Local Address:Port Peer Address:Port 11LISTEN 0 128 *:22 *:* 12LISTEN 0 50 172.30.1.105:8888 *:* 13LISTEN 0 50 *:10501 *:* 14LISTEN 0 128 :::22 :::* 15[root@node105.yinzhengjie.org.cn ~]# 16[root@node105.yinzhengjie.org.cn ~]# 17[root@node105.yinzhengjie.org.cn ~]# tail -100f /home/data/flume/log/flume-netcat.log 18Warning: JAVA_HOME is not set! 19Info: Including Hive libraries found via () for Hive access 20+ exec /home/softwares/jdk1.8.0_201/bin/java -Xmx20m -Dflume.monitoring.type=http -Dflume.monitoring.port=10501 -Dflume.root.logger=INFO,console -cp '/home/data/flume/job:/home/softwares/apache-flume-1.9.0-bin/lib/*:/lib/*' -Djava.library.path= org.apache.flume.node.App 21lication --conf-file=/home/data/flume/job/flume-netcat.conf --name yinzhengjielog4j:WARN No appenders could be found for logger (org.apache.flume.util.SSLUtil). 22log4j:WARN Please initialize the log4j system properly. 23log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. 24Warning: JAVA_HOME is not set! 25Info: Including Hive libraries found via () for Hive access 26+ exec /home/softwares/jdk1.8.0_201/bin/java -Xmx20m -Dflume.monitoring.type=http -Dflume.monitoring.port=10501 -Dflume.root.logger=INFO,console -cp '/home/softwares/apache-flume-1.9.0-bin/conf:/home/softwares/apache-flume-1.9.0-bin/lib/*:/lib/*' -Djava.library.path= or 27g.apache.flume.node.Application --conf-file=/home/data/flume/job/flume-netcat.conf --name yinzhengjie2019-07-19 15:14:19,044 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:62)] Configuration provider starting 282019-07-19 15:14:19,054 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:138)] Reloading configuration file:/home/data/flume/job/flume-netcat.conf 292019-07-19 15:14:19,060 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addComponentConfig(FlumeConfiguration.java:1203)] Processing:r1 302019-07-19 15:14:19,060 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addComponentConfig(FlumeConfiguration.java:1203)] Processing:c1 312019-07-19 15:14:19,060 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1117)] Added sinks: k1 Agent: yinzhengjie 322019-07-19 15:14:19,061 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addComponentConfig(FlumeConfiguration.java:1203)] Processing:c1 332019-07-19 15:14:19,061 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addComponentConfig(FlumeConfiguration.java:1203)] Processing:r1 342019-07-19 15:14:19,061 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addComponentConfig(FlumeConfiguration.java:1203)] Processing:k1 352019-07-19 15:14:19,061 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addComponentConfig(FlumeConfiguration.java:1203)] Processing:r1 362019-07-19 15:14:19,061 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addComponentConfig(FlumeConfiguration.java:1203)] Processing:r1 372019-07-19 15:14:19,061 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addComponentConfig(FlumeConfiguration.java:1203)] Processing:k1 382019-07-19 15:14:19,061 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addComponentConfig(FlumeConfiguration.java:1203)] Processing:c1 392019-07-19 15:14:19,062 (conf-file-poller-0) [WARN - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateConfigFilterSet(FlumeConfiguration.java:623)] Agent configuration for 'yinzhengjie' has no configfilters. 402019-07-19 15:14:19,080 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:163)] Post-validation flume configuration contains configuration for agents: [yinzhengjie] 412019-07-19 15:14:19,080 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:151)] Creating channels 422019-07-19 15:14:19,093 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:42)] Creating instance of channel c1 type memory 432019-07-19 15:14:19,098 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:205)] Created channel c1 442019-07-19 15:14:19,105 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:41)] Creating instance of source r1, type netcat 452019-07-19 15:14:19,110 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)] Creating instance of sink: k1, type: logger 462019-07-19 15:14:19,112 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:120)] Channel c1 connected to [r1, k1] 472019-07-19 15:14:19,129 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:162)] Starting new configuration:{ sourceRunners:{r1=EventDrivenSourceRunner: { source:org.apache.flume.source.NetcatSource{name:r1,state:IDLE} }} 48sinkRunners:{k1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@5b204d1f counterGroup:{ name:null counters:{} } }} channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1}} }2019-07-19 15:14:19,133 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:169)] Starting Channel c1 492019-07-19 15:14:19,135 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:184)] Waiting for channel: c1 to start. Sleeping for 500 ms 502019-07-19 15:14:19,221 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: CHANNEL, name: c1: Successfully registered new MBean. 512019-07-19 15:14:19,221 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: CHANNEL, name: c1 started 522019-07-19 15:14:19,636 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:196)] Starting Sink k1 532019-07-19 15:14:19,637 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:207)] Starting Source r1 542019-07-19 15:14:19,638 (lifecycleSupervisor-1-4) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:155)] Source starting 552019-07-19 15:14:19,681 (lifecycleSupervisor-1-4) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:166)] Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/172.30.1.105:8888] 562019-07-19 15:14:19,694 (conf-file-poller-0) [INFO - org.eclipse.jetty.util.log.Log.initialized(Log.java:192)] Logging initialized @1117ms to org.eclipse.jetty.util.log.Slf4jLog 572019-07-19 15:14:19,812 (conf-file-poller-0) [INFO - org.eclipse.jetty.server.Server.doStart(Server.java:372)] jetty-9.4.6.v20170531 582019-07-19 15:14:19,935 (conf-file-poller-0) [INFO - org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:280)] Started ServerConnector@48970ee9{HTTP/1.1,[http/1.1]}{0.0.0.0:10501} 592019-07-19 15:14:19,935 (conf-file-poller-0) [INFO - org.eclipse.jetty.server.Server.doStart(Server.java:444)] Started @1358ms 602019-07-19 15:15:05,701 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 79 69 6E 7A 68 65 6E 67 6A 69 65 20 64 61 6F 20 yinzhengjie dao }
[root@node105.yinzhengjie.org.cn ~]# tail -100f /home/data/flume/log/flume-netcat.log #启动flume并查看启动日志信息
1[root@node105.yinzhengjie.org.cn ~]# yum -y install epel-release 2Loaded plugins: fastestmirror 3Loading mirror speeds from cached hostfile 4 * base: mirror.bit.edu.cn 5 * extras: mirrors.aliyun.com 6 * updates: mirrors.aliyun.com 7Resolving Dependencies 8--> Running transaction check 9---> Package epel-release.noarch 0:7-11 will be installed 10--> Finished Dependency Resolution 11 12Dependencies Resolved 13 14============================================================================================================================================================================================================================================================================== 15 Package Arch Version Repository Size 16============================================================================================================================================================================================================================================================================== 17Installing: 18 epel-release noarch 7-11 extras 15 k 19 20Transaction Summary 21============================================================================================================================================================================================================================================================================== 22Install 1 Package 23 24Total download size: 15 k 25Installed size: 24 k 26Downloading packages: 27epel-release-7-11.noarch.rpm | 15 kB 00:00:00 28Running transaction check 29Running transaction test 30Transaction test succeeded 31Running transaction 32 Installing : epel-release-7-11.noarch 1/1 33 Verifying : epel-release-7-11.noarch 1/1 34 35Installed: 36 epel-release.noarch 0:7-11 37 38Complete! 39[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# yum -y install epel-release #安装EPEL源
1[root@node105.yinzhengjie.org.cn ~]# yum list jq 2Loaded plugins: fastestmirror 3Loading mirror speeds from cached hostfile 4epel/x86_64/metalink | 6.1 kB 00:00:00 5 * base: mirror.bit.edu.cn 6 * epel: mirrors.tuna.tsinghua.edu.cn 7 * extras: mirrors.aliyun.com 8 * updates: mirrors.aliyun.com 9epel | 5.3 kB 00:00:00 10(1/3): epel/x86_64/group_gz | 88 kB 00:00:00 11epel/x86_64/updateinfo FAILED 12http://ftp.jaist.ac.jp/pub/Linux/Fedora/epel/7/x86_64/repodata/52f0298e60c86c08c5a90ffdff1f223a1166be2d7e011c9015ecfc8dc8bdf38b-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found ] 0.0 B/s | 0 B --:--:-- ETA 13Trying other mirror. 14To address this issue please refer to the below wiki article 15 16https://wiki.centos.org/yum-errors 17 18If above article doesn't help to resolve this issue please use https://bugs.centos.org/. 19 20(2/3): epel/x86_64/updateinfo | 990 kB 00:00:00 21(3/3): epel/x86_64/primary_db | 6.8 MB 00:00:04 22Available Packages 23jq.x86_64 1.5-1.el7 epel 24[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# yum list jq #查看jq安装包是否存在
1[root@node105.yinzhengjie.org.cn ~]# yum -y install jq 2Loaded plugins: fastestmirror 3Loading mirror speeds from cached hostfile 4 * base: mirror.bit.edu.cn 5 * epel: mirrors.yun-idc.com 6 * extras: mirrors.aliyun.com 7 * updates: mirrors.aliyun.com 8Resolving Dependencies 9--> Running transaction check 10---> Package jq.x86_64 0:1.5-1.el7 will be installed 11--> Processing Dependency: libonig.so.2()(64bit) for package: jq-1.5-1.el7.x86_64 12--> Running transaction check 13---> Package oniguruma.x86_64 0:5.9.5-3.el7 will be installed 14--> Finished Dependency Resolution 15 16Dependencies Resolved 17 18============================================================================================================================================================================================================================================================================== 19 Package Arch Version Repository Size 20============================================================================================================================================================================================================================================================================== 21Installing: 22 jq x86_64 1.5-1.el7 epel 153 k 23Installing for dependencies: 24 oniguruma x86_64 5.9.5-3.el7 epel 129 k 25 26Transaction Summary 27============================================================================================================================================================================================================================================================================== 28Install 1 Package (+1 Dependent package) 29 30Total download size: 282 k 31Installed size: 906 k 32Downloading packages: 33warning: /var/cache/yum/x86_64/7/epel/packages/jq-1.5-1.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY 34Public key for jq-1.5-1.el7.x86_64.rpm is not installed 35(1/2): jq-1.5-1.el7.x86_64.rpm | 153 kB 00:00:00 36(2/2): oniguruma-5.9.5-3.el7.x86_64.rpm | 129 kB 00:00:02 37------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 38Total 118 kB/s | 282 kB 00:00:02 39Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 40Importing GPG key 0x352C64E5: 41 Userid : "Fedora EPEL (7) <epel@fedoraproject.org>" 42 Fingerprint: 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5 43 Package : epel-release-7-11.noarch (@extras) 44 From : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 45Running transaction check 46Running transaction test 47Transaction test succeeded 48Running transaction 49 Installing : oniguruma-5.9.5-3.el7.x86_64 1/2 50 Installing : jq-1.5-1.el7.x86_64 2/2 51 Verifying : oniguruma-5.9.5-3.el7.x86_64 1/2 52 Verifying : jq-1.5-1.el7.x86_64 2/2 53 54Installed: 55 jq.x86_64 0:1.5-1.el7 56 57Dependency Installed: 58 oniguruma.x86_64 0:5.9.5-3.el7 59 60Complete! 61[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# yum -y install jq #安装jq工具，便于我们查看json格式的内容
1[root@node105.yinzhengjie.org.cn ~]# curl http://node105.yinzhengjie.org.cn:10501/metrics | jq 2 % Total % Received % Xferd Average Speed Time Time Time Current 3 Dload Upload Total Spent Left Speed 4100 264 0 264 0 0 21605 0 --:--:-- --:--:-- --:--:-- 22000 5{ 6 "CHANNEL.c1": { #这是c1的CHANEL监控数据，c1该名称在flume-netcat.conf中配置文件中定义的。 7 "ChannelCapacity": "1000", #channel的容量，目前仅支持File Channel，Memory channel的统计数据。 8 "ChannelFillPercentage": "0.0", #channel已填入的百分比。 9 "Type": "CHANNEL", #很显然，这里是CHANNEL监控项，类型为CHANNEL。 10 "ChannelSize": "0", #目前channel中事件的总数量，目前仅支持File Channel，Memory channel的统计数据。 11 "EventTakeSuccessCount": "64", #sink成功从channel读取事件的总数量。 12 "EventTakeAttemptCount": "227", #sink尝试从channel拉取事件的总次数。这不意味着每次时间都被返回，因为sink拉取的时候channel可能没有任何数据。 13 "StartTime": "1563520459221", #channel启动时的毫秒值时间。 14 "EventPutAttemptCount": "64", #Source尝试写入Channe的事件总次数。 15 "EventPutSuccessCount": "64", #成功写入channel且提交的事件总次数。 16 "StopTime": "0" #channel停止时的毫秒值时间，为0表示一直在运行。 17 } 18} 19[root@node105.yinzhengjie.org.cn ~]# 20 21温馨提示： 22 如果你还要想了解更多度量值，可参考官方文档：http://flume.apache.org/FlumeUserGuide.html#monitoring。
1[root@node105.yinzhengjie.org.cn ~]# ss -ntl 2State Recv-Q Send-Q Local Address:Port Peer Address:Port 3LISTEN 0 128 *:22 *:* 4LISTEN 0 50 172.30.1.105:8888 *:* 5LISTEN 0 50 *:10501 *:* 6LISTEN 0 128 :::22 :::* 7[root@node105.yinzhengjie.org.cn ~]# 8[root@node105.yinzhengjie.org.cn ~]# netstat -untalp | grep 8888 9tcp 0 0 172.30.1.105:8888 0.0.0.0:* LISTEN 3816/java 10tcp 0 0 172.30.1.105:8888 172.30.1.105:47672 TIME_WAIT - 11[root@node105.yinzhengjie.org.cn ~]# 12[root@node105.yinzhengjie.org.cn ~]# 13[root@node105.yinzhengjie.org.cn ~]# jps 143816 Application 154426 Jps 16[root@node105.yinzhengjie.org.cn ~]# 17[root@node105.yinzhengjie.org.cn ~]# 18[root@node105.yinzhengjie.org.cn ~]# kill 3816 19[root@node105.yinzhengjie.org.cn ~]# 20[root@node105.yinzhengjie.org.cn ~]# jps 214440 Jps 22[root@node105.yinzhengjie.org.cn ~]# 23[root@node105.yinzhengjie.org.cn ~]# netstat -untalp | grep 8888 24tcp 0 0 172.30.1.105:8888 172.30.1.105:47672 TIME_WAIT - 25[root@node105.yinzhengjie.org.cn ~]# 26[root@node105.yinzhengjie.org.cn ~]# ss -ntl 27State Recv-Q Send-Q Local Address:Port Peer Address:Port 28LISTEN 0 128 *:22 *:* 29LISTEN 0 128 :::22 :::* 30[root@node105.yinzhengjie.org.cn ~]# 31[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# netstat -untalp | grep 8888 #杀掉对应的flume进程
2>.实时读取本地文件到HDFS集群（需要flume节点配置hadoop集群环境哟，exec source - memory channel - hdfs sink）
1[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/job/flume-hdfs.conf 2yinzhengjie2.sources = file_source 3yinzhengjie2.sinks = hdfs_sink 4yinzhengjie2.channels = memory_channel 5 6yinzhengjie2.sources.file_source.type = exec 7yinzhengjie2.sources.file_source.command = tail -F /var/log/messages 8yinzhengjie2.sources.file_source.shell = /bin/bash -c 9 10yinzhengjie2.sinks.hdfs_sink.type = hdfs 11yinzhengjie2.sinks.hdfs_sink.hdfs.path = hdfs://node101.yinzhengjie.org.cn:8020/flume/%Y%m%d/%H 12#上传文件的前缀 13yinzhengjie2.sinks.hdfs_sink.hdfs.filePrefix = 172.30.1.105- 14#是否按照时间滚动文件夹 15yinzhengjie2.sinks.hdfs_sink.hdfs.round = true 16#多少时间单位创建一个新的文件夹 17yinzhengjie2.sinks.hdfs_sink.hdfs.roundValue = 1 18#重新定义时间单位 19yinzhengjie2.sinks.hdfs_sink.hdfs.roundUnit = hour 20#是否使用本地时间戳 21yinzhengjie2.sinks.hdfs_sink.hdfs.useLocalTimeStamp = true 22#积攒多少个Event才flush到HDFS一次 23yinzhengjie2.sinks.hdfs_sink.hdfs.batchSize = 1000 24#设置文件类型，可支持压缩 25yinzhengjie2.sinks.hdfs_sink.hdfs.fileType = DataStream 26#多久生成一个新的文件 27yinzhengjie2.sinks.hdfs_sink.hdfs.rollInterval = 600 28#设置每个文件的滚动大小 29yinzhengjie2.sinks.hdfs_sink.hdfs.rollSize = 134217700 30#文件的滚动与Event数量无关 31yinzhengjie2.sinks.hdfs_sink.hdfs.rollCount = 0 32#最小副本数 33yinzhengjie2.sinks.hdfs_sink.hdfs.minBlockReplicas = 1 34 35yinzhengjie2.channels.memory_channel.type = memory 36yinzhengjie2.channels.memory_channel.capacity = 1000 37yinzhengjie2.channels.memory_channel.transactionCapacity = 1000 38 39yinzhengjie2.sources.file_source.channels = memory_channel 40yinzhengjie2.sinks.hdfs_sink.channel = memory_channel 41[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/job/flume-hdfs.conf #编写配置文件
1[root@node105.yinzhengjie.org.cn ~]# ss -ntl 2State Recv-Q Send-Q Local Address:Port Peer Address:Port 3LISTEN 0 128 *:22 *:* 4LISTEN 0 128 :::22 :::* 5[root@node105.yinzhengjie.org.cn ~]# 6[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/shell/start-hdfs.sh 7#!/bin/bash 8#@author :yinzhengjie 9#blog:http://www.cnblogs.com/yinzhengjie 10#EMAIL:y1053419035@qq.com 11#Data:Thu Oct 18 11:26:06 CST 2018 12 13#将监控数据发送给ganglia，需要指定ganglia服务器地址，使用请确认是否部署好ganglia服务！ 14#nohup flume-ng agent -c /home/softwares/apache-flume-1.9.0-bin/conf --conf-file=/home/data/flume/job/flume-hdfs.conf --name yinzhengjie2 -Dflume.monitoring.type=ganglia -Dflume.monitoring.hosts=node105.yinzhengjie.org.cn:864 159 -Dflume.root.logger=INFO,console >> /home/data/flume/log/flume-ganglia-flume-hdfs.log 2>&1 & 16 17#启动flume自身的监控参数，默认执行以下脚本 18nohup flume-ng agent -c /home/data/flume/job --conf-file=/home/data/flume/job/flume-hdfs.conf --name yinzhengjie2 -Dflume.monitoring.type=http -Dflume.monitoring.port=10502 -Dflume.root.logger=INFO,console >> /home/data/flu 19me/log/flume-hdfs.log 2>&1 &[root@node105.yinzhengjie.org.cn ~]# 20[root@node105.yinzhengjie.org.cn ~]# 21[root@node105.yinzhengjie.org.cn ~]# 22[root@node105.yinzhengjie.org.cn ~]# chmod +x /home/data/flume/shell/start-hdfs.sh 23[root@node105.yinzhengjie.org.cn ~]# 24[root@node105.yinzhengjie.org.cn ~]# /home/data/flume/shell/start-hdfs.sh 25[root@node105.yinzhengjie.org.cn ~]# 26[root@node105.yinzhengjie.org.cn ~]# ss -ntl 27State Recv-Q Send-Q Local Address:Port Peer Address:Port 28LISTEN 0 128 *:22 *:* 29LISTEN 0 50 *:10502 *:* 30LISTEN 0 128 :::22 :::* 31[root@node105.yinzhengjie.org.cn ~]# 32[root@node105.yinzhengjie.org.cn ~]# jps 335643 Application 345757 Jps 35[root@node105.yinzhengjie.org.cn ~]# 36[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/shell/start-hdfs.sh #编写启动脚本并启动flume
1[root@node105.yinzhengjie.org.cn ~]# tail -100f /home/data/flume/log/flume-hdfs.log 2Warning: JAVA_HOME is not set! 3Info: Including Hive libraries found via () for Hive access 4+ exec /home/softwares/jdk1.8.0_201/bin/java -Xmx20m -Dflume.monitoring.type=http -Dflume.monitoring.port=10502 -Dflume.root.logger=INFO,console -cp '/home/data/flume/job:/home/softwares/apache-flume-1.9.0-bin/lib/*:/lib/*' - 5Djava.library.path= org.apache.flume.node.Application --conf-file=/home/data/flume/job/flume-hdfs.conf --name yinzhengjie2log4j:WARN No appenders could be found for logger (org.apache.flume.util.SSLUtil). 6log4j:WARN Please initialize the log4j system properly. 7log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. 8Warning: JAVA_HOME is not set! 9Info: Including Hadoop libraries found via (/home/softwares/hadoop-2.6.0/bin/hadoop) for HDFS access 10Info: Including Hive libraries found via () for Hive access 11+ exec /home/softwares/jdk1.8.0_201/bin/java -Xmx20m -Dflume.monitoring.type=http -Dflume.monitoring.port=10502 -Dflume.root.logger=INFO,console -cp '/home/data/flume/job:/home/softwares/apache-flume-1.9.0-bin/lib/*:/home/sof 12twares/hadoop-2.6.0/etc/hadoop:/home/softwares/hadoop-2.6.0/share/hadoop/common/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/common/*:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs/*:/home/softwares/hadoop-2.6.0/share/hadoop/yarn/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/yarn/*:/home/softwares/hadoop-2.6.0/share/hadoop/mapreduce/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar:/lib/*' -Djava.library.path=:/home/softwares/hadoop-2.6.0/lib/native org.apache.flume.node.Application --conf-file=/home/data/flume/job/flume-hdfs.conf --name yinzhengjie2SLF4J: Class path contains multiple SLF4J bindings. 13SLF4J: Found binding in [jar:file:/home/softwares/apache-flume-1.9.0-bin/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] 14SLF4J: Found binding in [jar:file:/home/softwares/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] 15SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 16SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 1719/07/19 17:27:18 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 1819/07/19 17:27:18 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/home/data/flume/job/flume-hdfs.conf 1919/07/19 17:27:18 INFO conf.FlumeConfiguration: Processing:memory_channel 2019/07/19 17:27:18 INFO conf.FlumeConfiguration: Processing:file_source 2119/07/19 17:27:18 INFO conf.FlumeConfiguration: Processing:hdfs_sink 2219/07/19 17:27:18 INFO conf.FlumeConfiguration: Processing:hdfs_sink 2319/07/19 17:27:18 INFO conf.FlumeConfiguration: Processing:hdfs_sink 2419/07/19 17:27:18 INFO conf.FlumeConfiguration: Processing:hdfs_sink 2519/07/19 17:27:18 INFO conf.FlumeConfiguration: Processing:hdfs_sink 2619/07/19 17:27:18 INFO conf.FlumeConfiguration: Processing:memory_channel 2719/07/19 17:27:18 INFO conf.FlumeConfiguration: Processing:hdfs_sink 2819/07/19 17:27:18 INFO conf.FlumeConfiguration: Processing:file_source 2919/07/19 17:27:18 INFO conf.FlumeConfiguration: Processing:hdfs_sink 3019/07/19 17:27:18 INFO conf.FlumeConfiguration: Processing:hdfs_sink 3119/07/19 17:27:18 INFO conf.FlumeConfiguration: Processing:hdfs_sink 3219/07/19 17:27:18 INFO conf.FlumeConfiguration: Processing:file_source 3319/07/19 17:27:18 INFO conf.FlumeConfiguration: Processing:file_source 3419/07/19 17:27:18 INFO conf.FlumeConfiguration: Processing:memory_channel 3519/07/19 17:27:18 INFO conf.FlumeConfiguration: Processing:hdfs_sink 3619/07/19 17:27:18 INFO conf.FlumeConfiguration: Processing:hdfs_sink 3719/07/19 17:27:18 INFO conf.FlumeConfiguration: Processing:hdfs_sink 3819/07/19 17:27:18 INFO conf.FlumeConfiguration: Processing:hdfs_sink 3919/07/19 17:27:18 INFO conf.FlumeConfiguration: Processing:hdfs_sink 4019/07/19 17:27:18 INFO conf.FlumeConfiguration: Added sinks: hdfs_sink Agent: yinzhengjie2 4119/07/19 17:27:18 WARN conf.FlumeConfiguration: Agent configuration for 'yinzhengjie2' has no configfilters. 4219/07/19 17:27:18 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [yinzhengjie2] 4319/07/19 17:27:18 INFO node.AbstractConfigurationProvider: Creating channels 4419/07/19 17:27:18 INFO channel.DefaultChannelFactory: Creating instance of channel memory_channel type memory 4519/07/19 17:27:18 INFO node.AbstractConfigurationProvider: Created channel memory_channel 4619/07/19 17:27:18 INFO source.DefaultSourceFactory: Creating instance of source file_source, type exec 4719/07/19 17:27:18 INFO sink.DefaultSinkFactory: Creating instance of sink: hdfs_sink, type: hdfs 4819/07/19 17:27:18 INFO node.AbstractConfigurationProvider: Channel memory_channel connected to [file_source, hdfs_sink] 4919/07/19 17:27:18 INFO node.Application: Starting new configuration:{ sourceRunners:{file_source=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource{name:file_source,state:IDLE} }} sinkRunners:{hdfs_sink=Sink 50Runner: { policy:org.apache.flume.sink.DefaultSinkProcessor@331a821d counterGroup:{ name:null counters:{} } }} channels:{memory_channel=org.apache.flume.channel.MemoryChannel{name: memory_channel}} }19/07/19 17:27:18 INFO node.Application: Starting Channel memory_channel 5119/07/19 17:27:18 INFO node.Application: Waiting for channel: memory_channel to start. Sleeping for 500 ms 5219/07/19 17:27:18 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: memory_channel: Successfully registered new MBean. 5319/07/19 17:27:18 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: memory_channel started 5419/07/19 17:27:19 INFO node.Application: Starting Sink hdfs_sink 5519/07/19 17:27:19 INFO node.Application: Starting Source file_source 5619/07/19 17:27:19 INFO source.ExecSource: Exec source starting with command: tail -F /var/log/messages 5719/07/19 17:27:19 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: file_source: Successfully registered new MBean. 5819/07/19 17:27:19 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: file_source started 5919/07/19 17:27:19 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: hdfs_sink: Successfully registered new MBean. 6019/07/19 17:27:19 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: hdfs_sink started 6119/07/19 17:27:19 INFO util.log: Logging initialized @1347ms to org.eclipse.jetty.util.log.Slf4jLog 6219/07/19 17:27:19 INFO server.Server: jetty-9.4.6.v20170531 6319/07/19 17:27:19 INFO server.AbstractConnector: Started ServerConnector@3ab21218{HTTP/1.1,[http/1.1]}{0.0.0.0:10502} 6419/07/19 17:27:19 INFO server.Server: Started @1695ms 6519/07/19 17:27:23 INFO hdfs.HDFSDataStream: Serializer = TEXT, UseRawLocalFileSystem = false 6619/07/19 17:27:23 INFO hdfs.BucketWriter: Creating hdfs://node101.yinzhengjie.org.cn:8020/flume/20190719/17/172.30.1.105-.1563528443487.tmp 67 ^C 68[root@node105.yinzhengjie.org.cn ~]# 69[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# tail -100f /home/data/flume/log/flume-hdfs.log #查看flume日志收集信息
1[root@node105.yinzhengjie.org.cn ~]# hdfs dfs -ls /flume/20190719/17 2Found 1 items 3-rw-r--r-- 3 root supergroup 815 2019-07-19 17:27 /flume/20190719/17/172.30.1.105-.1563528443487.tmp 4[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# hdfs dfs -ls /flume/20190719/17 #查看hdfs对应目录是否生成相应的日志信息
1[root@node105.yinzhengjie.org.cn ~]# curl http://node105.yinzhengjie.org.cn:10502/metrics | jq 2 % Total % Received % Xferd Average Speed Time Time Time Current 3 Dload Upload Total Spent Left Speed 4100 951 0 951 0 0 72739 0 --:--:-- --:--:-- --:--:-- 73153 5{ 6 "SOURCE.file_source": { 7 "AppendBatchAcceptedCount": "0", #成功提交到channel的批次的总数量。 8 "GenericProcessingFail": "0", #常规处理失败的次数 9 "EventAcceptedCount": "9", #成功写出到channel的事件总数量。 10 "AppendReceivedCount": "0", #每批只有一个事件的事件总数量（与RPC调用的一个append调用相等）。 11 "StartTime": "1563528439426", #SOURCE启动时的毫秒值时间。 12 "AppendBatchReceivedCount": "0", #接收到事件批次的总数量。 13 "ChannelWriteFail": "0", #往CHANNEL写失败的次数 14 "EventReceivedCount": "9", #目前为止source已经接收到的事件总数量。 15 "EventReadFail": "0", #时间读取失败的次数 16 "Type": "SOURCE", #当前类型为SOURRCE 17 "AppendAcceptedCount": "0", #逐条录入的次数，单独传入的事件到Channel且成功返回的事件总数量。 18 "OpenConnectionCount": "0", #目前与客户端或sink保持连接的总数量，目前仅支持avro source展现该度量。 19 "StopTime": "0" #SOURCE停止时的毫秒值时间，0代表一直运行着 20 }, 21 "CHANNEL.memory_channel": { 22 "ChannelCapacity": "1000", #channel的容量，目前仅支持File Channel，Memory channel的统计数据。 23 "ChannelFillPercentage": "0.0", #channel已填入的百分比。 24 "Type": "CHANNEL", #当前类型为CHANNEL 25 "ChannelSize": "0", #目前channel中事件的总数量，目前仅支持File Channel，Memory channel的统计数据。 26 "EventTakeSuccessCount": "9", #sink成功从channel读取事件的总数量。 27 "EventTakeAttemptCount": "36", #sink尝试从channel拉取事件的总次数。这不意味着每次时间都被返回，因为sink拉取的时候channel可能没有任何数据。 28 "StartTime": "1563528438997", #CHANNEL启动时的毫秒值时间。 29 "EventPutAttemptCount": "9", #Source尝试写入Channe的事件总次数。 30 "EventPutSuccessCount": "9", #成功写入channel且提交的事件总次数。 31 "StopTime": "0" #CHANNEL停止时的毫秒值时间。 32 }, 33 "SINK.hdfs_sink": { 34 "ConnectionCreatedCount": "1", #下一个阶段（或存储系统）创建链接的数量（如HDFS创建一个文件）。 35 "BatchCompleteCount": "0", #批量处理event的个数等于批处理大小的数量。 36 "EventWriteFail": "0", #时间写失败的次数 37 "BatchEmptyCount": "26", #批量处理event的个数为0的数量（空的批量的数量），如果数量很大表示source写入数据的速度比sink处理数据的速度慢很多。 38 "EventDrainAttemptCount": "9", #sink尝试写出到存储的事件总数量。 39 "StartTime": "1563528439448", #SINK启动时的毫秒值时间。 40 "BatchUnderflowCount": "1", #批量处理event的个数小于批处理大小的数量（比sink配置使用的最大批量尺寸更小的批量的数量），如果该值很高也表示sink比source更快。 41 "ChannelReadFail": "0", #从CHANNEL读取失败的次数 42 "ConnectionFailedCount": "0", #连接失败的次数 43 "ConnectionClosedCount": "0", #连接关闭的次数 44 "Type": "SINK", #当前类型为SINK 45 "EventDrainSuccessCount": "9", #sink成功写出到存储的事件总数量。 46 "StopTime": "0" #SINK停止时的毫秒值时间。 47 } 48} 49[root@node105.yinzhengjie.org.cn ~]# 50[root@node105.yinzhengjie.org.cn ~]#
3>.实时指定目录文件内容到HDFS集群（需要flume节点配置hadoop集群环境哟，spooldir source - memory channel - hdfs sink）****
1[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/job/flume-dir.conf 2yinzhengjie3.sources = spooldir_source 3yinzhengjie3.sinks = hdfs_sink 4yinzhengjie3.channels = memory_channel 5 6# Describe/configure the source 7yinzhengjie3.sources.spooldir_source.type = spooldir 8yinzhengjie3.sources.spooldir_source.spoolDir = /yinzhengjie/data/flume/upload 9yinzhengjie3.sources.spooldir_source.fileSuffix = .COMPLETED 10yinzhengjie3.sources.spooldir_source.fileHeader = true 11#忽略所有以.tmp结尾的文件，不上传 12yinzhengjie3.sources.spooldir_source.ignorePattern = ([^ ]*\.tmp) 13#获取源文件名称，方便下面的sink调用变量fileName 14yinzhengjie3.sources.spooldir_source.basenameHeader = true 15yinzhengjie3.sources.spooldir_source.basenameHeaderKey = fileName 16 17 18# Describe the sink 19yinzhengjie3.sinks.hdfs_sink.type = hdfs 20yinzhengjie3.sinks.hdfs_sink.hdfs.path = hdfs://node101.yinzhengjie.org.cn:8020/flume 21#上传文件的前缀 22yinzhengjie3.sinks.hdfs_sink.hdfs.filePrefix = 172.30.1.105-upload- 23#是否按照时间滚动文件夹 24yinzhengjie3.sinks.hdfs_sink.hdfs.round = true 25#多少时间单位创建一个新的文件夹 26yinzhengjie3.sinks.hdfs_sink.hdfs.roundValue = 1 27#重新定义时间单位 28yinzhengjie3.sinks.hdfs_sink.hdfs.roundUnit = hour 29#是否使用本地时间戳 30yinzhengjie3.sinks.hdfs_sink.hdfs.useLocalTimeStamp = true 31#积攒多少个Event才flush到HDFS一次 32yinzhengjie3.sinks.hdfs_sink.hdfs.batchSize = 100 33#设置文件类型，可支持压缩 34yinzhengjie3.sinks.hdfs_sink.hdfs.fileType = DataStream 35#多久生成一个新的文件 36yinzhengjie3.sinks.hdfs_sink.hdfs.rollInterval = 60 37#设置每个文件的滚动大小大概是128M 38yinzhengjie3.sinks.hdfs_sink.hdfs.rollSize = 134217700 39#文件的滚动与Event数量无关 40yinzhengjie3.sinks.hdfs_sink.hdfs.rollCount = 0 41#最小冗余数 42yinzhengjie3.sinks.hdfs_sink.hdfs.minBlockReplicas = 1 43#和source的basenameHeader，basenameHeaderKey两个属性一起用可以保持原文件名称上传 44yinzhengjie3.sinks.hdfs_sink.hdfs.filePrefix = %{fileName} 45 46# Use a channel which buffers events in memory 47yinzhengjie3.channels.memory_channel.type = memory 48yinzhengjie3.channels.memory_channel.capacity = 1000 49yinzhengjie3.channels.memory_channel.transactionCapacity = 1000 50 51# Bind the source and sink to the channel 52yinzhengjie3.sources.spooldir_source.channels = memory_channel 53yinzhengjie3.sinks.hdfs_sink.channel = memory_channel 54[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/job/flume-dir.conf #编写flume配置文件
1[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/shell/start-dir.sh 2#!/bin/bash 3#@author :yinzhengjie 4#blog:http://www.cnblogs.com/yinzhengjie 5#EMAIL:y1053419035@qq.com 6#Data:Thu Oct 18 11:26:06 CST 2018 7 8#将监控数据发送给ganglia，需要指定ganglia服务器地址，使用请确认是否部署好ganglia服务！ 9#nohup flume-ng agent -c /home/data/flume/job --conf-file=/home/data/flume/job/flume-dir.conf --name yinzhengjie3 -Dflume.monitoring.type=ganglia -Dflume.monitoring.hosts=node105.yinzhengjie.org.cn:8649 -Dflume.root.logger=INFO,console >> /home/data/flume/log/flume-gang 10lia-flume-dir.log 2>&1 & 11 12#启动flume自身的监控参数，默认执行以下脚本 13nohup flume-ng agent -c /home/data/flume/job --conf-file=/home/data/flume/job/flume-dir.conf --name yinzhengjie3 -Dflume.monitoring.type=http -Dflume.monitoring.port=10503 -Dflume.root.logger=INFO,console >> /home/data/flume/log/flume-dir.log 2>&1 & 14[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/shell/start-dir.sh #编写启动脚本
1[root@node105.yinzhengjie.org.cn ~]# mkdir -pv /yinzhengjie/data/flume/upload 2mkdir: created directory ‘/yinzhengjie’ 3mkdir: created directory ‘/yinzhengjie/data’ 4mkdir: created directory ‘/yinzhengjie/data/flume’ 5mkdir: created directory ‘/yinzhengjie/data/flume/upload’ 6[root@node105.yinzhengjie.org.cn ~]# 7[root@node105.yinzhengjie.org.cn ~]# echo http://www.cnblogs.com/yinzhengjie>/yinzhengjie/data/flume/upload/yinzhengjie.blog 8[root@node105.yinzhengjie.org.cn ~]# 9[root@node105.yinzhengjie.org.cn ~]# echo http://www.cnblogs.com/yinzhengjie>/yinzhengjie/data/flume/upload/yinzhengjie2.tmp 10[root@node105.yinzhengjie.org.cn ~]# 11[root@node105.yinzhengjie.org.cn ~]# echo http://www.cnblogs.com/yinzhengjie>/yinzhengjie/data/flume/upload/yinzhengjie3.txt 12[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# mkdir -pv /yinzhengjie/data/flume/upload #创建测试数据
1[root@node105.yinzhengjie.org.cn ~]# ss -ntl 2State Recv-Q Send-Q Local Address:Port Peer Address:Port 3LISTEN 0 128 *:22 *:* 4LISTEN 0 128 :::22 :::* 5[root@node105.yinzhengjie.org.cn ~]# 6[root@node105.yinzhengjie.org.cn ~]# /home/data/flume/shell/start-dir.sh 7[root@node105.yinzhengjie.org.cn ~]# 8[root@node105.yinzhengjie.org.cn ~]# ss -ntl 9State Recv-Q Send-Q Local Address:Port Peer Address:Port 10LISTEN 0 50 *:10503 *:* 11LISTEN 0 128 *:22 *:* 12LISTEN 0 128 :::22 :::* 13[root@node105.yinzhengjie.org.cn ~]# 14[root@node105.yinzhengjie.org.cn ~]# tail -100f /home/data/flume/log/flume-dir.log 15Warning: JAVA_HOME is not set! 16Info: Including Hadoop libraries found via (/home/softwares/hadoop-2.6.0/bin/hadoop) for HDFS access 17Info: Including Hive libraries found via () for Hive access 18+ exec /home/softwares/jdk1.8.0_201/bin/java -Xmx20m -Dflume.monitoring.type=http -Dflume.monitoring.port=10503 -Dflume.root.logger=INFO,console -cp '/home/data/flume/job:/home/softwares/apache-flume-1.9.0-bin/lib/*:/home/softwares/hadoop-2.6.0/etc/hadoop:/home/software 19s/hadoop-2.6.0/share/hadoop/common/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/common/*:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs/*:/home/softwares/hadoop-2.6.0/share/hadoop/yarn/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/yarn/*:/home/softwares/hadoop-2.6.0/share/hadoop/mapreduce/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar:/lib/*' -Djava.library.path=:/home/softwares/hadoop-2.6.0/lib/native org.apache.flume.node.Application --conf-file=/home/data/flume/job/flume-dir.conf --name yinzhengjie3SLF4J: Class path contains multiple SLF4J bindings. 20SLF4J: Found binding in [jar:file:/home/softwares/apache-flume-1.9.0-bin/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] 21SLF4J: Found binding in [jar:file:/home/softwares/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] 22SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 23SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2419/07/19 18:28:43 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 2519/07/19 18:28:43 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/home/data/flume/job/flume-dir.conf 2619/07/19 18:28:43 INFO conf.FlumeConfiguration: Processing:hdfs_sink 2719/07/19 18:28:43 INFO conf.FlumeConfiguration: Processing:memory_channel 2819/07/19 18:28:43 INFO conf.FlumeConfiguration: Processing:hdfs_sink 2919/07/19 18:28:43 INFO conf.FlumeConfiguration: Processing:spooldir_source 3019/07/19 18:28:43 INFO conf.FlumeConfiguration: Processing:spooldir_source 3119/07/19 18:28:43 INFO conf.FlumeConfiguration: Processing:spooldir_source 3219/07/19 18:28:43 INFO conf.FlumeConfiguration: Processing:memory_channel 3319/07/19 18:28:43 INFO conf.FlumeConfiguration: Processing:hdfs_sink 3419/07/19 18:28:43 INFO conf.FlumeConfiguration: Processing:hdfs_sink 3519/07/19 18:28:43 INFO conf.FlumeConfiguration: Processing:spooldir_source 3619/07/19 18:28:43 INFO conf.FlumeConfiguration: Processing:hdfs_sink 3719/07/19 18:28:43 INFO conf.FlumeConfiguration: Processing:hdfs_sink 3819/07/19 18:28:43 INFO conf.FlumeConfiguration: Processing:hdfs_sink 3919/07/19 18:28:43 INFO conf.FlumeConfiguration: Processing:hdfs_sink 4019/07/19 18:28:43 INFO conf.FlumeConfiguration: Added sinks: hdfs_sink Agent: yinzhengjie3 4119/07/19 18:28:43 INFO conf.FlumeConfiguration: Processing:spooldir_source 4219/07/19 18:28:43 INFO conf.FlumeConfiguration: Processing:spooldir_source 4319/07/19 18:28:43 INFO conf.FlumeConfiguration: Processing:hdfs_sink 4419/07/19 18:28:43 INFO conf.FlumeConfiguration: Processing:hdfs_sink 4519/07/19 18:28:43 INFO conf.FlumeConfiguration: Processing:hdfs_sink 4619/07/19 18:28:43 INFO conf.FlumeConfiguration: Processing:memory_channel 4719/07/19 18:28:43 INFO conf.FlumeConfiguration: Processing:hdfs_sink 4819/07/19 18:28:43 INFO conf.FlumeConfiguration: Processing:spooldir_source 4919/07/19 18:28:43 INFO conf.FlumeConfiguration: Processing:spooldir_source 5019/07/19 18:28:43 INFO conf.FlumeConfiguration: Processing:hdfs_sink 5119/07/19 18:28:43 INFO conf.FlumeConfiguration: Processing:hdfs_sink 5219/07/19 18:28:43 WARN conf.FlumeConfiguration: Agent configuration for 'yinzhengjie3' has no configfilters. 5319/07/19 18:28:43 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [yinzhengjie3] 5419/07/19 18:28:43 INFO node.AbstractConfigurationProvider: Creating channels 5519/07/19 18:28:43 INFO channel.DefaultChannelFactory: Creating instance of channel memory_channel type memory 5619/07/19 18:28:43 INFO node.AbstractConfigurationProvider: Created channel memory_channel 5719/07/19 18:28:43 INFO source.DefaultSourceFactory: Creating instance of source spooldir_source, type spooldir 5819/07/19 18:28:43 INFO sink.DefaultSinkFactory: Creating instance of sink: hdfs_sink, type: hdfs 5919/07/19 18:28:43 INFO node.AbstractConfigurationProvider: Channel memory_channel connected to [spooldir_source, hdfs_sink] 6019/07/19 18:28:43 INFO node.Application: Starting new configuration:{ sourceRunners:{spooldir_source=EventDrivenSourceRunner: { source:Spool Directory source spooldir_source: { spoolDir: /yinzhengjie/data/flume/upload } }} sinkRunners:{hdfs_sink=SinkRunner: { policy:org 61.apache.flume.sink.DefaultSinkProcessor@440e91df counterGroup:{ name:null counters:{} } }} channels:{memory_channel=org.apache.flume.channel.MemoryChannel{name: memory_channel}} }19/07/19 18:28:43 INFO node.Application: Starting Channel memory_channel 6219/07/19 18:28:43 INFO node.Application: Waiting for channel: memory_channel to start. Sleeping for 500 ms 6319/07/19 18:28:43 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: memory_channel: Successfully registered new MBean. 6419/07/19 18:28:43 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: memory_channel started 6519/07/19 18:28:43 INFO node.Application: Starting Sink hdfs_sink 6619/07/19 18:28:43 INFO node.Application: Starting Source spooldir_source 6719/07/19 18:28:43 INFO source.SpoolDirectorySource: SpoolDirectorySource source starting with directory: /yinzhengjie/data/flume/upload 6819/07/19 18:28:43 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: hdfs_sink: Successfully registered new MBean. 6919/07/19 18:28:43 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: hdfs_sink started 7019/07/19 18:28:43 INFO util.log: Logging initialized @1358ms to org.eclipse.jetty.util.log.Slf4jLog 7119/07/19 18:28:43 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: spooldir_source: Successfully registered new MBean. 7219/07/19 18:28:43 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: spooldir_source started 7319/07/19 18:28:44 INFO server.Server: jetty-9.4.6.v20170531 7419/07/19 18:28:44 INFO server.AbstractConnector: Started ServerConnector@1d367324{HTTP/1.1,[http/1.1]}{0.0.0.0:10503} 7519/07/19 18:28:44 INFO server.Server: Started @1609ms 7619/07/19 18:29:16 INFO avro.ReliableSpoolingFileEventReader: Last read took us just up to a file boundary. Rolling to the next file, if there is one. 7719/07/19 18:29:16 INFO avro.ReliableSpoolingFileEventReader: Preparing to move file /yinzhengjie/data/flume/upload/yinzhengjie.blog to /yinzhengjie/data/flume/upload/yinzhengjie.blog.COMPLETED 7819/07/19 18:29:16 INFO hdfs.HDFSDataStream: Serializer = TEXT, UseRawLocalFileSystem = false 7919/07/19 18:29:16 INFO hdfs.BucketWriter: Creating hdfs://node101.yinzhengjie.org.cn:8020/flume/yinzhengjie.blog.1563532156286.tmp 8019/07/19 18:30:17 INFO hdfs.HDFSEventSink: Writer callback called. 8119/07/19 18:30:17 INFO hdfs.BucketWriter: Closing hdfs://node101.yinzhengjie.org.cn:8020/flume/yinzhengjie.blog.1563532156286.tmp 8219/07/19 18:30:18 INFO hdfs.BucketWriter: Renaming hdfs://node101.yinzhengjie.org.cn:8020/flume/yinzhengjie.blog.1563532156286.tmp to hdfs://node101.yinzhengjie.org.cn:8020/flume/yinzhengjie.blog.1563532156286 8319/07/19 18:31:03 INFO avro.ReliableSpoolingFileEventReader: Last read took us just up to a file boundary. Rolling to the next file, if there is one. 8419/07/19 18:31:03 INFO avro.ReliableSpoolingFileEventReader: Preparing to move file /yinzhengjie/data/flume/upload/yinzhengjie3.txt to /yinzhengjie/data/flume/upload/yinzhengjie3.txt.COMPLETED 8519/07/19 18:31:03 INFO hdfs.HDFSDataStream: Serializer = TEXT, UseRawLocalFileSystem = false 8619/07/19 18:31:03 INFO hdfs.BucketWriter: Creating hdfs://node101.yinzhengjie.org.cn:8020/flume/yinzhengjie3.txt.1563532263271.tmp 8719/07/19 18:32:03 INFO hdfs.HDFSEventSink: Writer callback called. 8819/07/19 18:32:03 INFO hdfs.BucketWriter: Closing hdfs://node101.yinzhengjie.org.cn:8020/flume/yinzhengjie3.txt.1563532263271.tmp 8919/07/19 18:32:03 INFO hdfs.BucketWriter: Renaming hdfs://node101.yinzhengjie.org.cn:8020/flume/yinzhengjie3.txt.1563532263271.tmp to hdfs://node101.yinzhengjie.org.cn:8020/flume/yinzhengjie3.txt.1563532263271
[root@node105.yinzhengjie.org.cn ~]# tail -100f /home/data/flume/log/flume-dir.log #启动脚本并查看对应的日志信息
1[root@node105.yinzhengjie.org.cn ~]# 2[root@node105.yinzhengjie.org.cn ~]# ll /yinzhengjie/data/flume/upload/ 3total 12 4-rw-r--r-- 1 root root 35 Jul 19 18:30 yinzhengjie2.tmp 5-rw-r--r-- 1 root root 35 Jul 19 18:31 yinzhengjie3.txt.COMPLETED 6-rw-r--r-- 1 root root 35 Jul 19 18:29 yinzhengjie.blog.COMPLETED 7[root@node105.yinzhengjie.org.cn ~]# 8[root@node105.yinzhengjie.org.cn ~]# 9[root@node105.yinzhengjie.org.cn ~]# hdfs dfs -ls /flume 10Found 3 items 11drwxr-xr-x - root supergroup 0 2019-07-19 18:01 /flume/20190719 12-rw-r--r-- 3 root supergroup 35 2019-07-19 18:30 /flume/yinzhengjie.blog.1563532156286 13-rw-r--r-- 3 root supergroup 35 2019-07-19 18:32 /flume/yinzhengjie3.txt.1563532263271 14[root@node105.yinzhengjie.org.cn ~]#
1[root@node105.yinzhengjie.org.cn ~]# curl http://node105.yinzhengjie.org.cn:10503/metrics | jq 2 % Total % Received % Xferd Average Speed Time Time Time Current 3 Dload Upload Total Spent Left Speed 4100 955 0 955 0 0 4936 0 --:--:-- --:--:-- --:--:-- 4948 5{ 6 "CHANNEL.memory_channel": { 7 "ChannelCapacity": "1000", 8 "ChannelFillPercentage": "0.0", 9 "Type": "CHANNEL", 10 "ChannelSize": "0", 11 "EventTakeSuccessCount": "2", 12 "EventTakeAttemptCount": "72", 13 "StartTime": "1563532123413", 14 "EventPutAttemptCount": "2", 15 "EventPutSuccessCount": "2", 16 "StopTime": "0" 17 }, 18 "SOURCE.spooldir_source": { 19 "AppendBatchAcceptedCount": "2", 20 "GenericProcessingFail": "0", 21 "EventAcceptedCount": "2", 22 "AppendReceivedCount": "0", 23 "StartTime": "1563532123986", 24 "AppendBatchReceivedCount": "2", 25 "ChannelWriteFail": "0", 26 "EventReceivedCount": "2", 27 "EventReadFail": "0", 28 "Type": "SOURCE", 29 "AppendAcceptedCount": "0", 30 "OpenConnectionCount": "0", 31 "StopTime": "0" 32 }, 33 "SINK.hdfs_sink": { 34 "ConnectionCreatedCount": "2", 35 "BatchCompleteCount": "0", 36 "EventWriteFail": "0", 37 "BatchEmptyCount": "68", 38 "EventDrainAttemptCount": "2", 39 "StartTime": "1563532123890", 40 "BatchUnderflowCount": "2", 41 "ChannelReadFail": "0", 42 "ConnectionFailedCount": "0", 43 "ConnectionClosedCount": "2", 44 "Type": "SINK", 45 "EventDrainSuccessCount": "2", 46 "StopTime": "0" 47 } 48} 49[root@node105.yinzhengjie.org.cn ~]# 50[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# curl http://node105.yinzhengjie.org.cn:10503/metrics | jq
4>.Flume与Flume之间数据传递，多Flume汇总数据到单Flume**（需要flume节点配置hadoop集群环境哟，大致架构如下图所示）******
flume-1监控文件yinzhengjie.log，flume-2监控某一个端口的数据流，flume-1与flume-2将数据发送给flume-3，flume3将最终数据写入到HDFS。
1[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/job/flume-aggregation.conf 2# Name the components on this agent 3aggregation.sources = avro_source 4aggregation.sinks = hdfs_sink 5aggregation.channels = memory_channel 6 7# Describe/configure the source 8aggregation.sources.avro_source.type = avro 9aggregation.sources.avro_source.bind = node105.yinzhengjie.org.cn 10aggregation.sources.avro_source.port = 6666 11 12# Describe the sink 13aggregation.sinks.hdfs_sink.type = hdfs 14aggregation.sinks.hdfs_sink.hdfs.path = hdfs://node101.yinzhengjie.org.cn:8020/flume/%Y%m%d/%H 15#上传文件的前缀 16aggregation.sinks.hdfs_sink.hdfs.filePrefix = 172.30.1.105- 17#是否按照时间滚动文件夹 18aggregation.sinks.hdfs_sink.hdfs.round = true 19#多少时间单位创建一个新的文件夹 20aggregation.sinks.hdfs_sink.hdfs.roundValue = 1 21#重新定义时间单位 22aggregation.sinks.hdfs_sink.hdfs.roundUnit = hour 23#是否使用本地时间戳 24aggregation.sinks.hdfs_sink.hdfs.useLocalTimeStamp = true 25#积攒多少个Event才flush到HDFS一次 26aggregation.sinks.hdfs_sink.hdfs.batchSize = 100 27#设置文件类型，可支持压缩 28aggregation.sinks.hdfs_sink.hdfs.fileType = DataStream 29#多久生成一个新的文件 30aggregation.sinks.hdfs_sink.hdfs.rollInterval = 600 31#设置每个文件的滚动大小大概是128M 32aggregation.sinks.hdfs_sink.hdfs.rollSize = 134217700 33#文件的滚动与Event数量无关 34aggregation.sinks.hdfs_sink.hdfs.rollCount = 0 35#最小冗余数 36aggregation.sinks.hdfs_sink.hdfs.minBlockReplicas = 1 37 38# Describe the channel 39aggregation.channels.memory_channel.type = memory 40aggregation.channels.memory_channel.capacity = 1000 41aggregation.channels.memory_channel.transactionCapacity = 100 42 43# Bind the source and sink to the channel 44aggregation.sources.avro_source.channels = memory_channel 45aggregation.sinks.hdfs_sink.channel = memory_channel 46[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/job/flume-aggregation.conf #编写负责收集数据到hdfs的配置文件
1[root@node105.yinzhengjie.org.cn ~]# vi /home/data/flume/shell/start-aggregation.sh 2[root@node105.yinzhengjie.org.cn ~]# 3[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/shell/start-aggregation.sh 4#!/bin/bash 5#@author :yinzhengjie 6#blog:http://www.cnblogs.com/yinzhengjie 7#EMAIL:y1053419035@qq.com 8#Data:Thu Oct 18 11:26:06 CST 2018 9 10#将监控数据发送给ganglia，需要指定ganglia服务器地址，使用请确认是否部署好ganglia服务！ 11#nohup flume-ng agent -c /home/data/flume/job --conf-file=/home/data/flume/job/flume-aggregation.conf --name aggregation -Dflume.monitoring.type=ganglia -Dflume.monitoring.hosts=node105.yinzhengjie.org.cn:8649 -Dflume.root.logger=INFO,console >> /home/data/flume/log/flu 12me-ganglia-flume-aggregation.log 2>&1 & 13 14#启动flume自身的监控参数，默认执行以下脚本 15nohup flume-ng agent -c /home/data/flume/job --conf-file=/home/data/flume/job/flume-aggregation.conf --name aggregation -Dflume.monitoring.type=http -Dflume.monitoring.port=10511 -Dflume.root.logger=INFO,console >> /home/data/flume/log/flume-aggregation.log 2>&1 & 16[root@node105.yinzhengjie.org.cn ~]# 17[root@node105.yinzhengjie.org.cn ~]# 18[root@node105.yinzhengjie.org.cn ~]# chmod +x /home/data/flume/shell/start-aggregation.sh 19[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/shell/start-aggregation.sh #编写相应的启动脚本
1[root@node105.yinzhengjie.org.cn ~]# /home/data/flume/shell/start-aggregation.sh 2[root@node105.yinzhengjie.org.cn ~]# 3[root@node105.yinzhengjie.org.cn ~]# jps 48147 Application 58207 Jps 6[root@node105.yinzhengjie.org.cn ~]# 7[root@node105.yinzhengjie.org.cn ~]# 8[root@node105.yinzhengjie.org.cn ~]# ss -ntl 9State Recv-Q Send-Q Local Address:Port Peer Address:Port 10LISTEN 0 50 172.30.1.105:6666 *:* 11LISTEN 0 50 *:10511 *:* 12LISTEN 0 128 *:22 *:* 13LISTEN 0 128 :::22 :::* 14[root@node105.yinzhengjie.org.cn ~]# 15[root@node105.yinzhengjie.org.cn ~]# tail -1000f /home/data/flume/log/flume-aggregation.log 16Warning: JAVA_HOME is not set! 17Info: Including Hadoop libraries found via (/home/softwares/hadoop-2.6.0/bin/hadoop) for HDFS access 18Info: Including Hive libraries found via () for Hive access 19+ exec /home/softwares/jdk1.8.0_201/bin/java -Xmx20m -Dflume.monitoring.type=http -Dflume.monitoring.port=10511 -Dflume.root.logger=INFO,console -cp '/home/data/flume/job:/home/softwares/apache-flume-1.9.0-bin/lib/*:/home/softwares/hadoop-2.6.0/etc/hadoop:/home/software 20s/hadoop-2.6.0/share/hadoop/common/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/common/*:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs/*:/home/softwares/hadoop-2.6.0/share/hadoop/yarn/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/yarn/*:/home/softwares/hadoop-2.6.0/share/hadoop/mapreduce/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar:/lib/*' -Djava.library.path=:/home/softwares/hadoop-2.6.0/lib/native org.apache.flume.node.Application --conf-file=/home/data/flume/job/flume-aggregation.conf --name aggregationSLF4J: Class path contains multiple SLF4J bindings. 21SLF4J: Found binding in [jar:file:/home/softwares/apache-flume-1.9.0-bin/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] 22SLF4J: Found binding in [jar:file:/home/softwares/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] 23SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 24SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2519/07/20 21:18:38 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 2619/07/20 21:18:38 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/home/data/flume/job/flume-aggregation.conf 2719/07/20 21:18:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 2819/07/20 21:18:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 2919/07/20 21:18:38 INFO conf.FlumeConfiguration: Processing:memory_channel 3019/07/20 21:18:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 3119/07/20 21:18:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 3219/07/20 21:18:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 3319/07/20 21:18:38 INFO conf.FlumeConfiguration: Processing:memory_channel 3419/07/20 21:18:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 3519/07/20 21:18:38 INFO conf.FlumeConfiguration: Added sinks: hdfs_sink Agent: aggregation 3619/07/20 21:18:38 INFO conf.FlumeConfiguration: Processing:avro_source 3719/07/20 21:18:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 3819/07/20 21:18:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 3919/07/20 21:18:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 4019/07/20 21:18:38 INFO conf.FlumeConfiguration: Processing:avro_source 4119/07/20 21:18:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 4219/07/20 21:18:38 INFO conf.FlumeConfiguration: Processing:avro_source 4319/07/20 21:18:38 INFO conf.FlumeConfiguration: Processing:memory_channel 4419/07/20 21:18:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 4519/07/20 21:18:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 4619/07/20 21:18:38 INFO conf.FlumeConfiguration: Processing:avro_source 4719/07/20 21:18:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 4819/07/20 21:18:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 4919/07/20 21:18:38 WARN conf.FlumeConfiguration: Agent configuration for 'aggregation' has no configfilters. 5019/07/20 21:18:39 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [aggregation] 5119/07/20 21:18:39 INFO node.AbstractConfigurationProvider: Creating channels 5219/07/20 21:18:39 INFO channel.DefaultChannelFactory: Creating instance of channel memory_channel type memory 5319/07/20 21:18:39 INFO node.AbstractConfigurationProvider: Created channel memory_channel 5419/07/20 21:18:39 INFO source.DefaultSourceFactory: Creating instance of source avro_source, type avro 5519/07/20 21:18:39 INFO sink.DefaultSinkFactory: Creating instance of sink: hdfs_sink, type: hdfs 5619/07/20 21:18:39 INFO node.AbstractConfigurationProvider: Channel memory_channel connected to [avro_source, hdfs_sink] 5719/07/20 21:18:39 INFO node.Application: Starting new configuration:{ sourceRunners:{avro_source=EventDrivenSourceRunner: { source:Avro source avro_source: { bindAddress: node105.yinzhengjie.org.cn, port: 6666 } }} sinkRunners:{hdfs_sink=SinkRunner: { policy:org.apache. 58flume.sink.DefaultSinkProcessor@77a41618 counterGroup:{ name:null counters:{} } }} channels:{memory_channel=org.apache.flume.channel.MemoryChannel{name: memory_channel}} }19/07/20 21:18:39 INFO node.Application: Starting Channel memory_channel 5919/07/20 21:18:39 INFO node.Application: Waiting for channel: memory_channel to start. Sleeping for 500 ms 6019/07/20 21:18:39 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: memory_channel: Successfully registered new MBean. 6119/07/20 21:18:39 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: memory_channel started 6219/07/20 21:18:39 INFO node.Application: Starting Sink hdfs_sink 6319/07/20 21:18:39 INFO node.Application: Starting Source avro_source 6419/07/20 21:18:39 INFO source.AvroSource: Starting Avro source avro_source: { bindAddress: node105.yinzhengjie.org.cn, port: 6666 }... 6519/07/20 21:18:39 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: hdfs_sink: Successfully registered new MBean. 6619/07/20 21:18:39 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: hdfs_sink started 6719/07/20 21:18:39 INFO util.log: Logging initialized @1309ms to org.eclipse.jetty.util.log.Slf4jLog 6819/07/20 21:18:39 INFO server.Server: jetty-9.4.6.v20170531 6919/07/20 21:18:40 INFO server.AbstractConnector: Started ServerConnector@72e46d99{HTTP/1.1,[http/1.1]}{0.0.0.0:10511} 7019/07/20 21:18:40 INFO server.Server: Started @1811ms 7119/07/20 21:18:40 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: avro_source: Successfully registered new MBean. 7219/07/20 21:18:40 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: avro_source started 7319/07/20 21:18:40 INFO source.AvroSource: Avro source avro_source started.
[root@node105.yinzhengjie.org.cn ~]# tail -1000f /home/data/flume/log/flume-aggregation.log #启动脚本并观察日志信息
1[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/job/flume-my_netcat.conf 2# Name the components on this agent 3my_netcat.sources = netcat_source 4my_netcat.sinks = avro_sink 5my_netcat.channels = memory_channel 6 7# Describe/configure the source 8my_netcat.sources.netcat_source.type = netcat 9my_netcat.sources.netcat_source.bind = node105.yinzhengjie.org.cn 10my_netcat.sources.netcat_source.port = 8888 11 12# Describe the sink 13my_netcat.sinks.avro_sink.type = avro 14my_netcat.sinks.avro_sink.hostname = node105.yinzhengjie.org.cn 15my_netcat.sinks.avro_sink.port = 6666 16 17# Use a channel which buffers events in memory 18my_netcat.channels.memory_channel.type = memory 19my_netcat.channels.memory_channel.capacity = 1000 20my_netcat.channels.memory_channel.transactionCapacity = 100 21 22# Bind the source and sink to the channel 23my_netcat.sources.netcat_source.channels = memory_channel 24my_netcat.sinks.avro_sink.channel = memory_channel 25[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/job/flume-my_netcat.conf #编写my_netcat的启动脚本
1[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/shell/start-my_netcat.sh 2#!/bin/bash 3#@author :yinzhengjie 4#blog:http://www.cnblogs.com/yinzhengjie 5#EMAIL:y1053419035@qq.com 6#Data:Thu Oct 18 11:26:06 CST 2018 7 8#将监控数据发送给ganglia，需要指定ganglia服务器地址，使用请确认是否部署好ganglia服务！ 9#nohup flume-ng agent -c /home/data/flume/job --conf-file=/home/data/flume/job/flume-my_netcat.conf --name my_netcat -Dflume.monitoring.type=ganglia -Dflume.monitoring.hosts=node105.yinzhengjie.org.cn:8649 -Dflume.root.logger=INFO,console >> /home/data/flume/log/flume-g 10anglia-flume-my_netcat.log 2>&1 & 11 12#启动flume自身的监控参数，默认执行以下脚本 13nohup flume-ng agent -c /home/data/flume/job --conf-file=/home/data/flume/job/flume-my_netcat.conf --name my_netcat -Dflume.monitoring.type=http -Dflume.monitoring.port=10512 -Dflume.root.logger=INFO,console >> /home/data/flume/log/flume-my_netcat.log 2>&1 & 14[root@node105.yinzhengjie.org.cn ~]# 15[root@node105.yinzhengjie.org.cn ~]# 16[root@node105.yinzhengjie.org.cn ~]# chmod +x /home/data/flume/shell/start-my_netcat.sh 17[root@node105.yinzhengjie.org.cn ~]# 18[root@node105.yinzhengjie.org.cn ~]# 19[root@node105.yinzhengjie.org.cn ~]# ss -ntl 20State Recv-Q Send-Q Local Address:Port Peer Address:Port 21LISTEN 0 50 172.30.1.105:6666 *:* 22LISTEN 0 50 *:10511 *:* 23LISTEN 0 128 *:22 *:* 24LISTEN 0 128 :::22 :::* 25[root@node105.yinzhengjie.org.cn ~]# 26[root@node105.yinzhengjie.org.cn ~]# /home/data/flume/shell/start-my_netcat.sh 27[root@node105.yinzhengjie.org.cn ~]# 28[root@node105.yinzhengjie.org.cn ~]# ss -ntl 29State Recv-Q Send-Q Local Address:Port Peer Address:Port 30LISTEN 0 50 172.30.1.105:6666 *:* 31LISTEN 0 50 *:10511 *:* 32LISTEN 0 50 *:10512 *:* 33LISTEN 0 128 *:22 *:* 34LISTEN 0 50 172.30.1.105:8888 *:* 35LISTEN 0 128 :::22 :::* 36[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/shell/start-my_netcat.sh #编写相应的启动脚本并启动
1[root@node105.yinzhengjie.org.cn ~]# tail -100f /home/data/flume/log/flume-my_netcat.log 2Warning: JAVA_HOME is not set! 3Info: Including Hadoop libraries found via (/home/softwares/hadoop-2.6.0/bin/hadoop) for HDFS access 4Info: Including Hive libraries found via () for Hive access 5+ exec /home/softwares/jdk1.8.0_201/bin/java -Xmx20m -Dflume.monitoring.type=http -Dflume.monitoring.port=10512 -Dflume.root.logger=INFO,console -cp '/home/data/flume/job:/home/softwares/apache-flume-1.9.0-bin/lib/*:/home/softwares/hadoop-2.6.0/etc/hadoop:/home/software 6s/hadoop-2.6.0/share/hadoop/common/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/common/*:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs/*:/home/softwares/hadoop-2.6.0/share/hadoop/yarn/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/yarn/*:/home/softwares/hadoop-2.6.0/share/hadoop/mapreduce/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar:/lib/*' -Djava.library.path=:/home/softwares/hadoop-2.6.0/lib/native org.apache.flume.node.Application --conf-file=/home/data/flume/job/flume-my_netcat.conf --name my_netcatSLF4J: Class path contains multiple SLF4J bindings. 7SLF4J: Found binding in [jar:file:/home/softwares/apache-flume-1.9.0-bin/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] 8SLF4J: Found binding in [jar:file:/home/softwares/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] 9SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 10SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 1119/07/20 21:28:56 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 1219/07/20 21:28:56 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/home/data/flume/job/flume-my_netcat.conf 1319/07/20 21:28:56 INFO conf.FlumeConfiguration: Processing:netcat_source 1419/07/20 21:28:56 INFO conf.FlumeConfiguration: Added sinks: avro_sink Agent: my_netcat 1519/07/20 21:28:56 INFO conf.FlumeConfiguration: Processing:netcat_source 1619/07/20 21:28:56 INFO conf.FlumeConfiguration: Processing:avro_sink 1719/07/20 21:28:56 INFO conf.FlumeConfiguration: Processing:avro_sink 1819/07/20 21:28:56 INFO conf.FlumeConfiguration: Processing:netcat_source 1919/07/20 21:28:56 INFO conf.FlumeConfiguration: Processing:avro_sink 2019/07/20 21:28:56 INFO conf.FlumeConfiguration: Processing:memory_channel 2119/07/20 21:28:56 INFO conf.FlumeConfiguration: Processing:netcat_source 2219/07/20 21:28:56 INFO conf.FlumeConfiguration: Processing:avro_sink 2319/07/20 21:28:56 INFO conf.FlumeConfiguration: Processing:memory_channel 2419/07/20 21:28:56 INFO conf.FlumeConfiguration: Processing:memory_channel 2519/07/20 21:28:56 WARN conf.FlumeConfiguration: Agent configuration for 'my_netcat' has no configfilters. 2619/07/20 21:28:56 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [my_netcat] 2719/07/20 21:28:56 INFO node.AbstractConfigurationProvider: Creating channels 2819/07/20 21:28:56 INFO channel.DefaultChannelFactory: Creating instance of channel memory_channel type memory 2919/07/20 21:28:56 INFO node.AbstractConfigurationProvider: Created channel memory_channel 3019/07/20 21:28:56 INFO source.DefaultSourceFactory: Creating instance of source netcat_source, type netcat 3119/07/20 21:28:56 INFO sink.DefaultSinkFactory: Creating instance of sink: avro_sink, type: avro 3219/07/20 21:28:57 INFO sink.AbstractRpcSink: Connection reset is set to 0. Will not reset connection to next hop 3319/07/20 21:28:57 INFO node.AbstractConfigurationProvider: Channel memory_channel connected to [netcat_source, avro_sink] 3419/07/20 21:28:57 INFO node.Application: Starting new configuration:{ sourceRunners:{netcat_source=EventDrivenSourceRunner: { source:org.apache.flume.source.NetcatSource{name:netcat_source,state:IDLE} }} sinkRunners:{avro_sink=SinkRunner: { policy:org.apache.flume.sink. 35DefaultSinkProcessor@54ee68c1 counterGroup:{ name:null counters:{} } }} channels:{memory_channel=org.apache.flume.channel.MemoryChannel{name: memory_channel}} }19/07/20 21:28:57 INFO node.Application: Starting Channel memory_channel 3619/07/20 21:28:57 INFO node.Application: Waiting for channel: memory_channel to start. Sleeping for 500 ms 3719/07/20 21:28:57 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: memory_channel: Successfully registered new MBean. 3819/07/20 21:28:57 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: memory_channel started 3919/07/20 21:28:57 INFO node.Application: Starting Sink avro_sink 4019/07/20 21:28:57 INFO node.Application: Starting Source netcat_source 4119/07/20 21:28:57 INFO source.NetcatSource: Source starting 4219/07/20 21:28:57 INFO source.NetcatSource: Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/172.30.1.105:8888] 4319/07/20 21:28:57 INFO sink.AbstractRpcSink: Starting RpcSink avro_sink { host: node105.yinzhengjie.org.cn, port: 6666 }... 4419/07/20 21:28:57 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: avro_sink: Successfully registered new MBean. 4519/07/20 21:28:57 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: avro_sink started 4619/07/20 21:28:57 INFO sink.AbstractRpcSink: Rpc sink avro_sink: Building RpcClient with hostname: node105.yinzhengjie.org.cn, port: 6666 4719/07/20 21:28:57 INFO sink.AvroSink: Attempting to create Avro Rpc client. 4819/07/20 21:28:57 INFO api.NettyAvroRpcClient: Using default maxIOWorkers 4919/07/20 21:28:57 INFO util.log: Logging initialized @1360ms to org.eclipse.jetty.util.log.Slf4jLog 5019/07/20 21:28:57 INFO server.Server: jetty-9.4.6.v20170531 5119/07/20 21:28:57 INFO server.AbstractConnector: Started ServerConnector@eb2eb8e{HTTP/1.1,[http/1.1]}{0.0.0.0:10512} 5219/07/20 21:28:57 INFO server.Server: Started @1716ms 5319/07/20 21:28:58 INFO sink.AbstractRpcSink: Rpc sink avro_sink started.
[root@node105.yinzhengjie.org.cn ~]# tail -100f /home/data/flume/log/flume-my_netcat.log #查看启动时日志
1[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/job/flume-my_exec.conf 2# Name the components on this agent 3my_exec.sources = exec_source 4my_exec.sinks = avro_sink 5my_exec.channels = memory_channel 6 7# Describe/configure the source 8my_exec.sources.exec_source.type = exec 9my_exec.sources.exec_source.command = tail -F /yinzhengjie/data/flume/blog.txt 10my_exec.sources.exec_source.shell = /bin/bash -c 11 12# Describe the sink 13my_exec.sinks.avro_sink.type = avro 14my_exec.sinks.avro_sink.hostname = node105.yinzhengjie.org.cn 15my_exec.sinks.avro_sink.port = 6666 16 17# Describe the channel 18my_exec.channels.memory_channel.type = memory 19my_exec.channels.memory_channel.capacity = 1000 20my_exec.channels.memory_channel.transactionCapacity = 100 21 22# Bind the source and sink to the channel 23my_exec.sources.exec_source.channels = memory_channel 24my_exec.sinks.avro_sink.channel = memory_channel 25[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/job/flume-my_exec.conf #编写my_exec配置文件
1[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/shell/start-my_exec.sh 2#!/bin/bash 3#@author :yinzhengjie 4#blog:http://www.cnblogs.com/yinzhengjie 5#EMAIL:y1053419035@qq.com 6#Data:Thu Oct 18 11:26:06 CST 2018 7 8#将监控数据发送给ganglia，需要指定ganglia服务器地址，使用请确认是否部署好ganglia服务！ 9#nohup flume-ng agent -c /home/data/flume/job --conf-file=/home/data/flume/job/flume-my_exec.conf --name my_exec -Dflume.monitoring.type=ganglia -Dflume.monitoring.hosts=node105.yinzhengjie.org.cn:8649 -Dflume.root.logger=INFO,console >> /home/data/flume/log/flume-gangl 10ia-flume-my_exec.log 2>&1 & 11 12#启动flume自身的监控参数，默认执行以下脚本 13nohup flume-ng agent -c /home/data/flume/job --conf-file=/home/data/flume/job/flume-my_exec.conf --name my_exec -Dflume.monitoring.type=http -Dflume.monitoring.port=10513 -Dflume.root.logger=INFO,console >> /home/data/flume/log/flume-my_exec.log 2>&1 & 14[root@node105.yinzhengjie.org.cn ~]# 15[root@node105.yinzhengjie.org.cn ~]# ss -ntl 16State Recv-Q Send-Q Local Address:Port Peer Address:Port 17LISTEN 0 50 172.30.1.105:6666 *:* 18LISTEN 0 50 *:10511 *:* 19LISTEN 0 50 *:10512 *:* 20LISTEN 0 128 *:22 *:* 21LISTEN 0 50 172.30.1.105:8888 *:* 22LISTEN 0 128 :::22 :::* 23[root@node105.yinzhengjie.org.cn ~]# 24[root@node105.yinzhengjie.org.cn ~]# 25[root@node105.yinzhengjie.org.cn ~]# /home/data/flume/shell/start-my_exec.sh 26[root@node105.yinzhengjie.org.cn ~]# 27[root@node105.yinzhengjie.org.cn ~]# ss -ntl 28State Recv-Q Send-Q Local Address:Port Peer Address:Port 29LISTEN 0 50 172.30.1.105:6666 *:* 30LISTEN 0 50 *:10511 *:* 31LISTEN 0 50 *:10512 *:* 32LISTEN 0 50 *:10513 *:* 33LISTEN 0 128 *:22 *:* 34LISTEN 0 50 172.30.1.105:8888 *:* 35LISTEN 0 128 :::22 :::* 36[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/shell/start-my_exec.sh #编写启动脚本并启动
1[root@node105.yinzhengjie.org.cn ~]# tail -100f /home/data/flume/log/flume-my_exec.log 2Warning: JAVA_HOME is not set! 3Info: Including Hadoop libraries found via (/home/softwares/hadoop-2.6.0/bin/hadoop) for HDFS access 4Info: Including Hive libraries found via () for Hive access 5+ exec /home/softwares/jdk1.8.0_201/bin/java -Xmx20m -Dflume.monitoring.type=http -Dflume.monitoring.port=10513 -Dflume.root.logger=INFO,console -cp '/home/data/flume/job:/home/softwares/apache-flume-1.9.0-bin/lib/*:/home/softwares/hadoop-2.6.0/etc/hadoop:/home/software 6s/hadoop-2.6.0/share/hadoop/common/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/common/*:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs/*:/home/softwares/hadoop-2.6.0/share/hadoop/yarn/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/yarn/*:/home/softwares/hadoop-2.6.0/share/hadoop/mapreduce/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar:/lib/*' -Djava.library.path=:/home/softwares/hadoop-2.6.0/lib/native org.apache.flume.node.Application --conf-file=/home/data/flume/job/flume-my_exec.conf --name my_execSLF4J: Class path contains multiple SLF4J bindings. 7SLF4J: Found binding in [jar:file:/home/softwares/apache-flume-1.9.0-bin/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] 8SLF4J: Found binding in [jar:file:/home/softwares/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] 9SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 10SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 1119/07/20 21:44:00 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 1219/07/20 21:44:00 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/home/data/flume/job/flume-my_exec.conf 1319/07/20 21:44:00 INFO conf.FlumeConfiguration: Processing:memory_channel 1419/07/20 21:44:00 INFO conf.FlumeConfiguration: Added sinks: avro_sink Agent: my_exec 1519/07/20 21:44:00 INFO conf.FlumeConfiguration: Processing:exec_source 1619/07/20 21:44:00 INFO conf.FlumeConfiguration: Processing:avro_sink 1719/07/20 21:44:00 INFO conf.FlumeConfiguration: Processing:avro_sink 1819/07/20 21:44:00 INFO conf.FlumeConfiguration: Processing:avro_sink 1919/07/20 21:44:00 INFO conf.FlumeConfiguration: Processing:exec_source 2019/07/20 21:44:00 INFO conf.FlumeConfiguration: Processing:avro_sink 2119/07/20 21:44:00 INFO conf.FlumeConfiguration: Processing:memory_channel 2219/07/20 21:44:00 INFO conf.FlumeConfiguration: Processing:exec_source 2319/07/20 21:44:00 INFO conf.FlumeConfiguration: Processing:exec_source 2419/07/20 21:44:00 INFO conf.FlumeConfiguration: Processing:memory_channel 2519/07/20 21:44:00 WARN conf.FlumeConfiguration: Agent configuration for 'my_exec' has no configfilters. 2619/07/20 21:44:00 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [my_exec] 2719/07/20 21:44:00 INFO node.AbstractConfigurationProvider: Creating channels 2819/07/20 21:44:00 INFO channel.DefaultChannelFactory: Creating instance of channel memory_channel type memory 2919/07/20 21:44:00 INFO node.AbstractConfigurationProvider: Created channel memory_channel 3019/07/20 21:44:00 INFO source.DefaultSourceFactory: Creating instance of source exec_source, type exec 3119/07/20 21:44:00 INFO sink.DefaultSinkFactory: Creating instance of sink: avro_sink, type: avro 3219/07/20 21:44:00 INFO sink.AbstractRpcSink: Connection reset is set to 0. Will not reset connection to next hop 3319/07/20 21:44:00 INFO node.AbstractConfigurationProvider: Channel memory_channel connected to [exec_source, avro_sink] 3419/07/20 21:44:00 INFO node.Application: Starting new configuration:{ sourceRunners:{exec_source=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource{name:exec_source,state:IDLE} }} sinkRunners:{avro_sink=SinkRunner: { policy:org.apache.flume.sink.Defaul 35tSinkProcessor@2739d05f counterGroup:{ name:null counters:{} } }} channels:{memory_channel=org.apache.flume.channel.MemoryChannel{name: memory_channel}} }19/07/20 21:44:00 INFO node.Application: Starting Channel memory_channel 3619/07/20 21:44:00 INFO node.Application: Waiting for channel: memory_channel to start. Sleeping for 500 ms 3719/07/20 21:44:00 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: memory_channel: Successfully registered new MBean. 3819/07/20 21:44:00 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: memory_channel started 3919/07/20 21:44:01 INFO node.Application: Starting Sink avro_sink 4019/07/20 21:44:01 INFO node.Application: Starting Source exec_source 4119/07/20 21:44:01 INFO source.ExecSource: Exec source starting with command: tail -F /yinzhengjie/data/flume/blog.txt 4219/07/20 21:44:01 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: exec_source: Successfully registered new MBean. 4319/07/20 21:44:01 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: exec_source started 4419/07/20 21:44:01 INFO sink.AbstractRpcSink: Starting RpcSink avro_sink { host: node105.yinzhengjie.org.cn, port: 6666 }... 4519/07/20 21:44:01 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: avro_sink: Successfully registered new MBean. 4619/07/20 21:44:01 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: avro_sink started 4719/07/20 21:44:01 INFO sink.AbstractRpcSink: Rpc sink avro_sink: Building RpcClient with hostname: node105.yinzhengjie.org.cn, port: 6666 4819/07/20 21:44:01 INFO sink.AvroSink: Attempting to create Avro Rpc client. 4919/07/20 21:44:01 INFO api.NettyAvroRpcClient: Using default maxIOWorkers 5019/07/20 21:44:01 INFO util.log: Logging initialized @1342ms to org.eclipse.jetty.util.log.Slf4jLog 5119/07/20 21:44:01 INFO server.Server: jetty-9.4.6.v20170531 5219/07/20 21:44:01 INFO server.AbstractConnector: Started ServerConnector@79d1663b{HTTP/1.1,[http/1.1]}{0.0.0.0:10513} 5319/07/20 21:44:01 INFO server.Server: Started @1660ms 5419/07/20 21:44:01 INFO sink.AbstractRpcSink: Rpc sink avro_sink started.
[root@node105.yinzhengjie.org.cn ~]# tail -100f /home/data/flume/log/flume-my_exec.log #查看日志信息
1[root@node105.yinzhengjie.org.cn ~]# telnet node105.yinzhengjie.org.cn 8888 2Trying 172.30.1.105... 3Connected to node105.yinzhengjie.org.cn. 4Escape character is '^]'. 5yinzhengjie dao ci yi you! 6OK
[root@node105.yinzhengjie.org.cn ~]# telnet node105.yinzhengjie.org.cn 8888 #测试一
1[root@node105.yinzhengjie.org.cn ~]# echo "https://www.cnblogs.com/yinzhengjie" >> /yinzhengjie/data/flume/blog.txt 2[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# echo "https://www.cnblogs.com/yinzhengjie" >> /yinzhengjie/data/flume/blog.txt #测试二
1[root@node105.yinzhengjie.org.cn ~]# hdfs dfs -cat /flume/20190720/21/172.30.1.105-.1563630502053.tmp #查看上面写入的2条测试数据 2yinzhengjie dao ci yi you! 3https://www.cnblogs.com/yinzhengjie 4[root@node105.yinzhengjie.org.cn ~]#
5>.挑选器案例
1channel selector: 2 通道挑选器，选择指定的event发送到指定的channel 3 4 (1)Replicating Channel Selector 5 默认为副本挑选器，事件均以副本方式输出，换句话说就是有几个channel就发送几个副本。 6 7 (2)multiplexing selector 8 多路复用挑选器，作用就是可以将不同的内容发送到指定的channel 9 10 详情请参考： 11 http://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html#flume-channel-selectors
1[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/job/flume-my_avro.conf 2# Name the components on this agent 3my_avro.sources = avro_source 4my_avro.sinks = hdfs_sink 5my_avro.channels = memory_channel 6 7# Describe/configure the source 8my_avro.sources.avro_source.type = avro 9my_avro.sources.avro_source.bind = node105.yinzhengjie.org.cn 10my_avro.sources.avro_source.port = 8888 11 12# 定义到hdfs的sink 13my_avro.sinks.hdfs_sink.type = hdfs 14my_avro.sinks.hdfs_sink.hdfs.path = hdfs://node101.yinzhengjie.org.cn:8020/flume/%Y%m%d/%H 15#上传文件的前缀 16my_avro.sinks.hdfs_sink.hdfs.filePrefix = 172.30.1.105- 17#是否按照时间滚动文件夹 18my_avro.sinks.hdfs_sink.hdfs.round = true 19#多少时间单位创建一个新的文件夹 20my_avro.sinks.hdfs_sink.hdfs.roundValue = 1 21#重新定义时间单位 22my_avro.sinks.hdfs_sink.hdfs.roundUnit = hour 23#是否使用本地时间戳 24my_avro.sinks.hdfs_sink.hdfs.useLocalTimeStamp = true 25#积攒多少个Event才flush到HDFS一次 26my_avro.sinks.hdfs_sink.hdfs.batchSize = 100 27#设置文件类型，可支持压缩 28my_avro.sinks.hdfs_sink.hdfs.fileType = DataStream 29#多久生成一个新的文件 30my_avro.sinks.hdfs_sink.hdfs.rollInterval = 60 31#设置每个文件的滚动大小大概是128M 32my_avro.sinks.hdfs_sink.hdfs.rollSize = 134210000 33#文件的滚动与Event数量无关 34my_avro.sinks.hdfs_sink.hdfs.rollCount = 0 35#最小冗余数 36my_avro.sinks.hdfs_sink.hdfs.minBlockReplicas = 1 37 38 39# Describe the channel 40my_avro.channels.memory_channel.type = memory 41my_avro.channels.memory_channel.capacity = 1000 42my_avro.channels.memory_channel.transactionCapacity = 100 43 44# Bind the source and sink to the channel 45my_avro.sources.avro_source.channels = memory_channel 46my_avro.sinks.hdfs_sink.channel = memory_channel 47[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/job/flume-my_avro.conf #编写启动文件
1[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/shell/start-my_avro.sh 2#!/bin/bash 3#@author :yinzhengjie 4#blog:http://www.cnblogs.com/yinzhengjie 5#EMAIL:y1053419035@qq.com 6#Data:Thu Oct 18 11:26:06 CST 2018 7 8#将监控数据发送给ganglia，需要指定ganglia服务器地址，使用请确认是否部署好ganglia服务！ 9#nohup flume-ng agent -c /home/data/flume/job --conf-file=/home/data/flume/job/flume-my_avro.conf --name my_avro -Dflume.monitoring.type=ganglia -Dflume.monitoring.hosts=node105.yinzhengjie.org.cn:8649 -Dflume.root.logger=INFO,console >> /home/data/flume/log/flume-gangl 10ia-flume-my_avro.log 2>&1 & 11 12#启动flume自身的监控参数，默认执行以下脚本 13nohup flume-ng agent -c /home/data/flume/job --conf-file=/home/data/flume/job/flume-my_avro.conf --name my_avro -Dflume.monitoring.type=http -Dflume.monitoring.port=10514 -Dflume.root.logger=INFO,console >> /home/data/flume/log/flume-my_avro.log 2>&1 & 14[root@node105.yinzhengjie.org.cn ~]# 15[root@node105.yinzhengjie.org.cn ~]# 16[root@node105.yinzhengjie.org.cn ~]# ss -ntl 17State Recv-Q Send-Q Local Address:Port Peer Address:Port 18LISTEN 0 128 *:22 *:* 19LISTEN 0 128 :::22 :::* 20[root@node105.yinzhengjie.org.cn ~]# 21[root@node105.yinzhengjie.org.cn ~]# 22[root@node105.yinzhengjie.org.cn ~]# /home/data/flume/shell/start-my_avro.sh 23[root@node105.yinzhengjie.org.cn ~]# 24[root@node105.yinzhengjie.org.cn ~]# ss -ntl 25State Recv-Q Send-Q Local Address:Port Peer Address:Port 26LISTEN 0 50 *:10514 *:* 27LISTEN 0 128 *:22 *:* 28LISTEN 0 50 172.30.1.105:8888 *:* 29LISTEN 0 128 :::22 :::* 30[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/shell/start-my_avro.sh #编写启动脚本并启动
1[root@node105.yinzhengjie.org.cn ~]# tail -100f /home/data/flume/log/flume-my_avro.log 2Warning: JAVA_HOME is not set! 3Info: Including Hadoop libraries found via (/home/softwares/hadoop-2.6.0/bin/hadoop) for HDFS access 4Info: Including Hive libraries found via () for Hive access 5+ exec /home/softwares/jdk1.8.0_201/bin/java -Xmx20m -Dflume.monitoring.type=http -Dflume.monitoring.port=10514 -Dflume.root.logger=INFO,console -cp '/home/data/flume/job:/home/softwares/apache-flume-1.9.0-bin/lib/*:/home/softwares/hadoop-2.6.0/etc/hadoop:/home/software 6s/hadoop-2.6.0/share/hadoop/common/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/common/*:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs/*:/home/softwares/hadoop-2.6.0/share/hadoop/yarn/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/yarn/*:/home/softwares/hadoop-2.6.0/share/hadoop/mapreduce/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar:/lib/*' -Djava.library.path=:/home/softwares/hadoop-2.6.0/lib/native org.apache.flume.node.Application --conf-file=/home/data/flume/job/flume-my_avro.conf --name my_avroSLF4J: Class path contains multiple SLF4J bindings. 7SLF4J: Found binding in [jar:file:/home/softwares/apache-flume-1.9.0-bin/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] 8SLF4J: Found binding in [jar:file:/home/softwares/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] 9SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 10SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 1119/07/20 22:16:38 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 1219/07/20 22:16:38 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/home/data/flume/job/flume-my_avro.conf 1319/07/20 22:16:38 INFO conf.FlumeConfiguration: Processing:avro_source 1419/07/20 22:16:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 1519/07/20 22:16:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 1619/07/20 22:16:38 INFO conf.FlumeConfiguration: Processing:avro_source 1719/07/20 22:16:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 1819/07/20 22:16:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 1919/07/20 22:16:38 INFO conf.FlumeConfiguration: Added sinks: hdfs_sink Agent: my_avro 2019/07/20 22:16:38 INFO conf.FlumeConfiguration: Processing:memory_channel 2119/07/20 22:16:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 2219/07/20 22:16:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 2319/07/20 22:16:38 INFO conf.FlumeConfiguration: Processing:memory_channel 2419/07/20 22:16:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 2519/07/20 22:16:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 2619/07/20 22:16:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 2719/07/20 22:16:38 INFO conf.FlumeConfiguration: Processing:avro_source 2819/07/20 22:16:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 2919/07/20 22:16:38 INFO conf.FlumeConfiguration: Processing:avro_source 3019/07/20 22:16:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 3119/07/20 22:16:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 3219/07/20 22:16:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 3319/07/20 22:16:38 INFO conf.FlumeConfiguration: Processing:hdfs_sink 3419/07/20 22:16:38 INFO conf.FlumeConfiguration: Processing:memory_channel 3519/07/20 22:16:38 WARN conf.FlumeConfiguration: Agent configuration for 'my_avro' has no configfilters. 3619/07/20 22:16:38 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [my_avro] 3719/07/20 22:16:38 INFO node.AbstractConfigurationProvider: Creating channels 3819/07/20 22:16:38 INFO channel.DefaultChannelFactory: Creating instance of channel memory_channel type memory 3919/07/20 22:16:38 INFO node.AbstractConfigurationProvider: Created channel memory_channel 4019/07/20 22:16:38 INFO source.DefaultSourceFactory: Creating instance of source avro_source, type avro 4119/07/20 22:16:38 INFO sink.DefaultSinkFactory: Creating instance of sink: hdfs_sink, type: hdfs 4219/07/20 22:16:38 INFO node.AbstractConfigurationProvider: Channel memory_channel connected to [avro_source, hdfs_sink] 4319/07/20 22:16:38 INFO node.Application: Starting new configuration:{ sourceRunners:{avro_source=EventDrivenSourceRunner: { source:Avro source avro_source: { bindAddress: node105.yinzhengjie.org.cn, port: 8888 } }} sinkRunners:{hdfs_sink=SinkRunner: { policy:org.apache. 44flume.sink.DefaultSinkProcessor@77a41618 counterGroup:{ name:null counters:{} } }} channels:{memory_channel=org.apache.flume.channel.MemoryChannel{name: memory_channel}} }19/07/20 22:16:38 INFO node.Application: Starting Channel memory_channel 4519/07/20 22:16:38 INFO node.Application: Waiting for channel: memory_channel to start. Sleeping for 500 ms 4619/07/20 22:16:38 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: memory_channel: Successfully registered new MBean. 4719/07/20 22:16:38 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: memory_channel started 4819/07/20 22:16:39 INFO node.Application: Starting Sink hdfs_sink 4919/07/20 22:16:39 INFO node.Application: Starting Source avro_source 5019/07/20 22:16:39 INFO source.AvroSource: Starting Avro source avro_source: { bindAddress: node105.yinzhengjie.org.cn, port: 8888 }... 5119/07/20 22:16:39 INFO util.log: Logging initialized @1334ms to org.eclipse.jetty.util.log.Slf4jLog 5219/07/20 22:16:39 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: hdfs_sink: Successfully registered new MBean. 5319/07/20 22:16:39 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: hdfs_sink started 5419/07/20 22:16:39 INFO server.Server: jetty-9.4.6.v20170531 5519/07/20 22:16:39 INFO server.AbstractConnector: Started ServerConnector@72e46d99{HTTP/1.1,[http/1.1]}{0.0.0.0:10514} 5619/07/20 22:16:39 INFO server.Server: Started @1740ms 5719/07/20 22:16:39 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: avro_source: Successfully registered new MBean. 5819/07/20 22:16:39 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: avro_source started 5919/07/20 22:16:39 INFO source.AvroSource: Avro source avro_source started.
[root@node105.yinzhengjie.org.cn ~]# tail -100f /home/data/flume/log/flume-my_avro.log #查看启动日志
1[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/job/flume-my_file_roll.conf 2# Name the components on this agent 3my_file_roll.sources = avro_source 4my_file_roll.sinks = file_roll_sink 5my_file_roll.channels = memory_channel 6 7# Describe/configure the source 8my_file_roll.sources.avro_source.type = avro 9my_file_roll.sources.avro_source.bind = node105.yinzhengjie.org.cn 10my_file_roll.sources.avro_source.port = 9999 11 12# Describe the sink 13my_file_roll.sinks.file_roll_sink.type = file_roll 14#输出的本地目录必须是已经存在的目录，如果该目录不存在，并不会创建新的目录。 15my_file_roll.sinks.file_roll_sink.sink.directory = /yinzhengjie/data/flume/output 16 17# Describe the channel 18my_file_roll.channels.memory_channel.type = memory 19my_file_roll.channels.memory_channel.capacity = 1000 20my_file_roll.channels.memory_channel.transactionCapacity = 100 21 22# Bind the source and sink to the channel 23my_file_roll.sources.avro_source.channels = memory_channel 24my_file_roll.sinks.file_roll_sink.channel = memory_channel 25[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/job/flume-my_file_roll.conf #编写配置文件
1[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/shell/start-my_file_roll.sh 2#!/bin/bash 3#@author :yinzhengjie 4#blog:http://www.cnblogs.com/yinzhengjie 5#EMAIL:y1053419035@qq.com 6#Data:Thu Oct 18 11:26:06 CST 2018 7 8#将监控数据发送给ganglia，需要指定ganglia服务器地址，使用请确认是否部署好ganglia服务！ 9#nohup flume-ng agent -c /home/data/flume/job --conf-file=/home/data/flume/job/flume-my_file_roll.conf --name my_file_roll -Dflume.monitoring.type=ganglia -Dflume.monitoring.hosts=node105.yinzhengjie.org.cn:8649 -Dflume.root.logger=INFO,console >> /home/data/flume/log/f 10lume-ganglia-flume-my_file_roll.log 2>&1 & 11 12#启动flume自身的监控参数，默认执行以下脚本 13nohup flume-ng agent -c /home/data/flume/job --conf-file=/home/data/flume/job/flume-my_file_roll.conf --name my_file_roll -Dflume.monitoring.type=http -Dflume.monitoring.port=10515 -Dflume.root.logger=INFO,console >> /home/data/flume/log/flume-my_file_roll.log 2>&1 & 14[root@node105.yinzhengjie.org.cn ~]# 15[root@node105.yinzhengjie.org.cn ~]# 16[root@node105.yinzhengjie.org.cn ~]# ss -ntl 17State Recv-Q Send-Q Local Address:Port Peer Address:Port 18LISTEN 0 50 *:10514 *:* 19LISTEN 0 128 *:22 *:* 20LISTEN 0 50 172.30.1.105:8888 *:* 21LISTEN 0 128 :::22 :::* 22[root@node105.yinzhengjie.org.cn ~]# 23[root@node105.yinzhengjie.org.cn ~]# /home/data/flume/shell/start-my_file_roll.sh 24[root@node105.yinzhengjie.org.cn ~]# 25[root@node105.yinzhengjie.org.cn ~]# ss -ntl 26State Recv-Q Send-Q Local Address:Port Peer Address:Port 27LISTEN 0 50 *:10514 *:* 28LISTEN 0 50 *:10515 *:* 29LISTEN 0 128 *:22 *:* 30LISTEN 0 50 172.30.1.105:8888 *:* 31LISTEN 0 128 :::22 :::* 32[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/shell/start-my_file_roll.sh #编写启动脚本并启动
1[root@node105.yinzhengjie.org.cn ~]# tail -100f /home/data/flume/log/flume-my_file_roll.log 2Warning: JAVA_HOME is not set! 3Info: Including Hadoop libraries found via (/home/softwares/hadoop-2.6.0/bin/hadoop) for HDFS access 4Info: Including Hive libraries found via () for Hive access 5+ exec /home/softwares/jdk1.8.0_201/bin/java -Xmx20m -Dflume.monitoring.type=http -Dflume.monitoring.port=10515 -Dflume.root.logger=INFO,console -cp '/home/data/flume/job:/home/softwares/apache-flume-1.9.0-bin/lib/*:/home/softwares/hadoop-2.6.0/etc/hadoop:/home/software 6s/hadoop-2.6.0/share/hadoop/common/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/common/*:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs/*:/home/softwares/hadoop-2.6.0/share/hadoop/yarn/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/yarn/*:/home/softwares/hadoop-2.6.0/share/hadoop/mapreduce/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar:/lib/*' -Djava.library.path=:/home/softwares/hadoop-2.6.0/lib/native org.apache.flume.node.Application --conf-file=/home/data/flume/job/flume-my_file_roll.conf --name my_file_rollSLF4J: Class path contains multiple SLF4J bindings. 7SLF4J: Found binding in [jar:file:/home/softwares/apache-flume-1.9.0-bin/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] 8SLF4J: Found binding in [jar:file:/home/softwares/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] 9SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 10SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 1119/07/20 22:25:32 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 1219/07/20 22:25:32 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/home/data/flume/job/flume-my_file_roll.conf 1319/07/20 22:25:32 INFO conf.FlumeConfiguration: Processing:file_roll_sink 1419/07/20 22:25:32 INFO conf.FlumeConfiguration: Processing:file_roll_sink 1519/07/20 22:25:32 INFO conf.FlumeConfiguration: Processing:memory_channel 1619/07/20 22:25:32 INFO conf.FlumeConfiguration: Processing:file_roll_sink 1719/07/20 22:25:32 INFO conf.FlumeConfiguration: Added sinks: file_roll_sink Agent: my_file_roll 1819/07/20 22:25:32 INFO conf.FlumeConfiguration: Processing:avro_source 1919/07/20 22:25:32 INFO conf.FlumeConfiguration: Processing:memory_channel 2019/07/20 22:25:32 INFO conf.FlumeConfiguration: Processing:avro_source 2119/07/20 22:25:32 INFO conf.FlumeConfiguration: Processing:memory_channel 2219/07/20 22:25:32 INFO conf.FlumeConfiguration: Processing:avro_source 2319/07/20 22:25:32 INFO conf.FlumeConfiguration: Processing:avro_source 2419/07/20 22:25:32 WARN conf.FlumeConfiguration: Agent configuration for 'my_file_roll' has no configfilters. 2519/07/20 22:25:32 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [my_file_roll] 2619/07/20 22:25:32 INFO node.AbstractConfigurationProvider: Creating channels 2719/07/20 22:25:32 INFO channel.DefaultChannelFactory: Creating instance of channel memory_channel type memory 2819/07/20 22:25:32 INFO node.AbstractConfigurationProvider: Created channel memory_channel 2919/07/20 22:25:32 INFO source.DefaultSourceFactory: Creating instance of source avro_source, type avro 3019/07/20 22:25:32 INFO sink.DefaultSinkFactory: Creating instance of sink: file_roll_sink, type: file_roll 3119/07/20 22:25:32 INFO node.AbstractConfigurationProvider: Channel memory_channel connected to [avro_source, file_roll_sink] 3219/07/20 22:25:32 INFO node.Application: Starting new configuration:{ sourceRunners:{avro_source=EventDrivenSourceRunner: { source:Avro source avro_source: { bindAddress: node105.yinzhengjie.org.cn, port: 9999 } }} sinkRunners:{file_roll_sink=SinkRunner: { policy:org.ap 33ache.flume.sink.DefaultSinkProcessor@5dda816 counterGroup:{ name:null counters:{} } }} channels:{memory_channel=org.apache.flume.channel.MemoryChannel{name: memory_channel}} }19/07/20 22:25:32 INFO node.Application: Starting Channel memory_channel 3419/07/20 22:25:32 INFO node.Application: Waiting for channel: memory_channel to start. Sleeping for 500 ms 3519/07/20 22:25:32 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: memory_channel: Successfully registered new MBean. 3619/07/20 22:25:32 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: memory_channel started 3719/07/20 22:25:33 INFO node.Application: Starting Sink file_roll_sink 3819/07/20 22:25:33 INFO node.Application: Starting Source avro_source 3919/07/20 22:25:33 INFO source.AvroSource: Starting Avro source avro_source: { bindAddress: node105.yinzhengjie.org.cn, port: 9999 }... 4019/07/20 22:25:33 INFO util.log: Logging initialized @1346ms to org.eclipse.jetty.util.log.Slf4jLog 4119/07/20 22:25:33 INFO sink.RollingFileSink: Starting org.apache.flume.sink.RollingFileSink{name:file_roll_sink, channel:memory_channel}... 4219/07/20 22:25:33 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: file_roll_sink: Successfully registered new MBean. 4319/07/20 22:25:33 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: file_roll_sink started 4419/07/20 22:25:33 INFO sink.RollingFileSink: RollingFileSink file_roll_sink started. 4519/07/20 22:25:33 INFO server.Server: jetty-9.4.6.v20170531 4619/07/20 22:25:33 INFO server.AbstractConnector: Started ServerConnector@1bca8c57{HTTP/1.1,[http/1.1]}{0.0.0.0:10515} 4719/07/20 22:25:33 INFO server.Server: Started @1818ms 4819/07/20 22:25:33 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: avro_source: Successfully registered new MBean. 4919/07/20 22:25:33 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: avro_source started 5019/07/20 22:25:33 INFO source.AvroSource: Avro source avro_source started.
[root@node105.yinzhengjie.org.cn ~]# tail -100f /home/data/flume/log/flume-my_file_roll.log #查看启动日志
1[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/job/flume-replica.conf 2# Name the components on this agent 3replica.sources = exec_source 4replica.sinks = hdfs_sink file_roll_sink 5replica.channels = hdfs_channel file_roll_channel 6 7# 将数据流复制给多个channel 8replica.sources.exec_source.selector.type = replicating 9 10# Describe/configure the source 11replica.sources.exec_source.type = exec 12replica.sources.exec_source.command = tail -F /yinzhengjie/data/flume/blog.txt 13replica.sources.exec_source.shell = /bin/bash -c 14 15# 定义要输出到hdfs的sink，注意端口号 16replica.sinks.hdfs_sink.type = avro 17replica.sinks.hdfs_sink.hostname = node105.yinzhengjie.org.cn 18replica.sinks.hdfs_sink.port = 8888 19 20# 定义要输出到local filesystem的sink 21replica.sinks.file_roll_sink.type = avro 22replica.sinks.file_roll_sink.hostname = node105.yinzhengjie.org.cn 23replica.sinks.file_roll_sink.port = 9999 24 25# Describe the channel 26replica.channels.hdfs_channel.type = memory 27replica.channels.hdfs_channel.capacity = 1000 28replica.channels.hdfs_channel.transactionCapacity = 100 29 30replica.channels.file_roll_channel.type = memory 31replica.channels.file_roll_channel.capacity = 1000 32replica.channels.file_roll_channel.transactionCapacity = 100 33 34# Bind the source and sink to the channel 35replica.sources.exec_source.channels = hdfs_channel file_roll_channel 36replica.sinks.hdfs_sink.channel = hdfs_channel 37replica.sinks.file_roll_sink.channel = file_roll_channel 38[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/job/flume-replica.conf #编写配置文件
1[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/shell/start-replica.sh 2#!/bin/bash 3#@author :yinzhengjie 4#blog:http://www.cnblogs.com/yinzhengjie 5#EMAIL:y1053419035@qq.com 6#Data:Thu Oct 18 11:26:06 CST 2018 7 8#将监控数据发送给ganglia，需要指定ganglia服务器地址，使用请确认是否部署好ganglia服务！ 9#nohup flume-ng agent -c /home/data/flume/job --conf-file=/home/data/flume/job/flume-replica.conf --name replica -Dflume.monitoring.type=ganglia -Dflume.monitoring.hosts=node105.yinzhengjie.org.cn:8649 -Dflume.root.logger=INFO,console >> /home/data/flume/log/flume-gangl 10ia-flume-replica.log 2>&1 & 11 12#启动flume自身的监控参数，默认执行以下脚本 13nohup flume-ng agent -c /home/data/flume/job --conf-file=/home/data/flume/job/flume-replica.conf --name replica -Dflume.monitoring.type=http -Dflume.monitoring.port=10516 -Dflume.root.logger=INFO,console >> /home/data/flume/log/flume-replica.log 2>&1 & 14[root@node105.yinzhengjie.org.cn ~]# 15[root@node105.yinzhengjie.org.cn ~]# ss -ntl 16State Recv-Q Send-Q Local Address:Port Peer Address:Port 17LISTEN 0 50 172.30.1.105:9999 *:* 18LISTEN 0 50 *:10514 *:* 19LISTEN 0 50 *:10515 *:* 20LISTEN 0 128 *:22 *:* 21LISTEN 0 50 172.30.1.105:8888 *:* 22LISTEN 0 128 :::22 :::* 23[root@node105.yinzhengjie.org.cn ~]# 24[root@node105.yinzhengjie.org.cn ~]# /home/data/flume/shell/start-replica.sh 25[root@node105.yinzhengjie.org.cn ~]# 26[root@node105.yinzhengjie.org.cn ~]# ss -ntl 27State Recv-Q Send-Q Local Address:Port Peer Address:Port 28LISTEN 0 50 172.30.1.105:9999 *:* 29LISTEN 0 50 *:10514 *:* 30LISTEN 0 50 *:10515 *:* 31LISTEN 0 50 *:10516 *:* 32LISTEN 0 128 *:22 *:* 33LISTEN 0 50 172.30.1.105:8888 *:* 34LISTEN 0 128 :::22 :::* 35[root@node105.yinzhengjie.org.cn ~]# 36[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/shell/start-replica.sh #编写启动脚本并启动
1[root@node105.yinzhengjie.org.cn ~]# tail -100f /home/data/flume/log/flume-replica.log 2Warning: JAVA_HOME is not set! 3Info: Including Hadoop libraries found via (/home/softwares/hadoop-2.6.0/bin/hadoop) for HDFS access 4Info: Including Hive libraries found via () for Hive access 5+ exec /home/softwares/jdk1.8.0_201/bin/java -Xmx20m -Dflume.monitoring.type=http -Dflume.monitoring.port=10516 -Dflume.root.logger=INFO,console -cp '/home/data/flume/job:/home/softwares/apache-flume-1.9.0-bin/lib/*:/home/softwares/hadoop-2.6.0/etc/hadoop:/home/software 6s/hadoop-2.6.0/share/hadoop/common/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/common/*:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs/*:/home/softwares/hadoop-2.6.0/share/hadoop/yarn/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/yarn/*:/home/softwares/hadoop-2.6.0/share/hadoop/mapreduce/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar:/lib/*' -Djava.library.path=:/home/softwares/hadoop-2.6.0/lib/native org.apache.flume.node.Application --conf-file=/home/data/flume/job/flume-replica.conf --name replicaSLF4J: Class path contains multiple SLF4J bindings. 7SLF4J: Found binding in [jar:file:/home/softwares/apache-flume-1.9.0-bin/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] 8SLF4J: Found binding in [jar:file:/home/softwares/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] 9SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 10SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 1119/07/20 22:35:04 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 1219/07/20 22:35:04 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/home/data/flume/job/flume-replica.conf 1319/07/20 22:35:04 INFO conf.FlumeConfiguration: Processing:hdfs_sink 1419/07/20 22:35:04 INFO conf.FlumeConfiguration: Processing:file_roll_channel 1519/07/20 22:35:04 INFO conf.FlumeConfiguration: Processing:hdfs_sink 1619/07/20 22:35:04 INFO conf.FlumeConfiguration: Processing:hdfs_channel 1719/07/20 22:35:04 INFO conf.FlumeConfiguration: Processing:hdfs_channel 1819/07/20 22:35:04 INFO conf.FlumeConfiguration: Processing:file_roll_channel 1919/07/20 22:35:04 INFO conf.FlumeConfiguration: Processing:file_roll_sink 2019/07/20 22:35:04 INFO conf.FlumeConfiguration: Processing:hdfs_channel 2119/07/20 22:35:04 INFO conf.FlumeConfiguration: Added sinks: hdfs_sink file_roll_sink Agent: replica 2219/07/20 22:35:04 INFO conf.FlumeConfiguration: Processing:file_roll_sink 2319/07/20 22:35:04 INFO conf.FlumeConfiguration: Processing:exec_source 2419/07/20 22:35:04 INFO conf.FlumeConfiguration: Processing:file_roll_sink 2519/07/20 22:35:04 INFO conf.FlumeConfiguration: Processing:exec_source 2619/07/20 22:35:04 INFO conf.FlumeConfiguration: Processing:exec_source 2719/07/20 22:35:04 INFO conf.FlumeConfiguration: Processing:exec_source 2819/07/20 22:35:04 INFO conf.FlumeConfiguration: Processing:hdfs_sink 2919/07/20 22:35:04 INFO conf.FlumeConfiguration: Processing:file_roll_channel 3019/07/20 22:35:04 INFO conf.FlumeConfiguration: Processing:hdfs_sink 3119/07/20 22:35:04 INFO conf.FlumeConfiguration: Processing:exec_source 3219/07/20 22:35:04 INFO conf.FlumeConfiguration: Processing:file_roll_sink 3319/07/20 22:35:04 WARN conf.FlumeConfiguration: Agent configuration for 'replica' has no configfilters. 3419/07/20 22:35:04 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [replica] 3519/07/20 22:35:04 INFO node.AbstractConfigurationProvider: Creating channels 3619/07/20 22:35:04 INFO channel.DefaultChannelFactory: Creating instance of channel hdfs_channel type memory 3719/07/20 22:35:04 INFO node.AbstractConfigurationProvider: Created channel hdfs_channel 3819/07/20 22:35:04 INFO channel.DefaultChannelFactory: Creating instance of channel file_roll_channel type memory 3919/07/20 22:35:04 INFO node.AbstractConfigurationProvider: Created channel file_roll_channel 4019/07/20 22:35:04 INFO source.DefaultSourceFactory: Creating instance of source exec_source, type exec 4119/07/20 22:35:04 INFO sink.DefaultSinkFactory: Creating instance of sink: file_roll_sink, type: avro 4219/07/20 22:35:04 INFO sink.AbstractRpcSink: Connection reset is set to 0. Will not reset connection to next hop 4319/07/20 22:35:04 INFO sink.DefaultSinkFactory: Creating instance of sink: hdfs_sink, type: avro 4419/07/20 22:35:04 INFO sink.AbstractRpcSink: Connection reset is set to 0. Will not reset connection to next hop 4519/07/20 22:35:04 INFO node.AbstractConfigurationProvider: Channel hdfs_channel connected to [exec_source, hdfs_sink] 4619/07/20 22:35:04 INFO node.AbstractConfigurationProvider: Channel file_roll_channel connected to [exec_source, file_roll_sink] 4719/07/20 22:35:04 INFO node.Application: Starting new configuration:{ sourceRunners:{exec_source=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource{name:exec_source,state:IDLE} }} sinkRunners:{file_roll_sink=SinkRunner: { policy:org.apache.flume.sink.D 48efaultSinkProcessor@444b86bf counterGroup:{ name:null counters:{} } }, hdfs_sink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@2d990d41 counterGroup:{ name:null counters:{} } }} channels:{hdfs_channel=org.apache.flume.channel.MemoryChannel{name: hdfs_channel}, file_roll_channel=org.apache.flume.channel.MemoryChannel{name: file_roll_channel}} }19/07/20 22:35:04 INFO node.Application: Starting Channel hdfs_channel 4919/07/20 22:35:04 INFO node.Application: Starting Channel file_roll_channel 5019/07/20 22:35:04 INFO node.Application: Waiting for channel: hdfs_channel to start. Sleeping for 500 ms 5119/07/20 22:35:04 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: file_roll_channel: Successfully registered new MBean. 5219/07/20 22:35:04 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: file_roll_channel started 5319/07/20 22:35:04 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: hdfs_channel: Successfully registered new MBean. 5419/07/20 22:35:04 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: hdfs_channel started 5519/07/20 22:35:04 INFO node.Application: Starting Sink file_roll_sink 5619/07/20 22:35:04 INFO node.Application: Starting Sink hdfs_sink 5719/07/20 22:35:04 INFO sink.AbstractRpcSink: Starting RpcSink hdfs_sink { host: node105.yinzhengjie.org.cn, port: 8888 }... 5819/07/20 22:35:04 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: hdfs_sink: Successfully registered new MBean. 5919/07/20 22:35:04 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: hdfs_sink started 6019/07/20 22:35:04 INFO sink.AbstractRpcSink: Rpc sink hdfs_sink: Building RpcClient with hostname: node105.yinzhengjie.org.cn, port: 8888 6119/07/20 22:35:04 INFO sink.AvroSink: Attempting to create Avro Rpc client. 6219/07/20 22:35:04 INFO api.NettyAvroRpcClient: Using default maxIOWorkers 6319/07/20 22:35:04 INFO sink.AbstractRpcSink: Starting RpcSink file_roll_sink { host: node105.yinzhengjie.org.cn, port: 9999 }... 6419/07/20 22:35:04 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: file_roll_sink: Successfully registered new MBean. 6519/07/20 22:35:04 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: file_roll_sink started 6619/07/20 22:35:04 INFO sink.AbstractRpcSink: Rpc sink file_roll_sink: Building RpcClient with hostname: node105.yinzhengjie.org.cn, port: 9999 6719/07/20 22:35:04 INFO sink.AvroSink: Attempting to create Avro Rpc client. 6819/07/20 22:35:04 INFO api.NettyAvroRpcClient: Using default maxIOWorkers 6919/07/20 22:35:04 INFO node.Application: Starting Source exec_source 7019/07/20 22:35:04 INFO source.ExecSource: Exec source starting with command: tail -F /yinzhengjie/data/flume/blog.txt 7119/07/20 22:35:04 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: exec_source: Successfully registered new MBean. 7219/07/20 22:35:04 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: exec_source started 7319/07/20 22:35:04 INFO util.log: Logging initialized @1370ms to org.eclipse.jetty.util.log.Slf4jLog 7419/07/20 22:35:05 INFO server.Server: jetty-9.4.6.v20170531 7519/07/20 22:35:05 INFO server.AbstractConnector: Started ServerConnector@3e973d1f{HTTP/1.1,[http/1.1]}{0.0.0.0:10516} 7619/07/20 22:35:05 INFO server.Server: Started @1627ms 7719/07/20 22:35:05 INFO sink.AbstractRpcSink: Rpc sink hdfs_sink started. 7819/07/20 22:35:05 INFO sink.AbstractRpcSink: Rpc sink file_roll_sink started.
[root@node105.yinzhengjie.org.cn ~]# tail -100f /home/data/flume/log/flume-replica.log #查看日志信息
1[root@node105.yinzhengjie.org.cn ~]# cat /yinzhengjie/data/flume/blog.txt 2https://www.cnblogs.com/yinzhengjie 3[root@node105.yinzhengjie.org.cn ~]# 4[root@node105.yinzhengjie.org.cn ~]# hdfs dfs -ls /flume/20190720/22 5Found 1 items 6-rw-r--r-- 3 root supergroup 36 2019-07-20 22:36 /flume/20190720/22/172.30.1.105-.1563633311415 7[root@node105.yinzhengjie.org.cn ~]# 8[root@node105.yinzhengjie.org.cn ~]# 9[root@node105.yinzhengjie.org.cn ~]# hdfs dfs -cat /flume/20190720/22/172.30.1.105-.1563633311415 10https://www.cnblogs.com/yinzhengjie 11[root@node105.yinzhengjie.org.cn ~]# 12[root@node105.yinzhengjie.org.cn ~]# 13[root@node105.yinzhengjie.org.cn ~]# ll /yinzhengjie/data/flume/output/ 14total 4 15-rw-r--r-- 1 root root 0 Jul 20 22:25 1563632732571-1 16-rw-r--r-- 1 root root 0 Jul 20 22:30 1563632732571-10 17-rw-r--r-- 1 root root 0 Jul 20 22:30 1563632732571-11 18-rw-r--r-- 1 root root 0 Jul 20 22:31 1563632732571-12 19-rw-r--r-- 1 root root 0 Jul 20 22:31 1563632732571-13 20-rw-r--r-- 1 root root 0 Jul 20 22:32 1563632732571-14 21-rw-r--r-- 1 root root 0 Jul 20 22:32 1563632732571-15 22-rw-r--r-- 1 root root 0 Jul 20 22:33 1563632732571-16 23-rw-r--r-- 1 root root 0 Jul 20 22:33 1563632732571-17 24-rw-r--r-- 1 root root 0 Jul 20 22:34 1563632732571-18 25-rw-r--r-- 1 root root 0 Jul 20 22:34 1563632732571-19 26-rw-r--r-- 1 root root 0 Jul 20 22:26 1563632732571-2 27-rw-r--r-- 1 root root 36 Jul 20 22:35 1563632732571-20 28-rw-r--r-- 1 root root 0 Jul 20 22:35 1563632732571-21 29-rw-r--r-- 1 root root 0 Jul 20 22:36 1563632732571-22 30-rw-r--r-- 1 root root 0 Jul 20 22:36 1563632732571-23 31-rw-r--r-- 1 root root 0 Jul 20 22:37 1563632732571-24 32-rw-r--r-- 1 root root 0 Jul 20 22:37 1563632732571-25 33-rw-r--r-- 1 root root 0 Jul 20 22:38 1563632732571-26 34-rw-r--r-- 1 root root 0 Jul 20 22:38 1563632732571-27 35-rw-r--r-- 1 root root 0 Jul 20 22:39 1563632732571-28 36-rw-r--r-- 1 root root 0 Jul 20 22:39 1563632732571-29 37-rw-r--r-- 1 root root 0 Jul 20 22:26 1563632732571-3 38-rw-r--r-- 1 root root 0 Jul 20 22:40 1563632732571-30 39-rw-r--r-- 1 root root 0 Jul 20 22:40 1563632732571-31 40-rw-r--r-- 1 root root 0 Jul 20 22:41 1563632732571-32 41-rw-r--r-- 1 root root 0 Jul 20 22:41 1563632732571-33 42-rw-r--r-- 1 root root 0 Jul 20 22:42 1563632732571-34 43-rw-r--r-- 1 root root 0 Jul 20 22:42 1563632732571-35 44-rw-r--r-- 1 root root 0 Jul 20 22:43 1563632732571-36 45-rw-r--r-- 1 root root 0 Jul 20 22:43 1563632732571-37 46-rw-r--r-- 1 root root 0 Jul 20 22:27 1563632732571-4 47-rw-r--r-- 1 root root 0 Jul 20 22:27 1563632732571-5 48-rw-r--r-- 1 root root 0 Jul 20 22:28 1563632732571-6 49-rw-r--r-- 1 root root 0 Jul 20 22:28 1563632732571-7 50-rw-r--r-- 1 root root 0 Jul 20 22:29 1563632732571-8 51-rw-r--r-- 1 root root 0 Jul 20 22:29 1563632732571-9 52[root@node105.yinzhengjie.org.cn ~]# 53[root@node105.yinzhengjie.org.cn ~]# cat /yinzhengjie/data/flume/output/1563632732571-20 54https://www.cnblogs.com/yinzhengjie 55[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# cat /yinzhengjie/data/flume/blog.txt #验证sink端是否有相应的数据生成
1[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/job/flume-my_multiplexing_selector.conf 2# Name the components on this agent 3my_multiplexing_selector.sources = avro_source 4my_multiplexing_selector.sinks = Java_sink Go_sink Python_sink 5my_multiplexing_selector.channels = Java_channel Go_channel Python_channel 6 7# Describe/configure the source 8my_multiplexing_selector.sources.avro_source.type = avro 9my_multiplexing_selector.sources.avro_source.bind = node105.yinzhengjie.org.cn 10my_multiplexing_selector.sources.avro_source.port = 8888 11# 指定挑选器类型为：多路复用 12my_multiplexing_selector.sources.avro_source.selector.type = multiplexing 13# 指定event header的key值 14my_multiplexing_selector.sources.avro_source.selector.header = language 15# 指定event header的key值对应的value值，编辑文件内容如：language java 16my_multiplexing_selector.sources.avro_source.selector.mapping.JAVA = Java_channel 17my_multiplexing_selector.sources.avro_source.selector.mapping.GOLANG = Go_channel 18my_multiplexing_selector.sources.avro_source.selector.default = Python_channel 19 20# Describe the sink 21my_multiplexing_selector.sinks.Java_sink.type = file_roll 22my_multiplexing_selector.sinks.Java_sink.sink.directory= /yinzhengjie/language/java 23my_multiplexing_selector.sinks.Java_sink.sink.rollInterval = 0 24my_multiplexing_selector.sinks.Go_sink.type = file_roll 25my_multiplexing_selector.sinks.Go_sink.sink.directory= /yinzhengjie/language/golang 26my_multiplexing_selector.sinks.Go_sink.sink.rollInterval = 0 27my_multiplexing_selector.sinks.Python_sink.type = file_roll 28my_multiplexing_selector.sinks.Python_sink.sink.directory= /yinzhengjie/language/python 29my_multiplexing_selector.sinks.Python_sink.sink.rollInterval = 0 30 31# Use a channel which buffers events in memory 32my_multiplexing_selector.channels.Java_channel.type = memory 33my_multiplexing_selector.channels.Java_channel.capacity = 100000 34my_multiplexing_selector.channels.Java_channel.transactionCapacity = 10000 35my_multiplexing_selector.channels.Go_channel.type = memory 36my_multiplexing_selector.channels.Go_channel.capacity = 100000 37my_multiplexing_selector.channels.Go_channel.transactionCapacity = 10000 38my_multiplexing_selector.channels.Python_channel.type = memory 39my_multiplexing_selector.channels.Python_channel.capacity = 100000 40my_multiplexing_selector.channels.Python_channel.transactionCapacity = 10000 41 42# Bind the source and sink to the channel 43my_multiplexing_selector.sources.avro_source.channels = Java_channel Go_channel Python_channel 44my_multiplexing_selector.sinks.Java_sink.channel = Java_channel 45my_multiplexing_selector.sinks.Go_sink.channel = Go_channel 46my_multiplexing_selector.sinks.Python_sink.channel = Python_channel 47[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/job/flume-my_multiplexing_selector.conf （不推荐使用）
1[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/shell/start-my_multiplexing_selector.sh 2#!/bin/bash 3#@author :yinzhengjie 4#blog:http://www.cnblogs.com/yinzhengjie 5#EMAIL:y1053419035@qq.com 6#Data:Thu Oct 18 11:26:06 CST 2018 7 8#将监控数据发送给ganglia，需要指定ganglia服务器地址，使用请确认是否部署好ganglia服务！ 9#nohup flume-ng agent -c /home/data/flume/job --conf-file=/home/data/flume/job/flume-my_multiplexing_selector.conf --name my_multiplexing_selector -Dflume.monitoring.type=ganglia -Dflume.monitoring.hosts=node105.yinzhengjie.org.cn:8649 -Dflume.root.logger=INFO,console > 10> /home/data/flume/log/flume-ganglia-flume-my_multiplexing_selector.log 2>&1 & 11 12#启动flume自身的监控参数，默认执行以下脚本 13nohup flume-ng agent -c /home/data/flume/job --conf-file=/home/data/flume/job/flume-my_multiplexing_selector.conf --name my_multiplexing_selector -Dflume.monitoring.type=http -Dflume.monitoring.port=10522 -Dflume.root.logger=INFO,console >> /home/data/flume/log/flume- 14my_multiplexing_selector.log 2>&1 &[root@node105.yinzhengjie.org.cn ~]# 15[root@node105.yinzhengjie.org.cn ~]# ss -ntl 16State Recv-Q Send-Q Local Address:Port Peer Address:Port 17LISTEN 0 128 *:22 *:* 18LISTEN 0 128 :::22 :::* 19[root@node105.yinzhengjie.org.cn ~]# 20[root@node105.yinzhengjie.org.cn ~]# 21[root@node105.yinzhengjie.org.cn ~]# /home/data/flume/shell/start-my_multiplexing_selector.sh 22[root@node105.yinzhengjie.org.cn ~]# 23[root@node105.yinzhengjie.org.cn ~]# ss -ntl 24State Recv-Q Send-Q Local Address:Port Peer Address:Port 25LISTEN 0 128 *:22 *:* 26LISTEN 0 50 172.30.1.105:8888 *:* 27LISTEN 0 50 *:10522 *:* 28LISTEN 0 128 :::22 :::* 29[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/shell/start-my_multiplexing_selector.sh #编写启动脚本并启动
1[root@node105.yinzhengjie.org.cn ~]# cat a.txt 2language java 3language php 4language shell 5language golang 6language python 7language scanla 8language js 9language vbs 10language c++ 11language linux 12[root@node105.yinzhengjie.org.cn ~]# 13[root@node105.yinzhengjie.org.cn ~]# cat header.txt 14language java 15[root@node105.yinzhengjie.org.cn ~]# 16[root@node105.yinzhengjie.org.cn ~]# flume-ng avro-client -H node105.yinzhengjie.org.cn -p 8888 -F a.txt -R header.txt 17Warning: No configuration directory set! Use --conf <dir> to override. 18Warning: JAVA_HOME is not set! 19Info: Including Hadoop libraries found via (/home/softwares/hadoop-2.6.0/bin/hadoop) for HDFS access 20Info: Including Hive libraries found via () for Hive access 21+ exec /home/softwares/jdk1.8.0_201/bin/java -Xmx20m -cp '/home/softwares/apache-flume-1.9.0-bin/lib/*:/home/softwares/hadoop-2.6.0/etc/hadoop:/home/softwares/hadoop-2.6.0/share/hadoop/common/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/common/*:/home/softwares/hadoo 22p-2.6.0/share/hadoop/hdfs:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs/*:/home/softwares/hadoop-2.6.0/share/hadoop/yarn/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/yarn/*:/home/softwares/hadoop-2.6.0/share/hadoop/mapreduce/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar:/lib/*' -Djava.library.path=:/home/softwares/hadoop-2.6.0/lib/native org.apache.flume.client.avro.AvroCLIClient -H node105.yinzhengjie.org.cn -p 8888 -F a.txt -R header.txtSLF4J: Class path contains multiple SLF4J bindings. 23SLF4J: Found binding in [jar:file:/home/softwares/apache-flume-1.9.0-bin/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] 24SLF4J: Found binding in [jar:file:/home/softwares/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] 25SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 26SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2719/07/21 00:09:22 INFO api.NettyAvroRpcClient: Using default maxIOWorkers 28[root@node105.yinzhengjie.org.cn ~]# 29[root@node105.yinzhengjie.org.cn ~]# 30[root@node105.yinzhengjie.org.cn ~]# ll /yinzhengjie/language/ -R 31/yinzhengjie/language/: 32total 0 33drwxr-xr-x 2 root root 29 Jul 21 00:08 golang 34drwxr-xr-x 2 root root 29 Jul 21 00:08 java 35drwxr-xr-x 2 root root 29 Jul 21 00:08 python 36 37/yinzhengjie/language/golang: 38total 0 39-rw-r--r-- 1 root root 0 Jul 21 00:08 1563638927737-1 40 41/yinzhengjie/language/java: 42total 0 43-rw-r--r-- 1 root root 0 Jul 21 00:08 1563638927710-1 44 45/yinzhengjie/language/python: 46total 4 47-rw-r--r-- 1 root root 143 Jul 21 00:09 1563638927710-1 48[root@node105.yinzhengjie.org.cn ~]# 49[root@node105.yinzhengjie.org.cn ~]# cat /yinzhengjie/language/python/1563638927710-1 50language java 51language php 52language shell 53language golang 54language python 55language scanla 56language js 57language vbs 58language c++ 59language linux 60[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# flume-ng avro-client -H node105.yinzhengjie.org.cn -p 8888 -F a.txt -R header.txt #启动avro客户端，指定头部，观察文件被发到哪个文件夹
1[root@node105.yinzhengjie.org.cn ~]# tail -100f /home/data/flume/log/flume-my_multiplexing_selector.log 2Warning: JAVA_HOME is not set! 3Info: Including Hadoop libraries found via (/home/softwares/hadoop-2.6.0/bin/hadoop) for HDFS access 4Info: Including Hive libraries found via () for Hive access 5+ exec /home/softwares/jdk1.8.0_201/bin/java -Xmx20m -Dflume.monitoring.type=http -Dflume.monitoring.port=10522 -Dflume.root.logger=INFO,console -cp '/home/data/flume/job:/home/softwares/apache-flume-1.9.0-bin/lib/*:/home/softwares/hadoop-2.6.0/etc/hadoop:/home/software 6s/hadoop-2.6.0/share/hadoop/common/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/common/*:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs/*:/home/softwares/hadoop-2.6.0/share/hadoop/yarn/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/yarn/*:/home/softwares/hadoop-2.6.0/share/hadoop/mapreduce/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar:/lib/*' -Djava.library.path=:/home/softwares/hadoop-2.6.0/lib/native org.apache.flume.node.Application --conf-file=/home/data/flume/job/flume-my_multiplexing_selector.conf --name my_multiplexing_selectorSLF4J: Class path contains multiple SLF4J bindings. 7SLF4J: Found binding in [jar:file:/home/softwares/apache-flume-1.9.0-bin/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] 8SLF4J: Found binding in [jar:file:/home/softwares/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] 9SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 10SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 1119/07/21 00:08:47 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 1219/07/21 00:08:47 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/home/data/flume/job/flume-my_multiplexing_selector.conf 1319/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:Java_sink 1419/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:Go_channel 1519/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:avro_source 1619/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:Go_channel 1719/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:avro_source 1819/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:avro_source 1919/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:Java_sink 2019/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:avro_source 2119/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:avro_source 2219/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:Python_sink 2319/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:Go_sink 2419/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:Java_channel 2519/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:avro_source 2619/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:avro_source 2719/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:Python_channel 2819/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:Python_sink 2919/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:Python_sink 3019/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:Go_sink 3119/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:Java_sink 3219/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:Go_channel 3319/07/21 00:08:47 INFO conf.FlumeConfiguration: Added sinks: Java_sink Go_sink Python_sink Agent: my_multiplexing_selector 3419/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:Python_channel 3519/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:Java_channel 3619/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:Python_channel 3719/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:Python_sink 3819/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:Java_sink 3919/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:avro_source 4019/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:Go_sink 4119/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:avro_source 4219/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:Java_channel 4319/07/21 00:08:47 INFO conf.FlumeConfiguration: Processing:Go_sink 4419/07/21 00:08:47 WARN conf.FlumeConfiguration: Agent configuration for 'my_multiplexing_selector' has no configfilters. 4519/07/21 00:08:47 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [my_multiplexing_selector] 4619/07/21 00:08:47 INFO node.AbstractConfigurationProvider: Creating channels 4719/07/21 00:08:47 INFO channel.DefaultChannelFactory: Creating instance of channel Java_channel type memory 4819/07/21 00:08:47 INFO node.AbstractConfigurationProvider: Created channel Java_channel 4919/07/21 00:08:47 INFO channel.DefaultChannelFactory: Creating instance of channel Go_channel type memory 5019/07/21 00:08:47 INFO node.AbstractConfigurationProvider: Created channel Go_channel 5119/07/21 00:08:47 INFO channel.DefaultChannelFactory: Creating instance of channel Python_channel type memory 5219/07/21 00:08:47 INFO node.AbstractConfigurationProvider: Created channel Python_channel 5319/07/21 00:08:47 INFO source.DefaultSourceFactory: Creating instance of source avro_source, type avro 5419/07/21 00:08:47 INFO sink.DefaultSinkFactory: Creating instance of sink: Java_sink, type: file_roll 5519/07/21 00:08:47 INFO sink.DefaultSinkFactory: Creating instance of sink: Python_sink, type: file_roll 5619/07/21 00:08:47 INFO sink.DefaultSinkFactory: Creating instance of sink: Go_sink, type: file_roll 5719/07/21 00:08:47 INFO node.AbstractConfigurationProvider: Channel Java_channel connected to [avro_source, Java_sink] 5819/07/21 00:08:47 INFO node.AbstractConfigurationProvider: Channel Go_channel connected to [avro_source, Go_sink] 5919/07/21 00:08:47 INFO node.AbstractConfigurationProvider: Channel Python_channel connected to [avro_source, Python_sink] 6019/07/21 00:08:47 INFO node.Application: Starting new configuration:{ sourceRunners:{avro_source=EventDrivenSourceRunner: { source:Avro source avro_source: { bindAddress: node105.yinzhengjie.org.cn, port: 8888 } }} sinkRunners:{Java_sink=SinkRunner: { policy:org.apache. 61flume.sink.DefaultSinkProcessor@12f20d45 counterGroup:{ name:null counters:{} } }, Python_sink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@40b91f49 counterGroup:{ name:null counters:{} } }, Go_sink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@328a1feb counterGroup:{ name:null counters:{} } }} channels:{Java_channel=org.apache.flume.channel.MemoryChannel{name: Java_channel}, Go_channel=org.apache.flume.channel.MemoryChannel{name: Go_channel}, Python_channel=org.apache.flume.channel.MemoryChannel{name: Python_channel}} }19/07/21 00:08:47 INFO node.Application: Starting Channel Java_channel 6219/07/21 00:08:47 INFO node.Application: Starting Channel Go_channel 6319/07/21 00:08:47 INFO node.Application: Starting Channel Python_channel 6419/07/21 00:08:47 INFO node.Application: Waiting for channel: Java_channel to start. Sleeping for 500 ms 6519/07/21 00:08:47 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: Go_channel: Successfully registered new MBean. 6619/07/21 00:08:47 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: Go_channel started 6719/07/21 00:08:47 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: Java_channel: Successfully registered new MBean. 6819/07/21 00:08:47 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: Java_channel started 6919/07/21 00:08:47 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: Python_channel: Successfully registered new MBean. 7019/07/21 00:08:47 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: Python_channel started 7119/07/21 00:08:48 INFO node.Application: Starting Sink Java_sink 7219/07/21 00:08:48 INFO node.Application: Starting Sink Python_sink 7319/07/21 00:08:48 INFO sink.RollingFileSink: Starting org.apache.flume.sink.RollingFileSink{name:Python_sink, channel:Python_channel}... 7419/07/21 00:08:48 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: Python_sink: Successfully registered new MBean. 7519/07/21 00:08:48 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: Python_sink started 7619/07/21 00:08:48 INFO sink.RollingFileSink: RollInterval is not valid, file rolling will not happen. 7719/07/21 00:08:48 INFO sink.RollingFileSink: RollingFileSink Python_sink started. 7819/07/21 00:08:48 INFO node.Application: Starting Sink Go_sink 7919/07/21 00:08:48 INFO node.Application: Starting Source avro_source 8019/07/21 00:08:48 INFO sink.RollingFileSink: Starting org.apache.flume.sink.RollingFileSink{name:Java_sink, channel:Java_channel}... 8119/07/21 00:08:48 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: Java_sink: Successfully registered new MBean. 8219/07/21 00:08:48 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: Java_sink started 8319/07/21 00:08:48 INFO sink.RollingFileSink: RollInterval is not valid, file rolling will not happen. 8419/07/21 00:08:48 INFO sink.RollingFileSink: RollingFileSink Java_sink started. 8519/07/21 00:08:48 INFO sink.RollingFileSink: Starting org.apache.flume.sink.RollingFileSink{name:Go_sink, channel:Go_channel}... 8619/07/21 00:08:48 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: Go_sink: Successfully registered new MBean. 8719/07/21 00:08:48 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: Go_sink started 8819/07/21 00:08:48 INFO sink.RollingFileSink: RollInterval is not valid, file rolling will not happen. 8919/07/21 00:08:48 INFO sink.RollingFileSink: RollingFileSink Go_sink started. 9019/07/21 00:08:48 INFO source.AvroSource: Starting Avro source avro_source: { bindAddress: node105.yinzhengjie.org.cn, port: 8888 }... 9119/07/21 00:08:48 INFO util.log: Logging initialized @1387ms to org.eclipse.jetty.util.log.Slf4jLog 9219/07/21 00:08:48 INFO server.Server: jetty-9.4.6.v20170531 9319/07/21 00:08:48 INFO server.AbstractConnector: Started ServerConnector@773090d9{HTTP/1.1,[http/1.1]}{0.0.0.0:10522} 9419/07/21 00:08:48 INFO server.Server: Started @1824ms 9519/07/21 00:08:48 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: avro_source: Successfully registered new MBean. 9619/07/21 00:08:48 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: avro_source started 9719/07/21 00:08:48 INFO source.AvroSource: Avro source avro_source started. 9819/07/21 00:09:22 INFO ipc.NettyServer: [id: 0x047099b3, /172.30.1.105:48044 => /172.30.1.105:8888] OPEN 9919/07/21 00:09:22 INFO ipc.NettyServer: [id: 0x047099b3, /172.30.1.105:48044 => /172.30.1.105:8888] BOUND: /172.30.1.105:8888 10019/07/21 00:09:22 INFO ipc.NettyServer: [id: 0x047099b3, /172.30.1.105:48044 => /172.30.1.105:8888] CONNECTED: /172.30.1.105:48044 10119/07/21 00:09:23 INFO ipc.NettyServer: [id: 0x047099b3, /172.30.1.105:48044 :> /172.30.1.105:8888] DISCONNECTED 10219/07/21 00:09:23 INFO ipc.NettyServer: [id: 0x047099b3, /172.30.1.105:48044 :> /172.30.1.105:8888] UNBOUND 10319/07/21 00:09:23 INFO ipc.NettyServer: [id: 0x047099b3, /172.30.1.105:48044 :> /172.30.1.105:8888] CLOSED 10419/07/21 00:09:23 INFO ipc.NettyServer: Connection to /172.30.1.105:48044 disconnected.
[root@node105.yinzhengjie.org.cn ~]# tail -100f /home/data/flume/log/flume-my_multiplexing_selector.log #查看日志信息
6>.主机拦截器案例
1拦截器(interceptor): 2 是source端的在处理过程中能够对数据(event)进行修改或丢弃的组件。常见的拦截器有： 3 (1)host interceptor 4 将发送的event添加主机名的header 5 6 (2)timestamp interceptor 7 将发送的event添加时间戳的header 8 9 更多拦截器可参考官方文档： 10 http://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html#flume-interceptors
1[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/job/flume-host_interceptor.conf 2# Name the components on this agent 3my_host_interceptor.sources = netcat_source 4my_host_interceptor.sinks = logger_sink 5my_host_interceptor.channels = memory_channel 6 7# Describe/configure the source 8my_host_interceptor.sources.netcat_source.type = netcat 9my_host_interceptor.sources.netcat_source.bind = node105.yinzhengjie.org.cn 10my_host_interceptor.sources.netcat_source.port = 8888 11 12# 指定添加拦截器 13my_host_interceptor.sources.netcat_source.interceptors = i1 14my_host_interceptor.sources.netcat_source.interceptors.i1.type = org.apache.flume.interceptor.HostInterceptor$Builder 15my_host_interceptor.sources.netcat_source.interceptors.i1.preserveExisting = false 16# 指定header的key 17my_host_interceptor.sources.netcat_source.interceptors.i1.hostHeader = hostname 18# 指定header的value为主机ip 19my_host_interceptor.sources.netcat_source.interceptors.i1.useIP = true 20 21# Describe the sink 22my_host_interceptor.sinks.logger_sink.type = logger 23 24# Use a channel which buffers events in memory 25my_host_interceptor.channels.memory_channel.type = memory 26my_host_interceptor.channels.memory_channel.capacity = 100000 27my_host_interceptor.channels.memory_channel.transactionCapacity = 10000 28 29# Bind the source and sink to the channel 30my_host_interceptor.sources.netcat_source.channels = memory_channel 31my_host_interceptor.sinks.logger_sink.channel = memory_channel 32[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/job/flume-host_interceptor.conf #编辑主机拦截器配置文件（案例一）
1[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/shell/start-my_host_interceptor.sh 2#!/bin/bash 3#@author :yinzhengjie 4#blog:http://www.cnblogs.com/yinzhengjie 5#EMAIL:y1053419035@qq.com 6#Data:Thu Oct 18 11:26:06 CST 2018 7 8#将监控数据发送给ganglia，需要指定ganglia服务器地址，使用请确认是否部署好ganglia服务！ 9#nohup flume-ng agent -c /home/data/flume/job --conf-file=/home/data/flume/job/flume-host_interceptor.conf --name my_host_interceptor -Dflume.monitoring.type=ganglia -Dflume.monitoring.hosts=node105.yinzhengjie.org.cn:8649 -Dflume.root.logger=INFO,console >> /home/data/ 10flume/log/flume-ganglia-flume-my_host_interceptor.log 2>&1 & 11 12#启动flume自身的监控参数，默认执行以下脚本 13nohup flume-ng agent -c /home/data/flume/job --conf-file=/home/data/flume/job/flume-host_interceptor.conf --name my_host_interceptor -Dflume.monitoring.type=http -Dflume.monitoring.port=10520 -Dflume.root.logger=INFO,console >> /home/data/flume/log/flume-my_host_inter 14ceptor.log 2>&1 &[root@node105.yinzhengjie.org.cn ~]# 15[root@node105.yinzhengjie.org.cn ~]# ss -ntl 16State Recv-Q Send-Q Local Address:Port Peer Address:Port 17LISTEN 0 128 *:22 *:* 18LISTEN 0 128 :::22 :::* 19[root@node105.yinzhengjie.org.cn ~]# 20[root@node105.yinzhengjie.org.cn ~]# /home/data/flume/shell/start-my_host_interceptor.sh 21[root@node105.yinzhengjie.org.cn ~]# 22[root@node105.yinzhengjie.org.cn ~]# ss -ntl 23State Recv-Q Send-Q Local Address:Port Peer Address:Port 24LISTEN 0 128 *:22 *:* 25LISTEN 0 50 *:10520 *:* 26LISTEN 0 50 172.30.1.105:8888 *:* 27LISTEN 0 128 :::22 :::* 28[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/shell/start-my_host_interceptor.sh #编写脚本并启动
1[root@node105.yinzhengjie.org.cn ~]# telnet node105.yinzhengjie.org.cn 8888 #连接到指定端口并发送测试数据 2Trying 172.30.1.105... 3Connected to node105.yinzhengjie.org.cn. 4Escape character is '^]'. 5yinzhengjie dao ci yi you! 6OK
1[root@node105.yinzhengjie.org.cn ~]# tail -100f /home/data/flume/log/flume-my_host_interceptor.log 2Warning: JAVA_HOME is not set! 3Info: Including Hadoop libraries found via (/home/softwares/hadoop-2.6.0/bin/hadoop) for HDFS access 4Info: Including Hive libraries found via () for Hive access 5+ exec /home/softwares/jdk1.8.0_201/bin/java -Xmx20m -Dflume.monitoring.type=http -Dflume.monitoring.port=10520 -Dflume.root.logger=INFO,console -cp '/home/data/flume/job:/home/softwares/apache-flume-1.9.0-bin/lib/*:/home/softwares/hadoop-2.6.0/etc/hadoop:/home/software 6s/hadoop-2.6.0/share/hadoop/common/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/common/*:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs/*:/home/softwares/hadoop-2.6.0/share/hadoop/yarn/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/yarn/*:/home/softwares/hadoop-2.6.0/share/hadoop/mapreduce/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar:/lib/*' -Djava.library.path=:/home/softwares/hadoop-2.6.0/lib/native org.apache.flume.node.Application --conf-file=/home/data/flume/job/flume-host_interceptor.conf --name my_host_interceptorSLF4J: Class path contains multiple SLF4J bindings. 7SLF4J: Found binding in [jar:file:/home/softwares/apache-flume-1.9.0-bin/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] 8SLF4J: Found binding in [jar:file:/home/softwares/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] 9SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 10SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 1119/07/20 23:10:03 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 1219/07/20 23:10:03 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/home/data/flume/job/flume-host_interceptor.conf 1319/07/20 23:10:03 INFO conf.FlumeConfiguration: Processing:netcat_source 1419/07/20 23:10:03 INFO conf.FlumeConfiguration: Processing:netcat_source 1519/07/20 23:10:03 INFO conf.FlumeConfiguration: Processing:memory_channel 1619/07/20 23:10:03 INFO conf.FlumeConfiguration: Processing:netcat_source 1719/07/20 23:10:03 INFO conf.FlumeConfiguration: Processing:netcat_source 1819/07/20 23:10:03 INFO conf.FlumeConfiguration: Processing:logger_sink 1919/07/20 23:10:03 INFO conf.FlumeConfiguration: Processing:memory_channel 2019/07/20 23:10:03 INFO conf.FlumeConfiguration: Added sinks: logger_sink Agent: my_host_interceptor 2119/07/20 23:10:03 INFO conf.FlumeConfiguration: Processing:netcat_source 2219/07/20 23:10:03 INFO conf.FlumeConfiguration: Processing:netcat_source 2319/07/20 23:10:03 INFO conf.FlumeConfiguration: Processing:memory_channel 2419/07/20 23:10:03 INFO conf.FlumeConfiguration: Processing:netcat_source 2519/07/20 23:10:03 INFO conf.FlumeConfiguration: Processing:netcat_source 2619/07/20 23:10:03 INFO conf.FlumeConfiguration: Processing:logger_sink 2719/07/20 23:10:03 INFO conf.FlumeConfiguration: Processing:netcat_source 2819/07/20 23:10:03 WARN conf.FlumeConfiguration: Agent configuration for 'my_host_interceptor' has no configfilters. 2919/07/20 23:10:03 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [my_host_interceptor] 3019/07/20 23:10:03 INFO node.AbstractConfigurationProvider: Creating channels 3119/07/20 23:10:03 INFO channel.DefaultChannelFactory: Creating instance of channel memory_channel type memory 3219/07/20 23:10:03 INFO node.AbstractConfigurationProvider: Created channel memory_channel 3319/07/20 23:10:03 INFO source.DefaultSourceFactory: Creating instance of source netcat_source, type netcat 3419/07/20 23:10:03 INFO sink.DefaultSinkFactory: Creating instance of sink: logger_sink, type: logger 3519/07/20 23:10:03 INFO node.AbstractConfigurationProvider: Channel memory_channel connected to [netcat_source, logger_sink] 3619/07/20 23:10:03 INFO node.Application: Starting new configuration:{ sourceRunners:{netcat_source=EventDrivenSourceRunner: { source:org.apache.flume.source.NetcatSource{name:netcat_source,state:IDLE} }} sinkRunners:{logger_sink=SinkRunner: { policy:org.apache.flume.sin 37k.DefaultSinkProcessor@2739d05f counterGroup:{ name:null counters:{} } }} channels:{memory_channel=org.apache.flume.channel.MemoryChannel{name: memory_channel}} }19/07/20 23:10:03 INFO node.Application: Starting Channel memory_channel 3819/07/20 23:10:03 INFO node.Application: Waiting for channel: memory_channel to start. Sleeping for 500 ms 3919/07/20 23:10:03 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: memory_channel: Successfully registered new MBean. 4019/07/20 23:10:03 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: memory_channel started 4119/07/20 23:10:04 INFO node.Application: Starting Sink logger_sink 4219/07/20 23:10:04 INFO node.Application: Starting Source netcat_source 4319/07/20 23:10:04 INFO source.NetcatSource: Source starting 4419/07/20 23:10:04 INFO source.NetcatSource: Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/172.30.1.105:8888] 4519/07/20 23:10:04 INFO util.log: Logging initialized @1344ms to org.eclipse.jetty.util.log.Slf4jLog 4619/07/20 23:10:04 INFO server.Server: jetty-9.4.6.v20170531 4719/07/20 23:10:04 INFO server.AbstractConnector: Started ServerConnector@6a5cf88c{HTTP/1.1,[http/1.1]}{0.0.0.0:10520} 4819/07/20 23:10:04 INFO server.Server: Started @1584ms 4919/07/20 23:10:21 INFO sink.LoggerSink: Event: { headers:{hostname=172.30.1.105} body: 79 69 6E 7A 68 65 6E 67 6A 69 65 20 64 61 6F 20 yinzhengjie dao }
[root@node105.yinzhengjie.org.cn ~]# tail -100f /home/data/flume/log/flume-my_host_interceptor.log #查看日志信息
1[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/job/flume-my_timestamp_interceptor.conf 2# Name the components on this agent 3my_timestamp_interceptor.sources = netcat_source 4my_timestamp_interceptor.sinks = logger_sink 5my_timestamp_interceptor.channels = memory_channel 6 7# Describe/configure the source 8my_timestamp_interceptor.sources.netcat_source.type = netcat 9my_timestamp_interceptor.sources.netcat_source.bind = node105.yinzhengjie.org.cn 10my_timestamp_interceptor.sources.netcat_source.port = 8888 11 12# 指定添加拦截器 13my_timestamp_interceptor.sources.netcat_source.interceptors = i1 14my_timestamp_interceptor.sources.netcat_source.interceptors.i1.type = timestamp 15 16# Describe the sink 17my_timestamp_interceptor.sinks.logger_sink.type = logger 18 19# Use a channel which buffers events in memory 20my_timestamp_interceptor.channels.memory_channel.type = memory 21my_timestamp_interceptor.channels.memory_channel.capacity = 100000 22my_timestamp_interceptor.channels.memory_channel.transactionCapacity = 10000 23 24# Bind the source and sink to the channel 25my_timestamp_interceptor.sources.netcat_source.channels = memory_channel 26my_timestamp_interceptor.sinks.logger_sink.channel = memory_channel 27[root@node105.yinzhengjie.org.cn ~]# 28[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/job/flume-my_timestamp_interceptor.conf #编写时间戳拦截器配置文件（案例二）
1[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/shell/start-my_timestamp_interceptor.sh 2#!/bin/bash 3#@author :yinzhengjie 4#blog:http://www.cnblogs.com/yinzhengjie 5#EMAIL:y1053419035@qq.com 6#Data:Thu Oct 18 11:26:06 CST 2018 7 8#将监控数据发送给ganglia，需要指定ganglia服务器地址，使用请确认是否部署好ganglia服务！ 9#nohup flume-ng agent -c /home/data/flume/job --conf-file=/home/data/flume/job/flume-my_timestamp_interceptor.conf --name my_timestamp_interceptor -Dflume.monitoring.type=ganglia -Dflume.monitoring.hosts=node105.yinzhengjie.org.cn:8649 -Dflume.root.logger=INFO,console > 10> /home/data/flume/log/flume-ganglia-flume-my_timestamp_interceptor.log 2>&1 & 11 12#启动flume自身的监控参数，默认执行以下脚本 13nohup flume-ng agent -c /home/data/flume/job --conf-file=/home/data/flume/job/flume-my_timestamp_interceptor.conf --name my_timestamp_interceptor -Dflume.monitoring.type=http -Dflume.monitoring.port=10521 -Dflume.root.logger=INFO,console >> /home/data/flume/log/flume- 14my_timestamp_interceptor.log 2>&1 &[root@node105.yinzhengjie.org.cn ~]# 15[root@node105.yinzhengjie.org.cn ~]# 16[root@node105.yinzhengjie.org.cn ~]# /home/data/flume/shell/start-my_timestamp_interceptor.sh 17[root@node105.yinzhengjie.org.cn ~]# 18[root@node105.yinzhengjie.org.cn ~]# ss -ntl 19State Recv-Q Send-Q Local Address:Port Peer Address:Port 20LISTEN 0 128 *:22 *:* 21LISTEN 0 50 172.30.1.105:8888 *:* 22LISTEN 0 50 *:10521 *:* 23LISTEN 0 128 :::22 :::* 24[root@node105.yinzhengjie.org.cn ~]# 25[root@node105.yinzhengjie.org.cn ~]#
[root@node105.yinzhengjie.org.cn ~]# cat /home/data/flume/shell/start-my_timestamp_interceptor.sh #编写启动脚本并启动
1[root@node105.yinzhengjie.org.cn ~]# tail -100f /home/data/flume/log/flume-my_timestamp_interceptor.log 2Warning: JAVA_HOME is not set! 3Info: Including Hadoop libraries found via (/home/softwares/hadoop-2.6.0/bin/hadoop) for HDFS access 4Info: Including Hive libraries found via () for Hive access 5+ exec /home/softwares/jdk1.8.0_201/bin/java -Xmx20m -Dflume.monitoring.type=http -Dflume.monitoring.port=10521 -Dflume.root.logger=INFO,console -cp '/home/data/flume/job:/home/softwares/apache-flume-1.9.0-bin/lib/*:/home/softwares/hadoop-2.6.0/etc/hadoop:/home/software 6s/hadoop-2.6.0/share/hadoop/common/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/common/*:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/hdfs/*:/home/softwares/hadoop-2.6.0/share/hadoop/yarn/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/yarn/*:/home/softwares/hadoop-2.6.0/share/hadoop/mapreduce/lib/*:/home/softwares/hadoop-2.6.0/share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar:/lib/*' -Djava.library.path=:/home/softwares/hadoop-2.6.0/lib/native org.apache.flume.node.Application --conf-file=/home/data/flume/job/flume-my_timestamp_interceptor.conf --name my_timestamp_interceptorSLF4J: Class path contains multiple SLF4J bindings. 7SLF4J: Found binding in [jar:file:/home/softwares/apache-flume-1.9.0-bin/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] 8SLF4J: Found binding in [jar:file:/home/softwares/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] 9SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 10SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 1119/07/20 23:25:03 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 1219/07/20 23:25:03 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/home/data/flume/job/flume-my_timestamp_interceptor.conf 1319/07/20 23:25:03 INFO conf.FlumeConfiguration: Processing:netcat_source 1419/07/20 23:25:03 INFO conf.FlumeConfiguration: Processing:memory_channel 1519/07/20 23:25:03 INFO conf.FlumeConfiguration: Added sinks: logger_sink Agent: my_timestamp_interceptor 1619/07/20 23:25:03 INFO conf.FlumeConfiguration: Processing:logger_sink 1719/07/20 23:25:03 INFO conf.FlumeConfiguration: Processing:netcat_source 1819/07/20 23:25:03 INFO conf.FlumeConfiguration: Processing:memory_channel 1919/07/20 23:25:03 INFO conf.FlumeConfiguration: Processing:netcat_source 2019/07/20 23:25:03 INFO conf.FlumeConfiguration: Processing:netcat_source 2119/07/20 23:25:03 INFO conf.FlumeConfiguration: Processing:logger_sink 2219/07/20 23:25:03 INFO conf.FlumeConfiguration: Processing:netcat_source 2319/07/20 23:25:03 INFO conf.FlumeConfiguration: Processing:memory_channel 2419/07/20 23:25:03 INFO conf.FlumeConfiguration: Processing:netcat_source 2519/07/20 23:25:03 WARN conf.FlumeConfiguration: Agent configuration for 'my_timestamp_interceptor' has no configfilters. 2619/07/20 23:25:04 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [my_timestamp_interceptor] 2719/07/20 23:25:04 INFO node.AbstractConfigurationProvider: Creating channels 2819/07/20 23:25:04 INFO channel.DefaultChannelFactory: Creating instance of channel memory_channel type memory 2919/07/20 23:25:04 INFO node.AbstractConfigurationProvider: Created channel memory_channel 3019/07/20 23:25:04 INFO source.DefaultSourceFactory: Creating instance of source netcat_source, type netcat 3119/07/20 23:25:04 INFO sink.DefaultSinkFactory: Creating instance of sink: logger_sink, type: logger 3219/07/20 23:25:04 INFO node.AbstractConfigurationProvider: Channel memory_channel connected to [netcat_source, logger_sink] 3319/07/20 23:25:04 INFO node.Application: Starting new configuration:{ sourceRunners:{netcat_source=EventDrivenSourceRunner: { source:org.apache.flume.source.NetcatSource{name:netcat_source,state:IDLE} }} sinkRunners:{logger_sink=SinkRunner: { policy:org.apache.flume.sin 34k.DefaultSinkProcessor@64de9bea counterGroup:{ name:null counters:{} } }} channels:{memory_channel=org.apache.flume.channel.MemoryChannel{name: memory_channel}} }19/07/20 23:25:04 INFO node.Application: Starting Channel memory_channel 3519/07/20 23:25:04 INFO node.Application: Waiting for channel: memory_channel to start. Sleeping for 500 ms 3619/07/20 23:25:04 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: memory_channel: Successfully registered new MBean. 3719/07/20 23:25:04 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: memory_channel started 3819/07/20 23:25:04 INFO node.Application: Starting Sink logger_sink 3919/07/20 23:25:04 INFO node.Application: Starting Source netcat_source 4019/07/20 23:25:04 INFO source.NetcatSource: Source starting 4119/07/20 23:25:04 INFO util.log: Logging initialized @1344ms to org.eclipse.jetty.util.log.Slf4jLog 4219/07/20 23:25:04 INFO source.NetcatSource: Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/172.30.1.105:8888] 4319/07/20 23:25:04 INFO server.Server: jetty-9.4.6.v20170531 4419/07/20 23:25:04 INFO server.AbstractConnector: Started ServerConnector@47d7718d{HTTP/1.1,[http/1.1]}{0.0.0.0:10521} 4519/07/20 23:25:04 INFO server.Server: Started @1597ms 4619/07/20 23:25:34 INFO sink.LoggerSink: Event: { headers:{timestamp=1563636333779} body: 79 69 6E 7A 68 65 6E 67 6A 69 65 20 64 61 6F 20 yinzhengjie dao }
[root@node105.yinzhengjie.org.cn ~]# tail -100f /home/data/flume/log/flume-my_timestamp_interceptor.log #测试方法和上述一样，测试后查看时间戳日志信息