作者：京东科技 刘恩浩
一、背景
基于K8s集群的私有化交付方案中，日志收集采用了ilogtail+logstash+kafka+es方案，其中ilogtail负责日志收集，logstash负责对数据转换，kafka负责对日志传递中的消峰进而减少es的写入压力，es用来保存日志数据。在私有化交付中本方案中涉及的中间件一般需要单独部署，但是在京东内网环境的部署考虑到kafka和es的高可用，则不推荐采用单独部署的方案。
二、新方案实践
1.新方案简介
在京东内网环境部署K8S并收集日志， kafka+es的替代方案考虑使用JMQ+JES，由于JMQ的底层是基于kafaka、JES的底层基于ES，所以该替换方案理论上是可行的
2.主要架构
数据流向大致如下
应用日志 -> ilogtail -> JMQ -> logstash -> JES
3.如何使用
核心改造点汇总
- ilogtail nameservers配置
增加解析JMQ域名的nameserver（京东云主机上无法直接解析.local域名）
1spec: 2 spec: 3 dnsPolicy: "None" 4 dnsConfig: 5 nameservers: 6 - x.x.x.x # 可以解析jmq域名的nameserver
- ilogtail flushers配置
调整发送到JMQ到配置
1apiVersion: v1 2kind: ConfigMap 3metadata: 4 name: ilogtail-user-cm 5 namespace: elastic-system 6data: 7 app_stdout.yaml: | 8 flushers: 9 - Type: flusher_stdout 10 OnlyStdout: true 11 - Type: flusher_kafka_v2 12 Brokers: 13 - nameserver.jmq.jd.local:80 # jmq元数据地址 14 Topic: ai-middle-k8s-log-prod # jmq topic 15 ClientID: ai4middle4log # Kafka的用户ID（识别客户端并设置其唯一性），对应jmq的Group名称，重要‼️ （https://ilogtail.gitbook.io/ilogtail-docs/plugins/input/service-kafka#cai-ji-pei-zhi-v2）
- logstash kafka&es配置
1apiVersion: v1 2kind: ConfigMap 3metadata: 4 name: logstash-config 5 namespace: elastic-system 6 labels: 7 elastic-app: logstash 8data: 9 logstash.conf: |- 10 input { 11 kafka { 12 bootstrap_servers => ["nameserver.jmq.jd.local:80"] #jmq的元数据地址 13 group_id => "ai4middle4log" # jmq的Group的名称 14 client_id => "ai4middle4log" # jmq的Group的名称，即jmq的省略了kafka中的client_id概念，用Group名称代替 15 consumer_threads => 2 16 decorate_events => true 17 topics => ["ai-middle-k8s-log-prod"] # jmp的topic 18 auto_offset_reset => "latest" 19 codec => json { charset => "UTF-8" } 20 } 21 } 22 output { 23 elasticsearch { 24 hosts => ["http://x.x.x.x:40000","http://x.x.x.x:40000","http://x.x.x.x:40000"] # es地址 25 index => "%{[@metadata][kafka][topic]}-%{+YYYY-MM-dd}" # 索引规则 26 user => "XXXXXX" #jes的用户名 27 password => "xxxxx" #jes的密码 28 ssl => "false" 29 ssl_certificate_verification => "false" 30 } 31 }
ilogtail 的配置如下
1# ilogtail-daemonset.yaml 2apiVersion: apps/v1 3kind: DaemonSet 4metadata: 5 name: ilogtail-ds 6 namespace: elastic-system 7 labels: 8 k8s-app: logtail-ds 9spec: 10 selector: 11 matchLabels: 12 k8s-app: logtail-ds 13 template: 14 metadata: 15 labels: 16 k8s-app: logtail-ds 17 spec: 18 dnsPolicy: "None" 19 dnsConfig: 20 nameservers: 21 - x.x.x.x # （京东云主机上）可以解析jmq域名的nameserver 22 tolerations: 23 - operator: Exists # deploy on all nodes 24 containers: 25 - name: logtail 26 env: 27 - name: ALIYUN_LOG_ENV_TAGS # add log tags from env 28 value: _node_name_|_node_ip_ 29 - name: _node_name_ 30 valueFrom: 31 fieldRef: 32 apiVersion: v1 33 fieldPath: spec.nodeName 34 - name: _node_ip_ 35 valueFrom: 36 fieldRef: 37 apiVersion: v1 38 fieldPath: status.hostIP 39 - name: cpu_usage_limit # iLogtail's self monitor cpu limit 40 value: "1" 41 - name: mem_usage_limit # iLogtail's self monitor mem limit 42 value: "512" 43 image: dockerhub.ai.jd.local/ai-middleware/ilogtail-community-edition/ilogtail:1.3.1 44 imagePullPolicy: IfNotPresent 45 resources: 46 limits: 47 cpu: 1000m 48 memory: 1Gi 49 requests: 50 cpu: 400m 51 memory: 384Mi 52 volumeMounts: 53 - mountPath: /var/run # for container runtime socket 54 name: run 55 - mountPath: /logtail_host # for log access on the node 56 mountPropagation: HostToContainer 57 name: root 58 readOnly: true 59 - mountPath: /usr/local/ilogtail/checkpoint # for checkpoint between container restart 60 name: checkpoint 61 - mountPath: /usr/local/ilogtail/user_yaml_config.d # mount config dir 62 name: user-config 63 readOnly: true 64 - mountPath: /usr/local/ilogtail/apsara_log_conf.json 65 name: apsara-log-config 66 readOnly: true 67 subPath: apsara_log_conf.json 68 dnsPolicy: ClusterFirst 69 hostNetwork: true 70 volumes: 71 - hostPath: 72 path: /var/run 73 type: Directory 74 name: run 75 - hostPath: 76 path: / 77 type: Directory 78 name: root 79 - hostPath: 80 path: /etc/ilogtail-ilogtail-ds/checkpoint 81 type: DirectoryOrCreate 82 name: checkpoint 83 - configMap: 84 defaultMode: 420 85 name: ilogtail-user-cm 86 name: user-config 87 - configMap: 88 defaultMode: 420 89 name: ilogtail-apsara-log-config-cm 90 name: apsara-log-config
1# ilogtail-user-configmap.yaml 2apiVersion: v1 3kind: ConfigMap 4metadata: 5 name: ilogtail-user-cm 6 namespace: elastic-system 7data: 8 app_stdout.yaml: | 9 enable: true 10 inputs: 11 - Type: service_docker_stdout 12 Stderr: true 13 Stdout: true 14 K8sNamespaceRegex: ai-train 15 ExternalK8sLabelTag: 16 platform/resource-name: k8s_label_resource-name 17 platform/task-identify: k8s_label_task-identify 18 task-id: k8s_label_task-id 19 run-id: k8s_label_run-id 20 request-id: k8s_label_request-id 21 processors: 22 - Type: processor_rename 23 SourceKeys: 24 - k8s_label_resource-name 25 - k8s_label_task-identify 26 - k8s_label_task-id 27 - k8s_label_run-id 28 - k8s_label_request-id 29 - _namespace_ 30 - _image_name_ 31 - _pod_uid_ 32 - _pod_name_ 33 - _container_name_ 34 - _container_ip_ 35 - __path__ 36 - _source_ 37 DestKeys: 38 - resource_name 39 - task_identify 40 - task_id 41 - run_id 42 - request_id 43 - namespace 44 - image_name 45 - pod_uid 46 - pod_name 47 - container_name 48 - container_ip 49 - path 50 - source 51 flushers: 52 - Type: flusher_stdout 53 OnlyStdout: true 54 - Type: flusher_kafka_v2 55 Brokers: 56 - nameserver.jmq.jd.local:80 # jmq元数据地址 57 Topic: ai-middle-k8s-log-prod # jmq topic 58 ClientID: ai4middle4log # Kafka的用户ID（识别客户端并设置其唯一性），对应jmq的Group名称，重要‼️ （https://ilogtail.gitbook.io/ilogtail-docs/plugins/input/service-kafka#cai-ji-pei-zhi-v2） 59 60 app_file_log.yaml: | 61 enable: true 62 inputs: 63 - Type: file_log 64 LogPath: /export/Logs/ai-dt-algorithm-tools 65 FilePattern: "*.log" 66 ContainerInfo: 67 K8sNamespaceRegex: ai-train 68 ExternalK8sLabelTag: 69 platform/resource-name: k8s_label_resource-name 70 platform/task-identify: k8s_label_task-identify 71 task-id: k8s_label_task-id 72 run-id: k8s_label_run-id 73 request-id: k8s_label_request-id 74 75 processors: 76 - Type: processor_add_fields 77 Fields: 78 source: file 79 - Type: processor_rename 80 SourceKeys: 81 - __tag__:k8s_label_resource-name 82 - __tag__:k8s_label_task-identify 83 - __tag__:k8s_label_task-id 84 - __tag__:k8s_label_run-id 85 - __tag__:k8s_label_request-id 86 - __tag__:_namespace_ 87 - __tag__:_image_name_ 88 - __tag__:_pod_uid_ 89 - __tag__:_pod_name_ 90 - __tag__:_container_name_ 91 - __tag__:_container_ip_ 92 - __tag__:__path__ 93 DestKeys: 94 - resource_name 95 - task_identify 96 - task_id 97 - run_id 98 - request_id 99 - namespace 100 - image_name 101 - pod_uid 102 - pod_name 103 - container_name 104 - container_ip 105 - path 106 107 flushers: 108 - Type: flusher_stdout 109 OnlyStdout: true 110 - Type: flusher_kafka_v2 111 Brokers: 112 - nameserver.jmq.jd.local:80 113 Topic: ai-middle-k8s-log-prod 114 ClientID: ai4middle4log
logstash 的配置如下
1# logstash-configmap.yaml 2--- 3apiVersion: v1 4kind: ConfigMap 5metadata: 6 name: logstash-config 7 namespace: elastic-system 8 labels: 9 elastic-app: logstash 10data: 11 logstash.conf: |- 12 input { 13 kafka { 14 bootstrap_servers => ["nameserver.jmq.jd.local:80"] #jmq的元数据地址 15 #group_id => "services" 16 group_id => "ai4middle4log" # jmq的Group的名称 17 client_id => "ai4middle4log" # jmq的Group的名称，即jmq的省略了kafka中的client_id概念，用Group名称代替 18 consumer_threads => 2 19 decorate_events => true 20 #topics_pattern => ".*" 21 topics => ["ai-middle-k8s-log-prod"] # jmp的topic 22 auto_offset_reset => "latest" 23 codec => json { charset => "UTF-8" } 24 } 25 } 26 27 28 filter { 29 ruby { 30 code => "event.set('index_date', event.get('@timestamp').time.localtime + 8*60*60)" 31 } 32 ruby { 33 code => "event.set('message',event.get('contents'))" 34 } 35 #ruby { 36 # code => "event.set('@timestamp',event.get('time').time.localtime)" 37 #} 38 39 mutate { 40 remove_field => ["contents"] 41 convert => ["index_date", "string"] 42 #convert => ["@timestamp", "string"] 43 gsub => ["index_date", "T.*Z",""] 44 #gsub => ["@timestamp", "T.*Z",""] 45 } 46 } 47 48 49 output { 50 elasticsearch { 51 #hosts => ["https://ai-middle-cluster-es-http:9200"] 52 hosts => ["http://x.x.x.x:40000","http://x.x.x.x:40000","http://x.x.x.x:40000"] # es地址 53 index => "%{[@metadata][kafka][topic]}-%{+YYYY-MM-dd}" # 索引规则 54 user => "XXXXXX" #jes的用户名 55 password => "xxxxx" #jes的密码 56 ssl => "false" 57 ssl_certificate_verification => "false" 58 #cacert => "/usr/share/logstash/cert/ca_logstash.cer" 59 } 60 stdout { 61 codec => rubydebug 62 } 63 }
4.核心价值
在私有化部署的基础上通过简单改造实现了与京东内部中间件的完美融合，使得系统在高可用性上适应性更强、可用范围更广。