flume 版本

flume 1.9.0使用指导，兼容flume1.8.0
引用：https://flume.liyifeng.org/
flume 指定版本下载，推荐使用1.8.0
引用：https://archive.apache.org/dist/flume/1.8.0/apache-flume-1.8.0-bin.tar.gz

分布式flume 架构图指导

flume 降低带宽使用率

（Avro是支持二进制序列化方式，可以便捷， 压缩，快速地处理大量数据，可持久化存储；
动态语言友好，Avro提供的机制使动态语言可以方便地处理Avro数据）
指导文档：https://flume.liyifeng.org/#avro-sink
使用Avro Source 方式跨机房传输的时候，流量压力、费用很多
使flume Avro Sink 参数开启 compression-type = deflate（无损数据压缩/压缩类型必须与上一级Avro Source 配置的一致）, compression-level = 6
总带宽使用率节省了50%以上，压缩传输速率问题通过增加Sink的方式解决，但会适当增加CPU和内存的损耗。

flume agent 启动

bin/flume-ng agent --conf-file example.conf --name a1 -Dflume.monitoring.type=http -Dflume.monitoring.port=34545

k8s 中使用 flume_exporter (监控系统：prometheus)

引用：https://github.com/woozhijun/flume_exporter

添加 configmap

ls config
# 以下文件在git主目录，看情况修改配置
config.yml  metrics.yml

kubectl create configmap flume-exporter-configs --from-file=config

k8s 多容器 yaml

kind: Deployment
apiVersion: apps/v1
spec:
  template:
    metadata:
      labels:
# 用于应用筛选
        app: tomcat
# 用于 exporter 自动发现
      annotations:
        prometheus.io/exporter: flume-exporter
        prometheus.io/path: metrics
        prometheus.io/port: '9360' 
    spec:
      volumes:
# 配置托管至configmap
        - name: flume-config
          configMap:
            name: flume-exporter-configs    
      containers:
        - name: flume-exporter
          image: 'zhijunwoo/flume_exporter:latest'
# 多flume容器多端口情况, 修改flume metric 连接地址
          command:
            - sh
            - '-c'
            - >-
              sed 's/localhost/'$MY_POD_NAME'/g' 
              /etc/flume_exporter/config.yml.back >
              /etc/flume_exporter/config.yml && /bin/flume_exporter
              --metric-file=/etc/flume_exporter/metrics.yml
              --config-file=/etc/flume_exporter/config.yml
          ports:
            - name: flume-exporter
              containerPort: 9360
              protocol: TCP
          env:
            - name: MY_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.podIP
            - name: MY_POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: PORT
              value: '9360'
          resources:
            limits:
              cpu: 1000m
              memory: 1024Mi
            requests:
              cpu: 500m
              memory: 512Mi
          volumeMounts:
            - name: flume-config
              mountPath: /etc/flume_exporter/config.yml.back
              subPath: config.yml
            - name: flume-config
              mountPath: /etc/flume_exporter/metrics.yml
              subPath: metrics.yml

prometheus 自动发现

global:
  scrape_interval:     15s
  evaluation_interval: 15s
scrape_configs:
- job_name: 'k8s_flume_exporter'
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_exporter]
    action: keep
    regex: flume-exporter
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name
  - source_labels: [__meta_kubernetes_pod_container_name, __meta_kubernetes_pod_container_port_number]
    target_label: k8s_pod_container_name
    regex: (.*);(\d+)
    action: replace
    replacement: $1:$2
  - source_labels: [__meta_kubernetes_service_name]
    action: replace
    replacement: "prod-k8s"
    target_label: cluster