可锐资源网

技术资源分享平台，提供编程学习、网站建设、脚本开发教程

2025年10月23日

ELK运维之路(Filebeat第一章-7.17.24)

1. 部署Filebeat

1.1 下载解压Filebeat

# 下载filebeat并解压到自己习惯的目录
root@ubuntu2204test99:~# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.17.24-linux-x86_64.tar.gz
root@ubuntu2204test99:~# mv filebeat-7.17.24-linux-x86_64 filebeat-7.17.24
root@ubuntu2204test99:~# mv filebeat-7.17.24 /usr/local/
root@ubuntu2204test99:~# mkdir /usr/local/filebeat-7.17.24/{logs,data}

1.2 使用Systemctl接管Filebeat服务

# 配置systemd启动filebeat
vi /lib/systemd/system/filebeat.service
[Unit]
Description=filebeat is a lightweight shipper for metrics.
Documentation=https://www.elastic.co/products/beats/filebeat
Wants=network-online.target
After=network-online.target
[Service]
Environment="BEAT_LOG_OPTS=-e"
Environment="BEAT_CONFIG_OPTS=-c /usr/local/filebeat-7.17.24/filebeat.yml"
Environment="BEAT_PATH_OPTS=-path.home /usr/local/filebeat-7.17.24 -path.config /usr/local/filebeat-7.17.24 -path.data /usr/local/filebeat-7.17.24/data -path.logs /usr/local/filebeat-7.17.24/logs"
ExecStart=/usr/local/filebeat-7.17.24/filebeat $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS
Restart=always
[Install]
WantedBy=multi-user.target


# 测试启动
root@ubuntu2204k8s220:~# systemctl daemon-reload
root@ubuntu2204k8s220:~# systemctl start filebeat.service
root@ubuntu2204k8s220:~# systemctl enable filebeat.service

# 查看启动状态
root@ubuntu2204test99:~# systemctl status filebeat.service
● filebeat.service - filebeat is a lightweight shipper for metrics.
     Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2025-10-09 09:55:56 UTC; 14s ago
       Docs: https://www.elastic.co/products/beats/filebeat
   Main PID: 542553 (filebeat)
      Tasks: 9 (limit: 9388)
     Memory: 32.4M
        CPU: 258ms
     CGroup: /system.slice/filebeat.service
             └─542553 /usr/local/filebeat-7.17.24/filebeat -e -c /usr/local/filebeat-7.17.24/filebeat.yml -path.home /usr/local/filebeat-7.17.24 -path.config /usr/local>

Oct 09 09:55:59 ubuntu2204test99 filebeat[542553]: 2025-10-09T09:55:59.685Z        INFO        memlog/store.go:119        Loading data file of '/usr/local/filebeat-7.17>
Oct 09 09:55:59 ubuntu2204test99 filebeat[542553]: 2025-10-09T09:55:59.685Z        INFO        memlog/store.go:124        Finished .....
lines 1-21/21 (END)

2. Filebeat默认配置

root@ubuntu2204test99:~# egrep -v "^*#|^#34; /usr/local/filebeat-7.17.24/filebeat.yml
filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: false
  paths:
    - /var/log/*.log
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.elasticsearch:
  hosts: ["localhost:9200"]
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

3.采集日志示例

将内容直接输出到ES当中

3.1 采集常规格式的Nginx日志

3.1.1 常规格式日志

root@ubuntu2204test99:~# cat nginx_log/nginx01.log
192.168.1.170 - - [28/Oct/2024:10:53:40 +0800] "GET /static/js/plugins/select2/select2.full.min.js HTTP/1.1" 200 78556 "http://jumpweb.muscledog.ink/core/auth/login/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:131.0) Gecko/20100101 Firefox/131.0" "-"
192.168.1.170 - - [28/Oct/2024:10:53:40 +0800] "GET /static/js/plugins/markdown-it.min.js HTTP/1.1" 200 47613 "http://jumpweb.muscledog.ink/core/auth/login/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:131.0) Gecko/20100101 Firefox/131.0" "-"
192.168.1.170 - - [28/Oct/2024:10:53:40 +0800] "GET /static/js/plugins/select2/select2.full.min.js HTTP/1.1" 200 78556 "http://jumpweb.muscledog.ink/core/auth/login/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:131.0) Gecko/20100101 Firefox/131.0" "-"
192.168.1.170 - - [28/Oct/2024:10:53:40 +0800] "GET /static/js/plugins/markdown-it.min.js HTTP/1.1" 200 47613 "http://jumpweb.muscledog.ink/core/auth/login/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:131.0) Gecko/20100101 Firefox/131.0" "-"

3.1.2 Filebeat配置

root@ubuntu2204test99:/usr/local/filebeat-7.17.24# cat filebeat-nginxlog-base.yml
filebeat.inputs:
- type: log
  enable: true
  tags: ["nginx"]
  paths:
    - /root/nginx_log/nginx01.log
  fields:
    python: true
    java: false
  fields_under_root: false
output.elasticsearch:
  hosts: ["http://192.168.1.100:9200","http://192.168.1.101:9200","http://192.168.1.102:9200"]
  username: "" # 账号
  password: "" # 密码
  index: "filebeat-nginxlog-base-%{+yyyy.MM.dd}"

setup.ilm.enabled: false

setup.template.name: "filebeat-nginxlog-base"
setup.template.pattern: "filebeat-nginxlog-base*"

setup.template.overwrite: false
setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 1
  
# 临时启动filebeat
root@ubuntu2204test99:/usr/local/filebeat-7.17.24# pwd
/usr/local/filebeat-7.17.24
root@ubuntu2204test99:/usr/local/filebeat-7.17.24# ./filebeat -e -c filebeat-nginxlog-base.yml --path.data /tmp/filebeat01/

image-20251016155656555

image-20251016145413084

image-20251016155802221

image-20251016155828824

image-20251016155942995

3.2 采集Json格式的Nginx日志

这里有一个前提，需要提前将Nginx的日志格式修改为Json，如果不知道如何修改可以参考网上

3.2.1 Nginx Json日志格式

# Nginx Json格式日志示例
root@ubuntu2204test99:/usr/local/filebeat-7.17.24# cat /root/nginx_log/nginxjson.log
{"@timestamp":"2024-11-26T15:47:34+08:00","@source":"192.168.1.39","@nginx_fields":{"http_x_forwarded_for":"","request":"POST /api/v4/jobs/request HTTP/1.1","status":"204","body_bytes_sent":"0","http_referer":"","client":"192.168.1.47","request_time":"0.001","upstream_response_time":"0.002","upstream_addr":"192.168.1.40:10080","request_method":"POST","domain":"gitlab.muscledog.top","url":"/api/v4/jobs/request","args":"","request_body":"{\"info\":{\"name\":\"gitlab-runner\",\"version\":\"17.3.1\",\"revision\":\"66269445\",\"platform\":\"linux\",\"architecture\":\"amd64\",\"executor\":\"shell\",\"shell\":\"bash\",\"features\":{\"variables\":true,\"image\":false,\"services\":false,\"artifacts\":true,\"cache\":true,\"fallback_cache_keys\":true,\"shared\":true,\"upload_multiple_artifacts\":true,\"upload_raw_artifacts\":true,\"session\":true,\"terminal\":true,\"refspecs\":true,\"masking\":true,\"proxy\":false,\"raw_variables\":true,\"artifacts_exclude\":true,\"multi_build_steps\":true,\"trace_reset\":true,\"trace_checksum\":true,\"trace_size\":true,\"vault_secrets\":true,\"cancelable\":true,\"return_exit_code\":true,\"service_variables\":false,\"service_multiple_aliases\":false,\"image_executor_opts\":false,\"service_executor_opts\":false,\"cancel_gracefully\":true},\"config\":{\"gpus\":\"\"}},\"token\":\"glrt-BetamQqihBBxfUghD5p6\",\"system_id\":\"s_9a48451a140f\",\"last_update\":\"ac9b5d563d539d7a3a08029a60d796ff\"}","http_user_agent":"gitlab-runner 17.3.1 (17-3-stable; go1.22.5; linux/amd64)","remote_addr":"192.168.1.47","proxy_add_x_forwarded_for":"192.168.1.47"}}
{"@timestamp":"2024-11-26T15:47:37+08:00","@source":"192.168.1.39","@nginx_fields":{"http_x_forwarded_for":"","request":"POST /api/v4/jobs/request HTTP/1.1","status":"204","body_bytes_sent":"0","http_referer":"","client":"192.168.1.47","request_time":"0.001","upstream_response_time":"0.002","upstream_addr":"192.168.1.40:10080","request_method":"POST","domain":"gitlab.muscledog.top","url":"/api/v4/jobs/request","args":"","request_body":"{\"info\":{\"name\":\"gitlab-runner\",\"version\":\"17.3.1\",\"revision\":\"66269445\",\"platform\":\"linux\",\"architecture\":\"amd64\",\"executor\":\"shell\",\"shell\":\"bash\",\"features\":{\"variables\":true,\"image\":false,\"services\":false,\"artifacts\":true,\"cache\":true,\"fallback_cache_keys\":true,\"shared\":true,\"upload_multiple_artifacts\":true,\"upload_raw_artifacts\":true,\"session\":true,\"terminal\":true,\"refspecs\":true,\"masking\":true,\"proxy\":false,\"raw_variables\":true,\"artifacts_exclude\":true,\"multi_build_steps\":true,\"trace_reset\":true,\"trace_checksum\":true,\"trace_size\":true,\"vault_secrets\":true,\"cancelable\":true,\"return_exit_code\":true,\"service_variables\":false,\"service_multiple_aliases\":false,\"image_executor_opts\":false,\"service_executor_opts\":false,\"cancel_gracefully\":true},\"config\":{\"gpus\":\"\"}},\"token\":\"glrt-BetamQqihBBxfUghD5p6\",\"system_id\":\"s_9a48451a140f\",\"last_update\":\"ac9b5d563d539d7a3a08029a60d796ff\"}","http_user_agent":"gitlab-runner 17.3.1 (17-3-stable; go1.22.5; linux/amd64)","remote_addr":"192.168.1.47","proxy_add_x_forwarded_for":"192.168.1.47"}}

3.2.2 Filebeat配置

root@ubuntu2204test99:/usr/local/filebeat-7.17.24# cat filebeat-nginxlog-json.yml
filebeat.inputs:
- type: log
  enable: true
  tags: ["nginxjson"]
  paths:
    - /root/nginx_log/nginxjson.log
  fields:
    python: true
    java: false
  fields_under_root: false
  json.keys_under_root: true #对Json格式的日志进行解析并放在顶级字段,如果不是json格式会有大量报错

setup.ilm.enabled: false
output.elasticsearch:
  hosts: ["http://192.168.1.99:9201","http://192.168.1.99:9202","http://192.168.1.99:9203"]
  username: "elastic" # 账号
  password: "123456" # 密码
  index: "filebeat-nginxlog-json-%{+yyyy.MM.dd}"

setup.template.name: "filebeat-nginxlog-json"
setup.template.pattern: "filebeat-nginxlog-json*"

setup.template.overwrite: false
setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 1
# 临时启动filebeat
root@ubuntu2204test99:/usr/local/filebeat-7.17.24# pwd
/usr/local/filebeat-7.17.24
root@ubuntu2204test99:/usr/local/filebeat-7.17.24# ./filebeat -e -c filebeat-nginxlog-json.yml --path.data /tmp/filebeat01/

可以看到json格式的日志，被解析后会非常清晰的进行展示，而不是像普通问个格式那样都堆在一块。目前我们公司在nginx日志方面是使用的json格式的。

3.3 使用Filebeat模块采集Nginx日志

Filebeat的模块会对Nginx日志进行序列化解析处理，然后再录入到ES，一般模块这种不太建议采用,仅做了解

3.3.1 修改模块的内容(需要先开启Filebeat对应模块)

root@ubuntu2204test99:/usr/local/filebeat-7.17.24# pwd
/usr/local/filebeat-7.17.24
root@ubuntu2204test99:/usr/local/filebeat-7.17.24# vi modules.d/nginx.yml
# Module: nginx
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-module-nginx.html

- module: nginx
  # Access logs
  access:
    enabled: true
    var.paths: ["/root/nginx_log/nginx01.log"]

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

  # Error logs
  error:
    enabled: false

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

  # Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
  ingress_controller:
    enabled: false

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

3.3.2 Filebeat模板

root@ubuntu2204test99:/usr/local/filebeat-7.17.24# cat filebeat-nginxlog-modules.yml
# 加载模块配置
filebeat.config.modules:
  path: /usr/local/filebeat-7.17.24/modules.d/*.yml
  reload.enabled: false
  #reload.period: 10s
output.elasticsearch:
  hosts: ["http://192.168.1.99:9201","http://192.168.1.99:9202","http://192.168.1.99:9203"]
  username: "elastic"
  password: "123456"
  index: "filebeat-nginxlog-modules-%{+yyyy.MM.dd}"
setup.ilm.enabled: false
setup.template.name: "filebeat-nginxlog-modules"
setup.template.pattern: "filebeat-nginxlog-modules*"
setup.template.overwrite: false
setup.template.settings:
  index.number_of_shards: 3
  index.number_of_replicas: 2

标签：jobs linux 

作者:keramox , 分类:技术教程 , 浏览:42 , 评论:0

控制面板
您好，欢迎到访网站！
  查看权限
网站分类
最新留言