ELK--grok正则分析日志与多日志收集

一、收集nginx日志

logstash配置文件

访问nginx，产生新的日志

有一条logstash-2019-08-28的索引

二、grok日志切割

192.168.0.2 - - [28/Aug/2019:22:35:03 +0800] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko Core/1.70.3722.400 QQBrowser/10.5.3776.400"

grok正则匹配

(?<clientip>[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) - - (?<requesttime>\[[0-9]{1,2}\/[A-z]+\/[0-9]{4}\:[0-9]{2}\:[0-9]{2}\:[0-9]{2} \+[0-9]*\]) "(?<requesttype>[A-Z]+) (?<requesturl>[^ ]+) (?<requestv>HTTP/\d\.\d)" (?<requestnode>[0-9]+) (?<requestsize>[0-9]+) "(?<content>[^ ]|(http|https)://[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\/)" "(?<ua>(a-Z|0-9| |.)+)"

修改配置文件，增加grok正则匹配

input {

  file {

    path => "/usr/local/nginx/logs/access.log"

filter {

    grok {

        match => {

            "message" => '(?<clientip>[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) - - (?<requesttime>\[[0-9]{1,2}\/[A-z]+\/[0-9]{4}\:[0-9]{2}\:[0-9]{2}\:[0-9]{2} \+[0-9]*\]) "(?<requesttype>[A-Z]+) (?<requesturl>[^ ]+) (?<requestv>HTTP/\d\.\d)" (?<requestnode>[0-9]+) (?<requestsize>[0-9]+) "(?<content>[^ ]|(http|https)://[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\/)" "(?<ua>(a-Z|0-9| |.)+)"'

output {

  elasticsearch {

    hosts => ["http://192.168.0.9:9200"]

重载配置文件

ps -aux | grep logstash

kill -1 pid

三、删除不需要的字段

上面我们可以看到，很多字段其实是不完全需要的

修改配置文件并重载配置文件

         remove_field => ["message","log","beat","offset","prospector","host","@version"]

web界面显示

四、logstash分析完整日志

修改配置文件

    start_position => "beginning"

    sincedb_path => "/dev/null"

删除索引后重新加载配置文件，这样收集的日志将从头开始分析

五、多日志收集，并对索引进行定义

修改配置文件

input {

  file {

    path => "/usr/local/nginx/logs/access.log"

    type => "nginx"

    start_position => "beginning"

    sincedb_path => "/dev/null"

  file {

    path => "/var/log/secure"

    type => "secure"

    start_position => "beginning"

    sincedb_path => "/dev/null"

filter {

    grok {

        match => {

            "message" => '(?<clientip>[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) - - (?<requesttime>\[[0-9]{1,2}\/[A-z]+\/[0-9]{4}\:[0-9]{2}\:[0-9]{2}\:[0-9]{2} \+[0-9]*\]) "(?<requesttype>[A-Z]+) (?<requesturl>[^ ]+) (?<requestv>HTTP/\d\.\d)" (?<requestnode>[0-9]+) (?<requestsize>[0-9]+) "(?<content>[^ ]|(http|https)://[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\/)" "(?<ua>(a-Z|0-9| |.)+)"'

         remove_field => ["message","log","beat","offset","prospector","host","@version"]

output {

 if [type] == "nginx" {

  elasticsearch {

    hosts => ["http://192.168.0.9:9200"]

    index => "nginx-%{+YYYY.MM.dd}"

  else if [type] == "secure" {

    elasticsearch {

    hosts => ["http://192.168.0.9:9200"]

    index => "secure-%{+YYYY.MM.dd}"

删除索引，重载配置文件