1. input-file Collect log information

 [yun@mini04 config]$ pwd
/app/logstash/config
[yun@mini04 config]$ cat file.conf
input{
file{
path => ["/var/log/messages", "/var/log/secure"]
type => "system-log"
start_position => "beginning"
}
} filter{
} output{
# es Yes 3 platform , Just choose any one It can also be multiple, such as ["127.0.0.1:9200","127.0.0.2:9200"]
elasticsearch {
hosts => ["mini01:9200", "mini02:9200", "mini03:9200"]
index => "system-log-%{+YYYY.MM}"
}
} ##################################################
[root@mini04 ~]# /app/logstash/bin/logstash -f /app/logstash/config/file.conf # start-up Need here root Only when the user starts , Otherwise, there is no permission
…………

1.1. Browser access

2. input-if Judge 【 Log multi-point collection 】

For convenience , I put logstatsh Deployed to mini03 On

The purpose of this section is : collect java journal 【 There are some flaws in the log collection , Inconvenient to view , The configuration needs to be improved 】

 [yun@mini03 config]$ pwd
/app/logstash/config
[yun@mini03 config]$ cat file2.conf
input{
file{
path => ["/var/log/messages", "/var/log/secure"]
type => "system-log"
start_position => "beginning"
} file{
path => ["/app/es-data/logs/zhang-es.log"]
type => "es-log"
start_position => "beginning"
}
} filter{
} output{
# es Yes 3 platform , Just choose any one It can also be multiple, such as ["127.0.0.1:9200","127.0.0.2:9200"]
if [type=] == "system-log" {
elasticsearch {
hosts => ["mini01:9200", "mini02:9200", "mini03:9200"]
index => "system-log-%{+YYYY.MM}"
}
} if [type] == "es-log" {
elasticsearch {
hosts => ["mini01:9200", "mini02:9200", "mini03:9200"]
index => "es-log-%{+YYYY.MM}"
}
}
} ##################################
[root@mini04 ~]# /app/logstash/bin/logstash -f /app/logstash/config/file2.conf # start-up Need here root Only when the user starts , Otherwise, there is no permission
…………

Browser access

 http://mini01:9100/ # head visit 

 http://mini01:5601 # kibana visit 

shortcoming :

java There are errors in the application log , So it's not convenient to collect it directly

3. codec-multiline—— Multi line merge collection 【 Can be used for java Program log collection 】

3.1. Command line I / O test

Multi line merge , With [ Start as a match

 # The configuration file 
[yun@mini03 config]$ pwd
/app/logstash/config
[yun@mini03 config]$ cat codec_test.conf
# Enter the configuration description :
# pattern => "^\[" matching [ Beginning line ;
# negate => "true" If not, put them together ;
# what => "previous" If it is "previous" Express , Anyone who doesn't [ The first line should be merged with the previous line .
# If "next" Express , Any [ The ending lines should be merged with the following lines .
input{
stdin{
codec => multiline {
pattern => "^\["
negate => "true"
what => "previous"
}
}
} filter{
} output{
stdout{
codec => rubydebug
}
}
# perform
[yun@mini03 config]$ /app/logstash/bin/logstash -f /app/logstash/config/codec_test.conf # perform
……………… [
{
"host" => "mini03",
"message" => "1111\n222\n333",
"@version" => "",
"tags" => [
[] "multiline"
],
"@timestamp" => --25T06::.486Z
} [
{
"host" => "mini03",
"message" => "[444\n555\n666\n8888",
"@version" => "",
"tags" => [
[] "multiline"
],
"@timestamp" => --25T06::.319Z
}

3.2. Collect again ES journal

3.2.1. stay ES Delete previously collected on mini03  ES journal

stop it mini03 Upper logstash Program

3.2.2. Delete logstash The tag

The plug-in works through a sincedb And record the current location in each file to track the current location . This will stop and restart Logstash, And keep it running where it ends , And don't miss out on log The line added to the file when it comes out .

 # Find the tag file 
[yun@mini03 logstash]$ pwd
/app/logstash
[yun@mini03 logstash]$ find . -type f | grep 'sincedb'
./data/plugins/inputs/file/.sincedb_1fb922e15ccea4ac0d028d33639ba3ea
./data/plugins/inputs/file/.sincedb_56a0ba191c6aa2202fcdc058933e33b0
##### mini03 es Log information
[yun@mini03 logs]$ pwd
/app/es-data/logs
[yun@mini03 logs]$ ll -i zhang-es.log
-rw-rw-r-- yun yun Aug : zhang-es.log # The first column is es Of inode Information
##### logstash sincedb File information for
[yun@mini03 file]$ pwd
/app/logstash/data/plugins/inputs/file
[yun@mini03 file]$ ll -a
total
drwxr-xr-x yun yun Aug : .
drwxr-xr-x yun yun Aug : ..
-rw-r--r-- yun yun Aug : .sincedb_1fb922e15ccea4ac0d028d33639ba3ea
-rw-r--r-- yun yun Aug : .sincedb_56a0ba191c6aa2202fcdc058933e33b0
[yun@mini03 file]$ cat .sincedb_56a0ba191c6aa2202fcdc058933e33b0 [yun@mini03 file]$ rm -f .sincedb_56a0ba191c6aa2202fcdc058933e33b0 # Delete es Of sincedb file

explain : among 33588216 Corresponding es The log inode Information , So delete .sincedb_56a0ba191c6aa2202fcdc058933e33b0 file , Then collect again es When the log , We'll start collecting again

3.2.3. logstash Configure and start

 [yun@mini03 config]$ pwd
/app/logstash/config
[yun@mini03 config]$ cat codec.conf
input{
file{
path => ["/var/log/messages", "/var/log/secure"]
type => "system-log"
start_position => "beginning"
} file{
path => ["/app/es-data/logs/zhang-es.log"]
type => "es-log"
start_position => "beginning"
codec => multiline {
pattern => "^\["
negate => "true"
what => "previous"
}
}
} filter{
} output{
# es Yes 3 platform , Just choose any one It can also be multiple, such as ["127.0.0.1:9200","127.0.0.2:9200"]
if [type=] == "system-log" {
elasticsearch {
hosts => ["mini01:9200", "mini02:9200", "mini03:9200"]
index => "system-log-%{+YYYY.MM}"
}
} if [type] == "es-log" {
elasticsearch {
hosts => ["mini01:9200", "mini02:9200", "mini03:9200"]
index => "es-log-%{+YYYY.MM}"
}
}
} #### Use root Permission start , Because there are "/var/log/messages", "/var/log/secure" Log collection
[root@mini03 ~]# /app/logstash/bin/logstash -f /app/logstash/config/codec.conf &

3.2.4. Browser pass kibana see

adopt kibana Inquire about , I learned that the logs collected this time really conform to our browsing habits .

4. codec-json【 collect Nginx Access log 】

Need to put Nginx The access log of is changed to json Format

4.1. nginx Partial log configuration

stay mini03  yum install Nginx

[root@mini03 ~]# vim /etc/nginx/nginx.conf
# For more information on configuration, see:
# * Official English Documentation: http://nginx.org/en/docs/
# * Official Russian Documentation: http://nginx.org/ru/docs/ user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid; # Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf; events {
worker_connections 1024;
} http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
# New configuration , The above configuration is not referenced , So you can ignore
# Bear in mind : Don't change lines *****
log_format access_log_json '{"user_ip":"$http_x_real_ip","lan_ip":"$remote_addr","log_time":"$time_iso8601","user_req":"$request","http_code":"$status","body_bytes_sent":"$body_bytes_sent","req_time":"$request_time","user_ua":"$http_user_agent"}'; # access_log /var/log/nginx/access.log main; # notes
access_log /var/log/nginx/access_log_json.log access_log_json; # newly added

4.2. logstash Configure and start

 [yun@mini03 config]$ pwd
/app/logstash/config
[yun@mini03 config]$ cat codec_json.conf
input{ file{
path => ["/var/log/nginx/access_log_json.log"]
type => "nginx-access-log"
codec => json
}
} filter{
} output{
# es Yes 3 platform , Just choose any one It can also be multiple, such as ["127.0.0.1:9200","127.0.0.2:9200"]
elasticsearch {
hosts => ["mini01:9200", "mini02:9200", "mini03:9200"]
index => "nginx-access-log-%{+YYYY.MM.dd}"
} } ##### need root jurisdiction , because Nginx yes yum Installed The access log is in /var/log/nginx/access_log_json.log in
[root@mini03 ~]# /app/logstash/bin/logstash -f /app/logstash/config/codec_json.conf &

4.3. Browser access Nginx

The access method is as follows :

 http://mini03/32t23t23t/ee # You can get 404 Status code 
  • stay mini01、mini02、mini03 Access through the following command
 # Need to install software 
yum -y install httpd-tools
# The access command is as follows
ab -n10 -c http://mini03/
ab -n10 -c http://mini03/aa/bbb/ccc # In order to get 404 Status code

4.4. Information view

adopt head see

adopt kibana see

5. input-rsyslog Log collection

requirement : collect mini01、mini02、mini03 Of rsyslog journal

5.1. rsyslog Log collection test

logstash To configure

 [yun@mini03 config]$ pwd
/app/logstash/config
[yun@mini03 config]$ cat rsyslog_test.conf
input{
syslog{
type => "system-rsyslog"
port =>
}
} filter{
} output{
stdout{
codec => rubydebug
}
} ##### Use root user , Otherwise, there are limits of authority
[root@mini03 ~]# /app/logstash/bin/logstash -f /app/logstash/config/rsyslog_test.conf

mini01、mini02、mini03 Configuration modification

 [root@mini01 ~]# tail -n5 /etc/rsyslog.conf # mini01、mini02、mini03
# remote host is: name/ip:port, e.g. 192.168.0.1:, port optional
#*.* @@remote-host:
# The configuration to be added below
*.* @@172.16.1.13: # ### end of the forwarding rule ###
[root@mini01 ~]# systemctl restart rsyslog.service # restart rsyslog

stay mini03 Of logstash On , It can be seen that rsyslog Brush it .

5.2. rsyslog Collect to ES To configure

among mini01、mini02、mini03 The configuration on has been modified as above , So there's no need to change .

logstash To configure

 [yun@mini03 config]$ pwd
/app/logstash/config
[yun@mini03 config]$ cat rsyslog.conf
input{
syslog{
type => "system-rsyslog"
port =>
}
} filter{
} output{
# es Yes 3 platform , Just choose any one It can also be multiple, such as ["127.0.0.1:9200","127.0.0.2:9200"]
elasticsearch {
hosts => ["mini01:9200", "mini02:9200", "mini03:9200"]
index => "system-rsyslog-%{+YYYY.MM}"
} } ##### Use root user , Otherwise, there are limits of authority
[root@mini03 ~]# /app/logstash/bin/logstash -f /app/logstash/config/rsyslog.conf &

5.3. Browser view

adopt head see

adopt kibana see

6. input-tcp collect

This time it's just a test , They don't collect ES 了 .

6.1. logstash To configure

 [yun@mini03 config]$ pwd
/app/logstash/config
[yun@mini03 config]$ cat tcp_test.conf
input{
tcp {
port =>
mode => "server"
type => "tcp_test"
}
} filter{
} output{
stdout{
codec => rubydebug
}
} ##########################
[yun@mini03 ~]$ /app/logstash/bin/logstash -f /app/logstash/config/tcp_test.conf # You can use ordinary users

6.2. stay mini02 test

 [yun@mini02 ~]$ echo "" | nc mini03
[yun@mini02 ~]$ echo "testinfo" | nc mini03
[yun@mini02 ~]$ nc mini03 < /etc/resolv.conf
[yun@mini02 ~]$ echo "myinfo" > /dev/tcp/mini03/

stay mini03 You can see , The command line has logstash Information output of

7. filter-Grok

The production environment is hardly used

reason :

1、grok It has a great impact on performance

2、 inflexible

Best practices : Separation , Attend to each one's own duties

 logstash => redis/kafka => logstash/python => ES

7.1. see grok Location and file

 [yun@mini03 patterns]$ pwd
/app/logstash/vendor/bundle/jruby/2.3./gems/logstash-patterns-core-4.1./patterns
[yun@mini03 patterns]$ ll
total
-rw-r--r-- yun yun Jul : aws
-rw-r--r-- yun yun Jul : bacula
-rw-r--r-- yun yun Jul : bind
-rw-r--r-- yun yun Jul : bro
-rw-r--r-- yun yun Jul : exim
-rw-r--r-- yun yun Jul : firewalls
-rw-r--r-- yun yun Jul : grok-patterns
-rw-r--r-- yun yun Jul : haproxy
-rw-r--r-- yun yun Jul : httpd
-rw-r--r-- yun yun Jul : java
-rw-r--r-- yun yun Jul : junos
-rw-r--r-- yun yun Jul : linux-syslog
-rw-r--r-- yun yun Jul : maven
-rw-r--r-- yun yun Jul : mcollective
-rw-r--r-- yun yun Jul : mcollective-patterns
-rw-r--r-- yun yun Jul : mongodb
-rw-r--r-- yun yun Jul : nagios
-rw-r--r-- yun yun Jul : postgresql
-rw-r--r-- yun yun Jul : rails
-rw-r--r-- yun yun Jul : redis
-rw-r--r-- yun yun Jul : ruby
-rw-r--r-- yun yun Jul : squid

7.2. Command line test

 [yun@mini03 config]$ pwd
/app/logstash/config
[yun@mini03 config]$
[yun@mini03 config]$ cat filter-grok_test.conf
input{
stdin{}
} filter{
grok {
match => { "message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" }
}
} output{
stdout{
codec => rubydebug
}
} #######################################
[yun@mini03 ~]$ /app/logstash/bin/logstash -f /app/logstash/config/filter-grok_test.conf # Use ordinary users
……………………
# Enter the following line of string
55.3.244.1 GET /index.html 0.043
{
"@version" => "",
"host" => "mini03",
"bytes" => "",
"message" => "55.3.244.1 GET /index.html 15824 0.043",
"client" => "55.3.244.1",
"duration" => "0.043",
"request" => "/index.html",
"@timestamp" => --28T13::.910Z,
"method" => "GET"
}

7.3. httpd Log collection command line test

[yun@mini03 config]$ pwd
/app/logstash/config
[yun@mini03 config]$ cat filter-grok_httpd-test.conf
input{
file{
path => ["/var/log/httpd/access_log"]
type => "httpd-access-log"
start_position => "beginning"
}
} filter{
grok {
match => { "message" => "%{HTTPD_COMBINEDLOG}" }
}
} output{
stdout{
codec => rubydebug
}
} ################# Use root user , It's about authority
[root@mini03 ~]# /app/logstash/bin/logstash -f /app/logstash/config/filter-grok_httpd-test.conf
……………………
# so httpd Our logs are collected , And it's parsed
{
"path" => "/var/log/httpd/access_log",
"referrer" => "\"http://mini03/\"",
"host" => "mini03",
"response" => "200",
"message" => "10.0.0.1 - - [28/Aug/2018:22:35:31 +0800] \"GET /images/poweredby.png HTTP/1.1\" 200 3956 \"http://mini03/\" \"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36\"",
"auth" => "-",
"timestamp" => "28/Aug/2018:22:35:31 +0800",
"bytes" => "3956",
"clientip" => "10.0.0.1",
"agent" => "\"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36\"",
"@version" => "1",
"@timestamp" => 2018-08-28T14:44:12.477Z,
"httpversion" => "1.1",
"type" => "httpd-access-log",
"ident" => "-",
"request" => "/images/poweredby.png",
"verb" => "GET"
}
………………

7.4. httpd Collect logs to ES

 [yun@mini03 config]$ pwd
/app/logstash/config
[yun@mini03 config]$ cat filter-grok_httpd.conf
input{
file{
path => ["/var/log/httpd/access_log"]
type => "httpd-access-log"
start_position => "beginning"
}
} filter{
grok {
match => { "message" => "%{HTTPD_COMBINEDLOG}" }
}
} output{
# es Yes 3 platform , Just choose any one It can also be multiple, such as ["127.0.0.1:9200","127.0.0.2:9200"]
elasticsearch {
hosts => ["mini01:9200", "mini02:9200", "mini03:9200"]
index => "httpd-access-log-%{+YYYY.MM.dd}"
}
} ########## Use root user , Involving authority
[root@mini03 ~]# /app/logstash/bin/logstash -f /app/logstash/config/filter-grok_httpd.conf
………………

7.5. Browser access httpd

browser

 # Through Google 、 firefox 、IE visit 
http://mini03/
http://mini03/indweg.html

Linux Command line access

 [yun@mini02 ~]$ ab -n40 -c http://mini03/
[yun@mini02 ~]$ ab -n40 -c http://mini03/wet/bdhw/

7.6. Information view

head visit

kibana see

ELK-logstash-6.3.2- More articles on common configurations

  1. ELK——Logstash 2.2 date plug-in unit 【 translate + practice 】

    Official website address The content of this article is grammar Test data Configurable options Reference material date The plug-in is the date plug-in , This plugin , Common but important . If not date plug-in unit , that Logstash Using the processing time as a timestamp . The timestamp field is Log ...

  2. ELK logstash Handle MySQL Slow query log ( preliminary )

    Write it at the front : Doing it ELK logstash Handle MySQL Slow query log when the problem : 1. There are no slow logs in the test database , So there's no log information , Lead to IP:9200/_plugin/head/ The interface is abnormal ( The number of logs suddenly appears ...

  3. ELK+SpringBoot+Logback Offline installation and configuration

    ELK+SpringBoot+Logback Offline installation and configuration edition v1.0 Time of writing 2018/6/11 Prepared by xxx     Catalog One . ELK Introduce 2 Two . Installation environment 2 3、 ... and . Elasticse ...

  4. logback General configuration details &lt;appender&gt;

    logback General configuration details  <appender> <appender>: <appender> yes <configuration> Child nodes of , I'm in charge of the journal ...

  5. 【 turn 】logback logback.xml General configuration details ( 3、 ... and ) &lt;filter&gt;

    Original article , Reprint please indicate the source :http://aub.iteye.com/blog/1110008, To respect others is to respect oneself Detailed arrangement of logback Common configuration , It's not a translation of the official website manual , It's using the summary , Designed to be faster and more transparent ...

  6. 【 turn 】logback logback.xml General configuration details ( Two )&lt;appender&gt;

    Original article , Reprint please indicate the source :http://aub.iteye.com/blog/1101260, To respect others is to respect oneself Detailed arrangement of logback Common configuration , It's not a translation of the official website manual , It's using the summary , Designed to be faster and more transparent ...

  7. 【 turn 】logback logback.xml General configuration details ( One )&lt;configuration&gt; and &lt;logger&gt;

    Original article , Reprint please indicate the source :http://aub.iteye.com/blog/1101260, To respect others is to respect oneself Detailed arrangement of logback Common configuration , It's not a translation of the official website manual , It's using the summary , Designed to be faster and more transparent ...

  8. 【 turn 】logback General configuration details ( order )logback brief introduction

    Original article , Reprint please indicate the source :http://aub.iteye.com/blog/1101222, To respect others is to respect oneself Detailed arrangement of logback Common configuration , It's not a translation of the official website manual , It's using the summary , Designed to be faster and more transparent ...

  9. logback General configuration details ( Two ) &lt;appender&gt;

    logback General configuration details ( Two ) <appender> <appender>: <appender> yes <configuration> Child nodes of , Is responsible for writing ...

  10. SpringBoot Introduction to common configurations

    SpringBoot Introduction to common configurations 1. SpringBoot A brief introduction to several commonly used configurations in A simple Spring.factories # Bootstrap components org.springf ...

Random recommendation

  1. Huang Cong :C# prohibit Webbrowser Script error prompt in , Automatically block pop ups

    using System; using System.Collections.Generic; using System.Text; using System.Windows.Forms; using ...

  2. js Expand more

    var introduces = { inIt : function(){ introduces.imgLoad(); introduces.showMore(0,'hioh',86); introd ...

  3. mysql Find a table with a specific name

    SELECT distinct TABLE_NAME FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME LIKE '%medias%'

  4. Linux Realization SSH Password-free login ( The setting of directory permissions is very detailed , You can refer to it )

    Hypothetical server IP The address is 192.168.1.1, machine name :cluster.hpc.org client IP The address is 172.16.16.1, machine name :p470-2.wangrx.sioc.ac.cn Client users yzha ...

  5. Huawei S5700 Basic configuration ---- Backup and restore configuration files

    One : Backup configuration files Equipment as FTP The server , user PC As FTP client # Configure the device's FTP Function and FTP User information . <HUAWEI> system-view [HUAWEI] ftp serve ...

  6. Android Fragment In the real sense onResume and onPause

    Fragment Although there are onResume and onPause Of , But the two methods are Activity Methods , The timing of the call is also related to Activity identical , and ViewPager It's very difficult to use this method together , It's not what you want ...

  7. In depth understanding of php kernel Write extension _III- resources

    original text :http://devzone.zend.com/article/1024-Extension-Writing-Part-III-Resources Write extension _III- resources Introduce resources Initial capital ...

  8. 032 Java To sum up again

    1. The outline How to use multithreading , Thread pool , Business , Communication mechanism and isolation level , Reflection , Generic , The difference between database engines , database merge, Window function ,fastJson,JVM tuning ,GC hook ,Linux Of awk,shell,HashM ...

  9. centos7 Lower installation docker(13.1docker Storage --data volume)

    We now know docker There are two ways to store :storage driver and data volume stroage driver This storage method is mainly used to store those stateless data , It's made up of the image layer and the container layer , and data ...

  10. log4j Introduction and use

    1.apache Launched the open source free log processing class library 2. Why you need a journal ?? 2.1 Write in project system.out.println(); Output to console , When the project is released to tomcat after , No console ( In the command interface, you can see ...