Logstash, ElasticSearch and Kibana

Overview

ข้อดีของมันคิอ ถ้ามีเครื่อง production หลายๆเครื่อง รัน load balance

จะหา log สามารถทำได้ง่าย แทนที่จะเข้าไปหาใน server load balance แต่ละอัน

Install dependency

apt-get install openjdk-7-jdk

Download

  • logstash-2.0.0 and extract
  • elasticsearch-2.0.0 and extract

NOTE: or we can use debian package

wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb http://packages.elastic.co/logstash/2.1/debian stable main" | sudo tee -a /etc/apt/sources.list

ref : https://www.elastic.co/guide/en/logstash/current/package-repositories.html

wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list

ref : https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-repositories.html

Run Elastic Search 

./bin/elasticsearch

logstash log to Elastic Search

bin/logstash -e 'input { stdin { } } output { elasticsearch { hosts => localhost } }'

Default settings used: Filter workers: 1
Logstash startup completed

Then type anything

Test the result

curl 'http://localhost:9200/_search?pretty'

{
"took" : 33,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 2,
"max_score" : 1.0,
"hits" : [ {
"_index" : "logstash-2015.11.17",
"_type" : "logs",
"_id" : "AVETlfOtUn7tncTXcx3F",
"_score" : 1.0,
"_source":{"message":"Then type anything","@version":"1","@timestamp":"2015-11-17T03:56:10.699Z","host":"pong-VirtualBox"}
}, {
"_index" : "logstash-2015.11.17",
"_type" : "logs",
"_id" : "AVETlfOtUn7tncTXcx3G",
"_score" : 1.0,
"_source":{"message":"","@version":"1","@timestamp":"2015-11-17T03:56:10.700Z","host":"pong-VirtualBox"}
} ]
}
}

get count

curl 'http://127.0.0.1:9200/_count?pretty'

Fetch the last row

curl 'http://localhost:9200/_search?sort=@timestamp:desc&size=1&pretty'


List all ElasticSearch index

http://localhost:9200/_cat/indices?v

Download
kibana-4.2.1-linux-x64

Configure config.js to point to elastic search ( default URL will be http://localhost:9200 )

./bin/kibana

auto refresh

kibana-4.2.1

Read the log

logstash-apache.conf

input {

file {

path => "/var/log/apache2/access.log"

start_position => beginning

}

}

filter {

if [path] =~ "access" {

mutate { replace => { "type" => "apache_access" } }

grok {

match => { "message" => "%{COMBINEDAPACHELOG}" }

}

}

date {

match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]

}

}

output {

elasticsearch {

hosts => localhost

}

stdout { codec => rubydebug }<span style="line-height: 1.7;">

NOTE : The pattern can be validated using http://grokdebug.herokuapp.com/

Test config file before run

bin/logstash --configtest -f logstash-apache.conf

Configuration OK

run

bin/logstash -f logstash-apache.conf

Default settings used: Filter workers: 1
Logstash startup completed

How the process look like

pong 10817 52.6 8.6 1477876 176976 pts/7 Sl+ 12:00 0:25 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Xmx500m -Xss2048k -Djffi.boot.library.path=/home/pong/VirtualBoxSharedFolder/logstash-2.0.0/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Xbootclasspath/a:/home/pong/VirtualBoxSharedFolder/logstash-2.0.0/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/home/pong/VirtualBoxSharedFolder/logstash-2.0.0/vendor/jruby -Djruby.lib=/home/pong/VirtualBoxSharedFolder/logstash-2.0.0/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /home/pong/VirtualBoxSharedFolder/logstash-2.0.0/lib/bootstrap/environment.rb logstash/runner.rb agent -f ../logstash-apache.conf

NOTE :

  • This conf contains 3 sections, input/filter/output
  • We can put it to /etc/logstash/conf.d/

and -f the whole directory

 bin/logstash -f conf.d

or you can start service

sudo service logstash start

This is how the process look like

logstash 10891 74.1 5.2 1437872 107212 pts/1 SNl 13:36 0:05 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/var/lib/logstash -Xmx500m -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/var/lib/logstash -XX:HeapDumpPath=/opt/logstash/heapdump.hprof -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log

or restart logstash with

sudo service logstash restart

ref : http://blog.redlinesoft.net/?p=1931

tail -f /var/log/logstash/logstash.log

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s