Add fluent 2 ES

Da Helpedia.

Install Fluent

curl -L http://toolbelt.treasuredata.com/sh/install-redhat.sh | sh

Attenzione: makes also a yum update!

Install Plugins

/usr/lib64/fluent/ruby/bin/fluent-gem install  fluent-plugin-elasticsearch
/usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-tail-ex

configure for apache

vi /etc/td-agent/td-agent.conf

<source>
  type tail_ex
  path /var/log/httpd/checkip02-access_log #...or where you placed your Apache access log
  pos_file /var/log/td-agent/httpd-access.log.pos # This is where you record file position
  tag ${hostname} #fluentd tag!
  #format apache2 # Do you have a custom format? You can write your own regex.
  format /^(?<clientip>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$/
time_format %d/%b/%Y:%H:%M:%S %z
</source>

<match **>
  type elasticsearch
  host 172.21.15.23 #(optional; default="localhost")
  logstash_format true
  type_name fluent_apache
  include_tag_key true
  tag_key hostname
  #logstash_prefix mylogs # defaults to "logstash"
  #port <port> #(optional; default=9200)
  #index_name <index name> #(optional; default=fluentd)
  #type_name <type name> #(optional; default=fluentd)
</match>


l'accesso ai log da parte di td-agent richiede:

chmod 645 /var/log/httpd/

Reduce log level of td.agent to ERROR

   -v, --verbose                    increase verbose level (-v: debug, -vv: trace)
   -q, --quiet                      decrease verbose level (-q: warn, -qq: error)


edit /etc/init.d/td-agent add -qq to TD_AGENT_ARGS restart td-agent

:::term # at /etc/init.d/td-agent … TD_AGENT_ARGS=“… -vv” …
Strumenti personali
Namespace
Varianti
Azioni
Navigazione
Strumenti