Centralized log server
Server for receiving logs in legacy BSD format
ElasticSearch
Install Java dependency (java 6 or newer)apt-get install default-jre -yGet Elasticsearch .deb from: http://www.elasticsearch.org/download/ and install it + dependencies
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.7.deb dpkg -i elasticsearch-0.90.7.deb apt-get install -f
Logstash
Logstash comes with a ready to run monolithic jar file, but I prefer .deb package including init script and sample configs since I find .deb more familiar to deploy and upgrade (E.G using Puppet)Install dependencies for creating .deb
apt-get install git rubygems -y gem install fpmCreate .deb:
git clone https://github.com/Yuav/logstash-packaging.git --depth=1 cd logstash-packaging ./package.shInstall:
cd .. dpkg -i logstash_1.2.2.debThis will install logstash init scripts and sample config. For a quick test to see if it's working, try starting logstash and access the web interface on port 9292 after it's done spinning up.
At this point, you might want to optimize ElasticSearch to minimize storage footprint, depending on your setup.
Also, you probably want to install Kibana as a web frontend, which is due to replace the default web interface in logstash core at a later time.
Thanks for posting this. Much easier than doing it all manually on RHEL/CentOS. There is one minor issue with the RC script though. It should have -c "$LS_USER":"$LS_GROUP", instead of "LS_USER":"$LS_USER".
ReplyDeleteMake sure to install git: apt-get install git
ReplyDeleteAfter the command 'git clone https://github.com/Yuav/logstash-packaging.git' type:
apt-get install rubygems
gem install fpm
Verify the path to the .deb package. My host dumped it in /.
dpkg -i /logstash_1.1.9.deb
I have that same problem - I need to find out where it put it
DeleteThis comment has been removed by the author.
ReplyDeleteubuntu /etc/init/elasticsearch.conf
ReplyDelete# ElasticSearch Service
description "ElasticSearch"
start on (net-device-up
and local-filesystems
and runlevel [2345])
stop on runlevel [016]
respawn limit 10 5
env ES_HOME=/usr/share/elasticsearch/home
env ES_MIN_MEM=256m
env ES_MAX_MEM=2g
env DAEMON="${ES_HOME}/bin/elasticsearch"
env DATA_DIR=/data/elasticsearch/data
env CONFIG_DIR=/etc/elasticsearch
console output
script
if [ -f /etc/default/elasticsearch ]; then
. /etc/default/elasticsearch
fi
su -s /bin/dash -c "/usr/share/elasticsearch/bin/elasticsearch -f -Des.path.conf=$CONFIG_DIR -Des.path.home=$ES_HOME -Des.path.logs=$LOG_DIR -Des.path.data=$DATA_DIR -Des.path.work=$WORK_DIR" elasticsearch
end script
You should remove the default-jre dependency. I have already installed openjdk-7-jre-headless
ReplyDeleteRemoved dependency from .deb package now :-)
DeleteHi, I google'd this page, thanks for the excellent info! One thing I noticed for my box is that it was spinning up a logstash that seems to be putting syslog in elasticsearch but it didn't start up a logstash end-user web interface on port 9292.
ReplyDeleteI still had to manually start a web interface logstash with: "java -jar /usr/share/logstash/logstash.jar web --backend elasticsearch://127.0.0.1/"
Nevermind, I had an error in my config I discovered.
DeleteThis doesnt seem to enable the web gui by default.
ReplyDeleteI tried adding "web" to the startup script, but it didnt seem to fix it.
How exactly do you edit the script to also start the web gui?
Nick, I modified the following line in /etc/init.d/logstash:
ReplyDeleteDAEMON_OPTS="agent -f ${CONF} --log ${LOG_FILE} -- web"
adding the -- web at the end
I need some help - 1st I can not find the .deb 2nd - how to get this working...
ReplyDelete