Thursday, January 17, 2013

Installing Logstash as syslog server on Ubuntu Server 12.04 LTS

Centralized log server

Server for receiving logs in legacy BSD format

ElasticSearch

Install Java dependency (java 6 or newer)
apt-get install default-jre -y
Get Elasticsearch .deb from: http://www.elasticsearch.org/download/ and install it + dependencies
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.7.deb
dpkg -i elasticsearch-0.90.7.deb
apt-get install -f

Logstash

Logstash comes with a ready to run monolithic jar file, but I prefer .deb package including init script and sample configs since I find .deb more familiar to deploy and upgrade (E.G using Puppet)

Install dependencies for creating .deb
 apt-get install git rubygems -y
 gem install fpm
Create .deb:
git clone https://github.com/Yuav/logstash-packaging.git --depth=1
cd logstash-packaging
./package.sh
Install:
cd ..
dpkg -i logstash_1.2.2.deb
This will install logstash init scripts and sample config. For a quick test to see if it's working, try starting logstash and access the web interface on port 9292 after it's done spinning up.

At this point, you might want to optimize ElasticSearch to minimize storage footprint, depending on your setup.

Also, you probably want to install Kibana as a web frontend, which is due to replace the default web interface in logstash core at a later time.

12 comments:

  1. Thanks for posting this. Much easier than doing it all manually on RHEL/CentOS. There is one minor issue with the RC script though. It should have -c "$LS_USER":"$LS_GROUP", instead of "LS_USER":"$LS_USER".

    ReplyDelete
  2. Make sure to install git: apt-get install git

    After the command 'git clone https://github.com/Yuav/logstash-packaging.git' type:

    apt-get install rubygems
    gem install fpm

    Verify the path to the .deb package. My host dumped it in /.

    dpkg -i /logstash_1.1.9.deb

    ReplyDelete
    Replies
    1. I have that same problem - I need to find out where it put it

      Delete
  3. This comment has been removed by the author.

    ReplyDelete
  4. ubuntu /etc/init/elasticsearch.conf


    # ElasticSearch Service

    description "ElasticSearch"

    start on (net-device-up
    and local-filesystems
    and runlevel [2345])

    stop on runlevel [016]

    respawn limit 10 5

    env ES_HOME=/usr/share/elasticsearch/home
    env ES_MIN_MEM=256m
    env ES_MAX_MEM=2g
    env DAEMON="${ES_HOME}/bin/elasticsearch"
    env DATA_DIR=/data/elasticsearch/data
    env CONFIG_DIR=/etc/elasticsearch

    console output

    script

    if [ -f /etc/default/elasticsearch ]; then
    . /etc/default/elasticsearch
    fi

    su -s /bin/dash -c "/usr/share/elasticsearch/bin/elasticsearch -f -Des.path.conf=$CONFIG_DIR -Des.path.home=$ES_HOME -Des.path.logs=$LOG_DIR -Des.path.data=$DATA_DIR -Des.path.work=$WORK_DIR" elasticsearch

    end script

    ReplyDelete
  5. You should remove the default-jre dependency. I have already installed openjdk-7-jre-headless

    ReplyDelete
  6. Hi, I google'd this page, thanks for the excellent info! One thing I noticed for my box is that it was spinning up a logstash that seems to be putting syslog in elasticsearch but it didn't start up a logstash end-user web interface on port 9292.

    I still had to manually start a web interface logstash with: "java -jar /usr/share/logstash/logstash.jar web --backend elasticsearch://127.0.0.1/"

    ReplyDelete
  7. This doesnt seem to enable the web gui by default.
    I tried adding "web" to the startup script, but it didnt seem to fix it.
    How exactly do you edit the script to also start the web gui?

    ReplyDelete
  8. Nick, I modified the following line in /etc/init.d/logstash:

    DAEMON_OPTS="agent -f ${CONF} --log ${LOG_FILE} -- web"

    adding the -- web at the end

    ReplyDelete
  9. I need some help - 1st I can not find the .deb 2nd - how to get this working...

    ReplyDelete