Yesterday started the Piweek IX, 9 times showing what a small company can do about innovation. Twice a year. You can follow us on twitter.
For this edition, I left aside my tradicional taste for Arduino and wanted to try a well known technology for monitorization (ELK). And I was very happy to know that my colleague Alex wanted to be part of the experiment.
As usual, I'm trying to build some useful information out from my learning process, so here you have a serie of posts about how to run ELK (Elasticsearch, Logstash, Kibana).
This post covers a brief explanation of each component of ELK and the instructions to have it up&running; in its simpliest mode. I've chosen avoid the installation process and use an amazing docker image.
Assumptions
This tutorial assumes that you work fine with the command line, Python and its ecosystem are not new to you, and you know basic concepts about logs. All the commands work in (my) ArchLinux, you may change commands to adapt to your operating system.
What's ELK?
Logstash: (site) is an application which reads several log sources, applies transformations (filters) and returns original logs with the transformations. Elasticsearch: with Elasticsearch we store the logs (after Logstash transformation); besides it's a search engine, so we can create complex queries to filter the documents. Kibana: (site) is a front-web application to visualize those logs in a fancy dashboard, built with different visual components.
Docker
Installation
We could install the 3 components from scratch, and there are many tutorials about this topic, but the goal of this specific tutorial is focused on the usage, so we're gonna use a docker image, with almost everything ready.
Install docker (system package) and start the service
yami $ sudo pacman -S docker
yami $ sudo systemctl start docker
After installation, we add our user to the group docker
yami $ sudo usermod -aG docker yami
Now, create a python3 virtualenv, where we're going to install docker-compose
yami $ mkvirtualenv monitoralia
(monitoralia) yami $ pip install docker-compose
Clone the repository (thanks to deviantony):
yami $ git clone https://github.com/deviantony/docker-elk.git
yami $ ls docker-elk
LICENSE README.md docker-compose.yml elasticsearch kibana logstash
After installation, we're going to review some configurations in docker-compose.yml file. In this file, we set the options to run the services.
Elasticsearch configuration
elasticsearch:
image: elasticsearch:latest
command: elasticsearch -Des.network.host=0.0.0.0
ports:
- "9200:9200"
- "9300:9300"
- use the latest image
- with command we configure the parameters to run the service
- ports: we map our local ports with ports in docker container
Logstash configuration
logstash:
image: logstash:latest
command: logstash agent --config /etc/logstash/conf.d/
volumes:
- ./logstash/config:/etc/logstash/conf.d
- ./taiga-logs:/var/log/taiga
ports:
- "5000:5000"
links:
- elasticsearch
- use the latest image
- with
command
we configure to run logstash reading configuration files in this folder (/etc/logstash/conf.d/) - volumes: we need to configure in docker the different folders that we'll use in logstash
- on one hand, configuration file. Here we configure where are the sources for the logs, we create transformations and the path to send the logs to their corresponding storage.
- on the other hand, we need to make docker know where are we storing logs. In our case, we see just a folder for logs
- ports: we map our local ports with ports in docker container
- links: here we say that we are going to need connection from logstash to elasticsearch
Kibana configuration
kibana:
build: kibana/
volumes:
- ./kibana/config/kibana.yml:/opt/kibana/config/kibana.yml
ports:
- "5601:5601"
links:
- elasticsearch
- instead of using an existing image, we're going to build a new one
- we set the volumes where kibana will find its configuration files
- ports: we map our local ports with ports in docker container
Run docker
Once we've done previous steps, we can launch docker:
(monitoralia) yami $ docker-compose up
Now you can go to http://localhost:5601 and you'll find the front web of kibana; you'll see tons of options with few sense right now. We'll see more detail in the following posts.