# IX PIWEEK: monitoring with ELK (I)

Yesterday started the Piweek IX, 9 times showing what a small company can do about innovation. Twice a year. You can follow us on twitter.

For this edition, I left aside my tradicional taste for Arduino and wanted to try a well known technology for monitorization (ELK). And I was very happy to know that my colleague Alex wanted to be part of the experiment.

As usual, I’m trying to build some useful information out from my learning process, so here you have a serie of posts about how to run ELK (Elasticsearch, Logstash, Kibana).

This post covers a brief explanation of each component of ELK and the instructions to have it up&running; in its simpliest mode. I’ve chosen avoid the installation process and use an amazing [docker image]().

## Assumptions

This tutorial assumes that you work fine with the command line, Python and its ecosystem are not new to you, and you know basic concepts about logs. All the commands work in (my) ArchLinux, you may change commands to adapt to your operating system.

## What’s ELK?

Logstash: (site) is an application which reads several log sources, applies transformations (filters) and returns original logs with the transformations. Elasticsearch: with Elasticsearch we store the logs (after Logstash transformation); besides it’s a search engine, so we can create complex queries to filter the documents. Kibana: (site) is a front-web application to visualize those logs in a fancy dashboard, built with different visual components.

## Docker

Installation

We could install the 3 components from scratch, and there are many tutorials about this topic, but the goal of this specific tutorial is focused on the usage, so we’re gonna use a docker image, with almost everything ready.

Install docker (system package) and start the service

yami $sudo pacman -S docker yami$ sudo systemctl start docker


After installation, we add our user to the group docker

yami $sudo usermod -aG docker yami  Now, create a python3 virtualenv, where we’re going to install docker-compose yami$ mkvirtualenv monitoralia
(monitoralia) yami $pip install docker-compose  Clone the repository (thanks to deviantony): yami$ git clone https://github.com/deviantony/docker-elk.git
yami $ls docker-elk LICENSE README.md docker-compose.yml elasticsearch kibana logstash  After installation, we’re going to review some configurations in docker-compose.yml file. In this file, we set the options to run the services. Elasticsearch configuration elasticsearch: image: elasticsearch:latest command: elasticsearch -Des.network.host=0.0.0.0 ports: - "9200:9200" - "9300:9300"  • use the latest image • with command we configure the parameters to run the service • ports: we map our local ports with ports in docker container Logstash configuration logstash: image: logstash:latest command: logstash agent --config /etc/logstash/conf.d/ volumes: - ./logstash/config:/etc/logstash/conf.d - ./taiga-logs:/var/log/taiga ports: - "5000:5000" links: - elasticsearch  • use the latest image • with command we configure to run logstash reading configuration files in this folder (/etc/logstash/conf.d/) • volumes: we need to configure in docker the different folders that we’ll use in logstash • on one hand, configuration file. Here we configure where are the sources for the logs, we create transformations and the path to send the logs to their corresponding storage. • on the other hand, we need to make docker know where are we storing logs. In our case, we see just a folder for logs • ports: we map our local ports with ports in docker container • links: here we say that we are going to need connection from logstash to elasticsearch Kibana configuration kibana: build: kibana/ volumes: - ./kibana/config/kibana.yml:/opt/kibana/config/kibana.yml ports: - "5601:5601" links: - elasticsearch  • instead of using an existing image, we’re going to build a new one • we set the volumes where kibana will find its configuration files • ports: we map our local ports with ports in docker container Run docker Once we’ve done previous steps, we can launch docker: (monitoralia) yami$ docker-compose up


Now you can go to http://localhost:5601 and you’ll find the front web of kibana; you’ll see tons of options with few sense right now. We’ll see more detail in the following posts.