What is the ELK stack ? and how to install it ?
The ELK stack is the combination of 3 tools :
This stack is supported by the elastic company. ELK is well known for centralising logs in infrastrutures. That mainly helps ops and dev to debug applications. They can analyse logs and do statistics with different vizulisation (charts, dashboards…).
But ELK is also well known to process data. Many data scientists use these softwares to manipulate a lot of datas.
What’s Elasticsearch ?
Elasticsearch is a nosql engine. The structure of this database type is the document. One document can store many fields. For example, in ecommerce, you can have a document for a product or a customer.
The strong point of ES is especially the fulltext search. Because Elasticsearch uses the Lucene library and an inverted index.
A cluster of elasticsearch nodes can be very wide due to the easy to scale up. ES allows you to distribute your data across the cluster with sharding. Sharding partition your database in segments and spread it on the cluster.
For the high availability, ES allows you to replicate your data (shard replication).
What’s Logstash ?
Logstash is an extract, transform and load software. So with it, you can collect data with many inputs (filebeat, postgresql, kafka, files, http). After the collect Logstash helps you to transform the data if necessary. For example, you can split your data or you can add the ip geolocalisation…
And finally you can send your data in many outputs like elasticsearch but also in kafka, mongodb…
What’s kibana ?
Kibana is the graphical user interface for elasticsearch (not for logstash). With kibana you can manage your elasticsearch cluster (only some tasks not all). You configure your cluster with kibana instead of the command line and yaml files. But Kibana helps you to analyse your datas with many visualization tools. For example you can create charts and dashboards. But you can also create maps or just query all documents in the elasticsearch cluster.
How to install the ELK stack ?
In our example, we install ELK on a debian server. But you could do the same logic for a redhat server.
First of all we add the elastic official repository :
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-7.x.list
sudo apt-get update
After that we can install elasticsearch :
sudo vim /etc/elasticsearch/jvm.options
sudo systemctl restart elasticsearch.service
sudo systemctl enable elasticsearch.service
Note : you can edit the JVM options (-Xmx and -Xms) to dicrease the ram consumption.
And to check you can send a curl on port 9200. The API of Elasticsearch listens on this port.
Secondly, we can install logstash :
sudo apt-get install logstash
sudo systemctl start logstash.service
sudo systemctl enable logstash.service
And finally we install the GUI kibana :
sudo apt-get install kibana
sudo systemctl status kibana.service
sudo vim /etc/kibana/kibana.yml
# change server.host to "0.0.0.0" or "external ip"
sudo systemctl start kibana.service
sudo systemctl enable kibana.service
Be careful you need to edit the configuration file in /etc/kibana/kibana.yml to change the line with the server.host parameter. Set the ip with 0.0.0.0 to listen on all interfaces.