Log Management systems comparison¶

Background¶

In order to fulfil our need for logging solution we need to be able to effectively collect and store logs from applications and systems deployed in Kubernetes as well as in EC2, Lambda and so on.

Previous experiments with k8s logging showed that collecting all logs produced in even medium sized k8s cluster leads to significant resource utilisation(CPU, storage, network throughput) and high costs. Thus logging solution must have functionality is selective log collection, which means one can easily enable/disable log collection on k8s workload whenever it's needed. The only fully fledged solution that fits this requirement as of today is Grafana Loki(Grafana - Loki), because it is able, alike Prometheus, to collect logs based on Pod labels.

Unfortunately Loki is more of troubleshooting tool and doesn't provide most of the functionality that Splunk, Graylog and ELK do.

Since Loki is lightweight, cost effective and cloud native solution we can consider having Loki in each k8s cluster(like with Prometheus) for troubleshooting purposes.

For long term, indexed and searchable log solution one of ELK or Graylog needs to be chosen(Splunk is enterprise and not considered as an option).

Worth to mention that there is also possibility to research Loki's log client called Promtail that can selectively collect logs based on Pod labels, and, if practical, modify it to send logs to ELK/Graylog instead, thus eliminating Loki from our infra.

Filebeat autodiscovery functionality also needs to be considered as an option to collect only needed logs: Configuration Autodiscover (This has been tested, functionality is included in OSS version of filebeat). Filebeat can be used for both ELK and Graylog.

Main requirements for centralised logging solution¶

1. Opensource
2. Has scalable HA log backend
3. Has dashboarding UI
4. Supports multiple inputs to accept logs
5. Easy to configure and maintain

ELK¶

ELK/EFK is built upon the following components:

• Elasticsearch (Lucene based distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents)
• Logstash/Filebeat (Log parsing/sending clients)
• Kibana (Dashboarding solution)

ELK has very powerful functionality both in terms of storing logs and graphing/dashboarding, although configuration and maintenance can be difficult, also Kibana interface is quite comprehensive and doesn't have basic dashboards preconfigured.

Pros:

• Very powerfull Kibana visualizations
• Scalable (ElasticSearch)
• Flexible data retention (indicies management in ElasticSearch)
• Plenty of input plugins available

Cons:

• Difficult to configure and maintain
• Kibana has steep learning curve and no default dashboards
• Kibana alerting feature is paid
• Elastic tends to move more and more features to paid section(recent conflict with AWS)

Still ELK can perfectly fit as logging solution since we have enough experience with it.

Graylog¶

Graylog like ELK stores logs in Elasticsearch but also relies on MongoDB. Graylog has much narrower functionality in terms of visualisations. Thus in case we would want to do more than just search though logs(like create World map visualization using Geo data in our logs) that will require to add other solutions. Its main functions like searching logs, different configurations(also logs processing and storage not only visualisations) it does very well and most available via UI.

Pros:

• Simple setup and maintenance
• AuthN/AuthZ out of the box
• Parsing, alerting and basic graphing out of the box and configurable via UI
• Scalable(ElasticSearch, MongoDB)

Cons:

• Limited functionality (mostly in terms of visualisations)
• Fewer plugins available than for ELK

Thus Graylog does very well what it was designed for - storing, parsing and searching logs.

Comparison¶