Deploy Elasticsearch, Fluentd and Kibana on Kubernetes
EFK Stack or Elasticseacrh, Fluentd and Kibana is a opensource for a centralized logging solution.
This module/application on this stack is have a specific role. The role of the applications is like this:
Elasticsearch is a real-time, distributed and scalable searc engine based on apache-lucent. It’s commonly used to index and search through large volumes of log data, but can also be used to search many different kinds of documents.
Usually Elasticseacrch is pairing with Logstasg, but this time i will use fluentd to collect, transform and ship log adata to elasticsearch backend. Fluentd is one of the CNCF project that graduated.
And Kibana is a visualization of the elasticsearch.
My environment uses 4 CPU cores and 8GB of RAM per node.
1 master nodes. 2 worker nodes. 200GB volume attached to worker nodes as a Persistent Volume. You can use your kubernetes environment. I would suggest you use 4 CPUs and 8GB of RAM per node with 1 Master and 2 Worker or more.
Kubernetes 1.10+ kubectl client
All of the YAML using in this posts is available at https://github.com/sdmoko/EFK-kubernetes
- Clone the repository on the master node
git clone https://github.com/sdmoko/EFK-kubernetes.git cd EFK-kubernetes
- Create a storage class, in this tutorial i’m using a local disk as a storage.
kubectl apply -f 00-storageClass.yaml
- Create a PV. Please check and change the size of pv and values for a worker node that have a storage disk.
kubectl apply -f 01-pv-0.yaml kubectl apply -f 02-pv-1.yaml kubectl apply -f 03-pv-2.yaml
- Then we need to create a namespaces for a logging system.
kubectl apply -f 04-namespaces.yaml
- After this create a service for elasticsearch cluster, so the Fluentd and Kibana can use it.
kubectl apply -f 05-elasticsearch_svc.yaml
- Then create a cluster elasticsearch which use the volume that we create before.
kubectl apply -f 06-elasticsearch_statefulset.yaml
- Then deploy Kibana and Fluentd to make it a complete logging stack. Where we can visualize the data.
kubectl apply -f 07-kibana.yaml kubectl apply -f 08-fluentd.yaml
Cheers, Moko :)
Syah Dwi Prihatmoko
Pelupa, makanya harus mencatat.