Skip to main content

Cài đặt và cấu hình Elasticsearch, Logstash, and Kibana (ELK Stack) trên CentOS 7

12,September 2018
cài đặt và cấu hình filebeat logstash elasticsearch kibana

Logstash: Đây là một công cụ sử dụng để thu thập, xử lý log được viết bằng java. Nhiệm vụ chính của logstash là thu thập log sau đó chuyển vào Elastichsearch. Mỗi dòng log của logstash được lưu trữ đưới dạng json.

Elasticsearch: sử dụng cơ sở dữ liệu NoSQL dựa trên nền tảng của Apache Lucene engine. Dùng để lưu trữ dữ liệu và cung cấp interface cho phép truy vấn đến cơ sở dữ liệu.

Kibana: Đây là giao diện sử dụng dành cho người dùng trên môi trường web. Kibana sẽ sử dụng Elashtichsearch để tìm kiếm các dữ liệu phù hợp với yêu cầu của người dùng.

I. Cài đặt và cấu hình ELK trên CentOS 7

1. Cài đặt Java 8

[root@centos74]# yum install wget
[root@centos74]# wget http://mirror.cnop.net/jdk/linux/jdk-8u77-linux-x64.rpm
[root@centos74]# yum -y localinstall jdk-8u77-linux-x64.rpm
[root@centos74]# java -version
java version "1.8.0_77"
Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)

 

2. Cài đặt và cấu hình elasticsearch

+ Cài đặt elasticsearch

;import the Elasticsearch public GPG key
[root@centos74]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
[root@centos74]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.1.1.rpm
[root@centos74]# rpm -ivh elasticsearch-5.1.1.rpm

+ Cấu hình elasticsearch

;Restrict outside access to your Elasticsearch instance (port 9200), so outsiders can't read your data or shutdown your Elasticsearch cluster through the HTTP API. Find the line that specifies network.host, uncomment it, and replace its value with "localhost" so it looks like this
[root@centos74]# vim /etc/elasticsearch/elasticsearch.yml
# Lock the memory on startup:
bootstrap.memory_lock: true
# In the 'Network' block, uncomment the network.host and http.port lines.
network.host: localhost
http.port: 9200

;Now edit the elasticsearch.service file for the memory lock configuration.
[root@centos74]# vim /usr/lib/systemd/system/elasticsearch.service
LimitMEMLOCK=infinity

;Edit the sysconfig configuration file for Elasticsearch.
[root@centos74]# vim /etc/sysconfig/elasticsearch
MAX_LOCKED_MEMORY=unlimited

;Elasticsearch will run on the localhost IP address on port 9200, we disabled memory swapping for it by enabling mlockall on the CentOS server
[root@centos74]# systemctl enable elasticsearch
[root@centos74]# systemctl start elasticsearch

;Check the memory lock to ensure that mlockall is enabled, and check that Elasticsearch is running with the commands below.
[root@centos74 ~]# curl -XGET 'localhost:9200/_nodes?filter_path=**.mlockall&pretty'
{
  "nodes" : {
    "QChpvGqkQVGQ5j8aMkC1kw" : {
      "process" : {
        "mlockall" : true
      }
    }
  }
}
[root@centos74 ~]# curl -XGET 'localhost:9200/?pretty'
{
  "name" : "QChpvGq",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "G8ya-TuFSVqsxJRRc3CCkw",
  "version" : {
    "number" : "5.1.1",
    "build_hash" : "5395e21",
    "build_date" : "2016-12-06T12:36:15.409Z",
    "build_snapshot" : false,
    "lucene_version" : "6.3.0"
  },
  "tagline" : "You Know, for Search"
}

 

3. Cài đặt và cấu hình Kibana

+ Cài đặt Kibana

[root@centos74]# wget https://artifacts.elastic.co/downloads/kibana/kibana-5.1.1-x86_64.rpm
[root@centos74]#  rpm -ivh kibana-5.1.1-x86_64.rpm

+ Cấu hình Kibana

[root@centos74]#  vim /etc/kibana/kibana.yml
#Uncomment the configuration lines for server.port, server.host and elasticsearch.url.
server.port: 5601
server.host: "localhost"
elasticsearch.url: "http://localhost:9200"

;start the Kibana service, and enable it
[root@centos74]# systemctl enable kibana
[root@centos74]# systemctl start kibana
[root@centos74]# netstat -ntlpu

 

4. Cài đặt và cấu hình Nginx

;Because we configured Kibana to listen on localhost, we must set up a reverse proxy to allow external access to it. We will use Nginx for this purpose.
;Add the EPEL repository to yum:
[root@centos74]# yum -y install epel-release
;Now use yum to install Nginx and httpd-tools:
[root@centos74]# yum -y install nginx httpd-tools
;Use htpasswd to create an admin user, called "admin"
[root@centos74 ~]# htpasswd -c /etc/nginx/.kibana-user admin
;Edit the Nginx configuration file and remove the 'server { }' block, so we can add a new virtual host configuration.
[root@centos74]# vim /etc/nginx/nginx.conf
include /etc/nginx/conf.d/*.conf;

;This configures Nginx to direct your server's HTTP traffic to the Kibana application, which is listening on localhost:5601. Also, Nginx will use the htpasswd.users file, that we created earlier, and require basic authentication
[root@centos74]# mv /etc/nginx/conf.d/default.conf /etc/nginx/conf.d/default.conf.bk
[root@centos74]# vim /etc/nginx/conf.d/kibana.conf
server {
    listen 80;
 
    server_name elk-stack.co;
 
    auth_basic "Restricted Access";
    auth_basic_user_file /etc/nginx/.kibana-user;
 
    location / {
        proxy_pass http://localhost:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

;Now start and enable Nginx to put our changes into effect
[root@centos74]#  nginx -t
[root@centos74]# systemctl enable nginx
[root@centos74]# systemctl start nginx
[root@centos74]# netstat -ntlpu

;This tutorial assumes that SELinux is disabled. If this is not the case, you may need to run the following command for Kibana to work properly: 
[root@centos74]# sudo setsebool -P httpd_can_network_connect 1

 

5. Cài đặt và cấu hình Logstash

+ Cài đặt logstash

- Logstash đóng vai trò là nơi nhận log từ client ( filebeat ), sau đó sẽ filter và chuyển qua elasticsearch

[root@centos74]# wget https://artifacts.elastic.co/downloads/logstash/logstash-5.1.1.rpm
[root@centos74]# rpm -ivh logstash-5.1.1.rpm

- Generate SSL Certificates: Khi chúng ta sử dụng Filebeat để đưa log từ client tới ELK server, chúng ta cần tạo SSL cert & key pair để Filebeat verify và identity với ELK Server

- Option 1: Khi sử dụng IP Address ( không thiết lập DNS ):

;Logstash package shares the same GPG Key as Elasticsearch
[root@centos74]# vim /etc/pki/tls/openssl.cnf
[ v3_ca ]
;subjectAltName = IP: ELK_server_private_ip
subjectAltName=IP:192.168.30.128

;generate the SSL certificate and private key in the appropriate locations (/etc/pki/tls/), with the following commands:
[root@centos74]# cd /etc/pki/tls
;Generate the certificate file with the openssl command
;The certificate files can be found in the '/etc/pki/tls/certs/' and '/etc/pki/tls/private/' directories.
[root@centos74]# openssl req -config /etc/pki/tls/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout /etc/pki/tls/private/logstash-forwarder.key -out /etc/pki/tls/certs/logstash-forwarder.crt

;The logstash-forwarder.crt file will be copied to all of the servers that will send logs to Logstash but we will do that a little later. 

- Option 2: FQDN (DNS)

[root@centos74]# cd /etc/pki/tls
[root@centos74]# openssl req -subj '/CN=ELK_server_fqdn/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

;The logstash-forwarder.crt file will be copied to all of the servers that will send logs to Logstash but we will do that a little later. Let's complete our Logstash configuration.

+ Cấu hình Logstash

- Tạo file cấu hình 'filebeat-input.conf' trong logstash để nhận log từ "filebeat".

;This specifies a beats input that will listen on tcp port 5443, and it will use the SSL certificate and private key that we created earlier.
[root@centos74]# vim /etc/logstash/conf.d/filebeat-input.conf
input {
  beats {
    port => 5443
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

;Create a 'syslog-filter.conf' file for syslog processing and the 'output-elasticsearch.conf' file to define the Elasticsearch output.
;This filter looks for logs that are labeled as "syslog" type (by Filebeat), and it will use a filter plugin named 'grok'  to parse incoming syslog logs to make it structured and query-able.
[root@centos74]# vim /etc/logstash/conf.d/syslog-filter.conf
filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

;Create the output configuration file 'output-elasticsearch.conf'.
[root@centos74]# vim /etc/logstash/conf.d/elasticsearch-output.conf
output {
  elasticsearch { hosts => ["localhost:9200"]
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

;This output basically configures Logstash to store the beats data in Elasticsearch which is running at localhost:9200, in an index named after the beat used (filebeat, in our case).
;If you want to add filters for other applications that use the Filebeat input, be sure to name the files so they sort between the input and the output configuration 

[root@centos74]# systemctl enable logstash
[root@centos74]# systemctl start logstash
[root@centos74]#  netstat -ntlpu
tcp6       0      0 :::5443                 :::*                    LISTEN      2651/java

 

II. Cài đặt và cấu hình filebeat

1. Trên CentOS 7

+ Filebeat đóng vai trò agent log sẽ gửi log data từ agent server tới Logstash server , sau đó logstash chuyển qua elasticsearch.

+ There are 4 beats available, 'Filebeat' for 'Log Files', 'Metricbeat' for 'Metrics', 'Packetbeat' for 'Network Data' and 'Winlogbeat' for the Windows client 'Event Log'.

+ Cài đặt filebeat - tạo certificate folder và copy cert file từ ELK server tạo ở trên về agent log

[root@centos74]# mkdir -p /etc/pki/tls/certs/ && cd /etc/pki/tls/certs/
[root@centos74]# scp root@elk-serverIP:~/logstash-forwarder.crt .
[root@centos74]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
[root@centos74]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.1.1-x86_64.rpm
[root@centos74]# rpm -ivh filebeat-5.1.1-x86_64.rpm

+ Cấu hình filebeat

[root@centos74]# vim /etc/filebeat/filebeat.yml
# We will add two files '/var/log/secure' for ssh activity and '/var/log/messages' for the server log.
  paths:
    - /var/log/auth.log
    - /var/log/syslog

# Filebeat is using Elasticsearch as the output target by default.Disable elasticsearch output.
#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
#  hosts: ["localhost:9200"]

# add the new logstash output configuration. Uncomment the logstash output configuration and change all value to the configuration that is shown below.
output.logstash:
  # The Logstash hosts
  hosts: ["192.168.30.128:5443"]
  bulk_max_size: 1024
  ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]
  template.name: "filebeat"
  template.path: "filebeat.template.json"
  template.overwrite: false

[root@centos74]# systemctl enable filebeat
[root@centos74]# systemctl start filebeat

 

2. Trên Ubuntu 16

+ Cài đặt filebeat - tạo certificate folder và copy cert file từ ELK server tạo ở trên về agent log

[root@ubuntu16]# mkdir -p /etc/pki/tls/certs/ && cd /etc/pki/tls/certs/
[root@ubuntu16]# scp root@elk-serverIP:~/logstash-forwarder.crt .
[root@ubuntu16]# wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
[root@ubuntu16]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.1.1-amd64.deb
[root@ubuntu16]# dpkg -i filebeat-5.1.1-amd64.deb

+ Cấu hình filebeat

[root@ubuntu16]# vim /etc/filebeat/filebeat.yml
# We will add two files '/var/log/secure' for ssh activity and '/var/log/messages' for the server log.
  paths:
    - /var/log/auth.log
    - /var/log/syslog

# Filebeat is using Elasticsearch as the output target by default.Disable elasticsearch output.
#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
#  hosts: ["localhost:9200"]

# add the new logstash output configuration. Uncomment the logstash output configuration and change all value to the configuration that is shown below.
output.logstash:
  # The Logstash hosts
  hosts: ["192.168.30.128:5443"]
  bulk_max_size: 1024
  ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]
  template.name: "filebeat"
  template.path: "filebeat.template.json"
  template.overwrite: false

[root@ubuntu16]# systemctl enable filebeat
[root@ubuntu16]# systemctl start filebeat
[root@ubuntu16]# systemctl status filebeat