Install the ELK stack on CentOS using Ansible
Create server, configure inventory
Create Droplet on Digital Ocean using CentOS 8.3 x64 with 4GB memory and 2 vCPUs configuring an SSH key to make easy access to it.
We can probably use other version of CentOS as well, but this is what I tried it with.
From the Digital Ocean web site copy the IP address of the drolet and add it to the inventory file replacing the IP address you find there:
examples/ansible/elk/inventory.yml
elastic:
hosts:
68.183.26.18
all:
children:
elastic:
Verify that we have access to the server using Ansible Ping
ansible NAME -m ping
First it will want to verify the fingerprint of the server:
The authenticity of host '134.122.123.157 (134.122.123.157)' can't be established. ECDSA key fingerprint is SHA256:L1jIJx45fOP3lFH/qQysD7tAdY9/rNoeC+eA2mO4ijY. Are you sure you want to continue connecting (yes/no/[fingerprint])?
Type in "yes" and press enter.
Then the response is expected to look like this:
134.122.123.157 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"ping": "pong"
}
Try the same with our first playbook:
ansible-playbook playbooks/ping.yml
The response is expected to be:
PLAY [all] **************************************************************************************** TASK [Gathering Facts] **************************************************************************** ok: [104.236.61.19] TASK [Ping] *************************************************************************************** ok: [104.236.61.19] PLAY RECAP **************************************************************************************** 104.236.61.19 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
examples/ansible/elk/playbooks/ping.yml
---
- hosts: all
tasks:
- name: Ping
ansible.builtin.ping:
See Ping module
Fetch the hostname of the server
OK, so this is not necessary to our task, but I like to see it working.
examples/ansible/elk/playbooks/hostname.yml
---
- hosts: all
tasks:
- name: Bash
ansible.builtin.shell: hostname
register: response
- debug: msg="{{ response.stdout }}"
ansible-playbook playbooks/hostname.yml
Output:
PLAY [all] ***************************************************************************************
TASK [Gathering Facts] ***************************************************************************
ok: [104.236.61.19]
TASK [Bash] **************************************************************************************
changed: [104.236.61.19]
TASK [debug] *************************************************************************************
ok: [104.236.61.19] => {
"msg": "elk1"
}
PLAY RECAP ***************************************************************************************
104.236.61.19 : ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
See Ansible shell module.
Install Elasticsearch
The version of ElasticSearch is baked into the playbook file. You can visit the download page of Elasticsearch to pick a different version.
examples/ansible/elk/playbooks/elasticsearch.yml
---
- hosts: all
gather_facts : no
vars:
elastic: elasticsearch-7.11.2-x86_64.rpm
project_root: /root
tasks:
- get_url:
url="https://artifacts.elastic.co/downloads/elasticsearch/{{elastic}}"
dest="{{project_root}}/{{elastic}}"
- name: Check if Elastic is installed
command: rpm -q elasticsearch
ignore_errors: True
register: rpm_check
- name: Install
ansible.builtin.shell: rpm -vi {{elastic}}
when: rpm_check.rc != 0
- name: Enable and Start
ansible.builtin.service:
name: elasticsearch
state: started
enabled: yes
- name: Copy configuration file
copy:
src: ../files/etc/elasticsearch/elasticsearch.yml
dest: /etc/elasticsearch/elasticsearch.yml
# - name: Configure Elasticsearch Cluster
# lineinfile:
# destfile: /etc/elasticsearch/elasticsearch.yml
# regexp: 'cluster.name:'
# line: 'cluster.name: code-maven-demo'
#
# - name: Configure Elasticsearch Node
# lineinfile:
# destfile: /etc/elasticsearch/elasticsearch.yml
# regexp: 'node.name:'
# line: 'node.name: code-maven-elastic-1'
#
# - name: Configure Elasticsearch Node Master
# lineinfile:
# destfile: /etc/elasticsearch/elasticsearch.yml
# regexp: 'node.master:'
# line: 'node.master: true'
#
# - name: Configure Elasticsearch Node Data
# lineinfile:
# destfile: /etc/elasticsearch/elasticsearch.yml
# regexp: 'node.data:'
# line: 'node.data: true'
#
# - name: Configure Elasticsearch Single-host
# lineinfile:
# destfile: /etc/elasticsearch/elasticsearch.yml
# regexp: 'discovery.type:'
# line: 'discovery.type: single-node'
- name: Restart
ansible.builtin.service:
name: elasticsearch
state: restarted
- name: Verify Elasticsearch
ansible.builtin.shell: curl http://localhost:9200
register: response
- debug: msg="{{ response.stdout }}"
ansible-playbook playbooks/elasticsearch.yml
examples/ansible/elk/files/etc/elasticsearch/elasticsearch.yml
# ======================== Elasticsearch Configuration ========================= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # cluster.name: code-maven-demo # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # node.name: code-maven-elastic-1 # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # path.data: /var/lib/elasticsearch # # Path to log files: # path.logs: /var/log/elasticsearch # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # #bootstrap.memory_lock: true # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # #network.host: 192.168.0.1 # # Set a custom port for HTTP: # #http.port: 9200 # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when this node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # #discovery.seed_hosts: ["host1", "host2"] # # Bootstrap the cluster using an initial set of master-eligible nodes: # #cluster.initial_master_nodes: ["node-1", "node-2"] # # For more information, consult the discovery and cluster formation module documentation. # # ---------------------------------- Gateway ----------------------------------- # # Block initial recovery after a full cluster restart until N nodes are started: # #gateway.recover_after_nodes: 3 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ----------------------------------- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true node.master: true node.data: true discovery.type: single-node #network.host: 0.0.0.0 #discovery.seed_hosts: ["1.2.3.4"]
Setup Nginx with simple authentication
Follow the instructions on how to configure http basic authentication for Nginx create one or more username/password pairs:
$ htpasswd -c .htpasswd user1 (pw: secret1) $ htpasswd .htpasswd user2 (pw: secret2)
examples/ansible/elk/files/etc/nginx/nginx.conf
# For more information on configuration, see:
# * Official English Documentation: http://nginx.org/en/docs/
# * Official Russian Documentation: http://nginx.org/ru/docs/
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/nginx/conf.d/*.conf;
# server {
# listen 80 default_server;
# listen [::]:80 default_server;
# server_name _;
# root /usr/share/nginx/html;
#
# # Load configuration files for the default server block.
# include /etc/nginx/default.d/*.conf;
#
# location / {
# }
#
# error_page 404 /404.html;
# location = /404.html {
# }
#
# error_page 500 502 503 504 /50x.html;
# location = /50x.html {
# }
# }
# Settings for a TLS enabled server.
#
# server {
# listen 443 ssl http2 default_server;
# listen [::]:443 ssl http2 default_server;
# server_name _;
# root /usr/share/nginx/html;
#
# ssl_certificate "/etc/pki/nginx/server.crt";
# ssl_certificate_key "/etc/pki/nginx/private/server.key";
# ssl_session_cache shared:SSL:1m;
# ssl_session_timeout 10m;
# ssl_ciphers HIGH:!aNULL:!MD5;
# ssl_prefer_server_ciphers on;
#
# # Load configuration files for the default server block.
# include /etc/nginx/default.d/*.conf;
#
# location / {
# }
#
# error_page 404 /404.html;
# location = /404.html {
# }
#
# error_page 500 502 503 504 /50x.html;
# location = /50x.html {
# }
# }
}
examples/ansible/elk/files/etc/nginx/conf.d/nginx-elk.conf
server {
listen [::]:80;
listen 80;
# server_name _;
location / {
auth_basic "Kibana area";
auth_basic_user_file /usr/share/nginx/html/.htpasswd;
proxy_pass http://localhost:5601;
proxy_redirect off;
proxy_buffering off;
proxy_http_version 1.1;
proxy_set_header Connection "Keep-Alive";
proxy_set_header Proxy-Connection "Keep-Alive";
}
}
server {
listen [::]:81;
listen 81;
# server_name _;
location / {
auth_basic "Elasticsearch";
auth_basic_user_file /usr/share/nginx/html/.htpasswd;
proxy_pass http://localhost:9200;
proxy_redirect off;
proxy_buffering off;
proxy_http_version 1.1;
proxy_set_header Connection "Keep-Alive";
proxy_set_header Proxy-Connection "Keep-Alive";
}
}
Then run the playbook:
ansible-playbook playbooks/nginx.yml
examples/ansible/elk/playbooks/nginx.yml
---
- hosts: all
gather_facts : no
vars:
project_root: /root
tasks:
- name: Enable epel
yum:
name: epel-release
state: present
- name: Install Nginx
yum:
name: nginx
state: present
- name: Enable and Start
ansible.builtin.service:
name: nginx
state: started
enabled: yes
- name: Nginx config file
copy:
src: ../files/etc/nginx/nginx.conf
dest: /etc/nginx/nginx.conf
- name: Nginx config file
copy:
src: ../files/etc/nginx/conf.d/nginx-elk.conf
dest: /etc/nginx/conf.d/nginx-elk.conf
- name: Nginx users file
copy:
src: ../files/usr/share/nginx/html/.htpasswd
dest: /usr/share/nginx/html/.htpasswd
- name: Allow Nginx to work as a reverse proxy
ansible.builtin.shell: setsebool httpd_can_network_connect on -P
- name: Restart
ansible.builtin.service:
name: nginx
state: restarted
Visit http://IP:81 after replacing IP with the IP address of your host to get access to Elasticsearch
Kibana
examples/ansible/elk/playbooks/kibana.yml
---
- hosts: all
gather_facts : no
vars:
kibana: kibana-7.11.2-x86_64.rpm
project_root: /root
tasks:
- get_url:
url="https://artifacts.elastic.co/downloads/kibana/{{kibana}}"
dest="{{project_root}}/{{kibana}}"
- name: Check if Kibana is installed
command: rpm -q kibana
ignore_errors: True
register: rpm_check
- name: Install
ansible.builtin.shell: rpm -vi {{kibana}}
when: rpm_check.rc != 0
- name: Enable and Start
ansible.builtin.service:
name: kibana
state: started
enabled: yes
- name: Copy configuration file
copy:
src: ../files/etc/kibana/kibana.yml
dest: /etc/kibana/kibana.yml
- name: Restart
ansible.builtin.service:
name: kibana
state: restarted
Metricbeat
examples/ansible/elk/playbooks/metricbeat.yml
---
- hosts: all
vars:
metricbeat: metricbeat-7.11.2-x86_64.rpm
project_root: /root
tasks:
- get_url:
url="https://artifacts.elastic.co/downloads/beats/metricbeat/{{metricbeat}}"
dest="{{project_root}}/{{metricbeat}}"
- name: Check if Metricbeat is installed
command: rpm -q metricbeat
ignore_errors: True
register: rpm_check
- name: Install
ansible.builtin.shell: rpm -vi {{metricbeat}}
when: rpm_check.rc != 0
- name: Enable and Start
ansible.builtin.service:
name: metricbeat
state: started
enabled: yes
- name: Copy config file
copy:
src: ../files/etc/metricbeat/metricbeat.yml
dest: /etc/metricbeat/metricbeat.yml
- name: Restart
ansible.builtin.service:
name: metricbeat
state: restarted
ELK
examples/ansible/elk/playbooks/elk.yml
--- - include: nginx.yml - include: elasticsearch.yml - include: kibana.yml - include: metricbeat.yml
Ansible Configuration file
examples/ansible/elk/ansible.cfg
[defaults] deprecation_warnings = False inventory = inventory.yml host_key_checking = True remote_user=root #ask_pass = True
Published on 2021-04-10
If you have any comments or questions, feel free to post them on the source of this page in GitHub. Source on GitHub.
Comment on this post