Create server, configure inventory
Create Droplet on Digital Ocean using CentOS 8.3 x64 with 4GB memory and 2 vCPUs configuring an SSH key to make easy access to it.
We can probably use other version of CentOS as well, but this is what I tried it with.
From the Digital Ocean web site copy the IP address of the drolet and add it to the inventory file replacing the IP address you find there:
examples/ansible/elk/inventory.yml
elastic:
hosts:
68.183.26.18
all:
children:
elastic:
Verify that we have access to the server using Ansible Ping
ansible NAME -m ping
First it will want to verify the fingerprint of the server:
The authenticity of host '134.122.123.157 (134.122.123.157)' can't be established.
ECDSA key fingerprint is SHA256:L1jIJx45fOP3lFH/qQysD7tAdY9/rNoeC+eA2mO4ijY.
Are you sure you want to continue connecting (yes/no/[fingerprint])?
Type in "yes" and press enter.
Then the response is expected to look like this:
134.122.123.157 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"ping": "pong"
}
Try the same with our first playbook:
ansible-playbook playbooks/ping.yml
The response is expected to be:
PLAY [all] ****************************************************************************************
TASK [Gathering Facts] ****************************************************************************
ok: [104.236.61.19]
TASK [Ping] ***************************************************************************************
ok: [104.236.61.19]
PLAY RECAP ****************************************************************************************
104.236.61.19 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
examples/ansible/elk/playbooks/ping.yml
---
- hosts: all
tasks:
- name: Ping
ansible.builtin.ping:
See Ping module
Fetch the hostname of the server
OK, so this is not necessary to our task, but I like to see it working.
examples/ansible/elk/playbooks/hostname.yml
---
- hosts: all
tasks:
- name: Bash
ansible.builtin.shell: hostname
register: response
- debug: msg="{{ response.stdout }}"
ansible-playbook playbooks/hostname.yml
Output:
PLAY [all] ***************************************************************************************
TASK [Gathering Facts] ***************************************************************************
ok: [104.236.61.19]
TASK [Bash] **************************************************************************************
changed: [104.236.61.19]
TASK [debug] *************************************************************************************
ok: [104.236.61.19] => {
"msg": "elk1"
}
PLAY RECAP ***************************************************************************************
104.236.61.19 : ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
See Ansible shell module.
Install Elasticsearch
The version of ElasticSearch is baked into the playbook file. You can visit the download page of Elasticsearch to pick a different version.
examples/ansible/elk/playbooks/elasticsearch.yml
---
- hosts: all
gather_facts : no
vars:
elastic: elasticsearch-7.11.2-x86_64.rpm
project_root: /root
tasks:
- get_url:
url="https://artifacts.elastic.co/downloads/elasticsearch/{{elastic}}"
dest="{{project_root}}/{{elastic}}"
- name: Check if Elastic is installed
command: rpm -q elasticsearch
ignore_errors: True
register: rpm_check
- name: Install
ansible.builtin.shell: rpm -vi {{elastic}}
when: rpm_check.rc != 0
- name: Enable and Start
ansible.builtin.service:
name: elasticsearch
state: started
enabled: yes
- name: Copy configuration file
copy:
src: ../files/etc/elasticsearch/elasticsearch.yml
dest: /etc/elasticsearch/elasticsearch.yml
# - name: Configure Elasticsearch Cluster
# lineinfile:
# destfile: /etc/elasticsearch/elasticsearch.yml
# regexp: 'cluster.name:'
# line: 'cluster.name: code-maven-demo'
#
# - name: Configure Elasticsearch Node
# lineinfile:
# destfile: /etc/elasticsearch/elasticsearch.yml
# regexp: 'node.name:'
# line: 'node.name: code-maven-elastic-1'
#
# - name: Configure Elasticsearch Node Master
# lineinfile:
# destfile: /etc/elasticsearch/elasticsearch.yml
# regexp: 'node.master:'
# line: 'node.master: true'
#
# - name: Configure Elasticsearch Node Data
# lineinfile:
# destfile: /etc/elasticsearch/elasticsearch.yml
# regexp: 'node.data:'
# line: 'node.data: true'
#
# - name: Configure Elasticsearch Single-host
# lineinfile:
# destfile: /etc/elasticsearch/elasticsearch.yml
# regexp: 'discovery.type:'
# line: 'discovery.type: single-node'
- name: Restart
ansible.builtin.service:
name: elasticsearch
state: restarted
- name: Verify Elasticsearch
ansible.builtin.shell: curl http://localhost:9200
register: response
- debug: msg="{{ response.stdout }}"
ansible-playbook playbooks/elasticsearch.yml
examples/ansible/elk/files/etc/elasticsearch/elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: code-maven-demo
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: code-maven-elastic-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 192.168.0.1
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
node.master: true
node.data: true
discovery.type: single-node
#network.host: 0.0.0.0
#discovery.seed_hosts: ["1.2.3.4"]
Setup Nginx with simple authentication
Follow the instructions on how to configure http basic authentication for Nginx create one or more username/password pairs:
$ htpasswd -c .htpasswd user1 (pw: secret1)
$ htpasswd .htpasswd user2 (pw: secret2)
examples/ansible/elk/files/etc/nginx/nginx.conf
# For more information on configuration, see:
# * Official English Documentation: http://nginx.org/en/docs/
# * Official Russian Documentation: http://nginx.org/ru/docs/
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/nginx/conf.d/*.conf;
# server {
# listen 80 default_server;
# listen [::]:80 default_server;
# server_name _;
# root /usr/share/nginx/html;
#
# # Load configuration files for the default server block.
# include /etc/nginx/default.d/*.conf;
#
# location / {
# }
#
# error_page 404 /404.html;
# location = /404.html {
# }
#
# error_page 500 502 503 504 /50x.html;
# location = /50x.html {
# }
# }
# Settings for a TLS enabled server.
#
# server {
# listen 443 ssl http2 default_server;
# listen [::]:443 ssl http2 default_server;
# server_name _;
# root /usr/share/nginx/html;
#
# ssl_certificate "/etc/pki/nginx/server.crt";
# ssl_certificate_key "/etc/pki/nginx/private/server.key";
# ssl_session_cache shared:SSL:1m;
# ssl_session_timeout 10m;
# ssl_ciphers HIGH:!aNULL:!MD5;
# ssl_prefer_server_ciphers on;
#
# # Load configuration files for the default server block.
# include /etc/nginx/default.d/*.conf;
#
# location / {
# }
#
# error_page 404 /404.html;
# location = /404.html {
# }
#
# error_page 500 502 503 504 /50x.html;
# location = /50x.html {
# }
# }
}
examples/ansible/elk/files/etc/nginx/conf.d/nginx-elk.conf
server {
listen [::]:80;
listen 80;
# server_name _;
location / {
auth_basic "Kibana area";
auth_basic_user_file /usr/share/nginx/html/.htpasswd;
proxy_pass http://localhost:5601;
proxy_redirect off;
proxy_buffering off;
proxy_http_version 1.1;
proxy_set_header Connection "Keep-Alive";
proxy_set_header Proxy-Connection "Keep-Alive";
}
}
server {
listen [::]:81;
listen 81;
# server_name _;
location / {
auth_basic "Elasticsearch";
auth_basic_user_file /usr/share/nginx/html/.htpasswd;
proxy_pass http://localhost:9200;
proxy_redirect off;
proxy_buffering off;
proxy_http_version 1.1;
proxy_set_header Connection "Keep-Alive";
proxy_set_header Proxy-Connection "Keep-Alive";
}
}
Then run the playbook:
ansible-playbook playbooks/nginx.yml
examples/ansible/elk/playbooks/nginx.yml
---
- hosts: all
gather_facts : no
vars:
project_root: /root
tasks:
- name: Enable epel
yum:
name: epel-release
state: present
- name: Install Nginx
yum:
name: nginx
state: present
- name: Enable and Start
ansible.builtin.service:
name: nginx
state: started
enabled: yes
- name: Nginx config file
copy:
src: ../files/etc/nginx/nginx.conf
dest: /etc/nginx/nginx.conf
- name: Nginx config file
copy:
src: ../files/etc/nginx/conf.d/nginx-elk.conf
dest: /etc/nginx/conf.d/nginx-elk.conf
- name: Nginx users file
copy:
src: ../files/usr/share/nginx/html/.htpasswd
dest: /usr/share/nginx/html/.htpasswd
- name: Allow Nginx to work as a reverse proxy
ansible.builtin.shell: setsebool httpd_can_network_connect on -P
- name: Restart
ansible.builtin.service:
name: nginx
state: restarted
Visit http://IP:81 after replacing IP with the IP address of your host to get access to Elasticsearch
Kibana
examples/ansible/elk/playbooks/kibana.yml
---
- hosts: all
gather_facts : no
vars:
kibana: kibana-7.11.2-x86_64.rpm
project_root: /root
tasks:
- get_url:
url="https://artifacts.elastic.co/downloads/kibana/{{kibana}}"
dest="{{project_root}}/{{kibana}}"
- name: Check if Kibana is installed
command: rpm -q kibana
ignore_errors: True
register: rpm_check
- name: Install
ansible.builtin.shell: rpm -vi {{kibana}}
when: rpm_check.rc != 0
- name: Enable and Start
ansible.builtin.service:
name: kibana
state: started
enabled: yes
- name: Copy configuration file
copy:
src: ../files/etc/kibana/kibana.yml
dest: /etc/kibana/kibana.yml
- name: Restart
ansible.builtin.service:
name: kibana
state: restarted
Metricbeat
examples/ansible/elk/playbooks/metricbeat.yml
---
- hosts: all
vars:
metricbeat: metricbeat-7.11.2-x86_64.rpm
project_root: /root
tasks:
- get_url:
url="https://artifacts.elastic.co/downloads/beats/metricbeat/{{metricbeat}}"
dest="{{project_root}}/{{metricbeat}}"
- name: Check if Metricbeat is installed
command: rpm -q metricbeat
ignore_errors: True
register: rpm_check
- name: Install
ansible.builtin.shell: rpm -vi {{metricbeat}}
when: rpm_check.rc != 0
- name: Enable and Start
ansible.builtin.service:
name: metricbeat
state: started
enabled: yes
- name: Copy config file
copy:
src: ../files/etc/metricbeat/metricbeat.yml
dest: /etc/metricbeat/metricbeat.yml
- name: Restart
ansible.builtin.service:
name: metricbeat
state: restarted
ELK
examples/ansible/elk/playbooks/elk.yml
---
- include: nginx.yml
- include: elasticsearch.yml
- include: kibana.yml
- include: metricbeat.yml
Ansible Configuration file
examples/ansible/elk/ansible.cfg
[defaults]
deprecation_warnings = False
inventory = inventory.yml
host_key_checking = True
remote_user=root
#ask_pass = True
