SyntropyStack — ANSIBLE — Logging solution with ElasticSearch, Kibana, Fluentd and Nginx — Part 2

In this tutorial, we will be creating a logging network with Kibana, ElasticSearch, Fluentd and Nginx (with Let’s Encrypt SSL certificates). To deploy our network, we will be using the Syntropy Stack, Docker and Ansible.

I would put you at the end of this article, a detailed link allowing you to mount this architecture.

Start Syntropy Agent with Docker

sudo docker run --network="host" --restart=on-failure:10 --cap-add=NET_ADMIN --cap-add=SYS_MODULE \
-v /var/run/docker.sock:/var/run/docker.sock:ro \
--device /dev/net/tun:/dev/net/tun --name=syntropynet-agent \
-e SYNTROPY_API_KEY=CHANGE ME \
-e SYNTROPY_TAGS=CHANGE ME \
-e SYNTROPY_PROVIDER=CHANGE ME \
-e SYNTROPY_AGENT_NAME=CHANGE ME \
-e SYNTROPY_NETWORK_API='docker' \
-d syntropynet/agent:stable

The VM of different vendors register on the SyntropyStack interface thanks to the agent launched by Docker on Ansible.

Certificat SSL LetsEncrypt / DNS

For my access to be secure, it was necessary to create a LetsEncrypt SSL certificate and pair it with a domain name. In this example, I use DuckDNS.

Duck DNS (www.duckdns.org)

The creation of several sub-networks

Launch services on each dedicated VM. Be careful by launching them on a different subnet

sudo docker network create --subnet 172.20.0.0/24 syntropynet
sudo docker network create --subnet 172.21.0.0/24 syntropynet
sudo docker network create --subnet 172.22.0.0/24 syntropynet

Here are some explanations for each service:

To distinguish VM and Services, it is necessary to mount them on different IP address plans. Ports were not to be exposed to the Internet, except for 443.

VM Providers

• VM1: Kibana + Nginx
• VM2: ElasticSearch
• VM3: Fluentd

Here are some explanations for each service:

• Nginx: is a web server which can also be used as a reverse proxy, load balancer, mail proxy and HTTP cache. The software was created by Igor Sysoev and first publicly released in 2004.
• ElastiSearch: is software that uses Lucene for indexing and data retrieval. It provides a distributed and multi-entity search engine through a REST interface. It is software written in Java distributed under the Elastic license.
• Fluentd: is an open source, cross-platform data collection software project originally developed at Treasure Data. It is written primarily in the Ruby programming language.

To distinguish VM and Services, it is necessary to mount them on different IP address plans. Ports were not to be exposed to the Internet, except for 443.

Ansible

Ansible is an open source software platform for configuring and managing computers. It combines the deployment of multi-node software, the execution of ad-hoc tasks, and configuration management. It manages the various nodes through SSH and does not require the installation of any additional software on them. Modules communicate through standard output in JSON notation and can be written in any programming language. The system uses YAML to express reusable descriptions of systems, called playbooks

In order for all the services to be built, several YAML files are made.

tree Ansible

Installation

Copy the entire roles directory to your controller server

Install the Syntropy Ansible Galaxy Collection.

ansible-galaxy collection install git@github.com:SyntropyNet/syntropy-ansible-collection.git

Navigate to your local ansible directory:

cd /root/.ansible/collections/ansible_collections/syntropynet/syntropy

Install the Python dependencies.

pip3 install -U -r requirements.txt

Authentication

Generate an API Token by logging in using the CLI:

syntropyctl login {syntropy stack user name} { syntropy stack password}

Provision your Virtual Machines

Info:

• For Python >= 2.7 [servers:vars] ansible_python_interpreter=/usr/bin/python3
• For Python <= 2.7 [servers:vars] ansible_python_interpreter=/usr/bin/python

[kibana]
yourfirstpubip ansible_python_interpreter=/usr/bin/python3
[elasticsearch]
yoursecondpubip ansible_python_interpreter=/usr/bin/python3
[fluentd]
localhost ansible_python_interpreter=/usr/bin/python3

Test Connection: ansible -m ping all

Output result:

localhost | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
*.*.*.* | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
*.*.*.* | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

Deploy Agent / Services / Network with Ansible Playbook

The power of Ansible is that everything is done from a single server. All this does automatically.

Kibana

Kibana is a data visualization plugin for Elasticsearch released under the Apache free license version 2. It provides visualization functions on content indexed in an Elasticsearch cluster.

• Simulate a log with curl command since Fluentd:

curl -X POST -d 'json={"message":"Hello Syntropy World"}' 172.22.0.2:9880/TEST

In conclusion, this exercise allowed us to highlight that with different VM connected via an encrypted tunnel, it is possible to build an architecture as if you were in a local network safely;-)
Thanks to Ansible, you make your life easier because everything is controlled from a single server.

Video tutorial

GitHub Tutorial

syntropynet-use-cases/elastic-kibana-fluentd-ansible at log2-aut-1 · lorenzo8769/syntropynet-use-cases (github.com)