Mauricio Magnani bio photo

Mauricio Magnani

A guy who loves technology and new challenges!

Email Twitter LinkedIn Github

Below are the chapters I, II and III:

WildFly Scalability

We will scale our instances manually using JBoss CLI and later this will be done automatically using a trigger via AlertManager.

So the first step is to test the creation of a new instance using JBoss CLI.

Use the master host itself for these tests. Create a file called create-server.cli:

/host=slave0/server-config=server-marketing-7:add(auto-start=true,socket-binding-port-offset=400,group=marketing)
/host=slave0/server-config=server-marketing-7/system-property=jboss.node.name:add(value=node-marketing-7)
/host=slave0/server-config=server-marketing-7/system-property=wildfly.balancer.name:add(value=marketing-lb)
/host=slave0/server=server-marketing-7:start

After that runs via command line:

[root@server-domain ~]# sh /usr/local/wildfly/wildfly-18.0.1.Final/bin/jboss-cli.sh -c --controller=10.0.0.68:9990 --user=admin --password=redhat --file=create-server.cli

The script creates a new WildFly instance (server-marketing-7) on slave0 and starts it immediately.

Note that a few seconds later the instance is already available on the balancer and also the application will be deployed automatically. This gives us a very interesting scalability in our WildFly environment.

Unfortunately this process is manual and failures can happen. We will automate it in the next steps.

WildFly Event-based Auto Scaling

To achieve the goal simply and effectively we will use Prometheus, AlertManager and Webhook.

  • Prometheus - https://prometheus.io
  • AlertManager - https://prometheus.io/docs/alerting/alertmanager
  • Webhook - https://github.com/adnanh/webhook

For this purpose I will use the server: monitor.mmagnani.lab - 10.0.0.117

The first step is to install Webhook.

[root@monitor ~]# cat /etc/redhat-release 
CentOS Linux release 7.7.1908 (Core)
[root@monitor ~]# systemctl stop firewalld
[root@monitor ~]# systemctl disable firewalld
[root@monitor ~]# setenforce 0
[root@monitor ~]# sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
[root@monitor ~]# mkdir -p /usr/local/webhook
[root@monitor ~]# wget https://github.com/adnanh/webhook/releases/download/2.6.11/webhook-linux-amd64.tar.gz -P /usr/local/webhook/
[root@monitor ~]# cd /usr/local/webhook/
[root@monitor ~]# tar -xzvf webhook-linux-amd64.tar.gz 
[root@monitor ~]# cd webhook-linux-amd64
[root@monitor ~]# ls
webhook
[root@monitor ~]# pwd
/usr/local/webhook/webhook-linux-amd64

Now create the hooks.json file that will contain the action to take when any event happens:

[root@monitor ~]# pwd
/usr/local/webhook/webhook-linux-amd64
[root@monitor ~]# vi /usr/local/webhook/webhook-linux-amd64/hooks.json
[
  {
    "id": "scale-wildfly",
    "execute-command": "/var/scripts/scale-wildfly.sh",
    "command-working-directory": "/var/webhook"
  }
]

Create the scale-wildfly.sh script that will actually contain JBoss CLI actions:

[root@monitor ~]# mkdir -p /var/scripts/
[root@monitor ~]# mkdir -p /var/webhook
[root@monitor ~]# vi /var/scripts/scale-wildfly.sh
#!/bin/bash
/usr/local/wildfly/wildfly-18.0.1.Final/bin/jboss-cli.sh -c --controller=10.0.0.68:9990  <<EOF
/host=slave0/server-config=server-marketing-8:add(auto-start=true,socket-binding-port-offset=400,group=marketing)
/host=slave0/server-config=server-marketing-8/system-property=jboss.node.name:add(value=node-marketing-8)
/host=slave0/server-config=server-marketing-8/system-property=wildfly.balancer.name:add(value=marketing-lb)
/host=slave0/server=server-marketing-8:start
EOF
[root@monitor ~]# chmod +x /var/scripts/scale-wildfly.sh

We will need the client which in this case is the JBoss CLI. For simplicity I will just unzip WildFly’s zip into a directory.

[root@monitor ~]# yum install java-1.8.0-openjdk-devel.x86_64 -y
[root@monitor ~]# mkdir -p /usr/local/wildfly
[root@monitor ~]# wget https://download.jboss.org/wildfly/18.0.1.Final/wildfly-18.0.1.Final.zip -P /usr/local/wildfly
[root@monitor ~]# cd /usr/local/wildfly
[root@monitor ~]# unzip wildfly-18.0.1.Final.zip

Now start webhook:

[root@monitor ~]# pwd
/usr/local/webhook/webhook-linux-amd64
[root@monitor ~]# ./webhook  -hooks hooks.json -verbose
 ####TODO put this as a service script

It will start up on default port 9000 and will provide you with one HTTP endpoint: http://monitor.mmagnani.lab:9000/hooks/scale-wildfly

By performing a simple HTTP GET or POST request to that endpoint, the specified script would be executed.

[webhook] 2019/12/27 17:25:59 [b23fed] incoming HTTP request from 10.0.0.114:52462
[webhook] 2019/12/27 17:25:59 [b23fed] scale-wildfly got matched
[webhook] 2019/12/27 17:25:59 [b23fed] error parsing body payload due to unsupported content type header: 
[webhook] 2019/12/27 17:25:59 [b23fed] scale-wildfly hook triggered successfully
[webhook] 2019/12/27 17:25:59 200 | 232.132µs | monitor.mmagnani.lab:9000 | GET /hooks/scale-wildfly 
[webhook] 2019/12/27 17:25:59 [b23fed] executing /var/scripts/scale-wildfly.sh (/var/scripts/scale-wildfly.sh) with arguments ["/var/scripts/scale-wildfly.sh"] and environment [] using /var/webhook as cwd
[webhook] 2019/12/27 17:26:02 [b23fed] command output: [domain@10.0.0.68:9990 /] /host=slave0/server-config=server-marketing-8:add(auto-start=true,socket-binding-port-offset=400,group=marketing)
{
    "outcome" => "success",
    "result" => undefined,
    "server-groups" => undefined
}
[domain@10.0.0.68:9990 /] /host=slave0/server-config=server-marketing-8/system-property=jboss.node.name:add(value=node-marketing-8)
{
    "outcome" => "success",
    "result" => undefined,
    "server-groups" => undefined
}
[domain@10.0.0.68:9990 /] /host=slave0/server-config=server-marketing-8/system-property=wildfly.balancer.name:add(value=marketing-lb)
{
    "outcome" => "success",
    "result" => undefined,
    "server-groups" => undefined
}
[domain@10.0.0.68:9990 /] /host=slave0/server=server-marketing-8:start
{
    "outcome" => "success",
    "result" => "STARTING"
}
[domain@10.0.0.68:9990 /] 

Note that the cluster was automatically escalated through a request using the endpoint.

Our goal was successfully achieved! The next step is to do this through AlertManager.

So at this point we will install Prometheus.

[root@monitor ~]# cd /tmp
[root@monitor ~]# wget https://github.com/prometheus/prometheus/releases/download/v2.15.1/prometheus-2.15.1.linux-amd64.tar.gz
[root@monitor ~]# useradd --no-create-home --shell /bin/false prometheus
[root@monitor ~]# mkdir /etc/prometheus
[root@monitor ~]# mkdir /var/lib/prometheus
[root@monitor ~]# chown prometheus:prometheus /etc/prometheus
[root@monitor ~]# chown prometheus:prometheus /var/lib/prometheus
[root@monitor ~]# tar -xvzf tar prometheus-2.15.1.linux-amd64.tar.gz
[root@monitor ~]# mv prometheus-2.15.1.linux-amd64 prometheus-package
[root@monitor ~]# cp prometheus-package/prometheus /usr/local/bin/
[root@monitor ~]# cp prometheus-package/promtool /usr/local/bin/
[root@monitor ~]# chown prometheus:prometheus /usr/local/bin/prometheus
[root@monitor ~]# chown prometheus:prometheus /usr/local/bin/promtool
[root@monitor ~]# cp -r prometheus-package/consoles /etc/prometheus
[root@monitor ~]# cp -r prometheus-package/console_libraries /etc/prometheus
[root@monitor ~]# chown -R prometheus:prometheus /etc/prometheus/consoles
[root@monitor ~]# chown -R prometheus:prometheus /etc/prometheus/console_libraries
[root@monitor ~]# vi /etc/prometheus/prometheus.yml

global:
  scrape_interval: 10s

scrape_configs:
  - job_name: 'prometheus_master'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

[root@monitor ~]# chown prometheus:prometheus /etc/prometheus/prometheus.yml
[root@monitor ~]# vi /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/lib/prometheus --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

[root@monitor ~]# systemctl daemon-reload
[root@monitor ~]# systemctl start prometheus
[root@monitor ~]# systemctl status prometheus

Use the following Url to access UI: http://monitor.mmagnani.lab:9090

Now let’s install the blackbox exporter. The blackbox exporter allows blackbox probing of endpoints over HTTP, HTTPS, DNS, TCP and ICMP.

[root@monitor ~]# cd /tmp
[root@monitor ~]# wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.16.0/blackbox_exporter-0.16.0.linux-amd64.tar.gz
[root@monitor ~]# useradd --no-create-home --shell /bin/false blackbox_exporter
[root@monitor ~]# tar -xvf blackbox_exporter-0.16.0.linux-amd64.tar.gz
[root@monitor ~]# cp blackbox_exporter-0.16.0.linux-amd64/blackbox_exporter /usr/local/bin/blackbox_exporter
[root@monitor ~]# chown blackbox_exporter:blackbox_exporter /usr/local/bin/blackbox_exporter
[root@monitor ~]# mkdir /etc/blackbox_exporter
[root@monitor ~]# vi /etc/blackbox_exporter/blackbox.yml

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: []
      method: GET

[root@monitor ~]# chown blackbox_exporter:blackbox_exporter /etc/blackbox_exporter/blackbox.yml
[root@monitor ~]# vi /etc/systemd/system/blackbox_exporter.service

[Unit]
Description=Blackbox Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=blackbox_exporter
Group=blackbox_exporter
Type=simple
ExecStart=/usr/local/bin/blackbox_exporter --config.file /etc/blackbox_exporter/blackbox.yml

[Install]
WantedBy=multi-user.target

[root@monitor ~]# systemctl daemon-reload
[root@monitor ~]# systemctl start blackbox_exporter
[root@monitor ~]# systemctl enable blackbox_exporter

Now edit the Prometheus configuration and add our target:

[root@monitor ~]# vi /etc/prometheus/prometheus.yml

  - job_name: 'blackbox-marketing'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - http://marketing.mmagnani.lab
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115

[root@monitor ~]# systemctl restart prometheus

Create a alarm definition:

[root@monitor ~]# vi /etc/prometheus/prometheus.rules.yml

groups:
- name: alert.rules
  rules:
  - alert: EndpointDown
    expr: probe_success == 0
    for: 10s
    labels:
      severity: "critical"
    annotations:
      summary: "Endpoint  down"

[root@monitor ~]# chown prometheus:prometheus /etc/prometheus/prometheus.rules.yml
[root@monitor ~]# promtool check rules /etc/prometheus/prometheus.rules.yml
[root@monitor ~]# systemctl restart prometheus

Please note that the Prometheus Target definitions already display our probe:

Finally install AlertManager.

[root@monitor ~]# cd /tmp
[root@monitor ~]# wget https://github.com/prometheus/alertmanager/releases/download/v0.20.0/alertmanager-0.20.0.linux-amd64.tar.gz
[root@monitor ~]# useradd --no-create-home --shell /bin/false alertmanager
[root@monitor ~]# tar -xvf alertmanager-0.20.0.linux-amd64.tar.gz 
[root@monitor ~]# cp alertmanager-0.20.0.linux-amd64/alertmanager /usr/local/bin/
[root@monitor ~]# cp alertmanager-0.20.0.linux-amd64/amtool /usr/local/bin/
[root@monitor ~]# chown alertmanager:alertmanager /usr/local/bin/alertmanager
[root@monitor ~]# chown alertmanager:alertmanager /usr/local/bin/amtool
[root@monitor ~]# mkdir /etc/alertmanager
[root@monitor ~]# vi /etc/alertmanager/alertmanager.yml

global:

route:
  group_by: ['instance', 'alert']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h
  receiver: default-receiver

receivers:
  - name: default-receiver
    webhook_configs:
      - url: "http://monitor.mmagnani.lab:9000/hooks/scale-wildfly"

[root@monitor ~]# chown alertmanager:alertmanager -R /etc/alertmanager
[root@monitor ~]# vi /etc/systemd/system/alertmanager.service

[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
WorkingDirectory=/etc/alertmanager/
ExecStart=/usr/local/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --web.external-url http://0.0.0.0:9093

[Install]
WantedBy=multi-user.target

[root@monitor ~]# vi /etc/prometheus/prometheus.yml
rule_files:
  - 'prometheus.rules.yml'
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - localhost:9093

[root@monitor ~]# systemctl restart prometheus
[root@monitor ~]# systemctl restart alertmanager

Note that the alert will be inactive: http://monitor.mmagnani.lab:9090/alerts

Check the AlertManager: http://monitor.mmagnani.lab:9093/#/alerts

Let’s stop all instances of the marketing group to make sure the app is down.

When we stop instances the application will be “down” and our alert will be firing.

Our endpoint that performs environment scaling will be triggered:

[webhook] 2019/12/29 19:55:33 [5f95ff] incoming HTTP request from 10.0.0.117:38598
[webhook] 2019/12/29 19:55:33 [5f95ff] scale-wildfly got matched
[webhook] 2019/12/29 19:55:33 [5f95ff] scale-wildfly hook triggered successfully
[webhook] 2019/12/29 19:55:33 200 | 130.171µs | monitor.mmagnani.lab:9000 | POST /hooks/scale-wildfly 
[webhook] 2019/12/29 19:55:33 [5f95ff] executing /var/scripts/scale-wildfly.sh (/var/scripts/scale-wildfly.sh) with arguments ["/var/scripts/scale-wildfly.sh"] and environment [] using /var/webhook as cwd
[webhook] 2019/12/29 19:55:36 [5f95ff] command output: [domain@10.0.0.68:9990 /] /host=slave0/server-config=server-marketing-8:add(auto-start=true,socket-binding-port-offset=500,group=marketing)
{
    "outcome" => "success",
    "result" => undefined,
    "server-groups" => undefined
}
[domain@10.0.0.68:9990 /] /host=slave0/server-config=server-marketing-8/system-property=jboss.node.name:add(value=node-marketing-8)
{
    "outcome" => "success",
    "result" => undefined,
    "server-groups" => undefined
}
[domain@10.0.0.68:9990 /] /host=slave0/server-config=server-marketing-8/system-property=wildfly.balancer.name:add(value=marketing-lb)
{
    "outcome" => "success",
    "result" => undefined,
    "server-groups" => undefined
}
[domain@10.0.0.68:9990 /] /host=slave0/server=server-marketing-8:start
{
    "outcome" => "success",
    "result" => "STARTING"
}
[domain@10.0.0.68:9990 /] 
[webhook] 2019/12/29 19:55:36 [5f95ff] finished handling scale-wildfly

A new instance has been created and our application is available again:

With this we have met our goal of having an elastic, scaling environment automatically based on events.

Any questions or suggestions let me know!

Best,