Practical Automation: Responding to Events with Ansible

Use Event-Driven Ansible to automatically remediate failed logins

You’ve probably heard the term “event-driven" when discussing modern application architecture. Writing applications that respond to change, instead of actively polling an environment to detect change, often results in more efficient and robust software. The same is true of infrastructure automation, and Event-Driven Ansible (EDA) is an exciting development in the Ansible ecosystem.

I often reach for Ansible when I need to perform a task that exceeds a basic shell script. It’s an extremely versatile automation tool that lends itself well to orchestration, configuration management, and even personal scripting to improve daily workflows. A classic problem with Ansible is the need to execute a playbook in response to change. Executing playbooks on a regular cadence is easy with tools like cron, systemd timers, or automation platforms like Rundeck. However, executing playbooks in response to changes in the external environment has often been cumbersome with these tools.

Event-Driven Ansible aims to solve this problem. EDA allows you to write special rulebooks that respond to incoming changes. Rulebooks run actions in response to external events. In this article, I’ll walk you through a realistic use case to introduce EDA. You’ll see how EDA can respond to failed login notifications and trigger a remediation playbook.

Understand a Practical Scenario

Consider a web environment with multiple web servers behind a load balancer. The web servers maintain logs of failed login attempts. It’s very useful to automatically block traffic on a load balancer when a large number of failed login attempts are detected on the backend web servers.

Traditionally, this would be accomplished by shipping all of the web server logs to a central Intrusion Detection System (IDS) and triggering automation based on centralized log analysis. This works great, but it can be ineffective in certain environments. Very small environments may not have central log analysis, while very large environments may find that it is more beneficial to distribute the task among the backend web servers.

Event-Driven Ansible works very well in all of these scenarios. In this article, I take the decentralized approach. When a backend web server detects too many failed login attempts within a window of time, it triggers a web request to EDA. EDA runs a playbook to ban the offending IP address on the load balancer.

The topology for this environment is shown below:

Scenario Topology

Set up the Environment

This sophisticated scenario can be accomplished with a surprisingly simple set of tools. The environment I use in this article contains the following components:

• A single load balancer running Nginx. The load balancer distributes traffic among the two backend servers.
• Two backend severs, also running Nginx. These servers are configured with HTTP basic authentication.
• Fail2ban running on each backend server. Fail2ban detects failed login attempts and triggers a remediation action.
• An automation host running Event-Driven Ansible with a webhook event source. This host listens for HTTP POST requests from the backend servers and triggers a playbook to block traffic.
• All hosts run Ubuntu 22.04

The load balancer has a default Nginx installation with a very simple configuration to forward traffic to the two backends:

upstream webapp {
    server 192.168.122.25;
    server 192.168.122.26;
}

server {
    listen 192.168.122.20:80 default_server;
    server_name _;

    proxy_set_header X-Forwarded-For $remote_addr;

    location / {
        proxy_pass http://webapp;
        include /etc/nginx/blocked_ips.conf;
     }
}

The configuration includes the file /etc/nginx/blocked_ips.conf. This is a file containing all of the IP addresses that have been blocked by the automated process:

touch /etc/nginx/blocked_ips.conf

The two backend web servers have a default Nginx installation with the root location protected by HTTP basic authentication:

server {
    listen 192.168.122.26:80 default_server;
    server_name _;

    root /var/www/html;
    index index.html index.htm index.nginx-debian.html;

    set_real_ip_from 192.168.122.0/24;
    real_ip_header X-Forwarded-For;

    location / {
        try_files $uri $uri/ =404;

        auth_basic "Enter password";
        auth_basic_user_file /etc/nginx/.htpasswd;
    }
}

Finally, the automation host has Event-Driven Ansible installed using the official installation instructions. The configuration of Fail2ban is discussed later in this article.

Write a Playbook

First, I need to define a playbook to automate the desired actions: blocking an IP address at the load balancer. The load balancer’s configuration file includes the /etc/nginx/blocked_ips.conf file, which will contain a list of IP addresses that are blocked using Nginx’s ngx_http_access module. The Ansible playbook only needs to add lines to this file, and the offending IP addresses will be blocked by the load balancer.

Ansible’s lineinfile module is ideal for this task. I’ve written this playbook to use an ip_state variable that dictates whether or not the IP is present in the blocked_ips.conf file. This allows the playbook to block or unblock an IP address, and this approach will be useful later when working with Fail2ban. Once the file is changed, the playbook restarts Nginx to pick up the changes:

---
# handle_ip.yaml

- name: Add or remove an IP address in the blocked IP list
  hosts: all
  gather_facts: false
  become: true
  vars:
    ip_address: ""
    ip_state: "present"
  tasks:

    - name: Block or unblock IP
      ansible.builtin.lineinfile:
        path: /etc/nginx/blocked_ips.conf
        line: "deny {{ ip_address }};"
        state: "{{ ip_state }}"
      when: ip_address
      notify: Reload Nginx

  handlers:

    - name: Reload Nginx
      ansible.builtin.service:
        name: nginx
        state: reloaded

The playbook runs against an inventory file with the load balancer host:

---
# inventory.yaml

ungrouped:
  hosts:
    loadbalancer.example.com:
      ansible_host: 192.168.122.20
      ansible_become_password: password

To run the playbook, specify the ip_address as an extra variable. By default, the playbook adds the IP address to the list of blocked IPs:

$ ansible-playbook -i inventory.yaml -e ip_address="192.168.1.155" handle_ip.yaml

PLAY [Add or remove an IP address in the blocked IP list] *******************************************

TASK [Block or unblock IP] **************************************************************************
changed: [loadbalancer.example.com]

RUNNING HANDLER [Reload Nginx] **********************************************************************
changed: [loadbalancer.example.com]

PLAY RECAP ******************************************************************************************
loadbalancer.example.com   : ok=2    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Checking the blocked_ips.conf list shows that the provided IP address is now blocked by the load balancer:

# cat /etc/nginx/blocked_ips.conf
deny 192.168.1.155;

To remove the IP address, simply re-run the playbook and set the ip_state to absent.

Write a Rulebook

EDA rulebooks provide a powerful mechanisms to listen for events, filter them, and respond by running playbooks or taking other actions. Rulebooks are composed of sources and rules. Sources define how EDA listens for events, while rules specify the actions that EDA takes in response to events.

EDA includes several event sources, but the webhook source is one of the most flexible and easy to use. The webhook event source exposes an HTTP server that listens for incoming HTTP POST requests. It provides information about the request path and JSON payload to use in the rulebook.

Create a rulebook.yaml file that defines a webhook server listening on 0.0.0.0:7777. EDA listens for any incoming HTTP POST requests on this endpoint, but it still needs a set of rules that define how to handle requests:

---
# rulebook.yaml

- name: Handle Fail2ban Ban and Unban Requests
  hosts: all
  sources:
    - ansible.eda.webhook:
        host: 0.0.0.0
        port: 7777

There are two scenarios to handle with this rulebook: banning an IP address and unbanning an IP address. EDA includes metadata about the event, making it easy to create an event for each situation.

Each rule defines a condition that specifies when it runs and an action to take. In this case, the rule should match the HTTP endpoint that the request is sent to: /ban-ip or /unban-ip. The rule action runs the handle_ip.yaml playbook, and it passes in two variables: the ip_address that it extracts from the JSON payload, and the ip_state. The playbook accepts the ip_state as a variable, so the playbook can easily be re-used between the two rules.

The full rulebook is below:

---
# rulebook.yaml

- name: Handle Fail2ban Ban and Unban Requests
  hosts: all
  sources:
    - ansible.eda.webhook:
        host: 0.0.0.0
        port: 7777
  rules:
    - name: Ban IP address
      condition: event.meta.endpoint == "ban-ip"
      action:
        run_playbook:
          name: handle_ip.yaml
          extra_vars:
            ip_address: "{{ event.payload.ip }}"
            ip_state: "present"

    - name: Unban IP address
      condition: event.meta.endpoint == "unban-ip"
      action:
        run_playbook:
          name: handle_ip.yaml
          extra_vars:
            ip_address: "{{ event.payload.ip }}"
            ip_state: "absent"

Trigger the Event

Next, it’s time to run and test the rulebook. I use the --verbose and --hot-reload flags when developing rulebooks, because they provide additional logging and automatically pick up any changes to the rulebook while I’m working on it:

ansible-rulebook --rulebook rulebook.yaml \
-i inventory.yaml --verbose --hot-reload

EDA is now running and listening on 0.0.0.0:7777 for incoming web requests. Test this out by sending a request to the /ban-ip endpoint:

# Construct a payload
$ cat -p payload.json
{
    "ip": "192.168.100.10"
}

$ curl -X POST --json @payload.json localhost:7777/ban-ip
ban-ip

The rulebook notices the event, matches it against the appropriate condition in the list of rules, and runs the playbook to ban an IP address:

2023-12-22 18:38:48,232 - aiohttp.access - INFO - 10.10.0.215 [22/Dec/2023:18:38:48 -0500] "POST /ban-ip HTTP/1.1" 200 157 "-" "curl/8.5.0"
2023-12-22 18:38:48 234 [main] INFO org.drools.ansible.rulebook.integration.api.rulesengine.RegisterOnlyAgendaFilter - Activation of effective rule "Ban IP address" with facts: {m={payload={ip=192.168.100.10}, meta={endpoint=ban-ip, headers={Host=localhost:7777, User-Agent=curl/8.5.0, Content-Type=application/json, Accept=application/json, Content-Length=31}, source={name=ansible.eda.webhook, type=ansible.eda.webhook}, received_at=2023-12-22T23:38:48.231906Z, uuid=f89fd2ee-6b81-4990-8319-b5954018938b}}}
2023-12-22 18:38:48,235 - ansible_rulebook.rule_generator - INFO - calling Ban IP address
2023-12-22 18:38:48,236 - ansible_rulebook.rule_set_runner - INFO - call_action run_playbook
2023-12-22 18:38:48,236 - ansible_rulebook.rule_set_runner - INFO - substitute_variables [{'name': 'handle_ip.yaml', 'extra_vars': {'ip_address': '{{ event.payload.ip }}', 'ip_state': 'present'}}] [{'event': {'payload': {'ip': '192.168.100.10'}, 'meta': {'endpoint': 'ban-ip', 'headers': {'Host': 'localhost:7777', 'User-Agent': 'curl/8.5.0', 'Content-Type': 'application/json', 'Accept': 'application/json', 'Content-Length': '31'}, 'source': {'name': 'ansible.eda.webhook', 'type': 'ansible.eda.webhook'}, 'received_at': '2023-12-22T23:38:48.231906Z', 'uuid': 'f89fd2ee-6b81-4990-8319-b5954018938b'}}}]
2023-12-22 18:38:48,238 - ansible_rulebook.rule_set_runner - INFO - action args: {'name': 'handle_ip.yaml', 'extra_vars': {'ip_address': '192.168.100.10', 'ip_state': 'present'}}
2023-12-22 18:38:48,238 - ansible_rulebook.action.run_playbook - INFO - ruleset: Handle Fail2ban Ban and Unban Requests, rule: Ban IP address
2023-12-22 18:38:48,509 - ansible_rulebook.action.run_playbook - INFO - Calling Ansible runner

PLAY [Add or remove an IP address in the blocked IP list] **********************

TASK [Block or unblock IP] *****************************************************
changed: [loadbalancer.example.com]

RUNNING HANDLER [Reload Nginx] *************************************************
changed: [loadbalancer.example.com]

PLAY RECAP *********************************************************************
loadbalancer.example.com   : ok=2    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
2023-12-22 18:38:54,422 - ansible_rulebook.action.runner - INFO - Ansible runner Queue task cancelled
2023-12-22 18:38:54,423 - ansible_rulebook.action.run_playbook - INFO - Ansible runner rc: 0, status: successful
2023-12-22 18:38:54,470 - ansible_rulebook.rule_set_runner - INFO - Task action::run_playbook::Handle Fail2ban Ban and Unban Requests::Ban IP address finished, active actions 0

Configure Fail2ban

Event-Driven Ansible is configured to properly handle incoming web requests and ban or unban IP addresses. Next, it’s time to use this configuration in a practical way by configuring Fail2ban to initiate ban and unban requests. Fail2ban is a powerful, lightweight service that scans log files for patterns and initiates actions, such as banning an IP address, in response to matches.

First, create a custom action for Fail2ban at /etc/fail2ban/action.d/ansible.local that sends an HTTP POST request to the EDA host for ban and unban actions. The request contains the offending IP address in its JSON payload:

# /etc/fail2ban/action.d/ansible.local

[Definition]
actionban   = curl -X POST \
                -d '{ "ip": "<ip>" }' \
                <ansible_webhook_host>:7777/ban-ip
actionunban = curl -X POST \
                -d '{ "ip": "<ip>" }' \
                <ansible_webhook_host>:7777/unban-ip

actioncheck =
actionstart =
actionstop =

[Init]

ansible_webhook_host = "192.168.122.1"

Fail2ban includes a built-in filter for NGINX HTTP authentication. Enable this filter and specify the newly created Ansible action in the /etc/fail2ban/jail.d/nginx.local file.

# /etc/fail2ban/jail.d/nginx.local
[nginx-http-auth]
enabled = true
action_ = ansible

Finally, restart fail2ban:

systemctl restart fail2ban

Test Everything Out

It’s time to test out the full automation flow. First, it’s helpful to think about the overall process one last time:

1. A user hits the load balancer with multiple HTTP requests with bad authentication credentials
2. The load balancer forwards these requests to the backend web servers
3. The backend web server logs the failed authentication attempts
4. Fail2ban notices that multiple failed authentication attempts are coming from the same IP address. It sends an HTTP POST request to the Event-Driven Ansible host.
5. EDA receives the incoming HTTP POST event, matches it against the appropriate rule in the rulebook, and executes the playbook to ban the IP address.
6. The load balancer is now reconfigured to drop the offending client’s traffic

Testing this setup is the easiest part of the process. First, make sure that the current blocked_ips.conf file is empty and restart Nginx on the load balancer:

echo -n > /etc/nginx/blocked_ips.conf
systemctl restart nginx

Next, confirm that valid requests are able to access the web page:

$ curl -u admin:password 192.168.122.20
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Finally, send several requests with invalid credentials to the web endpoint. You will receive HTTP 401 Unauthorized responses:

# Repeat this bad request several times
$ curl -u admin:badpassword 192.168.122.20
<html>
<head><title>401 Authorization Required</title></head>
<body>
<center><h1>401 Authorization Required</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

Eventually, Fail2ban will notice the influx of failed authentication attempts and send an HTTP request to the EDA host, triggering the playbook:

2023-12-22 18:40:47,746 - aiohttp.access - INFO - 192.168.122.207 [22/Dec/2023:18:40:47 -0500] "POST /ban-ip HTTP/1.1" 200 157 "-" "curl/7.81.0"
2023-12-22 18:40:47 748 [main] INFO org.drools.ansible.rulebook.integration.api.rulesengine.RegisterOnlyAgendaFilter - Activation of effective rule "Ban IP address" with facts: {m={payload={ip=192.168.122.100}, meta={endpoint=ban-ip, headers={Host=192.168.122.1:7777, User-Agent=curl/7.81.0, Accept=*/*, Content-Length=27, Content-Type=application/x-www-form-urlencoded}, source={name=ansible.eda.webhook, type=ansible.eda.webhook}, received_at=2023-12-22T23:40:47.746269Z, uuid=91013291-7e77-4106-a009-e5ba75b7e781}}}
2023-12-22 18:40:47,749 - ansible_rulebook.rule_generator - INFO - calling Ban IP address
2023-12-22 18:40:47,749 - ansible_rulebook.rule_set_runner - INFO - call_action run_playbook
2023-12-22 18:40:47,750 - ansible_rulebook.rule_set_runner - INFO - substitute_variables [{'name': 'handle_ip.yaml', 'extra_vars': {'ip_address': '{{ event.payload.ip }}', 'ip_state': 'present'}}] [{'event': {'payload': {'ip': '192.168.122.100'}, 'meta': {'endpoint': 'ban-ip', 'headers': {'Host': '192.168.122.1:7777', 'User-Agent': 'curl/7.81.0', 'Accept': '*/*', 'Content-Length': '27', 'Content-Type': 'application/x-www-form-urlencoded'}, 'source': {'name': 'ansible.eda.webhook', 'type': 'ansible.eda.webhook'}, 'received_at': '2023-12-22T23:40:47.746269Z', 'uuid': '91013291-7e77-4106-a009-e5ba75b7e781'}}}]
2023-12-22 18:40:47,751 - ansible_rulebook.rule_set_runner - INFO - action args: {'name': 'handle_ip.yaml', 'extra_vars': {'ip_address': '192.168.122.100', 'ip_state': 'present'}}
2023-12-22 18:40:47,751 - ansible_rulebook.action.run_playbook - INFO - ruleset: Handle Fail2ban Ban and Unban Requests, rule: Ban IP address
2023-12-22 18:40:48,034 - ansible_rulebook.action.run_playbook - INFO - Calling Ansible runner

PLAY [Add or remove an IP address in the blocked IP list] **********************

TASK [Block or unblock IP] *****************************************************
changed: [loadbalancer.example.com]

RUNNING HANDLER [Reload Nginx] *************************************************
changed: [loadbalancer.example.com]

PLAY RECAP *********************************************************************
loadbalancer.example.com   : ok=2    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
2023-12-22 18:40:53,871 - ansible_rulebook.action.runner - INFO - Ansible runner Queue task cancelled
2023-12-22 18:40:53,872 - ansible_rulebook.action.run_playbook - INFO - Ansible runner rc: 0, status: successful
2023-12-22 18:40:53,919 - ansible_rulebook.rule_set_runner - INFO - Task action::run_playbook::Handle Fail2ban Ban and Unban Requests::Ban IP address finished, active actions 0

Once the IP is blocked by the load balancer, the bad requests will begin receiving HTTP 403 Forbidden responses. These are sent by the load balancer, and the requests never reach the backend web servers:

$ curl -u admin:badpassword 192.168.122.20
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

If you want to test out the unban workflow, you can force Fail2ban to unban specific clients, or simply unban all clients. This will trigger the EDA rulebook to run and unban any IP addresses that are currently banned:

fail2ban-client unban --all

Wrapping Up

The event-driven architectural paradigm has gained widespread traction in software development for its ability to dynamically respond to change without regularly polling a system. In many cases, this results in more robust and efficient systems. This approach is gradually spreading to the world of operations: instead of Cron jobs that regularly poll system state, we are beginning to rely on even-driven approaches, such as Kubernetes Controllers.

Event-Driven Ansible is an excellent way to leverage the power of Ansible in response to changes in your environment. In this article, you saw a practical and easily adaptable example of EDA: triggering a playbook to reconfigure a load balancer in response to changes on backend web servers. This just scratches the surface of EDA’s capabilities, but it should provide you with a good base to start using EDA and harnessing the power of Ansible in event-driven environments.