2021-08-31 10:04:52 +00:00
2021-08-30 16:38:33 +00:00
2021-02-05 22:49:47 +00:00
2021-08-30 18:07:20 +01:00
2021-02-05 22:49:47 +00:00
2021-08-30 18:07:20 +01:00
2021-08-31 10:04:52 +00:00

Fail2Ban Prometheus Exporter

Go tool to collect and export metrics on Fail2Ban

Table of Contents

  1. How to use
  2. Docker
  3. CLI usage
  4. Metrics

1. How to use

The exporter can collect metrics from 2 locations: the fail2ban server socket, and the fail2ban server database.

Once the exporter is running, metrics are available at localhost:9191/metrics. The default port is 9191, but this can be modified with the -port flag.

1.1. Socket

The recommended way to run the exporter is to point it at the fail2ban server socket. This allows the exporter to communicate with the server in the same way fail2ban-client does and ensures the metrics it collects are exactly the same as the values reported by fail2ban-client status <jail>.

The default path to the socket is: /var/run/fail2ban/fail2ban.sock

1.2. Deprecated: Database

Reading metrics from the database has been deprecated in favour of using the socket. The reason being that database metrics do not always align with the output of fail2ban-client status <jail> and cause confusion. See #11 for more details.

To run the exporter in this mode:

Run the exporter with the path to the fail2ban database. The default path to the database is: /var/lib/fail2ban/fail2ban.sqlite3

2. Docker

An official docker image is available on the Gitlab container registry. Use it by pulling the following image:

registry.gitlab.com/hectorjsmith/fail2ban-prometheus-exporter:latest

Use the :latest tag to get the most up to date code (less stable) or use one of the version tagged images to use a specific release. See the registry page for all available tags.

2.1. Volumes

The docker image is designed to run by mounting either the fail2ban sqlite3 database of the fail2ban run folder.

  • The database should be mounted at: /app/fail2ban.sqlite3
  • The run folder should be mounted at: /var/run/fail2ban

Both paths can be mounted with readonly (ro) permissions.

NOTE: While it is possible to mount the fail2ban.sock file directly, I recommend mounting the parent folder instead. The .sock file is deleted by fail2ban on shutdown and then re-created on startup and this causes problems for the docker mount.

2.2. Docker run

Use the following command to run the forwarder as a docker container.

docker run -d \
    --name "fail2ban-exporter" \
    -v /var/lib/fail2ban/fail2ban.sqlite3:/app/fail2ban.sqlite3:ro \
    -v /var/run/fail2ban:/var/run/fail2ban:ro \
    -p "9191:9191"
    registry.gitlab.com/hectorjsmith/fail2ban-prometheus-exporter:latest

2.3. Docker compose

The following is a simple docker-compose file to run the exporter.

version: "2"
services:
  exporter:
    image: registry.gitlab.com/hectorjsmith/fail2ban-prometheus-exporter:latest
    volumes:
    - /var/lib/fail2ban/fail2ban.sqlite3:/app/fail2ban.sqlite3:ro
    - /var/run/fail2ban/:/var/run/fail2ban:ro
    ports:
    - "9191:9191"

3. CLI usage

$ fail2ban-prometheus-exporter -h

  -db string
        path to the fail2ban sqlite database (deprecated)
  -port int
        port to use for the metrics server (default 9191)
  -socket string
        path to the fail2ban server socket
  -version
        show version info and exit

4. Metrics

Access exported metrics at /metrics (on the provided port).

Fail2Ban Jails

fail2ban can be configured to process different log files and use different rules for each one. These separate configurations are referred to as jails.

For example, fail2ban can be configured to watch the system logs for failed SSH connections and Nextcloud logs for failed logins. In this configuration, there will be two jails - one for IPs banned from the SSH logs, and one for IPs banned from the Nextcloud logs.

This tool exports several metrics per jail, meaning that it is possible to track how many IPs are being banned in each jail as well as the overall total. This can be useful to track what services are seeing more failed logins.

4.1. Socket Metrics

Exposed metrics:

  • up - Returns 1 if the fail2ban server is up and connection succeeds
  • errors - Number of errors since startup
    • db - Errors connecting to the database
    • socket_conn - Errors connecting to the fail2ban socket (e.g. connection refused)
    • socket_req - Errors sending requests to the fail2ban server (e.g. invalid responses)
  • jail_count - Number of jails configured in fail2ban
  • jail_banned_current (per jail) - Number of IPs currently banned
  • jail_banned_total (per jail) - Total number of banned IPs since fail2ban startup (includes expired bans)
  • jail_failed_current (per jail) - Number of current failures
  • jail_failed_total (per jail) - Total number of failures since fail2ban startup

Sample

# HELP f2b_errors Number of errors found since startup
# TYPE f2b_errors counter
f2b_errors{type="db"} 0
f2b_errors{type="socket_conn"} 0
f2b_errors{type="socket_req"} 0
# HELP f2b_jail_banned_current Number of IPs currently banned in this jail
# TYPE f2b_jail_banned_current gauge
f2b_jail_banned_current{jail="recidive"} 5
f2b_jail_banned_current{jail="sshd"} 15
# HELP f2b_jail_banned_total Total number of IPs banned by this jail (includes expired bans)
# TYPE f2b_jail_banned_total gauge
f2b_jail_banned_total{jail="recidive"} 6
f2b_jail_banned_total{jail="sshd"} 31
# HELP f2b_jail_count Number of defined jails
# TYPE f2b_jail_count gauge
f2b_jail_count 2
# HELP f2b_jail_failed_current Number of current failures on this jail's filter
# TYPE f2b_jail_failed_current gauge
f2b_jail_failed_current{jail="recidive"} 5
f2b_jail_failed_current{jail="sshd"} 6
# HELP f2b_jail_failed_total Number of total failures on this jail's filter
# TYPE f2b_jail_failed_total gauge
f2b_jail_failed_total{jail="recidive"} 7
f2b_jail_failed_total{jail="sshd"} 125
# HELP f2b_up Check if the fail2ban server is up
# TYPE f2b_up gauge
f2b_up 1

The metrics above correspond to the matching fields in the fail2ban-client status <jail> command:

Status for the jail: sshd|- Filter
|  |- Currently failed: 6
|  |- Total failed:     125
|  `- File list:        /var/log/auth.log
`- Actions
   |- Currently banned: 15
   |- Total banned:     31
   `- Banned IP list:   ...

4.2. Database Metrics (deprecated)

Exposed metrics:

  • up - Returns 1 if the service is up
  • errors - Returns the number of errors found since startup
  • enabled_jails - Returns 1 for each jail that is enabled, 0 if disabled.
  • bad_ips (per jail)
    • A bad IP is defined as an IP that has been banned at least once in the past
    • Bad IPs are counted per jail
  • banned_ips (per jail)
    • A banned IP is defined as an IP that is currently banned on the firewall
    • Banned IPs are counted per jail

Sample

# HELP fail2ban_bad_ips Number of bad IPs stored in the database (per jail).
# TYPE fail2ban_bad_ips gauge
fail2ban_bad_ips{jail="jail1"} 6
fail2ban_bad_ips{jail="jail2"} 8
# HELP fail2ban_banned_ips Number of banned IPs stored in the database (per jail).
# TYPE fail2ban_banned_ips gauge
fail2ban_banned_ips{jail="jail1"} 3
fail2ban_banned_ips{jail="jail2"} 2
# HELP fail2ban_up Was the last fail2ban query successful.
# TYPE fail2ban_up gauge
fail2ban_up 1
# HELP fail2ban_enabled_jails Enabled jails.
# TYPE fail2ban_enabled_jails gauge
fail2ban_enabled_jails{jail="jail1"} 1
fail2ban_enabled_jails{jail="jail2"} 1
# HELP fail2ban_errors Number of errors found since startup.
# TYPE fail2ban_errors counter
fail2ban_errors{type="db"} 0
Description
Collect and export metrics on Fail2Ban
Readme MIT 11 MiB
Languages
Go 92.2%
Makefile 6.6%
Dockerfile 1.2%