Watcher concepts

With Version 1.0 (as of 2018) Watcher was modularized and broken up into:

Watcher Master
Dynamic loaders (fetching attacker lists from common resources; e.g. SpamHaus & NixSpam; both included for free with the master package)
Modules
- LG module WatchLG (Login/SSH)
- MX module WatchMX (Mail transport) and the companion process WatchMB that gets passed all issues that have to do with mailbox break-in (POP,IMAP,...) and transport authentification (SASL)
- WB module WatchWB (new with Watcher 1.3) tracks the logs from web servers and can track logs 'by instance'; i.e. separately by every definded vhost.

Watcher concept picture

In the early days (~2012-2015) what is the Watcher Master today was a monolithic BASH script "WatchFW" that got all the load and the mail services (PostFix) and WEB services were left to self-defense.

Also retrieving 'bad IP addresses' by `grep'ing or `tail'-reading log files was fairly awkward, since with such an asynchronous measure one cannot get a grasp on brute-force login attacks or other higher-level attack methods like DoS/DDoS or high-jacking of services. To cut these drawbacks down it takes 'realtime intrusion detection' with instant measures on detection. Recording determined attackers in files and scan these files by a 'linear scan' can slow down the entire detection process to be a crawler. In particular filling the firewall with iptables commands one-by-one can take hours after start-up or reboot if the collection of badfinger-IP addresses goes to several thousands and then even several ten-thousands as today's situation shows.

So the autonomous modules do not use normal files but store the detected bandits in exclusive databases established with Sqlite. This assures a fast retrieval of already registered bandits even in a growing 'list' of bandits stored in a database. On the average his is much faster and takes the same (short) time.

The DynLoaders have always been part of the Watcher project since the beginning in 2010 and are still part of it as they provide valuable information to get some protection for 'high-jackers' and 'SPAM spreaders' of commonly world-wide known internet bandits. (see SpamHaus DROP)

Why the module concept?

Infact the Watcher master now terminates after it has collected the firewall information from DynLoaders and Modules and finally has fired-up the modules that run completely autonomous if once started by the master.

Each module does its own job (for mailbox logins & mail transport attacks) and the load is spread accross several processes. This potentially speeds up attacker determination.
Each module now has its own exclusive database. So the concurrency on the database is potentially dimished (Note: The MB module shares the database of the MX module and provides in the 'origin' column the origin of the information)
A module can be stopped (e.g. for maintenance) without affecting other modules. One can even stop the entire Watcher service without affecting the fill state of the firewall. The firewall is just kept in its state - but no further entries are applied.
Even better: Since the modules are fed syncronously by the system logger through FiFos ('named pipes') the system logger continues to fill the FiFos and so buffers the log information which then is flushed and processed by the module if it is restarted; i.e. there is no loss of attack notification - it's just a little delayed ...
Each module does its own automatic house-keeping by expiration of the database on a regular basis (CRON driven, configurable) to keep the database compact and remove pointless IP address as a flooded firewall can have impact on the overall network performance of your services.
Each module has its own statistics tool (StatLG/StatMX/StatWB) that can send a statictics file in CSV format to a configured $REPORT_MAIL address that is globally configured in watcher.conf and can be overwritten by the module's configuration file in the module's installation directory.

Dynamic rule system

With Watcher Version 1.2 the 'dynamic rule system' was introduced for modules.

The rules are no longer 'hard coded' in the module's code but are in a sub-directory named 'rules' in separate "*.rule" files. These files are 'assembled' into a 'filter block' every time the module is started. This way -and in particular the MX module- can be adapted to any MTA (postfix, exim, qmail, etc.) and mailbox service (DBmail, dovecot, etc.) of choice.

The supplied rule-sets in the distribution are ready-made for:

sshd (LG module)
postfix & DBmail (MX/MB module)
apache (WB module; coming with watcher 1.3)

For other services (EXIM, DOVECOT, NGINX, ...) the rules can easily be changed by just modifying the 'Pattern' variable in the rule's heading:

RULE="NXdomain"
Pattern=': connect from unknown['
(... decision block ...)

All rules reside in a subdirectory 'rules' under the module's path. The rule files start with a number, since sometimes the order of the rules counts. Each rule then has the suffix '.rule'

[root@vmd28527 WatchMX]# ls rules/
100-NXDOMAIN.rule      150-Host.rule-disabled  300-MB-pass.rule  DISABLED
110-NXDOMAIN-TLS.rule  170-NOQUEUE.rule        check-all-rules   README
120-NXDOMAIN-SSL.rule  180-SSL-frauds.rule     check-rule        superflous_map

On module start-up the module code then simply collects the *.rule files into a 'filter' function and so dynamically creates the filter chain for the specific module.

(Schematically)

filter() {
   `cat rules/*.rule`
}

A 'superflous_map' file provides a REGEX list of patterns to be ignored by the scanner loop to skip log lines before the 'filter' function is reached. This speeds up tracking speed of the scanner tremendously.