Appendix A - Additional information

A.1 Watcher directory structure

Seen from a dynloader or module process the Master-Path is always 2 levels up; i.e.: ../../

A.2 Dynloaders

‘Dynamic loaders’ (dynload) are utility programs that read IP address lists from external resources: i.e. from outside the Watcher framework. This can be files on local filesystems, remote systems or from somewhere on the Internet of which the dynloader knows how to retrieve them.

A dynloader is kind of a ‘translator’ that conducts a transition of a proprietary external list format into a compliant format for the IPSET that the symlinked ‘Load’ program outputs for the exclusive ‘ipset.Loadfile-XXXX’ that is then in turn filled-in into exclusively assigned IPSET. This process is fully dynamic since Watcher release 1.3 and ‘reload’ of the entire firewall is no longer needed which speeds up the loading of the firewall tremendously. The entire load time for all dynloaders and modules is with Watcher release 1.3 usually below a second.

[root@vmd28527 Pool]# head -10 ipset.Loadfile-LG 
-exist create WatchLG-DB hash:ip comment
add WatchLG-DB 175.6.35.82 comment "DB,Login,NXDOMAIN"
add WatchLG-DB 106.54.253.9 comment "DB,Login,NXDOMAIN"
add WatchLG-DB 221.181.185.198 comment "DB,Login,NXDOMAIN"
add WatchLG-DB 176.112.79.111 comment "DB,Login,NXDOMAIN"
add WatchLG-DB 51.15.81.22 comment "DB,Login,FAKEHOST"
add WatchLG-DB 106.12.219.184 comment "DB,Login,NXDOMAIN"
add WatchLG-DB 221.181.185.159 comment "DB,Login,NXDOMAIN"
add WatchLG-DB 176.122.164.94 comment "DB,Login,TRUEHOST"
add WatchLG-DB 95.165.131.164 comment "DB,Login,TRUEHOST"
.
. (and so on)
.

Every dynloader has a ‘Load’ routine which is usually just a sym-link to the main program:

root@vmd28527 Watcher]# cd dynload/spamhaus
root@vmd28527 spamhaus]# ls -l Load spamhaus
lrwxrwxrwx   1 root root   8      Nov 5  11:24  Load -> spamhaus
-rwxr-xr-x   1 root root   1995   Nov 18 21:55  spamhaus

So one must not know the individual name of the loader routine. It can be just called by the name ‘Load’ with the full path.

This makes the entries in the CRONTAB for the updates easy to read.

This load routine is automatically called once by the Watcher service program during service start when the Watcher service includes ‘loader conf’ from the Watcher MASTER_PATH.

For the consecutive updates of provided data a CRONTAB entry is needed that matches the timely update policy of the external provider.

root@vmd28527 Watcher]# cat loader.conf
#----------------------------------------------------------------
#loader.conf (used by FILLFW - the Firewall filler)
#Startup lines for the modules and dynloader (dynamic loaders)
# 
#To disable load from a module or dynloader just clamp it off with
#a comment character in the first column
#----------------------------------------------------------------
modules/WatchLG/Load
modules/WatchMX/Load
dynload/spamhaus/Load
#dynload/nixspam/Load

A.2.1 Repeated dynloader calls

Dynloaders are called only once during startup of the Watcher service.

The Watcher service cannot know how external resources schedule their provisioning and the changes to it. So subsequent calls to a dynloader must be configured through CRONTAB entries that match the schedule of informations by the external providers.

(excerpts of crontab for 'root’)

#- DynLoader : Once per hour
10 * * * * cd /root/bin/Watcher/dynload/spamhaus && ./Load
30 * * * * cd /root/bin/Watcher/dynload/nixspam  && ./Load

The ‘Load’ routine of the dynloader then manages to retrieve refreshed information.

This will create a new ‘ipset.Loadfile-<dynloader name>’ in the framework’s loadpool:

../../Pool/ipset.Loadfile-xxxx
(as seen from the dynloader)

For the firewall the retrieval of new/changed information from an external provider is fully transparent (i.e. not seen).

With the consequent introduction of IPSETs in Watcher release 1.3 this process is fully dynamic and there is no longer any need to ‘reload’ the entire ‘chain’ in the firewall if changes happened.

Other dynloaders (or modules) are not affected by this. But you should keep in mind that the ‘refresh calls’ from external providers should run with a little time lag to avoid concurrency; i.e.: don’t run them at the same minute; i.e. keep them some minutes apart.

#- DynLoader : Once per hour
10 * * * * cd /root/bin/Watcher/dynload/spamhaus && ./Load
30 * * * * cd /root/bin/Watcher/dynload/nixspam  && ./Load

A.3 Rolling your own custom dynloader

What if you have some IP address list from other providers with ‘badfinger’ listings, that you would like to integrate with Watcher?

Here a schematic concept is shown on how to roll your dynloader.
To get a clue of the construction you may verify with the included dynloaders ‘spamhaus’ & ‘nixspam’.

It is assumed that the provider delivers the ‘attacker list’ as a simple *.txt file. Infact it can be any format. But that is abstracted/encapsulated by the ‘extraction function’ in your dynloader code.

First of all your code should contain the usual ‘initialization block’ in the program heading that is common for all processes in the Watcher concept for ‘positioning’ and ‘naming’.

#!/bin/bash
if [ "$1" == "debug"  ]; then set -x;   shift; fi
if [ "$1" == 'debug2' ]; then set -xvT; shift; fi
#
# Plug MyDynLoader lists dynamically into firewall ...
#
#-------------------------
REALPATH=`realpath $0`      # Where is my absolute file position?
WHERE=`dirname $REALPATH`   # What is my path? 
ME=`basename $REALPATH  `   # What’s my program name?
cd $WHERE                   # Finally hook to the path where we are called
#-------------------------
# Get common variable definitions and library functions from the master.
source ../../common.conf

trap cleanup 0 1 2 9 15

cleanup() {
        logger "$ME[$$]: Finished."
}

You may add to the cleanup() function as you like; e.g. removing temporary files that your code has generated. But do not place your ‘exit code’ here.

The program code for a dynloader is fairly straight forward on the whole.

Include the grey parts of the coding scheme at first as these comprise the Init, Main & Exit sections.

Then build-up your functions step-by-step and check the results that you get.

Check your code thoroughly before you integrate it in your loader.conf and/or have it fired up by a CRONTAB entry.

If the ipset.Loadfile-$ME from your custom dynloader produces a clean file to be read by ipset, then symlink your custom dynloader script to the common name ‘Load’ in the dynloader’s path:

# ln -s MyDynLoader Load

With this preparation you can include your custom dynloader to the ‘loader.conf' file in the MASTER_PATH.

...
modules/WatchLG/Load
modules/WatchMX/Load
dynload/spamhaus/Load
#dynload/nixspam/Load
dynload/MyDynLoader/Load

Last but not least create a CRONTAB entry in the crontab file for the super-user ‘root’ to get your custom dynloader started on a regular basis:

(crontab ‘root’)
...
# --- DynLoader: Once per hour
10 * * * *  cd /root/bin/Watcher/dynload/spamhaus     && ./Load
30 * * * *  cd /root/bin/Watcher/dynload/nixspam      && ./Load
50 * * * *  cd /root/bin/Watcher/dynload/MyDynLoader  && ./Load

Whether your custom dynloader really runs regularly you may check /var/log/messages file, since at least the cleanup() function writes to the log file when it lately has finished.