fwanalog: A firewall log summarizer that uses Analog

http://tud.at/programm/fwanalog/

Balázs Bárány, balazs@tud.at

Current version: 0.6.9

This program summarizes firewall logs and creates reports from them.

There are lots of programs that do this. But they weren't good enough for me:
- I use OpenBSD and GNU/Linux, so I need a solution that can handle both.
- I want pretty reports, see recent attacks, host names instead of IP
  addresses, HTML output.
- I want a daily report mailed to me, of course in text format.

I use the excellent logfile analyzer Analog ( http://www.analog.cx/ ) a lot.
It is very flexible. So I thought I could convert firewall logs to web
server logs Analog can use.

This shell script does exactly that.

1. It parses the firewall log (I can choose which one) and converts it to a
logfile that Analog understands. The fields in this file are faked, of course;
e.g. the network interface name is the virtual host.

2. It calls Analog with some nice options and so creates different reports:
one about all data; one about the last week; one about the current day in
HTML format and a short one in ASCII format for a report e-mailed daily.

3. Optionally, it can create a separate report for each host and each
blocked packet so you can look at the actions of a "bad guy" or answer 
the question "who scanned port 443?".

Requires: Perl, zegrep, awk, diff, sed, egrep, and of course Analog.
Most of these programs should be already installed on any Unix system.
You should really have the GNU versions somewhere and configure them in
fwanalog.opts.

Install Analog:
Debian GNU/Linux: 	"apt-get install analog"
Any modern BSD:		"cd /usr/ports/www/analog; make"
Other GNU/Linux:	probably there are RPMs for your distribution
Any other OS:		Download from http://www.analog.cx/ , compile, install

Installation
============
1. Decompress the distribution in some directory, e.g. /usr/local/fwanalog
2. Symlink, move or copy the fwanalog.opts.{your OS} to "fwanalog.opts"
3. Edit fwanalog.opts if necessary (most settings should be OK, though)
4. If your Analog version is not the newest stable one, find a language 
   file for it in the langfiles/ directory and copy it over fwanalog.lng 
5. On a non-free Unix (e.g. Solaris), modifiy the first line of the
   fwanalog.sh script to "#! /bin/bash" or where your bash or ksh shell
   is. Also, look if you have the GNU versions of the utilites listed in
   fwanalog.opts. 
6. Execute ./fwanalog.sh
7. There should be some HTML and text reports in the directory you specified
   in fwanalog.opts ("$outdir").

Customizing
===========
You can edit fwanalog.analog.conf.local to suit your taste, e.g. add pretty
icons and style sheets, switch reports on and off (however, the deactivated
reports don't make much sense with firewall logs). It is better not to edit
the master fwanalog.analog.conf yourself as that file probably will be
updated by me in the next fwanalog release.

You can also edit fwanalog.sh and change the Analog command line options,
deactivate reports and create a conversion function for your firewall if it is
not supported. It's easy. If you think that your changes made the program
better, please send them to me so I can include them in the next version.

Troubleshooting
===============
Some frequent problems:
- The language file doesn't match the Analog version. This can happen with
  new installations or after an upgrade of Analog or fwanalog. 
  Make sure that you use one the correct version of the language file; the
  major and the first minor version numbers must match. (E.g. analog 5.32
  works with the 5.3 langfile.)
- "It works perfectly when called from the command line but not when called
  from cron!" - Search for differences between your shell's and the cron
  shell's configuration. The cron path sometimes doesn't include
  /usr/local/bin where Analog can be etc.

If you have a problem with fwanalog, go to the homepage and read through the
mailing list archives. Many common problems are already solved there. If
not, subscribe to the mailing list and ask there so more people can help
you. I don't have time to answer e-mails with problems that can be solved by
reading the documentation and/or the knowledge in the mailing list archives.

"One host" mode
===============
You can set "onehost=" to true in fwanalog.opts if you are analyzing the
logs of only one host. This will cause fwanalog to show each packet source 
host (i.e. attacker) with the ports it tried. 
However, setting this option loses the information about the target IP
address. So don't set this if your firewall protects an entire network.
This feature is based on an idea by Kenneth Vestergaard Schmidt, who is
also the Debian maintainer for fwanalog.
There is also an option "onehost=dynip" based on an idea by Ralph Niere.
This is useful if the address of your firewall changes often, e.g. because
you are on a dial-up connection with dynamic IPs.

Creating separate reports of hosts and packets
==============================================
In fwanalog.opts, set sep_hosts and/or sep_packets to true. Note that this
will cause analog to run once for each host and each packet in the "current"
log. This shouldn't be a problem on a modern machine when fwanalog is run
periodically (e.g. once a day).
As this processes the current log, it will probably run for a long time when
you run fwanalog. If you update from an older version of fwanalog, it will
only process the new log entries since the last invocation, so only a few
packets and hosts will be linked in the reports. You can call analog with
"-a host" or "-p packet" to create a report for a host or a packet you are
interested in. In the future, this host or packet will always be linked in
the reports.

Services
========
fwanalog includes a services.conf file for Analog to convert port numbers
like 21 into service names like ftp. If you think that your services list
is better, feel free to use support/mkservices.conf.sh with your list.

There is also a well_known_services.conf file in support/. It includes lots
of port definitions of more-or-less well known ports. You can include that
file by simply appending it to services.conf or by editing
fwanalog.analog.conf.local to include it. However, so many aliases make 
analog slower.

Creating conversion functions for unsupported firewall formats
==============================================================
If your firewall is not supported, please contribute a conversion routine. 
It is not very hard:
1. Add your format in fwanalog.opts to the known ones.
2. Copy the ipf or iptables function in fwanalog.sh into a new function with
   the name of your firewall (the same you added to fwanalog.opts).
3. Grep the lines about blocked packets from your firewall log into the
   fwlog.current file in the output directory
4. Call mkdateconvscript and sed if your log file doesn't contain years.
5. Change the long perl regexp (or use any other tool if you like) so it
   changes all lines into the faked web server log format. It is not very
   hard if you know regular expressions. Be careful with \$!

Language files
==============
Language files define the strings in the fwanalog output. 
Most versions of analog require language files of a matching version.
You might find a language file for your Analog version (and your language,
if you prefer) in the langfiles/ subdirectory of the fwanalog distribution.
If you have a currently unsupported version of Analog, try the mklangfile
scripts in the support/ directory of the fwanalog distribution. Please 
contribute language files you have created by submitting them to the author.

Report mappings
===============
fwanalog renames some Analog reports. Not all reports are switched on by default.
(See http://www.analog.cx/docs/output.html for details on Analog's reports)

Analog report       fwanalog report     remarks
-------------       ---------------     -------
GENERAL             General summary     The first report, gives an overview
YEARLY              Yearly report       Makes sense if you have firewall logs for more than a year
QUARTERLY           Quarterly report    Makes sense if you have firewall logs for more than 3 months
MONTHLY             Monthly report      
WEEKLY              Weekly report       See also the "WEEKBEGINSON" in fwanalog.analog.conf.local
DAILYREP            Daily report        
DAILYSUM            Daily summary       Summary by weekdays
HOURLYREP           Hourly report       
HOURLYSUM           Hourly summary      Summary by hour of day
WEEKHOUR            Hour of the Week    Summary
QUARTERREP          Quarter-hour report 
QUARTERSUM          Quarter-hour summary
FIVEREP             Five-minute report
FIVESUM             Five-minute summary
HOST                Packet Source Host  Which hosts sent the packets that your firewall blocked
REDIRHOST           -                   Doesn't make sense with firewall logs
FAILHOST            -                   Doesn't make sense with firewall logs
ORGANISATION        Organization report 
DOMAIN              Domain report       top level domains
REQUEST             -					Not used: the directory report is better
										suited for fwanalog.
DIRECTORY           Blocked Packet      Detailed report of blocked packets.
                                        If onehost=false, the target address;
                                        if onehost=true, the source address and the target port.
FILETYPE            -                   Doesn't make sense with firewall logs
SIZE                Packet Size         Not many variations with some firewall settings
PROCTIME            Processing time     Not very interesting
REDIR               -                   Doesn't make sense with firewall logs
FAILURE             -                   Doesn't make sense with firewall logs
REFERRER            Source Port         Sometimes interesting, e.g. with port 21
REFSITE             -                   Doesn't make sense with firewall logs
SEARCHQUERY         -                   Doesn't make sense with firewall logs
SEARCHWORD          -                   Doesn't make sense with firewall logs
INTSEARCHQUERY      -                   Doesn't make sense with firewall logs
INTSEARCHWORD       -                   Doesn't make sense with firewall logs
REDIRREF            -                   Doesn't make sense with firewall logs
FAILREF             -                   Doesn't make sense with firewall logs
BROWSERREP          -                   MAC Address report (if your firewall logs them)
BROWSERSUM          -                   Doesn't make sense with firewall logs
OSREP               -                   Would be nice, but no firewall logs it 8-(
VHOST               Interface Report    You can turn it off if you have only one interface
REDIRVHOST          -                   Doesn't make sense with firewall logs
FAILVHOST           -                   Doesn't make sense with firewall logs
USER                Log Prefix Report	Only with iptables, if you set a log prefix
										Analog ignores this if it sees no data.
REDIRUSER           -                   Doesn't make sense with firewall logs
FAILUSER            -                   Doesn't make sense with firewall logs
STATUS              -                   Doesn't make sense with firewall logs

OpenBSD 3.x problem
===================
The developers of the new OpenBSD firewall "pf" decided that they log
blocked packets in a binary format instead of a text as usual on Unix.
This file can be only read by the OpenBSD version of tcpdump. So,
fwanalog must run on the OpenBSD 3.x machine itself in order to process
OpenBSD 3.x logfiles. 
All other logfiles can be handled on each architecture, e.g. a Linux 2.4
machine can process the logfiles of Solaris, or FreeBSD the logs of
Linux 2.2 etc. 

Other documentation
===================
See README.firewall for hints on configuring your firewall.
See README.sudo for information about running fwanalog as a non-root user.

Please mail your suggestions, patches, bugfixes etc. to balazs@tud.at .

$Id: README,v 1.30 2004/03/18 16:40:17 bb Exp $
