IFSTAT network data visualized

Posted: July 22nd, 2009 | Author: | Filed under: Network, Security | Comments Off on IFSTAT network data visualized

I have many computer systems on my network that stay powered on 24/7 (these are mostly servers but also many desktops systems stay on for a variety of reasons). Monitoring network bandwidth usage on such systems can be quite desirable. It can help to detect, let’s call them, “unofficial” activities by users. It can also help to detect presence of a network active malware. Whatever your reasons are, if you are interested in network monitoring, IFSTAT program is your friend.

IFSTAT – http://gael.roualland.free.fr/ifstat/ – is a great tool written by Gaël Roualland for network usage monitoring on individual computer hosts . IFSTAT runs on most computer platforms, it is very easy to install, and very easy to use.

IFSTAT generates large volumes of of data that is not easy to analyze especially when one has to deal with data coming from tens of hosts. A graphical presentation of IFSTAT data should substantially facilitate analysis. I developed a simple tool to collect and to visualize data collected by IFSTAT.

My tool is intended for networks of computers. It is not intended (and probably not very useful) for a home computer user with a couple of systems on the network.

Approach

IFSTAT is executed on individual computer hosts in a continuous or a semi-continuous manner. IFSTAT data is sent periodically to a Web server host via HTTP POST method. Perl scripts on the server process submitted data and make it available for presentation. Data is collected and presented per host and per calendar month. Color is used for presentation of network bandwidth values (in addition to numerical values). You can see an example of IFSTAT presentation page generated on the web server here.

Client configuration

Client means here a computer system that runs IFSTAT to collect network usage data. We need scripts to execute IFSTAT and to send IFSTAT output to a web server via HTTP POST method.

First, of course, you need to install IFSTAT executable. You can download source, as well as Debian package, and Windows binary from Gaël’s web site http://gael.roualland.free.fr/ifstat/ . You may also find IFSTAT binary available in your Linux distro’s repositories.

Next, you need to install scripts to execute IFSTAT and to send generated data to the web server. Below, you will find such scripts for Linux and Windows platforms. (I do not have anything for Mac OS X, as all my Macs are laptops. However, Linux scripts should probably work under Mac OS X.)

Now, you need to create a setup to execute IFSTAT for some period of time, to send its data to the web server, and to repeat the whole process again and again. So, we are talking about cron here (Linux) or Scheduled Tasks (Windows). (Presumably, you could also use the ‘at’ command, if you prefer.)

In my configuration, I launch 4 hour runs of IFSTAT during which data is collected in 10 minute intervals. After such 4 hour run, IFSTAT output is transferred to the web server for collection and presentation. Next, the process is repeated.

I schedule to run client script at determined hours, such as 4, 8, 12, 16, 20, and 0. However, this can be done in many other ways. One could start the script at each boot, and then restart after 4 hours. It is really up to your needs and imagination. I like my approach, although it means that data is not immediately collected after the boot.

Client scripts that you will find below are configured for 4 hour IFSTAT runs with 10 minute collection periods. Modify it, if this does not suit your needs. Man page for IFSTAT explains flags used by the program.

Windows client scripts

There are two scripts on Windows:

  • Batch script to execute IFSTAT binary and to launch VBScript script after the end of data collection
  • VBScript to submit IFSTAT output to a web server using HTTP POST method

Batch script – ifstat.bat – expects to find IFSTAT binary in your PATH. It also expect to find VBScript in your PATH. I place both of my scripts and the binary in the same folder, this is guaranteed to work. You will also need a subfolder named “data”, if you want to retain local copies of IFSTAT output files.

You need to modify one line in the VBScript (this script is named httppost.vbs) – the line that contains the URL to the CGI script on your web server that processes uploaded data, this is the line that reads:

   objHTTP.Open "POST", "http://seqweb/IT/ifstat_upload/accept.cgi", False

When everything is installed and configured, set up execution of the batch script ifstat.bat via Scheduled Tasks.

Linux client scripts

There are also two scripts needed on Linux clients:

  • bash script to execute IFSTAT
  • Perl script to upload IFSTAT output to your web server via HTTP POST

Bash script – ifstat.sh – is the one that needs to be launched from cron (or at startup if you prefer). The script expects to find the upload script – upload.pl – in the same directory. It looks for IFSTAT binary at /usr/local/bin/ifstat. If the binary is located somewhere else on your system, you need to modify the script accordingly.

You also need to modify in the uploading script upload.pl the URL pointing to the CGI script on your web server to accept uploaded data, i.e. the line that reads: 

   my $url = "http://seqweb/IT/ifstat_upload/accept.cgi";

Note that additional Perl modules may need to be installed on your Linux hosts (you can get them from CPAN), namely LWP::UserAgent, and File::Slurp, if they are not present in your Perl installations. These modules are required by upload.pl.

Web server configuration and scripts

You need a web server on your network to collect, process, and present IFSTAT data submitted by your client computers. I have used Apache under Linux, however, it should be possible to modify my Perl scripts for other web servers and/or other operating systems.

You need two directories in your web server area. One is used to collect the data, while the other one is used to present the data. In addition, you need a directory to store processed data – this directory can (and should) be outside of your web server area. However, remember to make it writable by the username under which your web server runs.

Access to the collection scripts cannot be secured by authentication, at least, in their present form. However, you could set up access limits based on IP addresses. The scripts expect only particular input, anything else is thrown into a pile of unprocessed data. Also, there is a limit of 10 MB on the submission size. As the result, any malicious activity is unlikely. However, I would recommend to run these scripts only on a web server situated on your LAN and without public (i.e. WAN) access to it.

Access to the directory with scripts that visualize processed data can be secured by any authentication means available on your web server. This is up to you.

The setup required on the web server is best illustrated by description of my configuration:

  1. /home/www/ifstat is the directory that all submitted data is stored (after processing). This directory is writable by user apache under which username my httpd daemon runs.

    /home/www/ifstat is also where I placed Perl module Ifstat.pm which contains settings for my CGI scripts. This is where all the settings go – modify them according to your configuration. At the minimum, you need to change 

    $ifstatdir = '/home/www/ifstat' ;

    of course, unless you keep /home/www/ifstat location . Note, that there is no sense to place Ifstat.pm under your Perl distribution directory structure. It is only usable for this particular application, and has absolutely no use outside. (Plus it is not exactly written with Perl development standards in mind.)

    • Download Perl Module Ifstat.pm as a tar archive

  2. /home/www/html/IT/ifstat_upload is where CGI script to process uploaded data resides – this script is called accept.cgi (/home/www/html is the document root of my Apache server). Client submit scripts need to refer to the URL of accept.cgi to deliver their payload – this is http://seqweb/IT/ifstat_upload/accept.cgi on my server. There is one more script in this directory – process.pl. This script is called by accept.cgi. The only thing that you should need to modify in any of these scripts is the Perl include line in process.pl script: 
      BEGIN { push(@INC, '/home/www/ifstat') } ;

    where you need to replace /home/www/ifstat with your location of the Ifstat.pm file.

    Note that you may alos need to install a few additional Perl modules – CGI, String::Random, File::Copy, File::Path – if they are not available in your Perl installation.

  3. /home/www/html/IT/IFSTAT is where CGI scripts to present processed data reside, and this is the URL to use to view IFSTAT results – http://seqweb/IT/IFSTAT. There are two CGI scripts there: index.cgi, and display.cgi, and supporting them stnd.css. The only thing to modify is the include path line in both CGI scripts: 
      BEGIN { push(@INC, '/home/www/ifstat') } ;

    if Ifstat.pm is not located in the above /home/www/ifstat location.

    Also, note that “index.cgi”‘ has been added to my DirectoryIndex directive in Apache configuration. Otherwise, one would need call the script index.cgi explicitly with the URL http://seqweb/IT/IFSTAT/index.cgi.

    Final thoughts

    My setup to collect and present IFSTAT results is not ideal – I am fully aware of it. Also, I am not a professional coder, and my scripts are certainly far from perfect. However, the described here setup fills my needs and provide me with a quick look at what is happening on the hosts on my network, especially after business hours. If you are interested in using it, it should be quite easy to implement assuming that you have some very basic knowledge of Perl, and web server configuration.

Comments are closed.