Network Interface Statistics Collection Agent

Changelog

v2.5

• Added 95th percentile processing to the text-only reports.
• Added "time zooming" to the text-only reports, too. It's cool.
• Added ability to "un-orphan" a host. If you stop monitoring a host (by 'deleting' it in the Admin section) and leave its collected stats in the database, it will be listed in a dropdown menu on the main SNMP Host Admin page. Selecting it will start the process of re-adding it just like it was a new host. Someday I'll add a toggle to just turn a host's monitoring on or off, but it's hairy, and until then, this will somewhat make up for it.
• Fixed a "bug" in the move_niscaconf_to script (I forgot to change the version number in the update-to-2.4 script's filename in move_niscaconf_to, which probably broke the update script for some people... sorry.)
• Replaced all array_push() usage with assignments to [] instead. If you don't understand, it's okay. :)
• Renamed all lib/* PHP includes to *.phps instead of *.php. It's just a little more secure that way, I guess; no other reason, really.
• Removed the "-q" from #!/usr/local/bin/php lines. PHP no longer uses this CLI switch.

v2.4

• A few minor bugfixes.
• Added 95th-percentile processing. Hope I did it right; it finds the biggest value in the report range and multiplies by 0.95 and calls that the max; any data point over that is reset to be whatever value was in the slot just before it, so you don't get odd negative spikes on graphs. Don't think I'll add that to the text reports too; not sure if it'd be all that useful.
• Added the neato CSS color preview thingy to the report admin page (adm_rpt2.phtml).
• Added Bits/Bytes and 95% toggles to stored reports.
• Since there are database changes, be sure to run update-to-2.4!
• Added nisca.conf setting for whether you have ucd-snmp or net-snmp installed.

v2.3.3

• Figured out what was wrong with the collector and fixed it. It should now do snmpv1/v2 correctly.
• Added the fontface CSS to the pages without it already.

v2.3.2

• A few minor cosmetic changes were made.
• A few bugs, major and minor, were corrected.
• The snmp_collect script I rewrote for v2.3.1 didn't work; I was using textual OIDs instead of numeric ones, so a simple fix to functions.php was all that was needed.
• Took the ".php" extension off all Nisca utilities; this way, there won't be any way to run one of them from a browser (which would be bad).
• Added the "Add All" button to the Javascript interface selection form on the main index page.
• Added the ability to change bit/byte display on a per-report basis; the setting in the global admin section is now just a default. (This is not saved in stored reports; that would have required database changes, and I don't want to do that yet.)
• Removed all requirement for "snmpget". Walk & bulkwalk are all that are needed now.
• The fonts used in Nisca have always been hardcoded into it. Not the ones on graphs; the ones on all the HTML pages. I've changed that; you need only define the set of fonts you like in one place (nisca.conf) now, and that's what it'll use on every single page. Thanks, Pier.

v2.3.1

• v2.3 turned out to be a hopeless mass of annoyance, so I've (hopefully) fixed all the bugs in it and called it 2.3.1. My apologies... just wasn't enough testing; it should have been called a Beta. 2.3.1 has been more thoroughly checked, and has even more enhancements than v2.3 had (all the enhancements from 2.3 are in 2.3.1). Fixed things: nisca_check, nisca_kill, snmp_collect, functions.php, report.phtml, the config files, and probably others I've since forgotten about.

• Replaced the "deprecated" ImageDashedLine function in report.phtml with ImageSetStyle instead so people using libgd2.x would actually see Y-axis gridlines.

• Figured out how to keep the Javascript interface selection box from clearing itself when you hit the "Back" button after viewing a report. Should be easier to use now.

v2.3

• First and foremost, got rid of the use of built-in SNMP functions in PHP. You no longer need the SNMP extension in PHP; all you need is the "snmpwalk", etc, binaries that come with ucd-snmp. Not sure if it's really faster or not, but at least it works now no matter what ucd-snmp version you have. Whew.

• Changed report.phtml to fix the hideous Giant Spiked Graph problem. (This fix has been back-ported to 2.2 as well.)

• ...and then rewrote it all again. With the Old Way, it first pulled all the stats out of the database as running totals, then converted them into counts-per-interval, then averaged them, in three separate operations. This has now all been consolidated into one operation. It hasn't resulted in much of a speed increase, but there is a definite improvement. At this point, there isn't much else I can do to speed it up. Sigh. As a nice side effect of all that, I fixed a few potential problems remaining with the Giant Spiked Graph Syndrome. After much consideration, I've back-ported it to 2.2 too.

• Something else that I've back-ported to 2.2 in report.phtml: there was a bug which would cause some report table rows to show "0" when they should be showing a large number. Silly me, I was using the whole byte counter instead of an average bytes/sec rate to compare with the interface's maximum speed. Thanks to Ambrus Csaba for pointing it out.

• Since there's apparently a bug in either PHP or net-snmp which causes UDP sockets used during SNMP collection to stay open, thus eating up file descriptors and eventually causing the script to come to a grinding halt, I've added the "nisca_kill.php" script. This, in conjunction with the process ID files the collectors now write, allows you to set a frequency (in crontab) with which to just kill off the collector and let it start over. Then, the "nisca_check" script will restart it. A quick-and-dirty solution which will be remedied in the next major overhaul, if I can ever get started on it...

• I've combined the snmp_collect and test_collect scripts. If you run snmp_collect.php with anything after it, like "./snmp_collect test", it will act like it was actually the test_collect.php script... that is, not write to the database and just show you what it encounters during collection cycles, including how long they took.

• Added support for putting a visible watermark on graphs from any JPG, PNG, or BMP image. Sorry, no GIFs; thanks again, Unisys.

• Fixed a bug in adm_snmp3.phtml where if you deleted a previously-entered manual ifSpeed entry on an interface, it caused a SQL error. Thanks, Pier.

• Added the two rangesX.phtml scripts to make it easier to figure out what time ranges you can get data from on a particular interface. An idea whose time had come.

• Added the ability to click on a graph image to zoom in to that time period. This may be the nicest new feature of all...

v2.2

• Added one of the neatest features so far, the "Data Re-Averaging Utility." If your database ever gets too large to handle, you can use this utility to trim out some of the statistics in your database (thus effectively re-averaging the data over a longer period). You can also use it if your statistics are too "fuzzy" (collecting at very frequent intervals on a fast interface will make your graphs fuzzy; the more frequent, the fuzzier it gets). It's like two features in one. Oh, and also, when you strip out database entries to re-average it or make it smaller, you have the option of telling it to create *compressed* text (SQL command) archive files so you can change your mind later.

• Modified the "Stats Deletion Utility" to create archive files like the Re-Averaging Utility does now.

• Added "multiple" report functionality. This means you can tell it to add up the stats on all the interfaces you have selected and put them all on one graph. This differs from MRTG's method, which requires that you have a special rule set up to collect the stats additively. Nisca allows you to just suddenly decide that you want to see the sum of the transfer rates on any number of interfaces on any number of machines over any time period; all that's required is that they have normal statistics already collected for them. This is only supported for the "In/OutOctets" (byte count) statistics, not packets, drops, or errors, though I could add support for multi-IF graphs on those kinds of data if people wanted it.

• Added "adm_showconf.phtml" script to the administration section. This script just shows you the contents of the "nisca.conf" and "db.conf" files it's using; I thought it might come in handy to some people. :)

• Added ability to use one of these methods to uniquely identify interfaces on a host: "ifDescr", "ifIndex", "ifName", "ifPhysAddress", "Cisco iflocDescr", or "Catalyst Portname". This was to help those people stuck with a Catalyst or other machine that has the same "ifDescr" entry on every interface.

• Added "Committed Information Rate" (CIR) on interfaces so you can see where your graphs go above it. Thanks to Mark Motley for the idea.

• Added "Host Aliases" so you aren't stuck with an IP or hostname to display a host as (similar to the user-definable Interface Aliases). Mark Motley's idea, too.

• Added the ability to change or add an "ifSpeed" entry for any SNMP or Localhost interface (adm_snmp2.phtml and adm_local1.phtml).

• Added the display of how much disk space your database consumes to the report index page. (This requires a bit of setting up; see the end of the "nisca.conf" file for details.)

• I've had a few requests from people who for whatever reason want to run their collectors from crontab instead of as a daemon. Now, all you have to do is change one variable at the top of "collect.php" or "snmp_collect.php" to toggle daemonic behavior.

• Added an SNMP timeout variable in snmp_collect.php. This can be adjusted by the user easily (it defaults to 15 seconds... I hope).

• Changed it so trying to save a report with an existing report's name isn't allowed anymore.

• Fixed a bug which caused interfaces without an ifSpeed entry to generate wildly inaccurate text-only reports. (This fix has been back-ported to v2.1 as well.) Thanks to Timothy Barbeisch for drawing it to my attention.

• Added the usual "update-to-2.2.php" script. It changes the version in your database to "2.2", adds two "multiple" columns to the "reports" table, adds a column to the "localhost" table for fake ifSpeed entries, and two columns to the "interfaces" table to handle host aliases and IF naming method.

• Added "PHP_HINTS" to the documentation for the purpose of explaining in excruciating detail how to install the proper Apache, MySQL, and PHP support for NISCA.

v2.1

• Added the "Stats Deletion Utility" that lets you get rid of unneeded statistics that are taking up lots of disk space.

• Changed the php_uname function in "nisca_check.php" to posix_uname (didn't notice that php_uname only returns information about the machine PHP was *built* on, not the machine it's *running* on).

• Added context-sensitive help, modularized so it's easy to add a help button to any page.

• Changed "adm_snmp3" so it won't delete that host's current config before checking to see if it'll do 64-bit counters; it now waits to the very end of the script before it deletes your current config and replaces it with the new one. Duh...

• Moved the "hostname", "community", and "if_name" fields out of the stats table and replaced them with a single mediumint identifier in the stats table tied to a record in the stats_ifs table. This has halved the disk space usage and doubled the report generation speed; not only that, but the index page now loads instantly every time. This limits you to monitoring 16.7 million interfaces; sorry about that :) It was either limiting it to that, or 65,000 interfaces; I decided to err on the side of "huge".

• Added ability to choose whether you want JPG or PNG format graphs created; added a config option and stored-report field to the database for it.

• Made "snmpv2" a per-interface option instead of per-host. I noticed a lot of machines which have some interfaces that do 64-bit counters while others don't; this is the only way to allow collecting some from v1 and some from v2 on the same host.

• The SNMP interface administration page (the one with all the checkboxes to turn on, adm_snmp2.phtml) now displays interfaces which support 64-bit counters in a kinda reddish watermelon-colored font so you can tell at a glance which interfaces on that host actually both support AND are using 64-bit counters. On an interface that doesn't support them, there will not be a checkbox available for enabling 64-bit counters on that interface.

• Also on adm_snmp2, interfaces that support SNMPv2 (whether or not they support 64-bit counters) will have their "ifAlias" SNMP entries read to populate the Alias entry fields. If you already manually set an alias for an interface, the alias you entered will be used regardless of its ifAlias entry.

• Added the "update-to-2.1.php" script for the usual reason. This script modifies an existing database thus:
  • Sets version to "2.1" in the "config" table

  • Creates "stats_ifs" table

  • Adds "if_id" field to "stats" table

  • For every unique hostname/community/if_name in "stats", adds an entry to "stats_ifs" and uses last_insert_id to populate the "if_id" column in the "stats" table

  • Deletes the "hostname", "community", and "if_name" fields and all keys and indices from the "stats" table

  • Creates new primary key for stats table (no indices needed)

  • If stats table was empty, simply creates new stats table

  • Adds the PNG/JPG toggle to "config" and "reports" tables.

• Altered mrtg_import script so that it's possible to import an MRTG log that was collected via SNMP on your localhost interface(s) into existing stats for localhost that were collected by NISCA via the proc/net/dev file by specifying "localhost" for the "community" option to mrtg_import.php.

v2.0

• Changed hostname/interface separator character from ":" to "!" because colons can appear in IF names. (Thanks to Eddy Lai.)

• Added "update_to_2.0.php" script; this modifies an existing database thus:
  • Sets version to "2.0"

  • Converts "reports" table (changes separator to "!")

  • Changes "stats.if_name" to varchar(64)

  • Adds the "stats.community" column

  • Fills the "stats.community" column with guessed values

  • Changes primary key on "stats" to include "community"

  • Adds the tables "config", "interfaces", "localhost", "users", and "sessions".

  • Fills "config", "interfaces", and "localhost" tables using pre-existing values (from functions.bak)

• Changed "stats.sql" to "create_stats.php" to make it easier to do a fresh installation.

• Utterly rewrote the hostname/interface selection method on the index form. If you have Javascript, you get a dropdown list for the hosts, another dynamically-generated list of the interfaces on the selected host, and a multi-select list where the entries selected are stored. If not, you get the same old giant long list as before. This was done so people who monitor a LOT of interfaces won't have to wade through a 1,000-entry list. Also added config option to turn this on and off. (Thanks to Pierfrancesco Caci for the suggestion.)

• Added ability to change community differently for each interface, not just each host.

• Moved all configuration options to the database. It was getting annoying having to redo functions.php every time you upgrade. Now, upgrades are a snap.

• Added "adm*.phtml", "log*.phtml", and "security.php" scripts for config admin. Administration scripts are now web-based and in a directory all by themselves (admin/).

• Security on the admin pages is assured through the use of cookies, PHP session variables, and database entries connecting them with a username.

• Absorbed "snmp_setup.php" script into the configuration interface.

• Added "security.php" and "log*.phtml" scripts to manage authentication on the admin pages.

• Made the date format a config option (date & time, date only, and time only). Thanks to Pierfrancesco Caci for telling me I needed to do this. :)

• Fixed graph displays so they don't all say "bytes/sec".

• Changed SNMP collection to use the numeric ISO OIDs instead of the textual ones; for example, it now uses ".1.3.6.1.2.1.2.2.1.1" instead of the more elegant "interfaces.ifTable.ifEntry.ifDescr.1".

• SNMP collection now uses "snmpget()" instead of "snmp_walk()" for collection to speed things up a bit.

• Added SNMPv2 support, though it requires a minor patch to the PHP source code (the INSTALL file tells you how)..

• Added "total bytes transferred" to the text reports (thanks to Sean for the idea).

• Added interface alias support. If you enter an alias for an interface, that will be used; otherwise, if there's an alias defined in the SNMPv2 OID "ifAlias", it will use that. Aliases are implemented for both localhost and SNMP interfaces.

• Changed config file location setup; locations are put into the "nisca.conf" file and used for ever after from it. It should always go into "/etc/nisca/". Thanks to Fabio Di Nitto for this idea.

• Added "nisca_check.php" script to run from crontab and make sure your collector(s) stay running.

• Fabio moved the Changelog into its own file (CHANGELOG) and redid the documentation so it'd make more sense.

• As you can see, I've been very busy. :)

v1.3

• Made all colors used to make graphs config options and changeable on the index form.

• Major revision to SNMP interface name coding; it's no longer a regular expression, just an array of names for each host in $hostname. This means it'll work with IF names containing slashes. It was also breaking the mrtg_import.php script. My apologies to anyone who hates NISCA now because of this... :) (Thanks to Pierfrancesco Caci and Fabio Di Nitto for helping me through this.)

• Added the $dynamic["*all*"] variable to "functions.php" and "snmp_collect.php".

v1.2

• Changed name of "functions.inc" to "functions.php".

• Moved *all* config options to "functions.php".

• Optimized stats table structure (removed "id" column and changed primary key to remove all possibility of duplicate data).

• Added "update.sql" script.

• Now including HTML versions of README, INSTALL, and LICENSE with the distribution.

• Added support for dynamically-changing SNMP interface numbers. That is, if interfaces go up and down and the SNMP server keeps changing which number it assigns to them, you can tell it to re-read the list of interface names and numbers every time it gets the rest of the SNMP data. (thanks to Esteve Chust for this idea.)

• More SNMP bugfixes (thanks to Jimmy Kaplowitz).

• Made SNMP community a config option (Jimmy's idea).

• Changed recommended way to run collector scripts by adding "nohup" to it to prevent death (Jimmy).

• Added "Saved Reports" functionality (Jimmy again).

• Created "reports" MySQL table.

• Added "rpt_*.phtml" scripts for stored report administration.

v1.1

• Upgraded report generator to allow for viewing reports on more than one host/if at once.

• Added option to show only text reports, graphs, or both.

• Some minor bugfixes in the SNMP scripts.

v1.01 beta

• Added the "mrtg_import.php" script; updated documentation to match.

v1.0 beta

• First release.