Logmonster ChangeLog 5.36 - Sep 2014 version bump, for consistent version numbers in all packages 3.12 - Apr 2013 testing changes, to resolve test errors on Windows 3.11 - Apr 2013 testing & packing changes 3.09 - Apr 2013 packaging changes 3.08 - Dec 2012 added missing URL dependency updated bin/install_deps.pl 3.07 - Oct 2012 removed Perl.pm updated Utility.pm to v 5.35 stripped out apache httpd.conf parsing. Instead, detect vhosts from the log files (works with any http server) added confdir (path to processor config files) statsdir is now a full path to a directory the 'drop logs into vhost docroot" feature is gone, pending a clever way to reliably deduce the vhost document root for various web servers. removed vhost config option added default_vhost option moved _progress subs into Logmonster.pm removed get_domains_list, check_stats_dir, get_domains_list_from_directories, get_vhosts_from_file, install_default_awstats_conf, turn_domains_into_sort_key abstracted new subs: get_log_files, spam_check, open_vhost_handle, report_matches, report_spam_hits, report_bad_hits, added lighttpd config entries to instructions 3.05 - Sep 2008 removed inc/* from distribution updated Makefile.PL, adding LICENSE and missing dependencies moved logmonster.pl into bin dir replaced bin/install_freebsd_deps.sh with install_deps.pl added 'make deps' target to Makefile.PL 3.04 - Nov 2007 included "inc"' in modules Utility.pm - only attempt to use sudo as a last resort Utility.pm & Utility.t- synced with Mail::Toaster moved Regexp::Log to inc dir 3.03 - Feb 17, 2007 Utility.pm - only attempt to use sudo as a last resort bin/install_freebsd_deps.sh - install Regexp::Log port 3.02 - Feb 16, 2007 added support for detecting dynamic mass virtual hosting evaluate log for referrer spam and discard the referral spam. optional referral spam reporting the hits-per-vhost was only counting the number of hits for the master domain. Domain alias hits weren't showing up in the hits-per-vhost report. added bin/install_freeebsd_deps.sh installer default log location is now /var/log/http (was /var/log/apache) added Regexp::Log::Logmonster - uses a compiled regexp to parse log entries (very useful for evaluate for referrer spam). 3.01 - Feb 2, 2007 Added Params::Validate dependency to Makefile.PL Fixed bug where time offset was being ignored. Checks more locations for awstats.pl (needs to become a config file settings). Removed an unnecessary dependency on Mail::Toaster in Utility.pm. 3.00rc2 - Oct 14, 2006 * updated usage examples * added FAQ * updated META.yml to new format * email report formatting added OUTPUT_AUTOFLUSH b/c printing to *STDERR flushes immediately and mixed print statements did not. Email report is better formatted now. * added inc/Module/Install * added Build.PL * added examples/daily_email_report.txt * Logmonster.pm debug was not being set properly, report formatting tweaks * Utility.pm added _progress sub 3.00rc1 - Oct 02, 2006 (no public release) massive code overhaul This new version is mostly about code quality and maintainability (not new features). The large chunks of code have been modularized into smaller subroutines and tests have been written to test the functionality of each sub. There are now 102 different tests (was 23) in the test suite. Added t/Test-coverage.pl, t/Logmonster.t, t/pod.t, t/pod-coverage, t/00.load Nearly all the "working" code has been moved into lib/Apache/Logmonster.pm. Logmonster.pl is now a "shell" consisting of a little bit of code and a lot of documentation. All the functions are now Object Oriented. Time will tell if that is a A Good Thing[TM] but it makes reading the code and understanding where all the calls are going much, much easier. added doc/* The documentation has been significantly updated, addressing many of the common questions and comments I have received. The reporting has been overhauled. You still get the same information but by default, if everything is okay it runs entirely silently. A single -v will output status messages that make for a nice birds eye view of your web log traffic. You can add additional -v options for even more verbose reporting. Interface change: instead of -m for month, -d for day, and -h for hour, you use a -i [hour|day|month] option. The old -mdh options are officially deprecated but will continue to work for the indefinte future. synced lib/Apache/Logmonster/Perl & Utility with Mail::Toaster 5 versions. 2.77 - no public release * create daily userlogs new feature submitted by Gernot Hueber. * autogenerate vhost stat dirs new feature funded by Lewis Bergman: statsdir can be automatically created if it does not exist. Set statsdir_policy = create in logmonster.conf to enable this feature. * incremental awstats config files. New feature by Lewis: Instead of having a full awstats.conf file for each vhost, each vhost has its own file that contains only specific info for that vhost and includes the generic /etc/awstats/awstats.conf file with the "global" settings. * added loguser and loggroup to .conf 2.76 - Jun 04, 2005 statsdir can now be an absolute path statsdir can now be an absolute path (ex: /var/www/html). If so, processor output will be stored in statsdir/vhost. Otherwise, it works as it used to and output goes to vhost/statsdir. This is useful if you have a seperate machine (not the web server) that does the processing and that system does not have access to the vhost docroot. 2.75 - Feb 05, 2005 * you can now select which stats processor is used for each virtual host. Create a .processor file in the stats dir and place the name of the processor on the first line. * test to make sure the log file exists before trying to compress it (supresses spurious warnings) * If vhosts are all in a directory, skip any files that end with ~ (vim) or .bak (user). * the code that hashed the domain list collected the settings from httpd.conf and stuffed them into a hash with the name of the vhost. This worked great, as long as ServerName was the first declaration in your httpd.conf. I discovered this fails otherwise. Now it stuffs the settings into a hash and then, after all the data is collected, moves the data into a new hash keyed off the vhost servername. The function now works regardless of order in the vhost container. * If a perl module was missing, the script would fail after attempting to load Mail::Toaster::Perl (which may not exist) Added Apache::Logmonster::Perl to distro 2.74 - Nov 26, 2004 package is now named Apache::Logmonster to fit nicely into a CPAN category bundled up for CPAN & freshmeat release Makefile.PL updated package NAME removed MATT::* dependency, added Compress::Zlib dep logmonster.pl updated package name added more example settings to logmonster.conf cleaned up pod docs for prettier web page formatting added TODO file remove MATT::Bundle reference from FAQ 2.73 - Nov 12, 2004 updated lib/Logmonster/Utility to latest fixed get_the_date bug in Utility added many more tests for Utility fixed a bug in my fileparse call (File::Basename) 2.72 - Nov 11, 2004 Removed MATT::Utility dependency added lib/logmonster/Utility (logmonster::Utility) removed Exporter updates for use with logmonster::Utility replaced StripLastDirFromPath with File::Basename 2.71 - Oct 22, 2004 Raymond Dijkxhoorn suggested not sorting the files if there is only one host. Shucks, that's a reasonable enough thing to do, for those of you with only one web server. ;-) Logmonster will now dutifully skip sorting logs if only one hostname is configured in logmonster.conf 2.70 - 0ct 22, 2004 allow for ServerName to have a :80 style suffix added verbose (-v) flag. fixed up reporting so quiet mode is really quiet unless there are errors normal output is prettier debugging output is much prettier apache config file parsing is now much more versatile if you have a folder full of files for vhosts, you can have multiple vhosts within a file now misc internal changes for efficiency prototyped all subroutines added additional comments here and there added test for FileHandle 2.6.6 - Jun 16, 2004 Added inline documentation to a few of the subs Modified SortVhostLogs so that it uses much less memory by writing to the log file as we sort instead of building an array and then writing the array contents to the file in GetDomainList, I forgot to add domains without aliases to %domains GetServerName: made the regexp search more reliable (no known problems but the potential existed). 2.6.5 - Jun 10, 2004 Switched date parsing regexp from / / to /\s+/ in SortVhostLogs per Earl Ruby (eruby@switchmanagement.com) for compatibility with Apache::Registry 2.6.4 - Apr 14, 2004 Added regexp notes Added additional debugging Check for files locally before trying to compress them 2.6.3 - Mar 28, 2004 Updated regexp to support numeric vhosts Updated logmonster.pl pod documentation Updated Makefile.PL, added README, FAQ files Added BSD copyright 2.6.2 - Mar 11, 2004 Corrected a typo in the logmonster.pl config file. 2.6.1 - Feb 24, 2004 Fixed problem where paths with caps weren't detected (search string was lower cased) 2.6 - Feb 21, 2004 Updated documentation and web site. More informative. 2.5.2 - Sep 24, 2003 Added a strip leading spaces function to GetVhostsFromFile Adjusted so FindTheBin will find awstats in its default location (/usr/local/www/cgi-bin/awstats.pl) 2.5 - Sep 15, 2003 Fixed a problem with HitsPerVHost not getting written Fixed a mis-feature where running logmonster -r was clobbering the active log processing dir. Oops. Fixed a couple problems related to interaction between script and MATT::Utility Made quite a few failures more graceful. This later proved to be only beneficial from a theoretical standpoint as it didn't solve the problem I was tracking down. 2.4 - Aug 14, 2003 Updated FetchLogFiles so that you don't need to ssh to localhost Updated URLs from matt.simerson.net to tnpi.biz Moved configuration from script to logmonster.conf Added Changes to the CVS depot Added support for AWStats log processor Added lots of documentation to logmonster.conf 2.3 - Jul 10, 03 Added pod documentation. Cleaned up SysCmd calls and now SysCmd calls use MATT::Utility 2.2 - Mar 24, 03 Moved many subs out of script into modules 2.1.1 - Mar 18, 03 Report hit counts (-r designed to be used with SNMP and RRDutil) 2.1 - Mar 17, 03 Moved code out of FetchLogFiles to GetTheLogDir for reuse Writes counters to $logdir/HitsPerVhost.txt Writes activity log to $logdir/Logmonster.txt Renamed $logdir to $logbase 2.0.3 - Mar 07, 03 Moved FAQ and Changelog to web site. Created web page for logmonster. Added support for http-analyze. It should work but I haven't used http-analzye in about 5 years so I might be forgetting something. 2.0.2 - Mar 06, 03 Added the httpd.conf parsing stuff. Now works with a vhost directory or parses out of your httpd.conf - cool :) 2.0.1 - Mar 06, 03 Moved vhost log pre-processer checks out of FeedTheMachine into CheckStatsDir. Run it before SortVhostLogs so we skip sorting any logs that we aren't going to store. Count up invalid lines in log files and report them instead of dying when we encounter them. Print out prettier logs. 2.0 - Mar 04, 03 An entire re-write Logic is much cleaner now and way more efficient Pulls most settings out of apache config files Reporting is much better :) Added -b (process archived logs) feature Added -h & -m (hourly & monthly processing) Added -n (dry run) so you can preview what it'll do Now clean enough that I'll make it publicly available 1.4 - Feb 22, 03 Major code cleanups Reworked fetch_the_files Added system_command Added FindTheBin 1.3 - Nov 25, 02 Added use strict (forced code cleanup) Added getopt::std instead of custom parser Moved $quiet to $opt_q, $debug to $opt_v 1.2.1 - Dec 14, 01 Made it work with matt.simerson.net 1.2 - Jun 28, 01 Cleaned up the code, added debug flag, expanded logic so v1.2 can replicate 1.0 & 1.1 behaviour with options 1.1 - Feb 14, 01 Added support for multiple domains 1.0 - Oct 29, 00 - Origin