Revision history for CPAN-Testers-Data-Generator

1.21    2015-08-26
        - uploads table now has a unique id as primary key.
        - cpanstats, release_data and release_summary tables now include an 
          uploadid field and index.

1.20    2015-04-19
        - fixed tail log searching.
        - ensure we are parsing a CPAN::Testers::Report object.
        - added examples/ to check GUID entries. 

1.19    2015-04-17
        - fixed date/time timezone bug (thanks Slaven).

1.18    2015-04-17
        - added insertion of PASS report data into passreports table.
        - added script to examples, for emergencies :)
        - maintain history of Bucket changes.

1.17    2014-09-11
        - fixed license fields in META.json to be lists.

1.16    2014-05-15
        - move to using DateTime to calculate dates and times.
        - fixed distribution name in META.
        - fixed _get_createdate guid pattern matching.

1.15    2014-03-23
        - finally upgraded to Metabase-1.003.
        - extended test suite.

1.14    2014-03-22
        - arch os no longer needs to decode JSON.

1.13    2014-03-20
        - enable mysql_auto_reconnect.
        - updates from live fixes.

1.12    2014-03-19
        - specify the end timestamp when getting the guids in a date range.
        - store complete report (using Sereal) in db as metabase.fact. Also 
          retrieve and de-reference facts hash from 'fact' field, as 'report' 
          field will be dropped once the 'fact' field is populated for all 
          reports (including historical ones).
        - rollback end timestamp (for now), until storage bug fixed.
        - expand test suite.
        - store perl6 reports with type=6. Amended to ignore these for the
          moment, as this needs to be determined by different metadata.
        - remove SQLite references.

1.11    2013-09-08
        - added Data::FlexSerializer functionality to compress reports.
        - added missing prerequisite, Metabase::Resource.

1.10    2013-05-12
        - fixed skip test count.
        - fixed cleanup of directory removal.

1.09    2013-04-14
        - date clarification for lookups.
        - additional logging.
        - added LICENSE file.
        - added cleanup process in test suite.
        - added Test::Database to test suite.

1.08    2012-12-15
        - fixes to make sure we move on if no reports processed.

1.07    2012-12-06
        - updated README
        - rewritten generate method.

1.06    2012-11-18
        - removed JSON & RSS Recent code.
        - fixed dates.

1.05    2012-11-13
        - parse message fix.
        - disabled SQLite updates.
        - new script:
            bin/cpanstats-sqlite                (v0.01)
        - script updates:
            bin/cpanstats                       (v1.05)
            bin/cpanstats-select                (v0.12)
            bin/cpanstats-update                (v0.10)
            bin/cpanstats-delete                (v0.15)

1.04    2012-10-23
        - script updates:
            bin/cpanstats                       (v1.04)
        - CHANGES => Changes.
        - added tail feature.

1.03    2012-10-21
        - spelling fixes. (thanks to Florian Schlichting & Gregor Herrmann).
        - AWS values are now configure settings, that can be overridden on the
          command line.
        - compensate for Amazon SimpleDB's brokeness with dates.
        - added parse function for specific GUIDs.
        - change file dates changed to meet W3CDTF standards.
        - META tests no longer require specific versions.
        - reworked Makefile.PL for clarity.
        - License updated to Artistic License v2.0.
        - added minimum perl version (5.006).
        - added missing prerequisite.

1.02    2012-04-01
        - reworked regenerate functionality and script.
        - reworked reparse functionality and script to use Metabase API.
        - added handling of localonly and check options.
        - updated script:
            bin/cpanstats                       (v1.02)

1.01    2012-01-22
        - generator now builds a JSON and RSS file of the last n reports.
        - added id to page_request insert (to speed up page builder).
        - reworked regenerator to allow a file of guid/date ranges.
        - update regenerate option and documentation:
            bin/cpanstats                       (v1.01)
        - added mappings for old badly formatted distributions.

1.00    2010-11-14
        - redesigned to use Metabase. All NNTP references removed.
        - removed ignore and nostore options.
        - added nonstop option.
        - added error report mail alert, to highlight problems quickly
        - improved validation of metabase results, due to the search sometimes
          returning older reports!
        - improved retrieval of tester fact
        - scripts removed:
        - added guid command line options and/or output:
            bin/cpanstats-select                (v0.11)
            bin/cpanstats-update                (v0.09)
            bin/cpanstats-delete                (v0.14)
        - major update of all SQL tables and databases
            examples/cpanstats-createdb         (v0.02)
        - added META.json
        - added regenerate() and retrieve_report() methods.
        - added regenerate option and documentation:
            bin/cpanstats                       (v1.00)
        - various data amendments, including _check_arch_os(), as the metabase
          uses the raw data.
        - new perl_version table incorporated to speed up perl version lookups.

0.41    2010-03-18
        - fixes to change the 'id' (was NNTP ID) to an auto incremental field.
        - reworked logic to better fit latest changes.
        - added repository to META.yml.
        - documentation updates.

0.40    2010-02-02
        - fixes to accommodate GUID changes.
        - added support for 'type' field.

0.39    2010-01-30
        - major update to include guid for Metabase/CT2.0.

0.38    2009-10-19
        - added functionality to update the osname table.
        - added cpanstats DB schema to examples directory.

0.37    2009-06-07
        - added page request functionality.
        - added release summary functionality.
        - due to the recent expansion, the 12generate.t test script now needs
          more complex logic. The intention is to use Test::Database for this,
          but unfortunately T::D isn't stable enough to use yet. As such this
          test script is now effectively disabled.

0.36    2009-02-16
        - clean up of pre-requisites.
        - CPAN::Testers::Data::Generator::Article abstracted into its own
          distribution CPAN-Testers-Common-Article.
        - fixed scripts due to above:
            bin/cpanstats         (v0.07)
            bin/cpanstats-reparse (v0.10)

0.35    2009-01-29
        - updated patterns when extracting perl version (
        - updated patterns when extracting osname (
        - updated cannot connect to NNTP server error message.

0.34    2009-01-12
        - updated Copyright info, minor fixes and added 'Getopt::ArgvFile', if
          not already used:
            bin/articles-verify   (v0.03)
            bin/articles-view     (v0.03)
            bin/cpanstats         (v0.06)
            bin/cpanstats-delete  (v0.13)
            bin/cpanstats-reparse (v0.09)
            bin/cpanstats-select  (v0.10)
            bin/cpanstats-update  (v0.08)
            bin/cpanstats-verify  (v0.24)
        - added to examples.
        - added missing log file command line option to bin/cpanstats (v0.06).
        - added missing prerequisites (thanks to CPANTS)

0.33    2008-12-29
        - updated help text to bin/cpanstats-verify  (v0.23)
        - fix to stop existing records being reinserted.

0.32    2008-12-19
        - added progress option to bin/cpanstats-verify (v0.22)
        - fixes to to attempt to correct missing osnames and osvers.
        - added articles scripts that have been used behind the scenes for
            bin/articles-verify   (v0.02)
            bin/articles-view     (v0.02)

0.31    2008-12-18
        - name change to fit below the CPAN::Testers namespace.
        - added epoch to date extraction from article.
        - added filename reference from CPAN-DistnameInfo.
        - added ability to provide SQL string in bin/cpanstats-select.
        - added better command line option parsing to bin/cpanstats-reparse.
        - convert to use CPAN::Testers::Common::DBUtils.
        - added MySQL cpanstats database to codebase.
        - new scripts added to examples to handle database creation:
            examples/cpanstats-createdb (v0.01)
            examples/cpanstats-convert  (v0.01)
        - update to bin scripts with latest code changes:
            bin/cpanstats         (v0.05)
            bin/cpanstats-delete  (v0.12)
            bin/cpanstats-reparse (v0.08)
            bin/cpanstats-select  (v0.09)
            bin/cpanstats-update  (v0.07)
            bin/cpanstats-verify  (v0.21)

0.30    2008-10-01
        - ensure upload pattern matching is for the end of the string.
        - add more option validation in bin/cpanstats-verify (v0.20).

0.29    2008-09-19
        - ignore any uploads that are not in a supported archive format used
          for testing.

0.28    2008-09-18
        - added SQLite pragma 'auto_vacuum' to database creation.
        - created an _init function to generate db/nntp objects in one place
        - add --localonly option to help screen in bin/cpanstats-verify (v0.19)
        - fixed test script to correctly do cmp_ok :)

0.27    2008-09-11
        - added 'grade' option to search in bin/cpanstats-select (v0.08)
        - fixed help text in bin/cpanstats-update (v0.06)
        - fixed test bug in 12generate.t (I can't count!)

0.26    2008-09-02
        - abstracted all database functionality out of into
, including database creation. All scripts updated too.
        - database links changed to reference CPAN Testers development site.
        - corrections to the database schema in POD
        - several POD updates across all files, including
            bin/cpanstats         (v0.04)
            bin/cpanstats-delete  (v0.11)
            bin/cpanstats-reparse (v0.07)
            bin/cpanstats-select  (v0.07)
            bin/cpanstats-update  (v0.05)
            bin/cpanstats-verify  (v0.18)
        - fixed dependency in cpanstats
        - consistently provided help|h and version|V options to all scripts
        - fix to support old form of PAUSE uploads
        - abstracted reparse code into from cpanstats-reparse
        - more tests

0.25    2008-08-28
        - added POD about bugs and see also sections.
        - fixed POD in scripts.
        - added author to upload metadata, to save lookups elsewhere.
        - renamed several supporting scripts:
            delstats    => bin/cpanstats-delete (v0.10)
            selectstats => bin/cpanstats-select (v0.06)
            upstats     => bin/cpanstats-update (v0.04)
            readstats   => bin/cpanstats-verify (v0.17)
        - added further functionality to cpanstats-select.
        - added bin/cpanstats-reparse (v0.06) to distribution
        - added nostore option to not retain old articles in the articles

0.24    2008-08-19
        - added bin/readstats (v0.16), bin/delstats (v0.09),
          bin/selectstats (v0.05) and bin/upstats (v0.03) from the non-CPAN
          cpanstats distribution.
        - updated all scripts to use an options hash rather than individual
        - added new module to handle all local DB interaction.
        - fixed test bug in t/12generate.t (thanks to Slaven Rezic).

0.23    2008-08-18
        - new maintainer: Barbie.
        - major overhaul to draw in CPAN Testers Statistics code.
        - added version to
        - added ability to decode QuotedPrintable and Base64 (
        - added functionality to parse the upload articles as well as the
          report articles (
        - added more reliable subject parsing.
        - added more reliable Perl -V parsing.
        - testers.db now renamed to cpanstats.db.
        - news.db now renamed to articles.db.
        - field names changed:
            status => state,
            distribution => dist,
            archname => platform.
        - new fields added: postdate,tester.
        - fixed prerequisites in modules, Makefile.PL and META.yml.
        - bin/cpan_www_testers_generate replaced with bin/
        - added more test samples
        - added more tests

0.22    2005-02-03T15:27:41
        - now includes perl + OS version in the testers.db
        - use SQLite 3.0
        - bin/cpan_www_testers_generate expanded to give more
          flexibility for execution
        - added Makefile.PL (as per TODO)
        - added tests
        - added more POD
        - (thanks to Barbie)

0.21    2004-03-07T18:31:35
        - the NNTP cache now uses SQLite instead of DB_File
        - rename Changes to CHANGES

0.20    2003-08-13T21:52:06
        - make sure to HTML escape everything
        - design update, now with CSS
        - Fix bug where reports are use A/AM/AMS/Crypt-TEA-1.22.tar.gz
          instead of Crypt-TEA-1.22.tar.gz
        - added a TODO
        - sort using version, not Sort::Version
        - renamed to CPAN::WWW::Testers::Generator
        - included HISTORY section by Elaine Ashton
        - moved examples/ to an installed script,

0.10    2003-07-31T23:09:25
        - initial release