-
-
12 Apr 2022 16:10:04 UTC
- Distribution: IO-Compress
- Source (raw)
- Browse (raw)
- Changes
- Homepage
- How to Contribute
- Repository
- Issues (0)
- Testers (476 / 5 / 0)
- Kwalitee
Bus factor: 5- 81.87% Coverage
- License: perl_5
- Activity
24 month- Tools
- Download (290.46KB)
- MetaCPAN Explorer
- Permissions
- Subscribe to distribution
- Permalinks
- This version
- Latest version
- Dependencies
- Compress::Raw::Bzip2
- Compress::Raw::Zlib
- Encode
- Scalar::Util
- and possibly others
- Reverse dependencies
- CPAN Testers List
- Dependency graph
- NAME
- DESCRIPTION
- GENERAL
- ZIP
- GZIP
- ZLIB
- Bzip2
- HTTP & NETWORK
- MISC
- SUPPORT
- SEE ALSO
- AUTHOR
- MODIFICATION HISTORY
- COPYRIGHT AND LICENSE
NAME
IO::Compress::FAQ -- Frequently Asked Questions about IO::Compress
DESCRIPTION
Common questions answered.
GENERAL
Compatibility with Unix compress/uncompress.
Although
Compress::Zlib
has a pair of functions calledcompress
anduncompress
, they are not related to the Unix programs of the same name. TheCompress::Zlib
module is not compatible with Unixcompress
.If you have the
uncompress
program available, you can use this to read compressed filesopen F, "uncompress -c $filename |"; while (<F>) { ...
Alternatively, if you have the
gunzip
program available, you can use this to read compressed filesopen F, "gunzip -c $filename |"; while (<F>) { ...
and this to write compress files, if you have the
compress
program availableopen F, "| compress -c $filename "; print F "data"; ... close F ;
Accessing .tar.Z files
The
Archive::Tar
module can optionally useCompress::Zlib
(via theIO::Zlib
module) to access tar files that have been compressed withgzip
. Unfortunately tar files compressed with the Unixcompress
utility cannot be read byCompress::Zlib
and so cannot be directly accessed byArchive::Tar
.If the
uncompress
orgunzip
programs are available, you can use one of these workarounds to read.tar.Z
files fromArchive::Tar
Firstly with
uncompress
use strict; use warnings; use Archive::Tar; open F, "uncompress -c $filename |"; my $tar = Archive::Tar->new(*F); ...
and this with
gunzip
use strict; use warnings; use Archive::Tar; open F, "gunzip -c $filename |"; my $tar = Archive::Tar->new(*F); ...
Similarly, if the
compress
program is available, you can use this to write a.tar.Z
fileuse strict; use warnings; use Archive::Tar; use IO::File; my $fh = IO::File->new( "| compress -c >$filename" ); my $tar = Archive::Tar->new(); ... $tar->write($fh); $fh->close ;
How do I recompress using a different compression?
This is easier that you might expect if you realise that all the
IO::Compress::*
objects are derived fromIO::File
and that all theIO::Uncompress::*
modules can read from anIO::File
filehandle.So, for example, say you have a file compressed with gzip that you want to recompress with bzip2. Here is all that is needed to carry out the recompression.
use IO::Uncompress::Gunzip ':all'; use IO::Compress::Bzip2 ':all'; my $gzipFile = "somefile.gz"; my $bzipFile = "somefile.bz2"; my $gunzip = IO::Uncompress::Gunzip->new( $gzipFile ) or die "Cannot gunzip $gzipFile: $GunzipError\n" ; bzip2 $gunzip => $bzipFile or die "Cannot bzip2 to $bzipFile: $Bzip2Error\n" ;
Note, there is a limitation of this technique. Some compression file formats store extra information along with the compressed data payload. For example, gzip can optionally store the original filename and Zip stores a lot of information about the original file. If the original compressed file contains any of this extra information, it will not be transferred to the new compressed file using the technique above.
ZIP
What Compression Types do IO::Compress::Zip & IO::Uncompress::Unzip support?
The following compression formats are supported by
IO::Compress::Zip
andIO::Uncompress::Unzip
Store (method 0)
No compression at all.
Deflate (method 8)
This is the default compression used when creating a zip file with
IO::Compress::Zip
.Bzip2 (method 12)
Only supported if the
IO-Compress-Bzip2
module is installed.Lzma (method 14)
Only supported if the
IO-Compress-Lzma
module is installed.
Can I Read/Write Zip files larger the 4 Gig?
Yes, both the
IO-Compress-Zip
andIO-Uncompress-Unzip
modules support the zip feature called Zip64. That allows them to read/write files/buffers larger than 4Gig.If you are creating a Zip file using the one-shot interface, and any of the input files is greater than 4Gig, a zip64 complaint zip file will be created.
zip "really-large-file" => "my.zip";
Similarly with the one-shot interface, if the input is a buffer larger than 4 Gig, a zip64 complaint zip file will be created.
zip \$really_large_buffer => "my.zip";
The one-shot interface allows you to force the creation of a zip64 zip file by including the
Zip64
option.zip $filehandle => "my.zip", Zip64 => 1;
If you want to create a zip64 zip file with the OO interface you must specify the
Zip64
option.my $zip = IO::Compress::Zip->new( "whatever", Zip64 => 1 );
When uncompressing with
IO-Uncompress-Unzip
, it will automatically detect if the zip file is zip64.If you intend to manipulate the Zip64 zip files created with
IO-Compress-Zip
using an external zip/unzip, make sure that it supports Zip64.In particular, if you are using Info-Zip you need to have zip version 3.x or better to update a Zip64 archive and unzip version 6.x to read a zip64 archive.
Can I write more that 64K entries is a Zip files?
Yes. Zip64 allows this. See previous question.
Zip Resources
The primary reference for zip files is the "appnote" document available at http://www.pkware.com/documents/casestudies/APPNOTE.TXT
An alternatively is the Info-Zip appnote. This is available from ftp://ftp.info-zip.org/pub/infozip/doc/
GZIP
Gzip Resources
The primary reference for gzip files is RFC 1952 https://datatracker.ietf.org/doc/html/rfc1952
The primary site for gzip is http://www.gzip.org.
Dealing with concatenated gzip files
If the gunzip program encounters a file containing multiple gzip files concatenated together it will automatically uncompress them all. The example below illustrates this behaviour
$ echo abc | gzip -c >x.gz $ echo def | gzip -c >>x.gz $ gunzip -c x.gz abc def
By default
IO::Uncompress::Gunzip
will not behave like the gunzip program. It will only uncompress the first gzip data stream in the file, as shown below$ perl -MIO::Uncompress::Gunzip=:all -e 'gunzip "x.gz" => \*STDOUT' abc
To force
IO::Uncompress::Gunzip
to uncompress all the gzip data streams, include theMultiStream
option, as shown below$ perl -MIO::Uncompress::Gunzip=:all -e 'gunzip "x.gz" => \*STDOUT, MultiStream => 1' abc def
Reading bgzip files with IO::Uncompress::Gunzip
A
bgzip
file consists of a series of valid gzip-compliant data streams concatenated together. To read a file created bybgzip
withIO::Uncompress::Gunzip
use theMultiStream
option as shown in the previous section.See the section titled "The BGZF compression format" in http://samtools.github.io/hts-specs/SAMv1.pdf for a definition of
bgzip
.ZLIB
Zlib Resources
The primary site for the zlib compression library is http://www.zlib.org.
Bzip2
Bzip2 Resources
The primary site for bzip2 is http://www.bzip.org.
Dealing with Concatenated bzip2 files
If the bunzip2 program encounters a file containing multiple bzip2 files concatenated together it will automatically uncompress them all. The example below illustrates this behaviour
$ echo abc | bzip2 -c >x.bz2 $ echo def | bzip2 -c >>x.bz2 $ bunzip2 -c x.bz2 abc def
By default
IO::Uncompress::Bunzip2
will not behave like the bunzip2 program. It will only uncompress the first bunzip2 data stream in the file, as shown below$ perl -MIO::Uncompress::Bunzip2=:all -e 'bunzip2 "x.bz2" => \*STDOUT' abc
To force
IO::Uncompress::Bunzip2
to uncompress all the bzip2 data streams, include theMultiStream
option, as shown below$ perl -MIO::Uncompress::Bunzip2=:all -e 'bunzip2 "x.bz2" => \*STDOUT, MultiStream => 1' abc def
Interoperating with Pbzip2
Pbzip2 (http://compression.ca/pbzip2/) is a parallel implementation of bzip2. The output from pbzip2 consists of a series of concatenated bzip2 data streams.
By default
IO::Uncompress::Bzip2
will only uncompress the first bzip2 data stream in a pbzip2 file. To uncompress the complete pbzip2 file you must include theMultiStream
option, like this.bunzip2 $input => \$output, MultiStream => 1 or die "bunzip2 failed: $Bunzip2Error\n";
HTTP & NETWORK
Apache::GZip Revisited
Below is a mod_perl Apache compression module, called
Apache::GZip
, taken from http://perl.apache.org/docs/tutorials/tips/mod_perl_tricks/mod_perl_tricks.html#On_the_Fly_Compressionpackage Apache::GZip; #File: Apache::GZip.pm use strict vars; use Apache::Constants ':common'; use Compress::Zlib; use IO::File; use constant GZIP_MAGIC => 0x1f8b; use constant OS_MAGIC => 0x03; sub handler { my $r = shift; my ($fh,$gz); my $file = $r->filename; return DECLINED unless $fh=IO::File->new($file); $r->header_out('Content-Encoding'=>'gzip'); $r->send_http_header; return OK if $r->header_only; tie *STDOUT,'Apache::GZip',$r; print($_) while <$fh>; untie *STDOUT; return OK; } sub TIEHANDLE { my($class,$r) = @_; # initialize a deflation stream my $d = deflateInit(-WindowBits=>-MAX_WBITS()) || return undef; # gzip header -- don't ask how I found out $r->print(pack("nccVcc",GZIP_MAGIC,Z_DEFLATED,0,time(),0,OS_MAGIC)); return bless { r => $r, crc => crc32(undef), d => $d, l => 0 },$class; } sub PRINT { my $self = shift; foreach (@_) { # deflate the data my $data = $self->{d}->deflate($_); $self->{r}->print($data); # keep track of its length and crc $self->{l} += length($_); $self->{crc} = crc32($_,$self->{crc}); } } sub DESTROY { my $self = shift; # flush the output buffers my $data = $self->{d}->flush; $self->{r}->print($data); # print the CRC and the total length (uncompressed) $self->{r}->print(pack("LL",@{$self}{qw/crc l/})); } 1;
Here's the Apache configuration entry you'll need to make use of it. Once set it will result in everything in the /compressed directory will be compressed automagically.
<Location /compressed> SetHandler perl-script PerlHandler Apache::GZip </Location>
Although at first sight there seems to be quite a lot going on in
Apache::GZip
, you could sum up what the code was doing as follows -- read the contents of the file in$r->filename
, compress it and write the compressed data to standard output. That's all.This code has to jump through a few hoops to achieve this because
The gzip support in
Compress::Zlib
version 1.x can only work with a real filesystem filehandle. The filehandles used by Apache modules are not associated with the filesystem.That means all the gzip support has to be done by hand - in this case by creating a tied filehandle to deal with creating the gzip header and trailer.
IO::Compress::Gzip
doesn't have that filehandle limitation (this was one of the reasons for writing it in the first place). So ifIO::Compress::Gzip
is used instead ofCompress::Zlib
the whole tied filehandle code can be removed. Here is the rewritten code.package Apache::GZip; use strict vars; use Apache::Constants ':common'; use IO::Compress::Gzip; use IO::File; sub handler { my $r = shift; my ($fh,$gz); my $file = $r->filename; return DECLINED unless $fh=IO::File->new($file); $r->header_out('Content-Encoding'=>'gzip'); $r->send_http_header; return OK if $r->header_only; my $gz = IO::Compress::Gzip->new( '-', Minimal => 1 ) or return DECLINED ; print $gz $_ while <$fh>; return OK; }
or even more succinctly, like this, using a one-shot gzip
package Apache::GZip; use strict vars; use Apache::Constants ':common'; use IO::Compress::Gzip qw(gzip); sub handler { my $r = shift; $r->header_out('Content-Encoding'=>'gzip'); $r->send_http_header; return OK if $r->header_only; gzip $r->filename => '-', Minimal => 1 or return DECLINED ; return OK; } 1;
The use of one-shot
gzip
above just reads from$r->filename
and writes the compressed data to standard output.Note the use of the
Minimal
option in the code above. When using gzip for Content-Encoding you should always use this option. In the example above it will prevent the filename being included in the gzip header and make the size of the gzip data stream a slight bit smaller.Compressed files and Net::FTP
The
Net::FTP
module provides two low-level methods calledstor
andretr
that both return filehandles. These filehandles can used with theIO::Compress/Uncompress
modules to compress or uncompress files read from or written to an FTP Server on the fly, without having to create a temporary file.Firstly, here is code that uses
retr
to uncompressed a file as it is read from the FTP Server.use Net::FTP; use IO::Uncompress::Gunzip qw(:all); my $ftp = Net::FTP->new( ... ) my $retr_fh = $ftp->retr($compressed_filename); gunzip $retr_fh => $outFilename, AutoClose => 1 or die "Cannot uncompress '$compressed_file': $GunzipError\n";
and this to compress a file as it is written to the FTP Server
use Net::FTP; use IO::Compress::Gzip qw(:all); my $stor_fh = $ftp->stor($filename); gzip "filename" => $stor_fh, AutoClose => 1 or die "Cannot compress '$filename': $GzipError\n";
MISC
Using
InputLength
to uncompress data embedded in a larger file/buffer.A fairly common use-case is where compressed data is embedded in a larger file/buffer and you want to read both.
As an example consider the structure of a zip file. This is a well-defined file format that mixes both compressed and uncompressed sections of data in a single file.
For the purposes of this discussion you can think of a zip file as sequence of compressed data streams, each of which is prefixed by an uncompressed local header. The local header contains information about the compressed data stream, including the name of the compressed file and, in particular, the length of the compressed data stream.
To illustrate how to use
InputLength
here is a script that walks a zip file and prints out how many lines are in each compressed file (if you intend write code to walking through a zip file for real see "Walking through a zip file" in IO::Uncompress::Unzip ). Also, although this example uses the zlib-based compression, the technique can be used by the otherIO::Uncompress::*
modules.use strict; use warnings; use IO::File; use IO::Uncompress::RawInflate qw(:all); use constant ZIP_LOCAL_HDR_SIG => 0x04034b50; use constant ZIP_LOCAL_HDR_LENGTH => 30; my $file = $ARGV[0] ; my $fh = IO::File->new( "<$file" ) or die "Cannot open '$file': $!\n"; while (1) { my $sig; my $buffer; my $x ; ($x = $fh->read($buffer, ZIP_LOCAL_HDR_LENGTH)) == ZIP_LOCAL_HDR_LENGTH or die "Truncated file: $!\n"; my $signature = unpack ("V", substr($buffer, 0, 4)); last unless $signature == ZIP_LOCAL_HDR_SIG; # Read Local Header my $gpFlag = unpack ("v", substr($buffer, 6, 2)); my $compressedMethod = unpack ("v", substr($buffer, 8, 2)); my $compressedLength = unpack ("V", substr($buffer, 18, 4)); my $uncompressedLength = unpack ("V", substr($buffer, 22, 4)); my $filename_length = unpack ("v", substr($buffer, 26, 2)); my $extra_length = unpack ("v", substr($buffer, 28, 2)); my $filename ; $fh->read($filename, $filename_length) == $filename_length or die "Truncated file\n"; $fh->read($buffer, $extra_length) == $extra_length or die "Truncated file\n"; if ($compressedMethod != 8 && $compressedMethod != 0) { warn "Skipping file '$filename' - not deflated $compressedMethod\n"; $fh->read($buffer, $compressedLength) == $compressedLength or die "Truncated file\n"; next; } if ($compressedMethod == 0 && $gpFlag & 8 == 8) { die "Streamed Stored not supported for '$filename'\n"; } next if $compressedLength == 0; # Done reading the Local Header my $inf = IO::Uncompress::RawInflate->new( $fh, Transparent => 1, InputLength => $compressedLength ) or die "Cannot uncompress $file [$filename]: $RawInflateError\n" ; my $line_count = 0; while (<$inf>) { ++ $line_count; } print "$filename: $line_count\n"; }
The majority of the code above is concerned with reading the zip local header data. The code that I want to focus on is at the bottom.
while (1) { # read local zip header data # get $filename # get $compressedLength my $inf = IO::Uncompress::RawInflate->new( $fh, Transparent => 1, InputLength => $compressedLength ) or die "Cannot uncompress $file [$filename]: $RawInflateError\n" ; my $line_count = 0; while (<$inf>) { ++ $line_count; } print "$filename: $line_count\n"; }
The call to
IO::Uncompress::RawInflate
creates a new filehandle$inf
that can be used to read from the parent filehandle$fh
, uncompressing it as it goes. The use of theInputLength
option will guarantee that at most$compressedLength
bytes of compressed data will be read from the$fh
filehandle (The only exception is for an error case like a truncated file or a corrupt data stream).This means that once RawInflate is finished
$fh
will be left at the byte directly after the compressed data stream.Now consider what the code looks like without
InputLength
while (1) { # read local zip header data # get $filename # get $compressedLength # read all the compressed data into $data read($fh, $data, $compressedLength); my $inf = IO::Uncompress::RawInflate->new( \$data, Transparent => 1 ) or die "Cannot uncompress $file [$filename]: $RawInflateError\n" ; my $line_count = 0; while (<$inf>) { ++ $line_count; } print "$filename: $line_count\n"; }
The difference here is the addition of the temporary variable
$data
. This is used to store a copy of the compressed data while it is being uncompressed.If you know that
$compressedLength
isn't that big then using temporary storage won't be a problem. But if$compressedLength
is very large or you are writing an application that other people will use, and so have no idea how big$compressedLength
will be, it could be an issue.Using
InputLength
avoids the use of temporary storage and means the application can cope with large compressed data streams.One final point -- obviously
InputLength
can only be used whenever you know the length of the compressed data beforehand, like here with a zip file.SUPPORT
General feedback/questions/bug reports should be sent to https://github.com/pmqs//issues (preferred) or https://rt.cpan.org/Public/Dist/Display.html?Name=.
SEE ALSO
Compress::Zlib, IO::Compress::Gzip, IO::Uncompress::Gunzip, IO::Compress::Deflate, IO::Uncompress::Inflate, IO::Compress::RawDeflate, IO::Uncompress::RawInflate, IO::Compress::Bzip2, IO::Uncompress::Bunzip2, IO::Compress::Lzma, IO::Uncompress::UnLzma, IO::Compress::Xz, IO::Uncompress::UnXz, IO::Compress::Lzip, IO::Uncompress::UnLzip, IO::Compress::Lzop, IO::Uncompress::UnLzop, IO::Compress::Lzf, IO::Uncompress::UnLzf, IO::Compress::Zstd, IO::Uncompress::UnZstd, IO::Uncompress::AnyInflate, IO::Uncompress::AnyUncompress
File::GlobMapper, Archive::Zip, Archive::Tar, IO::Zlib
AUTHOR
This module was written by Paul Marquess,
pmqs@cpan.org
.MODIFICATION HISTORY
See the Changes file.
COPYRIGHT AND LICENSE
Copyright (c) 2005-2022 Paul Marquess. All rights reserved.
This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
Module Install Instructions
To install Compress::Zlib, copy and paste the appropriate command in to your terminal.
cpanm Compress::Zlib
perl -MCPAN -e shell install Compress::Zlib
For more information on module installation, please visit the detailed CPAN module installation guide.