File-Rdiff/Rdiff.pm

=head1 NAME

File::Rdiff -- generate remote signatures and patch files using librsync

=head1 SYNOPSIS

 use File::Rdiff

=head1 DESCRIPTION

A more-or-less direct interface to librsync (L<http://librsync.sourceforge.net/>).

For usage examples (better than this very sparse documentation), see the
two example scripts below.

=over 4

=cut

package File::Rdiff;

require DynaLoader;
require Exporter;

$VERSION = '1.0';
@ISA = qw/DynaLoader Exporter/;

bootstrap File::Rdiff $VERSION;

{
   my @loglevels = qw(LOG_EMERG LOG_ALERT LOG_CRIT LOG_ERR LOG_WARNING LOG_NOTICE LOG_INFO LOG_DEBUG);
   my @result = qw(DONE BLOCKED RUNNING TEST_SKIPPED IO_ERROR SYNTAX_ERROR MEM_ERROR INPUT_ENDED BAD_MAGIC UNIMPLEMENTED CORRUPT INTERNAL_ERROR PARAM_ERROR);

   %EXPORT_TAGS = (
           loglevels   => [@loglevels],
           trace       => [@loglevels, qw(trace_level trace_to)],
           result      => [@result],
           error       => [@result, qw(strerror)],
           file        => [@result, qw(md4_file sig_file loadsig_file delta_file patch_file)],
           nonblocking => [@result],
   );

   my %export_ok;
   @export_ok{map @$_, values %EXPORT_TAGS} = ();
   @EXPORT_OK = keys %export_ok;
}

=item LIBRSYNC_VERSION

A constant describing the version of the rsync library used in this
module. My version claimed to be "0.9.5 librsync" when I wrote this
document ;)

=item $oldlevel = trace_level [$newlevel]

Return the current tracelevel and optionally set a new one.

=item $oldcb = trace_to [$newcb]

Return the current trace callback and optionally set a new one. The callback will be
called with the log level as the first argument and the log message as the second one.

Calling C<trace_to> with C<undef> will restore the default handler (which
currently prints the message to standard error).

=item supports_trace

Returns wether debugging traces are supported in this version of the library.

=item $msg = strerror $rcode

Returns a string representation of the given error code. You usually
just "exit(1)" or something when a function/method fails, as all (most?)
librsync functions log the error properly, so there is rarely a need to
call this function.

=item $md4 = md4_file $fh

=item sig_file $old_fh, $sig_fh[, $block_len[, $strong_len]]

=item $sig = loadsig_file $fh

=item delta_file $signature, $new_fh, $delta_fh

=item patch_file $base_fh, $delta_fh, $new_fh

=back

=head2 The File::Rdiff::Job class

=head2 The File::Rdiff::Buffers class

This class contains the input and output buffers for the non-blocking interface. It is slightly unusual
in that it allows direct manipulation of (some) of it's internal variables.

=over 4

=item new File::Rdiff::Buffers [$outsize]

Creates and initializes a new buffers structure. C<$outsize> specifies
the maximum number of bytes to be read into the output scalar until it is
considered full. The default is 64k.

=item $buffers->in($in)

Set the next block of input to consume. Data will be read from this scalar
(no copy will be made!) until all bytes have been consumed or a new input
scalar is set.

=item $out = $buffers->out

Return the current output data and create a new buffer. Returns C<undef> if no data has been accumulated.

=item $buffers->eof

Set the eof flag to true. This indicates that no data is following the current input scalar.

=item $buffers->avail_in

Returns the numer of bytes still available for input. If there are no
input bytes available but the eof flag is set, returns -1 (to make boolean
tests easy to check wether to supply more data easier).

=item $buffers->avail_out

Returns the number of bytes still available in the output buffer.

=item $buffers->size

The number of bytes that have been accumulated in the current buffer so far.

=back

=head2 The File::Rdiff::Job class

It is possible to have multiple jobs running at the same time. The idea
is to create job objects and then drive them incrementally with input or
output data until all date has been processed.

=over 4

=item new_sig File::Rdiff::Job [$new_block_len[, $strong_sum_len]]

Create a job that converts a base stream into a signature stream (i.e. creates signatures).

=item new_loadsig File::Rdiff::Job

Create a job that converts the input stream into a in-memory
File::Rdiff::Signature object. The signature object can be fetched anytime
with the C<signature>-method.

=item new_delta File::Rdiff::Job $signature

Creates a job that creates (outputs) a delta between the input stream (the
newer file) and the file represented by the given signature.

=item new_patch File::Rdiff::Job $callback_or_filehandle

Creates a job that patches a file according to the input stream (a delta
stream). The single argument is used to read the base file contents. If it
is a filehandle, it must be a seekable handle to the base file.

If it is a coderef, it will be called whenever base file data must be
read. Two arguments will be passed: the file offset and the length. The
callback should eithe return the data read (must be a string, not a
number!) or an error code.

=item $job->iter($buffers)

Do as much work as possible given the input and/or output data in the
File::Rdiff::Buffers structure and return either C<DONE> when the job is
finished, C<BLOCKED> if there aren't enough bytes available in the input
or output buffers (in which case you should deplete the output buffer
and/or fill the input buffer and loop), or some error code indicating that
the operation failed.

=item $job->signature

Only valid for C<new_loadsig>, so look there.

=back

=head1 EXAMPLE PROGRAM ONE

Very simple program that mimics librsync's rdiff, using the simple file
utility functions. see example below for the same program, written using
the nonblocking API.

   #!/usr/bin/perl

   use File::Rdiff qw(:trace :file);

   trace_level(LOG_INFO);

   if ($ARGV[0] eq "signature") {
      open $base, "<$ARGV[1]" or die "$ARGV[1]: $!";
      open $sig,  ">$ARGV[2]" or die "$ARGV[2]: $!";

      File::Rdiff::sig_file $base, $sig;
   } elsif ($ARGV[0] eq "delta") {
      open $sig,   "<$ARGV[1]" or die "$ARGV[1]: $!";
      open $new,   "<$ARGV[2]" or die "$ARGV[2]: $!";
      open $delta, ">$ARGV[3]" or die "$ARGV[3]: $!";

      $sig = loadsig_file $sig;

      ref $sig or exit 1;

      $sig->build_hash_table;

      File::Rdiff::delta_file $sig, $new, $delta;
   } elsif ($ARGV[0] eq "patch") {
      open $base,  "<$ARGV[1]" or die "$ARGV[1]: $!";
      open $delta, "<$ARGV[2]" or die "$ARGV[2]: $!";
      open $new,   ">$ARGV[3]" or die "$ARGV[3]: $!";

      File::Rdiff::patch_file $base, $delta, $new;
   } else {
      print <<EOF;
   $0 signature BASE SIGNATURE
   $0 delta SIGNATURE NEW DELTA
   $0 patch BASE DELTA NEW
   EOF
      exit (1);
   }

=head1 EXAMPLE PROGRAM TWO

Same as above, but written using the callback-based, "nonblocking", API.

   #!/usr/bin/perl

   use File::Rdiff qw(:trace :nonblocking);

   trace_level(LOG_INFO);

   if ($ARGV[0] eq "signature") {
      open $basis, "<", $ARGV[1]
         or die "$ARGV[1]: $!";
      open $sig, ">", $ARGV[2]
         or die "$ARGV[2]: $!";

      my $job = new_sig File::Rdiff::Job 128;
      my $buf = new File::Rdiff::Buffers 4096;

      while ($job->iter($buf) == BLOCKED) {
         # fetch more input data
         $buf->avail_in or do {
            my $in;
            65536 == sysread $basis, $in, 65536 or $buf->eof;
            $buf->in($in);
         };
         print $sig $buf->out;
      }
      print $sig $buf->out;

   } elsif ($ARGV[0] eq "delta") {
      open $sig,   "<$ARGV[1]" or die "$ARGV[1]: $!";
      open $new,   "<$ARGV[2]" or die "$ARGV[2]: $!";
      open $delta, ">$ARGV[3]" or die "$ARGV[3]: $!";

      # first load the signature into memory
      my $job = new_loadsig File::Rdiff::Job;
      my $buf = new File::Rdiff::Buffers 0;

      do {
         $buf->avail_in or do {
            my $in;
            65536 == sysread $sig, $in, 65536 or $buf->eof;
            $buf->in($in);
         };
      } while $job->iter($buf) == BLOCKED;

      $sig = $job->signature;

      $sig->build_hash_table;

      # now create the delta file
      my $job = new_delta File::Rdiff::Job $sig;
      my $buf = new File::Rdiff::Buffers 65536;

      do {
         $buf->avail_in or do {
            my $in;
            65536 == sysread $new, $in, 65536 or $buf->eof;
            $buf->in($in);
         };
         print $delta $buf->out;
      } while $job->iter($buf) == BLOCKED;
      print $delta $buf->out;

   } elsif ($ARGV[0] eq "patch") {
      open $base,  "<$ARGV[1]" or die "$ARGV[1]: $!";
      open $delta, "<$ARGV[2]" or die "$ARGV[2]: $!";
      open $new,   ">$ARGV[3]" or die "$ARGV[3]: $!";

      # NYI
      File::Rdiff::patch_file $base, $delta, $new;
   } else {
      print <<EOF;
   $0 signature BASIS SIGNATURE
   $0 delta SIGNATURE NEW DELTA
   $0 patch BASE DELTA NEW
   EOF
      exit (1);
   }

=head1 SEE ALSO

L<File::Rsync>, L<rdiff1> (usage example using simple file API), L<rdiff2> (example using nonblocking API).

=head1 BUGS

- not well-tested so far.

- low memory will result in segfaults rather than croaks.

- no access to statistics yet

- documentation leaves much to be deserved.

=head1 AUTHOR

 Marc Lehmann <schmorp@schmorp.de>
 http://home.schmorp.de/

=cut

1;

Revision:	1.6
Committed:	Thu Jul 24 21:05:17 2008 UTC (15 years, 9 months ago) by root
Branch:	MAIN
CVS Tags:	rel-1_0, HEAD
Changes since 1.5:	+128 -3 lines
Log Message:	1.0
#	User	Rev	Content
1	root	1.1	=head1 NAME
2
3	root	1.3	File::Rdiff -- generate remote signatures and patch files using librsync
4	root	1.1
5			=head1 SYNOPSIS
6
7			use File::Rdiff
8
9			=head1 DESCRIPTION
10
11	root	1.6	A more-or-less direct interface to librsync (L<http://librsync.sourceforge.net/>).
12	root	1.1
13			For usage examples (better than this very sparse documentation), see the
14	root	1.6	two example scripts below.
15	root	1.1
16			=over 4
17
18			=cut
19
20			package File::Rdiff;
21
22			require DynaLoader;
23			require Exporter;
24
25	root	1.6	$VERSION = '1.0';
26	root	1.1	@ISA = qw/DynaLoader Exporter/;
27
28			bootstrap File::Rdiff $VERSION;
29
30			{
31			my @loglevels = qw(LOG_EMERG LOG_ALERT LOG_CRIT LOG_ERR LOG_WARNING LOG_NOTICE LOG_INFO LOG_DEBUG);
32			my @result = qw(DONE BLOCKED RUNNING TEST_SKIPPED IO_ERROR SYNTAX_ERROR MEM_ERROR INPUT_ENDED BAD_MAGIC UNIMPLEMENTED CORRUPT INTERNAL_ERROR PARAM_ERROR);
33
34			%EXPORT_TAGS = (
35			loglevels => [@loglevels],
36			trace => [@loglevels, qw(trace_level trace_to)],
37			result => [@result],
38			error => [@result, qw(strerror)],
39			file => [@result, qw(md4_file sig_file loadsig_file delta_file patch_file)],
40			nonblocking => [@result],
41			);
42
43			my %export_ok;
44			@export_ok{map @$_, values %EXPORT_TAGS} = ();
45			@EXPORT_OK = keys %export_ok;
46			}
47
48			=item LIBRSYNC_VERSION
49
50			A constant describing the version of the rsync library used in this
51			module. My version claimed to be "0.9.5 librsync" when I wrote this
52			document ;)
53
54			=item $oldlevel = trace_level [$newlevel]
55
56			Return the current tracelevel and optionally set a new one.
57
58			=item $oldcb = trace_to [$newcb]
59
60			Return the current trace callback and optionally set a new one. The callback will be
61			called with the log level as the first argument and the log message as the second one.
62
63			Calling C<trace_to> with C<undef> will restore the default handler (which
64			currently prints the message to standard error).
65
66			=item supports_trace
67
68			Returns wether debugging traces are supported in this version of the library.
69
70			=item $msg = strerror $rcode
71
72			Returns a string representation of the given error code. You usually
73			just "exit(1)" or something when a function/method fails, as all (most?)
74			librsync functions log the error properly, so there is rarely a need to
75			call this function.
76
77			=item $md4 = md4_file $fh
78
79			=item sig_file $old_fh, $sig_fh[, $block_len[, $strong_len]]
80
81			=item $sig = loadsig_file $fh
82
83			=item delta_file $signature, $new_fh, $delta_fh
84
85			=item patch_file $base_fh, $delta_fh, $new_fh
86
87			=back
88
89			=head2 The File::Rdiff::Job class
90
91			=head2 The File::Rdiff::Buffers class
92
93			This class contains the input and output buffers for the non-blocking interface. It is slightly unusual
94			in that it allows direct manipulation of (some) of it's internal variables.
95
96			=over 4
97
98			=item new File::Rdiff::Buffers [$outsize]
99
100			Creates and initializes a new buffers structure. C<$outsize> specifies
101			the maximum number of bytes to be read into the output scalar until it is
102			considered full. The default is 64k.
103
104			=item $buffers->in($in)
105
106			Set the next block of input to consume. Data will be read from this scalar
107			(no copy will be made!) until all bytes have been consumed or a new input
108			scalar is set.
109
110			=item $out = $buffers->out
111
112			Return the current output data and create a new buffer. Returns C<undef> if no data has been accumulated.
113
114			=item $buffers->eof
115
116			Set the eof flag to true. This indicates that no data is following the current input scalar.
117
118			=item $buffers->avail_in
119
120	root	1.2	Returns the numer of bytes still available for input. If there are no
121			input bytes available but the eof flag is set, returns -1 (to make boolean
122			tests easy to check wether to supply more data easier).
123	root	1.1
124			=item $buffers->avail_out
125
126			Returns the number of bytes still available in the output buffer.
127
128			=item $buffers->size
129
130			The number of bytes that have been accumulated in the current buffer so far.
131
132			=back
133
134			=head2 The File::Rdiff::Job class
135
136	root	1.2	It is possible to have multiple jobs running at the same time. The idea
137			is to create job objects and then drive them incrementally with input or
138			output data until all date has been processed.
139
140	root	1.1	=over 4
141	root	1.2
142			=item new_sig File::Rdiff::Job [$new_block_len[, $strong_sum_len]]
143
144			Create a job that converts a base stream into a signature stream (i.e. creates signatures).
145
146			=item new_loadsig File::Rdiff::Job
147
148			Create a job that converts the input stream into a in-memory
149			File::Rdiff::Signature object. The signature object can be fetched anytime
150			with the C<signature>-method.
151
152			=item new_delta File::Rdiff::Job $signature
153
154			Creates a job that creates (outputs) a delta between the input stream (the
155			newer file) and the file represented by the given signature.
156
157			=item new_patch File::Rdiff::Job $callback_or_filehandle
158
159			Creates a job that patches a file according to the input stream (a delta
160			stream). The single argument is used to read the base file contents. If it
161			is a filehandle, it must be a seekable handle to the base file.
162
163			If it is a coderef, it will be called whenever base file data must be
164			read. Two arguments will be passed: the file offset and the length. The
165			callback should eithe return the data read (must be a string, not a
166			number!) or an error code.
167
168			=item $job->iter($buffers)
169
170			Do as much work as possible given the input and/or output data in the
171			File::Rdiff::Buffers structure and return either C<DONE> when the job is
172			finished, C<BLOCKED> if there aren't enough bytes available in the input
173			or output buffers (in which case you should deplete the output buffer
174			and/or fill the input buffer and loop), or some error code indicating that
175			the operation failed.
176
177			=item $job->signature
178
179			Only valid for C<new_loadsig>, so look there.
180	root	1.1
181			=back
182
183	root	1.6	=head1 EXAMPLE PROGRAM ONE
184
185			Very simple program that mimics librsync's rdiff, using the simple file
186			utility functions. see example below for the same program, written using
187			the nonblocking API.
188
189			#!/usr/bin/perl
190
191			use File::Rdiff qw(:trace :file);
192
193			trace_level(LOG_INFO);
194
195			if ($ARGV[0] eq "signature") {
196			open $base, "<$ARGV[1]" or die "$ARGV[1]: $!";
197			open $sig, ">$ARGV[2]" or die "$ARGV[2]: $!";
198
199			File::Rdiff::sig_file $base, $sig;
200			} elsif ($ARGV[0] eq "delta") {
201			open $sig, "<$ARGV[1]" or die "$ARGV[1]: $!";
202			open $new, "<$ARGV[2]" or die "$ARGV[2]: $!";
203			open $delta, ">$ARGV[3]" or die "$ARGV[3]: $!";
204
205			$sig = loadsig_file $sig;
206
207			ref $sig or exit 1;
208
209			$sig->build_hash_table;
210
211			File::Rdiff::delta_file $sig, $new, $delta;
212			} elsif ($ARGV[0] eq "patch") {
213			open $base, "<$ARGV[1]" or die "$ARGV[1]: $!";
214			open $delta, "<$ARGV[2]" or die "$ARGV[2]: $!";
215			open $new, ">$ARGV[3]" or die "$ARGV[3]: $!";
216
217			File::Rdiff::patch_file $base, $delta, $new;
218			} else {
219			print <<EOF;
220			$0 signature BASE SIGNATURE
221			$0 delta SIGNATURE NEW DELTA
222			$0 patch BASE DELTA NEW
223			EOF
224			exit (1);
225			}
226
227			=head1 EXAMPLE PROGRAM TWO
228
229			Same as above, but written using the callback-based, "nonblocking", API.
230
231			#!/usr/bin/perl
232
233			use File::Rdiff qw(:trace :nonblocking);
234
235			trace_level(LOG_INFO);
236
237			if ($ARGV[0] eq "signature") {
238			open $basis, "<", $ARGV[1]
239			or die "$ARGV[1]: $!";
240			open $sig, ">", $ARGV[2]
241			or die "$ARGV[2]: $!";
242
243			my $job = new_sig File::Rdiff::Job 128;
244			my $buf = new File::Rdiff::Buffers 4096;
245
246			while ($job->iter($buf) == BLOCKED) {
247			# fetch more input data
248			$buf->avail_in or do {
249			my $in;
250			65536 == sysread $basis, $in, 65536 or $buf->eof;
251			$buf->in($in);
252			};
253			print $sig $buf->out;
254			}
255			print $sig $buf->out;
256
257			} elsif ($ARGV[0] eq "delta") {
258			open $sig, "<$ARGV[1]" or die "$ARGV[1]: $!";
259			open $new, "<$ARGV[2]" or die "$ARGV[2]: $!";
260			open $delta, ">$ARGV[3]" or die "$ARGV[3]: $!";
261
262			# first load the signature into memory
263			my $job = new_loadsig File::Rdiff::Job;
264			my $buf = new File::Rdiff::Buffers 0;
265
266			do {
267			$buf->avail_in or do {
268			my $in;
269			65536 == sysread $sig, $in, 65536 or $buf->eof;
270			$buf->in($in);
271			};
272			} while $job->iter($buf) == BLOCKED;
273
274			$sig = $job->signature;
275
276			$sig->build_hash_table;
277
278			# now create the delta file
279			my $job = new_delta File::Rdiff::Job $sig;
280			my $buf = new File::Rdiff::Buffers 65536;
281
282			do {
283			$buf->avail_in or do {
284			my $in;
285			65536 == sysread $new, $in, 65536 or $buf->eof;
286			$buf->in($in);
287			};
288			print $delta $buf->out;
289			} while $job->iter($buf) == BLOCKED;
290			print $delta $buf->out;
291
292			} elsif ($ARGV[0] eq "patch") {
293			open $base, "<$ARGV[1]" or die "$ARGV[1]: $!";
294			open $delta, "<$ARGV[2]" or die "$ARGV[2]: $!";
295			open $new, ">$ARGV[3]" or die "$ARGV[3]: $!";
296
297			# NYI
298			File::Rdiff::patch_file $base, $delta, $new;
299			} else {
300			print <<EOF;
301			$0 signature BASIS SIGNATURE
302			$0 delta SIGNATURE NEW DELTA
303			$0 patch BASE DELTA NEW
304			EOF
305			exit (1);
306			}
307
308	root	1.1	=head1 SEE ALSO
309
310			L<File::Rsync>, L<rdiff1> (usage example using simple file API), L<rdiff2> (example using nonblocking API).
311
312			=head1 BUGS
313
314			- not well-tested so far.
315
316			- low memory will result in segfaults rather than croaks.
317
318			- no access to statistics yet
319
320			- documentation leaves much to be deserved.
321
322			=head1 AUTHOR
323
324	root	1.5	Marc Lehmann <schmorp@schmorp.de>
325	root	1.4	http://home.schmorp.de/
326	root	1.1
327			=cut
328
329			1;
330