ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/Async-Interrupt/Interrupt.pm
(Generate patch)

Comparing Async-Interrupt/Interrupt.pm (file contents):
Revision 1.5 by root, Fri Jul 3 21:11:22 2009 UTC vs.
Revision 1.21 by root, Thu Jul 30 03:59:47 2009 UTC

10 10
11This module implements a single feature only of interest to advanced perl 11This module implements a single feature only of interest to advanced perl
12modules, namely asynchronous interruptions (think "UNIX signals", which 12modules, namely asynchronous interruptions (think "UNIX signals", which
13are very similar). 13are very similar).
14 14
15Sometimes, modules wish to run code asynchronously (in another thread), 15Sometimes, modules wish to run code asynchronously (in another thread,
16and then signal the perl interpreter on certain events. One common way is 16or from a signal handler), and then signal the perl interpreter on
17to write some data to a pipe and use an event handling toolkit to watch 17certain events. One common way is to write some data to a pipe and use an
18for I/O events. Another way is to send a signal. Those methods are slow, 18event handling toolkit to watch for I/O events. Another way is to send
19and in the case of a pipe, also not asynchronous - it won't interrupt a 19a signal. Those methods are slow, and in the case of a pipe, also not
20running perl interpreter. 20asynchronous - it won't interrupt a running perl interpreter.
21 21
22This module implements asynchronous notifications that enable you to 22This module implements asynchronous notifications that enable you to
23signal running perl code form another thread, asynchronously, without 23signal running perl code from another thread, asynchronously, and
24issuing syscalls. 24sometimes even without using a single syscall.
25 25
26It works by creating an C<Async::Interrupt> object for each such use. This 26=head2 USAGE SCENARIOS
27object stores a perl and/or a C-level callback that is invoked when the 27
28C<Async::Interrupt> object gets signalled. It is executed at the next time 28=over 4
29the perl interpreter is running (i.e. it will interrupt a computation, but 29
30not an XS function or a syscall). 30=item Race-free signal handling
31
32There seems to be no way to do race-free signal handling in perl: to
33catch a signal, you have to execute Perl code, and between entering the
34interpreter C<select> function (or other blocking functions) and executing
35the select syscall is a small but relevant timespan during which signals
36will be queued, but perl signal handlers will not be executed and the
37blocking syscall will not be interrupted.
38
39You can use this module to bind a signal to a callback while at the same
40time activating an event pipe that you can C<select> on, fixing the race
41completely.
42
43This can be used to implement the signal hadling in event loops,
44e.g. L<AnyEvent>, L<POE>, L<IO::Async::Loop> and so on.
45
46=item Background threads want speedy reporting
47
48Assume you want very exact timing, and you can spare an extra cpu core
49for that. Then you can run an extra thread that signals your perl
50interpreter. This means you can get a very exact timing source while your
51perl code is number crunching, without even using a syscall to communicate
52between your threads.
53
54For example the deliantra game server uses a variant of this technique
55to interrupt background processes regularly to send map updates to game
56clients.
57
58Or L<EV::Loop::Async> uses an interrupt object to wake up perl when new
59events have arrived.
60
61L<IO::AIO> and L<BDB> could also use this to speed up result reporting.
62
63=item Speedy event loop invocation
64
65One could use this module e.g. in L<Coro> to interrupt a running coro-thread
66and cause it to enter the event loop.
67
68Or one could bind to C<SIGIO> and tell some important sockets to send this
69signal, causing the event loop to be entered to reduce network latency.
70
71=back
72
73=head2 HOW TO USE
74
75You can use this module by creating an C<Async::Interrupt> object for each
76such event source. This object stores a perl and/or a C-level callback
77that is invoked when the C<Async::Interrupt> object gets signalled. It is
78executed at the next time the perl interpreter is running (i.e. it will
79interrupt a computation, but not an XS function or a syscall).
31 80
32You can signal the C<Async::Interrupt> object either by calling it's C<< 81You can signal the C<Async::Interrupt> object either by calling it's C<<
33->signal >> method, or, more commonly, by calling a C function. 82->signal >> method, or, more commonly, by calling a C function. There is
83also the built-in (POSIX) signal source.
34 84
35The C<< ->signal_func >> returns the address of the C function that is to 85The C<< ->signal_func >> returns the address of the C function that is to
36be called (plus an argument to be used during the call). The signalling 86be called (plus an argument to be used during the call). The signalling
37function also takes an integer argument in the range SIG_ATOMIC_MIN to 87function also takes an integer argument in the range SIG_ATOMIC_MIN to
38SIG_ATOMIC_MAX (guaranteed to allow at least 0..127). 88SIG_ATOMIC_MAX (guaranteed to allow at least 0..127).
39 89
40Since this kind of interruption is fast, but can only interrupt a 90Since this kind of interruption is fast, but can only interrupt a
41I<running> interpreter, there is optional support for also signalling a 91I<running> interpreter, there is optional support for signalling a pipe
42pipe - that means you can also wait for the pipe to become readable (e.g. 92- that means you can also wait for the pipe to become readable (e.g. via
43via L<EV> or L<AnyEvent>). This, of course, incurs the overhead of a 93L<EV> or L<AnyEvent>). This, of course, incurs the overhead of a C<read>
44C<read> and C<write> syscall. 94and C<write> syscall.
95
96=head1 USAGE EXAMPLES
97
98=head2 Implementing race-free signal handling
99
100This example uses a single event pipe for all signals, and one
101Async::Interrupt per signal. This code is actually what the L<AnyEvent>
102module uses itself when Async::Interrupt is available.
103
104First, create the event pipe and hook it into the event loop
105
106 $SIGPIPE = new Async::Interrupt::EventPipe;
107 $SIGPIPE_W = AnyEvent->io (
108 fh => $SIGPIPE->fileno,
109 poll => "r",
110 cb => \&_signal_check, # defined later
111 );
112
113Then, for each signal to hook, create an Async::Interrupt object. The
114callback just sets a global variable, as we are only interested in
115synchronous signals (i.e. when the event loop polls), which is why the
116pipe draining is not done automatically.
117
118 my $interrupt = new Async::Interrupt
119 cb => sub { undef $SIGNAL_RECEIVED{$signum} }
120 signal => $signum,
121 pipe => [$SIGPIPE->filenos],
122 pipe_autodrain => 0,
123 ;
124
125Finally, the I/O callback for the event pipe handles the signals:
126
127 sub _signal_check {
128 # drain the pipe first
129 $SIGPIPE->drain;
130
131 # two loops, just to be sure
132 while (%SIGNAL_RECEIVED) {
133 for (keys %SIGNAL_RECEIVED) {
134 delete $SIGNAL_RECEIVED{$_};
135 warn "signal $_ received\n";
136 }
137 }
138 }
139
140=head2 Interrupt perl from another thread
141
142This example interrupts the Perl interpreter from another thread, via the
143XS API. This is used by e.g. the L<EV::Loop::Async> module.
144
145On the Perl level, a new loop object (which contains the thread)
146is created, by first calling some XS constructor, querying the
147C-level callback function and feeding that as the C<c_cb> into the
148Async::Interrupt constructor:
149
150 my $self = XS_thread_constructor;
151 my ($c_func, $c_arg) = _c_func $self; # return the c callback
152 my $asy = new Async::Interrupt c_cb => [$c_func, $c_arg];
153
154Then the newly created Interrupt object is queried for the signaling
155function that the newly created thread should call, and this is in turn
156told to the thread object:
157
158 _attach $self, $asy->signal_func;
159
160So to repeat: first the XS object is created, then it is queried for the
161callback that should be called when the Interrupt object gets signalled.
162
163Then the interrupt object is queried for the callback fucntion that the
164thread should call to signal the Interrupt object, and this callback is
165then attached to the thread.
166
167You have to be careful that your new thread is not signalling before the
168signal function was configured, for example by starting the background
169thread only within C<_attach>.
170
171That concludes the Perl part.
172
173The XS part consists of the actual constructor which creates a thread,
174which is not relevant for this example, and two functions, C<_c_func>,
175which returns the Perl-side callback, and C<_attach>, which configures
176the signalling functioon that is safe toc all from another thread. For
177simplicity, we will use global variables to store the functions, normally
178you would somehow attach them to C<$self>.
179
180The C<c_func> simply returns the address of a static function and arranges
181for the object pointed to by C<$self> to be passed to it, as an integer:
182
183 void
184 _c_func (SV *loop)
185 PPCODE:
186 EXTEND (SP, 2);
187 PUSHs (sv_2mortal (newSViv (PTR2IV (c_func))));
188 PUSHs (sv_2mortal (newSViv (SvRV (loop))));
189
190This would be the callback (since it runs in a normal Perl context, it is
191permissible to manipulate Perl values):
192
193 static void
194 c_func (pTHX_ void *loop_, int value)
195 {
196 SV *loop_object = (SV *)loop_;
197 ...
198 }
199
200And this attaches the signalling callback:
201
202 static void (*my_sig_func) (void *signal_arg, int value);
203 static void *my_sig_arg;
204
205 void
206 _attach (SV *loop_, IV sig_func, void *sig_arg)
207 CODE:
208 {
209 my_sig_func = sig_func;
210 my_sig_arg = sig_arg;
211
212 /* now run the thread */
213 thread_create (&u->tid, l_run, 0);
214 }
215
216And C<l_run> (the background thread) would eventually call the signaling
217function:
218
219 my_sig_func (my_sig_arg, 0);
220
221You can have a look at L<EV::Loop::Async> for an actual example using
222intra-thread communication, locking and so on.
223
224
225=head1 THE Async::Interrupt CLASS
45 226
46=over 4 227=over 4
47 228
48=cut 229=cut
49 230
50package Async::Interrupt; 231package Async::Interrupt;
51 232
52no warnings; 233use common::sense;
53 234
54BEGIN { 235BEGIN {
236 # the next line forces initialisation of internal
237 # signal handling variables, otherwise, PL_sig_pending
238 # etc. will be null pointers.
239 $SIG{KILL} = sub { };
240
55 $VERSION = '0.03'; 241 our $VERSION = '1.0';
56 242
57 require XSLoader; 243 require XSLoader;
58 XSLoader::load Async::Interrupt::, $VERSION; 244 XSLoader::load ("Async::Interrupt", $VERSION);
59} 245}
60 246
61our $DIED = sub { warn "$@" }; 247our $DIED = sub { warn "$@" };
62 248
63=item $async = new Async::Interrupt key => value... 249=item $async = new Async::Interrupt key => value...
83The exceptions are C<$!> and C<$@>, which are saved and restored by 269The exceptions are C<$!> and C<$@>, which are saved and restored by
84Async::Interrupt. 270Async::Interrupt.
85 271
86If the callback should throw an exception, then it will be caught, 272If the callback should throw an exception, then it will be caught,
87and C<$Async::Interrupt::DIED> will be called with C<$@> containing 273and C<$Async::Interrupt::DIED> will be called with C<$@> containing
88the exception. The default will simply C<warn> about the message and 274the exception. The default will simply C<warn> about the message and
89continue. 275continue.
90 276
91=item c_cb => [$c_func, $c_arg] 277=item c_cb => [$c_func, $c_arg]
92 278
93Registers a C callback the be invoked whenever the async interrupt is 279Registers a C callback the be invoked whenever the async interrupt is
106might use (the exception is C<errno>, which is saved and restored by 292might use (the exception is C<errno>, which is saved and restored by
107Async::Interrupt). The callback itself runs as part of the perl context, 293Async::Interrupt). The callback itself runs as part of the perl context,
108so you can call any perl functions and modify any perl data structures (in 294so you can call any perl functions and modify any perl data structures (in
109which case the requirements set out for C<cb> apply as well). 295which case the requirements set out for C<cb> apply as well).
110 296
297=item var => $scalar_ref
298
299When specified, then the given argument must be a reference to a
300scalar. The scalar will be set to C<0> initially. Signalling the interrupt
301object will set it to the passed value, handling the interrupt will reset
302it to C<0> again.
303
304Note that the only thing you are legally allowed to do is to is to check
305the variable in a boolean or integer context (e.g. comparing it with a
306string, or printing it, will I<destroy> it and might cause your program to
307crash or worse).
308
309=item signal => $signame_or_value
310
311When this parameter is specified, then the Async::Interrupt will hook the
312given signal, that is, it will effectively call C<< ->signal (0) >> each time
313the given signal is caught by the process.
314
315Only one async can hook a given signal, and the signal will be restored to
316defaults when the Async::Interrupt object gets destroyed.
317
318=item signal_hysteresis => $boolean
319
320Sets the initial signal hysteresis state, see the C<signal_hysteresis>
321method, below.
322
111=item pipe => [$fileno_or_fh_for_reading, $fileno_or_fh_for_writing] 323=item pipe => [$fileno_or_fh_for_reading, $fileno_or_fh_for_writing]
112 324
113Specifies two file descriptors (or file handles) that should be signalled 325Specifies two file descriptors (or file handles) that should be signalled
114whenever the async interrupt is signalled. This means a single octet will 326whenever the async interrupt is signalled. This means a single octet will
115be written to it, and before the callback is being invoked, it will be 327be written to it, and before the callback is being invoked, it will be
116read again. Due to races, it is unlikely but possible that multiple octets 328read again. Due to races, it is unlikely but possible that multiple octets
117are written. It is required that the file handles are both in nonblocking 329are written. It is required that the file handles are both in nonblocking
118mode. 330mode.
119 331
120(You can get a portable pipe and set non-blocking mode portably by using
121e.g. L<AnyEvent::Util> from the L<AnyEvent> distribution).
122
123The object will keep a reference to the file handles. 332The object will keep a reference to the file handles.
124 333
125This can be used to ensure that async notifications will interrupt event 334This can be used to ensure that async notifications will interrupt event
126frameworks as well. 335frameworks as well.
127 336
337Note that C<Async::Interrupt> will create a suitable signal fd
338automatically when your program requests one, so you don't have to specify
339this argument when all you want is an extra file descriptor to watch.
340
341If you want to share a single event pipe between multiple Async::Interrupt
342objects, you can use the C<Async::Interrupt::EventPipe> class to manage
343those.
344
345=item pipe_autodrain => $boolean
346
347Sets the initial autodrain state, see the C<pipe_autodrain> method, below.
348
128=back 349=back
129 350
130=cut 351=cut
131 352
132sub new { 353sub new {
133 my ($class, %arg) = @_; 354 my ($class, %arg) = @_;
134 355
135 bless \(_alloc $arg{cb}, @{$arg{c_cb}}[0,1], @{$arg{pipe}}[0,1]), $class 356 my $self = bless \(_alloc $arg{cb}, @{$arg{c_cb}}[0,1], @{$arg{pipe}}[0,1], $arg{signal}, $arg{var}), $class;
357
358 # urgs, reminds me of Event
359 for my $attr (qw(pipe_autodrain signal_hysteresis)) {
360 $self->$attr ($arg{$attr}) if exists $arg{$attr};
361 }
362
363 $self
136} 364}
137 365
138=item ($signal_func, $signal_arg) = $async->signal_func 366=item ($signal_func, $signal_arg) = $async->signal_func
139 367
140Returns the address of a function to call asynchronously. The function has 368Returns the address of a function to call asynchronously. The function
141the following prototype and needs to be passed the specified C<$c_arg>, 369has the following prototype and needs to be passed the specified
142which is a C<void *> cast to C<IV>: 370C<$signal_arg>, which is a C<void *> cast to C<IV>:
143 371
144 void (*signal_func) (void *signal_arg, int value) 372 void (*signal_func) (void *signal_arg, int value)
145 373
146An example call would look like: 374An example call would look like:
147 375
148 signal_func (signal_arg, 0); 376 signal_func (signal_arg, 0);
149 377
150The function is safe to call from within signal and thread contexts, at 378The function is safe to call from within signal and thread contexts, at
151any time. The specified C<value> is passed to both C and Perl callback. 379any time. The specified C<value> is passed to both C and Perl callback.
152 380
153C<$value> must be in the valid range for a C<sig_atomic_t> (0..127 is 381C<$value> must be in the valid range for a C<sig_atomic_t>, except C<0>
154portable). 382(1..127 is portable).
155 383
156If the function is called while the Async::Interrupt object is already 384If the function is called while the Async::Interrupt object is already
157signaled but before the callbacks are being executed, then the stored 385signaled but before the callbacks are being executed, then the stored
158C<value> is either the old or the new one. Due to the asynchronous 386C<value> is either the old or the new one. Due to the asynchronous
159nature of the code, the C<value> can even be passed to two consecutive 387nature of the code, the C<value> can even be passed to two consecutive
160invocations of the callback. 388invocations of the callback.
161 389
390=item $address = $async->c_var
391
392Returns the address (cast to IV) of an C<IV> variable. The variable is set
393to C<0> initially and gets set to the passed value whenever the object
394gets signalled, and reset to C<0> once the interrupt has been handled.
395
396Note that it is often beneficial to just call C<PERL_ASYNC_CHECK ()> to
397handle any interrupts.
398
399Example: call some XS function to store the address, then show C code
400waiting for it.
401
402 my_xs_func $async->c_var;
403
404 static IV *valuep;
405
406 void
407 my_xs_func (void *addr)
408 CODE:
409 valuep = (IV *)addr;
410
411 // code in a loop, waiting
412 while (!*valuep)
413 ; // do something
414
162=item $async->signal ($value=0) 415=item $async->signal ($value=1)
163 416
164This signals the given async object from Perl code. Semi-obviously, this 417This signals the given async object from Perl code. Semi-obviously, this
165will instantly trigger the callback invocation. 418will instantly trigger the callback invocation (it does not, as the name
419might imply, do anything with POSIX signals).
166 420
167C<$value> must be in the valid range for a C<sig_atomic_t> (0..127 is 421C<$value> must be in the valid range for a C<sig_atomic_t>, except C<0>
168portable). 422(1..127 is portable).
423
424=item $async->signal_hysteresis ($enable)
425
426Enables or disables signal hysteresis (default: disabled). If a POSIX
427signal is used as a signal source for the interrupt object, then enabling
428signal hysteresis causes Async::Interrupt to reset the signal action to
429C<SIG_IGN> in the signal handler and restore it just before handling the
430interruption.
431
432When you expect a lot of signals (e.g. when using SIGIO), then enabling
433signal hysteresis can reduce the number of handler invocations
434considerably, at the cost of two extra syscalls.
435
436Note that setting the signal to C<SIG_IGN> can have unintended side
437effects when you fork and exec other programs, as often they do nto expect
438signals to be ignored by default.
169 439
170=item $async->block 440=item $async->block
171 441
172=item $async->unblock 442=item $async->unblock
173 443
189the current scope is exited (via an exception, by canceling the Coro 459the current scope is exited (via an exception, by canceling the Coro
190thread, by calling last/goto etc.). 460thread, by calling last/goto etc.).
191 461
192This is the recommended (and fastest) way to implement critical sections. 462This is the recommended (and fastest) way to implement critical sections.
193 463
464=item ($block_func, $block_arg) = $async->scope_block_func
465
466Returns the address of a function that implements the C<scope_block>
467functionality.
468
469It has the following prototype and needs to be passed the specified
470C<$block_arg>, which is a C<void *> cast to C<IV>:
471
472 void (*block_func) (void *block_arg)
473
474An example call would look like:
475
476 block_func (block_arg);
477
478The function is safe to call only from within the toplevel of a perl XS
479function and will call C<LEAVE> and C<ENTER> (in this order!).
480
481=item $async->pipe_enable
482
483=item $async->pipe_disable
484
485Enable/disable signalling the pipe when the interrupt occurs (default is
486enabled). Writing to a pipe is relatively expensive, so it can be disabled
487when you know you are not waiting for it (for example, with L<EV> you
488could disable the pipe in a check watcher, and enable it in a prepare
489watcher).
490
491Note that currently, while C<pipe_disable> is in effect, no attempt to
492read from the pipe will be done when handling events. This might change as
493soon as I realize why this is a mistake.
494
495=item $fileno = $async->pipe_fileno
496
497Returns the reading side of the signalling pipe. If no signalling pipe is
498currently attached to the object, it will dynamically create one.
499
500Note that the only valid oepration on this file descriptor is to wait
501until it is readable. The fd might belong currently to a pipe, a tcp
502socket, or an eventfd, depending on the platform, and is guaranteed to be
503C<select>able.
504
505=item $async->pipe_autodrain ($enable)
506
507Enables (C<1>) or disables (C<0>) automatic draining of the pipe (default:
508enabled). When automatic draining is enabled, then Async::Interrupt will
509automatically clear the pipe. Otherwise the user is responsible for this
510draining.
511
512This is useful when you want to share one pipe among many Async::Interrupt
513objects.
514
515=item $async->post_fork
516
517The object will not normally be usable after a fork (as the pipe fd is
518shared between processes). Calling this method after a fork in the child
519ensures that the object will work as expected again. It only needs to be
520called when the async object is used in the child.
521
522This only works when the pipe was created by Async::Interrupt.
523
524Async::Interrupt ensures that the reading file descriptor does not change
525it's value.
526
527=item $signum = Async::Interrupt::sig2num $signame_or_number
528
529=item $signame = Async::Interrupt::sig2name $signame_or_number
530
531These two convenience functions simply convert a signal name or number to
532the corresponding name or number. They are not used by this module and
533exist just because perl doesn't have a nice way to do this on its own.
534
535They will return C<undef> on illegal names or numbers.
536
537=back
538
539=head1 THE Async::Interrupt::EventPipe CLASS
540
541Pipes are the predominent utility to make asynchronous signals
542synchronous. However, pipes are hard to come by: they don't exist on the
543broken windows platform, and on GNU/Linux systems, you might want to use
544an C<eventfd> instead.
545
546This class creates selectable event pipes in a portable fashion: on
547windows, it will try to create a tcp socket pair, on GNU/Linux, it will
548try to create an eventfd and everywhere else it will try to use a normal
549pipe.
550
551=over 4
552
553=item $epipe = new Async::Interrupt::EventPipe
554
555This creates and returns an eventpipe object. This object is simply a
556blessed array reference:
557
558=item ($r_fd, $w_fd) = $epipe->filenos
559
560Returns the read-side file descriptor and the write-side file descriptor.
561
562Example: pass an eventpipe object as pipe to the Async::Interrupt
563constructor, and create an AnyEvent watcher for the read side.
564
565 my $epipe = new Async::Interrupt::EventPipe;
566 my $asy = new Async::Interrupt pipe => [$epipe->filenos];
567 my $iow = AnyEvent->io (fh => $epipe->fileno, poll => 'r', cb => sub { });
568
569=item $r_fd = $epipe->fileno
570
571Return only the reading/listening side.
572
573=item $epipe->signal
574
575Write something to the pipe, in a portable fashion.
576
577=item $epipe->drain
578
579Drain (empty) the pipe.
580
581=item ($c_func, $c_arg) = $epipe->drain_func
582
583Returns a function pointer and C<void *> argument that can be called to
584have the effect of C<< $epipe->drain >> on the XS level.
585
586It has the following prototype and needs to be passed the specified
587C<$c_arg>, which is a C<void *> cast to C<IV>:
588
589 void (*c_func) (void *c_arg)
590
591An example call would look like:
592
593 c_func (c_arg);
594
595=item $epipe->renew
596
597Recreates the pipe (useful after a fork). The reading side will not change
598it's file descriptor number, but the writing side might.
599
600=item $epipe->wait
601
602This method blocks the process until there are events on the pipe. This is
603not a very event-based or ncie way of usign an event pipe, but it can be
604occasionally useful.
605
606=back
607
194=cut 608=cut
195 609
1961; 6101;
197 611
198=back
199
200=head1 EXAMPLE
201
202There really should be a complete C/XS example. Bug me about it.
203
204=head1 IMPLEMENTATION DETAILS AND LIMITATIONS 612=head1 IMPLEMENTATION DETAILS AND LIMITATIONS
205 613
206This module works by "hijacking" SIGKILL, which is guaranteed to be always 614This module works by "hijacking" SIGKILL, which is guaranteed to always
207available in perl, but also cannot be caught, so is always available. 615exist, but also cannot be caught, so is always available.
208 616
209Basically, this module fakes the receive of a SIGKILL signal and 617Basically, this module fakes the occurance of a SIGKILL signal and
210then catches it. This makes normal signal handling slower (probably 618then intercepts the interpreter handling it. This makes normal signal
211unmeasurably), but has the advantage of not requiring a special runops nor 619handling slower (probably unmeasurably, though), but has the advantage
212slowing down normal perl execution a bit. 620of not requiring a special runops function, nor slowing down normal perl
621execution a bit.
213 622
214It assumes that C<sig_atomic_t> and C<int> are both exception-safe to 623It assumes that C<sig_atomic_t>, C<int> and C<IV> are all async-safe to
215modify (C<sig_atomic_> is used by this module, and perl itself uses 624modify.
216C<int>, so we can assume that this is quite portable, at least w.r.t.
217signals).
218 625
219=head1 AUTHOR 626=head1 AUTHOR
220 627
221 Marc Lehmann <schmorp@schmorp.de> 628 Marc Lehmann <schmorp@schmorp.de>
222 http://home.schmorp.de/ 629 http://home.schmorp.de/

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines