ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-Fork/Fork.pm
(Generate patch)

Comparing AnyEvent-Fork/Fork.pm (file contents):
Revision 1.26 by root, Sat Apr 6 08:58:51 2013 UTC vs.
Revision 1.41 by root, Mon Apr 8 03:20:53 2013 UTC

… …
27 27
28Special care has been taken to make this module useful from other modules, 28Special care has been taken to make this module useful from other modules,
29while still supporting specialised environments such as L<App::Staticperl> 29while still supporting specialised environments such as L<App::Staticperl>
30or L<PAR::Packer>. 30or L<PAR::Packer>.
31 31
32=head1 WHAT THIS MODULE IS NOT 32=head2 WHAT THIS MODULE IS NOT
33 33
34This module only creates processes and lets you pass file handles and 34This module only creates processes and lets you pass file handles and
35strings to it, and run perl code. It does not implement any kind of RPC - 35strings to it, and run perl code. It does not implement any kind of RPC -
36there is no back channel from the process back to you, and there is no RPC 36there is no back channel from the process back to you, and there is no RPC
37or message passing going on. 37or message passing going on.
… …
40in whatever way you like, use some message-passing module such 40in whatever way you like, use some message-passing module such
41as L<AnyEvent::MP>, some pipe such as L<AnyEvent::ZeroMQ>, use 41as L<AnyEvent::MP>, some pipe such as L<AnyEvent::ZeroMQ>, use
42L<AnyEvent::Handle> on both sides to send e.g. JSON or Storable messages, 42L<AnyEvent::Handle> on both sides to send e.g. JSON or Storable messages,
43and so on. 43and so on.
44 44
45=head2 COMPARISON TO OTHER MODULES
46
47There is an abundance of modules on CPAN that do "something fork", such as
48L<Parallel::ForkManager>, L<AnyEvent::ForkManager>, L<AnyEvent::Worker>
49or L<AnyEvent::Subprocess>. There are modules that implement their own
50process management, such as L<AnyEvent::DBI>.
51
52The problems that all these modules try to solve are real, however, none
53of them (from what I have seen) tackle the very real problems of unwanted
54memory sharing, efficiency, not being able to use event processing or
55similar modules in the processes they create.
56
57This module doesn't try to replace any of them - instead it tries to solve
58the problem of creating processes with a minimum of fuss and overhead (and
59also luxury). Ideally, most of these would use AnyEvent::Fork internally,
60except they were written before AnyEvent:Fork was available, so obviously
61had to roll their own.
62
45=head1 PROBLEM STATEMENT 63=head2 PROBLEM STATEMENT
46 64
47There are two traditional ways to implement parallel processing on UNIX 65There are two traditional ways to implement parallel processing on UNIX
48like operating systems - fork and process, and fork+exec and process. They 66like operating systems - fork and process, and fork+exec and process. They
49have different advantages and disadvantages that I describe below, 67have different advantages and disadvantages that I describe below,
50together with how this module tries to mitigate the disadvantages. 68together with how this module tries to mitigate the disadvantages.
… …
152 170
153 # now $master_filehandle is connected to the 171 # now $master_filehandle is connected to the
154 # $slave_filehandle in the new process. 172 # $slave_filehandle in the new process.
155 }); 173 });
156 174
157 # MyModule::worker might look like this 175C<MyModule> might look like this:
176
177 package MyModule;
178
158 sub MyModule::worker { 179 sub worker {
159 my ($slave_filehandle) = @_; 180 my ($slave_filehandle) = @_;
160 181
161 # now $slave_filehandle is connected to the $master_filehandle 182 # now $slave_filehandle is connected to the $master_filehandle
162 # in the original prorcess. have fun! 183 # in the original prorcess. have fun!
163 } 184 }
… …
182 } 203 }
183 204
184 # now do other things - maybe use the filehandle provided by run 205 # now do other things - maybe use the filehandle provided by run
185 # to wait for the processes to die. or whatever. 206 # to wait for the processes to die. or whatever.
186 207
187 # My::Server::run might look like this 208C<My::Server> might look like this:
188 sub My::Server::run { 209
210 package My::Server;
211
212 sub run {
189 my ($slave, $listener, $id) = @_; 213 my ($slave, $listener, $id) = @_;
190 214
191 close $slave; # we do not use the socket, so close it to save resources 215 close $slave; # we do not use the socket, so close it to save resources
192 216
193 # we could go ballistic and use e.g. AnyEvent here, or IO::AIO, 217 # we could go ballistic and use e.g. AnyEvent here, or IO::AIO,
… …
197 } 221 }
198 } 222 }
199 223
200=head2 use AnyEvent::Fork as a faster fork+exec 224=head2 use AnyEvent::Fork as a faster fork+exec
201 225
202This runs /bin/echo hi, with stdout redirected to /tmp/log and stderr to 226This runs C</bin/echo hi>, with stdandard output redirected to /tmp/log
203the communications socket. It is usually faster than fork+exec, but still 227and standard error redirected to the communications socket. It is usually
204let's you prepare the environment. 228faster than fork+exec, but still lets you prepare the environment.
205 229
206 open my $output, ">/tmp/log" or die "$!"; 230 open my $output, ">/tmp/log" or die "$!";
207 231
208 AnyEvent::Fork 232 AnyEvent::Fork
209 ->new 233 ->new
210 ->eval (' 234 ->eval ('
235 # compile a helper function for later use
211 sub run { 236 sub run {
212 my ($fh, $output, @cmd) = @_; 237 my ($fh, $output, @cmd) = @_;
213 238
214 # perl will clear close-on-exec on STDOUT/STDERR 239 # perl will clear close-on-exec on STDOUT/STDERR
215 open STDOUT, ">&", $output or die; 240 open STDOUT, ">&", $output or die;
… …
309 my ($fork_fh) = @_; 334 my ($fork_fh) = @_;
310 }); 335 });
311 336
312=back 337=back
313 338
314=head1 FUNCTIONS 339=head1 THE C<AnyEvent::Fork> CLASS
340
341This module exports nothing, and only implements a single class -
342C<AnyEvent::Fork>.
343
344There are two class constructors that both create new processes - C<new>
345and C<new_exec>. The C<fork> method creates a new process by forking an
346existing one and could be considered a third constructor.
347
348Most of the remaining methods deal with preparing the new process, by
349loading code, evaluating code and sending data to the new process. They
350usually return the process object, so you can chain method calls.
351
352If a process object is destroyed before calling its C<run> method, then
353the process simply exits. After C<run> is called, all responsibility is
354passed to the specified function.
355
356As long as there is any outstanding work to be done, process objects
357resist being destroyed, so there is no reason to store them unless you
358need them later - configure and forget works just fine.
315 359
316=over 4 360=over 4
317 361
318=cut 362=cut
319 363
… …
326use AnyEvent; 370use AnyEvent;
327use AnyEvent::Util (); 371use AnyEvent::Util ();
328 372
329use IO::FDPass; 373use IO::FDPass;
330 374
331our $VERSION = 0.5; 375our $VERSION = 0.6;
332
333our $PERL; # the path to the perl interpreter, deduces with various forms of magic
334
335=item my $pool = new AnyEvent::Fork key => value...
336
337Create a new process pool. The following named parameters are supported:
338 376
339=over 4 377=over 4
340 378
341=back 379=back
342 380
… …
421 if ($pid eq 0) { 459 if ($pid eq 0) {
422 require AnyEvent::Fork::Serve; 460 require AnyEvent::Fork::Serve;
423 $AnyEvent::Fork::Serve::OWNER = $parent; 461 $AnyEvent::Fork::Serve::OWNER = $parent;
424 close $fh; 462 close $fh;
425 $0 = "$_[1] of $parent"; 463 $0 = "$_[1] of $parent";
426 $SIG{CHLD} = 'IGNORE';
427 AnyEvent::Fork::Serve::serve ($slave); 464 AnyEvent::Fork::Serve::serve ($slave);
428 exit 0; 465 exit 0;
429 } elsif (!$pid) { 466 } elsif (!$pid) {
430 die "AnyEvent::Fork::Early/Template: unable to fork template process: $!"; 467 die "AnyEvent::Fork::Early/Template: unable to fork template process: $!";
431 } 468 }
… …
438Create a new "empty" perl interpreter process and returns its process 475Create a new "empty" perl interpreter process and returns its process
439object for further manipulation. 476object for further manipulation.
440 477
441The new process is forked from a template process that is kept around 478The new process is forked from a template process that is kept around
442for this purpose. When it doesn't exist yet, it is created by a call to 479for this purpose. When it doesn't exist yet, it is created by a call to
443C<new_exec> and kept around for future calls. 480C<new_exec> first and then stays around for future calls.
444
445When the process object is destroyed, it will release the file handle
446that connects it with the new process. When the new process has not yet
447called C<run>, then the process will exit. Otherwise, what happens depends
448entirely on the code that is executed.
449 481
450=cut 482=cut
451 483
452sub new { 484sub new {
453 my $class = shift; 485 my $class = shift;
… …
543} 575}
544 576
545=item $pid = $proc->pid 577=item $pid = $proc->pid
546 578
547Returns the process id of the process I<iff it is a direct child of the 579Returns the process id of the process I<iff it is a direct child of the
548process> running AnyEvent::Fork, and C<undef> otherwise. 580process running AnyEvent::Fork>, and C<undef> otherwise.
549 581
550Normally, only processes created via C<< AnyEvent::Fork->new_exec >> and 582Normally, only processes created via C<< AnyEvent::Fork->new_exec >> and
551L<AnyEvent::Fork::Template> are direct children, and you are responsible 583L<AnyEvent::Fork::Template> are direct children, and you are responsible
552to clean up their zombies when they die. 584to clean up their zombies when they die.
553 585
554All other processes are not direct children, and will be cleaned up by 586All other processes are not direct children, and will be cleaned up by
555AnyEvent::Fork. 587AnyEvent::Fork itself.
556 588
557=cut 589=cut
558 590
559sub pid { 591sub pid {
560 $_[0][0] 592 $_[0][0]
… …
571 603
572The code will usually be executed after this call returns, and there is no 604The code will usually be executed after this call returns, and there is no
573way to pass anything back to the calling process. Any evaluation errors 605way to pass anything back to the calling process. Any evaluation errors
574will be reported to stderr and cause the process to exit. 606will be reported to stderr and cause the process to exit.
575 607
576If you want to execute some code to take over the process (see the 608If you want to execute some code (that isn't in a module) to take over the
577"fork+exec" example in the SYNOPSIS), you should compile a function via 609process, you should compile a function via C<eval> first, and then call
578C<eval> first, and then call it via C<run>. This also gives you access to 610it via C<run>. This also gives you access to any arguments passed via the
579any arguments passed via the C<send_xxx> methods, such as file handles. 611C<send_xxx> methods, such as file handles. See the L<use AnyEvent::Fork as
612a faster fork+exec> example to see it in action.
580 613
581Returns the process object for easy chaining of method calls. 614Returns the process object for easy chaining of method calls.
582 615
583=cut 616=cut
584 617
… …
610=item $proc = $proc->send_fh ($handle, ...) 643=item $proc = $proc->send_fh ($handle, ...)
611 644
612Send one or more file handles (I<not> file descriptors) to the process, 645Send one or more file handles (I<not> file descriptors) to the process,
613to prepare a call to C<run>. 646to prepare a call to C<run>.
614 647
615The process object keeps a reference to the handles until this is done, 648The process object keeps a reference to the handles until they have
616so you must not explicitly close the handles. This is most easily 649been passed over to the process, so you must not explicitly close the
617accomplished by simply not storing the file handles anywhere after passing 650handles. This is most easily accomplished by simply not storing the file
618them to this method. 651handles anywhere after passing them to this method - when AnyEvent::Fork
652is finished using them, perl will automatically close them.
619 653
620Returns the process object for easy chaining of method calls. 654Returns the process object for easy chaining of method calls.
621 655
622Example: pass a file handle to a process, and release it without 656Example: pass a file handle to a process, and release it without
623closing. It will be closed automatically when it is no longer used. 657closing. It will be closed automatically when it is no longer used.
… …
639} 673}
640 674
641=item $proc = $proc->send_arg ($string, ...) 675=item $proc = $proc->send_arg ($string, ...)
642 676
643Send one or more argument strings to the process, to prepare a call to 677Send one or more argument strings to the process, to prepare a call to
644C<run>. The strings can be any octet string. 678C<run>. The strings can be any octet strings.
645 679
646The protocol is optimised to pass a moderate number of relatively short 680The protocol is optimised to pass a moderate number of relatively short
647strings - while you can pass up to 4GB of data in one go, this is more 681strings - while you can pass up to 4GB of data in one go, this is more
648meant to pass some ID information or other startup info, not big chunks of 682meant to pass some ID information or other startup info, not big chunks of
649data. 683data.
… …
665Enter the function specified by the function name in C<$func> in the 699Enter the function specified by the function name in C<$func> in the
666process. The function is called with the communication socket as first 700process. The function is called with the communication socket as first
667argument, followed by all file handles and string arguments sent earlier 701argument, followed by all file handles and string arguments sent earlier
668via C<send_fh> and C<send_arg> methods, in the order they were called. 702via C<send_fh> and C<send_arg> methods, in the order they were called.
669 703
704The process object becomes unusable on return from this function - any
705further method calls result in undefined behaviour.
706
670The function name should be fully qualified, but if it isn't, it will be 707The function name should be fully qualified, but if it isn't, it will be
671looked up in the main package. 708looked up in the C<main> package.
672 709
673If the called function returns, doesn't exist, or any error occurs, the 710If the called function returns, doesn't exist, or any error occurs, the
674process exits. 711process exits.
675 712
676Preparing the process is done in the background - when all commands have 713Preparing the process is done in the background - when all commands have
677been sent, the callback is invoked with the local communications socket 714been sent, the callback is invoked with the local communications socket
678as argument. At this point you can start using the socket in any way you 715as argument. At this point you can start using the socket in any way you
679like. 716like.
680
681The process object becomes unusable on return from this function - any
682further method calls result in undefined behaviour.
683 717
684If the communication socket isn't used, it should be closed on both sides, 718If the communication socket isn't used, it should be closed on both sides,
685to save on kernel memory. 719to save on kernel memory.
686 720
687The socket is non-blocking in the parent, and blocking in the newly 721The socket is non-blocking in the parent, and blocking in the newly
… …
762 479 vfork+execs per second, using AnyEvent::Fork->new_exec 796 479 vfork+execs per second, using AnyEvent::Fork->new_exec
763 797
764So how can C<< AnyEvent->new >> be faster than a standard fork, even 798So how can C<< AnyEvent->new >> be faster than a standard fork, even
765though it uses the same operations, but adds a lot of overhead? 799though it uses the same operations, but adds a lot of overhead?
766 800
767The difference is simply the process size: forking the 6MB process takes 801The difference is simply the process size: forking the 5MB process takes
768so much longer than forking the 2.5MB template process that the overhead 802so much longer than forking the 2.5MB template process that the extra
769introduced is canceled out. 803overhead introduced is canceled out.
770 804
771If the benchmark process grows, the normal fork becomes even slower: 805If the benchmark process grows, the normal fork becomes even slower:
772 806
773 1340 new processes, manual fork in a 20MB process 807 1340 new processes, manual fork of a 20MB process
774 731 new processes, manual fork in a 200MB process 808 731 new processes, manual fork of a 200MB process
775 235 new processes, manual fork in a 2000MB process 809 235 new processes, manual fork of a 2000MB process
776 810
777What that means (to me) is that I can use this module without having a 811What that means (to me) is that I can use this module without having a bad
778very bad conscience because of the extra overhead required to start new 812conscience because of the extra overhead required to start new processes.
779processes.
780 813
781=head1 TYPICAL PROBLEMS 814=head1 TYPICAL PROBLEMS
782 815
783This section lists typical problems that remain. I hope by recognising 816This section lists typical problems that remain. I hope by recognising
784them, most can be avoided. 817them, most can be avoided.
785 818
786=over 4 819=over 4
787 820
788=item "leaked" file descriptors for exec'ed processes 821=item leaked file descriptors for exec'ed processes
789 822
790POSIX systems inherit file descriptors by default when exec'ing a new 823POSIX systems inherit file descriptors by default when exec'ing a new
791process. While perl itself laudably sets the close-on-exec flags on new 824process. While perl itself laudably sets the close-on-exec flags on new
792file handles, most C libraries don't care, and even if all cared, it's 825file handles, most C libraries don't care, and even if all cared, it's
793often not possible to set the flag in a race-free manner. 826often not possible to set the flag in a race-free manner.
… …
813libraries or the code that leaks those file descriptors. 846libraries or the code that leaks those file descriptors.
814 847
815Fortunately, most of these leaked descriptors do no harm, other than 848Fortunately, most of these leaked descriptors do no harm, other than
816sitting on some resources. 849sitting on some resources.
817 850
818=item "leaked" file descriptors for fork'ed processes 851=item leaked file descriptors for fork'ed processes
819 852
820Normally, L<AnyEvent::Fork> does start new processes by exec'ing them, 853Normally, L<AnyEvent::Fork> does start new processes by exec'ing them,
821which closes file descriptors not marked for being inherited. 854which closes file descriptors not marked for being inherited.
822 855
823However, L<AnyEvent::Fork::Early> and L<AnyEvent::Fork::Template> offer 856However, L<AnyEvent::Fork::Early> and L<AnyEvent::Fork::Template> offer
… …
832 865
833The solution is to either not load these modules before use'ing 866The solution is to either not load these modules before use'ing
834L<AnyEvent::Fork::Early> or L<AnyEvent::Fork::Template>, or to delay 867L<AnyEvent::Fork::Early> or L<AnyEvent::Fork::Template>, or to delay
835initialising them, for example, by calling C<init Gtk2> manually. 868initialising them, for example, by calling C<init Gtk2> manually.
836 869
837=item exit runs destructors 870=item exiting calls object destructors
838 871
839This only applies to users of Lc<AnyEvent::Fork:Early> and 872This only applies to users of L<AnyEvent::Fork:Early> and
840L<AnyEvent::Fork::Template>. 873L<AnyEvent::Fork::Template>, or when initialiasing code creates objects
874that reference external resources.
841 875
842When a process created by AnyEvent::Fork exits, it might do so by calling 876When a process created by AnyEvent::Fork exits, it might do so by calling
843exit, or simply letting perl reach the end of the program. At which point 877exit, or simply letting perl reach the end of the program. At which point
844Perl runs all destructors. 878Perl runs all destructors.
845 879
… …
864to make it so, mostly due to the bloody broken perl that nobody seems to 898to make it so, mostly due to the bloody broken perl that nobody seems to
865care about. The fork emulation is a bad joke - I have yet to see something 899care about. The fork emulation is a bad joke - I have yet to see something
866useful that you can do with it without running into memory corruption 900useful that you can do with it without running into memory corruption
867issues or other braindamage. Hrrrr. 901issues or other braindamage. Hrrrr.
868 902
869Cygwin perl is not supported at the moment, as it should implement fd 903Cygwin perl is not supported at the moment due to some hilarious
870passing, but doesn't, and rolling my own is hard, as cygwin doesn't 904shortcomings of its API - see L<IO::FDPoll> for more details.
871support enough functionality to do it.
872 905
873=head1 SEE ALSO 906=head1 SEE ALSO
874 907
875L<AnyEvent::Fork::Early> (to avoid executing a perl interpreter), 908L<AnyEvent::Fork::Early> (to avoid executing a perl interpreter),
876L<AnyEvent::Fork::Template> (to create a process by forking the main 909L<AnyEvent::Fork::Template> (to create a process by forking the main

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines