ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-Fork/Fork.pm
(Generate patch)

Comparing AnyEvent-Fork/Fork.pm (file contents):
Revision 1.29 by root, Sat Apr 6 09:15:49 2013 UTC vs.
Revision 1.44 by root, Thu Apr 18 10:49:59 2013 UTC

… …
27 27
28Special care has been taken to make this module useful from other modules, 28Special care has been taken to make this module useful from other modules,
29while still supporting specialised environments such as L<App::Staticperl> 29while still supporting specialised environments such as L<App::Staticperl>
30or L<PAR::Packer>. 30or L<PAR::Packer>.
31 31
32=head1 WHAT THIS MODULE IS NOT 32=head2 WHAT THIS MODULE IS NOT
33 33
34This module only creates processes and lets you pass file handles and 34This module only creates processes and lets you pass file handles and
35strings to it, and run perl code. It does not implement any kind of RPC - 35strings to it, and run perl code. It does not implement any kind of RPC -
36there is no back channel from the process back to you, and there is no RPC 36there is no back channel from the process back to you, and there is no RPC
37or message passing going on. 37or message passing going on.
38 38
39If you need some form of RPC, you can either implement it yourself 39If you need some form of RPC, you could use the L<AnyEvent::Fork::RPC>
40in whatever way you like, use some message-passing module such 40companion module, which adds simple RPC/job queueing to a process created
41as L<AnyEvent::MP>, some pipe such as L<AnyEvent::ZeroMQ>, use 41by this module.
42L<AnyEvent::Handle> on both sides to send e.g. JSON or Storable messages,
43and so on.
44 42
43Or you can implement it yourself in whatever way you like, use some
44message-passing module such as L<AnyEvent::MP>, some pipe such as
45L<AnyEvent::ZeroMQ>, use L<AnyEvent::Handle> on both sides to send
46e.g. JSON or Storable messages, and so on.
47
48=head2 COMPARISON TO OTHER MODULES
49
50There is an abundance of modules on CPAN that do "something fork", such as
51L<Parallel::ForkManager>, L<AnyEvent::ForkManager>, L<AnyEvent::Worker>
52or L<AnyEvent::Subprocess>. There are modules that implement their own
53process management, such as L<AnyEvent::DBI>.
54
55The problems that all these modules try to solve are real, however, none
56of them (from what I have seen) tackle the very real problems of unwanted
57memory sharing, efficiency, not being able to use event processing or
58similar modules in the processes they create.
59
60This module doesn't try to replace any of them - instead it tries to solve
61the problem of creating processes with a minimum of fuss and overhead (and
62also luxury). Ideally, most of these would use AnyEvent::Fork internally,
63except they were written before AnyEvent:Fork was available, so obviously
64had to roll their own.
65
45=head1 PROBLEM STATEMENT 66=head2 PROBLEM STATEMENT
46 67
47There are two traditional ways to implement parallel processing on UNIX 68There are two traditional ways to implement parallel processing on UNIX
48like operating systems - fork and process, and fork+exec and process. They 69like operating systems - fork and process, and fork+exec and process. They
49have different advantages and disadvantages that I describe below, 70have different advantages and disadvantages that I describe below,
50together with how this module tries to mitigate the disadvantages. 71together with how this module tries to mitigate the disadvantages.
… …
152 173
153 # now $master_filehandle is connected to the 174 # now $master_filehandle is connected to the
154 # $slave_filehandle in the new process. 175 # $slave_filehandle in the new process.
155 }); 176 });
156 177
157 # MyModule::worker might look like this 178C<MyModule> might look like this:
179
180 package MyModule;
181
158 sub MyModule::worker { 182 sub worker {
159 my ($slave_filehandle) = @_; 183 my ($slave_filehandle) = @_;
160 184
161 # now $slave_filehandle is connected to the $master_filehandle 185 # now $slave_filehandle is connected to the $master_filehandle
162 # in the original prorcess. have fun! 186 # in the original prorcess. have fun!
163 } 187 }
… …
182 } 206 }
183 207
184 # now do other things - maybe use the filehandle provided by run 208 # now do other things - maybe use the filehandle provided by run
185 # to wait for the processes to die. or whatever. 209 # to wait for the processes to die. or whatever.
186 210
187 # My::Server::run might look like this 211C<My::Server> might look like this:
188 sub My::Server::run { 212
213 package My::Server;
214
215 sub run {
189 my ($slave, $listener, $id) = @_; 216 my ($slave, $listener, $id) = @_;
190 217
191 close $slave; # we do not use the socket, so close it to save resources 218 close $slave; # we do not use the socket, so close it to save resources
192 219
193 # we could go ballistic and use e.g. AnyEvent here, or IO::AIO, 220 # we could go ballistic and use e.g. AnyEvent here, or IO::AIO,
… …
197 } 224 }
198 } 225 }
199 226
200=head2 use AnyEvent::Fork as a faster fork+exec 227=head2 use AnyEvent::Fork as a faster fork+exec
201 228
202This runs /bin/echo hi, with stdout redirected to /tmp/log and stderr to 229This runs C</bin/echo hi>, with standard output redirected to F</tmp/log>
203the communications socket. It is usually faster than fork+exec, but still 230and standard error redirected to the communications socket. It is usually
204let's you prepare the environment. 231faster than fork+exec, but still lets you prepare the environment.
205 232
206 open my $output, ">/tmp/log" or die "$!"; 233 open my $output, ">/tmp/log" or die "$!";
207 234
208 AnyEvent::Fork 235 AnyEvent::Fork
209 ->new 236 ->new
210 ->eval (' 237 ->eval ('
238 # compile a helper function for later use
211 sub run { 239 sub run {
212 my ($fh, $output, @cmd) = @_; 240 my ($fh, $output, @cmd) = @_;
213 241
214 # perl will clear close-on-exec on STDOUT/STDERR 242 # perl will clear close-on-exec on STDOUT/STDERR
215 open STDOUT, ">&", $output or die; 243 open STDOUT, ">&", $output or die;
… …
345use AnyEvent; 373use AnyEvent;
346use AnyEvent::Util (); 374use AnyEvent::Util ();
347 375
348use IO::FDPass; 376use IO::FDPass;
349 377
350our $VERSION = 0.5; 378our $VERSION = 0.6;
351
352our $PERL; # the path to the perl interpreter, deduces with various forms of magic
353
354=over 4
355
356=back
357
358=cut
359 379
360# the early fork template process 380# the early fork template process
361our $EARLY; 381our $EARLY;
362 382
363# the empty template process 383# the empty template process
364our $TEMPLATE; 384our $TEMPLATE;
385
386sub QUEUE() { 0 }
387sub FH() { 1 }
388sub WW() { 2 }
389sub PID() { 3 }
390sub CB() { 4 }
391
392sub _new {
393 my ($self, $fh, $pid) = @_;
394
395 AnyEvent::Util::fh_nonblocking $fh, 1;
396
397 $self = bless [
398 [], # write queue - strings or fd's
399 $fh,
400 undef, # AE watcher
401 $pid,
402 ], $self;
403
404 $self
405}
365 406
366sub _cmd { 407sub _cmd {
367 my $self = shift; 408 my $self = shift;
368 409
369 # ideally, we would want to use "a (w/a)*" as format string, but perl 410 # ideally, we would want to use "a (w/a)*" as format string, but perl
370 # versions from at least 5.8.9 to 5.16.3 are all buggy and can't unpack 411 # versions from at least 5.8.9 to 5.16.3 are all buggy and can't unpack
371 # it. 412 # it.
372 push @{ $self->[2] }, pack "a L/a*", $_[0], $_[1]; 413 push @{ $self->[QUEUE] }, pack "a L/a*", $_[0], $_[1];
373 414
374 $self->[3] ||= AE::io $self->[1], 1, sub { 415 $self->[WW] ||= AE::io $self->[FH], 1, sub {
375 do { 416 do {
376 # send the next "thing" in the queue - either a reference to an fh, 417 # send the next "thing" in the queue - either a reference to an fh,
377 # or a plain string. 418 # or a plain string.
378 419
379 if (ref $self->[2][0]) { 420 if (ref $self->[QUEUE][0]) {
380 # send fh 421 # send fh
381 unless (IO::FDPass::send fileno $self->[1], fileno ${ $self->[2][0] }) { 422 unless (IO::FDPass::send fileno $self->[FH], fileno ${ $self->[QUEUE][0] }) {
382 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK; 423 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK;
383 undef $self->[3]; 424 undef $self->[WW];
384 die "AnyEvent::Fork: file descriptor send failure: $!"; 425 die "AnyEvent::Fork: file descriptor send failure: $!";
385 } 426 }
386 427
387 shift @{ $self->[2] }; 428 shift @{ $self->[QUEUE] };
388 429
389 } else { 430 } else {
390 # send string 431 # send string
391 my $len = syswrite $self->[1], $self->[2][0]; 432 my $len = syswrite $self->[FH], $self->[QUEUE][0];
392 433
393 unless ($len) { 434 unless ($len) {
394 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK; 435 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK;
395 undef $self->[3]; 436 undef $self->[3];
396 die "AnyEvent::Fork: command write failure: $!"; 437 die "AnyEvent::Fork: command write failure: $!";
397 } 438 }
398 439
399 substr $self->[2][0], 0, $len, ""; 440 substr $self->[QUEUE][0], 0, $len, "";
400 shift @{ $self->[2] } unless length $self->[2][0]; 441 shift @{ $self->[QUEUE] } unless length $self->[QUEUE][0];
401 } 442 }
402 } while @{ $self->[2] }; 443 } while @{ $self->[QUEUE] };
403 444
404 # everything written 445 # everything written
405 undef $self->[3]; 446 undef $self->[WW];
406 447
407 # invoke run callback, if any 448 # invoke run callback, if any
408 $self->[4]->($self->[1]) if $self->[4]; 449 $self->[CB]->($self->[FH]) if $self->[CB];
409 }; 450 };
410 451
411 () # make sure we don't leak the watcher 452 () # make sure we don't leak the watcher
412}
413
414sub _new {
415 my ($self, $fh, $pid) = @_;
416
417 AnyEvent::Util::fh_nonblocking $fh, 1;
418
419 $self = bless [
420 $pid,
421 $fh,
422 [], # write queue - strings or fd's
423 undef, # AE watcher
424 ], $self;
425
426 $self
427} 453}
428 454
429# fork template from current process, used by AnyEvent::Fork::Early/Template 455# fork template from current process, used by AnyEvent::Fork::Early/Template
430sub _new_fork { 456sub _new_fork {
431 my ($fh, $slave) = AnyEvent::Util::portable_socketpair; 457 my ($fh, $slave) = AnyEvent::Util::portable_socketpair;
… …
436 if ($pid eq 0) { 462 if ($pid eq 0) {
437 require AnyEvent::Fork::Serve; 463 require AnyEvent::Fork::Serve;
438 $AnyEvent::Fork::Serve::OWNER = $parent; 464 $AnyEvent::Fork::Serve::OWNER = $parent;
439 close $fh; 465 close $fh;
440 $0 = "$_[1] of $parent"; 466 $0 = "$_[1] of $parent";
441 $SIG{CHLD} = 'IGNORE';
442 AnyEvent::Fork::Serve::serve ($slave); 467 AnyEvent::Fork::Serve::serve ($slave);
443 exit 0; 468 exit 0;
444 } elsif (!$pid) { 469 } elsif (!$pid) {
445 die "AnyEvent::Fork::Early/Template: unable to fork template process: $!"; 470 die "AnyEvent::Fork::Early/Template: unable to fork template process: $!";
446 } 471 }
… …
553} 578}
554 579
555=item $pid = $proc->pid 580=item $pid = $proc->pid
556 581
557Returns the process id of the process I<iff it is a direct child of the 582Returns the process id of the process I<iff it is a direct child of the
558process> running AnyEvent::Fork, and C<undef> otherwise. 583process running AnyEvent::Fork>, and C<undef> otherwise.
559 584
560Normally, only processes created via C<< AnyEvent::Fork->new_exec >> and 585Normally, only processes created via C<< AnyEvent::Fork->new_exec >> and
561L<AnyEvent::Fork::Template> are direct children, and you are responsible 586L<AnyEvent::Fork::Template> are direct children, and you are responsible
562to clean up their zombies when they die. 587to clean up their zombies when they die.
563 588
564All other processes are not direct children, and will be cleaned up by 589All other processes are not direct children, and will be cleaned up by
565AnyEvent::Fork. 590AnyEvent::Fork itself.
566 591
567=cut 592=cut
568 593
569sub pid { 594sub pid {
570 $_[0][0] 595 $_[0][PID]
571} 596}
572 597
573=item $proc = $proc->eval ($perlcode, @args) 598=item $proc = $proc->eval ($perlcode, @args)
574 599
575Evaluates the given C<$perlcode> as ... perl code, while setting C<@_> to 600Evaluates the given C<$perlcode> as ... Perl code, while setting C<@_> to
576the strings specified by C<@args>, in the "main" package. 601the strings specified by C<@args>, in the "main" package.
577 602
578This call is meant to do any custom initialisation that might be required 603This call is meant to do any custom initialisation that might be required
579(for example, the C<require> method uses it). It's not supposed to be used 604(for example, the C<require> method uses it). It's not supposed to be used
580to completely take over the process, use C<run> for that. 605to completely take over the process, use C<run> for that.
581 606
582The code will usually be executed after this call returns, and there is no 607The code will usually be executed after this call returns, and there is no
583way to pass anything back to the calling process. Any evaluation errors 608way to pass anything back to the calling process. Any evaluation errors
584will be reported to stderr and cause the process to exit. 609will be reported to stderr and cause the process to exit.
585 610
586If you want to execute some code to take over the process (see the 611If you want to execute some code (that isn't in a module) to take over the
587"fork+exec" example in the SYNOPSIS), you should compile a function via 612process, you should compile a function via C<eval> first, and then call
588C<eval> first, and then call it via C<run>. This also gives you access to 613it via C<run>. This also gives you access to any arguments passed via the
589any arguments passed via the C<send_xxx> methods, such as file handles. 614C<send_xxx> methods, such as file handles. See the L<use AnyEvent::Fork as
615a faster fork+exec> example to see it in action.
590 616
591Returns the process object for easy chaining of method calls. 617Returns the process object for easy chaining of method calls.
592 618
593=cut 619=cut
594 620
… …
620=item $proc = $proc->send_fh ($handle, ...) 646=item $proc = $proc->send_fh ($handle, ...)
621 647
622Send one or more file handles (I<not> file descriptors) to the process, 648Send one or more file handles (I<not> file descriptors) to the process,
623to prepare a call to C<run>. 649to prepare a call to C<run>.
624 650
625The process object keeps a reference to the handles until this is done, 651The process object keeps a reference to the handles until they have
626so you must not explicitly close the handles. This is most easily 652been passed over to the process, so you must not explicitly close the
627accomplished by simply not storing the file handles anywhere after passing 653handles. This is most easily accomplished by simply not storing the file
628them to this method. 654handles anywhere after passing them to this method - when AnyEvent::Fork
655is finished using them, perl will automatically close them.
629 656
630Returns the process object for easy chaining of method calls. 657Returns the process object for easy chaining of method calls.
631 658
632Example: pass a file handle to a process, and release it without 659Example: pass a file handle to a process, and release it without
633closing. It will be closed automatically when it is no longer used. 660closing. It will be closed automatically when it is no longer used.
… …
640sub send_fh { 667sub send_fh {
641 my ($self, @fh) = @_; 668 my ($self, @fh) = @_;
642 669
643 for my $fh (@fh) { 670 for my $fh (@fh) {
644 $self->_cmd ("h"); 671 $self->_cmd ("h");
645 push @{ $self->[2] }, \$fh; 672 push @{ $self->[QUEUE] }, \$fh;
646 } 673 }
647 674
648 $self 675 $self
649} 676}
650 677
651=item $proc = $proc->send_arg ($string, ...) 678=item $proc = $proc->send_arg ($string, ...)
652 679
653Send one or more argument strings to the process, to prepare a call to 680Send one or more argument strings to the process, to prepare a call to
654C<run>. The strings can be any octet string. 681C<run>. The strings can be any octet strings.
655 682
656The protocol is optimised to pass a moderate number of relatively short 683The protocol is optimised to pass a moderate number of relatively short
657strings - while you can pass up to 4GB of data in one go, this is more 684strings - while you can pass up to 4GB of data in one go, this is more
658meant to pass some ID information or other startup info, not big chunks of 685meant to pass some ID information or other startup info, not big chunks of
659data. 686data.
… …
675Enter the function specified by the function name in C<$func> in the 702Enter the function specified by the function name in C<$func> in the
676process. The function is called with the communication socket as first 703process. The function is called with the communication socket as first
677argument, followed by all file handles and string arguments sent earlier 704argument, followed by all file handles and string arguments sent earlier
678via C<send_fh> and C<send_arg> methods, in the order they were called. 705via C<send_fh> and C<send_arg> methods, in the order they were called.
679 706
707The process object becomes unusable on return from this function - any
708further method calls result in undefined behaviour.
709
680The function name should be fully qualified, but if it isn't, it will be 710The function name should be fully qualified, but if it isn't, it will be
681looked up in the main package. 711looked up in the C<main> package.
682 712
683If the called function returns, doesn't exist, or any error occurs, the 713If the called function returns, doesn't exist, or any error occurs, the
684process exits. 714process exits.
685 715
686Preparing the process is done in the background - when all commands have 716Preparing the process is done in the background - when all commands have
687been sent, the callback is invoked with the local communications socket 717been sent, the callback is invoked with the local communications socket
688as argument. At this point you can start using the socket in any way you 718as argument. At this point you can start using the socket in any way you
689like. 719like.
690
691The process object becomes unusable on return from this function - any
692further method calls result in undefined behaviour.
693 720
694If the communication socket isn't used, it should be closed on both sides, 721If the communication socket isn't used, it should be closed on both sides,
695to save on kernel memory. 722to save on kernel memory.
696 723
697The socket is non-blocking in the parent, and blocking in the newly 724The socket is non-blocking in the parent, and blocking in the newly
… …
736=cut 763=cut
737 764
738sub run { 765sub run {
739 my ($self, $func, $cb) = @_; 766 my ($self, $func, $cb) = @_;
740 767
741 $self->[4] = $cb; 768 $self->[CB] = $cb;
742 $self->_cmd (r => $func); 769 $self->_cmd (r => $func);
743} 770}
744 771
745=back 772=back
746 773
… …
772 479 vfork+execs per second, using AnyEvent::Fork->new_exec 799 479 vfork+execs per second, using AnyEvent::Fork->new_exec
773 800
774So how can C<< AnyEvent->new >> be faster than a standard fork, even 801So how can C<< AnyEvent->new >> be faster than a standard fork, even
775though it uses the same operations, but adds a lot of overhead? 802though it uses the same operations, but adds a lot of overhead?
776 803
777The difference is simply the process size: forking the 6MB process takes 804The difference is simply the process size: forking the 5MB process takes
778so much longer than forking the 2.5MB template process that the overhead 805so much longer than forking the 2.5MB template process that the extra
779introduced is canceled out. 806overhead is canceled out.
780 807
781If the benchmark process grows, the normal fork becomes even slower: 808If the benchmark process grows, the normal fork becomes even slower:
782 809
783 1340 new processes, manual fork in a 20MB process 810 1340 new processes, manual fork of a 20MB process
784 731 new processes, manual fork in a 200MB process 811 731 new processes, manual fork of a 200MB process
785 235 new processes, manual fork in a 2000MB process 812 235 new processes, manual fork of a 2000MB process
786 813
787What that means (to me) is that I can use this module without having a 814What that means (to me) is that I can use this module without having a bad
788very bad conscience because of the extra overhead required to start new 815conscience because of the extra overhead required to start new processes.
789processes.
790 816
791=head1 TYPICAL PROBLEMS 817=head1 TYPICAL PROBLEMS
792 818
793This section lists typical problems that remain. I hope by recognising 819This section lists typical problems that remain. I hope by recognising
794them, most can be avoided. 820them, most can be avoided.
795 821
796=over 4 822=over 4
797 823
798=item "leaked" file descriptors for exec'ed processes 824=item leaked file descriptors for exec'ed processes
799 825
800POSIX systems inherit file descriptors by default when exec'ing a new 826POSIX systems inherit file descriptors by default when exec'ing a new
801process. While perl itself laudably sets the close-on-exec flags on new 827process. While perl itself laudably sets the close-on-exec flags on new
802file handles, most C libraries don't care, and even if all cared, it's 828file handles, most C libraries don't care, and even if all cared, it's
803often not possible to set the flag in a race-free manner. 829often not possible to set the flag in a race-free manner.
… …
823libraries or the code that leaks those file descriptors. 849libraries or the code that leaks those file descriptors.
824 850
825Fortunately, most of these leaked descriptors do no harm, other than 851Fortunately, most of these leaked descriptors do no harm, other than
826sitting on some resources. 852sitting on some resources.
827 853
828=item "leaked" file descriptors for fork'ed processes 854=item leaked file descriptors for fork'ed processes
829 855
830Normally, L<AnyEvent::Fork> does start new processes by exec'ing them, 856Normally, L<AnyEvent::Fork> does start new processes by exec'ing them,
831which closes file descriptors not marked for being inherited. 857which closes file descriptors not marked for being inherited.
832 858
833However, L<AnyEvent::Fork::Early> and L<AnyEvent::Fork::Template> offer 859However, L<AnyEvent::Fork::Early> and L<AnyEvent::Fork::Template> offer
… …
842 868
843The solution is to either not load these modules before use'ing 869The solution is to either not load these modules before use'ing
844L<AnyEvent::Fork::Early> or L<AnyEvent::Fork::Template>, or to delay 870L<AnyEvent::Fork::Early> or L<AnyEvent::Fork::Template>, or to delay
845initialising them, for example, by calling C<init Gtk2> manually. 871initialising them, for example, by calling C<init Gtk2> manually.
846 872
847=item exit runs destructors 873=item exiting calls object destructors
848 874
849This only applies to users of Lc<AnyEvent::Fork:Early> and 875This only applies to users of L<AnyEvent::Fork:Early> and
850L<AnyEvent::Fork::Template>. 876L<AnyEvent::Fork::Template>, or when initialising code creates objects
877that reference external resources.
851 878
852When a process created by AnyEvent::Fork exits, it might do so by calling 879When a process created by AnyEvent::Fork exits, it might do so by calling
853exit, or simply letting perl reach the end of the program. At which point 880exit, or simply letting perl reach the end of the program. At which point
854Perl runs all destructors. 881Perl runs all destructors.
855 882
… …
874to make it so, mostly due to the bloody broken perl that nobody seems to 901to make it so, mostly due to the bloody broken perl that nobody seems to
875care about. The fork emulation is a bad joke - I have yet to see something 902care about. The fork emulation is a bad joke - I have yet to see something
876useful that you can do with it without running into memory corruption 903useful that you can do with it without running into memory corruption
877issues or other braindamage. Hrrrr. 904issues or other braindamage. Hrrrr.
878 905
879Cygwin perl is not supported at the moment, as it should implement fd 906Cygwin perl is not supported at the moment due to some hilarious
880passing, but doesn't, and rolling my own is hard, as cygwin doesn't 907shortcomings of its API - see L<IO::FDPoll> for more details.
881support enough functionality to do it.
882 908
883=head1 SEE ALSO 909=head1 SEE ALSO
884 910
885L<AnyEvent::Fork::Early> (to avoid executing a perl interpreter), 911L<AnyEvent::Fork::Early> (to avoid executing a perl interpreter),
886L<AnyEvent::Fork::Template> (to create a process by forking the main 912L<AnyEvent::Fork::Template> (to create a process by forking the main
887program at a convenient time). 913program at a convenient time), L<AnyEvent::Fork::RPC> (for simple RPC to
914child processes).
888 915
889=head1 AUTHOR 916=head1 AUTHOR AND CONTACT INFORMATION
890 917
891 Marc Lehmann <schmorp@schmorp.de> 918 Marc Lehmann <schmorp@schmorp.de>
892 http://home.schmorp.de/ 919 http://software.schmorp.de/pkg/AnyEvent-Fork
893 920
894=cut 921=cut
895 922
8961 9231
897 924

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines