ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-Fork/Fork.pm
(Generate patch)

Comparing AnyEvent-Fork/Fork.pm (file contents):
Revision 1.39 by root, Sat Apr 6 22:39:37 2013 UTC vs.
Revision 1.56 by root, Sun Apr 28 13:47:52 2013 UTC

34This module only creates processes and lets you pass file handles and 34This module only creates processes and lets you pass file handles and
35strings to it, and run perl code. It does not implement any kind of RPC - 35strings to it, and run perl code. It does not implement any kind of RPC -
36there is no back channel from the process back to you, and there is no RPC 36there is no back channel from the process back to you, and there is no RPC
37or message passing going on. 37or message passing going on.
38 38
39If you need some form of RPC, you can either implement it yourself 39If you need some form of RPC, you could use the L<AnyEvent::Fork::RPC>
40in whatever way you like, use some message-passing module such 40companion module, which adds simple RPC/job queueing to a process created
41as L<AnyEvent::MP>, some pipe such as L<AnyEvent::ZeroMQ>, use 41by this module.
42L<AnyEvent::Handle> on both sides to send e.g. JSON or Storable messages, 42
43and so on. 43And if you need some automatic process pool management on top of
44L<AnyEvent::Fork::RPC>, you can look at the L<AnyEvent::Fork::Pool>
45companion module.
46
47Or you can implement it yourself in whatever way you like: use some
48message-passing module such as L<AnyEvent::MP>, some pipe such as
49L<AnyEvent::ZeroMQ>, use L<AnyEvent::Handle> on both sides to send
50e.g. JSON or Storable messages, and so on.
44 51
45=head2 COMPARISON TO OTHER MODULES 52=head2 COMPARISON TO OTHER MODULES
46 53
47There is an abundance of modules on CPAN that do "something fork", such as 54There is an abundance of modules on CPAN that do "something fork", such as
48L<Parallel::ForkManager>, L<AnyEvent::ForkManager>, L<AnyEvent::Worker> 55L<Parallel::ForkManager>, L<AnyEvent::ForkManager>, L<AnyEvent::Worker>
221 } 228 }
222 } 229 }
223 230
224=head2 use AnyEvent::Fork as a faster fork+exec 231=head2 use AnyEvent::Fork as a faster fork+exec
225 232
226This runs C</bin/echo hi>, with stdandard output redirected to /tmp/log 233This runs C</bin/echo hi>, with standard output redirected to F</tmp/log>
227and standard error redirected to the communications socket. It is usually 234and standard error redirected to the communications socket. It is usually
228faster than fork+exec, but still lets you prepare the environment. 235faster than fork+exec, but still lets you prepare the environment.
229 236
230 open my $output, ">/tmp/log" or die "$!"; 237 open my $output, ">/tmp/log" or die "$!";
231 238
232 AnyEvent::Fork 239 AnyEvent::Fork
233 ->new 240 ->new
234 ->eval (' 241 ->eval ('
242 # compile a helper function for later use
235 sub run { 243 sub run {
236 my ($fh, $output, @cmd) = @_; 244 my ($fh, $output, @cmd) = @_;
237 245
238 # perl will clear close-on-exec on STDOUT/STDERR 246 # perl will clear close-on-exec on STDOUT/STDERR
239 open STDOUT, ">&", $output or die; 247 open STDOUT, ">&", $output or die;
246 ->send_arg ("/bin/echo", "hi") 254 ->send_arg ("/bin/echo", "hi")
247 ->run ("run", my $cv = AE::cv); 255 ->run ("run", my $cv = AE::cv);
248 256
249 my $stderr = $cv->recv; 257 my $stderr = $cv->recv;
250 258
259=head2 For stingy users: put the worker code into a C<DATA> section.
260
261When you want to be stingy with files, you cna put your code into the
262C<DATA> section of your module (or program):
263
264 use AnyEvent::Fork;
265
266 AnyEvent::Fork
267 ->new
268 ->eval (do { local $/; <DATA> })
269 ->run ("doit", sub { ... });
270
271 __DATA__
272
273 sub doit {
274 ... do something!
275 }
276
277=head2 For stingy standalone programs: do not rely on external files at
278all.
279
280For single-file scripts it can be inconvenient to rely on external
281files - even when using < C<DATA> section, you still need to C<exec>
282an external perl interpreter, which might not be available when using
283L<App::Staticperl>, L<Urlader> or L<PAR::Packer> for example.
284
285Two modules help here - L<AnyEvent::Fork::Early> forks a template process
286for all further calls to C<new_exec>, and L<AnyEvent::Fork::Template>
287forks the main program as a template process.
288
289Here is how your main program should look like:
290
291 #! perl
292
293 # optional, as the very first thing.
294 # in case modules want to create their own processes.
295 use AnyEvent::Fork::Early;
296
297 # next, load all modules you need in your template process
298 use Example::My::Module
299 use Example::Whatever;
300
301 # next, put your run function definition and anything else you
302 # need, but do not use code outside of BEGIN blocks.
303 sub worker_run {
304 my ($fh, @args) = @_;
305 ...
306 }
307
308 # now preserve everything so far as AnyEvent::Fork object
309 # in §TEMPLATE.
310 use AnyEvent::Fork::Template;
311
312 # do not put code outside of BEGIN blocks until here
313
314 # now use the $TEMPLATE process in any way you like
315
316 # for example: create 10 worker processes
317 my @worker;
318 my $cv = AE::cv;
319 for (1..10) {
320 $cv->begin;
321 $TEMPLATE->fork->send_arg ($_)->run ("worker_run", sub {
322 push @worker, shift;
323 $cv->end;
324 });
325 }
326 $cv->recv;
327
251=head1 CONCEPTS 328=head1 CONCEPTS
252 329
253This module can create new processes either by executing a new perl 330This module can create new processes either by executing a new perl
254process, or by forking from an existing "template" process. 331process, or by forking from an existing "template" process.
332
333All these processes are called "child processes" (whether they are direct
334children or not), while the process that manages them is called the
335"parent process".
255 336
256Each such process comes with its own file handle that can be used to 337Each such process comes with its own file handle that can be used to
257communicate with it (it's actually a socket - one end in the new process, 338communicate with it (it's actually a socket - one end in the new process,
258one end in the main process), and among the things you can do in it are 339one end in the main process), and among the things you can do in it are
259load modules, fork new processes, send file handles to it, and execute 340load modules, fork new processes, send file handles to it, and execute
369use AnyEvent; 450use AnyEvent;
370use AnyEvent::Util (); 451use AnyEvent::Util ();
371 452
372use IO::FDPass; 453use IO::FDPass;
373 454
374our $VERSION = 0.5; 455our $VERSION = 1.1;
375
376our $PERL; # the path to the perl interpreter, deduces with various forms of magic
377
378=over 4
379
380=back
381
382=cut
383 456
384# the early fork template process 457# the early fork template process
385our $EARLY; 458our $EARLY;
386 459
387# the empty template process 460# the empty template process
388our $TEMPLATE; 461our $TEMPLATE;
462
463sub QUEUE() { 0 }
464sub FH() { 1 }
465sub WW() { 2 }
466sub PID() { 3 }
467sub CB() { 4 }
468
469sub _new {
470 my ($self, $fh, $pid) = @_;
471
472 AnyEvent::Util::fh_nonblocking $fh, 1;
473
474 $self = bless [
475 [], # write queue - strings or fd's
476 $fh,
477 undef, # AE watcher
478 $pid,
479 ], $self;
480
481 $self
482}
389 483
390sub _cmd { 484sub _cmd {
391 my $self = shift; 485 my $self = shift;
392 486
393 # ideally, we would want to use "a (w/a)*" as format string, but perl 487 # ideally, we would want to use "a (w/a)*" as format string, but perl
394 # versions from at least 5.8.9 to 5.16.3 are all buggy and can't unpack 488 # versions from at least 5.8.9 to 5.16.3 are all buggy and can't unpack
395 # it. 489 # it.
396 push @{ $self->[2] }, pack "a L/a*", $_[0], $_[1]; 490 push @{ $self->[QUEUE] }, pack "a L/a*", $_[0], $_[1];
397 491
398 $self->[3] ||= AE::io $self->[1], 1, sub { 492 $self->[WW] ||= AE::io $self->[FH], 1, sub {
399 do { 493 do {
400 # send the next "thing" in the queue - either a reference to an fh, 494 # send the next "thing" in the queue - either a reference to an fh,
401 # or a plain string. 495 # or a plain string.
402 496
403 if (ref $self->[2][0]) { 497 if (ref $self->[QUEUE][0]) {
404 # send fh 498 # send fh
405 unless (IO::FDPass::send fileno $self->[1], fileno ${ $self->[2][0] }) { 499 unless (IO::FDPass::send fileno $self->[FH], fileno ${ $self->[QUEUE][0] }) {
406 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK; 500 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK;
407 undef $self->[3]; 501 undef $self->[WW];
408 die "AnyEvent::Fork: file descriptor send failure: $!"; 502 die "AnyEvent::Fork: file descriptor send failure: $!";
409 } 503 }
410 504
411 shift @{ $self->[2] }; 505 shift @{ $self->[QUEUE] };
412 506
413 } else { 507 } else {
414 # send string 508 # send string
415 my $len = syswrite $self->[1], $self->[2][0]; 509 my $len = syswrite $self->[FH], $self->[QUEUE][0];
416 510
417 unless ($len) { 511 unless ($len) {
418 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK; 512 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK;
419 undef $self->[3]; 513 undef $self->[WW];
420 die "AnyEvent::Fork: command write failure: $!"; 514 die "AnyEvent::Fork: command write failure: $!";
421 } 515 }
422 516
423 substr $self->[2][0], 0, $len, ""; 517 substr $self->[QUEUE][0], 0, $len, "";
424 shift @{ $self->[2] } unless length $self->[2][0]; 518 shift @{ $self->[QUEUE] } unless length $self->[QUEUE][0];
425 } 519 }
426 } while @{ $self->[2] }; 520 } while @{ $self->[QUEUE] };
427 521
428 # everything written 522 # everything written
429 undef $self->[3]; 523 undef $self->[WW];
430 524
431 # invoke run callback, if any 525 # invoke run callback, if any
432 $self->[4]->($self->[1]) if $self->[4]; 526 if ($self->[CB]) {
527 $self->[CB]->($self->[FH]);
528 @$self = ();
529 }
433 }; 530 };
434 531
435 () # make sure we don't leak the watcher 532 () # make sure we don't leak the watcher
436}
437
438sub _new {
439 my ($self, $fh, $pid) = @_;
440
441 AnyEvent::Util::fh_nonblocking $fh, 1;
442
443 $self = bless [
444 $pid,
445 $fh,
446 [], # write queue - strings or fd's
447 undef, # AE watcher
448 ], $self;
449
450 $self
451} 533}
452 534
453# fork template from current process, used by AnyEvent::Fork::Early/Template 535# fork template from current process, used by AnyEvent::Fork::Early/Template
454sub _new_fork { 536sub _new_fork {
455 my ($fh, $slave) = AnyEvent::Util::portable_socketpair; 537 my ($fh, $slave) = AnyEvent::Util::portable_socketpair;
460 if ($pid eq 0) { 542 if ($pid eq 0) {
461 require AnyEvent::Fork::Serve; 543 require AnyEvent::Fork::Serve;
462 $AnyEvent::Fork::Serve::OWNER = $parent; 544 $AnyEvent::Fork::Serve::OWNER = $parent;
463 close $fh; 545 close $fh;
464 $0 = "$_[1] of $parent"; 546 $0 = "$_[1] of $parent";
465 $SIG{CHLD} = 'IGNORE';
466 AnyEvent::Fork::Serve::serve ($slave); 547 AnyEvent::Fork::Serve::serve ($slave);
467 exit 0; 548 exit 0;
468 } elsif (!$pid) { 549 } elsif (!$pid) {
469 die "AnyEvent::Fork::Early/Template: unable to fork template process: $!"; 550 die "AnyEvent::Fork::Early/Template: unable to fork template process: $!";
470 } 551 }
528The path to the perl interpreter is divined using various methods - first 609The path to the perl interpreter is divined using various methods - first
529C<$^X> is investigated to see if the path ends with something that sounds 610C<$^X> is investigated to see if the path ends with something that sounds
530as if it were the perl interpreter. Failing this, the module falls back to 611as if it were the perl interpreter. Failing this, the module falls back to
531using C<$Config::Config{perlpath}>. 612using C<$Config::Config{perlpath}>.
532 613
614The path to perl can also be overriden by setting the global variable
615C<$AnyEvent::Fork::PERL> - it's value will be used for all subsequent
616invocations.
617
533=cut 618=cut
619
620our $PERL;
534 621
535sub new_exec { 622sub new_exec {
536 my ($self) = @_; 623 my ($self) = @_;
537 624
538 return $EARLY->fork 625 return $EARLY->fork
539 if $EARLY; 626 if $EARLY;
540 627
628 unless (defined $PERL) {
541 # first find path of perl 629 # first find path of perl
542 my $perl = $; 630 my $perl = $;
543 631
544 # first we try $^X, but the path must be absolute (always on win32), and end in sth. 632 # first we try $^X, but the path must be absolute (always on win32), and end in sth.
545 # that looks like perl. this obviously only works for posix and win32 633 # that looks like perl. this obviously only works for posix and win32
546 unless ( 634 unless (
547 ($^O eq "MSWin32" || $perl =~ m%^/%) 635 ($^O eq "MSWin32" || $perl =~ m%^/%)
548 && $perl =~ m%[/\\]perl(?:[0-9]+(\.[0-9]+)+)?(\.exe)?$%i 636 && $perl =~ m%[/\\]perl(?:[0-9]+(\.[0-9]+)+)?(\.exe)?$%i
549 ) { 637 ) {
550 # if it doesn't look perlish enough, try Config 638 # if it doesn't look perlish enough, try Config
551 require Config; 639 require Config;
552 $perl = $Config::Config{perlpath}; 640 $perl = $Config::Config{perlpath};
553 $perl =~ s/(?:\Q$Config::Config{_exe}\E)?$/$Config::Config{_exe}/; 641 $perl =~ s/(?:\Q$Config::Config{_exe}\E)?$/$Config::Config{_exe}/;
642 }
643
644 $PERL = $perl;
554 } 645 }
555 646
556 require Proc::FastSpawn; 647 require Proc::FastSpawn;
557 648
558 my ($fh, $slave) = AnyEvent::Util::portable_socketpair; 649 my ($fh, $slave) = AnyEvent::Util::portable_socketpair;
566 #local $ENV{PERL5LIB} = join ":", grep !ref, @INC; 657 #local $ENV{PERL5LIB} = join ":", grep !ref, @INC;
567 my %env = %ENV; 658 my %env = %ENV;
568 $env{PERL5LIB} = join +($^O eq "MSWin32" ? ";" : ":"), grep !ref, @INC; 659 $env{PERL5LIB} = join +($^O eq "MSWin32" ? ";" : ":"), grep !ref, @INC;
569 660
570 my $pid = Proc::FastSpawn::spawn ( 661 my $pid = Proc::FastSpawn::spawn (
571 $perl, 662 $PERL,
572 ["perl", "-MAnyEvent::Fork::Serve", "-e", "AnyEvent::Fork::Serve::me", fileno $slave, $$], 663 ["perl", "-MAnyEvent::Fork::Serve", "-e", "AnyEvent::Fork::Serve::me", fileno $slave, $$],
573 [map "$_=$env{$_}", keys %env], 664 [map "$_=$env{$_}", keys %env],
574 ) or die "unable to spawn AnyEvent::Fork server: $!"; 665 ) or die "unable to spawn AnyEvent::Fork server: $!";
575 666
576 $self->_new ($fh, $pid) 667 $self->_new ($fh, $pid)
589AnyEvent::Fork itself. 680AnyEvent::Fork itself.
590 681
591=cut 682=cut
592 683
593sub pid { 684sub pid {
594 $_[0][0] 685 $_[0][PID]
595} 686}
596 687
597=item $proc = $proc->eval ($perlcode, @args) 688=item $proc = $proc->eval ($perlcode, @args)
598 689
599Evaluates the given C<$perlcode> as ... perl code, while setting C<@_> to 690Evaluates the given C<$perlcode> as ... Perl code, while setting C<@_> to
600the strings specified by C<@args>, in the "main" package. 691the strings specified by C<@args>, in the "main" package.
601 692
602This call is meant to do any custom initialisation that might be required 693This call is meant to do any custom initialisation that might be required
603(for example, the C<require> method uses it). It's not supposed to be used 694(for example, the C<require> method uses it). It's not supposed to be used
604to completely take over the process, use C<run> for that. 695to completely take over the process, use C<run> for that.
666sub send_fh { 757sub send_fh {
667 my ($self, @fh) = @_; 758 my ($self, @fh) = @_;
668 759
669 for my $fh (@fh) { 760 for my $fh (@fh) {
670 $self->_cmd ("h"); 761 $self->_cmd ("h");
671 push @{ $self->[2] }, \$fh; 762 push @{ $self->[QUEUE] }, \$fh;
672 } 763 }
673 764
674 $self 765 $self
675} 766}
676 767
762=cut 853=cut
763 854
764sub run { 855sub run {
765 my ($self, $func, $cb) = @_; 856 my ($self, $func, $cb) = @_;
766 857
767 $self->[4] = $cb; 858 $self->[CB] = $cb;
768 $self->_cmd (r => $func); 859 $self->_cmd (r => $func);
860}
861
862=back
863
864=head2 EXPERIMENTAL METHODS
865
866These methods might go away completely or change behaviour, at any time.
867
868=over 4
869
870=item $proc->to_fh ($cb->($fh)) # EXPERIMENTAL, MIGHT BE REMOVED
871
872Flushes all commands out to the process and then calls the callback with
873the communications socket.
874
875The process object becomes unusable on return from this function - any
876further method calls result in undefined behaviour.
877
878The point of this method is to give you a file handle thta you cna pass
879to another process. In that other process, you can call C<new_from_fh
880AnyEvent::Fork> to create a new C<AnyEvent::Fork> object from it, thereby
881effectively passing a fork object to another process.
882
883=cut
884
885sub to_fh {
886 my ($self, $cb) = @_;
887
888 $self->[CB] = $cb;
889
890 unless ($self->[WW]) {
891 $self->[CB]->($self->[FH]);
892 @$self = ();
893 }
894}
895
896=item new_from_fh AnyEvent::Fork $fh # EXPERIMENTAL, MIGHT BE REMOVED
897
898Takes a file handle originally rceeived by the C<to_fh> method and creates
899a new C<AnyEvent:Fork> object. The child process itself will not change in
900any way, i.e. it will keep all the modifications done to it before calling
901C<to_fh>.
902
903The new object is very much like the original object, except that the
904C<pid> method will return C<undef> even if the process is a direct child.
905
906=cut
907
908sub new_from_fh {
909 my ($class, $fh) = @_;
910
911 $class->_new ($fh)
769} 912}
770 913
771=back 914=back
772 915
773=head1 PERFORMANCE 916=head1 PERFORMANCE
783 926
784 2079 new processes per second, using manual socketpair + fork 927 2079 new processes per second, using manual socketpair + fork
785 928
786Then I did the same thing, but instead of calling fork, I called 929Then I did the same thing, but instead of calling fork, I called
787AnyEvent::Fork->new->run ("CORE::exit") and then again waited for the 930AnyEvent::Fork->new->run ("CORE::exit") and then again waited for the
788socket form the child to close on exit. This does the same thing as manual 931socket from the child to close on exit. This does the same thing as manual
789socket pair + fork, except that what is forked is the template process 932socket pair + fork, except that what is forked is the template process
790(2440kB), and the socket needs to be passed to the server at the other end 933(2440kB), and the socket needs to be passed to the server at the other end
791of the socket first. 934of the socket first.
792 935
793 2307 new processes per second, using AnyEvent::Fork->new 936 2307 new processes per second, using AnyEvent::Fork->new
800So how can C<< AnyEvent->new >> be faster than a standard fork, even 943So how can C<< AnyEvent->new >> be faster than a standard fork, even
801though it uses the same operations, but adds a lot of overhead? 944though it uses the same operations, but adds a lot of overhead?
802 945
803The difference is simply the process size: forking the 5MB process takes 946The difference is simply the process size: forking the 5MB process takes
804so much longer than forking the 2.5MB template process that the extra 947so much longer than forking the 2.5MB template process that the extra
805overhead introduced is canceled out. 948overhead is canceled out.
806 949
807If the benchmark process grows, the normal fork becomes even slower: 950If the benchmark process grows, the normal fork becomes even slower:
808 951
809 1340 new processes, manual fork of a 20MB process 952 1340 new processes, manual fork of a 20MB process
810 731 new processes, manual fork of a 200MB process 953 731 new processes, manual fork of a 200MB process
870initialising them, for example, by calling C<init Gtk2> manually. 1013initialising them, for example, by calling C<init Gtk2> manually.
871 1014
872=item exiting calls object destructors 1015=item exiting calls object destructors
873 1016
874This only applies to users of L<AnyEvent::Fork:Early> and 1017This only applies to users of L<AnyEvent::Fork:Early> and
875L<AnyEvent::Fork::Template>, or when initialiasing code creates objects 1018L<AnyEvent::Fork::Template>, or when initialising code creates objects
876that reference external resources. 1019that reference external resources.
877 1020
878When a process created by AnyEvent::Fork exits, it might do so by calling 1021When a process created by AnyEvent::Fork exits, it might do so by calling
879exit, or simply letting perl reach the end of the program. At which point 1022exit, or simply letting perl reach the end of the program. At which point
880Perl runs all destructors. 1023Perl runs all destructors.
900to make it so, mostly due to the bloody broken perl that nobody seems to 1043to make it so, mostly due to the bloody broken perl that nobody seems to
901care about. The fork emulation is a bad joke - I have yet to see something 1044care about. The fork emulation is a bad joke - I have yet to see something
902useful that you can do with it without running into memory corruption 1045useful that you can do with it without running into memory corruption
903issues or other braindamage. Hrrrr. 1046issues or other braindamage. Hrrrr.
904 1047
1048Since fork is endlessly broken on win32 perls (it doesn't even remotely
1049work within it's documented limits) and quite obviously it's not getting
1050improved any time soon, the best way to proceed on windows would be to
1051always use C<new_exec> and thus never rely on perl's fork "emulation".
1052
905Cygwin perl is not supported at the moment due to some hilarious 1053Cygwin perl is not supported at the moment due to some hilarious
906shortcomings of its API - see L<IO::FDPoll> for more details. 1054shortcomings of its API - see L<IO::FDPoll> for more details. If you never
1055use C<send_fh> and always use C<new_exec> to create processes, it should
1056work though.
907 1057
908=head1 SEE ALSO 1058=head1 SEE ALSO
909 1059
910L<AnyEvent::Fork::Early> (to avoid executing a perl interpreter), 1060L<AnyEvent::Fork::Early>, to avoid executing a perl interpreter at all
1061(part of this distribution).
1062
911L<AnyEvent::Fork::Template> (to create a process by forking the main 1063L<AnyEvent::Fork::Template>, to create a process by forking the main
912program at a convenient time). 1064program at a convenient time (part of this distribution).
913 1065
914=head1 AUTHOR 1066L<AnyEvent::Fork::RPC>, for simple RPC to child processes (on CPAN).
1067
1068L<AnyEvent::Fork::Pool>, for simple worker process pool (on CPAN).
1069
1070=head1 AUTHOR AND CONTACT INFORMATION
915 1071
916 Marc Lehmann <schmorp@schmorp.de> 1072 Marc Lehmann <schmorp@schmorp.de>
917 http://home.schmorp.de/ 1073 http://software.schmorp.de/pkg/AnyEvent-Fork
918 1074
919=cut 1075=cut
920 1076
9211 10771
922 1078

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines