ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-Fork/Fork.pm
(Generate patch)

Comparing AnyEvent-Fork/Fork.pm (file contents):
Revision 1.40 by root, Sat Apr 6 22:41:56 2013 UTC vs.
Revision 1.56 by root, Sun Apr 28 13:47:52 2013 UTC

34This module only creates processes and lets you pass file handles and 34This module only creates processes and lets you pass file handles and
35strings to it, and run perl code. It does not implement any kind of RPC - 35strings to it, and run perl code. It does not implement any kind of RPC -
36there is no back channel from the process back to you, and there is no RPC 36there is no back channel from the process back to you, and there is no RPC
37or message passing going on. 37or message passing going on.
38 38
39If you need some form of RPC, you can either implement it yourself 39If you need some form of RPC, you could use the L<AnyEvent::Fork::RPC>
40in whatever way you like, use some message-passing module such 40companion module, which adds simple RPC/job queueing to a process created
41as L<AnyEvent::MP>, some pipe such as L<AnyEvent::ZeroMQ>, use 41by this module.
42L<AnyEvent::Handle> on both sides to send e.g. JSON or Storable messages, 42
43and so on. 43And if you need some automatic process pool management on top of
44L<AnyEvent::Fork::RPC>, you can look at the L<AnyEvent::Fork::Pool>
45companion module.
46
47Or you can implement it yourself in whatever way you like: use some
48message-passing module such as L<AnyEvent::MP>, some pipe such as
49L<AnyEvent::ZeroMQ>, use L<AnyEvent::Handle> on both sides to send
50e.g. JSON or Storable messages, and so on.
44 51
45=head2 COMPARISON TO OTHER MODULES 52=head2 COMPARISON TO OTHER MODULES
46 53
47There is an abundance of modules on CPAN that do "something fork", such as 54There is an abundance of modules on CPAN that do "something fork", such as
48L<Parallel::ForkManager>, L<AnyEvent::ForkManager>, L<AnyEvent::Worker> 55L<Parallel::ForkManager>, L<AnyEvent::ForkManager>, L<AnyEvent::Worker>
221 } 228 }
222 } 229 }
223 230
224=head2 use AnyEvent::Fork as a faster fork+exec 231=head2 use AnyEvent::Fork as a faster fork+exec
225 232
226This runs C</bin/echo hi>, with stdandard output redirected to /tmp/log 233This runs C</bin/echo hi>, with standard output redirected to F</tmp/log>
227and standard error redirected to the communications socket. It is usually 234and standard error redirected to the communications socket. It is usually
228faster than fork+exec, but still lets you prepare the environment. 235faster than fork+exec, but still lets you prepare the environment.
229 236
230 open my $output, ">/tmp/log" or die "$!"; 237 open my $output, ">/tmp/log" or die "$!";
231 238
247 ->send_arg ("/bin/echo", "hi") 254 ->send_arg ("/bin/echo", "hi")
248 ->run ("run", my $cv = AE::cv); 255 ->run ("run", my $cv = AE::cv);
249 256
250 my $stderr = $cv->recv; 257 my $stderr = $cv->recv;
251 258
259=head2 For stingy users: put the worker code into a C<DATA> section.
260
261When you want to be stingy with files, you cna put your code into the
262C<DATA> section of your module (or program):
263
264 use AnyEvent::Fork;
265
266 AnyEvent::Fork
267 ->new
268 ->eval (do { local $/; <DATA> })
269 ->run ("doit", sub { ... });
270
271 __DATA__
272
273 sub doit {
274 ... do something!
275 }
276
277=head2 For stingy standalone programs: do not rely on external files at
278all.
279
280For single-file scripts it can be inconvenient to rely on external
281files - even when using < C<DATA> section, you still need to C<exec>
282an external perl interpreter, which might not be available when using
283L<App::Staticperl>, L<Urlader> or L<PAR::Packer> for example.
284
285Two modules help here - L<AnyEvent::Fork::Early> forks a template process
286for all further calls to C<new_exec>, and L<AnyEvent::Fork::Template>
287forks the main program as a template process.
288
289Here is how your main program should look like:
290
291 #! perl
292
293 # optional, as the very first thing.
294 # in case modules want to create their own processes.
295 use AnyEvent::Fork::Early;
296
297 # next, load all modules you need in your template process
298 use Example::My::Module
299 use Example::Whatever;
300
301 # next, put your run function definition and anything else you
302 # need, but do not use code outside of BEGIN blocks.
303 sub worker_run {
304 my ($fh, @args) = @_;
305 ...
306 }
307
308 # now preserve everything so far as AnyEvent::Fork object
309 # in §TEMPLATE.
310 use AnyEvent::Fork::Template;
311
312 # do not put code outside of BEGIN blocks until here
313
314 # now use the $TEMPLATE process in any way you like
315
316 # for example: create 10 worker processes
317 my @worker;
318 my $cv = AE::cv;
319 for (1..10) {
320 $cv->begin;
321 $TEMPLATE->fork->send_arg ($_)->run ("worker_run", sub {
322 push @worker, shift;
323 $cv->end;
324 });
325 }
326 $cv->recv;
327
252=head1 CONCEPTS 328=head1 CONCEPTS
253 329
254This module can create new processes either by executing a new perl 330This module can create new processes either by executing a new perl
255process, or by forking from an existing "template" process. 331process, or by forking from an existing "template" process.
332
333All these processes are called "child processes" (whether they are direct
334children or not), while the process that manages them is called the
335"parent process".
256 336
257Each such process comes with its own file handle that can be used to 337Each such process comes with its own file handle that can be used to
258communicate with it (it's actually a socket - one end in the new process, 338communicate with it (it's actually a socket - one end in the new process,
259one end in the main process), and among the things you can do in it are 339one end in the main process), and among the things you can do in it are
260load modules, fork new processes, send file handles to it, and execute 340load modules, fork new processes, send file handles to it, and execute
370use AnyEvent; 450use AnyEvent;
371use AnyEvent::Util (); 451use AnyEvent::Util ();
372 452
373use IO::FDPass; 453use IO::FDPass;
374 454
375our $VERSION = 0.6; 455our $VERSION = 1.1;
376
377our $PERL; # the path to the perl interpreter, deduces with various forms of magic
378
379=over 4
380
381=back
382
383=cut
384 456
385# the early fork template process 457# the early fork template process
386our $EARLY; 458our $EARLY;
387 459
388# the empty template process 460# the empty template process
389our $TEMPLATE; 461our $TEMPLATE;
462
463sub QUEUE() { 0 }
464sub FH() { 1 }
465sub WW() { 2 }
466sub PID() { 3 }
467sub CB() { 4 }
468
469sub _new {
470 my ($self, $fh, $pid) = @_;
471
472 AnyEvent::Util::fh_nonblocking $fh, 1;
473
474 $self = bless [
475 [], # write queue - strings or fd's
476 $fh,
477 undef, # AE watcher
478 $pid,
479 ], $self;
480
481 $self
482}
390 483
391sub _cmd { 484sub _cmd {
392 my $self = shift; 485 my $self = shift;
393 486
394 # ideally, we would want to use "a (w/a)*" as format string, but perl 487 # ideally, we would want to use "a (w/a)*" as format string, but perl
395 # versions from at least 5.8.9 to 5.16.3 are all buggy and can't unpack 488 # versions from at least 5.8.9 to 5.16.3 are all buggy and can't unpack
396 # it. 489 # it.
397 push @{ $self->[2] }, pack "a L/a*", $_[0], $_[1]; 490 push @{ $self->[QUEUE] }, pack "a L/a*", $_[0], $_[1];
398 491
399 $self->[3] ||= AE::io $self->[1], 1, sub { 492 $self->[WW] ||= AE::io $self->[FH], 1, sub {
400 do { 493 do {
401 # send the next "thing" in the queue - either a reference to an fh, 494 # send the next "thing" in the queue - either a reference to an fh,
402 # or a plain string. 495 # or a plain string.
403 496
404 if (ref $self->[2][0]) { 497 if (ref $self->[QUEUE][0]) {
405 # send fh 498 # send fh
406 unless (IO::FDPass::send fileno $self->[1], fileno ${ $self->[2][0] }) { 499 unless (IO::FDPass::send fileno $self->[FH], fileno ${ $self->[QUEUE][0] }) {
407 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK; 500 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK;
408 undef $self->[3]; 501 undef $self->[WW];
409 die "AnyEvent::Fork: file descriptor send failure: $!"; 502 die "AnyEvent::Fork: file descriptor send failure: $!";
410 } 503 }
411 504
412 shift @{ $self->[2] }; 505 shift @{ $self->[QUEUE] };
413 506
414 } else { 507 } else {
415 # send string 508 # send string
416 my $len = syswrite $self->[1], $self->[2][0]; 509 my $len = syswrite $self->[FH], $self->[QUEUE][0];
417 510
418 unless ($len) { 511 unless ($len) {
419 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK; 512 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK;
420 undef $self->[3]; 513 undef $self->[WW];
421 die "AnyEvent::Fork: command write failure: $!"; 514 die "AnyEvent::Fork: command write failure: $!";
422 } 515 }
423 516
424 substr $self->[2][0], 0, $len, ""; 517 substr $self->[QUEUE][0], 0, $len, "";
425 shift @{ $self->[2] } unless length $self->[2][0]; 518 shift @{ $self->[QUEUE] } unless length $self->[QUEUE][0];
426 } 519 }
427 } while @{ $self->[2] }; 520 } while @{ $self->[QUEUE] };
428 521
429 # everything written 522 # everything written
430 undef $self->[3]; 523 undef $self->[WW];
431 524
432 # invoke run callback, if any 525 # invoke run callback, if any
433 $self->[4]->($self->[1]) if $self->[4]; 526 if ($self->[CB]) {
527 $self->[CB]->($self->[FH]);
528 @$self = ();
529 }
434 }; 530 };
435 531
436 () # make sure we don't leak the watcher 532 () # make sure we don't leak the watcher
437}
438
439sub _new {
440 my ($self, $fh, $pid) = @_;
441
442 AnyEvent::Util::fh_nonblocking $fh, 1;
443
444 $self = bless [
445 $pid,
446 $fh,
447 [], # write queue - strings or fd's
448 undef, # AE watcher
449 ], $self;
450
451 $self
452} 533}
453 534
454# fork template from current process, used by AnyEvent::Fork::Early/Template 535# fork template from current process, used by AnyEvent::Fork::Early/Template
455sub _new_fork { 536sub _new_fork {
456 my ($fh, $slave) = AnyEvent::Util::portable_socketpair; 537 my ($fh, $slave) = AnyEvent::Util::portable_socketpair;
461 if ($pid eq 0) { 542 if ($pid eq 0) {
462 require AnyEvent::Fork::Serve; 543 require AnyEvent::Fork::Serve;
463 $AnyEvent::Fork::Serve::OWNER = $parent; 544 $AnyEvent::Fork::Serve::OWNER = $parent;
464 close $fh; 545 close $fh;
465 $0 = "$_[1] of $parent"; 546 $0 = "$_[1] of $parent";
466 $SIG{CHLD} = 'IGNORE';
467 AnyEvent::Fork::Serve::serve ($slave); 547 AnyEvent::Fork::Serve::serve ($slave);
468 exit 0; 548 exit 0;
469 } elsif (!$pid) { 549 } elsif (!$pid) {
470 die "AnyEvent::Fork::Early/Template: unable to fork template process: $!"; 550 die "AnyEvent::Fork::Early/Template: unable to fork template process: $!";
471 } 551 }
529The path to the perl interpreter is divined using various methods - first 609The path to the perl interpreter is divined using various methods - first
530C<$^X> is investigated to see if the path ends with something that sounds 610C<$^X> is investigated to see if the path ends with something that sounds
531as if it were the perl interpreter. Failing this, the module falls back to 611as if it were the perl interpreter. Failing this, the module falls back to
532using C<$Config::Config{perlpath}>. 612using C<$Config::Config{perlpath}>.
533 613
614The path to perl can also be overriden by setting the global variable
615C<$AnyEvent::Fork::PERL> - it's value will be used for all subsequent
616invocations.
617
534=cut 618=cut
619
620our $PERL;
535 621
536sub new_exec { 622sub new_exec {
537 my ($self) = @_; 623 my ($self) = @_;
538 624
539 return $EARLY->fork 625 return $EARLY->fork
540 if $EARLY; 626 if $EARLY;
541 627
628 unless (defined $PERL) {
542 # first find path of perl 629 # first find path of perl
543 my $perl = $; 630 my $perl = $;
544 631
545 # first we try $^X, but the path must be absolute (always on win32), and end in sth. 632 # first we try $^X, but the path must be absolute (always on win32), and end in sth.
546 # that looks like perl. this obviously only works for posix and win32 633 # that looks like perl. this obviously only works for posix and win32
547 unless ( 634 unless (
548 ($^O eq "MSWin32" || $perl =~ m%^/%) 635 ($^O eq "MSWin32" || $perl =~ m%^/%)
549 && $perl =~ m%[/\\]perl(?:[0-9]+(\.[0-9]+)+)?(\.exe)?$%i 636 && $perl =~ m%[/\\]perl(?:[0-9]+(\.[0-9]+)+)?(\.exe)?$%i
550 ) { 637 ) {
551 # if it doesn't look perlish enough, try Config 638 # if it doesn't look perlish enough, try Config
552 require Config; 639 require Config;
553 $perl = $Config::Config{perlpath}; 640 $perl = $Config::Config{perlpath};
554 $perl =~ s/(?:\Q$Config::Config{_exe}\E)?$/$Config::Config{_exe}/; 641 $perl =~ s/(?:\Q$Config::Config{_exe}\E)?$/$Config::Config{_exe}/;
642 }
643
644 $PERL = $perl;
555 } 645 }
556 646
557 require Proc::FastSpawn; 647 require Proc::FastSpawn;
558 648
559 my ($fh, $slave) = AnyEvent::Util::portable_socketpair; 649 my ($fh, $slave) = AnyEvent::Util::portable_socketpair;
567 #local $ENV{PERL5LIB} = join ":", grep !ref, @INC; 657 #local $ENV{PERL5LIB} = join ":", grep !ref, @INC;
568 my %env = %ENV; 658 my %env = %ENV;
569 $env{PERL5LIB} = join +($^O eq "MSWin32" ? ";" : ":"), grep !ref, @INC; 659 $env{PERL5LIB} = join +($^O eq "MSWin32" ? ";" : ":"), grep !ref, @INC;
570 660
571 my $pid = Proc::FastSpawn::spawn ( 661 my $pid = Proc::FastSpawn::spawn (
572 $perl, 662 $PERL,
573 ["perl", "-MAnyEvent::Fork::Serve", "-e", "AnyEvent::Fork::Serve::me", fileno $slave, $$], 663 ["perl", "-MAnyEvent::Fork::Serve", "-e", "AnyEvent::Fork::Serve::me", fileno $slave, $$],
574 [map "$_=$env{$_}", keys %env], 664 [map "$_=$env{$_}", keys %env],
575 ) or die "unable to spawn AnyEvent::Fork server: $!"; 665 ) or die "unable to spawn AnyEvent::Fork server: $!";
576 666
577 $self->_new ($fh, $pid) 667 $self->_new ($fh, $pid)
590AnyEvent::Fork itself. 680AnyEvent::Fork itself.
591 681
592=cut 682=cut
593 683
594sub pid { 684sub pid {
595 $_[0][0] 685 $_[0][PID]
596} 686}
597 687
598=item $proc = $proc->eval ($perlcode, @args) 688=item $proc = $proc->eval ($perlcode, @args)
599 689
600Evaluates the given C<$perlcode> as ... perl code, while setting C<@_> to 690Evaluates the given C<$perlcode> as ... Perl code, while setting C<@_> to
601the strings specified by C<@args>, in the "main" package. 691the strings specified by C<@args>, in the "main" package.
602 692
603This call is meant to do any custom initialisation that might be required 693This call is meant to do any custom initialisation that might be required
604(for example, the C<require> method uses it). It's not supposed to be used 694(for example, the C<require> method uses it). It's not supposed to be used
605to completely take over the process, use C<run> for that. 695to completely take over the process, use C<run> for that.
667sub send_fh { 757sub send_fh {
668 my ($self, @fh) = @_; 758 my ($self, @fh) = @_;
669 759
670 for my $fh (@fh) { 760 for my $fh (@fh) {
671 $self->_cmd ("h"); 761 $self->_cmd ("h");
672 push @{ $self->[2] }, \$fh; 762 push @{ $self->[QUEUE] }, \$fh;
673 } 763 }
674 764
675 $self 765 $self
676} 766}
677 767
763=cut 853=cut
764 854
765sub run { 855sub run {
766 my ($self, $func, $cb) = @_; 856 my ($self, $func, $cb) = @_;
767 857
768 $self->[4] = $cb; 858 $self->[CB] = $cb;
769 $self->_cmd (r => $func); 859 $self->_cmd (r => $func);
860}
861
862=back
863
864=head2 EXPERIMENTAL METHODS
865
866These methods might go away completely or change behaviour, at any time.
867
868=over 4
869
870=item $proc->to_fh ($cb->($fh)) # EXPERIMENTAL, MIGHT BE REMOVED
871
872Flushes all commands out to the process and then calls the callback with
873the communications socket.
874
875The process object becomes unusable on return from this function - any
876further method calls result in undefined behaviour.
877
878The point of this method is to give you a file handle thta you cna pass
879to another process. In that other process, you can call C<new_from_fh
880AnyEvent::Fork> to create a new C<AnyEvent::Fork> object from it, thereby
881effectively passing a fork object to another process.
882
883=cut
884
885sub to_fh {
886 my ($self, $cb) = @_;
887
888 $self->[CB] = $cb;
889
890 unless ($self->[WW]) {
891 $self->[CB]->($self->[FH]);
892 @$self = ();
893 }
894}
895
896=item new_from_fh AnyEvent::Fork $fh # EXPERIMENTAL, MIGHT BE REMOVED
897
898Takes a file handle originally rceeived by the C<to_fh> method and creates
899a new C<AnyEvent:Fork> object. The child process itself will not change in
900any way, i.e. it will keep all the modifications done to it before calling
901C<to_fh>.
902
903The new object is very much like the original object, except that the
904C<pid> method will return C<undef> even if the process is a direct child.
905
906=cut
907
908sub new_from_fh {
909 my ($class, $fh) = @_;
910
911 $class->_new ($fh)
770} 912}
771 913
772=back 914=back
773 915
774=head1 PERFORMANCE 916=head1 PERFORMANCE
784 926
785 2079 new processes per second, using manual socketpair + fork 927 2079 new processes per second, using manual socketpair + fork
786 928
787Then I did the same thing, but instead of calling fork, I called 929Then I did the same thing, but instead of calling fork, I called
788AnyEvent::Fork->new->run ("CORE::exit") and then again waited for the 930AnyEvent::Fork->new->run ("CORE::exit") and then again waited for the
789socket form the child to close on exit. This does the same thing as manual 931socket from the child to close on exit. This does the same thing as manual
790socket pair + fork, except that what is forked is the template process 932socket pair + fork, except that what is forked is the template process
791(2440kB), and the socket needs to be passed to the server at the other end 933(2440kB), and the socket needs to be passed to the server at the other end
792of the socket first. 934of the socket first.
793 935
794 2307 new processes per second, using AnyEvent::Fork->new 936 2307 new processes per second, using AnyEvent::Fork->new
801So how can C<< AnyEvent->new >> be faster than a standard fork, even 943So how can C<< AnyEvent->new >> be faster than a standard fork, even
802though it uses the same operations, but adds a lot of overhead? 944though it uses the same operations, but adds a lot of overhead?
803 945
804The difference is simply the process size: forking the 5MB process takes 946The difference is simply the process size: forking the 5MB process takes
805so much longer than forking the 2.5MB template process that the extra 947so much longer than forking the 2.5MB template process that the extra
806overhead introduced is canceled out. 948overhead is canceled out.
807 949
808If the benchmark process grows, the normal fork becomes even slower: 950If the benchmark process grows, the normal fork becomes even slower:
809 951
810 1340 new processes, manual fork of a 20MB process 952 1340 new processes, manual fork of a 20MB process
811 731 new processes, manual fork of a 200MB process 953 731 new processes, manual fork of a 200MB process
871initialising them, for example, by calling C<init Gtk2> manually. 1013initialising them, for example, by calling C<init Gtk2> manually.
872 1014
873=item exiting calls object destructors 1015=item exiting calls object destructors
874 1016
875This only applies to users of L<AnyEvent::Fork:Early> and 1017This only applies to users of L<AnyEvent::Fork:Early> and
876L<AnyEvent::Fork::Template>, or when initialiasing code creates objects 1018L<AnyEvent::Fork::Template>, or when initialising code creates objects
877that reference external resources. 1019that reference external resources.
878 1020
879When a process created by AnyEvent::Fork exits, it might do so by calling 1021When a process created by AnyEvent::Fork exits, it might do so by calling
880exit, or simply letting perl reach the end of the program. At which point 1022exit, or simply letting perl reach the end of the program. At which point
881Perl runs all destructors. 1023Perl runs all destructors.
901to make it so, mostly due to the bloody broken perl that nobody seems to 1043to make it so, mostly due to the bloody broken perl that nobody seems to
902care about. The fork emulation is a bad joke - I have yet to see something 1044care about. The fork emulation is a bad joke - I have yet to see something
903useful that you can do with it without running into memory corruption 1045useful that you can do with it without running into memory corruption
904issues or other braindamage. Hrrrr. 1046issues or other braindamage. Hrrrr.
905 1047
1048Since fork is endlessly broken on win32 perls (it doesn't even remotely
1049work within it's documented limits) and quite obviously it's not getting
1050improved any time soon, the best way to proceed on windows would be to
1051always use C<new_exec> and thus never rely on perl's fork "emulation".
1052
906Cygwin perl is not supported at the moment due to some hilarious 1053Cygwin perl is not supported at the moment due to some hilarious
907shortcomings of its API - see L<IO::FDPoll> for more details. 1054shortcomings of its API - see L<IO::FDPoll> for more details. If you never
1055use C<send_fh> and always use C<new_exec> to create processes, it should
1056work though.
908 1057
909=head1 SEE ALSO 1058=head1 SEE ALSO
910 1059
911L<AnyEvent::Fork::Early> (to avoid executing a perl interpreter), 1060L<AnyEvent::Fork::Early>, to avoid executing a perl interpreter at all
1061(part of this distribution).
1062
912L<AnyEvent::Fork::Template> (to create a process by forking the main 1063L<AnyEvent::Fork::Template>, to create a process by forking the main
913program at a convenient time). 1064program at a convenient time (part of this distribution).
914 1065
915=head1 AUTHOR 1066L<AnyEvent::Fork::RPC>, for simple RPC to child processes (on CPAN).
1067
1068L<AnyEvent::Fork::Pool>, for simple worker process pool (on CPAN).
1069
1070=head1 AUTHOR AND CONTACT INFORMATION
916 1071
917 Marc Lehmann <schmorp@schmorp.de> 1072 Marc Lehmann <schmorp@schmorp.de>
918 http://home.schmorp.de/ 1073 http://software.schmorp.de/pkg/AnyEvent-Fork
919 1074
920=cut 1075=cut
921 1076
9221 10771
923 1078

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines