ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-Fork/Fork.pm
(Generate patch)

Comparing AnyEvent-Fork/Fork.pm (file contents):
Revision 1.41 by root, Mon Apr 8 03:20:53 2013 UTC vs.
Revision 1.57 by root, Sun Aug 25 17:38:43 2013 UTC

34This module only creates processes and lets you pass file handles and 34This module only creates processes and lets you pass file handles and
35strings to it, and run perl code. It does not implement any kind of RPC - 35strings to it, and run perl code. It does not implement any kind of RPC -
36there is no back channel from the process back to you, and there is no RPC 36there is no back channel from the process back to you, and there is no RPC
37or message passing going on. 37or message passing going on.
38 38
39If you need some form of RPC, you can either implement it yourself 39If you need some form of RPC, you could use the L<AnyEvent::Fork::RPC>
40in whatever way you like, use some message-passing module such 40companion module, which adds simple RPC/job queueing to a process created
41as L<AnyEvent::MP>, some pipe such as L<AnyEvent::ZeroMQ>, use 41by this module.
42L<AnyEvent::Handle> on both sides to send e.g. JSON or Storable messages, 42
43and so on. 43And if you need some automatic process pool management on top of
44L<AnyEvent::Fork::RPC>, you can look at the L<AnyEvent::Fork::Pool>
45companion module.
46
47Or you can implement it yourself in whatever way you like: use some
48message-passing module such as L<AnyEvent::MP>, some pipe such as
49L<AnyEvent::ZeroMQ>, use L<AnyEvent::Handle> on both sides to send
50e.g. JSON or Storable messages, and so on.
44 51
45=head2 COMPARISON TO OTHER MODULES 52=head2 COMPARISON TO OTHER MODULES
46 53
47There is an abundance of modules on CPAN that do "something fork", such as 54There is an abundance of modules on CPAN that do "something fork", such as
48L<Parallel::ForkManager>, L<AnyEvent::ForkManager>, L<AnyEvent::Worker> 55L<Parallel::ForkManager>, L<AnyEvent::ForkManager>, L<AnyEvent::Worker>
221 } 228 }
222 } 229 }
223 230
224=head2 use AnyEvent::Fork as a faster fork+exec 231=head2 use AnyEvent::Fork as a faster fork+exec
225 232
226This runs C</bin/echo hi>, with stdandard output redirected to /tmp/log 233This runs C</bin/echo hi>, with standard output redirected to F</tmp/log>
227and standard error redirected to the communications socket. It is usually 234and standard error redirected to the communications socket. It is usually
228faster than fork+exec, but still lets you prepare the environment. 235faster than fork+exec, but still lets you prepare the environment.
229 236
230 open my $output, ">/tmp/log" or die "$!"; 237 open my $output, ">/tmp/log" or die "$!";
231 238
247 ->send_arg ("/bin/echo", "hi") 254 ->send_arg ("/bin/echo", "hi")
248 ->run ("run", my $cv = AE::cv); 255 ->run ("run", my $cv = AE::cv);
249 256
250 my $stderr = $cv->recv; 257 my $stderr = $cv->recv;
251 258
259=head2 For stingy users: put the worker code into a C<DATA> section.
260
261When you want to be stingy with files, you cna put your code into the
262C<DATA> section of your module (or program):
263
264 use AnyEvent::Fork;
265
266 AnyEvent::Fork
267 ->new
268 ->eval (do { local $/; <DATA> })
269 ->run ("doit", sub { ... });
270
271 __DATA__
272
273 sub doit {
274 ... do something!
275 }
276
277=head2 For stingy standalone programs: do not rely on external files at
278all.
279
280For single-file scripts it can be inconvenient to rely on external
281files - even when using < C<DATA> section, you still need to C<exec>
282an external perl interpreter, which might not be available when using
283L<App::Staticperl>, L<Urlader> or L<PAR::Packer> for example.
284
285Two modules help here - L<AnyEvent::Fork::Early> forks a template process
286for all further calls to C<new_exec>, and L<AnyEvent::Fork::Template>
287forks the main program as a template process.
288
289Here is how your main program should look like:
290
291 #! perl
292
293 # optional, as the very first thing.
294 # in case modules want to create their own processes.
295 use AnyEvent::Fork::Early;
296
297 # next, load all modules you need in your template process
298 use Example::My::Module
299 use Example::Whatever;
300
301 # next, put your run function definition and anything else you
302 # need, but do not use code outside of BEGIN blocks.
303 sub worker_run {
304 my ($fh, @args) = @_;
305 ...
306 }
307
308 # now preserve everything so far as AnyEvent::Fork object
309 # in §TEMPLATE.
310 use AnyEvent::Fork::Template;
311
312 # do not put code outside of BEGIN blocks until here
313
314 # now use the $TEMPLATE process in any way you like
315
316 # for example: create 10 worker processes
317 my @worker;
318 my $cv = AE::cv;
319 for (1..10) {
320 $cv->begin;
321 $TEMPLATE->fork->send_arg ($_)->run ("worker_run", sub {
322 push @worker, shift;
323 $cv->end;
324 });
325 }
326 $cv->recv;
327
252=head1 CONCEPTS 328=head1 CONCEPTS
253 329
254This module can create new processes either by executing a new perl 330This module can create new processes either by executing a new perl
255process, or by forking from an existing "template" process. 331process, or by forking from an existing "template" process.
332
333All these processes are called "child processes" (whether they are direct
334children or not), while the process that manages them is called the
335"parent process".
256 336
257Each such process comes with its own file handle that can be used to 337Each such process comes with its own file handle that can be used to
258communicate with it (it's actually a socket - one end in the new process, 338communicate with it (it's actually a socket - one end in the new process,
259one end in the main process), and among the things you can do in it are 339one end in the main process), and among the things you can do in it are
260load modules, fork new processes, send file handles to it, and execute 340load modules, fork new processes, send file handles to it, and execute
370use AnyEvent; 450use AnyEvent;
371use AnyEvent::Util (); 451use AnyEvent::Util ();
372 452
373use IO::FDPass; 453use IO::FDPass;
374 454
375our $VERSION = 0.6; 455our $VERSION = 1.1;
376
377=over 4
378
379=back
380
381=cut
382 456
383# the early fork template process 457# the early fork template process
384our $EARLY; 458our $EARLY;
385 459
386# the empty template process 460# the empty template process
387our $TEMPLATE; 461our $TEMPLATE;
462
463sub QUEUE() { 0 }
464sub FH() { 1 }
465sub WW() { 2 }
466sub PID() { 3 }
467sub CB() { 4 }
468
469sub _new {
470 my ($self, $fh, $pid) = @_;
471
472 AnyEvent::Util::fh_nonblocking $fh, 1;
473
474 $self = bless [
475 [], # write queue - strings or fd's
476 $fh,
477 undef, # AE watcher
478 $pid,
479 ], $self;
480
481 $self
482}
388 483
389sub _cmd { 484sub _cmd {
390 my $self = shift; 485 my $self = shift;
391 486
392 # ideally, we would want to use "a (w/a)*" as format string, but perl 487 # ideally, we would want to use "a (w/a)*" as format string, but perl
393 # versions from at least 5.8.9 to 5.16.3 are all buggy and can't unpack 488 # versions from at least 5.8.9 to 5.16.3 are all buggy and can't unpack
394 # it. 489 # it.
395 push @{ $self->[2] }, pack "a L/a*", $_[0], $_[1]; 490 push @{ $self->[QUEUE] }, pack "a L/a*", $_[0], $_[1];
396 491
397 $self->[3] ||= AE::io $self->[1], 1, sub { 492 $self->[WW] ||= AE::io $self->[FH], 1, sub {
398 do { 493 do {
399 # send the next "thing" in the queue - either a reference to an fh, 494 # send the next "thing" in the queue - either a reference to an fh,
400 # or a plain string. 495 # or a plain string.
401 496
402 if (ref $self->[2][0]) { 497 if (ref $self->[QUEUE][0]) {
403 # send fh 498 # send fh
404 unless (IO::FDPass::send fileno $self->[1], fileno ${ $self->[2][0] }) { 499 unless (IO::FDPass::send fileno $self->[FH], fileno ${ $self->[QUEUE][0] }) {
405 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK; 500 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK;
406 undef $self->[3]; 501 undef $self->[WW];
407 die "AnyEvent::Fork: file descriptor send failure: $!"; 502 die "AnyEvent::Fork: file descriptor send failure: $!";
408 } 503 }
409 504
410 shift @{ $self->[2] }; 505 shift @{ $self->[QUEUE] };
411 506
412 } else { 507 } else {
413 # send string 508 # send string
414 my $len = syswrite $self->[1], $self->[2][0]; 509 my $len = syswrite $self->[FH], $self->[QUEUE][0];
415 510
416 unless ($len) { 511 unless ($len) {
417 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK; 512 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK;
418 undef $self->[3]; 513 undef $self->[WW];
419 die "AnyEvent::Fork: command write failure: $!"; 514 die "AnyEvent::Fork: command write failure: $!";
420 } 515 }
421 516
422 substr $self->[2][0], 0, $len, ""; 517 substr $self->[QUEUE][0], 0, $len, "";
423 shift @{ $self->[2] } unless length $self->[2][0]; 518 shift @{ $self->[QUEUE] } unless length $self->[QUEUE][0];
424 } 519 }
425 } while @{ $self->[2] }; 520 } while @{ $self->[QUEUE] };
426 521
427 # everything written 522 # everything written
428 undef $self->[3]; 523 undef $self->[WW];
429 524
430 # invoke run callback, if any 525 # invoke run callback, if any
431 $self->[4]->($self->[1]) if $self->[4]; 526 if ($self->[CB]) {
527 $self->[CB]->($self->[FH]);
528 @$self = ();
529 }
432 }; 530 };
433 531
434 () # make sure we don't leak the watcher 532 () # make sure we don't leak the watcher
435}
436
437sub _new {
438 my ($self, $fh, $pid) = @_;
439
440 AnyEvent::Util::fh_nonblocking $fh, 1;
441
442 $self = bless [
443 $pid,
444 $fh,
445 [], # write queue - strings or fd's
446 undef, # AE watcher
447 ], $self;
448
449 $self
450} 533}
451 534
452# fork template from current process, used by AnyEvent::Fork::Early/Template 535# fork template from current process, used by AnyEvent::Fork::Early/Template
453sub _new_fork { 536sub _new_fork {
454 my ($fh, $slave) = AnyEvent::Util::portable_socketpair; 537 my ($fh, $slave) = AnyEvent::Util::portable_socketpair;
522 605
523You should use C<new> whenever possible, except when having a template 606You should use C<new> whenever possible, except when having a template
524process around is unacceptable. 607process around is unacceptable.
525 608
526The path to the perl interpreter is divined using various methods - first 609The path to the perl interpreter is divined using various methods - first
527C<$^X> is investigated to see if the path ends with something that sounds 610C<$^X> is investigated to see if the path ends with something that looks
528as if it were the perl interpreter. Failing this, the module falls back to 611as if it were the perl interpreter. Failing this, the module falls back to
529using C<$Config::Config{perlpath}>. 612using C<$Config::Config{perlpath}>.
530 613
614The path to perl can also be overriden by setting the global variable
615C<$AnyEvent::Fork::PERL> - it's value will be used for all subsequent
616invocations.
617
531=cut 618=cut
619
620our $PERL;
532 621
533sub new_exec { 622sub new_exec {
534 my ($self) = @_; 623 my ($self) = @_;
535 624
536 return $EARLY->fork 625 return $EARLY->fork
537 if $EARLY; 626 if $EARLY;
538 627
628 unless (defined $PERL) {
539 # first find path of perl 629 # first find path of perl
540 my $perl = $; 630 my $perl = $;
541 631
542 # first we try $^X, but the path must be absolute (always on win32), and end in sth. 632 # first we try $^X, but the path must be absolute (always on win32), and end in sth.
543 # that looks like perl. this obviously only works for posix and win32 633 # that looks like perl. this obviously only works for posix and win32
544 unless ( 634 unless (
545 ($^O eq "MSWin32" || $perl =~ m%^/%) 635 ($^O eq "MSWin32" || $perl =~ m%^/%)
546 && $perl =~ m%[/\\]perl(?:[0-9]+(\.[0-9]+)+)?(\.exe)?$%i 636 && $perl =~ m%[/\\]perl(?:[0-9]+(\.[0-9]+)+)?(\.exe)?$%i
547 ) { 637 ) {
548 # if it doesn't look perlish enough, try Config 638 # if it doesn't look perlish enough, try Config
549 require Config; 639 require Config;
550 $perl = $Config::Config{perlpath}; 640 $perl = $Config::Config{perlpath};
551 $perl =~ s/(?:\Q$Config::Config{_exe}\E)?$/$Config::Config{_exe}/; 641 $perl =~ s/(?:\Q$Config::Config{_exe}\E)?$/$Config::Config{_exe}/;
642 }
643
644 $PERL = $perl;
552 } 645 }
553 646
554 require Proc::FastSpawn; 647 require Proc::FastSpawn;
555 648
556 my ($fh, $slave) = AnyEvent::Util::portable_socketpair; 649 my ($fh, $slave) = AnyEvent::Util::portable_socketpair;
564 #local $ENV{PERL5LIB} = join ":", grep !ref, @INC; 657 #local $ENV{PERL5LIB} = join ":", grep !ref, @INC;
565 my %env = %ENV; 658 my %env = %ENV;
566 $env{PERL5LIB} = join +($^O eq "MSWin32" ? ";" : ":"), grep !ref, @INC; 659 $env{PERL5LIB} = join +($^O eq "MSWin32" ? ";" : ":"), grep !ref, @INC;
567 660
568 my $pid = Proc::FastSpawn::spawn ( 661 my $pid = Proc::FastSpawn::spawn (
569 $perl, 662 $PERL,
570 ["perl", "-MAnyEvent::Fork::Serve", "-e", "AnyEvent::Fork::Serve::me", fileno $slave, $$], 663 ["perl", "-MAnyEvent::Fork::Serve", "-e", "AnyEvent::Fork::Serve::me", fileno $slave, $$],
571 [map "$_=$env{$_}", keys %env], 664 [map "$_=$env{$_}", keys %env],
572 ) or die "unable to spawn AnyEvent::Fork server: $!"; 665 ) or die "unable to spawn AnyEvent::Fork server: $!";
573 666
574 $self->_new ($fh, $pid) 667 $self->_new ($fh, $pid)
587AnyEvent::Fork itself. 680AnyEvent::Fork itself.
588 681
589=cut 682=cut
590 683
591sub pid { 684sub pid {
592 $_[0][0] 685 $_[0][PID]
593} 686}
594 687
595=item $proc = $proc->eval ($perlcode, @args) 688=item $proc = $proc->eval ($perlcode, @args)
596 689
597Evaluates the given C<$perlcode> as ... perl code, while setting C<@_> to 690Evaluates the given C<$perlcode> as ... Perl code, while setting C<@_> to
598the strings specified by C<@args>, in the "main" package. 691the strings specified by C<@args>, in the "main" package.
599 692
600This call is meant to do any custom initialisation that might be required 693This call is meant to do any custom initialisation that might be required
601(for example, the C<require> method uses it). It's not supposed to be used 694(for example, the C<require> method uses it). It's not supposed to be used
602to completely take over the process, use C<run> for that. 695to completely take over the process, use C<run> for that.
664sub send_fh { 757sub send_fh {
665 my ($self, @fh) = @_; 758 my ($self, @fh) = @_;
666 759
667 for my $fh (@fh) { 760 for my $fh (@fh) {
668 $self->_cmd ("h"); 761 $self->_cmd ("h");
669 push @{ $self->[2] }, \$fh; 762 push @{ $self->[QUEUE] }, \$fh;
670 } 763 }
671 764
672 $self 765 $self
673} 766}
674 767
760=cut 853=cut
761 854
762sub run { 855sub run {
763 my ($self, $func, $cb) = @_; 856 my ($self, $func, $cb) = @_;
764 857
765 $self->[4] = $cb; 858 $self->[CB] = $cb;
766 $self->_cmd (r => $func); 859 $self->_cmd (r => $func);
860}
861
862=back
863
864=head2 EXPERIMENTAL METHODS
865
866These methods might go away completely or change behaviour, at any time.
867
868=over 4
869
870=item $proc->to_fh ($cb->($fh)) # EXPERIMENTAL, MIGHT BE REMOVED
871
872Flushes all commands out to the process and then calls the callback with
873the communications socket.
874
875The process object becomes unusable on return from this function - any
876further method calls result in undefined behaviour.
877
878The point of this method is to give you a file handle thta you cna pass
879to another process. In that other process, you can call C<new_from_fh
880AnyEvent::Fork> to create a new C<AnyEvent::Fork> object from it, thereby
881effectively passing a fork object to another process.
882
883=cut
884
885sub to_fh {
886 my ($self, $cb) = @_;
887
888 $self->[CB] = $cb;
889
890 unless ($self->[WW]) {
891 $self->[CB]->($self->[FH]);
892 @$self = ();
893 }
894}
895
896=item new_from_fh AnyEvent::Fork $fh # EXPERIMENTAL, MIGHT BE REMOVED
897
898Takes a file handle originally rceeived by the C<to_fh> method and creates
899a new C<AnyEvent:Fork> object. The child process itself will not change in
900any way, i.e. it will keep all the modifications done to it before calling
901C<to_fh>.
902
903The new object is very much like the original object, except that the
904C<pid> method will return C<undef> even if the process is a direct child.
905
906=cut
907
908sub new_from_fh {
909 my ($class, $fh) = @_;
910
911 $class->_new ($fh)
767} 912}
768 913
769=back 914=back
770 915
771=head1 PERFORMANCE 916=head1 PERFORMANCE
781 926
782 2079 new processes per second, using manual socketpair + fork 927 2079 new processes per second, using manual socketpair + fork
783 928
784Then I did the same thing, but instead of calling fork, I called 929Then I did the same thing, but instead of calling fork, I called
785AnyEvent::Fork->new->run ("CORE::exit") and then again waited for the 930AnyEvent::Fork->new->run ("CORE::exit") and then again waited for the
786socket form the child to close on exit. This does the same thing as manual 931socket from the child to close on exit. This does the same thing as manual
787socket pair + fork, except that what is forked is the template process 932socket pair + fork, except that what is forked is the template process
788(2440kB), and the socket needs to be passed to the server at the other end 933(2440kB), and the socket needs to be passed to the server at the other end
789of the socket first. 934of the socket first.
790 935
791 2307 new processes per second, using AnyEvent::Fork->new 936 2307 new processes per second, using AnyEvent::Fork->new
798So how can C<< AnyEvent->new >> be faster than a standard fork, even 943So how can C<< AnyEvent->new >> be faster than a standard fork, even
799though it uses the same operations, but adds a lot of overhead? 944though it uses the same operations, but adds a lot of overhead?
800 945
801The difference is simply the process size: forking the 5MB process takes 946The difference is simply the process size: forking the 5MB process takes
802so much longer than forking the 2.5MB template process that the extra 947so much longer than forking the 2.5MB template process that the extra
803overhead introduced is canceled out. 948overhead is canceled out.
804 949
805If the benchmark process grows, the normal fork becomes even slower: 950If the benchmark process grows, the normal fork becomes even slower:
806 951
807 1340 new processes, manual fork of a 20MB process 952 1340 new processes, manual fork of a 20MB process
808 731 new processes, manual fork of a 200MB process 953 731 new processes, manual fork of a 200MB process
868initialising them, for example, by calling C<init Gtk2> manually. 1013initialising them, for example, by calling C<init Gtk2> manually.
869 1014
870=item exiting calls object destructors 1015=item exiting calls object destructors
871 1016
872This only applies to users of L<AnyEvent::Fork:Early> and 1017This only applies to users of L<AnyEvent::Fork:Early> and
873L<AnyEvent::Fork::Template>, or when initialiasing code creates objects 1018L<AnyEvent::Fork::Template>, or when initialising code creates objects
874that reference external resources. 1019that reference external resources.
875 1020
876When a process created by AnyEvent::Fork exits, it might do so by calling 1021When a process created by AnyEvent::Fork exits, it might do so by calling
877exit, or simply letting perl reach the end of the program. At which point 1022exit, or simply letting perl reach the end of the program. At which point
878Perl runs all destructors. 1023Perl runs all destructors.
898to make it so, mostly due to the bloody broken perl that nobody seems to 1043to make it so, mostly due to the bloody broken perl that nobody seems to
899care about. The fork emulation is a bad joke - I have yet to see something 1044care about. The fork emulation is a bad joke - I have yet to see something
900useful that you can do with it without running into memory corruption 1045useful that you can do with it without running into memory corruption
901issues or other braindamage. Hrrrr. 1046issues or other braindamage. Hrrrr.
902 1047
1048Since fork is endlessly broken on win32 perls (it doesn't even remotely
1049work within it's documented limits) and quite obviously it's not getting
1050improved any time soon, the best way to proceed on windows would be to
1051always use C<new_exec> and thus never rely on perl's fork "emulation".
1052
903Cygwin perl is not supported at the moment due to some hilarious 1053Cygwin perl is not supported at the moment due to some hilarious
904shortcomings of its API - see L<IO::FDPoll> for more details. 1054shortcomings of its API - see L<IO::FDPoll> for more details. If you never
1055use C<send_fh> and always use C<new_exec> to create processes, it should
1056work though.
905 1057
906=head1 SEE ALSO 1058=head1 SEE ALSO
907 1059
908L<AnyEvent::Fork::Early> (to avoid executing a perl interpreter), 1060L<AnyEvent::Fork::Early>, to avoid executing a perl interpreter at all
1061(part of this distribution).
1062
909L<AnyEvent::Fork::Template> (to create a process by forking the main 1063L<AnyEvent::Fork::Template>, to create a process by forking the main
910program at a convenient time). 1064program at a convenient time (part of this distribution).
911 1065
912=head1 AUTHOR 1066L<AnyEvent::Fork::Remote>, for another way to create processes that is
1067mostly compatible to this module and modules building on top of it, but
1068works better with remote processes.
1069
1070L<AnyEvent::Fork::RPC>, for simple RPC to child processes (on CPAN).
1071
1072L<AnyEvent::Fork::Pool>, for simple worker process pool (on CPAN).
1073
1074=head1 AUTHOR AND CONTACT INFORMATION
913 1075
914 Marc Lehmann <schmorp@schmorp.de> 1076 Marc Lehmann <schmorp@schmorp.de>
915 http://home.schmorp.de/ 1077 http://software.schmorp.de/pkg/AnyEvent-Fork
916 1078
917=cut 1079=cut
918 1080
9191 10811
920 1082

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines