ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-Fork/Fork.pm
(Generate patch)

Comparing AnyEvent-Fork/Fork.pm (file contents):
Revision 1.33 by root, Sat Apr 6 09:34:11 2013 UTC vs.
Revision 1.57 by root, Sun Aug 25 17:38:43 2013 UTC

27 27
28Special care has been taken to make this module useful from other modules, 28Special care has been taken to make this module useful from other modules,
29while still supporting specialised environments such as L<App::Staticperl> 29while still supporting specialised environments such as L<App::Staticperl>
30or L<PAR::Packer>. 30or L<PAR::Packer>.
31 31
32=head1 WHAT THIS MODULE IS NOT 32=head2 WHAT THIS MODULE IS NOT
33 33
34This module only creates processes and lets you pass file handles and 34This module only creates processes and lets you pass file handles and
35strings to it, and run perl code. It does not implement any kind of RPC - 35strings to it, and run perl code. It does not implement any kind of RPC -
36there is no back channel from the process back to you, and there is no RPC 36there is no back channel from the process back to you, and there is no RPC
37or message passing going on. 37or message passing going on.
38 38
39If you need some form of RPC, you can either implement it yourself 39If you need some form of RPC, you could use the L<AnyEvent::Fork::RPC>
40in whatever way you like, use some message-passing module such 40companion module, which adds simple RPC/job queueing to a process created
41as L<AnyEvent::MP>, some pipe such as L<AnyEvent::ZeroMQ>, use 41by this module.
42L<AnyEvent::Handle> on both sides to send e.g. JSON or Storable messages,
43and so on.
44 42
43And if you need some automatic process pool management on top of
44L<AnyEvent::Fork::RPC>, you can look at the L<AnyEvent::Fork::Pool>
45companion module.
46
47Or you can implement it yourself in whatever way you like: use some
48message-passing module such as L<AnyEvent::MP>, some pipe such as
49L<AnyEvent::ZeroMQ>, use L<AnyEvent::Handle> on both sides to send
50e.g. JSON or Storable messages, and so on.
51
52=head2 COMPARISON TO OTHER MODULES
53
54There is an abundance of modules on CPAN that do "something fork", such as
55L<Parallel::ForkManager>, L<AnyEvent::ForkManager>, L<AnyEvent::Worker>
56or L<AnyEvent::Subprocess>. There are modules that implement their own
57process management, such as L<AnyEvent::DBI>.
58
59The problems that all these modules try to solve are real, however, none
60of them (from what I have seen) tackle the very real problems of unwanted
61memory sharing, efficiency, not being able to use event processing or
62similar modules in the processes they create.
63
64This module doesn't try to replace any of them - instead it tries to solve
65the problem of creating processes with a minimum of fuss and overhead (and
66also luxury). Ideally, most of these would use AnyEvent::Fork internally,
67except they were written before AnyEvent:Fork was available, so obviously
68had to roll their own.
69
45=head1 PROBLEM STATEMENT 70=head2 PROBLEM STATEMENT
46 71
47There are two traditional ways to implement parallel processing on UNIX 72There are two traditional ways to implement parallel processing on UNIX
48like operating systems - fork and process, and fork+exec and process. They 73like operating systems - fork and process, and fork+exec and process. They
49have different advantages and disadvantages that I describe below, 74have different advantages and disadvantages that I describe below,
50together with how this module tries to mitigate the disadvantages. 75together with how this module tries to mitigate the disadvantages.
203 } 228 }
204 } 229 }
205 230
206=head2 use AnyEvent::Fork as a faster fork+exec 231=head2 use AnyEvent::Fork as a faster fork+exec
207 232
208This runs C</bin/echo hi>, with stdandard output redirected to /tmp/log 233This runs C</bin/echo hi>, with standard output redirected to F</tmp/log>
209and standard error redirected to the communications socket. It is usually 234and standard error redirected to the communications socket. It is usually
210faster than fork+exec, but still lets you prepare the environment. 235faster than fork+exec, but still lets you prepare the environment.
211 236
212 open my $output, ">/tmp/log" or die "$!"; 237 open my $output, ">/tmp/log" or die "$!";
213 238
214 AnyEvent::Fork 239 AnyEvent::Fork
215 ->new 240 ->new
216 ->eval (' 241 ->eval ('
242 # compile a helper function for later use
217 sub run { 243 sub run {
218 my ($fh, $output, @cmd) = @_; 244 my ($fh, $output, @cmd) = @_;
219 245
220 # perl will clear close-on-exec on STDOUT/STDERR 246 # perl will clear close-on-exec on STDOUT/STDERR
221 open STDOUT, ">&", $output or die; 247 open STDOUT, ">&", $output or die;
228 ->send_arg ("/bin/echo", "hi") 254 ->send_arg ("/bin/echo", "hi")
229 ->run ("run", my $cv = AE::cv); 255 ->run ("run", my $cv = AE::cv);
230 256
231 my $stderr = $cv->recv; 257 my $stderr = $cv->recv;
232 258
259=head2 For stingy users: put the worker code into a C<DATA> section.
260
261When you want to be stingy with files, you cna put your code into the
262C<DATA> section of your module (or program):
263
264 use AnyEvent::Fork;
265
266 AnyEvent::Fork
267 ->new
268 ->eval (do { local $/; <DATA> })
269 ->run ("doit", sub { ... });
270
271 __DATA__
272
273 sub doit {
274 ... do something!
275 }
276
277=head2 For stingy standalone programs: do not rely on external files at
278all.
279
280For single-file scripts it can be inconvenient to rely on external
281files - even when using < C<DATA> section, you still need to C<exec>
282an external perl interpreter, which might not be available when using
283L<App::Staticperl>, L<Urlader> or L<PAR::Packer> for example.
284
285Two modules help here - L<AnyEvent::Fork::Early> forks a template process
286for all further calls to C<new_exec>, and L<AnyEvent::Fork::Template>
287forks the main program as a template process.
288
289Here is how your main program should look like:
290
291 #! perl
292
293 # optional, as the very first thing.
294 # in case modules want to create their own processes.
295 use AnyEvent::Fork::Early;
296
297 # next, load all modules you need in your template process
298 use Example::My::Module
299 use Example::Whatever;
300
301 # next, put your run function definition and anything else you
302 # need, but do not use code outside of BEGIN blocks.
303 sub worker_run {
304 my ($fh, @args) = @_;
305 ...
306 }
307
308 # now preserve everything so far as AnyEvent::Fork object
309 # in §TEMPLATE.
310 use AnyEvent::Fork::Template;
311
312 # do not put code outside of BEGIN blocks until here
313
314 # now use the $TEMPLATE process in any way you like
315
316 # for example: create 10 worker processes
317 my @worker;
318 my $cv = AE::cv;
319 for (1..10) {
320 $cv->begin;
321 $TEMPLATE->fork->send_arg ($_)->run ("worker_run", sub {
322 push @worker, shift;
323 $cv->end;
324 });
325 }
326 $cv->recv;
327
233=head1 CONCEPTS 328=head1 CONCEPTS
234 329
235This module can create new processes either by executing a new perl 330This module can create new processes either by executing a new perl
236process, or by forking from an existing "template" process. 331process, or by forking from an existing "template" process.
332
333All these processes are called "child processes" (whether they are direct
334children or not), while the process that manages them is called the
335"parent process".
237 336
238Each such process comes with its own file handle that can be used to 337Each such process comes with its own file handle that can be used to
239communicate with it (it's actually a socket - one end in the new process, 338communicate with it (it's actually a socket - one end in the new process,
240one end in the main process), and among the things you can do in it are 339one end in the main process), and among the things you can do in it are
241load modules, fork new processes, send file handles to it, and execute 340load modules, fork new processes, send file handles to it, and execute
351use AnyEvent; 450use AnyEvent;
352use AnyEvent::Util (); 451use AnyEvent::Util ();
353 452
354use IO::FDPass; 453use IO::FDPass;
355 454
356our $VERSION = 0.5; 455our $VERSION = 1.1;
357
358our $PERL; # the path to the perl interpreter, deduces with various forms of magic
359
360=over 4
361
362=back
363
364=cut
365 456
366# the early fork template process 457# the early fork template process
367our $EARLY; 458our $EARLY;
368 459
369# the empty template process 460# the empty template process
370our $TEMPLATE; 461our $TEMPLATE;
462
463sub QUEUE() { 0 }
464sub FH() { 1 }
465sub WW() { 2 }
466sub PID() { 3 }
467sub CB() { 4 }
468
469sub _new {
470 my ($self, $fh, $pid) = @_;
471
472 AnyEvent::Util::fh_nonblocking $fh, 1;
473
474 $self = bless [
475 [], # write queue - strings or fd's
476 $fh,
477 undef, # AE watcher
478 $pid,
479 ], $self;
480
481 $self
482}
371 483
372sub _cmd { 484sub _cmd {
373 my $self = shift; 485 my $self = shift;
374 486
375 # ideally, we would want to use "a (w/a)*" as format string, but perl 487 # ideally, we would want to use "a (w/a)*" as format string, but perl
376 # versions from at least 5.8.9 to 5.16.3 are all buggy and can't unpack 488 # versions from at least 5.8.9 to 5.16.3 are all buggy and can't unpack
377 # it. 489 # it.
378 push @{ $self->[2] }, pack "a L/a*", $_[0], $_[1]; 490 push @{ $self->[QUEUE] }, pack "a L/a*", $_[0], $_[1];
379 491
380 $self->[3] ||= AE::io $self->[1], 1, sub { 492 $self->[WW] ||= AE::io $self->[FH], 1, sub {
381 do { 493 do {
382 # send the next "thing" in the queue - either a reference to an fh, 494 # send the next "thing" in the queue - either a reference to an fh,
383 # or a plain string. 495 # or a plain string.
384 496
385 if (ref $self->[2][0]) { 497 if (ref $self->[QUEUE][0]) {
386 # send fh 498 # send fh
387 unless (IO::FDPass::send fileno $self->[1], fileno ${ $self->[2][0] }) { 499 unless (IO::FDPass::send fileno $self->[FH], fileno ${ $self->[QUEUE][0] }) {
388 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK; 500 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK;
389 undef $self->[3]; 501 undef $self->[WW];
390 die "AnyEvent::Fork: file descriptor send failure: $!"; 502 die "AnyEvent::Fork: file descriptor send failure: $!";
391 } 503 }
392 504
393 shift @{ $self->[2] }; 505 shift @{ $self->[QUEUE] };
394 506
395 } else { 507 } else {
396 # send string 508 # send string
397 my $len = syswrite $self->[1], $self->[2][0]; 509 my $len = syswrite $self->[FH], $self->[QUEUE][0];
398 510
399 unless ($len) { 511 unless ($len) {
400 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK; 512 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK;
401 undef $self->[3]; 513 undef $self->[WW];
402 die "AnyEvent::Fork: command write failure: $!"; 514 die "AnyEvent::Fork: command write failure: $!";
403 } 515 }
404 516
405 substr $self->[2][0], 0, $len, ""; 517 substr $self->[QUEUE][0], 0, $len, "";
406 shift @{ $self->[2] } unless length $self->[2][0]; 518 shift @{ $self->[QUEUE] } unless length $self->[QUEUE][0];
407 } 519 }
408 } while @{ $self->[2] }; 520 } while @{ $self->[QUEUE] };
409 521
410 # everything written 522 # everything written
411 undef $self->[3]; 523 undef $self->[WW];
412 524
413 # invoke run callback, if any 525 # invoke run callback, if any
414 $self->[4]->($self->[1]) if $self->[4]; 526 if ($self->[CB]) {
527 $self->[CB]->($self->[FH]);
528 @$self = ();
529 }
415 }; 530 };
416 531
417 () # make sure we don't leak the watcher 532 () # make sure we don't leak the watcher
418}
419
420sub _new {
421 my ($self, $fh, $pid) = @_;
422
423 AnyEvent::Util::fh_nonblocking $fh, 1;
424
425 $self = bless [
426 $pid,
427 $fh,
428 [], # write queue - strings or fd's
429 undef, # AE watcher
430 ], $self;
431
432 $self
433} 533}
434 534
435# fork template from current process, used by AnyEvent::Fork::Early/Template 535# fork template from current process, used by AnyEvent::Fork::Early/Template
436sub _new_fork { 536sub _new_fork {
437 my ($fh, $slave) = AnyEvent::Util::portable_socketpair; 537 my ($fh, $slave) = AnyEvent::Util::portable_socketpair;
442 if ($pid eq 0) { 542 if ($pid eq 0) {
443 require AnyEvent::Fork::Serve; 543 require AnyEvent::Fork::Serve;
444 $AnyEvent::Fork::Serve::OWNER = $parent; 544 $AnyEvent::Fork::Serve::OWNER = $parent;
445 close $fh; 545 close $fh;
446 $0 = "$_[1] of $parent"; 546 $0 = "$_[1] of $parent";
447 $SIG{CHLD} = 'IGNORE';
448 AnyEvent::Fork::Serve::serve ($slave); 547 AnyEvent::Fork::Serve::serve ($slave);
449 exit 0; 548 exit 0;
450 } elsif (!$pid) { 549 } elsif (!$pid) {
451 die "AnyEvent::Fork::Early/Template: unable to fork template process: $!"; 550 die "AnyEvent::Fork::Early/Template: unable to fork template process: $!";
452 } 551 }
506 605
507You should use C<new> whenever possible, except when having a template 606You should use C<new> whenever possible, except when having a template
508process around is unacceptable. 607process around is unacceptable.
509 608
510The path to the perl interpreter is divined using various methods - first 609The path to the perl interpreter is divined using various methods - first
511C<$^X> is investigated to see if the path ends with something that sounds 610C<$^X> is investigated to see if the path ends with something that looks
512as if it were the perl interpreter. Failing this, the module falls back to 611as if it were the perl interpreter. Failing this, the module falls back to
513using C<$Config::Config{perlpath}>. 612using C<$Config::Config{perlpath}>.
514 613
614The path to perl can also be overriden by setting the global variable
615C<$AnyEvent::Fork::PERL> - it's value will be used for all subsequent
616invocations.
617
515=cut 618=cut
619
620our $PERL;
516 621
517sub new_exec { 622sub new_exec {
518 my ($self) = @_; 623 my ($self) = @_;
519 624
520 return $EARLY->fork 625 return $EARLY->fork
521 if $EARLY; 626 if $EARLY;
522 627
628 unless (defined $PERL) {
523 # first find path of perl 629 # first find path of perl
524 my $perl = $; 630 my $perl = $;
525 631
526 # first we try $^X, but the path must be absolute (always on win32), and end in sth. 632 # first we try $^X, but the path must be absolute (always on win32), and end in sth.
527 # that looks like perl. this obviously only works for posix and win32 633 # that looks like perl. this obviously only works for posix and win32
528 unless ( 634 unless (
529 ($^O eq "MSWin32" || $perl =~ m%^/%) 635 ($^O eq "MSWin32" || $perl =~ m%^/%)
530 && $perl =~ m%[/\\]perl(?:[0-9]+(\.[0-9]+)+)?(\.exe)?$%i 636 && $perl =~ m%[/\\]perl(?:[0-9]+(\.[0-9]+)+)?(\.exe)?$%i
531 ) { 637 ) {
532 # if it doesn't look perlish enough, try Config 638 # if it doesn't look perlish enough, try Config
533 require Config; 639 require Config;
534 $perl = $Config::Config{perlpath}; 640 $perl = $Config::Config{perlpath};
535 $perl =~ s/(?:\Q$Config::Config{_exe}\E)?$/$Config::Config{_exe}/; 641 $perl =~ s/(?:\Q$Config::Config{_exe}\E)?$/$Config::Config{_exe}/;
642 }
643
644 $PERL = $perl;
536 } 645 }
537 646
538 require Proc::FastSpawn; 647 require Proc::FastSpawn;
539 648
540 my ($fh, $slave) = AnyEvent::Util::portable_socketpair; 649 my ($fh, $slave) = AnyEvent::Util::portable_socketpair;
548 #local $ENV{PERL5LIB} = join ":", grep !ref, @INC; 657 #local $ENV{PERL5LIB} = join ":", grep !ref, @INC;
549 my %env = %ENV; 658 my %env = %ENV;
550 $env{PERL5LIB} = join +($^O eq "MSWin32" ? ";" : ":"), grep !ref, @INC; 659 $env{PERL5LIB} = join +($^O eq "MSWin32" ? ";" : ":"), grep !ref, @INC;
551 660
552 my $pid = Proc::FastSpawn::spawn ( 661 my $pid = Proc::FastSpawn::spawn (
553 $perl, 662 $PERL,
554 ["perl", "-MAnyEvent::Fork::Serve", "-e", "AnyEvent::Fork::Serve::me", fileno $slave, $$], 663 ["perl", "-MAnyEvent::Fork::Serve", "-e", "AnyEvent::Fork::Serve::me", fileno $slave, $$],
555 [map "$_=$env{$_}", keys %env], 664 [map "$_=$env{$_}", keys %env],
556 ) or die "unable to spawn AnyEvent::Fork server: $!"; 665 ) or die "unable to spawn AnyEvent::Fork server: $!";
557 666
558 $self->_new ($fh, $pid) 667 $self->_new ($fh, $pid)
571AnyEvent::Fork itself. 680AnyEvent::Fork itself.
572 681
573=cut 682=cut
574 683
575sub pid { 684sub pid {
576 $_[0][0] 685 $_[0][PID]
577} 686}
578 687
579=item $proc = $proc->eval ($perlcode, @args) 688=item $proc = $proc->eval ($perlcode, @args)
580 689
581Evaluates the given C<$perlcode> as ... perl code, while setting C<@_> to 690Evaluates the given C<$perlcode> as ... Perl code, while setting C<@_> to
582the strings specified by C<@args>, in the "main" package. 691the strings specified by C<@args>, in the "main" package.
583 692
584This call is meant to do any custom initialisation that might be required 693This call is meant to do any custom initialisation that might be required
585(for example, the C<require> method uses it). It's not supposed to be used 694(for example, the C<require> method uses it). It's not supposed to be used
586to completely take over the process, use C<run> for that. 695to completely take over the process, use C<run> for that.
591 700
592If you want to execute some code (that isn't in a module) to take over the 701If you want to execute some code (that isn't in a module) to take over the
593process, you should compile a function via C<eval> first, and then call 702process, you should compile a function via C<eval> first, and then call
594it via C<run>. This also gives you access to any arguments passed via the 703it via C<run>. This also gives you access to any arguments passed via the
595C<send_xxx> methods, such as file handles. See the L<use AnyEvent::Fork as 704C<send_xxx> methods, such as file handles. See the L<use AnyEvent::Fork as
596a faster fork+exec> example. 705a faster fork+exec> example to see it in action.
597 706
598Returns the process object for easy chaining of method calls. 707Returns the process object for easy chaining of method calls.
599 708
600=cut 709=cut
601 710
627=item $proc = $proc->send_fh ($handle, ...) 736=item $proc = $proc->send_fh ($handle, ...)
628 737
629Send one or more file handles (I<not> file descriptors) to the process, 738Send one or more file handles (I<not> file descriptors) to the process,
630to prepare a call to C<run>. 739to prepare a call to C<run>.
631 740
632The process object keeps a reference to the handles until this is done, 741The process object keeps a reference to the handles until they have
633so you must not explicitly close the handles. This is most easily 742been passed over to the process, so you must not explicitly close the
634accomplished by simply not storing the file handles anywhere after passing 743handles. This is most easily accomplished by simply not storing the file
635them to this method. 744handles anywhere after passing them to this method - when AnyEvent::Fork
745is finished using them, perl will automatically close them.
636 746
637Returns the process object for easy chaining of method calls. 747Returns the process object for easy chaining of method calls.
638 748
639Example: pass a file handle to a process, and release it without 749Example: pass a file handle to a process, and release it without
640closing. It will be closed automatically when it is no longer used. 750closing. It will be closed automatically when it is no longer used.
647sub send_fh { 757sub send_fh {
648 my ($self, @fh) = @_; 758 my ($self, @fh) = @_;
649 759
650 for my $fh (@fh) { 760 for my $fh (@fh) {
651 $self->_cmd ("h"); 761 $self->_cmd ("h");
652 push @{ $self->[2] }, \$fh; 762 push @{ $self->[QUEUE] }, \$fh;
653 } 763 }
654 764
655 $self 765 $self
656} 766}
657 767
658=item $proc = $proc->send_arg ($string, ...) 768=item $proc = $proc->send_arg ($string, ...)
659 769
660Send one or more argument strings to the process, to prepare a call to 770Send one or more argument strings to the process, to prepare a call to
661C<run>. The strings can be any octet string. 771C<run>. The strings can be any octet strings.
662 772
663The protocol is optimised to pass a moderate number of relatively short 773The protocol is optimised to pass a moderate number of relatively short
664strings - while you can pass up to 4GB of data in one go, this is more 774strings - while you can pass up to 4GB of data in one go, this is more
665meant to pass some ID information or other startup info, not big chunks of 775meant to pass some ID information or other startup info, not big chunks of
666data. 776data.
682Enter the function specified by the function name in C<$func> in the 792Enter the function specified by the function name in C<$func> in the
683process. The function is called with the communication socket as first 793process. The function is called with the communication socket as first
684argument, followed by all file handles and string arguments sent earlier 794argument, followed by all file handles and string arguments sent earlier
685via C<send_fh> and C<send_arg> methods, in the order they were called. 795via C<send_fh> and C<send_arg> methods, in the order they were called.
686 796
797The process object becomes unusable on return from this function - any
798further method calls result in undefined behaviour.
799
687The function name should be fully qualified, but if it isn't, it will be 800The function name should be fully qualified, but if it isn't, it will be
688looked up in the main package. 801looked up in the C<main> package.
689 802
690If the called function returns, doesn't exist, or any error occurs, the 803If the called function returns, doesn't exist, or any error occurs, the
691process exits. 804process exits.
692 805
693Preparing the process is done in the background - when all commands have 806Preparing the process is done in the background - when all commands have
694been sent, the callback is invoked with the local communications socket 807been sent, the callback is invoked with the local communications socket
695as argument. At this point you can start using the socket in any way you 808as argument. At this point you can start using the socket in any way you
696like. 809like.
697
698The process object becomes unusable on return from this function - any
699further method calls result in undefined behaviour.
700 810
701If the communication socket isn't used, it should be closed on both sides, 811If the communication socket isn't used, it should be closed on both sides,
702to save on kernel memory. 812to save on kernel memory.
703 813
704The socket is non-blocking in the parent, and blocking in the newly 814The socket is non-blocking in the parent, and blocking in the newly
743=cut 853=cut
744 854
745sub run { 855sub run {
746 my ($self, $func, $cb) = @_; 856 my ($self, $func, $cb) = @_;
747 857
748 $self->[4] = $cb; 858 $self->[CB] = $cb;
749 $self->_cmd (r => $func); 859 $self->_cmd (r => $func);
860}
861
862=back
863
864=head2 EXPERIMENTAL METHODS
865
866These methods might go away completely or change behaviour, at any time.
867
868=over 4
869
870=item $proc->to_fh ($cb->($fh)) # EXPERIMENTAL, MIGHT BE REMOVED
871
872Flushes all commands out to the process and then calls the callback with
873the communications socket.
874
875The process object becomes unusable on return from this function - any
876further method calls result in undefined behaviour.
877
878The point of this method is to give you a file handle thta you cna pass
879to another process. In that other process, you can call C<new_from_fh
880AnyEvent::Fork> to create a new C<AnyEvent::Fork> object from it, thereby
881effectively passing a fork object to another process.
882
883=cut
884
885sub to_fh {
886 my ($self, $cb) = @_;
887
888 $self->[CB] = $cb;
889
890 unless ($self->[WW]) {
891 $self->[CB]->($self->[FH]);
892 @$self = ();
893 }
894}
895
896=item new_from_fh AnyEvent::Fork $fh # EXPERIMENTAL, MIGHT BE REMOVED
897
898Takes a file handle originally rceeived by the C<to_fh> method and creates
899a new C<AnyEvent:Fork> object. The child process itself will not change in
900any way, i.e. it will keep all the modifications done to it before calling
901C<to_fh>.
902
903The new object is very much like the original object, except that the
904C<pid> method will return C<undef> even if the process is a direct child.
905
906=cut
907
908sub new_from_fh {
909 my ($class, $fh) = @_;
910
911 $class->_new ($fh)
750} 912}
751 913
752=back 914=back
753 915
754=head1 PERFORMANCE 916=head1 PERFORMANCE
764 926
765 2079 new processes per second, using manual socketpair + fork 927 2079 new processes per second, using manual socketpair + fork
766 928
767Then I did the same thing, but instead of calling fork, I called 929Then I did the same thing, but instead of calling fork, I called
768AnyEvent::Fork->new->run ("CORE::exit") and then again waited for the 930AnyEvent::Fork->new->run ("CORE::exit") and then again waited for the
769socket form the child to close on exit. This does the same thing as manual 931socket from the child to close on exit. This does the same thing as manual
770socket pair + fork, except that what is forked is the template process 932socket pair + fork, except that what is forked is the template process
771(2440kB), and the socket needs to be passed to the server at the other end 933(2440kB), and the socket needs to be passed to the server at the other end
772of the socket first. 934of the socket first.
773 935
774 2307 new processes per second, using AnyEvent::Fork->new 936 2307 new processes per second, using AnyEvent::Fork->new
779 479 vfork+execs per second, using AnyEvent::Fork->new_exec 941 479 vfork+execs per second, using AnyEvent::Fork->new_exec
780 942
781So how can C<< AnyEvent->new >> be faster than a standard fork, even 943So how can C<< AnyEvent->new >> be faster than a standard fork, even
782though it uses the same operations, but adds a lot of overhead? 944though it uses the same operations, but adds a lot of overhead?
783 945
784The difference is simply the process size: forking the 6MB process takes 946The difference is simply the process size: forking the 5MB process takes
785so much longer than forking the 2.5MB template process that the overhead 947so much longer than forking the 2.5MB template process that the extra
786introduced is canceled out. 948overhead is canceled out.
787 949
788If the benchmark process grows, the normal fork becomes even slower: 950If the benchmark process grows, the normal fork becomes even slower:
789 951
790 1340 new processes, manual fork in a 20MB process 952 1340 new processes, manual fork of a 20MB process
791 731 new processes, manual fork in a 200MB process 953 731 new processes, manual fork of a 200MB process
792 235 new processes, manual fork in a 2000MB process 954 235 new processes, manual fork of a 2000MB process
793 955
794What that means (to me) is that I can use this module without having a 956What that means (to me) is that I can use this module without having a bad
795very bad conscience because of the extra overhead required to start new 957conscience because of the extra overhead required to start new processes.
796processes.
797 958
798=head1 TYPICAL PROBLEMS 959=head1 TYPICAL PROBLEMS
799 960
800This section lists typical problems that remain. I hope by recognising 961This section lists typical problems that remain. I hope by recognising
801them, most can be avoided. 962them, most can be avoided.
802 963
803=over 4 964=over 4
804 965
805=item "leaked" file descriptors for exec'ed processes 966=item leaked file descriptors for exec'ed processes
806 967
807POSIX systems inherit file descriptors by default when exec'ing a new 968POSIX systems inherit file descriptors by default when exec'ing a new
808process. While perl itself laudably sets the close-on-exec flags on new 969process. While perl itself laudably sets the close-on-exec flags on new
809file handles, most C libraries don't care, and even if all cared, it's 970file handles, most C libraries don't care, and even if all cared, it's
810often not possible to set the flag in a race-free manner. 971often not possible to set the flag in a race-free manner.
830libraries or the code that leaks those file descriptors. 991libraries or the code that leaks those file descriptors.
831 992
832Fortunately, most of these leaked descriptors do no harm, other than 993Fortunately, most of these leaked descriptors do no harm, other than
833sitting on some resources. 994sitting on some resources.
834 995
835=item "leaked" file descriptors for fork'ed processes 996=item leaked file descriptors for fork'ed processes
836 997
837Normally, L<AnyEvent::Fork> does start new processes by exec'ing them, 998Normally, L<AnyEvent::Fork> does start new processes by exec'ing them,
838which closes file descriptors not marked for being inherited. 999which closes file descriptors not marked for being inherited.
839 1000
840However, L<AnyEvent::Fork::Early> and L<AnyEvent::Fork::Template> offer 1001However, L<AnyEvent::Fork::Early> and L<AnyEvent::Fork::Template> offer
849 1010
850The solution is to either not load these modules before use'ing 1011The solution is to either not load these modules before use'ing
851L<AnyEvent::Fork::Early> or L<AnyEvent::Fork::Template>, or to delay 1012L<AnyEvent::Fork::Early> or L<AnyEvent::Fork::Template>, or to delay
852initialising them, for example, by calling C<init Gtk2> manually. 1013initialising them, for example, by calling C<init Gtk2> manually.
853 1014
854=item exit runs destructors 1015=item exiting calls object destructors
855 1016
856This only applies to users of Lc<AnyEvent::Fork:Early> and 1017This only applies to users of L<AnyEvent::Fork:Early> and
857L<AnyEvent::Fork::Template>. 1018L<AnyEvent::Fork::Template>, or when initialising code creates objects
1019that reference external resources.
858 1020
859When a process created by AnyEvent::Fork exits, it might do so by calling 1021When a process created by AnyEvent::Fork exits, it might do so by calling
860exit, or simply letting perl reach the end of the program. At which point 1022exit, or simply letting perl reach the end of the program. At which point
861Perl runs all destructors. 1023Perl runs all destructors.
862 1024
881to make it so, mostly due to the bloody broken perl that nobody seems to 1043to make it so, mostly due to the bloody broken perl that nobody seems to
882care about. The fork emulation is a bad joke - I have yet to see something 1044care about. The fork emulation is a bad joke - I have yet to see something
883useful that you can do with it without running into memory corruption 1045useful that you can do with it without running into memory corruption
884issues or other braindamage. Hrrrr. 1046issues or other braindamage. Hrrrr.
885 1047
886Cygwin perl is not supported at the moment, as it should implement fd 1048Since fork is endlessly broken on win32 perls (it doesn't even remotely
887passing, but doesn't, and rolling my own is hard, as cygwin doesn't 1049work within it's documented limits) and quite obviously it's not getting
888support enough functionality to do it. 1050improved any time soon, the best way to proceed on windows would be to
1051always use C<new_exec> and thus never rely on perl's fork "emulation".
1052
1053Cygwin perl is not supported at the moment due to some hilarious
1054shortcomings of its API - see L<IO::FDPoll> for more details. If you never
1055use C<send_fh> and always use C<new_exec> to create processes, it should
1056work though.
889 1057
890=head1 SEE ALSO 1058=head1 SEE ALSO
891 1059
892L<AnyEvent::Fork::Early> (to avoid executing a perl interpreter), 1060L<AnyEvent::Fork::Early>, to avoid executing a perl interpreter at all
1061(part of this distribution).
1062
893L<AnyEvent::Fork::Template> (to create a process by forking the main 1063L<AnyEvent::Fork::Template>, to create a process by forking the main
894program at a convenient time). 1064program at a convenient time (part of this distribution).
895 1065
896=head1 AUTHOR 1066L<AnyEvent::Fork::Remote>, for another way to create processes that is
1067mostly compatible to this module and modules building on top of it, but
1068works better with remote processes.
1069
1070L<AnyEvent::Fork::RPC>, for simple RPC to child processes (on CPAN).
1071
1072L<AnyEvent::Fork::Pool>, for simple worker process pool (on CPAN).
1073
1074=head1 AUTHOR AND CONTACT INFORMATION
897 1075
898 Marc Lehmann <schmorp@schmorp.de> 1076 Marc Lehmann <schmorp@schmorp.de>
899 http://home.schmorp.de/ 1077 http://software.schmorp.de/pkg/AnyEvent-Fork
900 1078
901=cut 1079=cut
902 1080
9031 10811
904 1082

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines