ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-Fork/Fork.pm
(Generate patch)

Comparing AnyEvent-Fork/Fork.pm (file contents):
Revision 1.31 by root, Sat Apr 6 09:29:26 2013 UTC vs.
Revision 1.58 by root, Sun Aug 25 21:10:10 2013 UTC

27 27
28Special care has been taken to make this module useful from other modules, 28Special care has been taken to make this module useful from other modules,
29while still supporting specialised environments such as L<App::Staticperl> 29while still supporting specialised environments such as L<App::Staticperl>
30or L<PAR::Packer>. 30or L<PAR::Packer>.
31 31
32=head1 WHAT THIS MODULE IS NOT 32=head2 WHAT THIS MODULE IS NOT
33 33
34This module only creates processes and lets you pass file handles and 34This module only creates processes and lets you pass file handles and
35strings to it, and run perl code. It does not implement any kind of RPC - 35strings to it, and run perl code. It does not implement any kind of RPC -
36there is no back channel from the process back to you, and there is no RPC 36there is no back channel from the process back to you, and there is no RPC
37or message passing going on. 37or message passing going on.
38 38
39If you need some form of RPC, you can either implement it yourself 39If you need some form of RPC, you could use the L<AnyEvent::Fork::RPC>
40in whatever way you like, use some message-passing module such 40companion module, which adds simple RPC/job queueing to a process created
41as L<AnyEvent::MP>, some pipe such as L<AnyEvent::ZeroMQ>, use 41by this module.
42L<AnyEvent::Handle> on both sides to send e.g. JSON or Storable messages,
43and so on.
44 42
43And if you need some automatic process pool management on top of
44L<AnyEvent::Fork::RPC>, you can look at the L<AnyEvent::Fork::Pool>
45companion module.
46
47Or you can implement it yourself in whatever way you like: use some
48message-passing module such as L<AnyEvent::MP>, some pipe such as
49L<AnyEvent::ZeroMQ>, use L<AnyEvent::Handle> on both sides to send
50e.g. JSON or Storable messages, and so on.
51
52=head2 COMPARISON TO OTHER MODULES
53
54There is an abundance of modules on CPAN that do "something fork", such as
55L<Parallel::ForkManager>, L<AnyEvent::ForkManager>, L<AnyEvent::Worker>
56or L<AnyEvent::Subprocess>. There are modules that implement their own
57process management, such as L<AnyEvent::DBI>.
58
59The problems that all these modules try to solve are real, however, none
60of them (from what I have seen) tackle the very real problems of unwanted
61memory sharing, efficiency, not being able to use event processing or
62similar modules in the processes they create.
63
64This module doesn't try to replace any of them - instead it tries to solve
65the problem of creating processes with a minimum of fuss and overhead (and
66also luxury). Ideally, most of these would use AnyEvent::Fork internally,
67except they were written before AnyEvent:Fork was available, so obviously
68had to roll their own.
69
45=head1 PROBLEM STATEMENT 70=head2 PROBLEM STATEMENT
46 71
47There are two traditional ways to implement parallel processing on UNIX 72There are two traditional ways to implement parallel processing on UNIX
48like operating systems - fork and process, and fork+exec and process. They 73like operating systems - fork and process, and fork+exec and process. They
49have different advantages and disadvantages that I describe below, 74have different advantages and disadvantages that I describe below,
50together with how this module tries to mitigate the disadvantages. 75together with how this module tries to mitigate the disadvantages.
152 177
153 # now $master_filehandle is connected to the 178 # now $master_filehandle is connected to the
154 # $slave_filehandle in the new process. 179 # $slave_filehandle in the new process.
155 }); 180 });
156 181
157MyModule might look like this: 182C<MyModule> might look like this:
158 183
159 package MyModule; 184 package MyModule;
160 185
161 sub worker { 186 sub worker {
162 my ($slave_filehandle) = @_; 187 my ($slave_filehandle) = @_;
185 } 210 }
186 211
187 # now do other things - maybe use the filehandle provided by run 212 # now do other things - maybe use the filehandle provided by run
188 # to wait for the processes to die. or whatever. 213 # to wait for the processes to die. or whatever.
189 214
190My::Server might look like this: 215C<My::Server> might look like this:
191 216
192 package My::Server; 217 package My::Server;
193 218
194 sub run { 219 sub run {
195 my ($slave, $listener, $id) = @_; 220 my ($slave, $listener, $id) = @_;
203 } 228 }
204 } 229 }
205 230
206=head2 use AnyEvent::Fork as a faster fork+exec 231=head2 use AnyEvent::Fork as a faster fork+exec
207 232
208This runs /bin/echo hi, with stdout redirected to /tmp/log and stderr to 233This runs C</bin/echo hi>, with standard output redirected to F</tmp/log>
209the communications socket. It is usually faster than fork+exec, but still 234and standard error redirected to the communications socket. It is usually
210let's you prepare the environment. 235faster than fork+exec, but still lets you prepare the environment.
211 236
212 open my $output, ">/tmp/log" or die "$!"; 237 open my $output, ">/tmp/log" or die "$!";
213 238
214 AnyEvent::Fork 239 AnyEvent::Fork
215 ->new 240 ->new
216 ->eval (' 241 ->eval ('
242 # compile a helper function for later use
217 sub run { 243 sub run {
218 my ($fh, $output, @cmd) = @_; 244 my ($fh, $output, @cmd) = @_;
219 245
220 # perl will clear close-on-exec on STDOUT/STDERR 246 # perl will clear close-on-exec on STDOUT/STDERR
221 open STDOUT, ">&", $output or die; 247 open STDOUT, ">&", $output or die;
228 ->send_arg ("/bin/echo", "hi") 254 ->send_arg ("/bin/echo", "hi")
229 ->run ("run", my $cv = AE::cv); 255 ->run ("run", my $cv = AE::cv);
230 256
231 my $stderr = $cv->recv; 257 my $stderr = $cv->recv;
232 258
259=head2 For stingy users: put the worker code into a C<DATA> section.
260
261When you want to be stingy with files, you cna put your code into the
262C<DATA> section of your module (or program):
263
264 use AnyEvent::Fork;
265
266 AnyEvent::Fork
267 ->new
268 ->eval (do { local $/; <DATA> })
269 ->run ("doit", sub { ... });
270
271 __DATA__
272
273 sub doit {
274 ... do something!
275 }
276
277=head2 For stingy standalone programs: do not rely on external files at
278all.
279
280For single-file scripts it can be inconvenient to rely on external
281files - even when using < C<DATA> section, you still need to C<exec>
282an external perl interpreter, which might not be available when using
283L<App::Staticperl>, L<Urlader> or L<PAR::Packer> for example.
284
285Two modules help here - L<AnyEvent::Fork::Early> forks a template process
286for all further calls to C<new_exec>, and L<AnyEvent::Fork::Template>
287forks the main program as a template process.
288
289Here is how your main program should look like:
290
291 #! perl
292
293 # optional, as the very first thing.
294 # in case modules want to create their own processes.
295 use AnyEvent::Fork::Early;
296
297 # next, load all modules you need in your template process
298 use Example::My::Module
299 use Example::Whatever;
300
301 # next, put your run function definition and anything else you
302 # need, but do not use code outside of BEGIN blocks.
303 sub worker_run {
304 my ($fh, @args) = @_;
305 ...
306 }
307
308 # now preserve everything so far as AnyEvent::Fork object
309 # in §TEMPLATE.
310 use AnyEvent::Fork::Template;
311
312 # do not put code outside of BEGIN blocks until here
313
314 # now use the $TEMPLATE process in any way you like
315
316 # for example: create 10 worker processes
317 my @worker;
318 my $cv = AE::cv;
319 for (1..10) {
320 $cv->begin;
321 $TEMPLATE->fork->send_arg ($_)->run ("worker_run", sub {
322 push @worker, shift;
323 $cv->end;
324 });
325 }
326 $cv->recv;
327
233=head1 CONCEPTS 328=head1 CONCEPTS
234 329
235This module can create new processes either by executing a new perl 330This module can create new processes either by executing a new perl
236process, or by forking from an existing "template" process. 331process, or by forking from an existing "template" process.
332
333All these processes are called "child processes" (whether they are direct
334children or not), while the process that manages them is called the
335"parent process".
237 336
238Each such process comes with its own file handle that can be used to 337Each such process comes with its own file handle that can be used to
239communicate with it (it's actually a socket - one end in the new process, 338communicate with it (it's actually a socket - one end in the new process,
240one end in the main process), and among the things you can do in it are 339one end in the main process), and among the things you can do in it are
241load modules, fork new processes, send file handles to it, and execute 340load modules, fork new processes, send file handles to it, and execute
351use AnyEvent; 450use AnyEvent;
352use AnyEvent::Util (); 451use AnyEvent::Util ();
353 452
354use IO::FDPass; 453use IO::FDPass;
355 454
356our $VERSION = 0.5; 455our $VERSION = 1.1;
357
358our $PERL; # the path to the perl interpreter, deduces with various forms of magic
359
360=over 4
361
362=back
363
364=cut
365 456
366# the early fork template process 457# the early fork template process
367our $EARLY; 458our $EARLY;
368 459
369# the empty template process 460# the empty template process
370our $TEMPLATE; 461our $TEMPLATE;
462
463sub QUEUE() { 0 }
464sub FH() { 1 }
465sub WW() { 2 }
466sub PID() { 3 }
467sub CB() { 4 }
468
469sub _new {
470 my ($self, $fh, $pid) = @_;
471
472 AnyEvent::Util::fh_nonblocking $fh, 1;
473
474 $self = bless [
475 [], # write queue - strings or fd's
476 $fh,
477 undef, # AE watcher
478 $pid,
479 ], $self;
480
481 $self
482}
371 483
372sub _cmd { 484sub _cmd {
373 my $self = shift; 485 my $self = shift;
374 486
375 # ideally, we would want to use "a (w/a)*" as format string, but perl 487 # ideally, we would want to use "a (w/a)*" as format string, but perl
376 # versions from at least 5.8.9 to 5.16.3 are all buggy and can't unpack 488 # versions from at least 5.8.9 to 5.16.3 are all buggy and can't unpack
377 # it. 489 # it.
378 push @{ $self->[2] }, pack "a L/a*", $_[0], $_[1]; 490 push @{ $self->[QUEUE] }, pack "a L/a*", $_[0], $_[1];
379 491
380 $self->[3] ||= AE::io $self->[1], 1, sub { 492 $self->[WW] ||= AE::io $self->[FH], 1, sub {
381 do { 493 do {
382 # send the next "thing" in the queue - either a reference to an fh, 494 # send the next "thing" in the queue - either a reference to an fh,
383 # or a plain string. 495 # or a plain string.
384 496
385 if (ref $self->[2][0]) { 497 if (ref $self->[QUEUE][0]) {
386 # send fh 498 # send fh
387 unless (IO::FDPass::send fileno $self->[1], fileno ${ $self->[2][0] }) { 499 unless (IO::FDPass::send fileno $self->[FH], fileno ${ $self->[QUEUE][0] }) {
388 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK; 500 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK;
389 undef $self->[3]; 501 undef $self->[WW];
390 die "AnyEvent::Fork: file descriptor send failure: $!"; 502 die "AnyEvent::Fork: file descriptor send failure: $!";
391 } 503 }
392 504
393 shift @{ $self->[2] }; 505 shift @{ $self->[QUEUE] };
394 506
395 } else { 507 } else {
396 # send string 508 # send string
397 my $len = syswrite $self->[1], $self->[2][0]; 509 my $len = syswrite $self->[FH], $self->[QUEUE][0];
398 510
399 unless ($len) { 511 unless ($len) {
400 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK; 512 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK;
401 undef $self->[3]; 513 undef $self->[WW];
402 die "AnyEvent::Fork: command write failure: $!"; 514 die "AnyEvent::Fork: command write failure: $!";
403 } 515 }
404 516
405 substr $self->[2][0], 0, $len, ""; 517 substr $self->[QUEUE][0], 0, $len, "";
406 shift @{ $self->[2] } unless length $self->[2][0]; 518 shift @{ $self->[QUEUE] } unless length $self->[QUEUE][0];
407 } 519 }
408 } while @{ $self->[2] }; 520 } while @{ $self->[QUEUE] };
409 521
410 # everything written 522 # everything written
411 undef $self->[3]; 523 undef $self->[WW];
412 524
413 # invoke run callback, if any 525 # invoke run callback, if any
414 $self->[4]->($self->[1]) if $self->[4]; 526 if ($self->[CB]) {
527 $self->[CB]->($self->[FH]);
528 @$self = ();
529 }
415 }; 530 };
416 531
417 () # make sure we don't leak the watcher 532 () # make sure we don't leak the watcher
418}
419
420sub _new {
421 my ($self, $fh, $pid) = @_;
422
423 AnyEvent::Util::fh_nonblocking $fh, 1;
424
425 $self = bless [
426 $pid,
427 $fh,
428 [], # write queue - strings or fd's
429 undef, # AE watcher
430 ], $self;
431
432 $self
433} 533}
434 534
435# fork template from current process, used by AnyEvent::Fork::Early/Template 535# fork template from current process, used by AnyEvent::Fork::Early/Template
436sub _new_fork { 536sub _new_fork {
437 my ($fh, $slave) = AnyEvent::Util::portable_socketpair; 537 my ($fh, $slave) = AnyEvent::Util::portable_socketpair;
442 if ($pid eq 0) { 542 if ($pid eq 0) {
443 require AnyEvent::Fork::Serve; 543 require AnyEvent::Fork::Serve;
444 $AnyEvent::Fork::Serve::OWNER = $parent; 544 $AnyEvent::Fork::Serve::OWNER = $parent;
445 close $fh; 545 close $fh;
446 $0 = "$_[1] of $parent"; 546 $0 = "$_[1] of $parent";
447 $SIG{CHLD} = 'IGNORE';
448 AnyEvent::Fork::Serve::serve ($slave); 547 AnyEvent::Fork::Serve::serve ($slave);
449 exit 0; 548 exit 0;
450 } elsif (!$pid) { 549 } elsif (!$pid) {
451 die "AnyEvent::Fork::Early/Template: unable to fork template process: $!"; 550 die "AnyEvent::Fork::Early/Template: unable to fork template process: $!";
452 } 551 }
506 605
507You should use C<new> whenever possible, except when having a template 606You should use C<new> whenever possible, except when having a template
508process around is unacceptable. 607process around is unacceptable.
509 608
510The path to the perl interpreter is divined using various methods - first 609The path to the perl interpreter is divined using various methods - first
511C<$^X> is investigated to see if the path ends with something that sounds 610C<$^X> is investigated to see if the path ends with something that looks
512as if it were the perl interpreter. Failing this, the module falls back to 611as if it were the perl interpreter. Failing this, the module falls back to
513using C<$Config::Config{perlpath}>. 612using C<$Config::Config{perlpath}>.
514 613
614The path to perl can also be overriden by setting the global variable
615C<$AnyEvent::Fork::PERL> - it's value will be used for all subsequent
616invocations.
617
515=cut 618=cut
619
620our $PERL;
516 621
517sub new_exec { 622sub new_exec {
518 my ($self) = @_; 623 my ($self) = @_;
519 624
520 return $EARLY->fork 625 return $EARLY->fork
521 if $EARLY; 626 if $EARLY;
522 627
628 unless (defined $PERL) {
523 # first find path of perl 629 # first find path of perl
524 my $perl = $; 630 my $perl = $;
525 631
526 # first we try $^X, but the path must be absolute (always on win32), and end in sth. 632 # first we try $^X, but the path must be absolute (always on win32), and end in sth.
527 # that looks like perl. this obviously only works for posix and win32 633 # that looks like perl. this obviously only works for posix and win32
528 unless ( 634 unless (
529 ($^O eq "MSWin32" || $perl =~ m%^/%) 635 ($^O eq "MSWin32" || $perl =~ m%^/%)
530 && $perl =~ m%[/\\]perl(?:[0-9]+(\.[0-9]+)+)?(\.exe)?$%i 636 && $perl =~ m%[/\\]perl(?:[0-9]+(\.[0-9]+)+)?(\.exe)?$%i
531 ) { 637 ) {
532 # if it doesn't look perlish enough, try Config 638 # if it doesn't look perlish enough, try Config
533 require Config; 639 require Config;
534 $perl = $Config::Config{perlpath}; 640 $perl = $Config::Config{perlpath};
535 $perl =~ s/(?:\Q$Config::Config{_exe}\E)?$/$Config::Config{_exe}/; 641 $perl =~ s/(?:\Q$Config::Config{_exe}\E)?$/$Config::Config{_exe}/;
642 }
643
644 $PERL = $perl;
536 } 645 }
537 646
538 require Proc::FastSpawn; 647 require Proc::FastSpawn;
539 648
540 my ($fh, $slave) = AnyEvent::Util::portable_socketpair; 649 my ($fh, $slave) = AnyEvent::Util::portable_socketpair;
548 #local $ENV{PERL5LIB} = join ":", grep !ref, @INC; 657 #local $ENV{PERL5LIB} = join ":", grep !ref, @INC;
549 my %env = %ENV; 658 my %env = %ENV;
550 $env{PERL5LIB} = join +($^O eq "MSWin32" ? ";" : ":"), grep !ref, @INC; 659 $env{PERL5LIB} = join +($^O eq "MSWin32" ? ";" : ":"), grep !ref, @INC;
551 660
552 my $pid = Proc::FastSpawn::spawn ( 661 my $pid = Proc::FastSpawn::spawn (
553 $perl, 662 $PERL,
554 ["perl", "-MAnyEvent::Fork::Serve", "-e", "AnyEvent::Fork::Serve::me", fileno $slave, $$], 663 ["perl", "-MAnyEvent::Fork::Serve", "-e", "AnyEvent::Fork::Serve::me", fileno $slave, $$],
555 [map "$_=$env{$_}", keys %env], 664 [map "$_=$env{$_}", keys %env],
556 ) or die "unable to spawn AnyEvent::Fork server: $!"; 665 ) or die "unable to spawn AnyEvent::Fork server: $!";
557 666
558 $self->_new ($fh, $pid) 667 $self->_new ($fh, $pid)
559} 668}
560 669
561=item $pid = $proc->pid 670=item $pid = $proc->pid
562 671
563Returns the process id of the process I<iff it is a direct child of the 672Returns the process id of the process I<iff it is a direct child of the
564process> running AnyEvent::Fork, and C<undef> otherwise. 673process running AnyEvent::Fork>, and C<undef> otherwise.
565 674
566Normally, only processes created via C<< AnyEvent::Fork->new_exec >> and 675Normally, only processes created via C<< AnyEvent::Fork->new_exec >> and
567L<AnyEvent::Fork::Template> are direct children, and you are responsible 676L<AnyEvent::Fork::Template> are direct children, and you are responsible
568to clean up their zombies when they die. 677to clean up their zombies when they die.
569 678
571AnyEvent::Fork itself. 680AnyEvent::Fork itself.
572 681
573=cut 682=cut
574 683
575sub pid { 684sub pid {
576 $_[0][0] 685 $_[0][PID]
577} 686}
578 687
579=item $proc = $proc->eval ($perlcode, @args) 688=item $proc = $proc->eval ($perlcode, @args)
580 689
581Evaluates the given C<$perlcode> as ... perl code, while setting C<@_> to 690Evaluates the given C<$perlcode> as ... Perl code, while setting C<@_> to
582the strings specified by C<@args>, in the "main" package. 691the strings specified by C<@args>, in the "main" package.
583 692
584This call is meant to do any custom initialisation that might be required 693This call is meant to do any custom initialisation that might be required
585(for example, the C<require> method uses it). It's not supposed to be used 694(for example, the C<require> method uses it). It's not supposed to be used
586to completely take over the process, use C<run> for that. 695to completely take over the process, use C<run> for that.
587 696
588The code will usually be executed after this call returns, and there is no 697The code will usually be executed after this call returns, and there is no
589way to pass anything back to the calling process. Any evaluation errors 698way to pass anything back to the calling process. Any evaluation errors
590will be reported to stderr and cause the process to exit. 699will be reported to stderr and cause the process to exit.
591 700
592If you want to execute some code to take over the process (see the 701If you want to execute some code (that isn't in a module) to take over the
593"fork+exec" example in the SYNOPSIS), you should compile a function via 702process, you should compile a function via C<eval> first, and then call
594C<eval> first, and then call it via C<run>. This also gives you access to 703it via C<run>. This also gives you access to any arguments passed via the
595any arguments passed via the C<send_xxx> methods, such as file handles. 704C<send_xxx> methods, such as file handles. See the L<use AnyEvent::Fork as
705a faster fork+exec> example to see it in action.
596 706
597Returns the process object for easy chaining of method calls. 707Returns the process object for easy chaining of method calls.
598 708
599=cut 709=cut
600 710
626=item $proc = $proc->send_fh ($handle, ...) 736=item $proc = $proc->send_fh ($handle, ...)
627 737
628Send one or more file handles (I<not> file descriptors) to the process, 738Send one or more file handles (I<not> file descriptors) to the process,
629to prepare a call to C<run>. 739to prepare a call to C<run>.
630 740
631The process object keeps a reference to the handles until this is done, 741The process object keeps a reference to the handles until they have
632so you must not explicitly close the handles. This is most easily 742been passed over to the process, so you must not explicitly close the
633accomplished by simply not storing the file handles anywhere after passing 743handles. This is most easily accomplished by simply not storing the file
634them to this method. 744handles anywhere after passing them to this method - when AnyEvent::Fork
745is finished using them, perl will automatically close them.
635 746
636Returns the process object for easy chaining of method calls. 747Returns the process object for easy chaining of method calls.
637 748
638Example: pass a file handle to a process, and release it without 749Example: pass a file handle to a process, and release it without
639closing. It will be closed automatically when it is no longer used. 750closing. It will be closed automatically when it is no longer used.
646sub send_fh { 757sub send_fh {
647 my ($self, @fh) = @_; 758 my ($self, @fh) = @_;
648 759
649 for my $fh (@fh) { 760 for my $fh (@fh) {
650 $self->_cmd ("h"); 761 $self->_cmd ("h");
651 push @{ $self->[2] }, \$fh; 762 push @{ $self->[QUEUE] }, \$fh;
652 } 763 }
653 764
654 $self 765 $self
655} 766}
656 767
657=item $proc = $proc->send_arg ($string, ...) 768=item $proc = $proc->send_arg ($string, ...)
658 769
659Send one or more argument strings to the process, to prepare a call to 770Send one or more argument strings to the process, to prepare a call to
660C<run>. The strings can be any octet string. 771C<run>. The strings can be any octet strings.
661 772
662The protocol is optimised to pass a moderate number of relatively short 773The protocol is optimised to pass a moderate number of relatively short
663strings - while you can pass up to 4GB of data in one go, this is more 774strings - while you can pass up to 4GB of data in one go, this is more
664meant to pass some ID information or other startup info, not big chunks of 775meant to pass some ID information or other startup info, not big chunks of
665data. 776data.
681Enter the function specified by the function name in C<$func> in the 792Enter the function specified by the function name in C<$func> in the
682process. The function is called with the communication socket as first 793process. The function is called with the communication socket as first
683argument, followed by all file handles and string arguments sent earlier 794argument, followed by all file handles and string arguments sent earlier
684via C<send_fh> and C<send_arg> methods, in the order they were called. 795via C<send_fh> and C<send_arg> methods, in the order they were called.
685 796
797The process object becomes unusable on return from this function - any
798further method calls result in undefined behaviour.
799
686The function name should be fully qualified, but if it isn't, it will be 800The function name should be fully qualified, but if it isn't, it will be
687looked up in the main package. 801looked up in the C<main> package.
688 802
689If the called function returns, doesn't exist, or any error occurs, the 803If the called function returns, doesn't exist, or any error occurs, the
690process exits. 804process exits.
691 805
692Preparing the process is done in the background - when all commands have 806Preparing the process is done in the background - when all commands have
693been sent, the callback is invoked with the local communications socket 807been sent, the callback is invoked with the local communications socket
694as argument. At this point you can start using the socket in any way you 808as argument. At this point you can start using the socket in any way you
695like. 809like.
696 810
697The process object becomes unusable on return from this function - any
698further method calls result in undefined behaviour.
699
700If the communication socket isn't used, it should be closed on both sides, 811If the communication socket isn't used, it should be closed on both sides,
701to save on kernel memory. 812to save on kernel memory.
702 813
703The socket is non-blocking in the parent, and blocking in the newly 814The socket is non-blocking in the parent, and blocking in the newly
704created process. The close-on-exec flag is set in both. 815created process. The close-on-exec flag is set in both.
705 816
706Even if not used otherwise, the socket can be a good indicator for the 817Even if not used otherwise, the socket can be a good indicator for the
707existence of the process - if the other process exits, you get a readable 818existence of the process - if the other process exits, you get a readable
708event on it, because exiting the process closes the socket (if it didn't 819event on it, because exiting the process closes the socket (if it didn't
709create any children using fork). 820create any children using fork).
821
822=over 4
823
824=item Compatibility to L<AnyEvent::Fork::Remote>
825
826If you want to write code that works with both this module and
827L<AnyEvent::Fork::Remote>, you need to write your code so that it assumes
828there are two file handles for communications, which might not be unix
829domain sockets. The C<run> function should start like this:
830
831 sub run {
832 my ($rfh, @args) = @_; # @args is your normal arguments
833 my $wfh = fileno $rfh ? $rfh : *STDOUT;
834
835 # now use $rfh for reading and $wfh for writing
836 }
837
838This checks whether the passed file handle is, in fact, the process
839C<STDIN> handle. If it is, then the function was invoked visa
840L<AnyEvent::Fork::Remote>, so STDIN should be used for reading and
841C<STDOUT> should be used for writing.
842
843In all other cases, the function was called via this module, and there is
844only one file handle that should be sued for reading and writing.
845
846=back
710 847
711Example: create a template for a process pool, pass a few strings, some 848Example: create a template for a process pool, pass a few strings, some
712file handles, then fork, pass one more string, and run some code. 849file handles, then fork, pass one more string, and run some code.
713 850
714 my $pool = AnyEvent::Fork 851 my $pool = AnyEvent::Fork
742=cut 879=cut
743 880
744sub run { 881sub run {
745 my ($self, $func, $cb) = @_; 882 my ($self, $func, $cb) = @_;
746 883
747 $self->[4] = $cb; 884 $self->[CB] = $cb;
748 $self->_cmd (r => $func); 885 $self->_cmd (r => $func);
886}
887
888=back
889
890=head2 EXPERIMENTAL METHODS
891
892These methods might go away completely or change behaviour, at any time.
893
894=over 4
895
896=item $proc->to_fh ($cb->($fh)) # EXPERIMENTAL, MIGHT BE REMOVED
897
898Flushes all commands out to the process and then calls the callback with
899the communications socket.
900
901The process object becomes unusable on return from this function - any
902further method calls result in undefined behaviour.
903
904The point of this method is to give you a file handle that you can pass
905to another process. In that other process, you can call C<new_from_fh
906AnyEvent::Fork $fh> to create a new C<AnyEvent::Fork> object from it,
907thereby effectively passing a fork object to another process.
908
909=cut
910
911sub to_fh {
912 my ($self, $cb) = @_;
913
914 $self->[CB] = $cb;
915
916 unless ($self->[WW]) {
917 $self->[CB]->($self->[FH]);
918 @$self = ();
919 }
920}
921
922=item new_from_fh AnyEvent::Fork $fh # EXPERIMENTAL, MIGHT BE REMOVED
923
924Takes a file handle originally rceeived by the C<to_fh> method and creates
925a new C<AnyEvent:Fork> object. The child process itself will not change in
926any way, i.e. it will keep all the modifications done to it before calling
927C<to_fh>.
928
929The new object is very much like the original object, except that the
930C<pid> method will return C<undef> even if the process is a direct child.
931
932=cut
933
934sub new_from_fh {
935 my ($class, $fh) = @_;
936
937 $class->_new ($fh)
749} 938}
750 939
751=back 940=back
752 941
753=head1 PERFORMANCE 942=head1 PERFORMANCE
763 952
764 2079 new processes per second, using manual socketpair + fork 953 2079 new processes per second, using manual socketpair + fork
765 954
766Then I did the same thing, but instead of calling fork, I called 955Then I did the same thing, but instead of calling fork, I called
767AnyEvent::Fork->new->run ("CORE::exit") and then again waited for the 956AnyEvent::Fork->new->run ("CORE::exit") and then again waited for the
768socket form the child to close on exit. This does the same thing as manual 957socket from the child to close on exit. This does the same thing as manual
769socket pair + fork, except that what is forked is the template process 958socket pair + fork, except that what is forked is the template process
770(2440kB), and the socket needs to be passed to the server at the other end 959(2440kB), and the socket needs to be passed to the server at the other end
771of the socket first. 960of the socket first.
772 961
773 2307 new processes per second, using AnyEvent::Fork->new 962 2307 new processes per second, using AnyEvent::Fork->new
778 479 vfork+execs per second, using AnyEvent::Fork->new_exec 967 479 vfork+execs per second, using AnyEvent::Fork->new_exec
779 968
780So how can C<< AnyEvent->new >> be faster than a standard fork, even 969So how can C<< AnyEvent->new >> be faster than a standard fork, even
781though it uses the same operations, but adds a lot of overhead? 970though it uses the same operations, but adds a lot of overhead?
782 971
783The difference is simply the process size: forking the 6MB process takes 972The difference is simply the process size: forking the 5MB process takes
784so much longer than forking the 2.5MB template process that the overhead 973so much longer than forking the 2.5MB template process that the extra
785introduced is canceled out. 974overhead is canceled out.
786 975
787If the benchmark process grows, the normal fork becomes even slower: 976If the benchmark process grows, the normal fork becomes even slower:
788 977
789 1340 new processes, manual fork in a 20MB process 978 1340 new processes, manual fork of a 20MB process
790 731 new processes, manual fork in a 200MB process 979 731 new processes, manual fork of a 200MB process
791 235 new processes, manual fork in a 2000MB process 980 235 new processes, manual fork of a 2000MB process
792 981
793What that means (to me) is that I can use this module without having a 982What that means (to me) is that I can use this module without having a bad
794very bad conscience because of the extra overhead required to start new 983conscience because of the extra overhead required to start new processes.
795processes.
796 984
797=head1 TYPICAL PROBLEMS 985=head1 TYPICAL PROBLEMS
798 986
799This section lists typical problems that remain. I hope by recognising 987This section lists typical problems that remain. I hope by recognising
800them, most can be avoided. 988them, most can be avoided.
801 989
802=over 4 990=over 4
803 991
804=item "leaked" file descriptors for exec'ed processes 992=item leaked file descriptors for exec'ed processes
805 993
806POSIX systems inherit file descriptors by default when exec'ing a new 994POSIX systems inherit file descriptors by default when exec'ing a new
807process. While perl itself laudably sets the close-on-exec flags on new 995process. While perl itself laudably sets the close-on-exec flags on new
808file handles, most C libraries don't care, and even if all cared, it's 996file handles, most C libraries don't care, and even if all cared, it's
809often not possible to set the flag in a race-free manner. 997often not possible to set the flag in a race-free manner.
829libraries or the code that leaks those file descriptors. 1017libraries or the code that leaks those file descriptors.
830 1018
831Fortunately, most of these leaked descriptors do no harm, other than 1019Fortunately, most of these leaked descriptors do no harm, other than
832sitting on some resources. 1020sitting on some resources.
833 1021
834=item "leaked" file descriptors for fork'ed processes 1022=item leaked file descriptors for fork'ed processes
835 1023
836Normally, L<AnyEvent::Fork> does start new processes by exec'ing them, 1024Normally, L<AnyEvent::Fork> does start new processes by exec'ing them,
837which closes file descriptors not marked for being inherited. 1025which closes file descriptors not marked for being inherited.
838 1026
839However, L<AnyEvent::Fork::Early> and L<AnyEvent::Fork::Template> offer 1027However, L<AnyEvent::Fork::Early> and L<AnyEvent::Fork::Template> offer
848 1036
849The solution is to either not load these modules before use'ing 1037The solution is to either not load these modules before use'ing
850L<AnyEvent::Fork::Early> or L<AnyEvent::Fork::Template>, or to delay 1038L<AnyEvent::Fork::Early> or L<AnyEvent::Fork::Template>, or to delay
851initialising them, for example, by calling C<init Gtk2> manually. 1039initialising them, for example, by calling C<init Gtk2> manually.
852 1040
853=item exit runs destructors 1041=item exiting calls object destructors
854 1042
855This only applies to users of Lc<AnyEvent::Fork:Early> and 1043This only applies to users of L<AnyEvent::Fork:Early> and
856L<AnyEvent::Fork::Template>. 1044L<AnyEvent::Fork::Template>, or when initialising code creates objects
1045that reference external resources.
857 1046
858When a process created by AnyEvent::Fork exits, it might do so by calling 1047When a process created by AnyEvent::Fork exits, it might do so by calling
859exit, or simply letting perl reach the end of the program. At which point 1048exit, or simply letting perl reach the end of the program. At which point
860Perl runs all destructors. 1049Perl runs all destructors.
861 1050
880to make it so, mostly due to the bloody broken perl that nobody seems to 1069to make it so, mostly due to the bloody broken perl that nobody seems to
881care about. The fork emulation is a bad joke - I have yet to see something 1070care about. The fork emulation is a bad joke - I have yet to see something
882useful that you can do with it without running into memory corruption 1071useful that you can do with it without running into memory corruption
883issues or other braindamage. Hrrrr. 1072issues or other braindamage. Hrrrr.
884 1073
885Cygwin perl is not supported at the moment, as it should implement fd 1074Since fork is endlessly broken on win32 perls (it doesn't even remotely
886passing, but doesn't, and rolling my own is hard, as cygwin doesn't 1075work within it's documented limits) and quite obviously it's not getting
887support enough functionality to do it. 1076improved any time soon, the best way to proceed on windows would be to
1077always use C<new_exec> and thus never rely on perl's fork "emulation".
1078
1079Cygwin perl is not supported at the moment due to some hilarious
1080shortcomings of its API - see L<IO::FDPoll> for more details. If you never
1081use C<send_fh> and always use C<new_exec> to create processes, it should
1082work though.
888 1083
889=head1 SEE ALSO 1084=head1 SEE ALSO
890 1085
891L<AnyEvent::Fork::Early> (to avoid executing a perl interpreter), 1086L<AnyEvent::Fork::Early>, to avoid executing a perl interpreter at all
1087(part of this distribution).
1088
892L<AnyEvent::Fork::Template> (to create a process by forking the main 1089L<AnyEvent::Fork::Template>, to create a process by forking the main
893program at a convenient time). 1090program at a convenient time (part of this distribution).
894 1091
895=head1 AUTHOR 1092L<AnyEvent::Fork::Remote>, for another way to create processes that is
1093mostly compatible to this module and modules building on top of it, but
1094works better with remote processes.
1095
1096L<AnyEvent::Fork::RPC>, for simple RPC to child processes (on CPAN).
1097
1098L<AnyEvent::Fork::Pool>, for simple worker process pool (on CPAN).
1099
1100=head1 AUTHOR AND CONTACT INFORMATION
896 1101
897 Marc Lehmann <schmorp@schmorp.de> 1102 Marc Lehmann <schmorp@schmorp.de>
898 http://home.schmorp.de/ 1103 http://software.schmorp.de/pkg/AnyEvent-Fork
899 1104
900=cut 1105=cut
901 1106
9021 11071
903 1108

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines