ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-Fork/Fork.pm
(Generate patch)

Comparing AnyEvent-Fork/Fork.pm (file contents):
Revision 1.47 by root, Thu Apr 18 20:17:34 2013 UTC vs.
Revision 1.63 by root, Wed Nov 26 13:36:18 2014 UTC

38 38
39If you need some form of RPC, you could use the L<AnyEvent::Fork::RPC> 39If you need some form of RPC, you could use the L<AnyEvent::Fork::RPC>
40companion module, which adds simple RPC/job queueing to a process created 40companion module, which adds simple RPC/job queueing to a process created
41by this module. 41by this module.
42 42
43And if you need some automatic process pool management on top of
44L<AnyEvent::Fork::RPC>, you can look at the L<AnyEvent::Fork::Pool>
45companion module.
46
43Or you can implement it yourself in whatever way you like, use some 47Or you can implement it yourself in whatever way you like: use some
44message-passing module such as L<AnyEvent::MP>, some pipe such as 48message-passing module such as L<AnyEvent::MP>, some pipe such as
45L<AnyEvent::ZeroMQ>, use L<AnyEvent::Handle> on both sides to send 49L<AnyEvent::ZeroMQ>, use L<AnyEvent::Handle> on both sides to send
46e.g. JSON or Storable messages, and so on. 50e.g. JSON or Storable messages, and so on.
47 51
48=head2 COMPARISON TO OTHER MODULES 52=head2 COMPARISON TO OTHER MODULES
52or L<AnyEvent::Subprocess>. There are modules that implement their own 56or L<AnyEvent::Subprocess>. There are modules that implement their own
53process management, such as L<AnyEvent::DBI>. 57process management, such as L<AnyEvent::DBI>.
54 58
55The problems that all these modules try to solve are real, however, none 59The problems that all these modules try to solve are real, however, none
56of them (from what I have seen) tackle the very real problems of unwanted 60of them (from what I have seen) tackle the very real problems of unwanted
57memory sharing, efficiency, not being able to use event processing or 61memory sharing, efficiency or not being able to use event processing, GUI
58similar modules in the processes they create. 62toolkits or similar modules in the processes they create.
59 63
60This module doesn't try to replace any of them - instead it tries to solve 64This module doesn't try to replace any of them - instead it tries to solve
61the problem of creating processes with a minimum of fuss and overhead (and 65the problem of creating processes with a minimum of fuss and overhead (and
62also luxury). Ideally, most of these would use AnyEvent::Fork internally, 66also luxury). Ideally, most of these would use AnyEvent::Fork internally,
63except they were written before AnyEvent:Fork was available, so obviously 67except they were written before AnyEvent:Fork was available, so obviously
85 89
86=item Forking usually creates a copy-on-write copy of the parent 90=item Forking usually creates a copy-on-write copy of the parent
87process. 91process.
88 92
89For example, modules or data files that are loaded will not use additional 93For example, modules or data files that are loaded will not use additional
90memory after a fork. When exec'ing a new process, modules and data files 94memory after a fork. Exec'ing a new process, in contrast, means modules
91might need to be loaded again, at extra CPU and memory cost. But when 95and data files might need to be loaded again, at extra CPU and memory
92forking, literally all data structures are copied - if the program frees 96cost.
97
98But when forking, you still create a copy of your data structures - if
93them and replaces them by new data, the child processes will retain the 99the program frees them and replaces them by new data, the child processes
94old version even if it isn't used, which can suddenly and unexpectedly 100will retain the old version even if it isn't used, which can suddenly and
95increase memory usage when freeing memory. 101unexpectedly increase memory usage when freeing memory.
96 102
103For example, L<Gtk2::CV> is an image viewer optimised for large
104directories (millions of pictures). It also forks subprocesses for
105thumbnail generation, which inherit the data structure that stores all
106file information. If the user changes the directory, it gets freed in
107the main process, leaving a copy in the thumbnailer processes. This can
108lead to many times the memory usage that would actually be required. The
109solution is to fork early (and being unable to dynamically generate more
110subprocesses or do this from a module)... or to use L<AnyEvent:Fork>.
111
97The trade-off is between more sharing with fork (which can be good or 112There is a trade-off between more sharing with fork (which can be good or
98bad), and no sharing with exec. 113bad), and no sharing with exec.
99 114
100This module allows the main program to do a controlled fork, and allows 115This module allows the main program to do a controlled fork, and allows
101modules to exec processes safely at any time. When creating a custom 116modules to exec processes safely at any time. When creating a custom
102process pool you can take advantage of data sharing via fork without 117process pool you can take advantage of data sharing via fork without
107shared and what isn't, at all times. 122shared and what isn't, at all times.
108 123
109=item Exec'ing a new perl process might be difficult. 124=item Exec'ing a new perl process might be difficult.
110 125
111For example, it is not easy to find the correct path to the perl 126For example, it is not easy to find the correct path to the perl
112interpreter - C<$^X> might not be a perl interpreter at all. 127interpreter - C<$^X> might not be a perl interpreter at all. Worse, there
128might not even be a perl binary installed on the system.
113 129
114This module tries hard to identify the correct path to the perl 130This module tries hard to identify the correct path to the perl
115interpreter. With a cooperative main program, exec'ing the interpreter 131interpreter. With a cooperative main program, exec'ing the interpreter
116might not even be necessary, but even without help from the main program, 132might not even be necessary, but even without help from the main program,
117it will still work when used from a module. 133it will still work when used from a module.
123and modules are no longer loadable because they refer to a different 139and modules are no longer loadable because they refer to a different
124perl version, or parts of a distribution are newer than the ones already 140perl version, or parts of a distribution are newer than the ones already
125loaded. 141loaded.
126 142
127This module supports creating pre-initialised perl processes to be used as 143This module supports creating pre-initialised perl processes to be used as
128a template for new processes. 144a template for new processes at a later time, e.g. for use in a process
145pool.
129 146
130=item Forking might be impossible when a program is running. 147=item Forking might be impossible when a program is running.
131 148
132For example, POSIX makes it almost impossible to fork from a 149For example, POSIX makes it almost impossible to fork from a
133multi-threaded program while doing anything useful in the child - in 150multi-threaded program while doing anything useful in the child - in
134fact, if your perl program uses POSIX threads (even indirectly via 151fact, if your perl program uses POSIX threads (even indirectly via
135e.g. L<IO::AIO> or L<threads>), you cannot call fork on the perl level 152e.g. L<IO::AIO> or L<threads>), you cannot call fork on the perl level
136anymore without risking corruption issues on a number of operating 153anymore without risking memory corruption or worse on a number of
137systems. 154operating systems.
138 155
139This module can safely fork helper processes at any time, by calling 156This module can safely fork helper processes at any time, by calling
140fork+exec in C, in a POSIX-compatible way (via L<Proc::FastSpawn>). 157fork+exec in C, in a POSIX-compatible way (via L<Proc::FastSpawn>).
141 158
142=item Parallel processing with fork might be inconvenient or difficult 159=item Parallel processing with fork might be inconvenient or difficult
250 ->send_arg ("/bin/echo", "hi") 267 ->send_arg ("/bin/echo", "hi")
251 ->run ("run", my $cv = AE::cv); 268 ->run ("run", my $cv = AE::cv);
252 269
253 my $stderr = $cv->recv; 270 my $stderr = $cv->recv;
254 271
272=head2 For stingy users: put the worker code into a C<DATA> section.
273
274When you want to be stingy with files, you can put your code into the
275C<DATA> section of your module (or program):
276
277 use AnyEvent::Fork;
278
279 AnyEvent::Fork
280 ->new
281 ->eval (do { local $/; <DATA> })
282 ->run ("doit", sub { ... });
283
284 __DATA__
285
286 sub doit {
287 ... do something!
288 }
289
290=head2 For stingy standalone programs: do not rely on external files at
291all.
292
293For single-file scripts it can be inconvenient to rely on external
294files - even when using a C<DATA> section, you still need to C<exec> an
295external perl interpreter, which might not be available when using
296L<App::Staticperl>, L<Urlader> or L<PAR::Packer> for example.
297
298Two modules help here - L<AnyEvent::Fork::Early> forks a template process
299for all further calls to C<new_exec>, and L<AnyEvent::Fork::Template>
300forks the main program as a template process.
301
302Here is how your main program should look like:
303
304 #! perl
305
306 # optional, as the very first thing.
307 # in case modules want to create their own processes.
308 use AnyEvent::Fork::Early;
309
310 # next, load all modules you need in your template process
311 use Example::My::Module
312 use Example::Whatever;
313
314 # next, put your run function definition and anything else you
315 # need, but do not use code outside of BEGIN blocks.
316 sub worker_run {
317 my ($fh, @args) = @_;
318 ...
319 }
320
321 # now preserve everything so far as AnyEvent::Fork object
322 # in $TEMPLATE.
323 use AnyEvent::Fork::Template;
324
325 # do not put code outside of BEGIN blocks until here
326
327 # now use the $TEMPLATE process in any way you like
328
329 # for example: create 10 worker processes
330 my @worker;
331 my $cv = AE::cv;
332 for (1..10) {
333 $cv->begin;
334 $TEMPLATE->fork->send_arg ($_)->run ("worker_run", sub {
335 push @worker, shift;
336 $cv->end;
337 });
338 }
339 $cv->recv;
340
255=head1 CONCEPTS 341=head1 CONCEPTS
256 342
257This module can create new processes either by executing a new perl 343This module can create new processes either by executing a new perl
258process, or by forking from an existing "template" process. 344process, or by forking from an existing "template" process.
259 345
377use AnyEvent; 463use AnyEvent;
378use AnyEvent::Util (); 464use AnyEvent::Util ();
379 465
380use IO::FDPass; 466use IO::FDPass;
381 467
382our $VERSION = 0.7; 468our $VERSION = 1.2;
383 469
384# the early fork template process 470# the early fork template process
385our $EARLY; 471our $EARLY;
386 472
387# the empty template process 473# the empty template process
435 # send string 521 # send string
436 my $len = syswrite $self->[FH], $self->[QUEUE][0]; 522 my $len = syswrite $self->[FH], $self->[QUEUE][0];
437 523
438 unless ($len) { 524 unless ($len) {
439 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK; 525 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK;
440 undef $self->[3]; 526 undef $self->[WW];
441 die "AnyEvent::Fork: command write failure: $!"; 527 die "AnyEvent::Fork: command write failure: $!";
442 } 528 }
443 529
444 substr $self->[QUEUE][0], 0, $len, ""; 530 substr $self->[QUEUE][0], 0, $len, "";
445 shift @{ $self->[QUEUE] } unless length $self->[QUEUE][0]; 531 shift @{ $self->[QUEUE] } unless length $self->[QUEUE][0];
448 534
449 # everything written 535 # everything written
450 undef $self->[WW]; 536 undef $self->[WW];
451 537
452 # invoke run callback, if any 538 # invoke run callback, if any
539 if ($self->[CB]) {
453 $self->[CB]->($self->[FH]) if $self->[CB]; 540 $self->[CB]->($self->[FH]);
541 @$self = ();
542 }
454 }; 543 };
455 544
456 () # make sure we don't leak the watcher 545 () # make sure we don't leak the watcher
457} 546}
458 547
529 618
530You should use C<new> whenever possible, except when having a template 619You should use C<new> whenever possible, except when having a template
531process around is unacceptable. 620process around is unacceptable.
532 621
533The path to the perl interpreter is divined using various methods - first 622The path to the perl interpreter is divined using various methods - first
534C<$^X> is investigated to see if the path ends with something that sounds 623C<$^X> is investigated to see if the path ends with something that looks
535as if it were the perl interpreter. Failing this, the module falls back to 624as if it were the perl interpreter. Failing this, the module falls back to
536using C<$Config::Config{perlpath}>. 625using C<$Config::Config{perlpath}>.
537 626
627The path to perl can also be overriden by setting the global variable
628C<$AnyEvent::Fork::PERL> - it's value will be used for all subsequent
629invocations.
630
538=cut 631=cut
632
633our $PERL;
539 634
540sub new_exec { 635sub new_exec {
541 my ($self) = @_; 636 my ($self) = @_;
542 637
543 return $EARLY->fork 638 return $EARLY->fork
544 if $EARLY; 639 if $EARLY;
545 640
641 unless (defined $PERL) {
546 # first find path of perl 642 # first find path of perl
547 my $perl = $; 643 my $perl = $^X;
548 644
549 # first we try $^X, but the path must be absolute (always on win32), and end in sth. 645 # first we try $^X, but the path must be absolute (always on win32), and end in sth.
550 # that looks like perl. this obviously only works for posix and win32 646 # that looks like perl. this obviously only works for posix and win32
551 unless ( 647 unless (
552 ($^O eq "MSWin32" || $perl =~ m%^/%) 648 ($^O eq "MSWin32" || $perl =~ m%^/%)
553 && $perl =~ m%[/\\]perl(?:[0-9]+(\.[0-9]+)+)?(\.exe)?$%i 649 && $perl =~ m%[/\\]perl(?:[0-9]+(\.[0-9]+)+)?(\.exe)?$%i
554 ) { 650 ) {
555 # if it doesn't look perlish enough, try Config 651 # if it doesn't look perlish enough, try Config
556 require Config; 652 require Config;
557 $perl = $Config::Config{perlpath}; 653 $perl = $Config::Config{perlpath};
558 $perl =~ s/(?:\Q$Config::Config{_exe}\E)?$/$Config::Config{_exe}/; 654 $perl =~ s/(?:\Q$Config::Config{_exe}\E)?$/$Config::Config{_exe}/;
655 }
656
657 $PERL = $perl;
559 } 658 }
560 659
561 require Proc::FastSpawn; 660 require Proc::FastSpawn;
562 661
563 my ($fh, $slave) = AnyEvent::Util::portable_socketpair; 662 my ($fh, $slave) = AnyEvent::Util::portable_socketpair;
571 #local $ENV{PERL5LIB} = join ":", grep !ref, @INC; 670 #local $ENV{PERL5LIB} = join ":", grep !ref, @INC;
572 my %env = %ENV; 671 my %env = %ENV;
573 $env{PERL5LIB} = join +($^O eq "MSWin32" ? ";" : ":"), grep !ref, @INC; 672 $env{PERL5LIB} = join +($^O eq "MSWin32" ? ";" : ":"), grep !ref, @INC;
574 673
575 my $pid = Proc::FastSpawn::spawn ( 674 my $pid = Proc::FastSpawn::spawn (
576 $perl, 675 $PERL,
577 ["perl", "-MAnyEvent::Fork::Serve", "-e", "AnyEvent::Fork::Serve::me", fileno $slave, $$], 676 ["perl", "-MAnyEvent::Fork::Serve", "-e", "AnyEvent::Fork::Serve::me", fileno $slave, $$],
578 [map "$_=$env{$_}", keys %env], 677 [map "$_=$env{$_}", keys %env],
579 ) or die "unable to spawn AnyEvent::Fork server: $!"; 678 ) or die "unable to spawn AnyEvent::Fork server: $!";
580 679
581 $self->_new ($fh, $pid) 680 $self->_new ($fh, $pid)
582} 681}
583 682
584=item $pid = $proc->pid 683=item $pid = $proc->pid
585 684
586Returns the process id of the process I<iff it is a direct child of the 685Returns the process id of the process I<iff it is a direct child of the
587process running AnyEvent::Fork>, and C<undef> otherwise. 686process running AnyEvent::Fork>, and C<undef> otherwise. As a general
687rule (that you cannot rely upon), processes created via C<new_exec>,
688L<AnyEvent::Fork::Early> or L<AnyEvent::Fork::Template> are direct
689children, while all other processes are not.
588 690
589Normally, only processes created via C<< AnyEvent::Fork->new_exec >> and 691Or in other words, you do not normally have to take care of zombies for
590L<AnyEvent::Fork::Template> are direct children, and you are responsible 692processes created via C<new>, but when in doubt, or zombies are a problem,
591to clean up their zombies when they die. 693you need to check whether a process is a diretc child by calling this
592 694method, and possibly creating a child watcher or reap it manually.
593All other processes are not direct children, and will be cleaned up by
594AnyEvent::Fork itself.
595 695
596=cut 696=cut
597 697
598sub pid { 698sub pid {
599 $_[0][PID] 699 $_[0][PID]
730 830
731Even if not used otherwise, the socket can be a good indicator for the 831Even if not used otherwise, the socket can be a good indicator for the
732existence of the process - if the other process exits, you get a readable 832existence of the process - if the other process exits, you get a readable
733event on it, because exiting the process closes the socket (if it didn't 833event on it, because exiting the process closes the socket (if it didn't
734create any children using fork). 834create any children using fork).
835
836=over 4
837
838=item Compatibility to L<AnyEvent::Fork::Remote>
839
840If you want to write code that works with both this module and
841L<AnyEvent::Fork::Remote>, you need to write your code so that it assumes
842there are two file handles for communications, which might not be unix
843domain sockets. The C<run> function should start like this:
844
845 sub run {
846 my ($rfh, @args) = @_; # @args is your normal arguments
847 my $wfh = fileno $rfh ? $rfh : *STDOUT;
848
849 # now use $rfh for reading and $wfh for writing
850 }
851
852This checks whether the passed file handle is, in fact, the process
853C<STDIN> handle. If it is, then the function was invoked visa
854L<AnyEvent::Fork::Remote>, so STDIN should be used for reading and
855C<STDOUT> should be used for writing.
856
857In all other cases, the function was called via this module, and there is
858only one file handle that should be sued for reading and writing.
859
860=back
735 861
736Example: create a template for a process pool, pass a few strings, some 862Example: create a template for a process pool, pass a few strings, some
737file handles, then fork, pass one more string, and run some code. 863file handles, then fork, pass one more string, and run some code.
738 864
739 my $pool = AnyEvent::Fork 865 my $pool = AnyEvent::Fork
773 $self->_cmd (r => $func); 899 $self->_cmd (r => $func);
774} 900}
775 901
776=back 902=back
777 903
904=head2 EXPERIMENTAL METHODS
905
906These methods might go away completely or change behaviour, at any time.
907
908=over 4
909
910=item $proc->to_fh ($cb->($fh)) # EXPERIMENTAL, MIGHT BE REMOVED
911
912Flushes all commands out to the process and then calls the callback with
913the communications socket.
914
915The process object becomes unusable on return from this function - any
916further method calls result in undefined behaviour.
917
918The point of this method is to give you a file handle that you can pass
919to another process. In that other process, you can call C<new_from_fh
920AnyEvent::Fork $fh> to create a new C<AnyEvent::Fork> object from it,
921thereby effectively passing a fork object to another process.
922
923=cut
924
925sub to_fh {
926 my ($self, $cb) = @_;
927
928 $self->[CB] = $cb;
929
930 unless ($self->[WW]) {
931 $self->[CB]->($self->[FH]);
932 @$self = ();
933 }
934}
935
936=item new_from_fh AnyEvent::Fork $fh # EXPERIMENTAL, MIGHT BE REMOVED
937
938Takes a file handle originally rceeived by the C<to_fh> method and creates
939a new C<AnyEvent:Fork> object. The child process itself will not change in
940any way, i.e. it will keep all the modifications done to it before calling
941C<to_fh>.
942
943The new object is very much like the original object, except that the
944C<pid> method will return C<undef> even if the process is a direct child.
945
946=cut
947
948sub new_from_fh {
949 my ($class, $fh) = @_;
950
951 $class->_new ($fh)
952}
953
954=back
955
778=head1 PERFORMANCE 956=head1 PERFORMANCE
779 957
780Now for some unscientific benchmark numbers (all done on an amd64 958Now for some unscientific benchmark numbers (all done on an amd64
781GNU/Linux box). These are intended to give you an idea of the relative 959GNU/Linux box). These are intended to give you an idea of the relative
782performance you can expect, they are not meant to be absolute performance 960performance you can expect, they are not meant to be absolute performance
788 966
789 2079 new processes per second, using manual socketpair + fork 967 2079 new processes per second, using manual socketpair + fork
790 968
791Then I did the same thing, but instead of calling fork, I called 969Then I did the same thing, but instead of calling fork, I called
792AnyEvent::Fork->new->run ("CORE::exit") and then again waited for the 970AnyEvent::Fork->new->run ("CORE::exit") and then again waited for the
793socket form the child to close on exit. This does the same thing as manual 971socket from the child to close on exit. This does the same thing as manual
794socket pair + fork, except that what is forked is the template process 972socket pair + fork, except that what is forked is the template process
795(2440kB), and the socket needs to be passed to the server at the other end 973(2440kB), and the socket needs to be passed to the server at the other end
796of the socket first. 974of the socket first.
797 975
798 2307 new processes per second, using AnyEvent::Fork->new 976 2307 new processes per second, using AnyEvent::Fork->new
905to make it so, mostly due to the bloody broken perl that nobody seems to 1083to make it so, mostly due to the bloody broken perl that nobody seems to
906care about. The fork emulation is a bad joke - I have yet to see something 1084care about. The fork emulation is a bad joke - I have yet to see something
907useful that you can do with it without running into memory corruption 1085useful that you can do with it without running into memory corruption
908issues or other braindamage. Hrrrr. 1086issues or other braindamage. Hrrrr.
909 1087
1088Since fork is endlessly broken on win32 perls (it doesn't even remotely
1089work within it's documented limits) and quite obviously it's not getting
1090improved any time soon, the best way to proceed on windows would be to
1091always use C<new_exec> and thus never rely on perl's fork "emulation".
1092
910Cygwin perl is not supported at the moment due to some hilarious 1093Cygwin perl is not supported at the moment due to some hilarious
911shortcomings of its API - see L<IO::FDPoll> for more details. 1094shortcomings of its API - see L<IO::FDPoll> for more details. If you never
1095use C<send_fh> and always use C<new_exec> to create processes, it should
1096work though.
1097
1098=head1 USING AnyEvent::Fork IN SUBPROCESSES
1099
1100AnyEvent::Fork itself cannot generally be used in subprocesses. As long as
1101only one process ever forks new processes, sharing the template processes
1102is possible (you could use a pipe as a lock by writing a byte into it to
1103unlock, and reading the byte to lock for example)
1104
1105To make concurrent calls possible after fork, you should get rid of the
1106template and early fork processes. AnyEvent::Fork will create a new
1107template process as needed.
1108
1109 undef $AnyEvent::Fork::EARLY;
1110 undef $AnyEvent::Fork::TEMPLATE;
1111
1112It doesn't matter whether you get rid of them in the parent or child after
1113a fork.
912 1114
913=head1 SEE ALSO 1115=head1 SEE ALSO
914 1116
915L<AnyEvent::Fork::Early>, to avoid executing a perl interpreter at all 1117L<AnyEvent::Fork::Early>, to avoid executing a perl interpreter at all
916(part of this distribution). 1118(part of this distribution).
917 1119
918L<AnyEvent::Fork::Template>, to create a process by forking the main 1120L<AnyEvent::Fork::Template>, to create a process by forking the main
919program at a convenient time (part of this distribution). 1121program at a convenient time (part of this distribution).
920 1122
1123L<AnyEvent::Fork::Remote>, for another way to create processes that is
1124mostly compatible to this module and modules building on top of it, but
1125works better with remote processes.
1126
921L<AnyEvent::Fork::RPC>, for simple RPC to child processes (on CPAN). 1127L<AnyEvent::Fork::RPC>, for simple RPC to child processes (on CPAN).
1128
1129L<AnyEvent::Fork::Pool>, for simple worker process pool (on CPAN).
922 1130
923=head1 AUTHOR AND CONTACT INFORMATION 1131=head1 AUTHOR AND CONTACT INFORMATION
924 1132
925 Marc Lehmann <schmorp@schmorp.de> 1133 Marc Lehmann <schmorp@schmorp.de>
926 http://software.schmorp.de/pkg/AnyEvent-Fork 1134 http://software.schmorp.de/pkg/AnyEvent-Fork

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines