ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-Fork/Fork.pm
(Generate patch)

Comparing AnyEvent-Fork/Fork.pm (file contents):
Revision 1.50 by root, Sat Apr 20 19:58:06 2013 UTC vs.
Revision 1.68 by root, Sat May 21 07:01:58 2016 UTC

38 38
39If you need some form of RPC, you could use the L<AnyEvent::Fork::RPC> 39If you need some form of RPC, you could use the L<AnyEvent::Fork::RPC>
40companion module, which adds simple RPC/job queueing to a process created 40companion module, which adds simple RPC/job queueing to a process created
41by this module. 41by this module.
42 42
43And if you need some automatic process pool management on top of
44L<AnyEvent::Fork::RPC>, you can look at the L<AnyEvent::Fork::Pool>
45companion module.
46
43Or you can implement it yourself in whatever way you like, use some 47Or you can implement it yourself in whatever way you like: use some
44message-passing module such as L<AnyEvent::MP>, some pipe such as 48message-passing module such as L<AnyEvent::MP>, some pipe such as
45L<AnyEvent::ZeroMQ>, use L<AnyEvent::Handle> on both sides to send 49L<AnyEvent::ZeroMQ>, use L<AnyEvent::Handle> on both sides to send
46e.g. JSON or Storable messages, and so on. 50e.g. JSON or Storable messages, and so on.
47 51
48=head2 COMPARISON TO OTHER MODULES 52=head2 COMPARISON TO OTHER MODULES
52or L<AnyEvent::Subprocess>. There are modules that implement their own 56or L<AnyEvent::Subprocess>. There are modules that implement their own
53process management, such as L<AnyEvent::DBI>. 57process management, such as L<AnyEvent::DBI>.
54 58
55The problems that all these modules try to solve are real, however, none 59The problems that all these modules try to solve are real, however, none
56of them (from what I have seen) tackle the very real problems of unwanted 60of them (from what I have seen) tackle the very real problems of unwanted
57memory sharing, efficiency, not being able to use event processing or 61memory sharing, efficiency or not being able to use event processing, GUI
58similar modules in the processes they create. 62toolkits or similar modules in the processes they create.
59 63
60This module doesn't try to replace any of them - instead it tries to solve 64This module doesn't try to replace any of them - instead it tries to solve
61the problem of creating processes with a minimum of fuss and overhead (and 65the problem of creating processes with a minimum of fuss and overhead (and
62also luxury). Ideally, most of these would use AnyEvent::Fork internally, 66also luxury). Ideally, most of these would use AnyEvent::Fork internally,
63except they were written before AnyEvent:Fork was available, so obviously 67except they were written before AnyEvent:Fork was available, so obviously
85 89
86=item Forking usually creates a copy-on-write copy of the parent 90=item Forking usually creates a copy-on-write copy of the parent
87process. 91process.
88 92
89For example, modules or data files that are loaded will not use additional 93For example, modules or data files that are loaded will not use additional
90memory after a fork. When exec'ing a new process, modules and data files 94memory after a fork. Exec'ing a new process, in contrast, means modules
91might need to be loaded again, at extra CPU and memory cost. But when 95and data files might need to be loaded again, at extra CPU and memory
92forking, literally all data structures are copied - if the program frees 96cost.
97
98But when forking, you still create a copy of your data structures - if
93them and replaces them by new data, the child processes will retain the 99the program frees them and replaces them by new data, the child processes
94old version even if it isn't used, which can suddenly and unexpectedly 100will retain the old version even if it isn't used, which can suddenly and
95increase memory usage when freeing memory. 101unexpectedly increase memory usage when freeing memory.
96 102
103For example, L<Gtk2::CV> is an image viewer optimised for large
104directories (millions of pictures). It also forks subprocesses for
105thumbnail generation, which inherit the data structure that stores all
106file information. If the user changes the directory, it gets freed in
107the main process, leaving a copy in the thumbnailer processes. This can
108lead to many times the memory usage that would actually be required. The
109solution is to fork early (and being unable to dynamically generate more
110subprocesses or do this from a module)... or to use L<AnyEvent:Fork>.
111
97The trade-off is between more sharing with fork (which can be good or 112There is a trade-off between more sharing with fork (which can be good or
98bad), and no sharing with exec. 113bad), and no sharing with exec.
99 114
100This module allows the main program to do a controlled fork, and allows 115This module allows the main program to do a controlled fork, and allows
101modules to exec processes safely at any time. When creating a custom 116modules to exec processes safely at any time. When creating a custom
102process pool you can take advantage of data sharing via fork without 117process pool you can take advantage of data sharing via fork without
107shared and what isn't, at all times. 122shared and what isn't, at all times.
108 123
109=item Exec'ing a new perl process might be difficult. 124=item Exec'ing a new perl process might be difficult.
110 125
111For example, it is not easy to find the correct path to the perl 126For example, it is not easy to find the correct path to the perl
112interpreter - C<$^X> might not be a perl interpreter at all. 127interpreter - C<$^X> might not be a perl interpreter at all. Worse, there
128might not even be a perl binary installed on the system.
113 129
114This module tries hard to identify the correct path to the perl 130This module tries hard to identify the correct path to the perl
115interpreter. With a cooperative main program, exec'ing the interpreter 131interpreter. With a cooperative main program, exec'ing the interpreter
116might not even be necessary, but even without help from the main program, 132might not even be necessary, but even without help from the main program,
117it will still work when used from a module. 133it will still work when used from a module.
123and modules are no longer loadable because they refer to a different 139and modules are no longer loadable because they refer to a different
124perl version, or parts of a distribution are newer than the ones already 140perl version, or parts of a distribution are newer than the ones already
125loaded. 141loaded.
126 142
127This module supports creating pre-initialised perl processes to be used as 143This module supports creating pre-initialised perl processes to be used as
128a template for new processes. 144a template for new processes at a later time, e.g. for use in a process
145pool.
129 146
130=item Forking might be impossible when a program is running. 147=item Forking might be impossible when a program is running.
131 148
132For example, POSIX makes it almost impossible to fork from a 149For example, POSIX makes it almost impossible to fork from a
133multi-threaded program while doing anything useful in the child - in 150multi-threaded program while doing anything useful in the child - in
134fact, if your perl program uses POSIX threads (even indirectly via 151fact, if your perl program uses POSIX threads (even indirectly via
135e.g. L<IO::AIO> or L<threads>), you cannot call fork on the perl level 152e.g. L<IO::AIO> or L<threads>), you cannot call fork on the perl level
136anymore without risking corruption issues on a number of operating 153anymore without risking memory corruption or worse on a number of
137systems. 154operating systems.
138 155
139This module can safely fork helper processes at any time, by calling 156This module can safely fork helper processes at any time, by calling
140fork+exec in C, in a POSIX-compatible way (via L<Proc::FastSpawn>). 157fork+exec in C, in a POSIX-compatible way (via L<Proc::FastSpawn>).
141 158
142=item Parallel processing with fork might be inconvenient or difficult 159=item Parallel processing with fork might be inconvenient or difficult
161 178
162=back 179=back
163 180
164=head1 EXAMPLES 181=head1 EXAMPLES
165 182
183This is where the wall of text ends and code speaks.
184
166=head2 Create a single new process, tell it to run your worker function. 185=head2 Create a single new process, tell it to run your worker function.
167 186
168 AnyEvent::Fork 187 AnyEvent::Fork
169 ->new 188 ->new
170 ->require ("MyModule") 189 ->require ("MyModule")
181 200
182 sub worker { 201 sub worker {
183 my ($slave_filehandle) = @_; 202 my ($slave_filehandle) = @_;
184 203
185 # now $slave_filehandle is connected to the $master_filehandle 204 # now $slave_filehandle is connected to the $master_filehandle
186 # in the original prorcess. have fun! 205 # in the original process. have fun!
187 } 206 }
188 207
189=head2 Create a pool of server processes all accepting on the same socket. 208=head2 Create a pool of server processes all accepting on the same socket.
190 209
191 # create listener socket 210 # create listener socket
250 ->send_arg ("/bin/echo", "hi") 269 ->send_arg ("/bin/echo", "hi")
251 ->run ("run", my $cv = AE::cv); 270 ->run ("run", my $cv = AE::cv);
252 271
253 my $stderr = $cv->recv; 272 my $stderr = $cv->recv;
254 273
274=head2 For stingy users: put the worker code into a C<DATA> section.
275
276When you want to be stingy with files, you can put your code into the
277C<DATA> section of your module (or program):
278
279 use AnyEvent::Fork;
280
281 AnyEvent::Fork
282 ->new
283 ->eval (do { local $/; <DATA> })
284 ->run ("doit", sub { ... });
285
286 __DATA__
287
288 sub doit {
289 ... do something!
290 }
291
292=head2 For stingy standalone programs: do not rely on external files at
293all.
294
295For single-file scripts it can be inconvenient to rely on external
296files - even when using a C<DATA> section, you still need to C<exec> an
297external perl interpreter, which might not be available when using
298L<App::Staticperl>, L<Urlader> or L<PAR::Packer> for example.
299
300Two modules help here - L<AnyEvent::Fork::Early> forks a template process
301for all further calls to C<new_exec>, and L<AnyEvent::Fork::Template>
302forks the main program as a template process.
303
304Here is how your main program should look like:
305
306 #! perl
307
308 # optional, as the very first thing.
309 # in case modules want to create their own processes.
310 use AnyEvent::Fork::Early;
311
312 # next, load all modules you need in your template process
313 use Example::My::Module
314 use Example::Whatever;
315
316 # next, put your run function definition and anything else you
317 # need, but do not use code outside of BEGIN blocks.
318 sub worker_run {
319 my ($fh, @args) = @_;
320 ...
321 }
322
323 # now preserve everything so far as AnyEvent::Fork object
324 # in $TEMPLATE.
325 use AnyEvent::Fork::Template;
326
327 # do not put code outside of BEGIN blocks until here
328
329 # now use the $TEMPLATE process in any way you like
330
331 # for example: create 10 worker processes
332 my @worker;
333 my $cv = AE::cv;
334 for (1..10) {
335 $cv->begin;
336 $TEMPLATE->fork->send_arg ($_)->run ("worker_run", sub {
337 push @worker, shift;
338 $cv->end;
339 });
340 }
341 $cv->recv;
342
255=head1 CONCEPTS 343=head1 CONCEPTS
256 344
257This module can create new processes either by executing a new perl 345This module can create new processes either by executing a new perl
258process, or by forking from an existing "template" process. 346process, or by forking from an existing "template" process.
259 347
377use AnyEvent; 465use AnyEvent;
378use AnyEvent::Util (); 466use AnyEvent::Util ();
379 467
380use IO::FDPass; 468use IO::FDPass;
381 469
382our $VERSION = 0.7; 470our $VERSION = 1.3;
383 471
384# the early fork template process 472# the early fork template process
385our $EARLY; 473our $EARLY;
386 474
387# the empty template process 475# the empty template process
468 556
469 if ($pid eq 0) { 557 if ($pid eq 0) {
470 require AnyEvent::Fork::Serve; 558 require AnyEvent::Fork::Serve;
471 $AnyEvent::Fork::Serve::OWNER = $parent; 559 $AnyEvent::Fork::Serve::OWNER = $parent;
472 close $fh; 560 close $fh;
473 $0 = "$_[1] of $parent"; 561 $0 = "$AnyEvent::Fork::Serve::OWNER AnyEvent::Fork/exec";
474 AnyEvent::Fork::Serve::serve ($slave); 562 AnyEvent::Fork::Serve::serve ($slave);
475 exit 0; 563 exit 0;
476 } elsif (!$pid) { 564 } elsif (!$pid) {
477 die "AnyEvent::Fork::Early/Template: unable to fork template process: $!"; 565 die "AnyEvent::Fork::Early/Template: unable to fork template process: $!";
478 } 566 }
532 620
533You should use C<new> whenever possible, except when having a template 621You should use C<new> whenever possible, except when having a template
534process around is unacceptable. 622process around is unacceptable.
535 623
536The path to the perl interpreter is divined using various methods - first 624The path to the perl interpreter is divined using various methods - first
537C<$^X> is investigated to see if the path ends with something that sounds 625C<$^X> is investigated to see if the path ends with something that looks
538as if it were the perl interpreter. Failing this, the module falls back to 626as if it were the perl interpreter. Failing this, the module falls back to
539using C<$Config::Config{perlpath}>. 627using C<$Config::Config{perlpath}>.
540 628
629The path to perl can also be overriden by setting the global variable
630C<$AnyEvent::Fork::PERL> - it's value will be used for all subsequent
631invocations.
632
541=cut 633=cut
634
635our $PERL;
542 636
543sub new_exec { 637sub new_exec {
544 my ($self) = @_; 638 my ($self) = @_;
545 639
546 return $EARLY->fork 640 return $EARLY->fork
547 if $EARLY; 641 if $EARLY;
548 642
643 unless (defined $PERL) {
549 # first find path of perl 644 # first find path of perl
550 my $perl = $; 645 my $perl = $^X;
551 646
552 # first we try $^X, but the path must be absolute (always on win32), and end in sth. 647 # first we try $^X, but the path must be absolute (always on win32), and end in sth.
553 # that looks like perl. this obviously only works for posix and win32 648 # that looks like perl. this obviously only works for posix and win32
554 unless ( 649 unless (
555 ($^O eq "MSWin32" || $perl =~ m%^/%) 650 ($^O eq "MSWin32" || $perl =~ m%^/%)
556 && $perl =~ m%[/\\]perl(?:[0-9]+(\.[0-9]+)+)?(\.exe)?$%i 651 && $perl =~ m%[/\\]perl(?:[0-9]+(\.[0-9]+)+)?(\.exe)?$%i
557 ) { 652 ) {
558 # if it doesn't look perlish enough, try Config 653 # if it doesn't look perlish enough, try Config
559 require Config; 654 require Config;
560 $perl = $Config::Config{perlpath}; 655 $perl = $Config::Config{perlpath};
561 $perl =~ s/(?:\Q$Config::Config{_exe}\E)?$/$Config::Config{_exe}/; 656 $perl =~ s/(?:\Q$Config::Config{_exe}\E)?$/$Config::Config{_exe}/;
657 }
658
659 $PERL = $perl;
562 } 660 }
563 661
564 require Proc::FastSpawn; 662 require Proc::FastSpawn;
565 663
566 my ($fh, $slave) = AnyEvent::Util::portable_socketpair; 664 my ($fh, $slave) = AnyEvent::Util::portable_socketpair;
574 #local $ENV{PERL5LIB} = join ":", grep !ref, @INC; 672 #local $ENV{PERL5LIB} = join ":", grep !ref, @INC;
575 my %env = %ENV; 673 my %env = %ENV;
576 $env{PERL5LIB} = join +($^O eq "MSWin32" ? ";" : ":"), grep !ref, @INC; 674 $env{PERL5LIB} = join +($^O eq "MSWin32" ? ";" : ":"), grep !ref, @INC;
577 675
578 my $pid = Proc::FastSpawn::spawn ( 676 my $pid = Proc::FastSpawn::spawn (
579 $perl, 677 $PERL,
580 ["perl", "-MAnyEvent::Fork::Serve", "-e", "AnyEvent::Fork::Serve::me", fileno $slave, $$], 678 [$PERL, "-MAnyEvent::Fork::Serve", "-e", "AnyEvent::Fork::Serve::me", fileno $slave, $$],
581 [map "$_=$env{$_}", keys %env], 679 [map "$_=$env{$_}", keys %env],
582 ) or die "unable to spawn AnyEvent::Fork server: $!"; 680 ) or die "unable to spawn AnyEvent::Fork server: $!";
583 681
584 $self->_new ($fh, $pid) 682 $self->_new ($fh, $pid)
585} 683}
586 684
587=item $pid = $proc->pid 685=item $pid = $proc->pid
588 686
589Returns the process id of the process I<iff it is a direct child of the 687Returns the process id of the process I<iff it is a direct child of the
590process running AnyEvent::Fork>, and C<undef> otherwise. 688process running AnyEvent::Fork>, and C<undef> otherwise. As a general
689rule (that you cannot rely upon), processes created via C<new_exec>,
690L<AnyEvent::Fork::Early> or L<AnyEvent::Fork::Template> are direct
691children, while all other processes are not.
591 692
592Normally, only processes created via C<< AnyEvent::Fork->new_exec >> and 693Or in other words, you do not normally have to take care of zombies for
593L<AnyEvent::Fork::Template> are direct children, and you are responsible 694processes created via C<new>, but when in doubt, or zombies are a problem,
594to clean up their zombies when they die. 695you need to check whether a process is a diretc child by calling this
595 696method, and possibly creating a child watcher or reap it manually.
596All other processes are not direct children, and will be cleaned up by
597AnyEvent::Fork itself.
598 697
599=cut 698=cut
600 699
601sub pid { 700sub pid {
602 $_[0][PID] 701 $_[0][PID]
621C<send_xxx> methods, such as file handles. See the L<use AnyEvent::Fork as 720C<send_xxx> methods, such as file handles. See the L<use AnyEvent::Fork as
622a faster fork+exec> example to see it in action. 721a faster fork+exec> example to see it in action.
623 722
624Returns the process object for easy chaining of method calls. 723Returns the process object for easy chaining of method calls.
625 724
725It's common to want to call an iniitalisation function with some
726arguments. Make sure you actually pass C<@_> to that function (for example
727by using C<&name> syntax), and do not just specify a function name:
728
729 $proc->eval ('&MyModule::init', $string1, $string2);
730
626=cut 731=cut
627 732
628sub eval { 733sub eval {
629 my ($self, $code, @args) = @_; 734 my ($self, $code, @args) = @_;
630 735
733 838
734Even if not used otherwise, the socket can be a good indicator for the 839Even if not used otherwise, the socket can be a good indicator for the
735existence of the process - if the other process exits, you get a readable 840existence of the process - if the other process exits, you get a readable
736event on it, because exiting the process closes the socket (if it didn't 841event on it, because exiting the process closes the socket (if it didn't
737create any children using fork). 842create any children using fork).
843
844=over 4
845
846=item Compatibility to L<AnyEvent::Fork::Remote>
847
848If you want to write code that works with both this module and
849L<AnyEvent::Fork::Remote>, you need to write your code so that it assumes
850there are two file handles for communications, which might not be unix
851domain sockets. The C<run> function should start like this:
852
853 sub run {
854 my ($rfh, @args) = @_; # @args is your normal arguments
855 my $wfh = fileno $rfh ? $rfh : *STDOUT;
856
857 # now use $rfh for reading and $wfh for writing
858 }
859
860This checks whether the passed file handle is, in fact, the process
861C<STDIN> handle. If it is, then the function was invoked visa
862L<AnyEvent::Fork::Remote>, so STDIN should be used for reading and
863C<STDOUT> should be used for writing.
864
865In all other cases, the function was called via this module, and there is
866only one file handle that should be sued for reading and writing.
867
868=back
738 869
739Example: create a template for a process pool, pass a few strings, some 870Example: create a template for a process pool, pass a few strings, some
740file handles, then fork, pass one more string, and run some code. 871file handles, then fork, pass one more string, and run some code.
741 872
742 my $pool = AnyEvent::Fork 873 my $pool = AnyEvent::Fork
774 905
775 $self->[CB] = $cb; 906 $self->[CB] = $cb;
776 $self->_cmd (r => $func); 907 $self->_cmd (r => $func);
777} 908}
778 909
910=back
911
912=head2 EXPERIMENTAL METHODS
913
914These methods might go away completely or change behaviour, at any time.
915
916=over 4
917
779=item $proc->to_fh ($cb->($fh)) # EXPERIMENTAL, MIGHT BE REMOVED 918=item $proc->to_fh ($cb->($fh)) # EXPERIMENTAL, MIGHT BE REMOVED
780 919
781Flushes all commands out to the process and then calls the callback with 920Flushes all commands out to the process and then calls the callback with
782the communications socket. 921the communications socket.
783 922
784The process object becomes unusable on return from this function - any 923The process object becomes unusable on return from this function - any
785further method calls result in undefined behaviour. 924further method calls result in undefined behaviour.
786 925
787The point of this method is to give you a file handle thta you cna pass 926The point of this method is to give you a file handle that you can pass
788to another process. In that other process, you can call C<new_from_fh 927to another process. In that other process, you can call C<new_from_fh
789AnyEvent::Fork::RPC> to create a new C<AnyEvent::Fork> object from it, 928AnyEvent::Fork $fh> to create a new C<AnyEvent::Fork> object from it,
790thereby effectively passing a fork object to another process. 929thereby effectively passing a fork object to another process.
791 930
792=cut 931=cut
793 932
794sub to_fh { 933sub to_fh {
962Cygwin perl is not supported at the moment due to some hilarious 1101Cygwin perl is not supported at the moment due to some hilarious
963shortcomings of its API - see L<IO::FDPoll> for more details. If you never 1102shortcomings of its API - see L<IO::FDPoll> for more details. If you never
964use C<send_fh> and always use C<new_exec> to create processes, it should 1103use C<send_fh> and always use C<new_exec> to create processes, it should
965work though. 1104work though.
966 1105
1106=head1 USING AnyEvent::Fork IN SUBPROCESSES
1107
1108AnyEvent::Fork itself cannot generally be used in subprocesses. As long as
1109only one process ever forks new processes, sharing the template processes
1110is possible (you could use a pipe as a lock by writing a byte into it to
1111unlock, and reading the byte to lock for example)
1112
1113To make concurrent calls possible after fork, you should get rid of the
1114template and early fork processes. AnyEvent::Fork will create a new
1115template process as needed.
1116
1117 undef $AnyEvent::Fork::EARLY;
1118 undef $AnyEvent::Fork::TEMPLATE;
1119
1120It doesn't matter whether you get rid of them in the parent or child after
1121a fork.
1122
967=head1 SEE ALSO 1123=head1 SEE ALSO
968 1124
969L<AnyEvent::Fork::Early>, to avoid executing a perl interpreter at all 1125L<AnyEvent::Fork::Early>, to avoid executing a perl interpreter at all
970(part of this distribution). 1126(part of this distribution).
971 1127
972L<AnyEvent::Fork::Template>, to create a process by forking the main 1128L<AnyEvent::Fork::Template>, to create a process by forking the main
973program at a convenient time (part of this distribution). 1129program at a convenient time (part of this distribution).
974 1130
1131L<AnyEvent::Fork::Remote>, for another way to create processes that is
1132mostly compatible to this module and modules building on top of it, but
1133works better with remote processes.
1134
975L<AnyEvent::Fork::RPC>, for simple RPC to child processes (on CPAN). 1135L<AnyEvent::Fork::RPC>, for simple RPC to child processes (on CPAN).
1136
1137L<AnyEvent::Fork::Pool>, for simple worker process pool (on CPAN).
976 1138
977=head1 AUTHOR AND CONTACT INFORMATION 1139=head1 AUTHOR AND CONTACT INFORMATION
978 1140
979 Marc Lehmann <schmorp@schmorp.de> 1141 Marc Lehmann <schmorp@schmorp.de>
980 http://software.schmorp.de/pkg/AnyEvent-Fork 1142 http://software.schmorp.de/pkg/AnyEvent-Fork

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines