ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-Fork/Fork.pm
(Generate patch)

Comparing AnyEvent-Fork/Fork.pm (file contents):
Revision 1.54 by root, Fri Apr 26 17:24:05 2013 UTC vs.
Revision 1.66 by root, Sat Jun 20 13:24:54 2015 UTC

56or L<AnyEvent::Subprocess>. There are modules that implement their own 56or L<AnyEvent::Subprocess>. There are modules that implement their own
57process management, such as L<AnyEvent::DBI>. 57process management, such as L<AnyEvent::DBI>.
58 58
59The problems that all these modules try to solve are real, however, none 59The problems that all these modules try to solve are real, however, none
60of them (from what I have seen) tackle the very real problems of unwanted 60of them (from what I have seen) tackle the very real problems of unwanted
61memory sharing, efficiency, not being able to use event processing or 61memory sharing, efficiency or not being able to use event processing, GUI
62similar modules in the processes they create. 62toolkits or similar modules in the processes they create.
63 63
64This module doesn't try to replace any of them - instead it tries to solve 64This module doesn't try to replace any of them - instead it tries to solve
65the problem of creating processes with a minimum of fuss and overhead (and 65the problem of creating processes with a minimum of fuss and overhead (and
66also luxury). Ideally, most of these would use AnyEvent::Fork internally, 66also luxury). Ideally, most of these would use AnyEvent::Fork internally,
67except they were written before AnyEvent:Fork was available, so obviously 67except they were written before AnyEvent:Fork was available, so obviously
89 89
90=item Forking usually creates a copy-on-write copy of the parent 90=item Forking usually creates a copy-on-write copy of the parent
91process. 91process.
92 92
93For example, modules or data files that are loaded will not use additional 93For example, modules or data files that are loaded will not use additional
94memory after a fork. When exec'ing a new process, modules and data files 94memory after a fork. Exec'ing a new process, in contrast, means modules
95might need to be loaded again, at extra CPU and memory cost. But when 95and data files might need to be loaded again, at extra CPU and memory
96forking, literally all data structures are copied - if the program frees 96cost.
97
98But when forking, you still create a copy of your data structures - if
97them and replaces them by new data, the child processes will retain the 99the program frees them and replaces them by new data, the child processes
98old version even if it isn't used, which can suddenly and unexpectedly 100will retain the old version even if it isn't used, which can suddenly and
99increase memory usage when freeing memory. 101unexpectedly increase memory usage when freeing memory.
100 102
103For example, L<Gtk2::CV> is an image viewer optimised for large
104directories (millions of pictures). It also forks subprocesses for
105thumbnail generation, which inherit the data structure that stores all
106file information. If the user changes the directory, it gets freed in
107the main process, leaving a copy in the thumbnailer processes. This can
108lead to many times the memory usage that would actually be required. The
109solution is to fork early (and being unable to dynamically generate more
110subprocesses or do this from a module)... or to use L<AnyEvent:Fork>.
111
101The trade-off is between more sharing with fork (which can be good or 112There is a trade-off between more sharing with fork (which can be good or
102bad), and no sharing with exec. 113bad), and no sharing with exec.
103 114
104This module allows the main program to do a controlled fork, and allows 115This module allows the main program to do a controlled fork, and allows
105modules to exec processes safely at any time. When creating a custom 116modules to exec processes safely at any time. When creating a custom
106process pool you can take advantage of data sharing via fork without 117process pool you can take advantage of data sharing via fork without
111shared and what isn't, at all times. 122shared and what isn't, at all times.
112 123
113=item Exec'ing a new perl process might be difficult. 124=item Exec'ing a new perl process might be difficult.
114 125
115For example, it is not easy to find the correct path to the perl 126For example, it is not easy to find the correct path to the perl
116interpreter - C<$^X> might not be a perl interpreter at all. 127interpreter - C<$^X> might not be a perl interpreter at all. Worse, there
128might not even be a perl binary installed on the system.
117 129
118This module tries hard to identify the correct path to the perl 130This module tries hard to identify the correct path to the perl
119interpreter. With a cooperative main program, exec'ing the interpreter 131interpreter. With a cooperative main program, exec'ing the interpreter
120might not even be necessary, but even without help from the main program, 132might not even be necessary, but even without help from the main program,
121it will still work when used from a module. 133it will still work when used from a module.
127and modules are no longer loadable because they refer to a different 139and modules are no longer loadable because they refer to a different
128perl version, or parts of a distribution are newer than the ones already 140perl version, or parts of a distribution are newer than the ones already
129loaded. 141loaded.
130 142
131This module supports creating pre-initialised perl processes to be used as 143This module supports creating pre-initialised perl processes to be used as
132a template for new processes. 144a template for new processes at a later time, e.g. for use in a process
145pool.
133 146
134=item Forking might be impossible when a program is running. 147=item Forking might be impossible when a program is running.
135 148
136For example, POSIX makes it almost impossible to fork from a 149For example, POSIX makes it almost impossible to fork from a
137multi-threaded program while doing anything useful in the child - in 150multi-threaded program while doing anything useful in the child - in
138fact, if your perl program uses POSIX threads (even indirectly via 151fact, if your perl program uses POSIX threads (even indirectly via
139e.g. L<IO::AIO> or L<threads>), you cannot call fork on the perl level 152e.g. L<IO::AIO> or L<threads>), you cannot call fork on the perl level
140anymore without risking corruption issues on a number of operating 153anymore without risking memory corruption or worse on a number of
141systems. 154operating systems.
142 155
143This module can safely fork helper processes at any time, by calling 156This module can safely fork helper processes at any time, by calling
144fork+exec in C, in a POSIX-compatible way (via L<Proc::FastSpawn>). 157fork+exec in C, in a POSIX-compatible way (via L<Proc::FastSpawn>).
145 158
146=item Parallel processing with fork might be inconvenient or difficult 159=item Parallel processing with fork might be inconvenient or difficult
165 178
166=back 179=back
167 180
168=head1 EXAMPLES 181=head1 EXAMPLES
169 182
183This is where the wall of text ends and code speaks.
184
170=head2 Create a single new process, tell it to run your worker function. 185=head2 Create a single new process, tell it to run your worker function.
171 186
172 AnyEvent::Fork 187 AnyEvent::Fork
173 ->new 188 ->new
174 ->require ("MyModule") 189 ->require ("MyModule")
185 200
186 sub worker { 201 sub worker {
187 my ($slave_filehandle) = @_; 202 my ($slave_filehandle) = @_;
188 203
189 # now $slave_filehandle is connected to the $master_filehandle 204 # now $slave_filehandle is connected to the $master_filehandle
190 # in the original prorcess. have fun! 205 # in the original process. have fun!
191 } 206 }
192 207
193=head2 Create a pool of server processes all accepting on the same socket. 208=head2 Create a pool of server processes all accepting on the same socket.
194 209
195 # create listener socket 210 # create listener socket
256 271
257 my $stderr = $cv->recv; 272 my $stderr = $cv->recv;
258 273
259=head2 For stingy users: put the worker code into a C<DATA> section. 274=head2 For stingy users: put the worker code into a C<DATA> section.
260 275
261When you want to be stingy with files, you cna put your code into the 276When you want to be stingy with files, you can put your code into the
262C<DATA> section of your module (or program): 277C<DATA> section of your module (or program):
263 278
264 use AnyEvent::Fork; 279 use AnyEvent::Fork;
265 280
266 AnyEvent::Fork 281 AnyEvent::Fork
276 291
277=head2 For stingy standalone programs: do not rely on external files at 292=head2 For stingy standalone programs: do not rely on external files at
278all. 293all.
279 294
280For single-file scripts it can be inconvenient to rely on external 295For single-file scripts it can be inconvenient to rely on external
281files - even when using < C<DATA> section, you still need to C<exec> 296files - even when using a C<DATA> section, you still need to C<exec> an
282an external perl interpreter, which might not be available when using 297external perl interpreter, which might not be available when using
283L<App::Staticperl>, L<Urlader> or L<PAR::Packer> for example. 298L<App::Staticperl>, L<Urlader> or L<PAR::Packer> for example.
284 299
285Two modules help here - L<AnyEvent::Fork::Early> forks a template process 300Two modules help here - L<AnyEvent::Fork::Early> forks a template process
286for all further calls to C<new_exec>, and L<AnyEvent::Fork::Template> 301for all further calls to C<new_exec>, and L<AnyEvent::Fork::Template>
287forks the main program as a template process. 302forks the main program as a template process.
304 my ($fh, @args) = @_; 319 my ($fh, @args) = @_;
305 ... 320 ...
306 } 321 }
307 322
308 # now preserve everything so far as AnyEvent::Fork object 323 # now preserve everything so far as AnyEvent::Fork object
309 # in ยงTEMPLATE. 324 # in $TEMPLATE.
310 use AnyEvent::Fork::Template; 325 use AnyEvent::Fork::Template;
311 326
312 # do not put code outside of BEGIN blocks until here 327 # do not put code outside of BEGIN blocks until here
313 328
314 # now use the $TEMPLATE process in any way you like 329 # now use the $TEMPLATE process in any way you like
450use AnyEvent; 465use AnyEvent;
451use AnyEvent::Util (); 466use AnyEvent::Util ();
452 467
453use IO::FDPass; 468use IO::FDPass;
454 469
455our $VERSION = '1.0'; 470our $VERSION = 1.2;
456 471
457# the early fork template process 472# the early fork template process
458our $EARLY; 473our $EARLY;
459 474
460# the empty template process 475# the empty template process
485 my $self = shift; 500 my $self = shift;
486 501
487 # ideally, we would want to use "a (w/a)*" as format string, but perl 502 # ideally, we would want to use "a (w/a)*" as format string, but perl
488 # versions from at least 5.8.9 to 5.16.3 are all buggy and can't unpack 503 # versions from at least 5.8.9 to 5.16.3 are all buggy and can't unpack
489 # it. 504 # it.
490 push @{ $self->[QUEUE] }, pack "a N/a*", $_[0], $_[1]; 505 push @{ $self->[QUEUE] }, pack "a L/a*", $_[0], $_[1];
491 506
492 $self->[WW] ||= AE::io $self->[FH], 1, sub { 507 $self->[WW] ||= AE::io $self->[FH], 1, sub {
493 do { 508 do {
494 # send the next "thing" in the queue - either a reference to an fh, 509 # send the next "thing" in the queue - either a reference to an fh,
495 # or a plain string. 510 # or a plain string.
605 620
606You should use C<new> whenever possible, except when having a template 621You should use C<new> whenever possible, except when having a template
607process around is unacceptable. 622process around is unacceptable.
608 623
609The path to the perl interpreter is divined using various methods - first 624The path to the perl interpreter is divined using various methods - first
610C<$^X> is investigated to see if the path ends with something that sounds 625C<$^X> is investigated to see if the path ends with something that looks
611as if it were the perl interpreter. Failing this, the module falls back to 626as if it were the perl interpreter. Failing this, the module falls back to
612using C<$Config::Config{perlpath}>. 627using C<$Config::Config{perlpath}>.
613 628
629The path to perl can also be overriden by setting the global variable
630C<$AnyEvent::Fork::PERL> - it's value will be used for all subsequent
631invocations.
632
614=cut 633=cut
634
635our $PERL;
615 636
616sub new_exec { 637sub new_exec {
617 my ($self) = @_; 638 my ($self) = @_;
618 639
619 return $EARLY->fork 640 return $EARLY->fork
620 if $EARLY; 641 if $EARLY;
621 642
643 unless (defined $PERL) {
622 # first find path of perl 644 # first find path of perl
623 my $perl = $; 645 my $perl = $^X;
624 646
625 # first we try $^X, but the path must be absolute (always on win32), and end in sth. 647 # first we try $^X, but the path must be absolute (always on win32), and end in sth.
626 # that looks like perl. this obviously only works for posix and win32 648 # that looks like perl. this obviously only works for posix and win32
627 unless ( 649 unless (
628 ($^O eq "MSWin32" || $perl =~ m%^/%) 650 ($^O eq "MSWin32" || $perl =~ m%^/%)
629 && $perl =~ m%[/\\]perl(?:[0-9]+(\.[0-9]+)+)?(\.exe)?$%i 651 && $perl =~ m%[/\\]perl(?:[0-9]+(\.[0-9]+)+)?(\.exe)?$%i
630 ) { 652 ) {
631 # if it doesn't look perlish enough, try Config 653 # if it doesn't look perlish enough, try Config
632 require Config; 654 require Config;
633 $perl = $Config::Config{perlpath}; 655 $perl = $Config::Config{perlpath};
634 $perl =~ s/(?:\Q$Config::Config{_exe}\E)?$/$Config::Config{_exe}/; 656 $perl =~ s/(?:\Q$Config::Config{_exe}\E)?$/$Config::Config{_exe}/;
657 }
658
659 $PERL = $perl;
635 } 660 }
636 661
637 require Proc::FastSpawn; 662 require Proc::FastSpawn;
638 663
639 my ($fh, $slave) = AnyEvent::Util::portable_socketpair; 664 my ($fh, $slave) = AnyEvent::Util::portable_socketpair;
647 #local $ENV{PERL5LIB} = join ":", grep !ref, @INC; 672 #local $ENV{PERL5LIB} = join ":", grep !ref, @INC;
648 my %env = %ENV; 673 my %env = %ENV;
649 $env{PERL5LIB} = join +($^O eq "MSWin32" ? ";" : ":"), grep !ref, @INC; 674 $env{PERL5LIB} = join +($^O eq "MSWin32" ? ";" : ":"), grep !ref, @INC;
650 675
651 my $pid = Proc::FastSpawn::spawn ( 676 my $pid = Proc::FastSpawn::spawn (
652 $perl, 677 $PERL,
653 ["perl", "-MAnyEvent::Fork::Serve", "-e", "AnyEvent::Fork::Serve::me", fileno $slave, $$], 678 [$PERL, "-MAnyEvent::Fork::Serve", "-e", "AnyEvent::Fork::Serve::me", fileno $slave, $$],
654 [map "$_=$env{$_}", keys %env], 679 [map "$_=$env{$_}", keys %env],
655 ) or die "unable to spawn AnyEvent::Fork server: $!"; 680 ) or die "unable to spawn AnyEvent::Fork server: $!";
656 681
657 $self->_new ($fh, $pid) 682 $self->_new ($fh, $pid)
658} 683}
659 684
660=item $pid = $proc->pid 685=item $pid = $proc->pid
661 686
662Returns the process id of the process I<iff it is a direct child of the 687Returns the process id of the process I<iff it is a direct child of the
663process running AnyEvent::Fork>, and C<undef> otherwise. 688process running AnyEvent::Fork>, and C<undef> otherwise. As a general
689rule (that you cannot rely upon), processes created via C<new_exec>,
690L<AnyEvent::Fork::Early> or L<AnyEvent::Fork::Template> are direct
691children, while all other processes are not.
664 692
665Normally, only processes created via C<< AnyEvent::Fork->new_exec >> and 693Or in other words, you do not normally have to take care of zombies for
666L<AnyEvent::Fork::Template> are direct children, and you are responsible 694processes created via C<new>, but when in doubt, or zombies are a problem,
667to clean up their zombies when they die. 695you need to check whether a process is a diretc child by calling this
668 696method, and possibly creating a child watcher or reap it manually.
669All other processes are not direct children, and will be cleaned up by
670AnyEvent::Fork itself.
671 697
672=cut 698=cut
673 699
674sub pid { 700sub pid {
675 $_[0][PID] 701 $_[0][PID]
694C<send_xxx> methods, such as file handles. See the L<use AnyEvent::Fork as 720C<send_xxx> methods, such as file handles. See the L<use AnyEvent::Fork as
695a faster fork+exec> example to see it in action. 721a faster fork+exec> example to see it in action.
696 722
697Returns the process object for easy chaining of method calls. 723Returns the process object for easy chaining of method calls.
698 724
725It's common to want to call an iniitalisation function with some
726arguments. Make sure you actually pass C<@_> to that function (for example
727by using C<&name> syntax), and do not just specify a function name:
728
729 $proc->eval ('&MyModule::init', $string1, $string2);
730
699=cut 731=cut
700 732
701sub eval { 733sub eval {
702 my ($self, $code, @args) = @_; 734 my ($self, $code, @args) = @_;
703 735
806 838
807Even if not used otherwise, the socket can be a good indicator for the 839Even if not used otherwise, the socket can be a good indicator for the
808existence of the process - if the other process exits, you get a readable 840existence of the process - if the other process exits, you get a readable
809event on it, because exiting the process closes the socket (if it didn't 841event on it, because exiting the process closes the socket (if it didn't
810create any children using fork). 842create any children using fork).
843
844=over 4
845
846=item Compatibility to L<AnyEvent::Fork::Remote>
847
848If you want to write code that works with both this module and
849L<AnyEvent::Fork::Remote>, you need to write your code so that it assumes
850there are two file handles for communications, which might not be unix
851domain sockets. The C<run> function should start like this:
852
853 sub run {
854 my ($rfh, @args) = @_; # @args is your normal arguments
855 my $wfh = fileno $rfh ? $rfh : *STDOUT;
856
857 # now use $rfh for reading and $wfh for writing
858 }
859
860This checks whether the passed file handle is, in fact, the process
861C<STDIN> handle. If it is, then the function was invoked visa
862L<AnyEvent::Fork::Remote>, so STDIN should be used for reading and
863C<STDOUT> should be used for writing.
864
865In all other cases, the function was called via this module, and there is
866only one file handle that should be sued for reading and writing.
867
868=back
811 869
812Example: create a template for a process pool, pass a few strings, some 870Example: create a template for a process pool, pass a few strings, some
813file handles, then fork, pass one more string, and run some code. 871file handles, then fork, pass one more string, and run some code.
814 872
815 my $pool = AnyEvent::Fork 873 my $pool = AnyEvent::Fork
849 $self->_cmd (r => $func); 907 $self->_cmd (r => $func);
850} 908}
851 909
852=back 910=back
853 911
854=head2 ADVANCED METHODS
855
856=over 4
857
858=item new_from_stdio AnyEvent::Fork $fh
859
860Assume that you have a perl interpreter running (without any special
861options or a program) somewhere and it has it's STDIN and STDOUT connected
862to the C<$fh> somehow. I.e. exactly the state perl is in when you start it
863without any arguments:
864
865 perl
866
867Then you can create an C<AnyEvent::Fork> object out of this perl
868interpreter with this constructor.
869
870When the usefulness of this isn't immediately clear, imagine you manage to
871run a perl interpreter remotely (F<ssh remotemachine perl>), then you can
872manage it mostly like a local C<AnyEvent::Fork> child.
873
874This works without any module support, i.e. the remote F<perl> does not
875need to have any special modules installed.
876
877There are a number of limitations though: C<send_fh> will only work if the
878L<IO::FDPass> module is loadable by the remote perl and the two processes
879are connected in a way that let's L<IO::FDPass> do it's work.
880
881This will therefore not work over a network connection. From this follows
882that C<fork> will also not work under these circumstances, as it relies on
883C<send_fh> internally.
884
885Although not a limitation of this module, keep in mind that the
886"communications socket" is simply C<STDIN>, and depending on how you
887started F<perl> (e.g. via F<ssh>), it might only be half-duplex. This is
888fine for C<AnyEvent::Fork>, but your C<run> function might want to use
889C<STDIN> (or the "communications socket") for input and C<STDOUT> for
890output.
891
892You can support both cases by checking the C<fileno> of the handle passed
893to your run function:
894
895 sub run {
896 my ($rfh) = @_;
897
898 my $wfh = fileno $rfh ? $rfh : *STDOUT;
899
900 # now use $rfh for reading and $wfh for writing
901 }
902
903=cut
904
905sub new_from_stdio {
906 my ($class, $fh) = @_;
907
908 my $self = $class->_new ($fh);
909
910 # send startup code
911 push @{ $self->[QUEUE] },
912 (do "AnyEvent/Fork/serve.pl")
913 . <<'EOF';
914
915$OWNER = "another process";
916$0 = "AnyEvent::Fork/stdio of $OWNER";
917
918serve *STDIN;
919__END__
920EOF
921
922 # the data is only sent when the user requests additional things, which
923 # is likely early enough for our purposes.
924
925 $self
926}
927
928=back
929
930=head2 EXPERIMENTAL METHODS 912=head2 EXPERIMENTAL METHODS
931 913
932These methods might go away completely or change behaviour, a any time. 914These methods might go away completely or change behaviour, at any time.
933 915
934=over 4 916=over 4
935 917
936=item $proc->to_fh ($cb->($fh)) # EXPERIMENTAL, MIGHT BE REMOVED 918=item $proc->to_fh ($cb->($fh)) # EXPERIMENTAL, MIGHT BE REMOVED
937 919
939the communications socket. 921the communications socket.
940 922
941The process object becomes unusable on return from this function - any 923The process object becomes unusable on return from this function - any
942further method calls result in undefined behaviour. 924further method calls result in undefined behaviour.
943 925
944The point of this method is to give you a file handle thta you cna pass 926The point of this method is to give you a file handle that you can pass
945to another process. In that other process, you can call C<new_from_fh 927to another process. In that other process, you can call C<new_from_fh
946AnyEvent::Fork> to create a new C<AnyEvent::Fork> object from it, thereby 928AnyEvent::Fork $fh> to create a new C<AnyEvent::Fork> object from it,
947effectively passing a fork object to another process. 929thereby effectively passing a fork object to another process.
948 930
949=cut 931=cut
950 932
951sub to_fh { 933sub to_fh {
952 my ($self, $cb) = @_; 934 my ($self, $cb) = @_;
1119Cygwin perl is not supported at the moment due to some hilarious 1101Cygwin perl is not supported at the moment due to some hilarious
1120shortcomings of its API - see L<IO::FDPoll> for more details. If you never 1102shortcomings of its API - see L<IO::FDPoll> for more details. If you never
1121use C<send_fh> and always use C<new_exec> to create processes, it should 1103use C<send_fh> and always use C<new_exec> to create processes, it should
1122work though. 1104work though.
1123 1105
1106=head1 USING AnyEvent::Fork IN SUBPROCESSES
1107
1108AnyEvent::Fork itself cannot generally be used in subprocesses. As long as
1109only one process ever forks new processes, sharing the template processes
1110is possible (you could use a pipe as a lock by writing a byte into it to
1111unlock, and reading the byte to lock for example)
1112
1113To make concurrent calls possible after fork, you should get rid of the
1114template and early fork processes. AnyEvent::Fork will create a new
1115template process as needed.
1116
1117 undef $AnyEvent::Fork::EARLY;
1118 undef $AnyEvent::Fork::TEMPLATE;
1119
1120It doesn't matter whether you get rid of them in the parent or child after
1121a fork.
1122
1124=head1 SEE ALSO 1123=head1 SEE ALSO
1125 1124
1126L<AnyEvent::Fork::Early>, to avoid executing a perl interpreter at all 1125L<AnyEvent::Fork::Early>, to avoid executing a perl interpreter at all
1127(part of this distribution). 1126(part of this distribution).
1128 1127
1129L<AnyEvent::Fork::Template>, to create a process by forking the main 1128L<AnyEvent::Fork::Template>, to create a process by forking the main
1130program at a convenient time (part of this distribution). 1129program at a convenient time (part of this distribution).
1131 1130
1131L<AnyEvent::Fork::Remote>, for another way to create processes that is
1132mostly compatible to this module and modules building on top of it, but
1133works better with remote processes.
1134
1132L<AnyEvent::Fork::RPC>, for simple RPC to child processes (on CPAN). 1135L<AnyEvent::Fork::RPC>, for simple RPC to child processes (on CPAN).
1133 1136
1134L<AnyEvent::Fork::Pool>, for simple worker process pool (on CPAN). 1137L<AnyEvent::Fork::Pool>, for simple worker process pool (on CPAN).
1135 1138
1136=head1 AUTHOR AND CONTACT INFORMATION 1139=head1 AUTHOR AND CONTACT INFORMATION

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines