ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-Fork/Fork.pm
(Generate patch)

Comparing AnyEvent-Fork/Fork.pm (file contents):
Revision 1.55 by root, Fri Apr 26 23:24:30 2013 UTC vs.
Revision 1.63 by root, Wed Nov 26 13:36:18 2014 UTC

… …
56or L<AnyEvent::Subprocess>. There are modules that implement their own 56or L<AnyEvent::Subprocess>. There are modules that implement their own
57process management, such as L<AnyEvent::DBI>. 57process management, such as L<AnyEvent::DBI>.
58 58
59The problems that all these modules try to solve are real, however, none 59The problems that all these modules try to solve are real, however, none
60of them (from what I have seen) tackle the very real problems of unwanted 60of them (from what I have seen) tackle the very real problems of unwanted
61memory sharing, efficiency, not being able to use event processing or 61memory sharing, efficiency or not being able to use event processing, GUI
62similar modules in the processes they create. 62toolkits or similar modules in the processes they create.
63 63
64This module doesn't try to replace any of them - instead it tries to solve 64This module doesn't try to replace any of them - instead it tries to solve
65the problem of creating processes with a minimum of fuss and overhead (and 65the problem of creating processes with a minimum of fuss and overhead (and
66also luxury). Ideally, most of these would use AnyEvent::Fork internally, 66also luxury). Ideally, most of these would use AnyEvent::Fork internally,
67except they were written before AnyEvent:Fork was available, so obviously 67except they were written before AnyEvent:Fork was available, so obviously
… …
89 89
90=item Forking usually creates a copy-on-write copy of the parent 90=item Forking usually creates a copy-on-write copy of the parent
91process. 91process.
92 92
93For example, modules or data files that are loaded will not use additional 93For example, modules or data files that are loaded will not use additional
94memory after a fork. When exec'ing a new process, modules and data files 94memory after a fork. Exec'ing a new process, in contrast, means modules
95might need to be loaded again, at extra CPU and memory cost. But when 95and data files might need to be loaded again, at extra CPU and memory
96forking, literally all data structures are copied - if the program frees 96cost.
97
98But when forking, you still create a copy of your data structures - if
97them and replaces them by new data, the child processes will retain the 99the program frees them and replaces them by new data, the child processes
98old version even if it isn't used, which can suddenly and unexpectedly 100will retain the old version even if it isn't used, which can suddenly and
99increase memory usage when freeing memory. 101unexpectedly increase memory usage when freeing memory.
100 102
103For example, L<Gtk2::CV> is an image viewer optimised for large
104directories (millions of pictures). It also forks subprocesses for
105thumbnail generation, which inherit the data structure that stores all
106file information. If the user changes the directory, it gets freed in
107the main process, leaving a copy in the thumbnailer processes. This can
108lead to many times the memory usage that would actually be required. The
109solution is to fork early (and being unable to dynamically generate more
110subprocesses or do this from a module)... or to use L<AnyEvent:Fork>.
111
101The trade-off is between more sharing with fork (which can be good or 112There is a trade-off between more sharing with fork (which can be good or
102bad), and no sharing with exec. 113bad), and no sharing with exec.
103 114
104This module allows the main program to do a controlled fork, and allows 115This module allows the main program to do a controlled fork, and allows
105modules to exec processes safely at any time. When creating a custom 116modules to exec processes safely at any time. When creating a custom
106process pool you can take advantage of data sharing via fork without 117process pool you can take advantage of data sharing via fork without
… …
111shared and what isn't, at all times. 122shared and what isn't, at all times.
112 123
113=item Exec'ing a new perl process might be difficult. 124=item Exec'ing a new perl process might be difficult.
114 125
115For example, it is not easy to find the correct path to the perl 126For example, it is not easy to find the correct path to the perl
116interpreter - C<$^X> might not be a perl interpreter at all. 127interpreter - C<$^X> might not be a perl interpreter at all. Worse, there
128might not even be a perl binary installed on the system.
117 129
118This module tries hard to identify the correct path to the perl 130This module tries hard to identify the correct path to the perl
119interpreter. With a cooperative main program, exec'ing the interpreter 131interpreter. With a cooperative main program, exec'ing the interpreter
120might not even be necessary, but even without help from the main program, 132might not even be necessary, but even without help from the main program,
121it will still work when used from a module. 133it will still work when used from a module.
… …
127and modules are no longer loadable because they refer to a different 139and modules are no longer loadable because they refer to a different
128perl version, or parts of a distribution are newer than the ones already 140perl version, or parts of a distribution are newer than the ones already
129loaded. 141loaded.
130 142
131This module supports creating pre-initialised perl processes to be used as 143This module supports creating pre-initialised perl processes to be used as
132a template for new processes. 144a template for new processes at a later time, e.g. for use in a process
145pool.
133 146
134=item Forking might be impossible when a program is running. 147=item Forking might be impossible when a program is running.
135 148
136For example, POSIX makes it almost impossible to fork from a 149For example, POSIX makes it almost impossible to fork from a
137multi-threaded program while doing anything useful in the child - in 150multi-threaded program while doing anything useful in the child - in
138fact, if your perl program uses POSIX threads (even indirectly via 151fact, if your perl program uses POSIX threads (even indirectly via
139e.g. L<IO::AIO> or L<threads>), you cannot call fork on the perl level 152e.g. L<IO::AIO> or L<threads>), you cannot call fork on the perl level
140anymore without risking corruption issues on a number of operating 153anymore without risking memory corruption or worse on a number of
141systems. 154operating systems.
142 155
143This module can safely fork helper processes at any time, by calling 156This module can safely fork helper processes at any time, by calling
144fork+exec in C, in a POSIX-compatible way (via L<Proc::FastSpawn>). 157fork+exec in C, in a POSIX-compatible way (via L<Proc::FastSpawn>).
145 158
146=item Parallel processing with fork might be inconvenient or difficult 159=item Parallel processing with fork might be inconvenient or difficult
… …
256 269
257 my $stderr = $cv->recv; 270 my $stderr = $cv->recv;
258 271
259=head2 For stingy users: put the worker code into a C<DATA> section. 272=head2 For stingy users: put the worker code into a C<DATA> section.
260 273
261When you want to be stingy with files, you cna put your code into the 274When you want to be stingy with files, you can put your code into the
262C<DATA> section of your module (or program): 275C<DATA> section of your module (or program):
263 276
264 use AnyEvent::Fork; 277 use AnyEvent::Fork;
265 278
266 AnyEvent::Fork 279 AnyEvent::Fork
… …
276 289
277=head2 For stingy standalone programs: do not rely on external files at 290=head2 For stingy standalone programs: do not rely on external files at
278all. 291all.
279 292
280For single-file scripts it can be inconvenient to rely on external 293For single-file scripts it can be inconvenient to rely on external
281files - even when using < C<DATA> section, you still need to C<exec> 294files - even when using a C<DATA> section, you still need to C<exec> an
282an external perl interpreter, which might not be available when using 295external perl interpreter, which might not be available when using
283L<App::Staticperl>, L<Urlader> or L<PAR::Packer> for example. 296L<App::Staticperl>, L<Urlader> or L<PAR::Packer> for example.
284 297
285Two modules help here - L<AnyEvent::Fork::Early> forks a template process 298Two modules help here - L<AnyEvent::Fork::Early> forks a template process
286for all further calls to C<new_exec>, and L<AnyEvent::Fork::Template> 299for all further calls to C<new_exec>, and L<AnyEvent::Fork::Template>
287forks the main program as a template process. 300forks the main program as a template process.
… …
304 my ($fh, @args) = @_; 317 my ($fh, @args) = @_;
305 ... 318 ...
306 } 319 }
307 320
308 # now preserve everything so far as AnyEvent::Fork object 321 # now preserve everything so far as AnyEvent::Fork object
309 # in ยงTEMPLATE. 322 # in $TEMPLATE.
310 use AnyEvent::Fork::Template; 323 use AnyEvent::Fork::Template;
311 324
312 # do not put code outside of BEGIN blocks until here 325 # do not put code outside of BEGIN blocks until here
313 326
314 # now use the $TEMPLATE process in any way you like 327 # now use the $TEMPLATE process in any way you like
… …
450use AnyEvent; 463use AnyEvent;
451use AnyEvent::Util (); 464use AnyEvent::Util ();
452 465
453use IO::FDPass; 466use IO::FDPass;
454 467
455our $VERSION = '1.0'; 468our $VERSION = 1.2;
456 469
457# the early fork template process 470# the early fork template process
458our $EARLY; 471our $EARLY;
459 472
460# the empty template process 473# the empty template process
… …
605 618
606You should use C<new> whenever possible, except when having a template 619You should use C<new> whenever possible, except when having a template
607process around is unacceptable. 620process around is unacceptable.
608 621
609The path to the perl interpreter is divined using various methods - first 622The path to the perl interpreter is divined using various methods - first
610C<$^X> is investigated to see if the path ends with something that sounds 623C<$^X> is investigated to see if the path ends with something that looks
611as if it were the perl interpreter. Failing this, the module falls back to 624as if it were the perl interpreter. Failing this, the module falls back to
612using C<$Config::Config{perlpath}>. 625using C<$Config::Config{perlpath}>.
613 626
627The path to perl can also be overriden by setting the global variable
628C<$AnyEvent::Fork::PERL> - it's value will be used for all subsequent
629invocations.
630
614=cut 631=cut
632
633our $PERL;
615 634
616sub new_exec { 635sub new_exec {
617 my ($self) = @_; 636 my ($self) = @_;
618 637
619 return $EARLY->fork 638 return $EARLY->fork
620 if $EARLY; 639 if $EARLY;
621 640
641 unless (defined $PERL) {
622 # first find path of perl 642 # first find path of perl
623 my $perl = $; 643 my $perl = $^X;
624 644
625 # first we try $^X, but the path must be absolute (always on win32), and end in sth. 645 # first we try $^X, but the path must be absolute (always on win32), and end in sth.
626 # that looks like perl. this obviously only works for posix and win32 646 # that looks like perl. this obviously only works for posix and win32
627 unless ( 647 unless (
628 ($^O eq "MSWin32" || $perl =~ m%^/%) 648 ($^O eq "MSWin32" || $perl =~ m%^/%)
629 && $perl =~ m%[/\\]perl(?:[0-9]+(\.[0-9]+)+)?(\.exe)?$%i 649 && $perl =~ m%[/\\]perl(?:[0-9]+(\.[0-9]+)+)?(\.exe)?$%i
630 ) { 650 ) {
631 # if it doesn't look perlish enough, try Config 651 # if it doesn't look perlish enough, try Config
632 require Config; 652 require Config;
633 $perl = $Config::Config{perlpath}; 653 $perl = $Config::Config{perlpath};
634 $perl =~ s/(?:\Q$Config::Config{_exe}\E)?$/$Config::Config{_exe}/; 654 $perl =~ s/(?:\Q$Config::Config{_exe}\E)?$/$Config::Config{_exe}/;
655 }
656
657 $PERL = $perl;
635 } 658 }
636 659
637 require Proc::FastSpawn; 660 require Proc::FastSpawn;
638 661
639 my ($fh, $slave) = AnyEvent::Util::portable_socketpair; 662 my ($fh, $slave) = AnyEvent::Util::portable_socketpair;
… …
647 #local $ENV{PERL5LIB} = join ":", grep !ref, @INC; 670 #local $ENV{PERL5LIB} = join ":", grep !ref, @INC;
648 my %env = %ENV; 671 my %env = %ENV;
649 $env{PERL5LIB} = join +($^O eq "MSWin32" ? ";" : ":"), grep !ref, @INC; 672 $env{PERL5LIB} = join +($^O eq "MSWin32" ? ";" : ":"), grep !ref, @INC;
650 673
651 my $pid = Proc::FastSpawn::spawn ( 674 my $pid = Proc::FastSpawn::spawn (
652 $perl, 675 $PERL,
653 ["perl", "-MAnyEvent::Fork::Serve", "-e", "AnyEvent::Fork::Serve::me", fileno $slave, $$], 676 ["perl", "-MAnyEvent::Fork::Serve", "-e", "AnyEvent::Fork::Serve::me", fileno $slave, $$],
654 [map "$_=$env{$_}", keys %env], 677 [map "$_=$env{$_}", keys %env],
655 ) or die "unable to spawn AnyEvent::Fork server: $!"; 678 ) or die "unable to spawn AnyEvent::Fork server: $!";
656 679
657 $self->_new ($fh, $pid) 680 $self->_new ($fh, $pid)
658} 681}
659 682
660=item $pid = $proc->pid 683=item $pid = $proc->pid
661 684
662Returns the process id of the process I<iff it is a direct child of the 685Returns the process id of the process I<iff it is a direct child of the
663process running AnyEvent::Fork>, and C<undef> otherwise. 686process running AnyEvent::Fork>, and C<undef> otherwise. As a general
687rule (that you cannot rely upon), processes created via C<new_exec>,
688L<AnyEvent::Fork::Early> or L<AnyEvent::Fork::Template> are direct
689children, while all other processes are not.
664 690
665Normally, only processes created via C<< AnyEvent::Fork->new_exec >> and 691Or in other words, you do not normally have to take care of zombies for
666L<AnyEvent::Fork::Template> are direct children, and you are responsible 692processes created via C<new>, but when in doubt, or zombies are a problem,
667to clean up their zombies when they die. 693you need to check whether a process is a diretc child by calling this
668 694method, and possibly creating a child watcher or reap it manually.
669All other processes are not direct children, and will be cleaned up by
670AnyEvent::Fork itself.
671 695
672=cut 696=cut
673 697
674sub pid { 698sub pid {
675 $_[0][PID] 699 $_[0][PID]
… …
806 830
807Even if not used otherwise, the socket can be a good indicator for the 831Even if not used otherwise, the socket can be a good indicator for the
808existence of the process - if the other process exits, you get a readable 832existence of the process - if the other process exits, you get a readable
809event on it, because exiting the process closes the socket (if it didn't 833event on it, because exiting the process closes the socket (if it didn't
810create any children using fork). 834create any children using fork).
835
836=over 4
837
838=item Compatibility to L<AnyEvent::Fork::Remote>
839
840If you want to write code that works with both this module and
841L<AnyEvent::Fork::Remote>, you need to write your code so that it assumes
842there are two file handles for communications, which might not be unix
843domain sockets. The C<run> function should start like this:
844
845 sub run {
846 my ($rfh, @args) = @_; # @args is your normal arguments
847 my $wfh = fileno $rfh ? $rfh : *STDOUT;
848
849 # now use $rfh for reading and $wfh for writing
850 }
851
852This checks whether the passed file handle is, in fact, the process
853C<STDIN> handle. If it is, then the function was invoked visa
854L<AnyEvent::Fork::Remote>, so STDIN should be used for reading and
855C<STDOUT> should be used for writing.
856
857In all other cases, the function was called via this module, and there is
858only one file handle that should be sued for reading and writing.
859
860=back
811 861
812Example: create a template for a process pool, pass a few strings, some 862Example: create a template for a process pool, pass a few strings, some
813file handles, then fork, pass one more string, and run some code. 863file handles, then fork, pass one more string, and run some code.
814 864
815 my $pool = AnyEvent::Fork 865 my $pool = AnyEvent::Fork
… …
863the communications socket. 913the communications socket.
864 914
865The process object becomes unusable on return from this function - any 915The process object becomes unusable on return from this function - any
866further method calls result in undefined behaviour. 916further method calls result in undefined behaviour.
867 917
868The point of this method is to give you a file handle thta you cna pass 918The point of this method is to give you a file handle that you can pass
869to another process. In that other process, you can call C<new_from_fh 919to another process. In that other process, you can call C<new_from_fh
870AnyEvent::Fork> to create a new C<AnyEvent::Fork> object from it, thereby 920AnyEvent::Fork $fh> to create a new C<AnyEvent::Fork> object from it,
871effectively passing a fork object to another process. 921thereby effectively passing a fork object to another process.
872 922
873=cut 923=cut
874 924
875sub to_fh { 925sub to_fh {
876 my ($self, $cb) = @_; 926 my ($self, $cb) = @_;
… …
1043Cygwin perl is not supported at the moment due to some hilarious 1093Cygwin perl is not supported at the moment due to some hilarious
1044shortcomings of its API - see L<IO::FDPoll> for more details. If you never 1094shortcomings of its API - see L<IO::FDPoll> for more details. If you never
1045use C<send_fh> and always use C<new_exec> to create processes, it should 1095use C<send_fh> and always use C<new_exec> to create processes, it should
1046work though. 1096work though.
1047 1097
1098=head1 USING AnyEvent::Fork IN SUBPROCESSES
1099
1100AnyEvent::Fork itself cannot generally be used in subprocesses. As long as
1101only one process ever forks new processes, sharing the template processes
1102is possible (you could use a pipe as a lock by writing a byte into it to
1103unlock, and reading the byte to lock for example)
1104
1105To make concurrent calls possible after fork, you should get rid of the
1106template and early fork processes. AnyEvent::Fork will create a new
1107template process as needed.
1108
1109 undef $AnyEvent::Fork::EARLY;
1110 undef $AnyEvent::Fork::TEMPLATE;
1111
1112It doesn't matter whether you get rid of them in the parent or child after
1113a fork.
1114
1048=head1 SEE ALSO 1115=head1 SEE ALSO
1049 1116
1050L<AnyEvent::Fork::Early>, to avoid executing a perl interpreter at all 1117L<AnyEvent::Fork::Early>, to avoid executing a perl interpreter at all
1051(part of this distribution). 1118(part of this distribution).
1052 1119
1053L<AnyEvent::Fork::Template>, to create a process by forking the main 1120L<AnyEvent::Fork::Template>, to create a process by forking the main
1054program at a convenient time (part of this distribution). 1121program at a convenient time (part of this distribution).
1055 1122
1123L<AnyEvent::Fork::Remote>, for another way to create processes that is
1124mostly compatible to this module and modules building on top of it, but
1125works better with remote processes.
1126
1056L<AnyEvent::Fork::RPC>, for simple RPC to child processes (on CPAN). 1127L<AnyEvent::Fork::RPC>, for simple RPC to child processes (on CPAN).
1057 1128
1058L<AnyEvent::Fork::Pool>, for simple worker process pool (on CPAN). 1129L<AnyEvent::Fork::Pool>, for simple worker process pool (on CPAN).
1059 1130
1060=head1 AUTHOR AND CONTACT INFORMATION 1131=head1 AUTHOR AND CONTACT INFORMATION

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines