ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-Fork/Fork.pm
(Generate patch)

Comparing AnyEvent-Fork/Fork.pm (file contents):
Revision 1.43 by root, Thu Apr 18 10:47:48 2013 UTC vs.
Revision 1.57 by root, Sun Aug 25 17:38:43 2013 UTC

38 38
39If you need some form of RPC, you could use the L<AnyEvent::Fork::RPC> 39If you need some form of RPC, you could use the L<AnyEvent::Fork::RPC>
40companion module, which adds simple RPC/job queueing to a process created 40companion module, which adds simple RPC/job queueing to a process created
41by this module. 41by this module.
42 42
43And if you need some automatic process pool management on top of
44L<AnyEvent::Fork::RPC>, you can look at the L<AnyEvent::Fork::Pool>
45companion module.
46
43Or you can implement it yourself in whatever way you like, use some 47Or you can implement it yourself in whatever way you like: use some
44message-passing module such as L<AnyEvent::MP>, some pipe such as 48message-passing module such as L<AnyEvent::MP>, some pipe such as
45L<AnyEvent::ZeroMQ>, use L<AnyEvent::Handle> on both sides to send 49L<AnyEvent::ZeroMQ>, use L<AnyEvent::Handle> on both sides to send
46e.g. JSON or Storable messages, and so on. 50e.g. JSON or Storable messages, and so on.
47 51
48=head2 COMPARISON TO OTHER MODULES 52=head2 COMPARISON TO OTHER MODULES
224 } 228 }
225 } 229 }
226 230
227=head2 use AnyEvent::Fork as a faster fork+exec 231=head2 use AnyEvent::Fork as a faster fork+exec
228 232
229This runs C</bin/echo hi>, with stdandard output redirected to /tmp/log 233This runs C</bin/echo hi>, with standard output redirected to F</tmp/log>
230and standard error redirected to the communications socket. It is usually 234and standard error redirected to the communications socket. It is usually
231faster than fork+exec, but still lets you prepare the environment. 235faster than fork+exec, but still lets you prepare the environment.
232 236
233 open my $output, ">/tmp/log" or die "$!"; 237 open my $output, ">/tmp/log" or die "$!";
234 238
250 ->send_arg ("/bin/echo", "hi") 254 ->send_arg ("/bin/echo", "hi")
251 ->run ("run", my $cv = AE::cv); 255 ->run ("run", my $cv = AE::cv);
252 256
253 my $stderr = $cv->recv; 257 my $stderr = $cv->recv;
254 258
259=head2 For stingy users: put the worker code into a C<DATA> section.
260
261When you want to be stingy with files, you cna put your code into the
262C<DATA> section of your module (or program):
263
264 use AnyEvent::Fork;
265
266 AnyEvent::Fork
267 ->new
268 ->eval (do { local $/; <DATA> })
269 ->run ("doit", sub { ... });
270
271 __DATA__
272
273 sub doit {
274 ... do something!
275 }
276
277=head2 For stingy standalone programs: do not rely on external files at
278all.
279
280For single-file scripts it can be inconvenient to rely on external
281files - even when using < C<DATA> section, you still need to C<exec>
282an external perl interpreter, which might not be available when using
283L<App::Staticperl>, L<Urlader> or L<PAR::Packer> for example.
284
285Two modules help here - L<AnyEvent::Fork::Early> forks a template process
286for all further calls to C<new_exec>, and L<AnyEvent::Fork::Template>
287forks the main program as a template process.
288
289Here is how your main program should look like:
290
291 #! perl
292
293 # optional, as the very first thing.
294 # in case modules want to create their own processes.
295 use AnyEvent::Fork::Early;
296
297 # next, load all modules you need in your template process
298 use Example::My::Module
299 use Example::Whatever;
300
301 # next, put your run function definition and anything else you
302 # need, but do not use code outside of BEGIN blocks.
303 sub worker_run {
304 my ($fh, @args) = @_;
305 ...
306 }
307
308 # now preserve everything so far as AnyEvent::Fork object
309 # in §TEMPLATE.
310 use AnyEvent::Fork::Template;
311
312 # do not put code outside of BEGIN blocks until here
313
314 # now use the $TEMPLATE process in any way you like
315
316 # for example: create 10 worker processes
317 my @worker;
318 my $cv = AE::cv;
319 for (1..10) {
320 $cv->begin;
321 $TEMPLATE->fork->send_arg ($_)->run ("worker_run", sub {
322 push @worker, shift;
323 $cv->end;
324 });
325 }
326 $cv->recv;
327
255=head1 CONCEPTS 328=head1 CONCEPTS
256 329
257This module can create new processes either by executing a new perl 330This module can create new processes either by executing a new perl
258process, or by forking from an existing "template" process. 331process, or by forking from an existing "template" process.
332
333All these processes are called "child processes" (whether they are direct
334children or not), while the process that manages them is called the
335"parent process".
259 336
260Each such process comes with its own file handle that can be used to 337Each such process comes with its own file handle that can be used to
261communicate with it (it's actually a socket - one end in the new process, 338communicate with it (it's actually a socket - one end in the new process,
262one end in the main process), and among the things you can do in it are 339one end in the main process), and among the things you can do in it are
263load modules, fork new processes, send file handles to it, and execute 340load modules, fork new processes, send file handles to it, and execute
373use AnyEvent; 450use AnyEvent;
374use AnyEvent::Util (); 451use AnyEvent::Util ();
375 452
376use IO::FDPass; 453use IO::FDPass;
377 454
378our $VERSION = 0.6; 455our $VERSION = 1.1;
379 456
380# the early fork template process 457# the early fork template process
381our $EARLY; 458our $EARLY;
382 459
383# the empty template process 460# the empty template process
431 # send string 508 # send string
432 my $len = syswrite $self->[FH], $self->[QUEUE][0]; 509 my $len = syswrite $self->[FH], $self->[QUEUE][0];
433 510
434 unless ($len) { 511 unless ($len) {
435 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK; 512 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK;
436 undef $self->[3]; 513 undef $self->[WW];
437 die "AnyEvent::Fork: command write failure: $!"; 514 die "AnyEvent::Fork: command write failure: $!";
438 } 515 }
439 516
440 substr $self->[QUEUE][0], 0, $len, ""; 517 substr $self->[QUEUE][0], 0, $len, "";
441 shift @{ $self->[QUEUE] } unless length $self->[QUEUE][0]; 518 shift @{ $self->[QUEUE] } unless length $self->[QUEUE][0];
444 521
445 # everything written 522 # everything written
446 undef $self->[WW]; 523 undef $self->[WW];
447 524
448 # invoke run callback, if any 525 # invoke run callback, if any
526 if ($self->[CB]) {
449 $self->[CB]->($self->[FH]) if $self->[CB]; 527 $self->[CB]->($self->[FH]);
528 @$self = ();
529 }
450 }; 530 };
451 531
452 () # make sure we don't leak the watcher 532 () # make sure we don't leak the watcher
453} 533}
454 534
525 605
526You should use C<new> whenever possible, except when having a template 606You should use C<new> whenever possible, except when having a template
527process around is unacceptable. 607process around is unacceptable.
528 608
529The path to the perl interpreter is divined using various methods - first 609The path to the perl interpreter is divined using various methods - first
530C<$^X> is investigated to see if the path ends with something that sounds 610C<$^X> is investigated to see if the path ends with something that looks
531as if it were the perl interpreter. Failing this, the module falls back to 611as if it were the perl interpreter. Failing this, the module falls back to
532using C<$Config::Config{perlpath}>. 612using C<$Config::Config{perlpath}>.
533 613
614The path to perl can also be overriden by setting the global variable
615C<$AnyEvent::Fork::PERL> - it's value will be used for all subsequent
616invocations.
617
534=cut 618=cut
619
620our $PERL;
535 621
536sub new_exec { 622sub new_exec {
537 my ($self) = @_; 623 my ($self) = @_;
538 624
539 return $EARLY->fork 625 return $EARLY->fork
540 if $EARLY; 626 if $EARLY;
541 627
628 unless (defined $PERL) {
542 # first find path of perl 629 # first find path of perl
543 my $perl = $; 630 my $perl = $;
544 631
545 # first we try $^X, but the path must be absolute (always on win32), and end in sth. 632 # first we try $^X, but the path must be absolute (always on win32), and end in sth.
546 # that looks like perl. this obviously only works for posix and win32 633 # that looks like perl. this obviously only works for posix and win32
547 unless ( 634 unless (
548 ($^O eq "MSWin32" || $perl =~ m%^/%) 635 ($^O eq "MSWin32" || $perl =~ m%^/%)
549 && $perl =~ m%[/\\]perl(?:[0-9]+(\.[0-9]+)+)?(\.exe)?$%i 636 && $perl =~ m%[/\\]perl(?:[0-9]+(\.[0-9]+)+)?(\.exe)?$%i
550 ) { 637 ) {
551 # if it doesn't look perlish enough, try Config 638 # if it doesn't look perlish enough, try Config
552 require Config; 639 require Config;
553 $perl = $Config::Config{perlpath}; 640 $perl = $Config::Config{perlpath};
554 $perl =~ s/(?:\Q$Config::Config{_exe}\E)?$/$Config::Config{_exe}/; 641 $perl =~ s/(?:\Q$Config::Config{_exe}\E)?$/$Config::Config{_exe}/;
642 }
643
644 $PERL = $perl;
555 } 645 }
556 646
557 require Proc::FastSpawn; 647 require Proc::FastSpawn;
558 648
559 my ($fh, $slave) = AnyEvent::Util::portable_socketpair; 649 my ($fh, $slave) = AnyEvent::Util::portable_socketpair;
567 #local $ENV{PERL5LIB} = join ":", grep !ref, @INC; 657 #local $ENV{PERL5LIB} = join ":", grep !ref, @INC;
568 my %env = %ENV; 658 my %env = %ENV;
569 $env{PERL5LIB} = join +($^O eq "MSWin32" ? ";" : ":"), grep !ref, @INC; 659 $env{PERL5LIB} = join +($^O eq "MSWin32" ? ";" : ":"), grep !ref, @INC;
570 660
571 my $pid = Proc::FastSpawn::spawn ( 661 my $pid = Proc::FastSpawn::spawn (
572 $perl, 662 $PERL,
573 ["perl", "-MAnyEvent::Fork::Serve", "-e", "AnyEvent::Fork::Serve::me", fileno $slave, $$], 663 ["perl", "-MAnyEvent::Fork::Serve", "-e", "AnyEvent::Fork::Serve::me", fileno $slave, $$],
574 [map "$_=$env{$_}", keys %env], 664 [map "$_=$env{$_}", keys %env],
575 ) or die "unable to spawn AnyEvent::Fork server: $!"; 665 ) or die "unable to spawn AnyEvent::Fork server: $!";
576 666
577 $self->_new ($fh, $pid) 667 $self->_new ($fh, $pid)
595 $_[0][PID] 685 $_[0][PID]
596} 686}
597 687
598=item $proc = $proc->eval ($perlcode, @args) 688=item $proc = $proc->eval ($perlcode, @args)
599 689
600Evaluates the given C<$perlcode> as ... perl code, while setting C<@_> to 690Evaluates the given C<$perlcode> as ... Perl code, while setting C<@_> to
601the strings specified by C<@args>, in the "main" package. 691the strings specified by C<@args>, in the "main" package.
602 692
603This call is meant to do any custom initialisation that might be required 693This call is meant to do any custom initialisation that might be required
604(for example, the C<require> method uses it). It's not supposed to be used 694(for example, the C<require> method uses it). It's not supposed to be used
605to completely take over the process, use C<run> for that. 695to completely take over the process, use C<run> for that.
769 $self->_cmd (r => $func); 859 $self->_cmd (r => $func);
770} 860}
771 861
772=back 862=back
773 863
864=head2 EXPERIMENTAL METHODS
865
866These methods might go away completely or change behaviour, at any time.
867
868=over 4
869
870=item $proc->to_fh ($cb->($fh)) # EXPERIMENTAL, MIGHT BE REMOVED
871
872Flushes all commands out to the process and then calls the callback with
873the communications socket.
874
875The process object becomes unusable on return from this function - any
876further method calls result in undefined behaviour.
877
878The point of this method is to give you a file handle thta you cna pass
879to another process. In that other process, you can call C<new_from_fh
880AnyEvent::Fork> to create a new C<AnyEvent::Fork> object from it, thereby
881effectively passing a fork object to another process.
882
883=cut
884
885sub to_fh {
886 my ($self, $cb) = @_;
887
888 $self->[CB] = $cb;
889
890 unless ($self->[WW]) {
891 $self->[CB]->($self->[FH]);
892 @$self = ();
893 }
894}
895
896=item new_from_fh AnyEvent::Fork $fh # EXPERIMENTAL, MIGHT BE REMOVED
897
898Takes a file handle originally rceeived by the C<to_fh> method and creates
899a new C<AnyEvent:Fork> object. The child process itself will not change in
900any way, i.e. it will keep all the modifications done to it before calling
901C<to_fh>.
902
903The new object is very much like the original object, except that the
904C<pid> method will return C<undef> even if the process is a direct child.
905
906=cut
907
908sub new_from_fh {
909 my ($class, $fh) = @_;
910
911 $class->_new ($fh)
912}
913
914=back
915
774=head1 PERFORMANCE 916=head1 PERFORMANCE
775 917
776Now for some unscientific benchmark numbers (all done on an amd64 918Now for some unscientific benchmark numbers (all done on an amd64
777GNU/Linux box). These are intended to give you an idea of the relative 919GNU/Linux box). These are intended to give you an idea of the relative
778performance you can expect, they are not meant to be absolute performance 920performance you can expect, they are not meant to be absolute performance
784 926
785 2079 new processes per second, using manual socketpair + fork 927 2079 new processes per second, using manual socketpair + fork
786 928
787Then I did the same thing, but instead of calling fork, I called 929Then I did the same thing, but instead of calling fork, I called
788AnyEvent::Fork->new->run ("CORE::exit") and then again waited for the 930AnyEvent::Fork->new->run ("CORE::exit") and then again waited for the
789socket form the child to close on exit. This does the same thing as manual 931socket from the child to close on exit. This does the same thing as manual
790socket pair + fork, except that what is forked is the template process 932socket pair + fork, except that what is forked is the template process
791(2440kB), and the socket needs to be passed to the server at the other end 933(2440kB), and the socket needs to be passed to the server at the other end
792of the socket first. 934of the socket first.
793 935
794 2307 new processes per second, using AnyEvent::Fork->new 936 2307 new processes per second, using AnyEvent::Fork->new
871initialising them, for example, by calling C<init Gtk2> manually. 1013initialising them, for example, by calling C<init Gtk2> manually.
872 1014
873=item exiting calls object destructors 1015=item exiting calls object destructors
874 1016
875This only applies to users of L<AnyEvent::Fork:Early> and 1017This only applies to users of L<AnyEvent::Fork:Early> and
876L<AnyEvent::Fork::Template>, or when initialiasing code creates objects 1018L<AnyEvent::Fork::Template>, or when initialising code creates objects
877that reference external resources. 1019that reference external resources.
878 1020
879When a process created by AnyEvent::Fork exits, it might do so by calling 1021When a process created by AnyEvent::Fork exits, it might do so by calling
880exit, or simply letting perl reach the end of the program. At which point 1022exit, or simply letting perl reach the end of the program. At which point
881Perl runs all destructors. 1023Perl runs all destructors.
901to make it so, mostly due to the bloody broken perl that nobody seems to 1043to make it so, mostly due to the bloody broken perl that nobody seems to
902care about. The fork emulation is a bad joke - I have yet to see something 1044care about. The fork emulation is a bad joke - I have yet to see something
903useful that you can do with it without running into memory corruption 1045useful that you can do with it without running into memory corruption
904issues or other braindamage. Hrrrr. 1046issues or other braindamage. Hrrrr.
905 1047
1048Since fork is endlessly broken on win32 perls (it doesn't even remotely
1049work within it's documented limits) and quite obviously it's not getting
1050improved any time soon, the best way to proceed on windows would be to
1051always use C<new_exec> and thus never rely on perl's fork "emulation".
1052
906Cygwin perl is not supported at the moment due to some hilarious 1053Cygwin perl is not supported at the moment due to some hilarious
907shortcomings of its API - see L<IO::FDPoll> for more details. 1054shortcomings of its API - see L<IO::FDPoll> for more details. If you never
1055use C<send_fh> and always use C<new_exec> to create processes, it should
1056work though.
908 1057
909=head1 SEE ALSO 1058=head1 SEE ALSO
910 1059
911L<AnyEvent::Fork::Early> (to avoid executing a perl interpreter), 1060L<AnyEvent::Fork::Early>, to avoid executing a perl interpreter at all
1061(part of this distribution).
1062
912L<AnyEvent::Fork::Template> (to create a process by forking the main 1063L<AnyEvent::Fork::Template>, to create a process by forking the main
913program at a convenient time). 1064program at a convenient time (part of this distribution).
1065
1066L<AnyEvent::Fork::Remote>, for another way to create processes that is
1067mostly compatible to this module and modules building on top of it, but
1068works better with remote processes.
1069
1070L<AnyEvent::Fork::RPC>, for simple RPC to child processes (on CPAN).
1071
1072L<AnyEvent::Fork::Pool>, for simple worker process pool (on CPAN).
914 1073
915=head1 AUTHOR AND CONTACT INFORMATION 1074=head1 AUTHOR AND CONTACT INFORMATION
916 1075
917 Marc Lehmann <schmorp@schmorp.de> 1076 Marc Lehmann <schmorp@schmorp.de>
918 http://software.schmorp.de/pkg/AnyEvent-Fork 1077 http://software.schmorp.de/pkg/AnyEvent-Fork

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines