ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-Fork/Fork.pm
(Generate patch)

Comparing AnyEvent-Fork/Fork.pm (file contents):
Revision 1.24 by root, Sat Apr 6 08:32:23 2013 UTC vs.
Revision 1.72 by root, Tue Nov 5 02:44:27 2019 UTC

27 27
28Special care has been taken to make this module useful from other modules, 28Special care has been taken to make this module useful from other modules,
29while still supporting specialised environments such as L<App::Staticperl> 29while still supporting specialised environments such as L<App::Staticperl>
30or L<PAR::Packer>. 30or L<PAR::Packer>.
31 31
32=head1 WHAT THIS MODULE IS NOT 32=head2 WHAT THIS MODULE IS NOT
33 33
34This module only creates processes and lets you pass file handles and 34This module only creates processes and lets you pass file handles and
35strings to it, and run perl code. It does not implement any kind of RPC - 35strings to it, and run perl code. It does not implement any kind of RPC -
36there is no back channel from the process back to you, and there is no RPC 36there is no back channel from the process back to you, and there is no RPC
37or message passing going on. 37or message passing going on.
38 38
39If you need some form of RPC, you can either implement it yourself 39If you need some form of RPC, you could use the L<AnyEvent::Fork::RPC>
40in whatever way you like, use some message-passing module such 40companion module, which adds simple RPC/job queueing to a process created
41as L<AnyEvent::MP>, some pipe such as L<AnyEvent::ZeroMQ>, use 41by this module.
42L<AnyEvent::Handle> on both sides to send e.g. JSON or Storable messages, 42
43and so on. 43And if you need some automatic process pool management on top of
44L<AnyEvent::Fork::RPC>, you can look at the L<AnyEvent::Fork::Pool>
45companion module.
46
47Or you can implement it yourself in whatever way you like: use some
48message-passing module such as L<AnyEvent::MP>, some pipe such as
49L<AnyEvent::ZeroMQ>, use L<AnyEvent::Handle> on both sides to send
50e.g. JSON or Storable messages, and so on.
51
52=head2 COMPARISON TO OTHER MODULES
53
54There is an abundance of modules on CPAN that do "something fork", such as
55L<Parallel::ForkManager>, L<AnyEvent::ForkManager>, L<AnyEvent::Worker>
56or L<AnyEvent::Subprocess>. There are modules that implement their own
57process management, such as L<AnyEvent::DBI>.
58
59The problems that all these modules try to solve are real, however, none
60of them (from what I have seen) tackle the very real problems of unwanted
61memory sharing, efficiency or not being able to use event processing, GUI
62toolkits or similar modules in the processes they create.
63
64This module doesn't try to replace any of them - instead it tries to solve
65the problem of creating processes with a minimum of fuss and overhead (and
66also luxury). Ideally, most of these would use AnyEvent::Fork internally,
67except they were written before AnyEvent:Fork was available, so obviously
68had to roll their own.
69
70=head2 PROBLEM STATEMENT
71
72There are two traditional ways to implement parallel processing on UNIX
73like operating systems - fork and process, and fork+exec and process. They
74have different advantages and disadvantages that I describe below,
75together with how this module tries to mitigate the disadvantages.
76
77=over 4
78
79=item Forking from a big process can be very slow.
80
81A 5GB process needs 0.05s to fork on my 3.6GHz amd64 GNU/Linux box. This
82overhead is often shared with exec (because you have to fork first), but
83in some circumstances (e.g. when vfork is used), fork+exec can be much
84faster.
85
86This module can help here by telling a small(er) helper process to fork,
87which is faster then forking the main process, and also uses vfork where
88possible. This gives the speed of vfork, with the flexibility of fork.
89
90=item Forking usually creates a copy-on-write copy of the parent
91process.
92
93For example, modules or data files that are loaded will not use additional
94memory after a fork. Exec'ing a new process, in contrast, means modules
95and data files might need to be loaded again, at extra CPU and memory
96cost.
97
98But when forking, you still create a copy of your data structures - if
99the program frees them and replaces them by new data, the child processes
100will retain the old version even if it isn't used, which can suddenly and
101unexpectedly increase memory usage when freeing memory.
102
103For example, L<Gtk2::CV> is an image viewer optimised for large
104directories (millions of pictures). It also forks subprocesses for
105thumbnail generation, which inherit the data structure that stores all
106file information. If the user changes the directory, it gets freed in
107the main process, leaving a copy in the thumbnailer processes. This can
108lead to many times the memory usage that would actually be required. The
109solution is to fork early (and being unable to dynamically generate more
110subprocesses or do this from a module)... or to use L<AnyEvent:Fork>.
111
112There is a trade-off between more sharing with fork (which can be good or
113bad), and no sharing with exec.
114
115This module allows the main program to do a controlled fork, and allows
116modules to exec processes safely at any time. When creating a custom
117process pool you can take advantage of data sharing via fork without
118risking to share large dynamic data structures that will blow up child
119memory usage.
120
121In other words, this module puts you into control over what is being
122shared and what isn't, at all times.
123
124=item Exec'ing a new perl process might be difficult.
125
126For example, it is not easy to find the correct path to the perl
127interpreter - C<$^X> might not be a perl interpreter at all. Worse, there
128might not even be a perl binary installed on the system.
129
130This module tries hard to identify the correct path to the perl
131interpreter. With a cooperative main program, exec'ing the interpreter
132might not even be necessary, but even without help from the main program,
133it will still work when used from a module.
134
135=item Exec'ing a new perl process might be slow, as all necessary modules
136have to be loaded from disk again, with no guarantees of success.
137
138Long running processes might run into problems when perl is upgraded
139and modules are no longer loadable because they refer to a different
140perl version, or parts of a distribution are newer than the ones already
141loaded.
142
143This module supports creating pre-initialised perl processes to be used as
144a template for new processes at a later time, e.g. for use in a process
145pool.
146
147=item Forking might be impossible when a program is running.
148
149For example, POSIX makes it almost impossible to fork from a
150multi-threaded program while doing anything useful in the child - in
151fact, if your perl program uses POSIX threads (even indirectly via
152e.g. L<IO::AIO> or L<threads>), you cannot call fork on the perl level
153anymore without risking memory corruption or worse on a number of
154operating systems.
155
156This module can safely fork helper processes at any time, by calling
157fork+exec in C, in a POSIX-compatible way (via L<Proc::FastSpawn>).
158
159=item Parallel processing with fork might be inconvenient or difficult
160to implement. Modules might not work in both parent and child.
161
162For example, when a program uses an event loop and creates watchers it
163becomes very hard to use the event loop from a child program, as the
164watchers already exist but are only meaningful in the parent. Worse, a
165module might want to use such a module, not knowing whether another module
166or the main program also does, leading to problems.
167
168Apart from event loops, graphical toolkits also commonly fall into the
169"unsafe module" category, or just about anything that communicates with
170the external world, such as network libraries and file I/O modules, which
171usually don't like being copied and then allowed to continue in two
172processes.
173
174With this module only the main program is allowed to create new processes
175by forking (because only the main program can know when it is still safe
176to do so) - all other processes are created via fork+exec, which makes it
177possible to use modules such as event loops or window interfaces safely.
178
179=back
44 180
45=head1 EXAMPLES 181=head1 EXAMPLES
182
183This is where the wall of text ends and code speaks.
46 184
47=head2 Create a single new process, tell it to run your worker function. 185=head2 Create a single new process, tell it to run your worker function.
48 186
49 AnyEvent::Fork 187 AnyEvent::Fork
50 ->new 188 ->new
54 192
55 # now $master_filehandle is connected to the 193 # now $master_filehandle is connected to the
56 # $slave_filehandle in the new process. 194 # $slave_filehandle in the new process.
57 }); 195 });
58 196
59 # MyModule::worker might look like this 197C<MyModule> might look like this:
198
199 package MyModule;
200
60 sub MyModule::worker { 201 sub worker {
61 my ($slave_filehandle) = @_; 202 my ($slave_filehandle) = @_;
62 203
63 # now $slave_filehandle is connected to the $master_filehandle 204 # now $slave_filehandle is connected to the $master_filehandle
64 # in the original prorcess. have fun! 205 # in the original process. have fun!
65 } 206 }
66 207
67=head2 Create a pool of server processes all accepting on the same socket. 208=head2 Create a pool of server processes all accepting on the same socket.
68 209
69 # create listener socket 210 # create listener socket
84 } 225 }
85 226
86 # now do other things - maybe use the filehandle provided by run 227 # now do other things - maybe use the filehandle provided by run
87 # to wait for the processes to die. or whatever. 228 # to wait for the processes to die. or whatever.
88 229
89 # My::Server::run might look like this 230C<My::Server> might look like this:
90 sub My::Server::run { 231
232 package My::Server;
233
234 sub run {
91 my ($slave, $listener, $id) = @_; 235 my ($slave, $listener, $id) = @_;
92 236
93 close $slave; # we do not use the socket, so close it to save resources 237 close $slave; # we do not use the socket, so close it to save resources
94 238
95 # we could go ballistic and use e.g. AnyEvent here, or IO::AIO, 239 # we could go ballistic and use e.g. AnyEvent here, or IO::AIO,
99 } 243 }
100 } 244 }
101 245
102=head2 use AnyEvent::Fork as a faster fork+exec 246=head2 use AnyEvent::Fork as a faster fork+exec
103 247
104This runs /bin/echo hi, with stdout redirected to /tmp/log and stderr to 248This runs C</bin/echo hi>, with standard output redirected to F</tmp/log>
105the communications socket. It is usually faster than fork+exec, but still 249and standard error redirected to the communications socket. It is usually
106let's you prepare the environment. 250faster than fork+exec, but still lets you prepare the environment.
107 251
108 open my $output, ">/tmp/log" or die "$!"; 252 open my $output, ">/tmp/log" or die "$!";
109 253
110 AnyEvent::Fork 254 AnyEvent::Fork
111 ->new 255 ->new
112 ->eval (' 256 ->eval ('
257 # compile a helper function for later use
113 sub run { 258 sub run {
114 my ($fh, $output, @cmd) = @_; 259 my ($fh, $output, @cmd) = @_;
115 260
116 # perl will clear close-on-exec on STDOUT/STDERR 261 # perl will clear close-on-exec on STDOUT/STDERR
117 open STDOUT, ">&", $output or die; 262 open STDOUT, ">&", $output or die;
124 ->send_arg ("/bin/echo", "hi") 269 ->send_arg ("/bin/echo", "hi")
125 ->run ("run", my $cv = AE::cv); 270 ->run ("run", my $cv = AE::cv);
126 271
127 my $stderr = $cv->recv; 272 my $stderr = $cv->recv;
128 273
129=head1 PROBLEM STATEMENT 274=head2 For stingy users: put the worker code into a C<DATA> section.
130 275
131There are two ways to implement parallel processing on UNIX like operating 276When you want to be stingy with files, you can put your code into the
132systems - fork and process, and fork+exec and process. They have different 277C<DATA> section of your module (or program):
133advantages and disadvantages that I describe below, together with how this
134module tries to mitigate the disadvantages.
135 278
136=over 4 279 use AnyEvent::Fork;
137 280
138=item Forking from a big process can be very slow (a 5GB process needs 281 AnyEvent::Fork
1390.05s to fork on my 3.6GHz amd64 GNU/Linux box for example). This overhead 282 ->new
140is often shared with exec (because you have to fork first), but in some 283 ->eval (do { local $/; <DATA> })
141circumstances (e.g. when vfork is used), fork+exec can be much faster. 284 ->run ("doit", sub { ... });
142 285
143This module can help here by telling a small(er) helper process to fork, 286 __DATA__
144or fork+exec instead.
145 287
146=item Forking usually creates a copy-on-write copy of the parent 288 sub doit {
147process. Memory (for example, modules or data files that have been 289 ... do something!
148will not take additional memory). When exec'ing a new process, modules 290 }
149and data files might need to be loaded again, at extra CPU and memory
150cost. Likewise when forking, all data structures are copied as well - if
151the program frees them and replaces them by new data, the child processes
152will retain the memory even if it isn't used.
153 291
154This module allows the main program to do a controlled fork, and allows 292=head2 For stingy standalone programs: do not rely on external files at
155modules to exec processes safely at any time. When creating a custom 293all.
156process pool you can take advantage of data sharing via fork without
157risking to share large dynamic data structures that will blow up child
158memory usage.
159 294
160=item Exec'ing a new perl process might be difficult and slow. For 295For single-file scripts it can be inconvenient to rely on external
161example, it is not easy to find the correct path to the perl interpreter, 296files - even when using a C<DATA> section, you still need to C<exec> an
162and all modules have to be loaded from disk again. Long running processes 297external perl interpreter, which might not be available when using
163might run into problems when perl is upgraded for example. 298L<App::Staticperl>, L<Urlader> or L<PAR::Packer> for example.
164 299
165This module supports creating pre-initialised perl processes to be used 300Two modules help here - L<AnyEvent::Fork::Early> forks a template process
166as template, and also tries hard to identify the correct path to the perl 301for all further calls to C<new_exec>, and L<AnyEvent::Fork::Template>
167interpreter. With a cooperative main program, exec'ing the interpreter 302forks the main program as a template process.
168might not even be necessary.
169 303
170=item Forking might be impossible when a program is running. For example, 304Here is how your main program should look like:
171POSIX makes it almost impossible to fork from a multi-threaded program and
172do anything useful in the child - strictly speaking, if your perl program
173uses posix threads (even indirectly via e.g. L<IO::AIO> or L<threads>),
174you cannot call fork on the perl level anymore, at all.
175 305
176This module can safely fork helper processes at any time, by calling 306 #! perl
177fork+exec in C, in a POSIX-compatible way.
178 307
179=item Parallel processing with fork might be inconvenient or difficult 308 # optional, as the very first thing.
180to implement. For example, when a program uses an event loop and creates 309 # in case modules want to create their own processes.
181watchers it becomes very hard to use the event loop from a child 310 use AnyEvent::Fork::Early;
182program, as the watchers already exist but are only meaningful in the
183parent. Worse, a module might want to use such a system, not knowing
184whether another module or the main program also does, leading to problems.
185 311
186This module only lets the main program create pools by forking (because 312 # next, load all modules you need in your template process
187only the main program can know when it is still safe to do so) - all other 313 use Example::My::Module
188pools are created by fork+exec, after which such modules can again be 314 use Example::Whatever;
189loaded.
190 315
191=back 316 # next, put your run function definition and anything else you
317 # need, but do not use code outside of BEGIN blocks.
318 sub worker_run {
319 my ($fh, @args) = @_;
320 ...
321 }
322
323 # now preserve everything so far as AnyEvent::Fork object
324 # in $TEMPLATE.
325 use AnyEvent::Fork::Template;
326
327 # do not put code outside of BEGIN blocks until here
328
329 # now use the $TEMPLATE process in any way you like
330
331 # for example: create 10 worker processes
332 my @worker;
333 my $cv = AE::cv;
334 for (1..10) {
335 $cv->begin;
336 $TEMPLATE->fork->send_arg ($_)->run ("worker_run", sub {
337 push @worker, shift;
338 $cv->end;
339 });
340 }
341 $cv->recv;
192 342
193=head1 CONCEPTS 343=head1 CONCEPTS
194 344
195This module can create new processes either by executing a new perl 345This module can create new processes either by executing a new perl
196process, or by forking from an existing "template" process. 346process, or by forking from an existing "template" process.
347
348All these processes are called "child processes" (whether they are direct
349children or not), while the process that manages them is called the
350"parent process".
197 351
198Each such process comes with its own file handle that can be used to 352Each such process comes with its own file handle that can be used to
199communicate with it (it's actually a socket - one end in the new process, 353communicate with it (it's actually a socket - one end in the new process,
200one end in the main process), and among the things you can do in it are 354one end in the main process), and among the things you can do in it are
201load modules, fork new processes, send file handles to it, and execute 355load modules, fork new processes, send file handles to it, and execute
275 my ($fork_fh) = @_; 429 my ($fork_fh) = @_;
276 }); 430 });
277 431
278=back 432=back
279 433
280=head1 FUNCTIONS 434=head1 THE C<AnyEvent::Fork> CLASS
435
436This module exports nothing, and only implements a single class -
437C<AnyEvent::Fork>.
438
439There are two class constructors that both create new processes - C<new>
440and C<new_exec>. The C<fork> method creates a new process by forking an
441existing one and could be considered a third constructor.
442
443Most of the remaining methods deal with preparing the new process, by
444loading code, evaluating code and sending data to the new process. They
445usually return the process object, so you can chain method calls.
446
447If a process object is destroyed before calling its C<run> method, then
448the process simply exits. After C<run> is called, all responsibility is
449passed to the specified function.
450
451As long as there is any outstanding work to be done, process objects
452resist being destroyed, so there is no reason to store them unless you
453need them later - configure and forget works just fine.
281 454
282=over 4 455=over 4
283 456
284=cut 457=cut
285 458
292use AnyEvent; 465use AnyEvent;
293use AnyEvent::Util (); 466use AnyEvent::Util ();
294 467
295use IO::FDPass; 468use IO::FDPass;
296 469
297our $VERSION = 0.5; 470our $VERSION = 1.31;
298
299our $PERL; # the path to the perl interpreter, deduces with various forms of magic
300
301=item my $pool = new AnyEvent::Fork key => value...
302
303Create a new process pool. The following named parameters are supported:
304
305=over 4
306
307=back
308
309=cut
310 471
311# the early fork template process 472# the early fork template process
312our $EARLY; 473our $EARLY;
313 474
314# the empty template process 475# the empty template process
315our $TEMPLATE; 476our $TEMPLATE;
477
478sub QUEUE() { 0 }
479sub FH() { 1 }
480sub WW() { 2 }
481sub PID() { 3 }
482sub CB() { 4 }
483
484sub _new {
485 my ($self, $fh, $pid) = @_;
486
487 AnyEvent::Util::fh_nonblocking $fh, 1;
488
489 $self = bless [
490 [], # write queue - strings or fd's
491 $fh,
492 undef, # AE watcher
493 $pid,
494 ], $self;
495
496 $self
497}
316 498
317sub _cmd { 499sub _cmd {
318 my $self = shift; 500 my $self = shift;
319 501
320 # ideally, we would want to use "a (w/a)*" as format string, but perl 502 # ideally, we would want to use "a (w/a)*" as format string, but perl
321 # versions from at least 5.8.9 to 5.16.3 are all buggy and can't unpack 503 # versions from at least 5.8.9 to 5.16.3 are all buggy and can't unpack
322 # it. 504 # it.
323 push @{ $self->[2] }, pack "a L/a*", $_[0], $_[1]; 505 push @{ $self->[QUEUE] }, pack "a L/a*", $_[0], $_[1];
324 506
325 $self->[3] ||= AE::io $self->[1], 1, sub { 507 $self->[WW] ||= AE::io $self->[FH], 1, sub {
326 do { 508 do {
327 # send the next "thing" in the queue - either a reference to an fh, 509 # send the next "thing" in the queue - either a reference to an fh,
328 # or a plain string. 510 # or a plain string.
329 511
330 if (ref $self->[2][0]) { 512 if (ref $self->[QUEUE][0]) {
331 # send fh 513 # send fh
332 unless (IO::FDPass::send fileno $self->[1], fileno ${ $self->[2][0] }) { 514 unless (IO::FDPass::send fileno $self->[FH], fileno ${ $self->[QUEUE][0] }) {
333 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK; 515 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK;
334 undef $self->[3]; 516 undef $self->[WW];
335 die "AnyEvent::Fork: file descriptor send failure: $!"; 517 die "AnyEvent::Fork: file descriptor send failure: $!";
336 } 518 }
337 519
338 shift @{ $self->[2] }; 520 shift @{ $self->[QUEUE] };
339 521
340 } else { 522 } else {
341 # send string 523 # send string
342 my $len = syswrite $self->[1], $self->[2][0]; 524 my $len = syswrite $self->[FH], $self->[QUEUE][0];
343 525
344 unless ($len) { 526 unless ($len) {
345 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK; 527 return if $! == Errno::EAGAIN || $! == Errno::EWOULDBLOCK;
346 undef $self->[3]; 528 undef $self->[WW];
347 die "AnyEvent::Fork: command write failure: $!"; 529 die "AnyEvent::Fork: command write failure: $!";
348 } 530 }
349 531
350 substr $self->[2][0], 0, $len, ""; 532 substr $self->[QUEUE][0], 0, $len, "";
351 shift @{ $self->[2] } unless length $self->[2][0]; 533 shift @{ $self->[QUEUE] } unless length $self->[QUEUE][0];
352 } 534 }
353 } while @{ $self->[2] }; 535 } while @{ $self->[QUEUE] };
354 536
355 # everything written 537 # everything written
356 undef $self->[3]; 538 undef $self->[WW];
357 539
358 # invoke run callback, if any 540 # invoke run callback, if any
359 $self->[4]->($self->[1]) if $self->[4]; 541 if ($self->[CB]) {
542 $self->[CB]->($self->[FH]);
543 @$self = ();
544 }
360 }; 545 };
361 546
362 () # make sure we don't leak the watcher 547 () # make sure we don't leak the watcher
363}
364
365sub _new {
366 my ($self, $fh, $pid) = @_;
367
368 AnyEvent::Util::fh_nonblocking $fh, 1;
369
370 $self = bless [
371 $pid,
372 $fh,
373 [], # write queue - strings or fd's
374 undef, # AE watcher
375 ], $self;
376
377 $self
378} 548}
379 549
380# fork template from current process, used by AnyEvent::Fork::Early/Template 550# fork template from current process, used by AnyEvent::Fork::Early/Template
381sub _new_fork { 551sub _new_fork {
382 my ($fh, $slave) = AnyEvent::Util::portable_socketpair; 552 my ($fh, $slave) = AnyEvent::Util::portable_socketpair;
386 556
387 if ($pid eq 0) { 557 if ($pid eq 0) {
388 require AnyEvent::Fork::Serve; 558 require AnyEvent::Fork::Serve;
389 $AnyEvent::Fork::Serve::OWNER = $parent; 559 $AnyEvent::Fork::Serve::OWNER = $parent;
390 close $fh; 560 close $fh;
391 $0 = "$_[1] of $parent"; 561 $0 = "$parent AnyEvent::Fork/exec";
392 $SIG{CHLD} = 'IGNORE';
393 AnyEvent::Fork::Serve::serve ($slave); 562 AnyEvent::Fork::Serve::serve ($slave);
394 exit 0; 563 exit 0;
395 } elsif (!$pid) { 564 } elsif (!$pid) {
396 die "AnyEvent::Fork::Early/Template: unable to fork template process: $!"; 565 die "AnyEvent::Fork::Early/Template: unable to fork template process: $!";
397 } 566 }
404Create a new "empty" perl interpreter process and returns its process 573Create a new "empty" perl interpreter process and returns its process
405object for further manipulation. 574object for further manipulation.
406 575
407The new process is forked from a template process that is kept around 576The new process is forked from a template process that is kept around
408for this purpose. When it doesn't exist yet, it is created by a call to 577for this purpose. When it doesn't exist yet, it is created by a call to
409C<new_exec> and kept around for future calls. 578C<new_exec> first and then stays around for future calls.
410
411When the process object is destroyed, it will release the file handle
412that connects it with the new process. When the new process has not yet
413called C<run>, then the process will exit. Otherwise, what happens depends
414entirely on the code that is executed.
415 579
416=cut 580=cut
417 581
418sub new { 582sub new {
419 my $class = shift; 583 my $class = shift;
456 620
457You should use C<new> whenever possible, except when having a template 621You should use C<new> whenever possible, except when having a template
458process around is unacceptable. 622process around is unacceptable.
459 623
460The path to the perl interpreter is divined using various methods - first 624The path to the perl interpreter is divined using various methods - first
461C<$^X> is investigated to see if the path ends with something that sounds 625C<$^X> is investigated to see if the path ends with something that looks
462as if it were the perl interpreter. Failing this, the module falls back to 626as if it were the perl interpreter. Failing this, the module falls back to
463using C<$Config::Config{perlpath}>. 627using C<$Config::Config{perlpath}>.
464 628
629The path to perl can also be overridden by setting the global variable
630C<$AnyEvent::Fork::PERL> - it's value will be used for all subsequent
631invocations.
632
465=cut 633=cut
634
635our $PERL;
466 636
467sub new_exec { 637sub new_exec {
468 my ($self) = @_; 638 my ($self) = @_;
469 639
470 return $EARLY->fork 640 return $EARLY->fork
471 if $EARLY; 641 if $EARLY;
472 642
643 unless (defined $PERL) {
473 # first find path of perl 644 # first find path of perl
474 my $perl = $; 645 my $perl = $^X;
475 646
476 # first we try $^X, but the path must be absolute (always on win32), and end in sth. 647 # first we try $^X, but the path must be absolute (always on win32), and end in sth.
477 # that looks like perl. this obviously only works for posix and win32 648 # that looks like perl. this obviously only works for posix and win32
478 unless ( 649 unless (
479 ($^O eq "MSWin32" || $perl =~ m%^/%) 650 ($^O eq "MSWin32" || $perl =~ m%^/%)
480 && $perl =~ m%[/\\]perl(?:[0-9]+(\.[0-9]+)+)?(\.exe)?$%i 651 && $perl =~ m%[/\\]perl(?:[0-9]+(\.[0-9]+)+)?(\.exe)?$%i
481 ) { 652 ) {
482 # if it doesn't look perlish enough, try Config 653 # if it doesn't look perlish enough, try Config
483 require Config; 654 require Config;
484 $perl = $Config::Config{perlpath}; 655 $perl = $Config::Config{perlpath};
485 $perl =~ s/(?:\Q$Config::Config{_exe}\E)?$/$Config::Config{_exe}/; 656 $perl =~ s/(?:\Q$Config::Config{_exe}\E)?$/$Config::Config{_exe}/;
657 }
658
659 $PERL = $perl;
486 } 660 }
487 661
488 require Proc::FastSpawn; 662 require Proc::FastSpawn;
489 663
490 my ($fh, $slave) = AnyEvent::Util::portable_socketpair; 664 my ($fh, $slave) = AnyEvent::Util::portable_socketpair;
498 #local $ENV{PERL5LIB} = join ":", grep !ref, @INC; 672 #local $ENV{PERL5LIB} = join ":", grep !ref, @INC;
499 my %env = %ENV; 673 my %env = %ENV;
500 $env{PERL5LIB} = join +($^O eq "MSWin32" ? ";" : ":"), grep !ref, @INC; 674 $env{PERL5LIB} = join +($^O eq "MSWin32" ? ";" : ":"), grep !ref, @INC;
501 675
502 my $pid = Proc::FastSpawn::spawn ( 676 my $pid = Proc::FastSpawn::spawn (
503 $perl, 677 $PERL,
504 ["perl", "-MAnyEvent::Fork::Serve", "-e", "AnyEvent::Fork::Serve::me", fileno $slave, $$], 678 [$PERL, "-MAnyEvent::Fork::Serve", "-e", "AnyEvent::Fork::Serve::me", fileno $slave, $$],
505 [map "$_=$env{$_}", keys %env], 679 [map "$_=$env{$_}", keys %env],
506 ) or die "unable to spawn AnyEvent::Fork server: $!"; 680 ) or die "unable to spawn AnyEvent::Fork server: $!";
507 681
508 $self->_new ($fh, $pid) 682 $self->_new ($fh, $pid)
509} 683}
510 684
511=item $pid = $proc->pid 685=item $pid = $proc->pid
512 686
513Returns the process id of the process I<iff it is a direct child of the 687Returns the process id of the process I<iff it is a direct child of the
514process> running AnyEvent::Fork, and C<undef> otherwise. 688process running AnyEvent::Fork>, and C<undef> otherwise. As a general
689rule (that you cannot rely upon), processes created via C<new_exec>,
690L<AnyEvent::Fork::Early> or L<AnyEvent::Fork::Template> are direct
691children, while all other processes are not.
515 692
516Normally, only processes created via C<< AnyEvent::Fork->new_exec >> and 693Or in other words, you do not normally have to take care of zombies for
517L<AnyEvent::Fork::Template> are direct children, and you are responsible 694processes created via C<new>, but when in doubt, or zombies are a problem,
518to clean up their zombies when they die. 695you need to check whether a process is a diretc child by calling this
519 696method, and possibly creating a child watcher or reap it manually.
520All other processes are not direct children, and will be cleaned up by
521AnyEvent::Fork.
522 697
523=cut 698=cut
524 699
525sub pid { 700sub pid {
526 $_[0][0] 701 $_[0][PID]
527} 702}
528 703
529=item $proc = $proc->eval ($perlcode, @args) 704=item $proc = $proc->eval ($perlcode, @args)
530 705
531Evaluates the given C<$perlcode> as ... perl code, while setting C<@_> to 706Evaluates the given C<$perlcode> as ... Perl code, while setting C<@_>
532the strings specified by C<@args>, in the "main" package. 707to the strings specified by C<@args>, in the "main" package (so you can
708access the args using C<$_[0]> and so on, but not using implicit C<shit>
709as the latter works on C<@ARGV>).
533 710
534This call is meant to do any custom initialisation that might be required 711This call is meant to do any custom initialisation that might be required
535(for example, the C<require> method uses it). It's not supposed to be used 712(for example, the C<require> method uses it). It's not supposed to be used
536to completely take over the process, use C<run> for that. 713to completely take over the process, use C<run> for that.
537 714
538The code will usually be executed after this call returns, and there is no 715The code will usually be executed after this call returns, and there is no
539way to pass anything back to the calling process. Any evaluation errors 716way to pass anything back to the calling process. Any evaluation errors
540will be reported to stderr and cause the process to exit. 717will be reported to stderr and cause the process to exit.
541 718
542If you want to execute some code to take over the process (see the 719If you want to execute some code (that isn't in a module) to take over the
543"fork+exec" example in the SYNOPSIS), you should compile a function via 720process, you should compile a function via C<eval> first, and then call
544C<eval> first, and then call it via C<run>. This also gives you access to 721it via C<run>. This also gives you access to any arguments passed via the
545any arguments passed via the C<send_xxx> methods, such as file handles. 722C<send_xxx> methods, such as file handles. See the L<use AnyEvent::Fork as
723a faster fork+exec> example to see it in action.
546 724
547Returns the process object for easy chaining of method calls. 725Returns the process object for easy chaining of method calls.
726
727It's common to want to call an iniitalisation function with some
728arguments. Make sure you actually pass C<@_> to that function (for example
729by using C<&name> syntax), and do not just specify a function name:
730
731 $proc->eval ('&MyModule::init', $string1, $string2);
548 732
549=cut 733=cut
550 734
551sub eval { 735sub eval {
552 my ($self, $code, @args) = @_; 736 my ($self, $code, @args) = @_;
576=item $proc = $proc->send_fh ($handle, ...) 760=item $proc = $proc->send_fh ($handle, ...)
577 761
578Send one or more file handles (I<not> file descriptors) to the process, 762Send one or more file handles (I<not> file descriptors) to the process,
579to prepare a call to C<run>. 763to prepare a call to C<run>.
580 764
581The process object keeps a reference to the handles until this is done, 765The process object keeps a reference to the handles until they have
582so you must not explicitly close the handles. This is most easily 766been passed over to the process, so you must not explicitly close the
583accomplished by simply not storing the file handles anywhere after passing 767handles. This is most easily accomplished by simply not storing the file
584them to this method. 768handles anywhere after passing them to this method - when AnyEvent::Fork
769is finished using them, perl will automatically close them.
585 770
586Returns the process object for easy chaining of method calls. 771Returns the process object for easy chaining of method calls.
587 772
588Example: pass a file handle to a process, and release it without 773Example: pass a file handle to a process, and release it without
589closing. It will be closed automatically when it is no longer used. 774closing. It will be closed automatically when it is no longer used.
596sub send_fh { 781sub send_fh {
597 my ($self, @fh) = @_; 782 my ($self, @fh) = @_;
598 783
599 for my $fh (@fh) { 784 for my $fh (@fh) {
600 $self->_cmd ("h"); 785 $self->_cmd ("h");
601 push @{ $self->[2] }, \$fh; 786 push @{ $self->[QUEUE] }, \$fh;
602 } 787 }
603 788
604 $self 789 $self
605} 790}
606 791
607=item $proc = $proc->send_arg ($string, ...) 792=item $proc = $proc->send_arg ($string, ...)
608 793
609Send one or more argument strings to the process, to prepare a call to 794Send one or more argument strings to the process, to prepare a call to
610C<run>. The strings can be any octet string. 795C<run>. The strings can be any octet strings.
611 796
612The protocol is optimised to pass a moderate number of relatively short 797The protocol is optimised to pass a moderate number of relatively short
613strings - while you can pass up to 4GB of data in one go, this is more 798strings - while you can pass up to 4GB of data in one go, this is more
614meant to pass some ID information or other startup info, not big chunks of 799meant to pass some ID information or other startup info, not big chunks of
615data. 800data.
631Enter the function specified by the function name in C<$func> in the 816Enter the function specified by the function name in C<$func> in the
632process. The function is called with the communication socket as first 817process. The function is called with the communication socket as first
633argument, followed by all file handles and string arguments sent earlier 818argument, followed by all file handles and string arguments sent earlier
634via C<send_fh> and C<send_arg> methods, in the order they were called. 819via C<send_fh> and C<send_arg> methods, in the order they were called.
635 820
821The process object becomes unusable on return from this function - any
822further method calls result in undefined behaviour.
823
636The function name should be fully qualified, but if it isn't, it will be 824The function name should be fully qualified, but if it isn't, it will be
637looked up in the main package. 825looked up in the C<main> package.
638 826
639If the called function returns, doesn't exist, or any error occurs, the 827If the called function returns, doesn't exist, or any error occurs, the
640process exits. 828process exits.
641 829
642Preparing the process is done in the background - when all commands have 830Preparing the process is done in the background - when all commands have
643been sent, the callback is invoked with the local communications socket 831been sent, the callback is invoked with the local communications socket
644as argument. At this point you can start using the socket in any way you 832as argument. At this point you can start using the socket in any way you
645like. 833like.
646 834
647The process object becomes unusable on return from this function - any
648further method calls result in undefined behaviour.
649
650If the communication socket isn't used, it should be closed on both sides, 835If the communication socket isn't used, it should be closed on both sides,
651to save on kernel memory. 836to save on kernel memory.
652 837
653The socket is non-blocking in the parent, and blocking in the newly 838The socket is non-blocking in the parent, and blocking in the newly
654created process. The close-on-exec flag is set in both. 839created process. The close-on-exec flag is set in both.
655 840
656Even if not used otherwise, the socket can be a good indicator for the 841Even if not used otherwise, the socket can be a good indicator for the
657existence of the process - if the other process exits, you get a readable 842existence of the process - if the other process exits, you get a readable
658event on it, because exiting the process closes the socket (if it didn't 843event on it, because exiting the process closes the socket (if it didn't
659create any children using fork). 844create any children using fork).
845
846=over 4
847
848=item Compatibility to L<AnyEvent::Fork::Remote>
849
850If you want to write code that works with both this module and
851L<AnyEvent::Fork::Remote>, you need to write your code so that it assumes
852there are two file handles for communications, which might not be unix
853domain sockets. The C<run> function should start like this:
854
855 sub run {
856 my ($rfh, @args) = @_; # @args is your normal arguments
857 my $wfh = fileno $rfh ? $rfh : *STDOUT;
858
859 # now use $rfh for reading and $wfh for writing
860 }
861
862This checks whether the passed file handle is, in fact, the process
863C<STDIN> handle. If it is, then the function was invoked visa
864L<AnyEvent::Fork::Remote>, so STDIN should be used for reading and
865C<STDOUT> should be used for writing.
866
867In all other cases, the function was called via this module, and there is
868only one file handle that should be sued for reading and writing.
869
870=back
660 871
661Example: create a template for a process pool, pass a few strings, some 872Example: create a template for a process pool, pass a few strings, some
662file handles, then fork, pass one more string, and run some code. 873file handles, then fork, pass one more string, and run some code.
663 874
664 my $pool = AnyEvent::Fork 875 my $pool = AnyEvent::Fork
692=cut 903=cut
693 904
694sub run { 905sub run {
695 my ($self, $func, $cb) = @_; 906 my ($self, $func, $cb) = @_;
696 907
697 $self->[4] = $cb; 908 $self->[CB] = $cb;
698 $self->_cmd (r => $func); 909 $self->_cmd (r => $func);
910}
911
912=back
913
914
915=head2 CHILD PROCESS INTERFACE
916
917This module has a limited API for use in child processes.
918
919=over 4
920
921=item @args = AnyEvent::Fork::Serve::run_args
922
923This function, which only exists before the C<run> method is called,
924returns the arguments that would be passed to the run function, and clears
925them.
926
927This is mainly useful to get any file handles passed via C<send_fh>, but
928works for any arguments passed via C<< send_I<xxx> >> methods.
929
930=back
931
932
933=head2 EXPERIMENTAL METHODS
934
935These methods might go away completely or change behaviour, at any time.
936
937=over 4
938
939=item $proc->to_fh ($cb->($fh)) # EXPERIMENTAL, MIGHT BE REMOVED
940
941Flushes all commands out to the process and then calls the callback with
942the communications socket.
943
944The process object becomes unusable on return from this function - any
945further method calls result in undefined behaviour.
946
947The point of this method is to give you a file handle that you can pass
948to another process. In that other process, you can call C<new_from_fh
949AnyEvent::Fork $fh> to create a new C<AnyEvent::Fork> object from it,
950thereby effectively passing a fork object to another process.
951
952=cut
953
954sub to_fh {
955 my ($self, $cb) = @_;
956
957 $self->[CB] = $cb;
958
959 unless ($self->[WW]) {
960 $self->[CB]->($self->[FH]);
961 @$self = ();
962 }
963}
964
965=item new_from_fh AnyEvent::Fork $fh # EXPERIMENTAL, MIGHT BE REMOVED
966
967Takes a file handle originally rceeived by the C<to_fh> method and creates
968a new C<AnyEvent:Fork> object. The child process itself will not change in
969any way, i.e. it will keep all the modifications done to it before calling
970C<to_fh>.
971
972The new object is very much like the original object, except that the
973C<pid> method will return C<undef> even if the process is a direct child.
974
975=cut
976
977sub new_from_fh {
978 my ($class, $fh) = @_;
979
980 $class->_new ($fh)
699} 981}
700 982
701=back 983=back
702 984
703=head1 PERFORMANCE 985=head1 PERFORMANCE
713 995
714 2079 new processes per second, using manual socketpair + fork 996 2079 new processes per second, using manual socketpair + fork
715 997
716Then I did the same thing, but instead of calling fork, I called 998Then I did the same thing, but instead of calling fork, I called
717AnyEvent::Fork->new->run ("CORE::exit") and then again waited for the 999AnyEvent::Fork->new->run ("CORE::exit") and then again waited for the
718socket form the child to close on exit. This does the same thing as manual 1000socket from the child to close on exit. This does the same thing as manual
719socket pair + fork, except that what is forked is the template process 1001socket pair + fork, except that what is forked is the template process
720(2440kB), and the socket needs to be passed to the server at the other end 1002(2440kB), and the socket needs to be passed to the server at the other end
721of the socket first. 1003of the socket first.
722 1004
723 2307 new processes per second, using AnyEvent::Fork->new 1005 2307 new processes per second, using AnyEvent::Fork->new
728 479 vfork+execs per second, using AnyEvent::Fork->new_exec 1010 479 vfork+execs per second, using AnyEvent::Fork->new_exec
729 1011
730So how can C<< AnyEvent->new >> be faster than a standard fork, even 1012So how can C<< AnyEvent->new >> be faster than a standard fork, even
731though it uses the same operations, but adds a lot of overhead? 1013though it uses the same operations, but adds a lot of overhead?
732 1014
733The difference is simply the process size: forking the 6MB process takes 1015The difference is simply the process size: forking the 5MB process takes
734so much longer than forking the 2.5MB template process that the overhead 1016so much longer than forking the 2.5MB template process that the extra
735introduced is canceled out. 1017overhead is canceled out.
736 1018
737If the benchmark process grows, the normal fork becomes even slower: 1019If the benchmark process grows, the normal fork becomes even slower:
738 1020
739 1340 new processes, manual fork in a 20MB process 1021 1340 new processes, manual fork of a 20MB process
740 731 new processes, manual fork in a 200MB process 1022 731 new processes, manual fork of a 200MB process
741 235 new processes, manual fork in a 2000MB process 1023 235 new processes, manual fork of a 2000MB process
742 1024
743What that means (to me) is that I can use this module without having a 1025What that means (to me) is that I can use this module without having a bad
744very bad conscience because of the extra overhead required to start new 1026conscience because of the extra overhead required to start new processes.
745processes.
746 1027
747=head1 TYPICAL PROBLEMS 1028=head1 TYPICAL PROBLEMS
748 1029
749This section lists typical problems that remain. I hope by recognising 1030This section lists typical problems that remain. I hope by recognising
750them, most can be avoided. 1031them, most can be avoided.
751 1032
752=over 4 1033=over 4
753 1034
754=item "leaked" file descriptors for exec'ed processes 1035=item leaked file descriptors for exec'ed processes
755 1036
756POSIX systems inherit file descriptors by default when exec'ing a new 1037POSIX systems inherit file descriptors by default when exec'ing a new
757process. While perl itself laudably sets the close-on-exec flags on new 1038process. While perl itself laudably sets the close-on-exec flags on new
758file handles, most C libraries don't care, and even if all cared, it's 1039file handles, most C libraries don't care, and even if all cared, it's
759often not possible to set the flag in a race-free manner. 1040often not possible to set the flag in a race-free manner.
779libraries or the code that leaks those file descriptors. 1060libraries or the code that leaks those file descriptors.
780 1061
781Fortunately, most of these leaked descriptors do no harm, other than 1062Fortunately, most of these leaked descriptors do no harm, other than
782sitting on some resources. 1063sitting on some resources.
783 1064
784=item "leaked" file descriptors for fork'ed processes 1065=item leaked file descriptors for fork'ed processes
785 1066
786Normally, L<AnyEvent::Fork> does start new processes by exec'ing them, 1067Normally, L<AnyEvent::Fork> does start new processes by exec'ing them,
787which closes file descriptors not marked for being inherited. 1068which closes file descriptors not marked for being inherited.
788 1069
789However, L<AnyEvent::Fork::Early> and L<AnyEvent::Fork::Template> offer 1070However, L<AnyEvent::Fork::Early> and L<AnyEvent::Fork::Template> offer
798 1079
799The solution is to either not load these modules before use'ing 1080The solution is to either not load these modules before use'ing
800L<AnyEvent::Fork::Early> or L<AnyEvent::Fork::Template>, or to delay 1081L<AnyEvent::Fork::Early> or L<AnyEvent::Fork::Template>, or to delay
801initialising them, for example, by calling C<init Gtk2> manually. 1082initialising them, for example, by calling C<init Gtk2> manually.
802 1083
803=item exit runs destructors 1084=item exiting calls object destructors
804 1085
805This only applies to users of Lc<AnyEvent::Fork:Early> and 1086This only applies to users of L<AnyEvent::Fork:Early> and
806L<AnyEvent::Fork::Template>. 1087L<AnyEvent::Fork::Template>, or when initialising code creates objects
1088that reference external resources.
807 1089
808When a process created by AnyEvent::Fork exits, it might do so by calling 1090When a process created by AnyEvent::Fork exits, it might do so by calling
809exit, or simply letting perl reach the end of the program. At which point 1091exit, or simply letting perl reach the end of the program. At which point
810Perl runs all destructors. 1092Perl runs all destructors.
811 1093
830to make it so, mostly due to the bloody broken perl that nobody seems to 1112to make it so, mostly due to the bloody broken perl that nobody seems to
831care about. The fork emulation is a bad joke - I have yet to see something 1113care about. The fork emulation is a bad joke - I have yet to see something
832useful that you can do with it without running into memory corruption 1114useful that you can do with it without running into memory corruption
833issues or other braindamage. Hrrrr. 1115issues or other braindamage. Hrrrr.
834 1116
835Cygwin perl is not supported at the moment, as it should implement fd 1117Since fork is endlessly broken on win32 perls (it doesn't even remotely
836passing, but doesn't, and rolling my own is hard, as cygwin doesn't 1118work within it's documented limits) and quite obviously it's not getting
837support enough functionality to do it. 1119improved any time soon, the best way to proceed on windows would be to
1120always use C<new_exec> and thus never rely on perl's fork "emulation".
1121
1122Cygwin perl is not supported at the moment due to some hilarious
1123shortcomings of its API - see L<IO::FDPoll> for more details. If you never
1124use C<send_fh> and always use C<new_exec> to create processes, it should
1125work though.
1126
1127=head1 USING AnyEvent::Fork IN SUBPROCESSES
1128
1129AnyEvent::Fork itself cannot generally be used in subprocesses. As long as
1130only one process ever forks new processes, sharing the template processes
1131is possible (you could use a pipe as a lock by writing a byte into it to
1132unlock, and reading the byte to lock for example)
1133
1134To make concurrent calls possible after fork, you should get rid of the
1135template and early fork processes. AnyEvent::Fork will create a new
1136template process as needed.
1137
1138 undef $AnyEvent::Fork::EARLY;
1139 undef $AnyEvent::Fork::TEMPLATE;
1140
1141It doesn't matter whether you get rid of them in the parent or child after
1142a fork.
838 1143
839=head1 SEE ALSO 1144=head1 SEE ALSO
840 1145
841L<AnyEvent::Fork::Early> (to avoid executing a perl interpreter), 1146L<AnyEvent::Fork::Early>, to avoid executing a perl interpreter at all
1147(part of this distribution).
1148
842L<AnyEvent::Fork::Template> (to create a process by forking the main 1149L<AnyEvent::Fork::Template>, to create a process by forking the main
843program at a convenient time). 1150program at a convenient time (part of this distribution).
844 1151
845=head1 AUTHOR 1152L<AnyEvent::Fork::Remote>, for another way to create processes that is
1153mostly compatible to this module and modules building on top of it, but
1154works better with remote processes.
1155
1156L<AnyEvent::Fork::RPC>, for simple RPC to child processes (on CPAN).
1157
1158L<AnyEvent::Fork::Pool>, for simple worker process pool (on CPAN).
1159
1160=head1 AUTHOR AND CONTACT INFORMATION
846 1161
847 Marc Lehmann <schmorp@schmorp.de> 1162 Marc Lehmann <schmorp@schmorp.de>
848 http://home.schmorp.de/ 1163 http://software.schmorp.de/pkg/AnyEvent-Fork
849 1164
850=cut 1165=cut
851 1166
8521 11671
853 1168

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines