ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-Fork/README
(Generate patch)

Comparing AnyEvent-Fork/README (file contents):
Revision 1.5 by root, Sat Apr 6 22:41:56 2013 UTC vs.
Revision 1.9 by root, Wed Sep 25 11:05:30 2013 UTC

… …
30 This module only creates processes and lets you pass file handles and 30 This module only creates processes and lets you pass file handles and
31 strings to it, and run perl code. It does not implement any kind of RPC 31 strings to it, and run perl code. It does not implement any kind of RPC
32 - there is no back channel from the process back to you, and there is no 32 - there is no back channel from the process back to you, and there is no
33 RPC or message passing going on. 33 RPC or message passing going on.
34 34
35 If you need some form of RPC, you can either implement it yourself in 35 If you need some form of RPC, you could use the AnyEvent::Fork::RPC
36 whatever way you like, use some message-passing module such as 36 companion module, which adds simple RPC/job queueing to a process
37 AnyEvent::MP, some pipe such as AnyEvent::ZeroMQ, use AnyEvent::Handle 37 created by this module.
38 on both sides to send e.g. JSON or Storable messages, and so on. 38
39 And if you need some automatic process pool management on top of
40 AnyEvent::Fork::RPC, you can look at the AnyEvent::Fork::Pool companion
41 module.
42
43 Or you can implement it yourself in whatever way you like: use some
44 message-passing module such as AnyEvent::MP, some pipe such as
45 AnyEvent::ZeroMQ, use AnyEvent::Handle on both sides to send e.g. JSON
46 or Storable messages, and so on.
39 47
40 COMPARISON TO OTHER MODULES 48 COMPARISON TO OTHER MODULES
41 There is an abundance of modules on CPAN that do "something fork", such 49 There is an abundance of modules on CPAN that do "something fork", such
42 as Parallel::ForkManager, AnyEvent::ForkManager, AnyEvent::Worker or 50 as Parallel::ForkManager, AnyEvent::ForkManager, AnyEvent::Worker or
43 AnyEvent::Subprocess. There are modules that implement their own process 51 AnyEvent::Subprocess. There are modules that implement their own process
… …
202 # do sth. with new socket 210 # do sth. with new socket
203 } 211 }
204 } 212 }
205 213
206 use AnyEvent::Fork as a faster fork+exec 214 use AnyEvent::Fork as a faster fork+exec
207 This runs "/bin/echo hi", with stdandard output redirected to /tmp/log 215 This runs "/bin/echo hi", with standard output redirected to /tmp/log
208 and standard error redirected to the communications socket. It is 216 and standard error redirected to the communications socket. It is
209 usually faster than fork+exec, but still lets you prepare the 217 usually faster than fork+exec, but still lets you prepare the
210 environment. 218 environment.
211 219
212 open my $output, ">/tmp/log" or die "$!"; 220 open my $output, ">/tmp/log" or die "$!";
… …
229 ->send_arg ("/bin/echo", "hi") 237 ->send_arg ("/bin/echo", "hi")
230 ->run ("run", my $cv = AE::cv); 238 ->run ("run", my $cv = AE::cv);
231 239
232 my $stderr = $cv->recv; 240 my $stderr = $cv->recv;
233 241
242 For stingy users: put the worker code into a "DATA" section.
243 When you want to be stingy with files, you cna put your code into the
244 "DATA" section of your module (or program):
245
246 use AnyEvent::Fork;
247
248 AnyEvent::Fork
249 ->new
250 ->eval (do { local $/; <DATA> })
251 ->run ("doit", sub { ... });
252
253 __DATA__
254
255 sub doit {
256 ... do something!
257 }
258
259 For stingy standalone programs: do not rely on external files at
260all.
261 For single-file scripts it can be inconvenient to rely on external files
262 - even when using < "DATA" section, you still need to "exec" an external
263 perl interpreter, which might not be available when using
264 App::Staticperl, Urlader or PAR::Packer for example.
265
266 Two modules help here - AnyEvent::Fork::Early forks a template process
267 for all further calls to "new_exec", and AnyEvent::Fork::Template forks
268 the main program as a template process.
269
270 Here is how your main program should look like:
271
272 #! perl
273
274 # optional, as the very first thing.
275 # in case modules want to create their own processes.
276 use AnyEvent::Fork::Early;
277
278 # next, load all modules you need in your template process
279 use Example::My::Module
280 use Example::Whatever;
281
282 # next, put your run function definition and anything else you
283 # need, but do not use code outside of BEGIN blocks.
284 sub worker_run {
285 my ($fh, @args) = @_;
286 ...
287 }
288
289 # now preserve everything so far as AnyEvent::Fork object
290 # in §TEMPLATE.
291 use AnyEvent::Fork::Template;
292
293 # do not put code outside of BEGIN blocks until here
294
295 # now use the $TEMPLATE process in any way you like
296
297 # for example: create 10 worker processes
298 my @worker;
299 my $cv = AE::cv;
300 for (1..10) {
301 $cv->begin;
302 $TEMPLATE->fork->send_arg ($_)->run ("worker_run", sub {
303 push @worker, shift;
304 $cv->end;
305 });
306 }
307 $cv->recv;
308
234CONCEPTS 309CONCEPTS
235 This module can create new processes either by executing a new perl 310 This module can create new processes either by executing a new perl
236 process, or by forking from an existing "template" process. 311 process, or by forking from an existing "template" process.
312
313 All these processes are called "child processes" (whether they are
314 direct children or not), while the process that manages them is called
315 the "parent process".
237 316
238 Each such process comes with its own file handle that can be used to 317 Each such process comes with its own file handle that can be used to
239 communicate with it (it's actually a socket - one end in the new 318 communicate with it (it's actually a socket - one end in the new
240 process, one end in the main process), and among the things you can do 319 process, one end in the main process), and among the things you can do
241 in it are load modules, fork new processes, send file handles to it, and 320 in it are load modules, fork new processes, send file handles to it, and
… …
329 408
330 As long as there is any outstanding work to be done, process objects 409 As long as there is any outstanding work to be done, process objects
331 resist being destroyed, so there is no reason to store them unless you 410 resist being destroyed, so there is no reason to store them unless you
332 need them later - configure and forget works just fine. 411 need them later - configure and forget works just fine.
333 412
334 my $proc = new AnyEvent::Fork 413 my $proc = new AnyEvent::Fork
335
336 Create a new "empty" perl interpreter process and returns its 414 Create a new "empty" perl interpreter process and returns its
337 process object for further manipulation. 415 process object for further manipulation.
338 416
339 The new process is forked from a template process that is kept 417 The new process is forked from a template process that is kept
340 around for this purpose. When it doesn't exist yet, it is created by 418 around for this purpose. When it doesn't exist yet, it is created by
341 a call to "new_exec" first and then stays around for future calls. 419 a call to "new_exec" first and then stays around for future calls.
342 420
343 $new_proc = $proc->fork 421 $new_proc = $proc->fork
344
345 Forks $proc, creating a new process, and returns the process object 422 Forks $proc, creating a new process, and returns the process object
346 of the new process. 423 of the new process.
347 424
348 If any of the "send_" functions have been called before fork, then 425 If any of the "send_" functions have been called before fork, then
349 they will be cloned in the child. For example, in a pre-forked 426 they will be cloned in the child. For example, in a pre-forked
350 server, you might "send_fh" the listening socket into the template 427 server, you might "send_fh" the listening socket into the template
351 process, and then keep calling "fork" and "run". 428 process, and then keep calling "fork" and "run".
352 429
353 my $proc = new_exec AnyEvent::Fork 430 my $proc = new_exec AnyEvent::Fork
354
355 Create a new "empty" perl interpreter process and returns its 431 Create a new "empty" perl interpreter process and returns its
356 process object for further manipulation. 432 process object for further manipulation.
357 433
358 Unlike the "new" method, this method *always* spawns a new perl 434 Unlike the "new" method, this method *always* spawns a new perl
359 process (except in some cases, see AnyEvent::Fork::Early for 435 process (except in some cases, see AnyEvent::Fork::Early for
… …
363 You should use "new" whenever possible, except when having a 439 You should use "new" whenever possible, except when having a
364 template process around is unacceptable. 440 template process around is unacceptable.
365 441
366 The path to the perl interpreter is divined using various methods - 442 The path to the perl interpreter is divined using various methods -
367 first $^X is investigated to see if the path ends with something 443 first $^X is investigated to see if the path ends with something
368 that sounds as if it were the perl interpreter. Failing this, the 444 that looks as if it were the perl interpreter. Failing this, the
369 module falls back to using $Config::Config{perlpath}. 445 module falls back to using $Config::Config{perlpath}.
370 446
447 The path to perl can also be overriden by setting the global
448 variable $AnyEvent::Fork::PERL - it's value will be used for all
449 subsequent invocations.
450
371 $pid = $proc->pid 451 $pid = $proc->pid
372
373 Returns the process id of the process *iff it is a direct child of 452 Returns the process id of the process *iff it is a direct child of
374 the process running AnyEvent::Fork*, and "undef" otherwise. 453 the process running AnyEvent::Fork*, and "undef" otherwise. As a
454 general rule (that you cannot rely upon), processes created via
455 "new_exec", AnyEvent::Fork::Early or AnyEvent::Fork::Template are
456 direct children, while all other processes are not.
375 457
376 Normally, only processes created via "AnyEvent::Fork->new_exec" and 458 Or in other words, you do not normally have to take care of zombies
377 AnyEvent::Fork::Template are direct children, and you are 459 for processes created via "new", but when in doubt, or zombies are a
378 responsible to clean up their zombies when they die. 460 problem, you need to check whether a process is a diretc child by
461 calling this method, and possibly creating a child watcher or reap
462 it manually.
379 463
380 All other processes are not direct children, and will be cleaned up
381 by AnyEvent::Fork itself.
382
383 $proc = $proc->eval ($perlcode, @args) 464 $proc = $proc->eval ($perlcode, @args)
384
385 Evaluates the given $perlcode as ... perl code, while setting @_ to 465 Evaluates the given $perlcode as ... Perl code, while setting @_ to
386 the strings specified by @args, in the "main" package. 466 the strings specified by @args, in the "main" package.
387 467
388 This call is meant to do any custom initialisation that might be 468 This call is meant to do any custom initialisation that might be
389 required (for example, the "require" method uses it). It's not 469 required (for example, the "require" method uses it). It's not
390 supposed to be used to completely take over the process, use "run" 470 supposed to be used to completely take over the process, use "run"
… …
402 See the "use AnyEvent::Fork as a faster fork+exec" example to see it 482 See the "use AnyEvent::Fork as a faster fork+exec" example to see it
403 in action. 483 in action.
404 484
405 Returns the process object for easy chaining of method calls. 485 Returns the process object for easy chaining of method calls.
406 486
407 $proc = $proc->require ($module, ...) 487 $proc = $proc->require ($module, ...)
408
409 Tries to load the given module(s) into the process 488 Tries to load the given module(s) into the process
410 489
411 Returns the process object for easy chaining of method calls. 490 Returns the process object for easy chaining of method calls.
412 491
413 $proc = $proc->send_fh ($handle, ...) 492 $proc = $proc->send_fh ($handle, ...)
414
415 Send one or more file handles (*not* file descriptors) to the 493 Send one or more file handles (*not* file descriptors) to the
416 process, to prepare a call to "run". 494 process, to prepare a call to "run".
417 495
418 The process object keeps a reference to the handles until they have 496 The process object keeps a reference to the handles until they have
419 been passed over to the process, so you must not explicitly close 497 been passed over to the process, so you must not explicitly close
… …
428 closing. It will be closed automatically when it is no longer used. 506 closing. It will be closed automatically when it is no longer used.
429 507
430 $proc->send_fh ($my_fh); 508 $proc->send_fh ($my_fh);
431 undef $my_fh; # free the reference if you want, but DO NOT CLOSE IT 509 undef $my_fh; # free the reference if you want, but DO NOT CLOSE IT
432 510
433 $proc = $proc->send_arg ($string, ...) 511 $proc = $proc->send_arg ($string, ...)
434
435 Send one or more argument strings to the process, to prepare a call 512 Send one or more argument strings to the process, to prepare a call
436 to "run". The strings can be any octet strings. 513 to "run". The strings can be any octet strings.
437 514
438 The protocol is optimised to pass a moderate number of relatively 515 The protocol is optimised to pass a moderate number of relatively
439 short strings - while you can pass up to 4GB of data in one go, this 516 short strings - while you can pass up to 4GB of data in one go, this
440 is more meant to pass some ID information or other startup info, not 517 is more meant to pass some ID information or other startup info, not
441 big chunks of data. 518 big chunks of data.
442 519
443 Returns the process object for easy chaining of method calls. 520 Returns the process object for easy chaining of method calls.
444 521
445 $proc->run ($func, $cb->($fh)) 522 $proc->run ($func, $cb->($fh))
446
447 Enter the function specified by the function name in $func in the 523 Enter the function specified by the function name in $func in the
448 process. The function is called with the communication socket as 524 process. The function is called with the communication socket as
449 first argument, followed by all file handles and string arguments 525 first argument, followed by all file handles and string arguments
450 sent earlier via "send_fh" and "send_arg" methods, in the order they 526 sent earlier via "send_fh" and "send_arg" methods, in the order they
451 were called. 527 were called.
… …
472 548
473 Even if not used otherwise, the socket can be a good indicator for 549 Even if not used otherwise, the socket can be a good indicator for
474 the existence of the process - if the other process exits, you get a 550 the existence of the process - if the other process exits, you get a
475 readable event on it, because exiting the process closes the socket 551 readable event on it, because exiting the process closes the socket
476 (if it didn't create any children using fork). 552 (if it didn't create any children using fork).
553
554 Compatibility to AnyEvent::Fork::Remote
555 If you want to write code that works with both this module and
556 AnyEvent::Fork::Remote, you need to write your code so that it
557 assumes there are two file handles for communications, which
558 might not be unix domain sockets. The "run" function should
559 start like this:
560
561 sub run {
562 my ($rfh, @args) = @_; # @args is your normal arguments
563 my $wfh = fileno $rfh ? $rfh : *STDOUT;
564
565 # now use $rfh for reading and $wfh for writing
566 }
567
568 This checks whether the passed file handle is, in fact, the
569 process "STDIN" handle. If it is, then the function was invoked
570 visa AnyEvent::Fork::Remote, so STDIN should be used for reading
571 and "STDOUT" should be used for writing.
572
573 In all other cases, the function was called via this module, and
574 there is only one file handle that should be sued for reading
575 and writing.
477 576
478 Example: create a template for a process pool, pass a few strings, 577 Example: create a template for a process pool, pass a few strings,
479 some file handles, then fork, pass one more string, and run some 578 some file handles, then fork, pass one more string, and run some
480 code. 579 code.
481 580
… …
505 my ($fh, $str1, $str2, $fh1, $fh2, $str3) = @_; 604 my ($fh, $str1, $str2, $fh1, $fh2, $str3) = @_;
506 605
507 print scalar <$fh>; # prints "hi #1\n" and "hi #2\n" in any order 606 print scalar <$fh>; # prints "hi #1\n" and "hi #2\n" in any order
508 } 607 }
509 608
609 EXPERIMENTAL METHODS
610 These methods might go away completely or change behaviour, at any time.
611
612 $proc->to_fh ($cb->($fh)) # EXPERIMENTAL, MIGHT BE REMOVED
613 Flushes all commands out to the process and then calls the callback
614 with the communications socket.
615
616 The process object becomes unusable on return from this function -
617 any further method calls result in undefined behaviour.
618
619 The point of this method is to give you a file handle that you can
620 pass to another process. In that other process, you can call
621 "new_from_fh AnyEvent::Fork $fh" to create a new "AnyEvent::Fork"
622 object from it, thereby effectively passing a fork object to another
623 process.
624
625 new_from_fh AnyEvent::Fork $fh # EXPERIMENTAL, MIGHT BE REMOVED
626 Takes a file handle originally rceeived by the "to_fh" method and
627 creates a new "AnyEvent:Fork" object. The child process itself will
628 not change in any way, i.e. it will keep all the modifications done
629 to it before calling "to_fh".
630
631 The new object is very much like the original object, except that
632 the "pid" method will return "undef" even if the process is a direct
633 child.
634
510PERFORMANCE 635PERFORMANCE
511 Now for some unscientific benchmark numbers (all done on an amd64 636 Now for some unscientific benchmark numbers (all done on an amd64
512 GNU/Linux box). These are intended to give you an idea of the relative 637 GNU/Linux box). These are intended to give you an idea of the relative
513 performance you can expect, they are not meant to be absolute 638 performance you can expect, they are not meant to be absolute
514 performance numbers. 639 performance numbers.
… …
520 645
521 2079 new processes per second, using manual socketpair + fork 646 2079 new processes per second, using manual socketpair + fork
522 647
523 Then I did the same thing, but instead of calling fork, I called 648 Then I did the same thing, but instead of calling fork, I called
524 AnyEvent::Fork->new->run ("CORE::exit") and then again waited for the 649 AnyEvent::Fork->new->run ("CORE::exit") and then again waited for the
525 socket form the child to close on exit. This does the same thing as 650 socket from the child to close on exit. This does the same thing as
526 manual socket pair + fork, except that what is forked is the template 651 manual socket pair + fork, except that what is forked is the template
527 process (2440kB), and the socket needs to be passed to the server at the 652 process (2440kB), and the socket needs to be passed to the server at the
528 other end of the socket first. 653 other end of the socket first.
529 654
530 2307 new processes per second, using AnyEvent::Fork->new 655 2307 new processes per second, using AnyEvent::Fork->new
… …
537 So how can "AnyEvent->new" be faster than a standard fork, even though 662 So how can "AnyEvent->new" be faster than a standard fork, even though
538 it uses the same operations, but adds a lot of overhead? 663 it uses the same operations, but adds a lot of overhead?
539 664
540 The difference is simply the process size: forking the 5MB process takes 665 The difference is simply the process size: forking the 5MB process takes
541 so much longer than forking the 2.5MB template process that the extra 666 so much longer than forking the 2.5MB template process that the extra
542 overhead introduced is canceled out. 667 overhead is canceled out.
543 668
544 If the benchmark process grows, the normal fork becomes even slower: 669 If the benchmark process grows, the normal fork becomes even slower:
545 670
546 1340 new processes, manual fork of a 20MB process 671 1340 new processes, manual fork of a 20MB process
547 731 new processes, manual fork of a 200MB process 672 731 new processes, manual fork of a 200MB process
… …
603 AnyEvent::Fork::Early or AnyEvent::Fork::Template, or to delay 728 AnyEvent::Fork::Early or AnyEvent::Fork::Template, or to delay
604 initialising them, for example, by calling "init Gtk2" manually. 729 initialising them, for example, by calling "init Gtk2" manually.
605 730
606 exiting calls object destructors 731 exiting calls object destructors
607 This only applies to users of AnyEvent::Fork:Early and 732 This only applies to users of AnyEvent::Fork:Early and
608 AnyEvent::Fork::Template, or when initialiasing code creates objects 733 AnyEvent::Fork::Template, or when initialising code creates objects
609 that reference external resources. 734 that reference external resources.
610 735
611 When a process created by AnyEvent::Fork exits, it might do so by 736 When a process created by AnyEvent::Fork exits, it might do so by
612 calling exit, or simply letting perl reach the end of the program. 737 calling exit, or simply letting perl reach the end of the program.
613 At which point Perl runs all destructors. 738 At which point Perl runs all destructors.
… …
630 and sweat to make it so, mostly due to the bloody broken perl that 755 and sweat to make it so, mostly due to the bloody broken perl that
631 nobody seems to care about. The fork emulation is a bad joke - I have 756 nobody seems to care about. The fork emulation is a bad joke - I have
632 yet to see something useful that you can do with it without running into 757 yet to see something useful that you can do with it without running into
633 memory corruption issues or other braindamage. Hrrrr. 758 memory corruption issues or other braindamage. Hrrrr.
634 759
760 Since fork is endlessly broken on win32 perls (it doesn't even remotely
761 work within it's documented limits) and quite obviously it's not getting
762 improved any time soon, the best way to proceed on windows would be to
763 always use "new_exec" and thus never rely on perl's fork "emulation".
764
635 Cygwin perl is not supported at the moment due to some hilarious 765 Cygwin perl is not supported at the moment due to some hilarious
636 shortcomings of its API - see IO::FDPoll for more details. 766 shortcomings of its API - see IO::FDPoll for more details. If you never
767 use "send_fh" and always use "new_exec" to create processes, it should
768 work though.
637 769
638SEE ALSO 770SEE ALSO
639 AnyEvent::Fork::Early (to avoid executing a perl interpreter), 771 AnyEvent::Fork::Early, to avoid executing a perl interpreter at all
772 (part of this distribution).
773
640 AnyEvent::Fork::Template (to create a process by forking the main 774 AnyEvent::Fork::Template, to create a process by forking the main
641 program at a convenient time). 775 program at a convenient time (part of this distribution).
642 776
643AUTHOR 777 AnyEvent::Fork::Remote, for another way to create processes that is
778 mostly compatible to this module and modules building on top of it, but
779 works better with remote processes.
780
781 AnyEvent::Fork::RPC, for simple RPC to child processes (on CPAN).
782
783 AnyEvent::Fork::Pool, for simple worker process pool (on CPAN).
784
785AUTHOR AND CONTACT INFORMATION
644 Marc Lehmann <schmorp@schmorp.de> 786 Marc Lehmann <schmorp@schmorp.de>
645 http://home.schmorp.de/ 787 http://software.schmorp.de/pkg/AnyEvent-Fork
646 788
647POD ERRORS
648 Hey! The above document had some coding errors, which are explained
649 below:
650
651 Around line 360:
652 You can't have =items (as at line 476) unless the first thing after
653 the =over is an =item
654

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines