ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-Fork/README
(Generate patch)

Comparing AnyEvent-Fork/README (file contents):
Revision 1.8 by root, Sun Apr 28 13:47:52 2013 UTC vs.
Revision 1.12 by root, Wed Jan 26 16:44:16 2022 UTC

… …
51 AnyEvent::Subprocess. There are modules that implement their own process 51 AnyEvent::Subprocess. There are modules that implement their own process
52 management, such as AnyEvent::DBI. 52 management, such as AnyEvent::DBI.
53 53
54 The problems that all these modules try to solve are real, however, none 54 The problems that all these modules try to solve are real, however, none
55 of them (from what I have seen) tackle the very real problems of 55 of them (from what I have seen) tackle the very real problems of
56 unwanted memory sharing, efficiency, not being able to use event 56 unwanted memory sharing, efficiency or not being able to use event
57 processing or similar modules in the processes they create. 57 processing, GUI toolkits or similar modules in the processes they
58 create.
58 59
59 This module doesn't try to replace any of them - instead it tries to 60 This module doesn't try to replace any of them - instead it tries to
60 solve the problem of creating processes with a minimum of fuss and 61 solve the problem of creating processes with a minimum of fuss and
61 overhead (and also luxury). Ideally, most of these would use 62 overhead (and also luxury). Ideally, most of these would use
62 AnyEvent::Fork internally, except they were written before AnyEvent:Fork 63 AnyEvent::Fork internally, except they were written before AnyEvent:Fork
… …
79 vfork where possible. This gives the speed of vfork, with the 80 vfork where possible. This gives the speed of vfork, with the
80 flexibility of fork. 81 flexibility of fork.
81 82
82 Forking usually creates a copy-on-write copy of the parent process. 83 Forking usually creates a copy-on-write copy of the parent process.
83 For example, modules or data files that are loaded will not use 84 For example, modules or data files that are loaded will not use
84 additional memory after a fork. When exec'ing a new process, modules 85 additional memory after a fork. Exec'ing a new process, in contrast,
85 and data files might need to be loaded again, at extra CPU and 86 means modules and data files might need to be loaded again, at extra
86 memory cost. But when forking, literally all data structures are 87 CPU and memory cost.
88
89 But when forking, you still create a copy of your data structures -
87 copied - if the program frees them and replaces them by new data, 90 if the program frees them and replaces them by new data, the child
88 the child processes will retain the old version even if it isn't 91 processes will retain the old version even if it isn't used, which
89 used, which can suddenly and unexpectedly increase memory usage when 92 can suddenly and unexpectedly increase memory usage when freeing
90 freeing memory. 93 memory.
91 94
95 For example, Gtk2::CV is an image viewer optimised for large
96 directories (millions of pictures). It also forks subprocesses for
97 thumbnail generation, which inherit the data structure that stores
98 all file information. If the user changes the directory, it gets
99 freed in the main process, leaving a copy in the thumbnailer
100 processes. This can lead to many times the memory usage that would
101 actually be required. The solution is to fork early (and being
102 unable to dynamically generate more subprocesses or do this from a
103 module)... or to use <AnyEvent:Fork>.
104
92 The trade-off is between more sharing with fork (which can be good 105 There is a trade-off between more sharing with fork (which can be
93 or bad), and no sharing with exec. 106 good or bad), and no sharing with exec.
94 107
95 This module allows the main program to do a controlled fork, and 108 This module allows the main program to do a controlled fork, and
96 allows modules to exec processes safely at any time. When creating a 109 allows modules to exec processes safely at any time. When creating a
97 custom process pool you can take advantage of data sharing via fork 110 custom process pool you can take advantage of data sharing via fork
98 without risking to share large dynamic data structures that will 111 without risking to share large dynamic data structures that will
… …
101 In other words, this module puts you into control over what is being 114 In other words, this module puts you into control over what is being
102 shared and what isn't, at all times. 115 shared and what isn't, at all times.
103 116
104 Exec'ing a new perl process might be difficult. 117 Exec'ing a new perl process might be difficult.
105 For example, it is not easy to find the correct path to the perl 118 For example, it is not easy to find the correct path to the perl
106 interpreter - $^X might not be a perl interpreter at all. 119 interpreter - $^X might not be a perl interpreter at all. Worse,
120 there might not even be a perl binary installed on the system.
107 121
108 This module tries hard to identify the correct path to the perl 122 This module tries hard to identify the correct path to the perl
109 interpreter. With a cooperative main program, exec'ing the 123 interpreter. With a cooperative main program, exec'ing the
110 interpreter might not even be necessary, but even without help from 124 interpreter might not even be necessary, but even without help from
111 the main program, it will still work when used from a module. 125 the main program, it will still work when used from a module.
… …
116 and modules are no longer loadable because they refer to a different 130 and modules are no longer loadable because they refer to a different
117 perl version, or parts of a distribution are newer than the ones 131 perl version, or parts of a distribution are newer than the ones
118 already loaded. 132 already loaded.
119 133
120 This module supports creating pre-initialised perl processes to be 134 This module supports creating pre-initialised perl processes to be
121 used as a template for new processes. 135 used as a template for new processes at a later time, e.g. for use
136 in a process pool.
122 137
123 Forking might be impossible when a program is running. 138 Forking might be impossible when a program is running.
124 For example, POSIX makes it almost impossible to fork from a 139 For example, POSIX makes it almost impossible to fork from a
125 multi-threaded program while doing anything useful in the child - in 140 multi-threaded program while doing anything useful in the child - in
126 fact, if your perl program uses POSIX threads (even indirectly via 141 fact, if your perl program uses POSIX threads (even indirectly via
127 e.g. IO::AIO or threads), you cannot call fork on the perl level 142 e.g. IO::AIO or threads), you cannot call fork on the perl level
128 anymore without risking corruption issues on a number of operating 143 anymore without risking memory corruption or worse on a number of
129 systems. 144 operating systems.
130 145
131 This module can safely fork helper processes at any time, by calling 146 This module can safely fork helper processes at any time, by calling
132 fork+exec in C, in a POSIX-compatible way (via Proc::FastSpawn). 147 fork+exec in C, in a POSIX-compatible way (via Proc::FastSpawn).
133 148
134 Parallel processing with fork might be inconvenient or difficult to 149 Parallel processing with fork might be inconvenient or difficult to
… …
150 is still safe to do so) - all other processes are created via 165 is still safe to do so) - all other processes are created via
151 fork+exec, which makes it possible to use modules such as event 166 fork+exec, which makes it possible to use modules such as event
152 loops or window interfaces safely. 167 loops or window interfaces safely.
153 168
154EXAMPLES 169EXAMPLES
170 This is where the wall of text ends and code speaks.
171
155 Create a single new process, tell it to run your worker function. 172 Create a single new process, tell it to run your worker function.
156 AnyEvent::Fork 173 AnyEvent::Fork
157 ->new 174 ->new
158 ->require ("MyModule") 175 ->require ("MyModule")
159 ->run ("MyModule::worker, sub { 176 ->run ("MyModule::worker, sub {
… …
169 186
170 sub worker { 187 sub worker {
171 my ($slave_filehandle) = @_; 188 my ($slave_filehandle) = @_;
172 189
173 # now $slave_filehandle is connected to the $master_filehandle 190 # now $slave_filehandle is connected to the $master_filehandle
174 # in the original prorcess. have fun! 191 # in the original process. have fun!
175 } 192 }
176 193
177 Create a pool of server processes all accepting on the same socket. 194 Create a pool of server processes all accepting on the same socket.
178 # create listener socket 195 # create listener socket
179 my $listener = ...; 196 my $listener = ...;
… …
238 ->run ("run", my $cv = AE::cv); 255 ->run ("run", my $cv = AE::cv);
239 256
240 my $stderr = $cv->recv; 257 my $stderr = $cv->recv;
241 258
242 For stingy users: put the worker code into a "DATA" section. 259 For stingy users: put the worker code into a "DATA" section.
243 When you want to be stingy with files, you cna put your code into the 260 When you want to be stingy with files, you can put your code into the
244 "DATA" section of your module (or program): 261 "DATA" section of your module (or program):
245 262
246 use AnyEvent::Fork; 263 use AnyEvent::Fork;
247 264
248 AnyEvent::Fork 265 AnyEvent::Fork
… …
257 } 274 }
258 275
259 For stingy standalone programs: do not rely on external files at 276 For stingy standalone programs: do not rely on external files at
260all. 277all.
261 For single-file scripts it can be inconvenient to rely on external files 278 For single-file scripts it can be inconvenient to rely on external files
262 - even when using < "DATA" section, you still need to "exec" an external 279 - even when using a "DATA" section, you still need to "exec" an external
263 perl interpreter, which might not be available when using 280 perl interpreter, which might not be available when using
264 App::Staticperl, Urlader or PAR::Packer for example. 281 App::Staticperl, Urlader or PAR::Packer for example.
265 282
266 Two modules help here - AnyEvent::Fork::Early forks a template process 283 Two modules help here - AnyEvent::Fork::Early forks a template process
267 for all further calls to "new_exec", and AnyEvent::Fork::Template forks 284 for all further calls to "new_exec", and AnyEvent::Fork::Template forks
… …
285 my ($fh, @args) = @_; 302 my ($fh, @args) = @_;
286 ... 303 ...
287 } 304 }
288 305
289 # now preserve everything so far as AnyEvent::Fork object 306 # now preserve everything so far as AnyEvent::Fork object
290 # in ยงTEMPLATE. 307 # in $TEMPLATE.
291 use AnyEvent::Fork::Template; 308 use AnyEvent::Fork::Template;
292 309
293 # do not put code outside of BEGIN blocks until here 310 # do not put code outside of BEGIN blocks until here
294 311
295 # now use the $TEMPLATE process in any way you like 312 # now use the $TEMPLATE process in any way you like
… …
439 You should use "new" whenever possible, except when having a 456 You should use "new" whenever possible, except when having a
440 template process around is unacceptable. 457 template process around is unacceptable.
441 458
442 The path to the perl interpreter is divined using various methods - 459 The path to the perl interpreter is divined using various methods -
443 first $^X is investigated to see if the path ends with something 460 first $^X is investigated to see if the path ends with something
444 that sounds as if it were the perl interpreter. Failing this, the 461 that looks as if it were the perl interpreter. Failing this, the
445 module falls back to using $Config::Config{perlpath}. 462 module falls back to using $Config::Config{perlpath}.
446 463
447 The path to perl can also be overriden by setting the global 464 The path to perl can also be overridden by setting the global
448 variable $AnyEvent::Fork::PERL - it's value will be used for all 465 variable $AnyEvent::Fork::PERL - it's value will be used for all
449 subsequent invocations. 466 subsequent invocations.
450 467
451 $pid = $proc->pid 468 $pid = $proc->pid
452 Returns the process id of the process *iff it is a direct child of 469 Returns the process id of the process *iff it is a direct child of
453 the process running AnyEvent::Fork*, and "undef" otherwise. 470 the process running AnyEvent::Fork*, and "undef" otherwise. As a
471 general rule (that you cannot rely upon), processes created via
472 "new_exec", AnyEvent::Fork::Early or AnyEvent::Fork::Template are
473 direct children, while all other processes are not.
454 474
455 Normally, only processes created via "AnyEvent::Fork->new_exec" and 475 Or in other words, you do not normally have to take care of zombies
456 AnyEvent::Fork::Template are direct children, and you are 476 for processes created via "new", but when in doubt, or zombies are a
457 responsible to clean up their zombies when they die. 477 problem, you need to check whether a process is a diretc child by
458 478 calling this method, and possibly creating a child watcher or reap
459 All other processes are not direct children, and will be cleaned up 479 it manually.
460 by AnyEvent::Fork itself.
461 480
462 $proc = $proc->eval ($perlcode, @args) 481 $proc = $proc->eval ($perlcode, @args)
463 Evaluates the given $perlcode as ... Perl code, while setting @_ to 482 Evaluates the given $perlcode as ... Perl code, while setting @_ to
464 the strings specified by @args, in the "main" package. 483 the strings specified by @args, in the "main" package (so you can
484 access the args using $_[0] and so on, but not using implicit "shit"
485 as the latter works on @ARGV).
465 486
466 This call is meant to do any custom initialisation that might be 487 This call is meant to do any custom initialisation that might be
467 required (for example, the "require" method uses it). It's not 488 required (for example, the "require" method uses it). It's not
468 supposed to be used to completely take over the process, use "run" 489 supposed to be used to completely take over the process, use "run"
469 for that. 490 for that.
… …
480 See the "use AnyEvent::Fork as a faster fork+exec" example to see it 501 See the "use AnyEvent::Fork as a faster fork+exec" example to see it
481 in action. 502 in action.
482 503
483 Returns the process object for easy chaining of method calls. 504 Returns the process object for easy chaining of method calls.
484 505
506 It's common to want to call an iniitalisation function with some
507 arguments. Make sure you actually pass @_ to that function (for
508 example by using &name syntax), and do not just specify a function
509 name:
510
511 $proc->eval ('&MyModule::init', $string1, $string2);
512
485 $proc = $proc->require ($module, ...) 513 $proc = $proc->require ($module, ...)
486 Tries to load the given module(s) into the process 514 Tries to load the given module(s) into the process
487 515
488 Returns the process object for easy chaining of method calls. 516 Returns the process object for easy chaining of method calls.
489 517
… …
547 Even if not used otherwise, the socket can be a good indicator for 575 Even if not used otherwise, the socket can be a good indicator for
548 the existence of the process - if the other process exits, you get a 576 the existence of the process - if the other process exits, you get a
549 readable event on it, because exiting the process closes the socket 577 readable event on it, because exiting the process closes the socket
550 (if it didn't create any children using fork). 578 (if it didn't create any children using fork).
551 579
580 Compatibility to AnyEvent::Fork::Remote
581 If you want to write code that works with both this module and
582 AnyEvent::Fork::Remote, you need to write your code so that it
583 assumes there are two file handles for communications, which
584 might not be unix domain sockets. The "run" function should
585 start like this:
586
587 sub run {
588 my ($rfh, @args) = @_; # @args is your normal arguments
589 my $wfh = fileno $rfh ? $rfh : *STDOUT;
590
591 # now use $rfh for reading and $wfh for writing
592 }
593
594 This checks whether the passed file handle is, in fact, the
595 process "STDIN" handle. If it is, then the function was invoked
596 visa AnyEvent::Fork::Remote, so STDIN should be used for reading
597 and "STDOUT" should be used for writing.
598
599 In all other cases, the function was called via this module, and
600 there is only one file handle that should be sued for reading
601 and writing.
602
552 Example: create a template for a process pool, pass a few strings, 603 Example: create a template for a process pool, pass a few strings,
553 some file handles, then fork, pass one more string, and run some 604 some file handles, then fork, pass one more string, and run some
554 code. 605 code.
555 606
556 my $pool = AnyEvent::Fork 607 my $pool = AnyEvent::Fork
… …
579 my ($fh, $str1, $str2, $fh1, $fh2, $str3) = @_; 630 my ($fh, $str1, $str2, $fh1, $fh2, $str3) = @_;
580 631
581 print scalar <$fh>; # prints "hi #1\n" and "hi #2\n" in any order 632 print scalar <$fh>; # prints "hi #1\n" and "hi #2\n" in any order
582 } 633 }
583 634
635 CHILD PROCESS INTERFACE
636 This module has a limited API for use in child processes.
637
638 @args = AnyEvent::Fork::Serve::run_args
639 This function, which only exists before the "run" method is called,
640 returns the arguments that would be passed to the run function, and
641 clears them.
642
643 This is mainly useful to get any file handles passed via "send_fh",
644 but works for any arguments passed via "send_*xxx*" methods.
645
584 EXPERIMENTAL METHODS 646 EXPERIMENTAL METHODS
585 These methods might go away completely or change behaviour, at any time. 647 These methods might go away completely or change behaviour, at any time.
586 648
587 $proc->to_fh ($cb->($fh)) # EXPERIMENTAL, MIGHT BE REMOVED 649 $proc->to_fh ($cb->($fh)) # EXPERIMENTAL, MIGHT BE REMOVED
588 Flushes all commands out to the process and then calls the callback 650 Flushes all commands out to the process and then calls the callback
589 with the communications socket. 651 with the communications socket.
590 652
591 The process object becomes unusable on return from this function - 653 The process object becomes unusable on return from this function -
592 any further method calls result in undefined behaviour. 654 any further method calls result in undefined behaviour.
593 655
594 The point of this method is to give you a file handle thta you cna 656 The point of this method is to give you a file handle that you can
595 pass to another process. In that other process, you can call 657 pass to another process. In that other process, you can call
596 "new_from_fh AnyEvent::Fork" to create a new "AnyEvent::Fork" object 658 "new_from_fh AnyEvent::Fork $fh" to create a new "AnyEvent::Fork"
597 from it, thereby effectively passing a fork object to another 659 object from it, thereby effectively passing a fork object to another
598 process. 660 process.
599 661
600 new_from_fh AnyEvent::Fork $fh # EXPERIMENTAL, MIGHT BE REMOVED 662 new_from_fh AnyEvent::Fork $fh # EXPERIMENTAL, MIGHT BE REMOVED
601 Takes a file handle originally rceeived by the "to_fh" method and 663 Takes a file handle originally rceeived by the "to_fh" method and
602 creates a new "AnyEvent:Fork" object. The child process itself will 664 creates a new "AnyEvent:Fork" object. The child process itself will
… …
740 Cygwin perl is not supported at the moment due to some hilarious 802 Cygwin perl is not supported at the moment due to some hilarious
741 shortcomings of its API - see IO::FDPoll for more details. If you never 803 shortcomings of its API - see IO::FDPoll for more details. If you never
742 use "send_fh" and always use "new_exec" to create processes, it should 804 use "send_fh" and always use "new_exec" to create processes, it should
743 work though. 805 work though.
744 806
807USING AnyEvent::Fork IN SUBPROCESSES
808 AnyEvent::Fork itself cannot generally be used in subprocesses. As long
809 as only one process ever forks new processes, sharing the template
810 processes is possible (you could use a pipe as a lock by writing a byte
811 into it to unlock, and reading the byte to lock for example)
812
813 To make concurrent calls possible after fork, you should get rid of the
814 template and early fork processes. AnyEvent::Fork will create a new
815 template process as needed.
816
817 undef $AnyEvent::Fork::EARLY;
818 undef $AnyEvent::Fork::TEMPLATE;
819
820 It doesn't matter whether you get rid of them in the parent or child
821 after a fork.
822
745SEE ALSO 823SEE ALSO
746 AnyEvent::Fork::Early, to avoid executing a perl interpreter at all 824 AnyEvent::Fork::Early, to avoid executing a perl interpreter at all
747 (part of this distribution). 825 (part of this distribution).
748 826
749 AnyEvent::Fork::Template, to create a process by forking the main 827 AnyEvent::Fork::Template, to create a process by forking the main
750 program at a convenient time (part of this distribution). 828 program at a convenient time (part of this distribution).
751 829
830 AnyEvent::Fork::Remote, for another way to create processes that is
831 mostly compatible to this module and modules building on top of it, but
832 works better with remote processes.
833
752 AnyEvent::Fork::RPC, for simple RPC to child processes (on CPAN). 834 AnyEvent::Fork::RPC, for simple RPC to child processes (on CPAN).
753 835
754 AnyEvent::Fork::Pool, for simple worker process pool (on CPAN). 836 AnyEvent::Fork::Pool, for simple worker process pool (on CPAN).
755 837
756AUTHOR AND CONTACT INFORMATION 838AUTHOR AND CONTACT INFORMATION

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines