ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-Fork/README
(Generate patch)

Comparing AnyEvent-Fork/README (file contents):
Revision 1.8 by root, Sun Apr 28 13:47:52 2013 UTC vs.
Revision 1.11 by root, Thu Nov 10 16:31:03 2016 UTC

… …
51 AnyEvent::Subprocess. There are modules that implement their own process 51 AnyEvent::Subprocess. There are modules that implement their own process
52 management, such as AnyEvent::DBI. 52 management, such as AnyEvent::DBI.
53 53
54 The problems that all these modules try to solve are real, however, none 54 The problems that all these modules try to solve are real, however, none
55 of them (from what I have seen) tackle the very real problems of 55 of them (from what I have seen) tackle the very real problems of
56 unwanted memory sharing, efficiency, not being able to use event 56 unwanted memory sharing, efficiency or not being able to use event
57 processing or similar modules in the processes they create. 57 processing, GUI toolkits or similar modules in the processes they
58 create.
58 59
59 This module doesn't try to replace any of them - instead it tries to 60 This module doesn't try to replace any of them - instead it tries to
60 solve the problem of creating processes with a minimum of fuss and 61 solve the problem of creating processes with a minimum of fuss and
61 overhead (and also luxury). Ideally, most of these would use 62 overhead (and also luxury). Ideally, most of these would use
62 AnyEvent::Fork internally, except they were written before AnyEvent:Fork 63 AnyEvent::Fork internally, except they were written before AnyEvent:Fork
… …
79 vfork where possible. This gives the speed of vfork, with the 80 vfork where possible. This gives the speed of vfork, with the
80 flexibility of fork. 81 flexibility of fork.
81 82
82 Forking usually creates a copy-on-write copy of the parent process. 83 Forking usually creates a copy-on-write copy of the parent process.
83 For example, modules or data files that are loaded will not use 84 For example, modules or data files that are loaded will not use
84 additional memory after a fork. When exec'ing a new process, modules 85 additional memory after a fork. Exec'ing a new process, in contrast,
85 and data files might need to be loaded again, at extra CPU and 86 means modules and data files might need to be loaded again, at extra
86 memory cost. But when forking, literally all data structures are 87 CPU and memory cost.
88
89 But when forking, you still create a copy of your data structures -
87 copied - if the program frees them and replaces them by new data, 90 if the program frees them and replaces them by new data, the child
88 the child processes will retain the old version even if it isn't 91 processes will retain the old version even if it isn't used, which
89 used, which can suddenly and unexpectedly increase memory usage when 92 can suddenly and unexpectedly increase memory usage when freeing
90 freeing memory. 93 memory.
91 94
95 For example, Gtk2::CV is an image viewer optimised for large
96 directories (millions of pictures). It also forks subprocesses for
97 thumbnail generation, which inherit the data structure that stores
98 all file information. If the user changes the directory, it gets
99 freed in the main process, leaving a copy in the thumbnailer
100 processes. This can lead to many times the memory usage that would
101 actually be required. The solution is to fork early (and being
102 unable to dynamically generate more subprocesses or do this from a
103 module)... or to use <AnyEvent:Fork>.
104
92 The trade-off is between more sharing with fork (which can be good 105 There is a trade-off between more sharing with fork (which can be
93 or bad), and no sharing with exec. 106 good or bad), and no sharing with exec.
94 107
95 This module allows the main program to do a controlled fork, and 108 This module allows the main program to do a controlled fork, and
96 allows modules to exec processes safely at any time. When creating a 109 allows modules to exec processes safely at any time. When creating a
97 custom process pool you can take advantage of data sharing via fork 110 custom process pool you can take advantage of data sharing via fork
98 without risking to share large dynamic data structures that will 111 without risking to share large dynamic data structures that will
… …
101 In other words, this module puts you into control over what is being 114 In other words, this module puts you into control over what is being
102 shared and what isn't, at all times. 115 shared and what isn't, at all times.
103 116
104 Exec'ing a new perl process might be difficult. 117 Exec'ing a new perl process might be difficult.
105 For example, it is not easy to find the correct path to the perl 118 For example, it is not easy to find the correct path to the perl
106 interpreter - $^X might not be a perl interpreter at all. 119 interpreter - $^X might not be a perl interpreter at all. Worse,
120 there might not even be a perl binary installed on the system.
107 121
108 This module tries hard to identify the correct path to the perl 122 This module tries hard to identify the correct path to the perl
109 interpreter. With a cooperative main program, exec'ing the 123 interpreter. With a cooperative main program, exec'ing the
110 interpreter might not even be necessary, but even without help from 124 interpreter might not even be necessary, but even without help from
111 the main program, it will still work when used from a module. 125 the main program, it will still work when used from a module.
… …
116 and modules are no longer loadable because they refer to a different 130 and modules are no longer loadable because they refer to a different
117 perl version, or parts of a distribution are newer than the ones 131 perl version, or parts of a distribution are newer than the ones
118 already loaded. 132 already loaded.
119 133
120 This module supports creating pre-initialised perl processes to be 134 This module supports creating pre-initialised perl processes to be
121 used as a template for new processes. 135 used as a template for new processes at a later time, e.g. for use
136 in a process pool.
122 137
123 Forking might be impossible when a program is running. 138 Forking might be impossible when a program is running.
124 For example, POSIX makes it almost impossible to fork from a 139 For example, POSIX makes it almost impossible to fork from a
125 multi-threaded program while doing anything useful in the child - in 140 multi-threaded program while doing anything useful in the child - in
126 fact, if your perl program uses POSIX threads (even indirectly via 141 fact, if your perl program uses POSIX threads (even indirectly via
127 e.g. IO::AIO or threads), you cannot call fork on the perl level 142 e.g. IO::AIO or threads), you cannot call fork on the perl level
128 anymore without risking corruption issues on a number of operating 143 anymore without risking memory corruption or worse on a number of
129 systems. 144 operating systems.
130 145
131 This module can safely fork helper processes at any time, by calling 146 This module can safely fork helper processes at any time, by calling
132 fork+exec in C, in a POSIX-compatible way (via Proc::FastSpawn). 147 fork+exec in C, in a POSIX-compatible way (via Proc::FastSpawn).
133 148
134 Parallel processing with fork might be inconvenient or difficult to 149 Parallel processing with fork might be inconvenient or difficult to
… …
150 is still safe to do so) - all other processes are created via 165 is still safe to do so) - all other processes are created via
151 fork+exec, which makes it possible to use modules such as event 166 fork+exec, which makes it possible to use modules such as event
152 loops or window interfaces safely. 167 loops or window interfaces safely.
153 168
154EXAMPLES 169EXAMPLES
170 This is where the wall of text ends and code speaks.
171
155 Create a single new process, tell it to run your worker function. 172 Create a single new process, tell it to run your worker function.
156 AnyEvent::Fork 173 AnyEvent::Fork
157 ->new 174 ->new
158 ->require ("MyModule") 175 ->require ("MyModule")
159 ->run ("MyModule::worker, sub { 176 ->run ("MyModule::worker, sub {
… …
169 186
170 sub worker { 187 sub worker {
171 my ($slave_filehandle) = @_; 188 my ($slave_filehandle) = @_;
172 189
173 # now $slave_filehandle is connected to the $master_filehandle 190 # now $slave_filehandle is connected to the $master_filehandle
174 # in the original prorcess. have fun! 191 # in the original process. have fun!
175 } 192 }
176 193
177 Create a pool of server processes all accepting on the same socket. 194 Create a pool of server processes all accepting on the same socket.
178 # create listener socket 195 # create listener socket
179 my $listener = ...; 196 my $listener = ...;
… …
238 ->run ("run", my $cv = AE::cv); 255 ->run ("run", my $cv = AE::cv);
239 256
240 my $stderr = $cv->recv; 257 my $stderr = $cv->recv;
241 258
242 For stingy users: put the worker code into a "DATA" section. 259 For stingy users: put the worker code into a "DATA" section.
243 When you want to be stingy with files, you cna put your code into the 260 When you want to be stingy with files, you can put your code into the
244 "DATA" section of your module (or program): 261 "DATA" section of your module (or program):
245 262
246 use AnyEvent::Fork; 263 use AnyEvent::Fork;
247 264
248 AnyEvent::Fork 265 AnyEvent::Fork
… …
257 } 274 }
258 275
259 For stingy standalone programs: do not rely on external files at 276 For stingy standalone programs: do not rely on external files at
260all. 277all.
261 For single-file scripts it can be inconvenient to rely on external files 278 For single-file scripts it can be inconvenient to rely on external files
262 - even when using < "DATA" section, you still need to "exec" an external 279 - even when using a "DATA" section, you still need to "exec" an external
263 perl interpreter, which might not be available when using 280 perl interpreter, which might not be available when using
264 App::Staticperl, Urlader or PAR::Packer for example. 281 App::Staticperl, Urlader or PAR::Packer for example.
265 282
266 Two modules help here - AnyEvent::Fork::Early forks a template process 283 Two modules help here - AnyEvent::Fork::Early forks a template process
267 for all further calls to "new_exec", and AnyEvent::Fork::Template forks 284 for all further calls to "new_exec", and AnyEvent::Fork::Template forks
… …
285 my ($fh, @args) = @_; 302 my ($fh, @args) = @_;
286 ... 303 ...
287 } 304 }
288 305
289 # now preserve everything so far as AnyEvent::Fork object 306 # now preserve everything so far as AnyEvent::Fork object
290 # in ยงTEMPLATE. 307 # in $TEMPLATE.
291 use AnyEvent::Fork::Template; 308 use AnyEvent::Fork::Template;
292 309
293 # do not put code outside of BEGIN blocks until here 310 # do not put code outside of BEGIN blocks until here
294 311
295 # now use the $TEMPLATE process in any way you like 312 # now use the $TEMPLATE process in any way you like
… …
439 You should use "new" whenever possible, except when having a 456 You should use "new" whenever possible, except when having a
440 template process around is unacceptable. 457 template process around is unacceptable.
441 458
442 The path to the perl interpreter is divined using various methods - 459 The path to the perl interpreter is divined using various methods -
443 first $^X is investigated to see if the path ends with something 460 first $^X is investigated to see if the path ends with something
444 that sounds as if it were the perl interpreter. Failing this, the 461 that looks as if it were the perl interpreter. Failing this, the
445 module falls back to using $Config::Config{perlpath}. 462 module falls back to using $Config::Config{perlpath}.
446 463
447 The path to perl can also be overriden by setting the global 464 The path to perl can also be overriden by setting the global
448 variable $AnyEvent::Fork::PERL - it's value will be used for all 465 variable $AnyEvent::Fork::PERL - it's value will be used for all
449 subsequent invocations. 466 subsequent invocations.
450 467
451 $pid = $proc->pid 468 $pid = $proc->pid
452 Returns the process id of the process *iff it is a direct child of 469 Returns the process id of the process *iff it is a direct child of
453 the process running AnyEvent::Fork*, and "undef" otherwise. 470 the process running AnyEvent::Fork*, and "undef" otherwise. As a
471 general rule (that you cannot rely upon), processes created via
472 "new_exec", AnyEvent::Fork::Early or AnyEvent::Fork::Template are
473 direct children, while all other processes are not.
454 474
455 Normally, only processes created via "AnyEvent::Fork->new_exec" and 475 Or in other words, you do not normally have to take care of zombies
456 AnyEvent::Fork::Template are direct children, and you are 476 for processes created via "new", but when in doubt, or zombies are a
457 responsible to clean up their zombies when they die. 477 problem, you need to check whether a process is a diretc child by
458 478 calling this method, and possibly creating a child watcher or reap
459 All other processes are not direct children, and will be cleaned up 479 it manually.
460 by AnyEvent::Fork itself.
461 480
462 $proc = $proc->eval ($perlcode, @args) 481 $proc = $proc->eval ($perlcode, @args)
463 Evaluates the given $perlcode as ... Perl code, while setting @_ to 482 Evaluates the given $perlcode as ... Perl code, while setting @_ to
464 the strings specified by @args, in the "main" package. 483 the strings specified by @args, in the "main" package.
465 484
… …
480 See the "use AnyEvent::Fork as a faster fork+exec" example to see it 499 See the "use AnyEvent::Fork as a faster fork+exec" example to see it
481 in action. 500 in action.
482 501
483 Returns the process object for easy chaining of method calls. 502 Returns the process object for easy chaining of method calls.
484 503
504 It's common to want to call an iniitalisation function with some
505 arguments. Make sure you actually pass @_ to that function (for
506 example by using &name syntax), and do not just specify a function
507 name:
508
509 $proc->eval ('&MyModule::init', $string1, $string2);
510
485 $proc = $proc->require ($module, ...) 511 $proc = $proc->require ($module, ...)
486 Tries to load the given module(s) into the process 512 Tries to load the given module(s) into the process
487 513
488 Returns the process object for easy chaining of method calls. 514 Returns the process object for easy chaining of method calls.
489 515
… …
547 Even if not used otherwise, the socket can be a good indicator for 573 Even if not used otherwise, the socket can be a good indicator for
548 the existence of the process - if the other process exits, you get a 574 the existence of the process - if the other process exits, you get a
549 readable event on it, because exiting the process closes the socket 575 readable event on it, because exiting the process closes the socket
550 (if it didn't create any children using fork). 576 (if it didn't create any children using fork).
551 577
578 Compatibility to AnyEvent::Fork::Remote
579 If you want to write code that works with both this module and
580 AnyEvent::Fork::Remote, you need to write your code so that it
581 assumes there are two file handles for communications, which
582 might not be unix domain sockets. The "run" function should
583 start like this:
584
585 sub run {
586 my ($rfh, @args) = @_; # @args is your normal arguments
587 my $wfh = fileno $rfh ? $rfh : *STDOUT;
588
589 # now use $rfh for reading and $wfh for writing
590 }
591
592 This checks whether the passed file handle is, in fact, the
593 process "STDIN" handle. If it is, then the function was invoked
594 visa AnyEvent::Fork::Remote, so STDIN should be used for reading
595 and "STDOUT" should be used for writing.
596
597 In all other cases, the function was called via this module, and
598 there is only one file handle that should be sued for reading
599 and writing.
600
552 Example: create a template for a process pool, pass a few strings, 601 Example: create a template for a process pool, pass a few strings,
553 some file handles, then fork, pass one more string, and run some 602 some file handles, then fork, pass one more string, and run some
554 code. 603 code.
555 604
556 my $pool = AnyEvent::Fork 605 my $pool = AnyEvent::Fork
… …
579 my ($fh, $str1, $str2, $fh1, $fh2, $str3) = @_; 628 my ($fh, $str1, $str2, $fh1, $fh2, $str3) = @_;
580 629
581 print scalar <$fh>; # prints "hi #1\n" and "hi #2\n" in any order 630 print scalar <$fh>; # prints "hi #1\n" and "hi #2\n" in any order
582 } 631 }
583 632
633 CHILD PROCESS INTERFACE
634 This module has a limited API for use in child processes.
635
636 @args = AnyEvent::Fork::Serve::run_args
637 This function, which only exists before the "run" method is called,
638 returns the arguments that would be passed to the run function, and
639 clears them.
640
641 This is mainly useful to get any file handles passed via "send_fh",
642 but works for any arguments passed via "send_*xxx*" methods.
643
584 EXPERIMENTAL METHODS 644 EXPERIMENTAL METHODS
585 These methods might go away completely or change behaviour, at any time. 645 These methods might go away completely or change behaviour, at any time.
586 646
587 $proc->to_fh ($cb->($fh)) # EXPERIMENTAL, MIGHT BE REMOVED 647 $proc->to_fh ($cb->($fh)) # EXPERIMENTAL, MIGHT BE REMOVED
588 Flushes all commands out to the process and then calls the callback 648 Flushes all commands out to the process and then calls the callback
589 with the communications socket. 649 with the communications socket.
590 650
591 The process object becomes unusable on return from this function - 651 The process object becomes unusable on return from this function -
592 any further method calls result in undefined behaviour. 652 any further method calls result in undefined behaviour.
593 653
594 The point of this method is to give you a file handle thta you cna 654 The point of this method is to give you a file handle that you can
595 pass to another process. In that other process, you can call 655 pass to another process. In that other process, you can call
596 "new_from_fh AnyEvent::Fork" to create a new "AnyEvent::Fork" object 656 "new_from_fh AnyEvent::Fork $fh" to create a new "AnyEvent::Fork"
597 from it, thereby effectively passing a fork object to another 657 object from it, thereby effectively passing a fork object to another
598 process. 658 process.
599 659
600 new_from_fh AnyEvent::Fork $fh # EXPERIMENTAL, MIGHT BE REMOVED 660 new_from_fh AnyEvent::Fork $fh # EXPERIMENTAL, MIGHT BE REMOVED
601 Takes a file handle originally rceeived by the "to_fh" method and 661 Takes a file handle originally rceeived by the "to_fh" method and
602 creates a new "AnyEvent:Fork" object. The child process itself will 662 creates a new "AnyEvent:Fork" object. The child process itself will
… …
740 Cygwin perl is not supported at the moment due to some hilarious 800 Cygwin perl is not supported at the moment due to some hilarious
741 shortcomings of its API - see IO::FDPoll for more details. If you never 801 shortcomings of its API - see IO::FDPoll for more details. If you never
742 use "send_fh" and always use "new_exec" to create processes, it should 802 use "send_fh" and always use "new_exec" to create processes, it should
743 work though. 803 work though.
744 804
805USING AnyEvent::Fork IN SUBPROCESSES
806 AnyEvent::Fork itself cannot generally be used in subprocesses. As long
807 as only one process ever forks new processes, sharing the template
808 processes is possible (you could use a pipe as a lock by writing a byte
809 into it to unlock, and reading the byte to lock for example)
810
811 To make concurrent calls possible after fork, you should get rid of the
812 template and early fork processes. AnyEvent::Fork will create a new
813 template process as needed.
814
815 undef $AnyEvent::Fork::EARLY;
816 undef $AnyEvent::Fork::TEMPLATE;
817
818 It doesn't matter whether you get rid of them in the parent or child
819 after a fork.
820
745SEE ALSO 821SEE ALSO
746 AnyEvent::Fork::Early, to avoid executing a perl interpreter at all 822 AnyEvent::Fork::Early, to avoid executing a perl interpreter at all
747 (part of this distribution). 823 (part of this distribution).
748 824
749 AnyEvent::Fork::Template, to create a process by forking the main 825 AnyEvent::Fork::Template, to create a process by forking the main
750 program at a convenient time (part of this distribution). 826 program at a convenient time (part of this distribution).
751 827
828 AnyEvent::Fork::Remote, for another way to create processes that is
829 mostly compatible to this module and modules building on top of it, but
830 works better with remote processes.
831
752 AnyEvent::Fork::RPC, for simple RPC to child processes (on CPAN). 832 AnyEvent::Fork::RPC, for simple RPC to child processes (on CPAN).
753 833
754 AnyEvent::Fork::Pool, for simple worker process pool (on CPAN). 834 AnyEvent::Fork::Pool, for simple worker process pool (on CPAN).
755 835
756AUTHOR AND CONTACT INFORMATION 836AUTHOR AND CONTACT INFORMATION

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines