ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-Fork/README
(Generate patch)

Comparing AnyEvent-Fork/README (file contents):
Revision 1.9 by root, Wed Sep 25 11:05:30 2013 UTC vs.
Revision 1.10 by root, Thu May 12 16:54:01 2016 UTC

51 AnyEvent::Subprocess. There are modules that implement their own process 51 AnyEvent::Subprocess. There are modules that implement their own process
52 management, such as AnyEvent::DBI. 52 management, such as AnyEvent::DBI.
53 53
54 The problems that all these modules try to solve are real, however, none 54 The problems that all these modules try to solve are real, however, none
55 of them (from what I have seen) tackle the very real problems of 55 of them (from what I have seen) tackle the very real problems of
56 unwanted memory sharing, efficiency, not being able to use event 56 unwanted memory sharing, efficiency or not being able to use event
57 processing or similar modules in the processes they create. 57 processing, GUI toolkits or similar modules in the processes they
58 create.
58 59
59 This module doesn't try to replace any of them - instead it tries to 60 This module doesn't try to replace any of them - instead it tries to
60 solve the problem of creating processes with a minimum of fuss and 61 solve the problem of creating processes with a minimum of fuss and
61 overhead (and also luxury). Ideally, most of these would use 62 overhead (and also luxury). Ideally, most of these would use
62 AnyEvent::Fork internally, except they were written before AnyEvent:Fork 63 AnyEvent::Fork internally, except they were written before AnyEvent:Fork
79 vfork where possible. This gives the speed of vfork, with the 80 vfork where possible. This gives the speed of vfork, with the
80 flexibility of fork. 81 flexibility of fork.
81 82
82 Forking usually creates a copy-on-write copy of the parent process. 83 Forking usually creates a copy-on-write copy of the parent process.
83 For example, modules or data files that are loaded will not use 84 For example, modules or data files that are loaded will not use
84 additional memory after a fork. When exec'ing a new process, modules 85 additional memory after a fork. Exec'ing a new process, in contrast,
85 and data files might need to be loaded again, at extra CPU and 86 means modules and data files might need to be loaded again, at extra
86 memory cost. But when forking, literally all data structures are 87 CPU and memory cost.
88
89 But when forking, you still create a copy of your data structures -
87 copied - if the program frees them and replaces them by new data, 90 if the program frees them and replaces them by new data, the child
88 the child processes will retain the old version even if it isn't 91 processes will retain the old version even if it isn't used, which
89 used, which can suddenly and unexpectedly increase memory usage when 92 can suddenly and unexpectedly increase memory usage when freeing
90 freeing memory. 93 memory.
91 94
95 For example, Gtk2::CV is an image viewer optimised for large
96 directories (millions of pictures). It also forks subprocesses for
97 thumbnail generation, which inherit the data structure that stores
98 all file information. If the user changes the directory, it gets
99 freed in the main process, leaving a copy in the thumbnailer
100 processes. This can lead to many times the memory usage that would
101 actually be required. The solution is to fork early (and being
102 unable to dynamically generate more subprocesses or do this from a
103 module)... or to use <AnyEvent:Fork>.
104
92 The trade-off is between more sharing with fork (which can be good 105 There is a trade-off between more sharing with fork (which can be
93 or bad), and no sharing with exec. 106 good or bad), and no sharing with exec.
94 107
95 This module allows the main program to do a controlled fork, and 108 This module allows the main program to do a controlled fork, and
96 allows modules to exec processes safely at any time. When creating a 109 allows modules to exec processes safely at any time. When creating a
97 custom process pool you can take advantage of data sharing via fork 110 custom process pool you can take advantage of data sharing via fork
98 without risking to share large dynamic data structures that will 111 without risking to share large dynamic data structures that will
101 In other words, this module puts you into control over what is being 114 In other words, this module puts you into control over what is being
102 shared and what isn't, at all times. 115 shared and what isn't, at all times.
103 116
104 Exec'ing a new perl process might be difficult. 117 Exec'ing a new perl process might be difficult.
105 For example, it is not easy to find the correct path to the perl 118 For example, it is not easy to find the correct path to the perl
106 interpreter - $^X might not be a perl interpreter at all. 119 interpreter - $^X might not be a perl interpreter at all. Worse,
120 there might not even be a perl binary installed on the system.
107 121
108 This module tries hard to identify the correct path to the perl 122 This module tries hard to identify the correct path to the perl
109 interpreter. With a cooperative main program, exec'ing the 123 interpreter. With a cooperative main program, exec'ing the
110 interpreter might not even be necessary, but even without help from 124 interpreter might not even be necessary, but even without help from
111 the main program, it will still work when used from a module. 125 the main program, it will still work when used from a module.
116 and modules are no longer loadable because they refer to a different 130 and modules are no longer loadable because they refer to a different
117 perl version, or parts of a distribution are newer than the ones 131 perl version, or parts of a distribution are newer than the ones
118 already loaded. 132 already loaded.
119 133
120 This module supports creating pre-initialised perl processes to be 134 This module supports creating pre-initialised perl processes to be
121 used as a template for new processes. 135 used as a template for new processes at a later time, e.g. for use
136 in a process pool.
122 137
123 Forking might be impossible when a program is running. 138 Forking might be impossible when a program is running.
124 For example, POSIX makes it almost impossible to fork from a 139 For example, POSIX makes it almost impossible to fork from a
125 multi-threaded program while doing anything useful in the child - in 140 multi-threaded program while doing anything useful in the child - in
126 fact, if your perl program uses POSIX threads (even indirectly via 141 fact, if your perl program uses POSIX threads (even indirectly via
127 e.g. IO::AIO or threads), you cannot call fork on the perl level 142 e.g. IO::AIO or threads), you cannot call fork on the perl level
128 anymore without risking corruption issues on a number of operating 143 anymore without risking memory corruption or worse on a number of
129 systems. 144 operating systems.
130 145
131 This module can safely fork helper processes at any time, by calling 146 This module can safely fork helper processes at any time, by calling
132 fork+exec in C, in a POSIX-compatible way (via Proc::FastSpawn). 147 fork+exec in C, in a POSIX-compatible way (via Proc::FastSpawn).
133 148
134 Parallel processing with fork might be inconvenient or difficult to 149 Parallel processing with fork might be inconvenient or difficult to
150 is still safe to do so) - all other processes are created via 165 is still safe to do so) - all other processes are created via
151 fork+exec, which makes it possible to use modules such as event 166 fork+exec, which makes it possible to use modules such as event
152 loops or window interfaces safely. 167 loops or window interfaces safely.
153 168
154EXAMPLES 169EXAMPLES
170 This is where the wall of text ends and code speaks.
171
155 Create a single new process, tell it to run your worker function. 172 Create a single new process, tell it to run your worker function.
156 AnyEvent::Fork 173 AnyEvent::Fork
157 ->new 174 ->new
158 ->require ("MyModule") 175 ->require ("MyModule")
159 ->run ("MyModule::worker, sub { 176 ->run ("MyModule::worker, sub {
169 186
170 sub worker { 187 sub worker {
171 my ($slave_filehandle) = @_; 188 my ($slave_filehandle) = @_;
172 189
173 # now $slave_filehandle is connected to the $master_filehandle 190 # now $slave_filehandle is connected to the $master_filehandle
174 # in the original prorcess. have fun! 191 # in the original process. have fun!
175 } 192 }
176 193
177 Create a pool of server processes all accepting on the same socket. 194 Create a pool of server processes all accepting on the same socket.
178 # create listener socket 195 # create listener socket
179 my $listener = ...; 196 my $listener = ...;
238 ->run ("run", my $cv = AE::cv); 255 ->run ("run", my $cv = AE::cv);
239 256
240 my $stderr = $cv->recv; 257 my $stderr = $cv->recv;
241 258
242 For stingy users: put the worker code into a "DATA" section. 259 For stingy users: put the worker code into a "DATA" section.
243 When you want to be stingy with files, you cna put your code into the 260 When you want to be stingy with files, you can put your code into the
244 "DATA" section of your module (or program): 261 "DATA" section of your module (or program):
245 262
246 use AnyEvent::Fork; 263 use AnyEvent::Fork;
247 264
248 AnyEvent::Fork 265 AnyEvent::Fork
257 } 274 }
258 275
259 For stingy standalone programs: do not rely on external files at 276 For stingy standalone programs: do not rely on external files at
260all. 277all.
261 For single-file scripts it can be inconvenient to rely on external files 278 For single-file scripts it can be inconvenient to rely on external files
262 - even when using < "DATA" section, you still need to "exec" an external 279 - even when using a "DATA" section, you still need to "exec" an external
263 perl interpreter, which might not be available when using 280 perl interpreter, which might not be available when using
264 App::Staticperl, Urlader or PAR::Packer for example. 281 App::Staticperl, Urlader or PAR::Packer for example.
265 282
266 Two modules help here - AnyEvent::Fork::Early forks a template process 283 Two modules help here - AnyEvent::Fork::Early forks a template process
267 for all further calls to "new_exec", and AnyEvent::Fork::Template forks 284 for all further calls to "new_exec", and AnyEvent::Fork::Template forks
285 my ($fh, @args) = @_; 302 my ($fh, @args) = @_;
286 ... 303 ...
287 } 304 }
288 305
289 # now preserve everything so far as AnyEvent::Fork object 306 # now preserve everything so far as AnyEvent::Fork object
290 # in ยงTEMPLATE. 307 # in $TEMPLATE.
291 use AnyEvent::Fork::Template; 308 use AnyEvent::Fork::Template;
292 309
293 # do not put code outside of BEGIN blocks until here 310 # do not put code outside of BEGIN blocks until here
294 311
295 # now use the $TEMPLATE process in any way you like 312 # now use the $TEMPLATE process in any way you like
482 See the "use AnyEvent::Fork as a faster fork+exec" example to see it 499 See the "use AnyEvent::Fork as a faster fork+exec" example to see it
483 in action. 500 in action.
484 501
485 Returns the process object for easy chaining of method calls. 502 Returns the process object for easy chaining of method calls.
486 503
504 It's common to want to call an iniitalisation function with some
505 arguments. Make sure you actually pass @_ to that function (for
506 example by using &name syntax), and do not just specify a function
507 name:
508
509 $proc->eval ('&MyModule::init', $string1, $string2);
510
487 $proc = $proc->require ($module, ...) 511 $proc = $proc->require ($module, ...)
488 Tries to load the given module(s) into the process 512 Tries to load the given module(s) into the process
489 513
490 Returns the process object for easy chaining of method calls. 514 Returns the process object for easy chaining of method calls.
491 515
765 Cygwin perl is not supported at the moment due to some hilarious 789 Cygwin perl is not supported at the moment due to some hilarious
766 shortcomings of its API - see IO::FDPoll for more details. If you never 790 shortcomings of its API - see IO::FDPoll for more details. If you never
767 use "send_fh" and always use "new_exec" to create processes, it should 791 use "send_fh" and always use "new_exec" to create processes, it should
768 work though. 792 work though.
769 793
794USING AnyEvent::Fork IN SUBPROCESSES
795 AnyEvent::Fork itself cannot generally be used in subprocesses. As long
796 as only one process ever forks new processes, sharing the template
797 processes is possible (you could use a pipe as a lock by writing a byte
798 into it to unlock, and reading the byte to lock for example)
799
800 To make concurrent calls possible after fork, you should get rid of the
801 template and early fork processes. AnyEvent::Fork will create a new
802 template process as needed.
803
804 undef $AnyEvent::Fork::EARLY;
805 undef $AnyEvent::Fork::TEMPLATE;
806
807 It doesn't matter whether you get rid of them in the parent or child
808 after a fork.
809
770SEE ALSO 810SEE ALSO
771 AnyEvent::Fork::Early, to avoid executing a perl interpreter at all 811 AnyEvent::Fork::Early, to avoid executing a perl interpreter at all
772 (part of this distribution). 812 (part of this distribution).
773 813
774 AnyEvent::Fork::Template, to create a process by forking the main 814 AnyEvent::Fork::Template, to create a process by forking the main

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines