[ViewVC] Diff of: cvs/AnyEvent-Fork/Fork.pm

Comparing AnyEvent-Fork/Fork.pm (file contents):
Revision 1.26 by root, Sat Apr 6 08:58:51 2013 UTC vs.
Revision 1.49 by root, Fri Apr 19 12:56:53 2013 UTC

…		…
27		27
28	Special care has been taken to make this module useful from other modules,	28	Special care has been taken to make this module useful from other modules,
29	while still supporting specialised environments such as L<App::Staticperl>	29	while still supporting specialised environments such as L<App::Staticperl>
30	or L<PAR::Packer>.	30	or L<PAR::Packer>.
31		31
32	=head1 WHAT THIS MODULE IS NOT	32	=head2 WHAT THIS MODULE IS NOT
33		33
34	This module only creates processes and lets you pass file handles and	34	This module only creates processes and lets you pass file handles and
35	strings to it, and run perl code. It does not implement any kind of RPC -	35	strings to it, and run perl code. It does not implement any kind of RPC -
36	there is no back channel from the process back to you, and there is no RPC	36	there is no back channel from the process back to you, and there is no RPC
37	or message passing going on.	37	or message passing going on.
38		38
39	If you need some form of RPC, you can either implement it yourself	39	If you need some form of RPC, you could use the L<AnyEvent::Fork::RPC>
40	in whatever way you like, use some message-passing module such	40	companion module, which adds simple RPC/job queueing to a process created
41	as L<AnyEvent::MP>, some pipe such as L<AnyEvent::ZeroMQ>, use	41	by this module.
42	L<AnyEvent::Handle> on both sides to send e.g. JSON or Storable messages,
43	and so on.
44		42
		43	Or you can implement it yourself in whatever way you like, use some
		44	message-passing module such as L<AnyEvent::MP>, some pipe such as
		45	L<AnyEvent::ZeroMQ>, use L<AnyEvent::Handle> on both sides to send
		46	e.g. JSON or Storable messages, and so on.
		47
		48	=head2 COMPARISON TO OTHER MODULES
		49
		50	There is an abundance of modules on CPAN that do "something fork", such as
		51	L<Parallel::ForkManager>, L<AnyEvent::ForkManager>, L<AnyEvent::Worker>
		52	or L<AnyEvent::Subprocess>. There are modules that implement their own
		53	process management, such as L<AnyEvent::DBI>.
		54
		55	The problems that all these modules try to solve are real, however, none
		56	of them (from what I have seen) tackle the very real problems of unwanted
		57	memory sharing, efficiency, not being able to use event processing or
		58	similar modules in the processes they create.
		59
		60	This module doesn't try to replace any of them - instead it tries to solve
		61	the problem of creating processes with a minimum of fuss and overhead (and
		62	also luxury). Ideally, most of these would use AnyEvent::Fork internally,
		63	except they were written before AnyEvent:Fork was available, so obviously
		64	had to roll their own.
		65
45	=head1 PROBLEM STATEMENT	66	=head2 PROBLEM STATEMENT
46		67
47	There are two traditional ways to implement parallel processing on UNIX	68	There are two traditional ways to implement parallel processing on UNIX
48	like operating systems - fork and process, and fork+exec and process. They	69	like operating systems - fork and process, and fork+exec and process. They
49	have different advantages and disadvantages that I describe below,	70	have different advantages and disadvantages that I describe below,
50	together with how this module tries to mitigate the disadvantages.	71	together with how this module tries to mitigate the disadvantages.
…		…
152		173
153	# now $master_filehandle is connected to the	174	# now $master_filehandle is connected to the
154	# $slave_filehandle in the new process.	175	# $slave_filehandle in the new process.
155	});	176	});
156		177
157	# MyModule::worker might look like this	178	C<MyModule> might look like this:
		179
		180	package MyModule;
		181
158	sub MyModule::worker {	182	sub worker {
159	my ($slave_filehandle) = @_;	183	my ($slave_filehandle) = @_;
160		184
161	# now $slave_filehandle is connected to the $master_filehandle	185	# now $slave_filehandle is connected to the $master_filehandle
162	# in the original prorcess. have fun!	186	# in the original prorcess. have fun!
163	}	187	}
…		…
182	}	206	}
183		207
184	# now do other things - maybe use the filehandle provided by run	208	# now do other things - maybe use the filehandle provided by run
185	# to wait for the processes to die. or whatever.	209	# to wait for the processes to die. or whatever.
186		210
187	# My::Server::run might look like this	211	C<My::Server> might look like this:
188	sub My::Server::run {	212
		213	package My::Server;
		214
		215	sub run {
189	my ($slave, $listener, $id) = @_;	216	my ($slave, $listener, $id) = @_;
190		217
191	close $slave; # we do not use the socket, so close it to save resources	218	close $slave; # we do not use the socket, so close it to save resources
192		219
193	# we could go ballistic and use e.g. AnyEvent here, or IO::AIO,	220	# we could go ballistic and use e.g. AnyEvent here, or IO::AIO,
…		…
197	}	224	}
198	}	225	}
199		226
200	=head2 use AnyEvent::Fork as a faster fork+exec	227	=head2 use AnyEvent::Fork as a faster fork+exec
201		228
202	This runs /bin/echo hi, with stdout redirected to /tmp/log and stderr to	229	This runs C</bin/echo hi>, with standard output redirected to F</tmp/log>
203	the communications socket. It is usually faster than fork+exec, but still	230	and standard error redirected to the communications socket. It is usually
204	let's you prepare the environment.	231	faster than fork+exec, but still lets you prepare the environment.
205		232
206	open my $output, ">/tmp/log" or die "$!";	233	open my $output, ">/tmp/log" or die "$!";
207		234
208	AnyEvent::Fork	235	AnyEvent::Fork
209	->new	236	->new
210	->eval ('	237	->eval ('
		238	# compile a helper function for later use
211	sub run {	239	sub run {
212	my ($fh, $output, @cmd) = @_;	240	my ($fh, $output, @cmd) = @_;
213		241
214	# perl will clear close-on-exec on STDOUT/STDERR	242	# perl will clear close-on-exec on STDOUT/STDERR
215	open STDOUT, ">&", $output or die;	243	open STDOUT, ">&", $output or die;
…		…
226		254
227	=head1 CONCEPTS	255	=head1 CONCEPTS
228		256
229	This module can create new processes either by executing a new perl	257	This module can create new processes either by executing a new perl
230	process, or by forking from an existing "template" process.	258	process, or by forking from an existing "template" process.
		259
		260	All these processes are called "child processes" (whether they are direct
		261	children or not), while the process that manages them is called the
		262	"parent process".
231		263
232	Each such process comes with its own file handle that can be used to	264	Each such process comes with its own file handle that can be used to
233	communicate with it (it's actually a socket - one end in the new process,	265	communicate with it (it's actually a socket - one end in the new process,
234	one end in the main process), and among the things you can do in it are	266	one end in the main process), and among the things you can do in it are
235	load modules, fork new processes, send file handles to it, and execute	267	load modules, fork new processes, send file handles to it, and execute
…		…
309	my ($fork_fh) = @_;	341	my ($fork_fh) = @_;
310	});	342	});
311		343
312	=back	344	=back
313		345
314	=head1 FUNCTIONS	346	=head1 THE C<AnyEvent::Fork> CLASS
		347
		348	This module exports nothing, and only implements a single class -
		349	C<AnyEvent::Fork>.
		350
		351	There are two class constructors that both create new processes - C<new>
		352	and C<new_exec>. The C<fork> method creates a new process by forking an
		353	existing one and could be considered a third constructor.
		354
		355	Most of the remaining methods deal with preparing the new process, by
		356	loading code, evaluating code and sending data to the new process. They
		357	usually return the process object, so you can chain method calls.
		358
		359	If a process object is destroyed before calling its C<run> method, then
		360	the process simply exits. After C<run> is called, all responsibility is
		361	passed to the specified function.
		362
		363	As long as there is any outstanding work to be done, process objects
		364	resist being destroyed, so there is no reason to store them unless you
		365	need them later - configure and forget works just fine.
315		366
316	=over 4	367	=over 4
317		368
318	=cut	369	=cut
319		370
…		…
326	use AnyEvent;	377	use AnyEvent;
327	use AnyEvent::Util ();	378	use AnyEvent::Util ();
328		379
329	use IO::FDPass;	380	use IO::FDPass;
330		381
331	our $VERSION = 0.5;	382	our $VERSION = 0.7;
332
333	our $PERL; # the path to the perl interpreter, deduces with various forms of magic
334
335	=item my $pool = new AnyEvent::Fork key => value...
336
337	Create a new process pool. The following named parameters are supported:
338
339	=over 4
340
341	=back
342
343	=cut
344		383
345	# the early fork template process	384	# the early fork template process
346	our $EARLY;	385	our $EARLY;
347		386
348	# the empty template process	387	# the empty template process
349	our $TEMPLATE;	388	our $TEMPLATE;
		389
		390	sub QUEUE() { 0 }
		391	sub FH() { 1 }
		392	sub WW() { 2 }
		393	sub PID() { 3 }
		394	sub CB() { 4 }
		395
		396	sub _new {
		397	my ($self, $fh, $pid) = @_;
		398
		399	AnyEvent::Util::fh_nonblocking $fh, 1;
		400
		401	$self = bless [
		402	[], # write queue - strings or fd's
		403	$fh,
		404	undef, # AE watcher
		405	$pid,
		406	], $self;
		407
		408	$self
		409	}
350		410
351	sub _cmd {	411	sub _cmd {
352	my $self = shift;	412	my $self = shift;
353		413
354	# ideally, we would want to use "a (w/a)*" as format string, but perl	414	# ideally, we would want to use "a (w/a)*" as format string, but perl
355	# versions from at least 5.8.9 to 5.16.3 are all buggy and can't unpack	415	# versions from at least 5.8.9 to 5.16.3 are all buggy and can't unpack
356	# it.	416	# it.
357	push @{ $self->[2] }, pack "a L/a*", $_[0], $_[1];	417	push @{ $self->[QUEUE] }, pack "a L/a*", $_[0], $_[1];
358		418
359	$self->[3] \|\|= AE::io $self->[1], 1, sub {	419	$self->[WW] \|\|= AE::io $self->[FH], 1, sub {
360	do {	420	do {
361	# send the next "thing" in the queue - either a reference to an fh,	421	# send the next "thing" in the queue - either a reference to an fh,
362	# or a plain string.	422	# or a plain string.
363		423
364	if (ref $self->[2][0]) {	424	if (ref $self->[QUEUE][0]) {
365	# send fh	425	# send fh
366	unless (IO::FDPass::send fileno $self->[1], fileno ${ $self->[2][0] }) {	426	unless (IO::FDPass::send fileno $self->[FH], fileno ${ $self->[QUEUE][0] }) {
367	return if $! == Errno::EAGAIN \|\| $! == Errno::EWOULDBLOCK;	427	return if $! == Errno::EAGAIN \|\| $! == Errno::EWOULDBLOCK;
368	undef $self->[3];	428	undef $self->[WW];
369	die "AnyEvent::Fork: file descriptor send failure: $!";	429	die "AnyEvent::Fork: file descriptor send failure: $!";
370	}	430	}
371		431
372	shift @{ $self->[2] };	432	shift @{ $self->[QUEUE] };
373		433
374	} else {	434	} else {
375	# send string	435	# send string
376	my $len = syswrite $self->[1], $self->[2][0];	436	my $len = syswrite $self->[FH], $self->[QUEUE][0];
377		437
378	unless ($len) {	438	unless ($len) {
379	return if $! == Errno::EAGAIN \|\| $! == Errno::EWOULDBLOCK;	439	return if $! == Errno::EAGAIN \|\| $! == Errno::EWOULDBLOCK;
380	undef $self->[3];	440	undef $self->[3];
381	die "AnyEvent::Fork: command write failure: $!";	441	die "AnyEvent::Fork: command write failure: $!";
382	}	442	}
383		443
384	substr $self->[2][0], 0, $len, "";	444	substr $self->[QUEUE][0], 0, $len, "";
385	shift @{ $self->[2] } unless length $self->[2][0];	445	shift @{ $self->[QUEUE] } unless length $self->[QUEUE][0];
386	}	446	}
387	} while @{ $self->[2] };	447	} while @{ $self->[QUEUE] };
388		448
389	# everything written	449	# everything written
390	undef $self->[3];	450	undef $self->[WW];
391		451
392	# invoke run callback, if any	452	# invoke run callback, if any
393	$self->[4]->($self->[1]) if $self->[4];	453	if ($self->[CB]) {
		454	$self->[CB]->($self->[FH]);
		455	@$self = ();
		456	}
394	};	457	};
395		458
396	() # make sure we don't leak the watcher	459	() # make sure we don't leak the watcher
397	}
398
399	sub _new {
400	my ($self, $fh, $pid) = @_;
401
402	AnyEvent::Util::fh_nonblocking $fh, 1;
403
404	$self = bless [
405	$pid,
406	$fh,
407	[], # write queue - strings or fd's
408	undef, # AE watcher
409	], $self;
410
411	$self
412	}	460	}
413		461
414	# fork template from current process, used by AnyEvent::Fork::Early/Template	462	# fork template from current process, used by AnyEvent::Fork::Early/Template
415	sub _new_fork {	463	sub _new_fork {
416	my ($fh, $slave) = AnyEvent::Util::portable_socketpair;	464	my ($fh, $slave) = AnyEvent::Util::portable_socketpair;
…		…
421	if ($pid eq 0) {	469	if ($pid eq 0) {
422	require AnyEvent::Fork::Serve;	470	require AnyEvent::Fork::Serve;
423	$AnyEvent::Fork::Serve::OWNER = $parent;	471	$AnyEvent::Fork::Serve::OWNER = $parent;
424	close $fh;	472	close $fh;
425	$0 = "$_[1] of $parent";	473	$0 = "$_[1] of $parent";
426	$SIG{CHLD} = 'IGNORE';
427	AnyEvent::Fork::Serve::serve ($slave);	474	AnyEvent::Fork::Serve::serve ($slave);
428	exit 0;	475	exit 0;
429	} elsif (!$pid) {	476	} elsif (!$pid) {
430	die "AnyEvent::Fork::Early/Template: unable to fork template process: $!";	477	die "AnyEvent::Fork::Early/Template: unable to fork template process: $!";
431	}	478	}
…		…
438	Create a new "empty" perl interpreter process and returns its process	485	Create a new "empty" perl interpreter process and returns its process
439	object for further manipulation.	486	object for further manipulation.
440		487
441	The new process is forked from a template process that is kept around	488	The new process is forked from a template process that is kept around
442	for this purpose. When it doesn't exist yet, it is created by a call to	489	for this purpose. When it doesn't exist yet, it is created by a call to
443	C<new_exec> and kept around for future calls.	490	C<new_exec> first and then stays around for future calls.
444
445	When the process object is destroyed, it will release the file handle
446	that connects it with the new process. When the new process has not yet
447	called C<run>, then the process will exit. Otherwise, what happens depends
448	entirely on the code that is executed.
449		491
450	=cut	492	=cut
451		493
452	sub new {	494	sub new {
453	my $class = shift;	495	my $class = shift;
…		…
543	}	585	}
544		586
545	=item $pid = $proc->pid	587	=item $pid = $proc->pid
546		588
547	Returns the process id of the process I<iff it is a direct child of the	589	Returns the process id of the process I<iff it is a direct child of the
548	process> running AnyEvent::Fork, and C<undef> otherwise.	590	process running AnyEvent::Fork>, and C<undef> otherwise.
549		591
550	Normally, only processes created via C<< AnyEvent::Fork->new_exec >> and	592	Normally, only processes created via C<< AnyEvent::Fork->new_exec >> and
551	L<AnyEvent::Fork::Template> are direct children, and you are responsible	593	L<AnyEvent::Fork::Template> are direct children, and you are responsible
552	to clean up their zombies when they die.	594	to clean up their zombies when they die.
553		595
554	All other processes are not direct children, and will be cleaned up by	596	All other processes are not direct children, and will be cleaned up by
555	AnyEvent::Fork.	597	AnyEvent::Fork itself.
556		598
557	=cut	599	=cut
558		600
559	sub pid {	601	sub pid {
560	$_[0][0]	602	$_[0][PID]
561	}	603	}
562		604
563	=item $proc = $proc->eval ($perlcode, @args)	605	=item $proc = $proc->eval ($perlcode, @args)
564		606
565	Evaluates the given C<$perlcode> as ... perl code, while setting C<@_> to	607	Evaluates the given C<$perlcode> as ... Perl code, while setting C<@_> to
566	the strings specified by C<@args>, in the "main" package.	608	the strings specified by C<@args>, in the "main" package.
567		609
568	This call is meant to do any custom initialisation that might be required	610	This call is meant to do any custom initialisation that might be required
569	(for example, the C<require> method uses it). It's not supposed to be used	611	(for example, the C<require> method uses it). It's not supposed to be used
570	to completely take over the process, use C<run> for that.	612	to completely take over the process, use C<run> for that.
571		613
572	The code will usually be executed after this call returns, and there is no	614	The code will usually be executed after this call returns, and there is no
573	way to pass anything back to the calling process. Any evaluation errors	615	way to pass anything back to the calling process. Any evaluation errors
574	will be reported to stderr and cause the process to exit.	616	will be reported to stderr and cause the process to exit.
575		617
576	If you want to execute some code to take over the process (see the	618	If you want to execute some code (that isn't in a module) to take over the
577	"fork+exec" example in the SYNOPSIS), you should compile a function via	619	process, you should compile a function via C<eval> first, and then call
578	C<eval> first, and then call it via C<run>. This also gives you access to	620	it via C<run>. This also gives you access to any arguments passed via the
579	any arguments passed via the C<send_xxx> methods, such as file handles.	621	C<send_xxx> methods, such as file handles. See the L<use AnyEvent::Fork as
		622	a faster fork+exec> example to see it in action.
580		623
581	Returns the process object for easy chaining of method calls.	624	Returns the process object for easy chaining of method calls.
582		625
583	=cut	626	=cut
584		627
…		…
610	=item $proc = $proc->send_fh ($handle, ...)	653	=item $proc = $proc->send_fh ($handle, ...)
611		654
612	Send one or more file handles (I<not> file descriptors) to the process,	655	Send one or more file handles (I<not> file descriptors) to the process,
613	to prepare a call to C<run>.	656	to prepare a call to C<run>.
614		657
615	The process object keeps a reference to the handles until this is done,	658	The process object keeps a reference to the handles until they have
616	so you must not explicitly close the handles. This is most easily	659	been passed over to the process, so you must not explicitly close the
617	accomplished by simply not storing the file handles anywhere after passing	660	handles. This is most easily accomplished by simply not storing the file
618	them to this method.	661	handles anywhere after passing them to this method - when AnyEvent::Fork
		662	is finished using them, perl will automatically close them.
619		663
620	Returns the process object for easy chaining of method calls.	664	Returns the process object for easy chaining of method calls.
621		665
622	Example: pass a file handle to a process, and release it without	666	Example: pass a file handle to a process, and release it without
623	closing. It will be closed automatically when it is no longer used.	667	closing. It will be closed automatically when it is no longer used.
…		…
630	sub send_fh {	674	sub send_fh {
631	my ($self, @fh) = @_;	675	my ($self, @fh) = @_;
632		676
633	for my $fh (@fh) {	677	for my $fh (@fh) {
634	$self->_cmd ("h");	678	$self->_cmd ("h");
635	push @{ $self->[2] }, \$fh;	679	push @{ $self->[QUEUE] }, \$fh;
636	}	680	}
637		681
638	$self	682	$self
639	}	683	}
640		684
641	=item $proc = $proc->send_arg ($string, ...)	685	=item $proc = $proc->send_arg ($string, ...)
642		686
643	Send one or more argument strings to the process, to prepare a call to	687	Send one or more argument strings to the process, to prepare a call to
644	C<run>. The strings can be any octet string.	688	C<run>. The strings can be any octet strings.
645		689
646	The protocol is optimised to pass a moderate number of relatively short	690	The protocol is optimised to pass a moderate number of relatively short
647	strings - while you can pass up to 4GB of data in one go, this is more	691	strings - while you can pass up to 4GB of data in one go, this is more
648	meant to pass some ID information or other startup info, not big chunks of	692	meant to pass some ID information or other startup info, not big chunks of
649	data.	693	data.
…		…
665	Enter the function specified by the function name in C<$func> in the	709	Enter the function specified by the function name in C<$func> in the
666	process. The function is called with the communication socket as first	710	process. The function is called with the communication socket as first
667	argument, followed by all file handles and string arguments sent earlier	711	argument, followed by all file handles and string arguments sent earlier
668	via C<send_fh> and C<send_arg> methods, in the order they were called.	712	via C<send_fh> and C<send_arg> methods, in the order they were called.
669		713
		714	The process object becomes unusable on return from this function - any
		715	further method calls result in undefined behaviour.
		716
670	The function name should be fully qualified, but if it isn't, it will be	717	The function name should be fully qualified, but if it isn't, it will be
671	looked up in the main package.	718	looked up in the C<main> package.
672		719
673	If the called function returns, doesn't exist, or any error occurs, the	720	If the called function returns, doesn't exist, or any error occurs, the
674	process exits.	721	process exits.
675		722
676	Preparing the process is done in the background - when all commands have	723	Preparing the process is done in the background - when all commands have
677	been sent, the callback is invoked with the local communications socket	724	been sent, the callback is invoked with the local communications socket
678	as argument. At this point you can start using the socket in any way you	725	as argument. At this point you can start using the socket in any way you
679	like.	726	like.
680
681	The process object becomes unusable on return from this function - any
682	further method calls result in undefined behaviour.
683		727
684	If the communication socket isn't used, it should be closed on both sides,	728	If the communication socket isn't used, it should be closed on both sides,
685	to save on kernel memory.	729	to save on kernel memory.
686		730
687	The socket is non-blocking in the parent, and blocking in the newly	731	The socket is non-blocking in the parent, and blocking in the newly
…		…
726	=cut	770	=cut
727		771
728	sub run {	772	sub run {
729	my ($self, $func, $cb) = @_;	773	my ($self, $func, $cb) = @_;
730		774
731	$self->[4] = $cb;	775	$self->[CB] = $cb;
732	$self->_cmd (r => $func);	776	$self->_cmd (r => $func);
		777	}
		778
		779	=item $proc->to_fh ($cb->($fh))
		780
		781	Flushes all commands out to the process and then calls the callback with
		782	the communications socket.
		783
		784	The process object becomes unusable on return from this function - any
		785	further method calls result in undefined behaviour.
		786
		787	The point of this method is to give you a file handle thta you cna pass
		788	to another process. In that other process, you can call C<new_from_fh
		789	AnyEvent::Fork::RPC> to create a new C<AnyEvent::Fork> object from it,
		790	thereby effectively passing a fork object to another process.
		791
		792	=cut
		793
		794	sub to_fh {
		795	my ($self, $cb) = @_;
		796
		797	$self->[CB] = $cb;
		798
		799	unless ($self->[WW]) {
		800	$self->[CB]->($self->[FH]);
		801	@$self = ();
		802	}
		803	}
		804
		805	=item new_from_fh AnyEvent::Fork $fh
		806
		807	Takes a file handle originally rceeived by the C<to_fh> method and creates
		808	a new C<AnyEvent:Fork> object. The child process itself will not change in
		809	any way, i.e. it will keep all the modifications done to it before calling
		810	C<to_fh>.
		811
		812	The new object is very much like the original object, except that the
		813	C<pid> method will return C<undef> even if the process is a direct child.
		814
		815	=cut
		816
		817	sub new_from_fh {
		818	my ($class, $fh) = @_;
		819
		820	$class->_new ($fh)
733	}	821	}
734		822
735	=back	823	=back
736		824
737	=head1 PERFORMANCE	825	=head1 PERFORMANCE
…		…
747		835
748	2079 new processes per second, using manual socketpair + fork	836	2079 new processes per second, using manual socketpair + fork
749		837
750	Then I did the same thing, but instead of calling fork, I called	838	Then I did the same thing, but instead of calling fork, I called
751	AnyEvent::Fork->new->run ("CORE::exit") and then again waited for the	839	AnyEvent::Fork->new->run ("CORE::exit") and then again waited for the
752	socket form the child to close on exit. This does the same thing as manual	840	socket from the child to close on exit. This does the same thing as manual
753	socket pair + fork, except that what is forked is the template process	841	socket pair + fork, except that what is forked is the template process
754	(2440kB), and the socket needs to be passed to the server at the other end	842	(2440kB), and the socket needs to be passed to the server at the other end
755	of the socket first.	843	of the socket first.
756		844
757	2307 new processes per second, using AnyEvent::Fork->new	845	2307 new processes per second, using AnyEvent::Fork->new
…		…
762	479 vfork+execs per second, using AnyEvent::Fork->new_exec	850	479 vfork+execs per second, using AnyEvent::Fork->new_exec
763		851
764	So how can C<< AnyEvent->new >> be faster than a standard fork, even	852	So how can C<< AnyEvent->new >> be faster than a standard fork, even
765	though it uses the same operations, but adds a lot of overhead?	853	though it uses the same operations, but adds a lot of overhead?
766		854
767	The difference is simply the process size: forking the 6MB process takes	855	The difference is simply the process size: forking the 5MB process takes
768	so much longer than forking the 2.5MB template process that the overhead	856	so much longer than forking the 2.5MB template process that the extra
769	introduced is canceled out.	857	overhead is canceled out.
770		858
771	If the benchmark process grows, the normal fork becomes even slower:	859	If the benchmark process grows, the normal fork becomes even slower:
772		860
773	1340 new processes, manual fork in a 20MB process	861	1340 new processes, manual fork of a 20MB process
774	731 new processes, manual fork in a 200MB process	862	731 new processes, manual fork of a 200MB process
775	235 new processes, manual fork in a 2000MB process	863	235 new processes, manual fork of a 2000MB process
776		864
777	What that means (to me) is that I can use this module without having a	865	What that means (to me) is that I can use this module without having a bad
778	very bad conscience because of the extra overhead required to start new	866	conscience because of the extra overhead required to start new processes.
779	processes.
780		867
781	=head1 TYPICAL PROBLEMS	868	=head1 TYPICAL PROBLEMS
782		869
783	This section lists typical problems that remain. I hope by recognising	870	This section lists typical problems that remain. I hope by recognising
784	them, most can be avoided.	871	them, most can be avoided.
785		872
786	=over 4	873	=over 4
787		874
788	=item "leaked" file descriptors for exec'ed processes	875	=item leaked file descriptors for exec'ed processes
789		876
790	POSIX systems inherit file descriptors by default when exec'ing a new	877	POSIX systems inherit file descriptors by default when exec'ing a new
791	process. While perl itself laudably sets the close-on-exec flags on new	878	process. While perl itself laudably sets the close-on-exec flags on new
792	file handles, most C libraries don't care, and even if all cared, it's	879	file handles, most C libraries don't care, and even if all cared, it's
793	often not possible to set the flag in a race-free manner.	880	often not possible to set the flag in a race-free manner.
…		…
813	libraries or the code that leaks those file descriptors.	900	libraries or the code that leaks those file descriptors.
814		901
815	Fortunately, most of these leaked descriptors do no harm, other than	902	Fortunately, most of these leaked descriptors do no harm, other than
816	sitting on some resources.	903	sitting on some resources.
817		904
818	=item "leaked" file descriptors for fork'ed processes	905	=item leaked file descriptors for fork'ed processes
819		906
820	Normally, L<AnyEvent::Fork> does start new processes by exec'ing them,	907	Normally, L<AnyEvent::Fork> does start new processes by exec'ing them,
821	which closes file descriptors not marked for being inherited.	908	which closes file descriptors not marked for being inherited.
822		909
823	However, L<AnyEvent::Fork::Early> and L<AnyEvent::Fork::Template> offer	910	However, L<AnyEvent::Fork::Early> and L<AnyEvent::Fork::Template> offer
…		…
832		919
833	The solution is to either not load these modules before use'ing	920	The solution is to either not load these modules before use'ing
834	L<AnyEvent::Fork::Early> or L<AnyEvent::Fork::Template>, or to delay	921	L<AnyEvent::Fork::Early> or L<AnyEvent::Fork::Template>, or to delay
835	initialising them, for example, by calling C<init Gtk2> manually.	922	initialising them, for example, by calling C<init Gtk2> manually.
836		923
837	=item exit runs destructors	924	=item exiting calls object destructors
838		925
839	This only applies to users of Lc<AnyEvent::Fork:Early> and	926	This only applies to users of L<AnyEvent::Fork:Early> and
840	L<AnyEvent::Fork::Template>.	927	L<AnyEvent::Fork::Template>, or when initialising code creates objects
		928	that reference external resources.
841		929
842	When a process created by AnyEvent::Fork exits, it might do so by calling	930	When a process created by AnyEvent::Fork exits, it might do so by calling
843	exit, or simply letting perl reach the end of the program. At which point	931	exit, or simply letting perl reach the end of the program. At which point
844	Perl runs all destructors.	932	Perl runs all destructors.
845		933
…		…
864	to make it so, mostly due to the bloody broken perl that nobody seems to	952	to make it so, mostly due to the bloody broken perl that nobody seems to
865	care about. The fork emulation is a bad joke - I have yet to see something	953	care about. The fork emulation is a bad joke - I have yet to see something
866	useful that you can do with it without running into memory corruption	954	useful that you can do with it without running into memory corruption
867	issues or other braindamage. Hrrrr.	955	issues or other braindamage. Hrrrr.
868		956
869	Cygwin perl is not supported at the moment, as it should implement fd	957	Since fork is endlessly broken on win32 perls (it doesn't even remotely
870	passing, but doesn't, and rolling my own is hard, as cygwin doesn't	958	work within it's documented limits) and quite obviously it's not getting
871	support enough functionality to do it.	959	improved any time soon, the best way to proceed on windows would be to
		960	always use C<new_exec> and thus never rely on perl's fork "emulation".
		961
		962	Cygwin perl is not supported at the moment due to some hilarious
		963	shortcomings of its API - see L<IO::FDPoll> for more details. If you never
		964	use C<send_fh> and always use C<new_exec> to create processes, it should
		965	work though.
872		966
873	=head1 SEE ALSO	967	=head1 SEE ALSO
874		968
875	L<AnyEvent::Fork::Early> (to avoid executing a perl interpreter),	969	L<AnyEvent::Fork::Early>, to avoid executing a perl interpreter at all
		970	(part of this distribution).
		971
876	L<AnyEvent::Fork::Template> (to create a process by forking the main	972	L<AnyEvent::Fork::Template>, to create a process by forking the main
877	program at a convenient time).	973	program at a convenient time (part of this distribution).
878		974
879	=head1 AUTHOR	975	L<AnyEvent::Fork::RPC>, for simple RPC to child processes (on CPAN).
		976
		977	=head1 AUTHOR AND CONTACT INFORMATION
880		978
881	Marc Lehmann <schmorp@schmorp.de>	979	Marc Lehmann <schmorp@schmorp.de>
882	http://home.schmorp.de/	980	http://software.schmorp.de/pkg/AnyEvent-Fork
883		981
884	=cut	982	=cut
885		983
886	1	984	1
887		985

Diff Legend

-–
+Removed lines
-+
+Added lines
-<
+Changed lines
->
+Changed lines

Comparing AnyEvent-Fork/Fork.pm (file contents): Revision 1.26 by root, Sat Apr 6 08:58:51 2013 UTC vs. Revision 1.49 by root, Fri Apr 19 12:56:53 2013 UTC

Diff Legend

Comparing AnyEvent-Fork/Fork.pm (file contents):
Revision 1.26 by root, Sat Apr 6 08:58:51 2013 UTC vs.
Revision 1.49 by root, Fri Apr 19 12:56:53 2013 UTC