[ViewVC] Diff of: cvs/AnyEvent-Fork/Fork.pm

Comparing AnyEvent-Fork/Fork.pm (file contents):
Revision 1.29 by root, Sat Apr 6 09:15:49 2013 UTC vs.
Revision 1.42 by root, Mon Apr 8 05:44:23 2013 UTC

…		…
27		27
28	Special care has been taken to make this module useful from other modules,	28	Special care has been taken to make this module useful from other modules,
29	while still supporting specialised environments such as L<App::Staticperl>	29	while still supporting specialised environments such as L<App::Staticperl>
30	or L<PAR::Packer>.	30	or L<PAR::Packer>.
31		31
32	=head1 WHAT THIS MODULE IS NOT	32	=head2 WHAT THIS MODULE IS NOT
33		33
34	This module only creates processes and lets you pass file handles and	34	This module only creates processes and lets you pass file handles and
35	strings to it, and run perl code. It does not implement any kind of RPC -	35	strings to it, and run perl code. It does not implement any kind of RPC -
36	there is no back channel from the process back to you, and there is no RPC	36	there is no back channel from the process back to you, and there is no RPC
37	or message passing going on.	37	or message passing going on.
…		…
40	in whatever way you like, use some message-passing module such	40	in whatever way you like, use some message-passing module such
41	as L<AnyEvent::MP>, some pipe such as L<AnyEvent::ZeroMQ>, use	41	as L<AnyEvent::MP>, some pipe such as L<AnyEvent::ZeroMQ>, use
42	L<AnyEvent::Handle> on both sides to send e.g. JSON or Storable messages,	42	L<AnyEvent::Handle> on both sides to send e.g. JSON or Storable messages,
43	and so on.	43	and so on.
44		44
		45	=head2 COMPARISON TO OTHER MODULES
		46
		47	There is an abundance of modules on CPAN that do "something fork", such as
		48	L<Parallel::ForkManager>, L<AnyEvent::ForkManager>, L<AnyEvent::Worker>
		49	or L<AnyEvent::Subprocess>. There are modules that implement their own
		50	process management, such as L<AnyEvent::DBI>.
		51
		52	The problems that all these modules try to solve are real, however, none
		53	of them (from what I have seen) tackle the very real problems of unwanted
		54	memory sharing, efficiency, not being able to use event processing or
		55	similar modules in the processes they create.
		56
		57	This module doesn't try to replace any of them - instead it tries to solve
		58	the problem of creating processes with a minimum of fuss and overhead (and
		59	also luxury). Ideally, most of these would use AnyEvent::Fork internally,
		60	except they were written before AnyEvent:Fork was available, so obviously
		61	had to roll their own.
		62
45	=head1 PROBLEM STATEMENT	63	=head2 PROBLEM STATEMENT
46		64
47	There are two traditional ways to implement parallel processing on UNIX	65	There are two traditional ways to implement parallel processing on UNIX
48	like operating systems - fork and process, and fork+exec and process. They	66	like operating systems - fork and process, and fork+exec and process. They
49	have different advantages and disadvantages that I describe below,	67	have different advantages and disadvantages that I describe below,
50	together with how this module tries to mitigate the disadvantages.	68	together with how this module tries to mitigate the disadvantages.
…		…
152		170
153	# now $master_filehandle is connected to the	171	# now $master_filehandle is connected to the
154	# $slave_filehandle in the new process.	172	# $slave_filehandle in the new process.
155	});	173	});
156		174
157	# MyModule::worker might look like this	175	C<MyModule> might look like this:
		176
		177	package MyModule;
		178
158	sub MyModule::worker {	179	sub worker {
159	my ($slave_filehandle) = @_;	180	my ($slave_filehandle) = @_;
160		181
161	# now $slave_filehandle is connected to the $master_filehandle	182	# now $slave_filehandle is connected to the $master_filehandle
162	# in the original prorcess. have fun!	183	# in the original prorcess. have fun!
163	}	184	}
…		…
182	}	203	}
183		204
184	# now do other things - maybe use the filehandle provided by run	205	# now do other things - maybe use the filehandle provided by run
185	# to wait for the processes to die. or whatever.	206	# to wait for the processes to die. or whatever.
186		207
187	# My::Server::run might look like this	208	C<My::Server> might look like this:
188	sub My::Server::run {	209
		210	package My::Server;
		211
		212	sub run {
189	my ($slave, $listener, $id) = @_;	213	my ($slave, $listener, $id) = @_;
190		214
191	close $slave; # we do not use the socket, so close it to save resources	215	close $slave; # we do not use the socket, so close it to save resources
192		216
193	# we could go ballistic and use e.g. AnyEvent here, or IO::AIO,	217	# we could go ballistic and use e.g. AnyEvent here, or IO::AIO,
…		…
197	}	221	}
198	}	222	}
199		223
200	=head2 use AnyEvent::Fork as a faster fork+exec	224	=head2 use AnyEvent::Fork as a faster fork+exec
201		225
202	This runs /bin/echo hi, with stdout redirected to /tmp/log and stderr to	226	This runs C</bin/echo hi>, with stdandard output redirected to /tmp/log
203	the communications socket. It is usually faster than fork+exec, but still	227	and standard error redirected to the communications socket. It is usually
204	let's you prepare the environment.	228	faster than fork+exec, but still lets you prepare the environment.
205		229
206	open my $output, ">/tmp/log" or die "$!";	230	open my $output, ">/tmp/log" or die "$!";
207		231
208	AnyEvent::Fork	232	AnyEvent::Fork
209	->new	233	->new
210	->eval ('	234	->eval ('
		235	# compile a helper function for later use
211	sub run {	236	sub run {
212	my ($fh, $output, @cmd) = @_;	237	my ($fh, $output, @cmd) = @_;
213		238
214	# perl will clear close-on-exec on STDOUT/STDERR	239	# perl will clear close-on-exec on STDOUT/STDERR
215	open STDOUT, ">&", $output or die;	240	open STDOUT, ">&", $output or die;
…		…
345	use AnyEvent;	370	use AnyEvent;
346	use AnyEvent::Util ();	371	use AnyEvent::Util ();
347		372
348	use IO::FDPass;	373	use IO::FDPass;
349		374
350	our $VERSION = 0.5;	375	our $VERSION = 0.6;
351
352	our $PERL; # the path to the perl interpreter, deduces with various forms of magic
353		376
354	=over 4	377	=over 4
355		378
356	=back	379	=back
357		380
…		…
360	# the early fork template process	383	# the early fork template process
361	our $EARLY;	384	our $EARLY;
362		385
363	# the empty template process	386	# the empty template process
364	our $TEMPLATE;	387	our $TEMPLATE;
		388
		389	sub QUEUE() { 0 }
		390	sub FH() { 1 }
		391	sub WW() { 2 }
		392	sub PID() { 3 }
		393	sub CB() { 4 }
		394
		395	sub _new {
		396	my ($self, $fh, $pid) = @_;
		397
		398	AnyEvent::Util::fh_nonblocking $fh, 1;
		399
		400	$self = bless [
		401	[], # write queue - strings or fd's
		402	$fh,
		403	undef, # AE watcher
		404	$pid,
		405	], $self;
		406
		407	$self
		408	}
365		409
366	sub _cmd {	410	sub _cmd {
367	my $self = shift;	411	my $self = shift;
368		412
369	# ideally, we would want to use "a (w/a)*" as format string, but perl	413	# ideally, we would want to use "a (w/a)*" as format string, but perl
370	# versions from at least 5.8.9 to 5.16.3 are all buggy and can't unpack	414	# versions from at least 5.8.9 to 5.16.3 are all buggy and can't unpack
371	# it.	415	# it.
372	push @{ $self->[2] }, pack "a L/a*", $_[0], $_[1];	416	push @{ $self->[QUEUE] }, pack "a L/a*", $_[0], $_[1];
373		417
374	$self->[3] \|\|= AE::io $self->[1], 1, sub {	418	$self->[WW] \|\|= AE::io $self->[FH], 1, sub {
375	do {	419	do {
376	# send the next "thing" in the queue - either a reference to an fh,	420	# send the next "thing" in the queue - either a reference to an fh,
377	# or a plain string.	421	# or a plain string.
378		422
379	if (ref $self->[2][0]) {	423	if (ref $self->[QUEUE][0]) {
380	# send fh	424	# send fh
381	unless (IO::FDPass::send fileno $self->[1], fileno ${ $self->[2][0] }) {	425	unless (IO::FDPass::send fileno $self->[FH], fileno ${ $self->[QUEUE][0] }) {
382	return if $! == Errno::EAGAIN \|\| $! == Errno::EWOULDBLOCK;	426	return if $! == Errno::EAGAIN \|\| $! == Errno::EWOULDBLOCK;
383	undef $self->[3];	427	undef $self->[WW];
384	die "AnyEvent::Fork: file descriptor send failure: $!";	428	die "AnyEvent::Fork: file descriptor send failure: $!";
385	}	429	}
386		430
387	shift @{ $self->[2] };	431	shift @{ $self->[QUEUE] };
388		432
389	} else {	433	} else {
390	# send string	434	# send string
391	my $len = syswrite $self->[1], $self->[2][0];	435	my $len = syswrite $self->[FH], $self->[QUEUE][0];
392		436
393	unless ($len) {	437	unless ($len) {
394	return if $! == Errno::EAGAIN \|\| $! == Errno::EWOULDBLOCK;	438	return if $! == Errno::EAGAIN \|\| $! == Errno::EWOULDBLOCK;
395	undef $self->[3];	439	undef $self->[3];
396	die "AnyEvent::Fork: command write failure: $!";	440	die "AnyEvent::Fork: command write failure: $!";
397	}	441	}
398		442
399	substr $self->[2][0], 0, $len, "";	443	substr $self->[QUEUE][0], 0, $len, "";
400	shift @{ $self->[2] } unless length $self->[2][0];	444	shift @{ $self->[QUEUE] } unless length $self->[QUEUE][0];
401	}	445	}
402	} while @{ $self->[2] };	446	} while @{ $self->[QUEUE] };
403		447
404	# everything written	448	# everything written
405	undef $self->[3];	449	undef $self->[WW];
406		450
407	# invoke run callback, if any	451	# invoke run callback, if any
408	$self->[4]->($self->[1]) if $self->[4];	452	$self->[CB]->($self->[FH]) if $self->[CB];
409	};	453	};
410		454
411	() # make sure we don't leak the watcher	455	() # make sure we don't leak the watcher
412	}
413
414	sub _new {
415	my ($self, $fh, $pid) = @_;
416
417	AnyEvent::Util::fh_nonblocking $fh, 1;
418
419	$self = bless [
420	$pid,
421	$fh,
422	[], # write queue - strings or fd's
423	undef, # AE watcher
424	], $self;
425
426	$self
427	}	456	}
428		457
429	# fork template from current process, used by AnyEvent::Fork::Early/Template	458	# fork template from current process, used by AnyEvent::Fork::Early/Template
430	sub _new_fork {	459	sub _new_fork {
431	my ($fh, $slave) = AnyEvent::Util::portable_socketpair;	460	my ($fh, $slave) = AnyEvent::Util::portable_socketpair;
…		…
436	if ($pid eq 0) {	465	if ($pid eq 0) {
437	require AnyEvent::Fork::Serve;	466	require AnyEvent::Fork::Serve;
438	$AnyEvent::Fork::Serve::OWNER = $parent;	467	$AnyEvent::Fork::Serve::OWNER = $parent;
439	close $fh;	468	close $fh;
440	$0 = "$_[1] of $parent";	469	$0 = "$_[1] of $parent";
441	$SIG{CHLD} = 'IGNORE';
442	AnyEvent::Fork::Serve::serve ($slave);	470	AnyEvent::Fork::Serve::serve ($slave);
443	exit 0;	471	exit 0;
444	} elsif (!$pid) {	472	} elsif (!$pid) {
445	die "AnyEvent::Fork::Early/Template: unable to fork template process: $!";	473	die "AnyEvent::Fork::Early/Template: unable to fork template process: $!";
446	}	474	}
…		…
553	}	581	}
554		582
555	=item $pid = $proc->pid	583	=item $pid = $proc->pid
556		584
557	Returns the process id of the process I<iff it is a direct child of the	585	Returns the process id of the process I<iff it is a direct child of the
558	process> running AnyEvent::Fork, and C<undef> otherwise.	586	process running AnyEvent::Fork>, and C<undef> otherwise.
559		587
560	Normally, only processes created via C<< AnyEvent::Fork->new_exec >> and	588	Normally, only processes created via C<< AnyEvent::Fork->new_exec >> and
561	L<AnyEvent::Fork::Template> are direct children, and you are responsible	589	L<AnyEvent::Fork::Template> are direct children, and you are responsible
562	to clean up their zombies when they die.	590	to clean up their zombies when they die.
563		591
564	All other processes are not direct children, and will be cleaned up by	592	All other processes are not direct children, and will be cleaned up by
565	AnyEvent::Fork.	593	AnyEvent::Fork itself.
566		594
567	=cut	595	=cut
568		596
569	sub pid {	597	sub pid {
570	$_[0][0]	598	$_[0][PID]
571	}	599	}
572		600
573	=item $proc = $proc->eval ($perlcode, @args)	601	=item $proc = $proc->eval ($perlcode, @args)
574		602
575	Evaluates the given C<$perlcode> as ... perl code, while setting C<@_> to	603	Evaluates the given C<$perlcode> as ... perl code, while setting C<@_> to
…		…
581		609
582	The code will usually be executed after this call returns, and there is no	610	The code will usually be executed after this call returns, and there is no
583	way to pass anything back to the calling process. Any evaluation errors	611	way to pass anything back to the calling process. Any evaluation errors
584	will be reported to stderr and cause the process to exit.	612	will be reported to stderr and cause the process to exit.
585		613
586	If you want to execute some code to take over the process (see the	614	If you want to execute some code (that isn't in a module) to take over the
587	"fork+exec" example in the SYNOPSIS), you should compile a function via	615	process, you should compile a function via C<eval> first, and then call
588	C<eval> first, and then call it via C<run>. This also gives you access to	616	it via C<run>. This also gives you access to any arguments passed via the
589	any arguments passed via the C<send_xxx> methods, such as file handles.	617	C<send_xxx> methods, such as file handles. See the L<use AnyEvent::Fork as
		618	a faster fork+exec> example to see it in action.
590		619
591	Returns the process object for easy chaining of method calls.	620	Returns the process object for easy chaining of method calls.
592		621
593	=cut	622	=cut
594		623
…		…
620	=item $proc = $proc->send_fh ($handle, ...)	649	=item $proc = $proc->send_fh ($handle, ...)
621		650
622	Send one or more file handles (I<not> file descriptors) to the process,	651	Send one or more file handles (I<not> file descriptors) to the process,
623	to prepare a call to C<run>.	652	to prepare a call to C<run>.
624		653
625	The process object keeps a reference to the handles until this is done,	654	The process object keeps a reference to the handles until they have
626	so you must not explicitly close the handles. This is most easily	655	been passed over to the process, so you must not explicitly close the
627	accomplished by simply not storing the file handles anywhere after passing	656	handles. This is most easily accomplished by simply not storing the file
628	them to this method.	657	handles anywhere after passing them to this method - when AnyEvent::Fork
		658	is finished using them, perl will automatically close them.
629		659
630	Returns the process object for easy chaining of method calls.	660	Returns the process object for easy chaining of method calls.
631		661
632	Example: pass a file handle to a process, and release it without	662	Example: pass a file handle to a process, and release it without
633	closing. It will be closed automatically when it is no longer used.	663	closing. It will be closed automatically when it is no longer used.
…		…
640	sub send_fh {	670	sub send_fh {
641	my ($self, @fh) = @_;	671	my ($self, @fh) = @_;
642		672
643	for my $fh (@fh) {	673	for my $fh (@fh) {
644	$self->_cmd ("h");	674	$self->_cmd ("h");
645	push @{ $self->[2] }, \$fh;	675	push @{ $self->[QUEUE] }, \$fh;
646	}	676	}
647		677
648	$self	678	$self
649	}	679	}
650		680
651	=item $proc = $proc->send_arg ($string, ...)	681	=item $proc = $proc->send_arg ($string, ...)
652		682
653	Send one or more argument strings to the process, to prepare a call to	683	Send one or more argument strings to the process, to prepare a call to
654	C<run>. The strings can be any octet string.	684	C<run>. The strings can be any octet strings.
655		685
656	The protocol is optimised to pass a moderate number of relatively short	686	The protocol is optimised to pass a moderate number of relatively short
657	strings - while you can pass up to 4GB of data in one go, this is more	687	strings - while you can pass up to 4GB of data in one go, this is more
658	meant to pass some ID information or other startup info, not big chunks of	688	meant to pass some ID information or other startup info, not big chunks of
659	data.	689	data.
…		…
675	Enter the function specified by the function name in C<$func> in the	705	Enter the function specified by the function name in C<$func> in the
676	process. The function is called with the communication socket as first	706	process. The function is called with the communication socket as first
677	argument, followed by all file handles and string arguments sent earlier	707	argument, followed by all file handles and string arguments sent earlier
678	via C<send_fh> and C<send_arg> methods, in the order they were called.	708	via C<send_fh> and C<send_arg> methods, in the order they were called.
679		709
		710	The process object becomes unusable on return from this function - any
		711	further method calls result in undefined behaviour.
		712
680	The function name should be fully qualified, but if it isn't, it will be	713	The function name should be fully qualified, but if it isn't, it will be
681	looked up in the main package.	714	looked up in the C<main> package.
682		715
683	If the called function returns, doesn't exist, or any error occurs, the	716	If the called function returns, doesn't exist, or any error occurs, the
684	process exits.	717	process exits.
685		718
686	Preparing the process is done in the background - when all commands have	719	Preparing the process is done in the background - when all commands have
687	been sent, the callback is invoked with the local communications socket	720	been sent, the callback is invoked with the local communications socket
688	as argument. At this point you can start using the socket in any way you	721	as argument. At this point you can start using the socket in any way you
689	like.	722	like.
690
691	The process object becomes unusable on return from this function - any
692	further method calls result in undefined behaviour.
693		723
694	If the communication socket isn't used, it should be closed on both sides,	724	If the communication socket isn't used, it should be closed on both sides,
695	to save on kernel memory.	725	to save on kernel memory.
696		726
697	The socket is non-blocking in the parent, and blocking in the newly	727	The socket is non-blocking in the parent, and blocking in the newly
…		…
736	=cut	766	=cut
737		767
738	sub run {	768	sub run {
739	my ($self, $func, $cb) = @_;	769	my ($self, $func, $cb) = @_;
740		770
741	$self->[4] = $cb;	771	$self->[CB] = $cb;
742	$self->_cmd (r => $func);	772	$self->_cmd (r => $func);
743	}	773	}
744		774
745	=back	775	=back
746		776
…		…
772	479 vfork+execs per second, using AnyEvent::Fork->new_exec	802	479 vfork+execs per second, using AnyEvent::Fork->new_exec
773		803
774	So how can C<< AnyEvent->new >> be faster than a standard fork, even	804	So how can C<< AnyEvent->new >> be faster than a standard fork, even
775	though it uses the same operations, but adds a lot of overhead?	805	though it uses the same operations, but adds a lot of overhead?
776		806
777	The difference is simply the process size: forking the 6MB process takes	807	The difference is simply the process size: forking the 5MB process takes
778	so much longer than forking the 2.5MB template process that the overhead	808	so much longer than forking the 2.5MB template process that the extra
779	introduced is canceled out.	809	overhead introduced is canceled out.
780		810
781	If the benchmark process grows, the normal fork becomes even slower:	811	If the benchmark process grows, the normal fork becomes even slower:
782		812
783	1340 new processes, manual fork in a 20MB process	813	1340 new processes, manual fork of a 20MB process
784	731 new processes, manual fork in a 200MB process	814	731 new processes, manual fork of a 200MB process
785	235 new processes, manual fork in a 2000MB process	815	235 new processes, manual fork of a 2000MB process
786		816
787	What that means (to me) is that I can use this module without having a	817	What that means (to me) is that I can use this module without having a bad
788	very bad conscience because of the extra overhead required to start new	818	conscience because of the extra overhead required to start new processes.
789	processes.
790		819
791	=head1 TYPICAL PROBLEMS	820	=head1 TYPICAL PROBLEMS
792		821
793	This section lists typical problems that remain. I hope by recognising	822	This section lists typical problems that remain. I hope by recognising
794	them, most can be avoided.	823	them, most can be avoided.
795		824
796	=over 4	825	=over 4
797		826
798	=item "leaked" file descriptors for exec'ed processes	827	=item leaked file descriptors for exec'ed processes
799		828
800	POSIX systems inherit file descriptors by default when exec'ing a new	829	POSIX systems inherit file descriptors by default when exec'ing a new
801	process. While perl itself laudably sets the close-on-exec flags on new	830	process. While perl itself laudably sets the close-on-exec flags on new
802	file handles, most C libraries don't care, and even if all cared, it's	831	file handles, most C libraries don't care, and even if all cared, it's
803	often not possible to set the flag in a race-free manner.	832	often not possible to set the flag in a race-free manner.
…		…
823	libraries or the code that leaks those file descriptors.	852	libraries or the code that leaks those file descriptors.
824		853
825	Fortunately, most of these leaked descriptors do no harm, other than	854	Fortunately, most of these leaked descriptors do no harm, other than
826	sitting on some resources.	855	sitting on some resources.
827		856
828	=item "leaked" file descriptors for fork'ed processes	857	=item leaked file descriptors for fork'ed processes
829		858
830	Normally, L<AnyEvent::Fork> does start new processes by exec'ing them,	859	Normally, L<AnyEvent::Fork> does start new processes by exec'ing them,
831	which closes file descriptors not marked for being inherited.	860	which closes file descriptors not marked for being inherited.
832		861
833	However, L<AnyEvent::Fork::Early> and L<AnyEvent::Fork::Template> offer	862	However, L<AnyEvent::Fork::Early> and L<AnyEvent::Fork::Template> offer
…		…
842		871
843	The solution is to either not load these modules before use'ing	872	The solution is to either not load these modules before use'ing
844	L<AnyEvent::Fork::Early> or L<AnyEvent::Fork::Template>, or to delay	873	L<AnyEvent::Fork::Early> or L<AnyEvent::Fork::Template>, or to delay
845	initialising them, for example, by calling C<init Gtk2> manually.	874	initialising them, for example, by calling C<init Gtk2> manually.
846		875
847	=item exit runs destructors	876	=item exiting calls object destructors
848		877
849	This only applies to users of Lc<AnyEvent::Fork:Early> and	878	This only applies to users of L<AnyEvent::Fork:Early> and
850	L<AnyEvent::Fork::Template>.	879	L<AnyEvent::Fork::Template>, or when initialiasing code creates objects
		880	that reference external resources.
851		881
852	When a process created by AnyEvent::Fork exits, it might do so by calling	882	When a process created by AnyEvent::Fork exits, it might do so by calling
853	exit, or simply letting perl reach the end of the program. At which point	883	exit, or simply letting perl reach the end of the program. At which point
854	Perl runs all destructors.	884	Perl runs all destructors.
855		885
…		…
874	to make it so, mostly due to the bloody broken perl that nobody seems to	904	to make it so, mostly due to the bloody broken perl that nobody seems to
875	care about. The fork emulation is a bad joke - I have yet to see something	905	care about. The fork emulation is a bad joke - I have yet to see something
876	useful that you can do with it without running into memory corruption	906	useful that you can do with it without running into memory corruption
877	issues or other braindamage. Hrrrr.	907	issues or other braindamage. Hrrrr.
878		908
879	Cygwin perl is not supported at the moment, as it should implement fd	909	Cygwin perl is not supported at the moment due to some hilarious
880	passing, but doesn't, and rolling my own is hard, as cygwin doesn't	910	shortcomings of its API - see L<IO::FDPoll> for more details.
881	support enough functionality to do it.
882		911
883	=head1 SEE ALSO	912	=head1 SEE ALSO
884		913
885	L<AnyEvent::Fork::Early> (to avoid executing a perl interpreter),	914	L<AnyEvent::Fork::Early> (to avoid executing a perl interpreter),
886	L<AnyEvent::Fork::Template> (to create a process by forking the main	915	L<AnyEvent::Fork::Template> (to create a process by forking the main

Diff Legend

-–
+Removed lines
-+
+Added lines
-<
+Changed lines
->
+Changed lines

Comparing AnyEvent-Fork/Fork.pm (file contents): Revision 1.29 by root, Sat Apr 6 09:15:49 2013 UTC vs. Revision 1.42 by root, Mon Apr 8 05:44:23 2013 UTC

Diff Legend

Comparing AnyEvent-Fork/Fork.pm (file contents):
Revision 1.29 by root, Sat Apr 6 09:15:49 2013 UTC vs.
Revision 1.42 by root, Mon Apr 8 05:44:23 2013 UTC