[ViewVC] Diff of: cvs/AnyEvent-Fork/Fork.pm

Comparing AnyEvent-Fork/Fork.pm (file contents):
Revision 1.40 by root, Sat Apr 6 22:41:56 2013 UTC vs.
Revision 1.49 by root, Fri Apr 19 12:56:53 2013 UTC

…		…
34	This module only creates processes and lets you pass file handles and	34	This module only creates processes and lets you pass file handles and
35	strings to it, and run perl code. It does not implement any kind of RPC -	35	strings to it, and run perl code. It does not implement any kind of RPC -
36	there is no back channel from the process back to you, and there is no RPC	36	there is no back channel from the process back to you, and there is no RPC
37	or message passing going on.	37	or message passing going on.
38		38
39	If you need some form of RPC, you can either implement it yourself	39	If you need some form of RPC, you could use the L<AnyEvent::Fork::RPC>
40	in whatever way you like, use some message-passing module such	40	companion module, which adds simple RPC/job queueing to a process created
41	as L<AnyEvent::MP>, some pipe such as L<AnyEvent::ZeroMQ>, use	41	by this module.
42	L<AnyEvent::Handle> on both sides to send e.g. JSON or Storable messages,	42
43	and so on.	43	Or you can implement it yourself in whatever way you like, use some
		44	message-passing module such as L<AnyEvent::MP>, some pipe such as
		45	L<AnyEvent::ZeroMQ>, use L<AnyEvent::Handle> on both sides to send
		46	e.g. JSON or Storable messages, and so on.
44		47
45	=head2 COMPARISON TO OTHER MODULES	48	=head2 COMPARISON TO OTHER MODULES
46		49
47	There is an abundance of modules on CPAN that do "something fork", such as	50	There is an abundance of modules on CPAN that do "something fork", such as
48	L<Parallel::ForkManager>, L<AnyEvent::ForkManager>, L<AnyEvent::Worker>	51	L<Parallel::ForkManager>, L<AnyEvent::ForkManager>, L<AnyEvent::Worker>
…		…
221	}	224	}
222	}	225	}
223		226
224	=head2 use AnyEvent::Fork as a faster fork+exec	227	=head2 use AnyEvent::Fork as a faster fork+exec
225		228
226	This runs C</bin/echo hi>, with stdandard output redirected to /tmp/log	229	This runs C</bin/echo hi>, with standard output redirected to F</tmp/log>
227	and standard error redirected to the communications socket. It is usually	230	and standard error redirected to the communications socket. It is usually
228	faster than fork+exec, but still lets you prepare the environment.	231	faster than fork+exec, but still lets you prepare the environment.
229		232
230	open my $output, ">/tmp/log" or die "$!";	233	open my $output, ">/tmp/log" or die "$!";
231		234
…		…
251		254
252	=head1 CONCEPTS	255	=head1 CONCEPTS
253		256
254	This module can create new processes either by executing a new perl	257	This module can create new processes either by executing a new perl
255	process, or by forking from an existing "template" process.	258	process, or by forking from an existing "template" process.
		259
		260	All these processes are called "child processes" (whether they are direct
		261	children or not), while the process that manages them is called the
		262	"parent process".
256		263
257	Each such process comes with its own file handle that can be used to	264	Each such process comes with its own file handle that can be used to
258	communicate with it (it's actually a socket - one end in the new process,	265	communicate with it (it's actually a socket - one end in the new process,
259	one end in the main process), and among the things you can do in it are	266	one end in the main process), and among the things you can do in it are
260	load modules, fork new processes, send file handles to it, and execute	267	load modules, fork new processes, send file handles to it, and execute
…		…
370	use AnyEvent;	377	use AnyEvent;
371	use AnyEvent::Util ();	378	use AnyEvent::Util ();
372		379
373	use IO::FDPass;	380	use IO::FDPass;
374		381
375	our $VERSION = 0.6;	382	our $VERSION = 0.7;
376
377	our $PERL; # the path to the perl interpreter, deduces with various forms of magic
378
379	=over 4
380
381	=back
382
383	=cut
384		383
385	# the early fork template process	384	# the early fork template process
386	our $EARLY;	385	our $EARLY;
387		386
388	# the empty template process	387	# the empty template process
389	our $TEMPLATE;	388	our $TEMPLATE;
		389
		390	sub QUEUE() { 0 }
		391	sub FH() { 1 }
		392	sub WW() { 2 }
		393	sub PID() { 3 }
		394	sub CB() { 4 }
		395
		396	sub _new {
		397	my ($self, $fh, $pid) = @_;
		398
		399	AnyEvent::Util::fh_nonblocking $fh, 1;
		400
		401	$self = bless [
		402	[], # write queue - strings or fd's
		403	$fh,
		404	undef, # AE watcher
		405	$pid,
		406	], $self;
		407
		408	$self
		409	}
390		410
391	sub _cmd {	411	sub _cmd {
392	my $self = shift;	412	my $self = shift;
393		413
394	# ideally, we would want to use "a (w/a)*" as format string, but perl	414	# ideally, we would want to use "a (w/a)*" as format string, but perl
395	# versions from at least 5.8.9 to 5.16.3 are all buggy and can't unpack	415	# versions from at least 5.8.9 to 5.16.3 are all buggy and can't unpack
396	# it.	416	# it.
397	push @{ $self->[2] }, pack "a L/a*", $_[0], $_[1];	417	push @{ $self->[QUEUE] }, pack "a L/a*", $_[0], $_[1];
398		418
399	$self->[3] \|\|= AE::io $self->[1], 1, sub {	419	$self->[WW] \|\|= AE::io $self->[FH], 1, sub {
400	do {	420	do {
401	# send the next "thing" in the queue - either a reference to an fh,	421	# send the next "thing" in the queue - either a reference to an fh,
402	# or a plain string.	422	# or a plain string.
403		423
404	if (ref $self->[2][0]) {	424	if (ref $self->[QUEUE][0]) {
405	# send fh	425	# send fh
406	unless (IO::FDPass::send fileno $self->[1], fileno ${ $self->[2][0] }) {	426	unless (IO::FDPass::send fileno $self->[FH], fileno ${ $self->[QUEUE][0] }) {
407	return if $! == Errno::EAGAIN \|\| $! == Errno::EWOULDBLOCK;	427	return if $! == Errno::EAGAIN \|\| $! == Errno::EWOULDBLOCK;
408	undef $self->[3];	428	undef $self->[WW];
409	die "AnyEvent::Fork: file descriptor send failure: $!";	429	die "AnyEvent::Fork: file descriptor send failure: $!";
410	}	430	}
411		431
412	shift @{ $self->[2] };	432	shift @{ $self->[QUEUE] };
413		433
414	} else {	434	} else {
415	# send string	435	# send string
416	my $len = syswrite $self->[1], $self->[2][0];	436	my $len = syswrite $self->[FH], $self->[QUEUE][0];
417		437
418	unless ($len) {	438	unless ($len) {
419	return if $! == Errno::EAGAIN \|\| $! == Errno::EWOULDBLOCK;	439	return if $! == Errno::EAGAIN \|\| $! == Errno::EWOULDBLOCK;
420	undef $self->[3];	440	undef $self->[3];
421	die "AnyEvent::Fork: command write failure: $!";	441	die "AnyEvent::Fork: command write failure: $!";
422	}	442	}
423		443
424	substr $self->[2][0], 0, $len, "";	444	substr $self->[QUEUE][0], 0, $len, "";
425	shift @{ $self->[2] } unless length $self->[2][0];	445	shift @{ $self->[QUEUE] } unless length $self->[QUEUE][0];
426	}	446	}
427	} while @{ $self->[2] };	447	} while @{ $self->[QUEUE] };
428		448
429	# everything written	449	# everything written
430	undef $self->[3];	450	undef $self->[WW];
431		451
432	# invoke run callback, if any	452	# invoke run callback, if any
433	$self->[4]->($self->[1]) if $self->[4];	453	if ($self->[CB]) {
		454	$self->[CB]->($self->[FH]);
		455	@$self = ();
		456	}
434	};	457	};
435		458
436	() # make sure we don't leak the watcher	459	() # make sure we don't leak the watcher
437	}
438
439	sub _new {
440	my ($self, $fh, $pid) = @_;
441
442	AnyEvent::Util::fh_nonblocking $fh, 1;
443
444	$self = bless [
445	$pid,
446	$fh,
447	[], # write queue - strings or fd's
448	undef, # AE watcher
449	], $self;
450
451	$self
452	}	460	}
453		461
454	# fork template from current process, used by AnyEvent::Fork::Early/Template	462	# fork template from current process, used by AnyEvent::Fork::Early/Template
455	sub _new_fork {	463	sub _new_fork {
456	my ($fh, $slave) = AnyEvent::Util::portable_socketpair;	464	my ($fh, $slave) = AnyEvent::Util::portable_socketpair;
…		…
461	if ($pid eq 0) {	469	if ($pid eq 0) {
462	require AnyEvent::Fork::Serve;	470	require AnyEvent::Fork::Serve;
463	$AnyEvent::Fork::Serve::OWNER = $parent;	471	$AnyEvent::Fork::Serve::OWNER = $parent;
464	close $fh;	472	close $fh;
465	$0 = "$_[1] of $parent";	473	$0 = "$_[1] of $parent";
466	$SIG{CHLD} = 'IGNORE';
467	AnyEvent::Fork::Serve::serve ($slave);	474	AnyEvent::Fork::Serve::serve ($slave);
468	exit 0;	475	exit 0;
469	} elsif (!$pid) {	476	} elsif (!$pid) {
470	die "AnyEvent::Fork::Early/Template: unable to fork template process: $!";	477	die "AnyEvent::Fork::Early/Template: unable to fork template process: $!";
471	}	478	}
…		…
590	AnyEvent::Fork itself.	597	AnyEvent::Fork itself.
591		598
592	=cut	599	=cut
593		600
594	sub pid {	601	sub pid {
595	$_[0][0]	602	$_[0][PID]
596	}	603	}
597		604
598	=item $proc = $proc->eval ($perlcode, @args)	605	=item $proc = $proc->eval ($perlcode, @args)
599		606
600	Evaluates the given C<$perlcode> as ... perl code, while setting C<@_> to	607	Evaluates the given C<$perlcode> as ... Perl code, while setting C<@_> to
601	the strings specified by C<@args>, in the "main" package.	608	the strings specified by C<@args>, in the "main" package.
602		609
603	This call is meant to do any custom initialisation that might be required	610	This call is meant to do any custom initialisation that might be required
604	(for example, the C<require> method uses it). It's not supposed to be used	611	(for example, the C<require> method uses it). It's not supposed to be used
605	to completely take over the process, use C<run> for that.	612	to completely take over the process, use C<run> for that.
…		…
667	sub send_fh {	674	sub send_fh {
668	my ($self, @fh) = @_;	675	my ($self, @fh) = @_;
669		676
670	for my $fh (@fh) {	677	for my $fh (@fh) {
671	$self->_cmd ("h");	678	$self->_cmd ("h");
672	push @{ $self->[2] }, \$fh;	679	push @{ $self->[QUEUE] }, \$fh;
673	}	680	}
674		681
675	$self	682	$self
676	}	683	}
677		684
…		…
763	=cut	770	=cut
764		771
765	sub run {	772	sub run {
766	my ($self, $func, $cb) = @_;	773	my ($self, $func, $cb) = @_;
767		774
768	$self->[4] = $cb;	775	$self->[CB] = $cb;
769	$self->_cmd (r => $func);	776	$self->_cmd (r => $func);
		777	}
		778
		779	=item $proc->to_fh ($cb->($fh))
		780
		781	Flushes all commands out to the process and then calls the callback with
		782	the communications socket.
		783
		784	The process object becomes unusable on return from this function - any
		785	further method calls result in undefined behaviour.
		786
		787	The point of this method is to give you a file handle thta you cna pass
		788	to another process. In that other process, you can call C<new_from_fh
		789	AnyEvent::Fork::RPC> to create a new C<AnyEvent::Fork> object from it,
		790	thereby effectively passing a fork object to another process.
		791
		792	=cut
		793
		794	sub to_fh {
		795	my ($self, $cb) = @_;
		796
		797	$self->[CB] = $cb;
		798
		799	unless ($self->[WW]) {
		800	$self->[CB]->($self->[FH]);
		801	@$self = ();
		802	}
		803	}
		804
		805	=item new_from_fh AnyEvent::Fork $fh
		806
		807	Takes a file handle originally rceeived by the C<to_fh> method and creates
		808	a new C<AnyEvent:Fork> object. The child process itself will not change in
		809	any way, i.e. it will keep all the modifications done to it before calling
		810	C<to_fh>.
		811
		812	The new object is very much like the original object, except that the
		813	C<pid> method will return C<undef> even if the process is a direct child.
		814
		815	=cut
		816
		817	sub new_from_fh {
		818	my ($class, $fh) = @_;
		819
		820	$class->_new ($fh)
770	}	821	}
771		822
772	=back	823	=back
773		824
774	=head1 PERFORMANCE	825	=head1 PERFORMANCE
…		…
784		835
785	2079 new processes per second, using manual socketpair + fork	836	2079 new processes per second, using manual socketpair + fork
786		837
787	Then I did the same thing, but instead of calling fork, I called	838	Then I did the same thing, but instead of calling fork, I called
788	AnyEvent::Fork->new->run ("CORE::exit") and then again waited for the	839	AnyEvent::Fork->new->run ("CORE::exit") and then again waited for the
789	socket form the child to close on exit. This does the same thing as manual	840	socket from the child to close on exit. This does the same thing as manual
790	socket pair + fork, except that what is forked is the template process	841	socket pair + fork, except that what is forked is the template process
791	(2440kB), and the socket needs to be passed to the server at the other end	842	(2440kB), and the socket needs to be passed to the server at the other end
792	of the socket first.	843	of the socket first.
793		844
794	2307 new processes per second, using AnyEvent::Fork->new	845	2307 new processes per second, using AnyEvent::Fork->new
…		…
801	So how can C<< AnyEvent->new >> be faster than a standard fork, even	852	So how can C<< AnyEvent->new >> be faster than a standard fork, even
802	though it uses the same operations, but adds a lot of overhead?	853	though it uses the same operations, but adds a lot of overhead?
803		854
804	The difference is simply the process size: forking the 5MB process takes	855	The difference is simply the process size: forking the 5MB process takes
805	so much longer than forking the 2.5MB template process that the extra	856	so much longer than forking the 2.5MB template process that the extra
806	overhead introduced is canceled out.	857	overhead is canceled out.
807		858
808	If the benchmark process grows, the normal fork becomes even slower:	859	If the benchmark process grows, the normal fork becomes even slower:
809		860
810	1340 new processes, manual fork of a 20MB process	861	1340 new processes, manual fork of a 20MB process
811	731 new processes, manual fork of a 200MB process	862	731 new processes, manual fork of a 200MB process
…		…
871	initialising them, for example, by calling C<init Gtk2> manually.	922	initialising them, for example, by calling C<init Gtk2> manually.
872		923
873	=item exiting calls object destructors	924	=item exiting calls object destructors
874		925
875	This only applies to users of L<AnyEvent::Fork:Early> and	926	This only applies to users of L<AnyEvent::Fork:Early> and
876	L<AnyEvent::Fork::Template>, or when initialiasing code creates objects	927	L<AnyEvent::Fork::Template>, or when initialising code creates objects
877	that reference external resources.	928	that reference external resources.
878		929
879	When a process created by AnyEvent::Fork exits, it might do so by calling	930	When a process created by AnyEvent::Fork exits, it might do so by calling
880	exit, or simply letting perl reach the end of the program. At which point	931	exit, or simply letting perl reach the end of the program. At which point
881	Perl runs all destructors.	932	Perl runs all destructors.
…		…
901	to make it so, mostly due to the bloody broken perl that nobody seems to	952	to make it so, mostly due to the bloody broken perl that nobody seems to
902	care about. The fork emulation is a bad joke - I have yet to see something	953	care about. The fork emulation is a bad joke - I have yet to see something
903	useful that you can do with it without running into memory corruption	954	useful that you can do with it without running into memory corruption
904	issues or other braindamage. Hrrrr.	955	issues or other braindamage. Hrrrr.
905		956
		957	Since fork is endlessly broken on win32 perls (it doesn't even remotely
		958	work within it's documented limits) and quite obviously it's not getting
		959	improved any time soon, the best way to proceed on windows would be to
		960	always use C<new_exec> and thus never rely on perl's fork "emulation".
		961
906	Cygwin perl is not supported at the moment due to some hilarious	962	Cygwin perl is not supported at the moment due to some hilarious
907	shortcomings of its API - see L<IO::FDPoll> for more details.	963	shortcomings of its API - see L<IO::FDPoll> for more details. If you never
		964	use C<send_fh> and always use C<new_exec> to create processes, it should
		965	work though.
908		966
909	=head1 SEE ALSO	967	=head1 SEE ALSO
910		968
911	L<AnyEvent::Fork::Early> (to avoid executing a perl interpreter),	969	L<AnyEvent::Fork::Early>, to avoid executing a perl interpreter at all
		970	(part of this distribution).
		971
912	L<AnyEvent::Fork::Template> (to create a process by forking the main	972	L<AnyEvent::Fork::Template>, to create a process by forking the main
913	program at a convenient time).	973	program at a convenient time (part of this distribution).
914		974
915	=head1 AUTHOR	975	L<AnyEvent::Fork::RPC>, for simple RPC to child processes (on CPAN).
		976
		977	=head1 AUTHOR AND CONTACT INFORMATION
916		978
917	Marc Lehmann <schmorp@schmorp.de>	979	Marc Lehmann <schmorp@schmorp.de>
918	http://home.schmorp.de/	980	http://software.schmorp.de/pkg/AnyEvent-Fork
919		981
920	=cut	982	=cut
921		983
922	1	984	1
923		985

Diff Legend

-–
+Removed lines
-+
+Added lines
-<
+Changed lines
->
+Changed lines

Comparing AnyEvent-Fork/Fork.pm (file contents): Revision 1.40 by root, Sat Apr 6 22:41:56 2013 UTC vs. Revision 1.49 by root, Fri Apr 19 12:56:53 2013 UTC

Diff Legend

Comparing AnyEvent-Fork/Fork.pm (file contents):
Revision 1.40 by root, Sat Apr 6 22:41:56 2013 UTC vs.
Revision 1.49 by root, Fri Apr 19 12:56:53 2013 UTC