[ViewVC] Diff of: cvs/AnyEvent-Fork/Fork.pm

Comparing AnyEvent-Fork/Fork.pm (file contents):
Revision 1.8 by root, Thu Apr 4 01:54:40 2013 UTC vs.
Revision 1.18 by root, Sat Apr 6 01:33:56 2013 UTC

…		…
3	AnyEvent::Fork - everything you wanted to use fork() for, but couldn't	3	AnyEvent::Fork - everything you wanted to use fork() for, but couldn't
4		4
5	=head1 SYNOPSIS	5	=head1 SYNOPSIS
6		6
7	use AnyEvent::Fork;	7	use AnyEvent::Fork;
		8
		9	##################################################################
		10	# create a single new process, tell it to run your worker function
		11
		12	AnyEvent::Fork
		13	->new
		14	->require ("MyModule")
		15	->run ("MyModule::worker, sub {
		16	my ($master_filehandle) = @_;
		17
		18	# now $master_filehandle is connected to the
		19	# $slave_filehandle in the new process.
		20	});
		21
		22	# MyModule::worker might look like this
		23	sub MyModule::worker {
		24	my ($slave_filehandle) = @_;
		25
		26	# now $slave_filehandle is connected to the $master_filehandle
		27	# in the original prorcess. have fun!
		28	}
		29
		30	##################################################################
		31	# create a pool of server processes all accepting on the same socket
		32
		33	# create listener socket
		34	my $listener = ...;
		35
		36	# create a pool template, initialise it and give it the socket
		37	my $pool = AnyEvent::Fork
		38	->new
		39	->require ("Some::Stuff", "My::Server")
		40	->send_fh ($listener);
		41
		42	# now create 10 identical workers
		43	for my $id (1..10) {
		44	$pool
		45	->fork
		46	->send_arg ($id)
		47	->run ("My::Server::run");
		48	}
		49
		50	# now do other things - maybe use the filehandle provided by run
		51	# to wait for the processes to die. or whatever.
		52
		53	# My::Server::run might look like this
		54	sub My::Server::run {
		55	my ($slave, $listener, $id) = @_;
		56
		57	close $slave; # we do not use the socket, so close it to save resources
		58
		59	# we could go ballistic and use e.g. AnyEvent here, or IO::AIO,
		60	# or anything we usually couldn't do in a process forked normally.
		61	while (my $socket = $listener->accept) {
		62	# do sth. with new socket
		63	}
		64	}
8		65
9	=head1 DESCRIPTION	66	=head1 DESCRIPTION
10		67
11	This module allows you to create new processes, without actually forking	68	This module allows you to create new processes, without actually forking
12	them from your current process (avoiding the problems of forking), but	69	them from your current process (avoiding the problems of forking), but
13	preserving most of the advantages of fork.	70	preserving most of the advantages of fork.
14		71
15	It can be used to create new worker processes or new independent	72	It can be used to create new worker processes or new independent
16	subprocesses for short- and long-running jobs, process pools (e.g. for use	73	subprocesses for short- and long-running jobs, process pools (e.g. for use
17	in pre-forked servers) but also to spawn new external processes (such as	74	in pre-forked servers) but also to spawn new external processes (such as
18	CGI scripts from a webserver), which can be faster (and more well behaved)	75	CGI scripts from a web server), which can be faster (and more well behaved)
19	than using fork+exec in big processes.	76	than using fork+exec in big processes.
20		77
21	Special care has been taken to make this module useful from other modules,	78	Special care has been taken to make this module useful from other modules,
22	while still supporting specialised environments such as L<App::Staticperl>	79	while still supporting specialised environments such as L<App::Staticperl>
23	or L<PAR::Packer>.	80	or L<PAR::Packer>.
		81
		82	=head1 WHAT THIS MODULE IS NOT
		83
		84	This module only creates processes and lets you pass file handles and
		85	strings to it, and run perl code. It does not implement any kind of RPC -
		86	there is no back channel from the process back to you, and there is no RPC
		87	or message passing going on.
		88
		89	If you need some form of RPC, you can either implement it yourself
		90	in whatever way you like, use some message-passing module such
		91	as L<AnyEvent::MP>, some pipe such as L<AnyEvent::ZeroMQ>, use
		92	L<AnyEvent::Handle> on both sides to send e.g. JSON or Storable messages,
		93	and so on.
24		94
25	=head1 PROBLEM STATEMENT	95	=head1 PROBLEM STATEMENT
26		96
27	There are two ways to implement parallel processing on UNIX like operating	97	There are two ways to implement parallel processing on UNIX like operating
28	systems - fork and process, and fork+exec and process. They have different	98	systems - fork and process, and fork+exec and process. They have different
…		…
40	or fork+exec instead.	110	or fork+exec instead.
41		111
42	=item Forking usually creates a copy-on-write copy of the parent	112	=item Forking usually creates a copy-on-write copy of the parent
43	process. Memory (for example, modules or data files that have been	113	process. Memory (for example, modules or data files that have been
44	will not take additional memory). When exec'ing a new process, modules	114	will not take additional memory). When exec'ing a new process, modules
45	and data files might need to be loaded again, at extra cpu and memory	115	and data files might need to be loaded again, at extra CPU and memory
46	cost. Likewise when forking, all data structures are copied as well - if	116	cost. Likewise when forking, all data structures are copied as well - if
47	the program frees them and replaces them by new data, the child processes	117	the program frees them and replaces them by new data, the child processes
48	will retain the memory even if it isn't used.	118	will retain the memory even if it isn't used.
49		119
50	This module allows the main program to do a controlled fork, and allows	120	This module allows the main program to do a controlled fork, and allows
…		…
62	as template, and also tries hard to identify the correct path to the perl	132	as template, and also tries hard to identify the correct path to the perl
63	interpreter. With a cooperative main program, exec'ing the interpreter	133	interpreter. With a cooperative main program, exec'ing the interpreter
64	might not even be necessary.	134	might not even be necessary.
65		135
66	=item Forking might be impossible when a program is running. For example,	136	=item Forking might be impossible when a program is running. For example,
67	POSIX makes it almost impossible to fork from a multithreaded program and	137	POSIX makes it almost impossible to fork from a multi-threaded program and
68	do anything useful in the child - strictly speaking, if your perl program	138	do anything useful in the child - strictly speaking, if your perl program
69	uses posix threads (even indirectly via e.g. L<IO::AIO> or L<threads>),	139	uses posix threads (even indirectly via e.g. L<IO::AIO> or L<threads>),
70	you cannot call fork on the perl level anymore, at all.	140	you cannot call fork on the perl level anymore, at all.
71		141
72	This module can safely fork helper processes at any time, by caling	142	This module can safely fork helper processes at any time, by calling
73	fork+exec in C, in a POSIX-compatible way.	143	fork+exec in C, in a POSIX-compatible way.
74		144
75	=item Parallel processing with fork might be inconvenient or difficult	145	=item Parallel processing with fork might be inconvenient or difficult
76	to implement. For example, when a program uses an event loop and creates	146	to implement. For example, when a program uses an event loop and creates
77	watchers it becomes very hard to use the event loop from a child	147	watchers it becomes very hard to use the event loop from a child
…		…
109	needed the first time. Forking from this process shares the memory used	179	needed the first time. Forking from this process shares the memory used
110	for the perl interpreter with the new process, but loading modules takes	180	for the perl interpreter with the new process, but loading modules takes
111	time, and the memory is not shared with anything else.	181	time, and the memory is not shared with anything else.
112		182
113	This is ideal for when you only need one extra process of a kind, with the	183	This is ideal for when you only need one extra process of a kind, with the
114	option of starting and stipping it on demand.	184	option of starting and stopping it on demand.
		185
		186	Example:
		187
		188	AnyEvent::Fork
		189	->new
		190	->require ("Some::Module")
		191	->run ("Some::Module::run", sub {
		192	my ($fork_fh) = @_;
		193	});
115		194
116	=item fork a new template process, load code, then fork processes off of	195	=item fork a new template process, load code, then fork processes off of
117	it and run the code	196	it and run the code
118		197
119	When you need to have a bunch of processes that all execute the same (or	198	When you need to have a bunch of processes that all execute the same (or
…		…
125	modules you loaded) is shared between the processes, and each new process	204	modules you loaded) is shared between the processes, and each new process
126	consumes relatively little memory of its own.	205	consumes relatively little memory of its own.
127		206
128	The disadvantage of this approach is that you need to create a template	207	The disadvantage of this approach is that you need to create a template
129	process for the sole purpose of forking new processes from it, but if you	208	process for the sole purpose of forking new processes from it, but if you
130	only need a fixed number of proceses you can create them, and then destroy	209	only need a fixed number of processes you can create them, and then destroy
131	the template process.	210	the template process.
		211
		212	Example:
		213
		214	my $template = AnyEvent::Fork->new->require ("Some::Module");
		215
		216	for (1..10) {
		217	$template->fork->run ("Some::Module::run", sub {
		218	my ($fork_fh) = @_;
		219	});
		220	}
		221
		222	# at this point, you can keep $template around to fork new processes
		223	# later, or you can destroy it, which causes it to vanish.
132		224
133	=item execute a new perl interpreter, load some code, run it	225	=item execute a new perl interpreter, load some code, run it
134		226
135	This is relatively slow, and doesn't allow you to share memory between	227	This is relatively slow, and doesn't allow you to share memory between
136	multiple processes.	228	multiple processes.
…		…
138	The only advantage is that you don't have to have a template process	230	The only advantage is that you don't have to have a template process
139	hanging around all the time to fork off some new processes, which might be	231	hanging around all the time to fork off some new processes, which might be
140	an advantage when there are long time spans where no extra processes are	232	an advantage when there are long time spans where no extra processes are
141	needed.	233	needed.
142		234
		235	Example:
		236
		237	AnyEvent::Fork
		238	->new_exec
		239	->require ("Some::Module")
		240	->run ("Some::Module::run", sub {
		241	my ($fork_fh) = @_;
		242	});
		243
143	=back	244	=back
144		245
145	=head1 FUNCTIONS	246	=head1 FUNCTIONS
146		247
147	=over 4	248	=over 4
…		…
150		251
151	package AnyEvent::Fork;	252	package AnyEvent::Fork;
152		253
153	use common::sense;	254	use common::sense;
154		255
155	use Socket ();	256	use Errno ();
156		257
157	use AnyEvent;	258	use AnyEvent;
158	use AnyEvent::Fork::Util;
159	use AnyEvent::Util ();	259	use AnyEvent::Util ();
		260
		261	use IO::FDPass;
		262
		263	our $VERSION = 0.2;
160		264
161	our $PERL; # the path to the perl interpreter, deduces with various forms of magic	265	our $PERL; # the path to the perl interpreter, deduces with various forms of magic
162		266
163	=item my $pool = new AnyEvent::Fork key => value...	267	=item my $pool = new AnyEvent::Fork key => value...
164		268
…		…
177	our $TEMPLATE;	281	our $TEMPLATE;
178		282
179	sub _cmd {	283	sub _cmd {
180	my $self = shift;	284	my $self = shift;
181		285
182	# ideally, we would want to use "a (w/a)*" as format string, but perl versions	286	# ideally, we would want to use "a (w/a)*" as format string, but perl
183	# from at least 5.8.9 to 5.16.3 are all buggy and can't unpack it.	287	# versions from at least 5.8.9 to 5.16.3 are all buggy and can't unpack
		288	# it.
184	push @{ $self->[2] }, pack "N/a", pack "(w/a)*", @_;	289	push @{ $self->[2] }, pack "L/a", pack "(w/a)*", @_;
185		290
186	$self->[3] \|\|= AE::io $self->[1], 1, sub {	291	unless ($self->[3]) {
		292	my $wcb = sub {
		293	do {
		294	# send the next "thing" in the queue - either a reference to an fh,
		295	# or a plain string.
		296
187	if (ref $self->[2][0]) {	297	if (ref $self->[2][0]) {
		298	# send fh
188	AnyEvent::Fork::Util::fd_send fileno $self->[1], fileno ${ $self->[2][0] }	299	unless (IO::FDPass::send fileno $self->[1], fileno ${ $self->[2][0] }) {
		300	return if $! == Errno::EAGAIN \|\| $! == Errno::EWOULDBLOCK;
		301	undef $self->[3];
		302	die "AnyEvent::Fork: file descriptor send failure: $!";
		303	}
		304
189	and shift @{ $self->[2] };	305	shift @{ $self->[2] };
190		306
191	} else {	307	} else {
		308	# send string
192	my $len = syswrite $self->[1], $self->[2][0]	309	my $len = syswrite $self->[1], $self->[2][0];
		310
		311	unless ($len) {
		312	return if $! == Errno::EAGAIN \|\| $! == Errno::EWOULDBLOCK;
		313	undef $self->[3];
193	or do { undef $self->[3]; die "AnyEvent::Fork: command write failure: $!" };	314	die "AnyEvent::Fork: command write failure: $!";
		315	}
194		316
195	substr $self->[2][0], 0, $len, "";	317	substr $self->[2][0], 0, $len, "";
196	shift @{ $self->[2] } unless length $self->[2][0];	318	shift @{ $self->[2] } unless length $self->[2][0];
197	}	319	}
		320	} while @{ $self->[2] };
198		321
199	unless (@{ $self->[2] }) {	322	# everything written
200	undef $self->[3];	323	undef $self->[3];
		324	# invoke run callback
201	$self->[0]->($self->[1]) if $self->[0];	325	$self->[0]->($self->[1]) if $self->[0];
202	}	326	};
		327
		328	$wcb->();
		329
		330	$self->[3] \|\|= AE::io $self->[1], 1, $wcb
		331	if @{ $self->[2] };
203	};	332	}
		333
		334	() # make sure we don't leak the watcher
204	}	335	}
205		336
206	sub _new {	337	sub _new {
207	my ($self, $fh) = @_;	338	my ($self, $fh) = @_;
208		339
…		…
213	$fh,	344	$fh,
214	[], # write queue - strings or fd's	345	[], # write queue - strings or fd's
215	undef, # AE watcher	346	undef, # AE watcher
216	], $self;	347	], $self;
217		348
218	# my ($a, $b) = AnyEvent::Util::portable_socketpair;
219
220	# queue_cmd $template, "Iabc";
221	# push @{ $template->[2] }, \$b;
222
223	# use Coro::AnyEvent; Coro::AnyEvent::sleep 1;
224	# undef $b;
225	# die "x" . <$a>;
226
227	$self	349	$self
228	}	350	}
229		351
230	# fork template from current process, used by AnyEvent::Fork::Early/Template	352	# fork template from current process, used by AnyEvent::Fork::Early/Template
231	sub _new_fork {	353	sub _new_fork {
…		…
237	if ($pid eq 0) {	359	if ($pid eq 0) {
238	require AnyEvent::Fork::Serve;	360	require AnyEvent::Fork::Serve;
239	$AnyEvent::Fork::Serve::OWNER = $parent;	361	$AnyEvent::Fork::Serve::OWNER = $parent;
240	close $fh;	362	close $fh;
241	$0 = "$_[1] of $parent";	363	$0 = "$_[1] of $parent";
		364	$SIG{CHLD} = 'IGNORE';
242	AnyEvent::Fork::Serve::serve ($slave);	365	AnyEvent::Fork::Serve::serve ($slave);
243	AnyEvent::Fork::Util::_exit 0;	366	exit 0;
244	} elsif (!$pid) {	367	} elsif (!$pid) {
245	die "AnyEvent::Fork::Early/Template: unable to fork template process: $!";	368	die "AnyEvent::Fork::Early/Template: unable to fork template process: $!";
246	}	369	}
247		370
248	AnyEvent::Fork->_new ($fh)	371	AnyEvent::Fork->_new ($fh)
…		…
254	object for further manipulation.	377	object for further manipulation.
255		378
256	The new process is forked from a template process that is kept around	379	The new process is forked from a template process that is kept around
257	for this purpose. When it doesn't exist yet, it is created by a call to	380	for this purpose. When it doesn't exist yet, it is created by a call to
258	C<new_exec> and kept around for future calls.	381	C<new_exec> and kept around for future calls.
		382
		383	When the process object is destroyed, it will release the file handle
		384	that connects it with the new process. When the new process has not yet
		385	called C<run>, then the process will exit. Otherwise, what happens depends
		386	entirely on the code that is executed.
259		387
260	=cut	388	=cut
261		389
262	sub new {	390	sub new {
263	my $class = shift;	391	my $class = shift;
…		…
299	reduces the amount of memory sharing that is possible, and is also slower.	427	reduces the amount of memory sharing that is possible, and is also slower.
300		428
301	You should use C<new> whenever possible, except when having a template	429	You should use C<new> whenever possible, except when having a template
302	process around is unacceptable.	430	process around is unacceptable.
303		431
304	The path to the perl interpreter is divined usign various methods - first	432	The path to the perl interpreter is divined using various methods - first
305	C<$^X> is investigated to see if the path ends with something that sounds	433	C<$^X> is investigated to see if the path ends with something that sounds
306	as if it were the perl interpreter. Failing this, the module falls back to	434	as if it were the perl interpreter. Failing this, the module falls back to
307	using C<$Config::Config{perlpath}>.	435	using C<$Config::Config{perlpath}>.
308		436
309	=cut	437	=cut
…		…
318	my $perl = $;	446	my $perl = $;
319		447
320	# first we try $^X, but the path must be absolute (always on win32), and end in sth.	448	# first we try $^X, but the path must be absolute (always on win32), and end in sth.
321	# that looks like perl. this obviously only works for posix and win32	449	# that looks like perl. this obviously only works for posix and win32
322	unless (	450	unless (
323	(AnyEvent::Fork::Util::WIN32 \|\| $perl =~ m%^/%)	451	($^O eq "MSWin32" \|\| $perl =~ m%^/%)
324	&& $perl =~ m%[/\\]perl(?:[0-9]+(\.[0-9]+)+)?(\.exe)?$%i	452	&& $perl =~ m%[/\\]perl(?:[0-9]+(\.[0-9]+)+)?(\.exe)?$%i
325	) {	453	) {
326	# if it doesn't look perlish enough, try Config	454	# if it doesn't look perlish enough, try Config
327	require Config;	455	require Config;
328	$perl = $Config::Config{perlpath};	456	$perl = $Config::Config{perlpath};
…		…
332	require Proc::FastSpawn;	460	require Proc::FastSpawn;
333		461
334	my ($fh, $slave) = AnyEvent::Util::portable_socketpair;	462	my ($fh, $slave) = AnyEvent::Util::portable_socketpair;
335	Proc::FastSpawn::fd_inherit (fileno $slave);	463	Proc::FastSpawn::fd_inherit (fileno $slave);
336		464
		465	# new fh's should always be set cloexec (due to $^F),
		466	# but hey, not on win32, so we always clear the inherit flag.
		467	Proc::FastSpawn::fd_inherit (fileno $fh, 0);
		468
337	# quick. also doesn't work in win32. of course. what did you expect	469	# quick. also doesn't work in win32. of course. what did you expect
338	#local $ENV{PERL5LIB} = join ":", grep !ref, @INC;	470	#local $ENV{PERL5LIB} = join ":", grep !ref, @INC;
339	my %env = %ENV;	471	my %env = %ENV;
340	$env{PERL5LIB} = join +(AnyEvent::Fork::Util::WIN32 ? ";" : ":"), grep !ref, @INC;	472	$env{PERL5LIB} = join +($^O eq "MSWin32" ? ";" : ":"), grep !ref, @INC;
341		473
342	Proc::FastSpawn::spawn (	474	Proc::FastSpawn::spawn (
343	$perl,	475	$perl,
344	["perl", "-MAnyEvent::Fork::Serve", "-e", "AnyEvent::Fork::Serve::me", fileno $slave, $$],	476	["perl", "-MAnyEvent::Fork::Serve", "-e", "AnyEvent::Fork::Serve::me", fileno $slave, $$],
345	[map "$_=$env{$_}", keys %env],	477	[map "$_=$env{$_}", keys %env],
346	) or die "unable to spawn AnyEvent::Fork server: $!";	478	) or die "unable to spawn AnyEvent::Fork server: $!";
347		479
348	$self->_new ($fh)	480	$self->_new ($fh)
349	}	481	}
350		482
		483	=item $proc = $proc->eval ($perlcode, @args)
		484
		485	Evaluates the given C<$perlcode> as ... perl code, while setting C<@_> to
		486	the strings specified by C<@args>.
		487
		488	This call is meant to do any custom initialisation that might be required
		489	(for example, the C<require> method uses it). It's not supposed to be used
		490	to completely take over the process, use C<run> for that.
		491
		492	The code will usually be executed after this call returns, and there is no
		493	way to pass anything back to the calling process. Any evaluation errors
		494	will be reported to stderr and cause the process to exit.
		495
		496	Returns the process object for easy chaining of method calls.
		497
		498	=cut
		499
		500	sub eval {
		501	my ($self, $code, @args) = @_;
		502
		503	$self->_cmd (e => $code, @args);
		504
		505	$self
		506	}
		507
351	=item $proc = $proc->require ($module, ...)	508	=item $proc = $proc->require ($module, ...)
352		509
353	Tries to load the given modules into the process	510	Tries to load the given module(s) into the process
354		511
355	Returns the process object for easy chaining of method calls.	512	Returns the process object for easy chaining of method calls.
		513
		514	=cut
		515
		516	sub require {
		517	my ($self, @modules) = @_;
		518
		519	s%::%/%g for @modules;
		520	$self->eval ('require "$_.pm" for @_', @modules);
		521
		522	$self
		523	}
356		524
357	=item $proc = $proc->send_fh ($handle, ...)	525	=item $proc = $proc->send_fh ($handle, ...)
358		526
359	Send one or more file handles (I<not> file descriptors) to the process,	527	Send one or more file handles (I<not> file descriptors) to the process,
360	to prepare a call to C<run>.	528	to prepare a call to C<run>.
…		…
364	accomplished by simply not storing the file handles anywhere after passing	532	accomplished by simply not storing the file handles anywhere after passing
365	them to this method.	533	them to this method.
366		534
367	Returns the process object for easy chaining of method calls.	535	Returns the process object for easy chaining of method calls.
368		536
		537	Example: pass a file handle to a process, and release it without
		538	closing. It will be closed automatically when it is no longer used.
		539
		540	$proc->send_fh ($my_fh);
		541	undef $my_fh; # free the reference if you want, but DO NOT CLOSE IT
		542
369	=cut	543	=cut
370		544
371	sub send_fh {	545	sub send_fh {
372	my ($self, @fh) = @_;	546	my ($self, @fh) = @_;
373		547
374	for my $fh (@fh) {	548	for my $fh (@fh) {
375	$self->_cmd ("h");	549	$self->_cmd ("h");
376	push @{ $self->[2] }, \$fh;	550	push @{ $self->[2] }, \$fh;
377	push @$self, $fh; # dire hack
378	}	551	}
379		552
380	$self	553	$self
381	}	554	}
382		555
383	=item $proc = $proc->send_arg ($string, ...)	556	=item $proc = $proc->send_arg ($string, ...)
384		557
385	Send one or more argument strings to the process, to prepare a call to	558	Send one or more argument strings to the process, to prepare a call to
386	C<run>. The strings can be any octet string.	559	C<run>. The strings can be any octet string.
387		560
		561	The protocol is optimised to pass a moderate number of relatively short
		562	strings - while you can pass up to 4GB of data in one go, this is more
		563	meant to pass some ID information or other startup info, not big chunks of
		564	data.
		565
388	Returns the process object for easy chaining of emthod calls.	566	Returns the process object for easy chaining of method calls.
389		567
390	=cut	568	=cut
391		569
392	sub send_arg {	570	sub send_arg {
393	my ($self, @arg) = @_;	571	my ($self, @arg) = @_;
…		…
414	If the communication socket isn't used, it should be closed on both sides,	592	If the communication socket isn't used, it should be closed on both sides,
415	to save on kernel memory.	593	to save on kernel memory.
416		594
417	The socket is non-blocking in the parent, and blocking in the newly	595	The socket is non-blocking in the parent, and blocking in the newly
418	created process. The close-on-exec flag is set on both. Even if not used	596	created process. The close-on-exec flag is set on both. Even if not used
419	otherwise, the socket can be a good indicator for the existance of the	597	otherwise, the socket can be a good indicator for the existence of the
420	process - if the other process exits, you get a readable event on it,	598	process - if the other process exits, you get a readable event on it,
421	because exiting the process closes the socket (if it didn't create any	599	because exiting the process closes the socket (if it didn't create any
422	children using fork).	600	children using fork).
423		601
		602	Example: create a template for a process pool, pass a few strings, some
		603	file handles, then fork, pass one more string, and run some code.
		604
		605	my $pool = AnyEvent::Fork
		606	->new
		607	->send_arg ("str1", "str2")
		608	->send_fh ($fh1, $fh2);
		609
		610	for (1..2) {
		611	$pool
		612	->fork
		613	->send_arg ("str3")
		614	->run ("Some::function", sub {
		615	my ($fh) = @_;
		616
		617	# fh is nonblocking, but we trust that the OS can accept these
		618	# extra 3 octets anyway.
		619	syswrite $fh, "hi #$_\n";
		620
		621	# $fh is being closed here, as we don't store it anywhere
		622	});
		623	}
		624
		625	# Some::function might look like this - all parameters passed before fork
		626	# and after will be passed, in order, after the communications socket.
		627	sub Some::function {
		628	my ($fh, $str1, $str2, $fh1, $fh2, $str3) = @_;
		629
		630	print scalar <$fh>; # prints "hi 1\n" and "hi 2\n"
		631	}
		632
424	=cut	633	=cut
425		634
426	sub run {	635	sub run {
427	my ($self, $func, $cb) = @_;	636	my ($self, $func, $cb) = @_;
428		637
429	$self->[0] = $cb;	638	$self->[0] = $cb;
430	$self->_cmd ("r", $func);	639	$self->_cmd (r => $func);
431	}	640	}
432		641
433	=back	642	=back
434		643
		644	=head1 PERFORMANCE
		645
		646	Now for some unscientific benchmark numbers (all done on an amd64
		647	GNU/Linux box). These are intended to give you an idea of the relative
		648	performance you can expect, they are not meant to be absolute performance
		649	numbers.
		650
		651	OK, so, I ran a simple benchmark that creates a socket pair, forks, calls
		652	exit in the child and waits for the socket to close in the parent. I did
		653	load AnyEvent, EV and AnyEvent::Fork, for a total process size of 5100kB.
		654
		655	2079 new processes per second, using manual socketpair + fork
		656
		657	Then I did the same thing, but instead of calling fork, I called
		658	AnyEvent::Fork->new->run ("CORE::exit") and then again waited for the
		659	socket form the child to close on exit. This does the same thing as manual
		660	socket pair + fork, except that what is forked is the template process
		661	(2440kB), and the socket needs to be passed to the server at the other end
		662	of the socket first.
		663
		664	2307 new processes per second, using AnyEvent::Fork->new
		665
		666	And finally, using C<new_exec> instead C<new>, using vforks+execs to exec
		667	a new perl interpreter and compile the small server each time, I get:
		668
		669	479 vfork+execs per second, using AnyEvent::Fork->new_exec
		670
		671	So how can C<< AnyEvent->new >> be faster than a standard fork, even
		672	though it uses the same operations, but adds a lot of overhead?
		673
		674	The difference is simply the process size: forking the 6MB process takes
		675	so much longer than forking the 2.5MB template process that the overhead
		676	introduced is canceled out.
		677
		678	If the benchmark process grows, the normal fork becomes even slower:
		679
		680	1340 new processes, manual fork in a 20MB process
		681	731 new processes, manual fork in a 200MB process
		682	235 new processes, manual fork in a 2000MB process
		683
		684	What that means (to me) is that I can use this module without having a
		685	very bad conscience because of the extra overhead required to start new
		686	processes.
		687
		688	=head1 TYPICAL PROBLEMS
		689
		690	This section lists typical problems that remain. I hope by recognising
		691	them, most can be avoided.
		692
		693	=over 4
		694
		695	=item exit runs destructors
		696
		697	=item "leaked" file descriptors for exec'ed processes
		698
		699	POSIX systems inherit file descriptors by default when exec'ing a new
		700	process. While perl itself laudably sets the close-on-exec flags on new
		701	file handles, most C libraries don't care, and even if all cared, it's
		702	often not possible to set the flag in a race-free manner.
		703
		704	That means some file descriptors can leak through. And since it isn't
		705	possible to know which file descriptors are "good" and "necessary" (or
		706	even to know which file descriptors are open), there is no good way to
		707	close the ones that might harm.
		708
		709	As an example of what "harm" can be done consider a web server that
		710	accepts connections and afterwards some module uses AnyEvent::Fork for the
		711	first time, causing it to fork and exec a new process, which might inherit
		712	the network socket. When the server closes the socket, it is still open
		713	in the child (which doesn't even know that) and the client might conclude
		714	that the connection is still fine.
		715
		716	For the main program, there are multiple remedies available -
		717	L<AnyEvent::Fork::Early> is one, creating a process early and not using
		718	C<new_exec> is another, as in both cases, the first process can be exec'ed
		719	well before many random file descriptors are open.
		720
		721	In general, the solution for these kind of problems is to fix the
		722	libraries or the code that leaks those file descriptors.
		723
		724	Fortunately, most of these leaked descriptors do no harm, other than
		725	sitting on some resources.
		726
		727	=item "leaked" file descriptors for fork'ed processes
		728
		729	Normally, L<AnyEvent::Fork> does start new processes by exec'ing them,
		730	which closes file descriptors not marked for being inherited.
		731
		732	However, L<AnyEvent::Fork::Early> and L<AnyEvent::Fork::Template> offer
		733	a way to create these processes by forking, and this leaks more file
		734	descriptors than exec'ing them, as there is no way to mark descriptors as
		735	"close on fork".
		736
		737	An example would be modules like L<EV>, L<IO::AIO> or L<Gtk2>. Both create
		738	pipes for internal uses, and L<Gtk2> might open a connection to the X
		739	server. L<EV> and L<IO::AIO> can deal with fork, but Gtk2 might have
		740	trouble with a fork.
		741
		742	The solution is to either not load these modules before use'ing
		743	L<AnyEvent::Fork::Early> or L<AnyEvent::Fork::Template>, or to delay
		744	initialising them, for example, by calling C<init Gtk2> manually.
		745
		746	=back
		747
435	=head1 PORTABILITY NOTES	748	=head1 PORTABILITY NOTES
436		749
437	Win32 is a loser - code has been written for this platform, pain has been	750	Native win32 perls are somewhat supported (AnyEvent::Fork::Early is a nop,
438	felt, but in the end, this platform is just too broken - maybe a later	751	and ::Template is not going to work), and it cost a lot of blood and sweat
439	version can do it.	752	to make it so, mostly due to the bloody broken perl that nobody seems to
		753	care about. The fork emulation is a bad joke - I have yet to see something
		754	useful that you can do with it without running into memory corruption
		755	issues or other braindamage. Hrrrr.
		756
		757	Cygwin perl is not supported at the moment, as it should implement fd
		758	passing, but doesn't, and rolling my own is hard, as cygwin doesn't
		759	support enough functionality to do it.
		760
		761	=head1 SEE ALSO
		762
		763	L<AnyEvent::Fork::Early> (to avoid executing a perl interpreter),
		764	L<AnyEvent::Fork::Template> (to create a process by forking the main
		765	program at a convenient time).
440		766
441	=head1 AUTHOR	767	=head1 AUTHOR
442		768
443	Marc Lehmann <schmorp@schmorp.de>	769	Marc Lehmann <schmorp@schmorp.de>
444	http://home.schmorp.de/	770	http://home.schmorp.de/

Diff Legend

-–
+Removed lines
-+
+Added lines
-<
+Changed lines
->
+Changed lines

Comparing AnyEvent-Fork/Fork.pm (file contents): Revision 1.8 by root, Thu Apr 4 01:54:40 2013 UTC vs. Revision 1.18 by root, Sat Apr 6 01:33:56 2013 UTC

Diff Legend

Comparing AnyEvent-Fork/Fork.pm (file contents):
Revision 1.8 by root, Thu Apr 4 01:54:40 2013 UTC vs.
Revision 1.18 by root, Sat Apr 6 01:33:56 2013 UTC