[ViewVC] Diff of: cvs/AnyEvent-Fork-RPC/RPC.pm

Comparing AnyEvent-Fork-RPC/RPC.pm (file contents):
Revision 1.4 by root, Wed Apr 17 19:38:25 2013 UTC vs.
Revision 1.34 by root, Wed Nov 20 15:26:56 2013 UTC

…		…
12	->require ("MyModule")	12	->require ("MyModule")
13	->AnyEvent::Fork::RPC::run (	13	->AnyEvent::Fork::RPC::run (
14	"MyModule::server",	14	"MyModule::server",
15	);	15	);
16		16
		17	use AnyEvent;
		18
17	my $cv = AE::cv;	19	my $cv = AE::cv;
18		20
19	$rpc->(1, 2, 3, sub {	21	$rpc->(1, 2, 3, sub {
20	print "MyModule::server returned @_\n";	22	print "MyModule::server returned @_\n";
21	$cv->send;	23	$cv->send;
…		…
24	$cv->recv;	26	$cv->recv;
25		27
26	=head1 DESCRIPTION	28	=head1 DESCRIPTION
27		29
28	This module implements a simple RPC protocol and backend for processes	30	This module implements a simple RPC protocol and backend for processes
29	created via L<AnyEvent::Fork>, allowing you to call a function in the	31	created via L<AnyEvent::Fork> or L<AnyEvent::Fork::Remote>, allowing you
30	child process and receive its return values (up to 4GB serialised).	32	to call a function in the child process and receive its return values (up
		33	to 4GB serialised).
31		34
32	It implements two different backends: a synchronous one that works like a	35	It implements two different backends: a synchronous one that works like a
33	normal function call, and an asynchronous one that can run multiple jobs	36	normal function call, and an asynchronous one that can run multiple jobs
34	concurrently in the child, using AnyEvent.	37	concurrently in the child, using AnyEvent.
35		38
36	It also implements an asynchronous event mechanism from the child to the	39	It also implements an asynchronous event mechanism from the child to the
37	parent, that could be used for progress indications or other information.	40	parent, that could be used for progress indications or other information.
38		41
39	=head1 EXAMPLES	42	=head1 EXAMPLES
40		43
41	=head2 Synchronous Backend	44	=head2 Example 1: Synchronous Backend
42		45
43	Here is a simple example that implements a backend that executes C<unlink>	46	Here is a simple example that implements a backend that executes C<unlink>
44	and C<rmdir> calls, and reports their status back. It also reports the	47	and C<rmdir> calls, and reports their status back. It also reports the
45	number of requests it has processed every three requests, which is clearly	48	number of requests it has processed every three requests, which is clearly
46	silly, but illustrates the use of events.	49	silly, but illustrates the use of events.
…		…
55		58
56	my $rpc = AnyEvent::Fork	59	my $rpc = AnyEvent::Fork
57	->new	60	->new
58	->require ("MyWorker")	61	->require ("MyWorker")
59	->AnyEvent::Fork::RPC::run ("MyWorker::run",	62	->AnyEvent::Fork::RPC::run ("MyWorker::run",
		63	on_error => sub { warn "ERROR: $_[0]"; exit 1 },
60	on_event => sub { warn "$_[0] requests handled\n" },	64	on_event => sub { warn "$_[0] requests handled\n" },
61	on_destroy => $done,	65	on_destroy => $done,
62	);	66	);
63		67
64	for my $id (1..6) {	68	for my $id (1..6) {
…		…
102	dies with a fatal error - obviously, you must never let this happen :).	106	dies with a fatal error - obviously, you must never let this happen :).
103		107
104	Eventually it returns the status value true if the command was successful,	108	Eventually it returns the status value true if the command was successful,
105	or the status value 0 and the stringified error message.	109	or the status value 0 and the stringified error message.
106		110
107	On my system, running the first cdoe fragment with the given	111	On my system, running the first code fragment with the given
108	F<MyWorker.pm> in the current directory yields:	112	F<MyWorker.pm> in the current directory yields:
109		113
110	/tmp/somepath/1: No such file or directory	114	/tmp/somepath/1: No such file or directory
111	/tmp/somepath/2: No such file or directory	115	/tmp/somepath/2: No such file or directory
112	3 requests handled	116	3 requests handled
…		…
133		137
134	And as a final remark, there is a fine module on CPAN that can	138	And as a final remark, there is a fine module on CPAN that can
135	asynchronously C<rmdir> and C<unlink> and a lot more, and more efficiently	139	asynchronously C<rmdir> and C<unlink> and a lot more, and more efficiently
136	than this example, namely L<IO::AIO>.	140	than this example, namely L<IO::AIO>.
137		141
		142	=head3 Example 1a: the same with the asynchronous backend
		143
		144	This example only shows what needs to be changed to use the async backend
		145	instead. Doing this is not very useful, the purpose of this example is
		146	to show the minimum amount of change that is required to go from the
		147	synchronous to the asynchronous backend.
		148
		149	To use the async backend in the previous example, you need to add the
		150	C<async> parameter to the C<AnyEvent::Fork::RPC::run> call:
		151
		152	->AnyEvent::Fork::RPC::run ("MyWorker::run",
		153	async => 1,
		154	...
		155
		156	And since the function call protocol is now changed, you need to adopt
		157	C<MyWorker::run> to the async API.
		158
		159	First, you need to accept the extra initial C<$done> callback:
		160
		161	sub run {
		162	my ($done, $cmd, $path) = @_;
		163
		164	And since a response is now generated when C<$done> is called, as opposed
		165	to when the function returns, we need to call the C<$done> function with
		166	the status:
		167
		168	$done->($status or (0, "$!"));
		169
		170	A few remarks are in order. First, it's quite pointless to use the async
		171	backend for this example - but it I<is> possible. Second, you can call
		172	C<$done> before or after returning from the function. Third, having both
		173	returned from the function and having called the C<$done> callback, the
		174	child process may exit at any time, so you should call C<$done> only when
		175	you really I<are> done.
		176
		177	=head2 Example 2: Asynchronous Backend
		178
		179	This example implements multiple count-downs in the child, using
		180	L<AnyEvent> timers. While this is a bit silly (one could use timers in the
		181	parent just as well), it illustrates the ability to use AnyEvent in the
		182	child and the fact that responses can arrive in a different order then the
		183	requests.
		184
		185	It also shows how to embed the actual child code into a C<__DATA__>
		186	section, so it doesn't need any external files at all.
		187
		188	And when your parent process is often busy, and you have stricter timing
		189	requirements, then running timers in a child process suddenly doesn't look
		190	so silly anymore.
		191
		192	Without further ado, here is the code:
		193
		194	use AnyEvent;
		195	use AnyEvent::Fork;
		196	use AnyEvent::Fork::RPC;
		197
		198	my $done = AE::cv;
		199
		200	my $rpc = AnyEvent::Fork
		201	->new
		202	->require ("AnyEvent::Fork::RPC::Async")
		203	->eval (do { local $/; <DATA> })
		204	->AnyEvent::Fork::RPC::run ("run",
		205	async => 1,
		206	on_error => sub { warn "ERROR: $_[0]"; exit 1 },
		207	on_event => sub { print $_[0] },
		208	on_destroy => $done,
		209	);
		210
		211	for my $count (3, 2, 1) {
		212	$rpc->($count, sub {
		213	warn "job $count finished\n";
		214	});
		215	}
		216
		217	undef $rpc;
		218
		219	$done->recv;
		220
		221	__DATA__
		222
		223	# this ends up in main, as we don't use a package declaration
		224
		225	use AnyEvent;
		226
		227	sub run {
		228	my ($done, $count) = @_;
		229
		230	my $n;
		231
		232	AnyEvent::Fork::RPC::event "starting to count up to $count\n";
		233
		234	my $w; $w = AE::timer 1, 1, sub {
		235	++$n;
		236
		237	AnyEvent::Fork::RPC::event "count $n of $count\n";
		238
		239	if ($n == $count) {
		240	undef $w;
		241	$done->();
		242	}
		243	};
		244	}
		245
		246	The parent part (the one before the C<__DATA__> section) isn't very
		247	different from the earlier examples. It sets async mode, preloads
		248	the backend module (so the C<AnyEvent::Fork::RPC::event> function is
		249	declared), uses a slightly different C<on_event> handler (which we use
		250	simply for logging purposes) and then, instead of loading a module with
		251	the actual worker code, it C<eval>'s the code from the data section in the
		252	child process.
		253
		254	It then starts three countdowns, from 3 to 1 seconds downwards, destroys
		255	the rpc object so the example finishes eventually, and then just waits for
		256	the stuff to trickle in.
		257
		258	The worker code uses the event function to log some progress messages, but
		259	mostly just creates a recurring one-second timer.
		260
		261	The timer callback increments a counter, logs a message, and eventually,
		262	when the count has been reached, calls the finish callback.
		263
		264	On my system, this results in the following output. Since all timers fire
		265	at roughly the same time, the actual order isn't guaranteed, but the order
		266	shown is very likely what you would get, too.
		267
		268	starting to count up to 3
		269	starting to count up to 2
		270	starting to count up to 1
		271	count 1 of 3
		272	count 1 of 2
		273	count 1 of 1
		274	job 1 finished
		275	count 2 of 2
		276	job 2 finished
		277	count 2 of 3
		278	count 3 of 3
		279	job 3 finished
		280
		281	While the overall ordering isn't guaranteed, the async backend still
		282	guarantees that events and responses are delivered to the parent process
		283	in the exact same ordering as they were generated in the child process.
		284
		285	And unless your system is I<very> busy, it should clearly show that the
		286	job started last will finish first, as it has the lowest count.
		287
		288	This concludes the async example. Since L<AnyEvent::Fork> does not
		289	actually fork, you are free to use about any module in the child, not just
		290	L<AnyEvent>, but also L<IO::AIO>, or L<Tk> for example.
		291
		292	=head2 Example 3: Asynchronous backend with Coro
		293
		294	With L<Coro> you can create a nice asynchronous backend implementation by
		295	defining an rpc server function that creates a new Coro thread for every
		296	request that calls a function "normally", i.e. the parameters from the
		297	parent process are passed to it, and any return values are returned to the
		298	parent process, e.g.:
		299
		300	package My::Arith;
		301
		302	sub add {
		303	return $_[0] + $_[1];
		304	}
		305
		306	sub mul {
		307	return $_[0] * $_[1];
		308	}
		309
		310	sub run {
		311	my ($done, $func, @arg) = @_;
		312
		313	Coro::async_pool {
		314	$done->($func->(@arg));
		315	};
		316	}
		317
		318	The C<run> function creates a new thread for every invocation, using the
		319	first argument as function name, and calls the C<$done> callback on it's
		320	return values. This makes it quite natural to define the C<add> and C<mul>
		321	functions to add or multiply two numbers and return the result.
		322
		323	Since this is the asynchronous backend, it's quite possible to define RPC
		324	function that do I/O or wait for external events - their execution will
		325	overlap as needed.
		326
		327	The above could be used like this:
		328
		329	my $rpc = AnyEvent::Fork
		330	->new
		331	->require ("MyWorker")
		332	->AnyEvent::Fork::RPC::run ("My::Arith::run",
		333	on_error => ..., on_event => ..., on_destroy => ...,
		334	);
		335
		336	$rpc->(add => 1, 3, Coro::rouse_cb); say Coro::rouse_wait;
		337	$rpc->(mul => 3, 2, Coro::rouse_cb); say Coro::rouse_wait;
		338
		339	The C<say>'s will print C<4> and C<6>.
		340
		341	=head2 Example 4: Forward AnyEvent::Log messages using C<on_event>
		342
		343	This partial example shows how to use the C<event> function to forward
		344	L<AnyEvent::Log> messages to the parent.
		345
		346	For this, the parent needs to provide a suitable C<on_event>:
		347
		348	->AnyEvent::Fork::RPC::run (
		349	on_event => sub {
		350	if ($_[0] eq "ae_log") {
		351	my (undef, $level, $message) = @_;
		352	AE::log $level, $message;
		353	} else {
		354	# other event types
		355	}
		356	},
		357	)
		358
		359	In the child, as early as possible, the following code should reconfigure
		360	L<AnyEvent::Log> to log via C<AnyEvent::Fork::RPC::event>:
		361
		362	$AnyEvent::Log::LOG->log_cb (sub {
		363	my ($timestamp, $orig_ctx, $level, $message) = @{+shift};
		364
		365	if (defined &AnyEvent::Fork::RPC::event) {
		366	AnyEvent::Fork::RPC::event (ae_log => $level, $message);
		367	} else {
		368	warn "[$$ before init] $message\n";
		369	}
		370	});
		371
		372	There is an important twist - the C<AnyEvent::Fork::RPC::event> function
		373	is only defined when the child is fully initialised. If you redirect the
		374	log messages in your C<init> function for example, then the C<event>
		375	function might not yet be available. This is why the log callback checks
		376	whether the fucntion is there using C<defined>, and only then uses it to
		377	log the message.
		378
138	=head1 PARENT PROCESS USAGE	379	=head1 PARENT PROCESS USAGE
139		380
140	This module exports nothing, and only implements a single function:	381	This module exports nothing, and only implements a single function:
141		382
142	=over 4	383	=over 4
…		…
149		390
150	use Errno ();	391	use Errno ();
151	use Guard ();	392	use Guard ();
152		393
153	use AnyEvent;	394	use AnyEvent;
154	#use AnyEvent::Fork;
155		395
156	our $VERSION = 0.1;	396	our $VERSION = 1.21;
157		397
158	=item my $rpc = AnyEvent::Fork::RPC::run $fork, $function, [key => value...]	398	=item my $rpc = AnyEvent::Fork::RPC::run $fork, $function, [key => value...]
159		399
160	The traditional way to call it. But it is way cooler to call it in the	400	The traditional way to call it. But it is way cooler to call it in the
161	following way:	401	following way:
…		…
181	Called on (fatal) errors, with a descriptive (hopefully) message. If	421	Called on (fatal) errors, with a descriptive (hopefully) message. If
182	this callback is not provided, but C<on_event> is, then the C<on_event>	422	this callback is not provided, but C<on_event> is, then the C<on_event>
183	callback is called with the first argument being the string C<error>,	423	callback is called with the first argument being the string C<error>,
184	followed by the error message.	424	followed by the error message.
185		425
186	If neither handler is provided it prints the error to STDERR and will	426	If neither handler is provided, then the error is reported with loglevel
187	start failing badly.	427	C<error> via C<AE::log>.
188		428
189	=item on_event => $cb->(...)	429	=item on_event => $cb->(...)
190		430
191	Called for every call to the C<AnyEvent::Fork::RPC::event> function in the	431	Called for every call to the C<AnyEvent::Fork::RPC::event> function in the
192	child, with the arguments of that function passed to the callback.	432	child, with the arguments of that function passed to the callback.
…		…
197		437
198	Called when the C<$rpc> object has been destroyed and all requests have	438	Called when the C<$rpc> object has been destroyed and all requests have
199	been successfully handled. This is useful when you queue some requests and	439	been successfully handled. This is useful when you queue some requests and
200	want the child to go away after it has handled them. The problem is that	440	want the child to go away after it has handled them. The problem is that
201	the parent must not exit either until all requests have been handled, and	441	the parent must not exit either until all requests have been handled, and
202	this cna be accomplished by waiting for this callback.	442	this can be accomplished by waiting for this callback.
203		443
204	=item init => $function (default none)	444	=item init => $function (default none)
205		445
206	When specified (by name), this function is called in the child as the very	446	When specified (by name), this function is called in the child as the very
207	first thing when taking over the process, with all the arguments normally	447	first thing when taking over the process, with all the arguments normally
…		…
214	It is called very early - before the serialisers are created or the	454	It is called very early - before the serialisers are created or the
215	C<$function> name is resolved into a function reference, so it could be	455	C<$function> name is resolved into a function reference, so it could be
216	used to load any modules that provide the serialiser or function. It can	456	used to load any modules that provide the serialiser or function. It can
217	not, however, create events.	457	not, however, create events.
218		458
		459	=item done => $function (default C<CORE::exit>)
		460
		461	The function to call when the asynchronous backend detects an end of file
		462	condition when reading from the communications socket I<and> there are no
		463	outstanding requests. It's ignored by the synchronous backend.
		464
		465	By overriding this you can prolong the life of a RPC process after e.g.
		466	the parent has exited by running the event loop in the provided function
		467	(or simply calling it, for example, when your child process uses L<EV> you
		468	could provide L<EV::loop> as C<done> function).
		469
		470	Of course, in that case you are responsible for exiting at the appropriate
		471	time and not returning from
		472
219	=item async => $boolean (default: 0)	473	=item async => $boolean (default: 0)
220		474
221	The default server used in the child does all I/O blockingly, and only	475	The default server used in the child does all I/O blockingly, and only
222	allows a single RPC call to execute concurrently.	476	allows a single RPC call to execute concurrently.
223		477
224	Setting C<async> to a true value switches to another implementation that	478	Setting C<async> to a true value switches to another implementation that
225	uses L<AnyEvent> in the child and allows multiple concurrent RPC calls.	479	uses L<AnyEvent> in the child and allows multiple concurrent RPC calls (it
		480	does not support recursion in the event loop however, blocking condvar
		481	calls will fail).
226		482
227	The actual API in the child is documented in the section that describes	483	The actual API in the child is documented in the section that describes
228	the calling semantics of the returned C<$rpc> function.	484	the calling semantics of the returned C<$rpc> function.
229		485
230	If you want to pre-load the actual back-end modules to enable memory	486	If you want to pre-load the actual back-end modules to enable memory
231	sharing, then you should load C<AnyEvent::Fork::RPC::Sync> for	487	sharing, then you should load C<AnyEvent::Fork::RPC::Sync> for
232	synchronous, and C<AnyEvent::Fork::RPC::Async> for asynchronous mode.	488	synchronous, and C<AnyEvent::Fork::RPC::Async> for asynchronous mode.
233		489
234	If you use a template process and want to fork both sync and async	490	If you use a template process and want to fork both sync and async
235	children, then it is permissible to laod both modules.	491	children, then it is permissible to load both modules.
236		492
237	=item serialiser => $string (default: '(sub { pack "(w/a)", @_ }, sub { unpack "(w/a)", shift })')	493	=item serialiser => $string (default: $AnyEvent::Fork::RPC::STRING_SERIALISER)
238		494
239	All arguments, result data and event data have to be serialised to be	495	All arguments, result data and event data have to be serialised to be
240	transferred between the processes. For this, they have to be frozen and	496	transferred between the processes. For this, they have to be frozen and
241	thawed in both parent and child processes.	497	thawed in both parent and child processes.
242		498
243	By default, only octet strings can be passed between the processes, which	499	By default, only octet strings can be passed between the processes, which
244	is reasonably fast and efficient.	500	is reasonably fast and efficient and requires no extra modules.
245		501
246	For more complicated use cases, you can provide your own freeze and thaw	502	For more complicated use cases, you can provide your own freeze and thaw
247	functions, by specifying a string with perl source code. It's supposed to	503	functions, by specifying a string with perl source code. It's supposed to
248	return two code references when evaluated: the first receives a list of	504	return two code references when evaluated: the first receives a list of
249	perl values and must return an octet string. The second receives the octet	505	perl values and must return an octet string. The second receives the octet
…		…
251		507
252	If you need an external module for serialisation, then you can either	508	If you need an external module for serialisation, then you can either
253	pre-load it into your L<AnyEvent::Fork> process, or you can add a C<use>	509	pre-load it into your L<AnyEvent::Fork> process, or you can add a C<use>
254	or C<require> statement into the serialiser string. Or both.	510	or C<require> statement into the serialiser string. Or both.
255		511
		512	Here are some examples - some of them are also available as global
		513	variables that make them easier to use.
		514
		515	=over 4
		516
		517	=item octet strings - C<$AnyEvent::Fork::RPC::STRING_SERIALISER>
		518
		519	This serialiser concatenates length-prefixes octet strings, and is the
		520	default. That means you can only pass (and return) strings containing
		521	character codes 0-255.
		522
		523	Implementation:
		524
		525	(
		526	sub { pack "(w/a)", @_ },
		527	sub { unpack "(w/a)", shift }
		528	)
		529
		530	=item json - C<$AnyEvent::Fork::RPC::JSON_SERIALISER>
		531
		532	This serialiser creates JSON arrays - you have to make sure the L<JSON>
		533	module is installed for this serialiser to work. It can be beneficial for
		534	sharing when you preload the L<JSON> module in a template process.
		535
		536	L<JSON> (with L<JSON::XS> installed) is slower than the octet string
		537	serialiser, but usually much faster than L<Storable>, unless big chunks of
		538	binary data need to be transferred.
		539
		540	Implementation:
		541
		542	use JSON ();
		543	(
		544	sub { JSON::encode_json \@_ },
		545	sub { @{ JSON::decode_json shift } }
		546	)
		547
		548	=item storable - C<$AnyEvent::Fork::RPC::STORABLE_SERIALISER>
		549
		550	This serialiser uses L<Storable>, which means it has high chance of
		551	serialising just about anything you throw at it, at the cost of having
		552	very high overhead per operation. It also comes with perl. It should be
		553	used when you need to serialise complex data structures.
		554
		555	Implementation:
		556
		557	use Storable ();
		558	(
		559	sub { Storable::freeze \@_ },
		560	sub { @{ Storable::thaw shift } }
		561	)
		562
		563	=item portable storable - C<$AnyEvent::Fork::RPC::NSTORABLE_SERIALISER>
		564
		565	This serialiser also uses L<Storable>, but uses it's "network" format
		566	to serialise data, which makes it possible to talk to different
		567	perl binaries (for example, when talking to a process created with
		568	L<AnyEvent::Fork::Remote>).
		569
		570	Implementation:
		571
		572	use Storable ();
		573	(
		574	sub { Storable::nfreeze \@_ },
		575	sub { @{ Storable::thaw shift } }
		576	)
		577
256	=back	578	=back
257		579
		580	=back
		581
		582	See the examples section earlier in this document for some actual
		583	examples.
		584
258	=cut	585	=cut
259		586
260	our $STRING_SERIALISER = '(sub { pack "(w/a)", @_ }, sub { unpack "(w/a)", shift })';	587	our $STRING_SERIALISER = '(sub { pack "(w/a)", @_ }, sub { unpack "(w/a)", shift })';
		588	our $JSON_SERIALISER = 'use JSON (); (sub { JSON::encode_json \@_ }, sub { @{ JSON::decode_json shift } })';
		589	our $STORABLE_SERIALISER = 'use Storable (); (sub { Storable::freeze \@_ }, sub { @{ Storable::thaw shift } })';
		590	our $NSTORABLE_SERIALISER = 'use Storable (); (sub { Storable::nfreeze \@_ }, sub { @{ Storable::thaw shift } })';
261		591
262	sub run {	592	sub run {
263	my ($self, $function, %arg) = @_;	593	my ($self, $function, %arg) = @_;
264		594
265	my $serialiser = delete $arg{serialiser} \|\| $STRING_SERIALISER;	595	my $serialiser = delete $arg{serialiser} \|\| $STRING_SERIALISER;
…		…
268	my $on_destroy = delete $arg{on_destroy};	598	my $on_destroy = delete $arg{on_destroy};
269		599
270	# default for on_error is to on_event, if specified	600	# default for on_error is to on_event, if specified
271	$on_error \|\|= $on_event	601	$on_error \|\|= $on_event
272	? sub { $on_event->(error => shift) }	602	? sub { $on_event->(error => shift) }
273	: sub { die "AnyEvent::Fork::RPC: uncaught error: $_[0].\n" };	603	: sub { AE::log die => "AnyEvent::Fork::RPC: uncaught error: $_[0]." };
274		604
275	# default for on_event is to raise an error	605	# default for on_event is to raise an error
276	$on_event \|\|= sub { $on_error->("event received, but no on_event handler") };	606	$on_event \|\|= sub { $on_error->("event received, but no on_event handler") };
277		607
278	my ($f, $t) = eval $serialiser; die $@ if $@;	608	my ($f, $t) = eval $serialiser; die $@ if $@;
279		609
280	my (@rcb, $fh, $shutdown, $wbuf, $ww, $rw);	610	my (@rcb, %rcb, $fh, $shutdown, $wbuf, $ww);
281	my ($rlen, $rbuf) = 512 - 16;	611	my ($rlen, $rbuf, $rw) = 512 - 16;
282		612
283	my $wcb = sub {	613	my $wcb = sub {
284	my $len = syswrite $fh, $wbuf;	614	my $len = syswrite $fh, $wbuf;
285		615
286	if (!defined $len) {	616	unless (defined $len) {
287	if ($! != Errno::EAGAIN && $! != Errno::EWOULDBLOCK) {	617	if ($! != Errno::EAGAIN && $! != Errno::EWOULDBLOCK) {
288	undef $rw; undef $ww; # it ends here	618	undef $rw; undef $ww; # it ends here
289	$on_error->("$!");	619	$on_error->("$!");
290	}	620	}
291	}	621	}
…		…
299	};	629	};
300		630
301	my $module = "AnyEvent::Fork::RPC::" . ($arg{async} ? "Async" : "Sync");	631	my $module = "AnyEvent::Fork::RPC::" . ($arg{async} ? "Async" : "Sync");
302		632
303	$self->require ($module)	633	$self->require ($module)
304	->send_arg ($function, $arg{init}, $serialiser)	634	->send_arg ($function, $arg{init}, $serialiser, $arg{done} \|\| "$module\::do_exit")
305	->run ("$module\::run", sub {	635	->run ("$module\::run", sub {
306	$fh = shift;	636	$fh = shift;
		637
		638	my ($id, $len);
307	$rw = AE::io $fh, 0, sub {	639	$rw = AE::io $fh, 0, sub {
308	$rlen = $rlen * 2 + 16 if $rlen - 128 < length $rbuf;	640	$rlen = $rlen * 2 + 16 if $rlen - 128 < length $rbuf;
309	my $len = sysread $fh, $rbuf, $rlen - length $rbuf, length $rbuf;	641	$len = sysread $fh, $rbuf, $rlen - length $rbuf, length $rbuf;
310		642
311	if ($len) {	643	if ($len) {
312	while (5 <= length $rbuf) {	644	while (8 <= length $rbuf) {
313	$len = unpack "L", $rbuf;	645	($id, $len) = unpack "NN", $rbuf;
314	4 + $len <= length $rbuf	646	8 + $len <= length $rbuf
315	or last;	647	or last;
316		648
317	my @r = $t->(substr $rbuf, 4, $len);	649	my @r = $t->(substr $rbuf, 8, $len);
318	substr $rbuf, 0, $len + 4, "";	650	substr $rbuf, 0, 8 + $len, "";
		651
		652	if ($id) {
		653	if (@rcb) {
		654	(shift @rcb)->(@r);
		655	} elsif (my $cb = delete $rcb{$id}) {
		656	$cb->(@r);
		657	} else {
		658	undef $rw; undef $ww;
		659	$on_error->("unexpected data from child");
319		660	}
320	if (pop @r) {	661	} else {
321	$on_event->(@r);	662	$on_event->(@r);
322	} elsif (@rcb) {
323	(shift @rcb)->(@r);
324	} else {
325	undef $rw; undef $ww;
326	$on_error->("unexpected data from child");
327	}	663	}
328	}	664	}
329	} elsif (defined $len) {	665	} elsif (defined $len) {
330	undef $rw; undef $ww; # it ends here	666	undef $rw; undef $ww; # it ends here
331		667
332	if (@rcb) {	668	if (@rcb \|\| %rcb) {
333	$on_error->("unexpected eof");	669	$on_error->("unexpected eof");
334	} else {	670	} else {
335	$on_destroy->();	671	$on_destroy->()
		672	if $on_destroy;
336	}	673	}
337	} elsif ($! != Errno::EAGAIN && $! != Errno::EWOULDBLOCK) {	674	} elsif ($! != Errno::EAGAIN && $! != Errno::EWOULDBLOCK) {
338	undef $rw; undef $ww; # it ends here	675	undef $rw; undef $ww; # it ends here
339	$on_error->("read: $!");	676	$on_error->("read: $!");
340	}	677	}
…		…
343	$ww \|\|= AE::io $fh, 1, $wcb;	680	$ww \|\|= AE::io $fh, 1, $wcb;
344	});	681	});
345		682
346	my $guard = Guard::guard {	683	my $guard = Guard::guard {
347	$shutdown = 1;	684	$shutdown = 1;
348	$ww \|\|= $fh && AE::io $fh, 1, $wcb;	685
		686	shutdown $fh, 1 if $fh && !$ww;
349	};	687	};
350		688
		689	my $id;
		690
		691	$arg{async}
351	sub {	692	? sub {
352	push @rcb, pop;	693	$id = ($id == 0xffffffff ? 0 : $id) + 1;
		694	$id = ($id == 0xffffffff ? 0 : $id) + 1 while exists $rcb{$id}; # rarely loops
353		695
		696	$rcb{$id} = pop;
		697
354	$guard; # keep it alive	698	$guard if 0; # keep it alive
355		699
356	$wbuf .= pack "L/a*", &$f;	700	$wbuf .= pack "NN/a*", $id, &$f;
357	$ww \|\|= $fh && AE::io $fh, 1, $wcb;	701	$ww \|\|= $fh && AE::io $fh, 1, $wcb;
358	}	702	}
		703	: sub {
		704	push @rcb, pop;
		705
		706	$guard; # keep it alive
		707
		708	$wbuf .= pack "N/a*", &$f;
		709	$ww \|\|= $fh && AE::io $fh, 1, $wcb;
		710	}
359	}	711	}
360		712
361	=item $rpc->(..., $cb->(...))	713	=item $rpc->(..., $cb->(...))
362		714
363	The RPC object returned by C<AnyEvent::Fork::RPC::run> is actually a code	715	The RPC object returned by C<AnyEvent::Fork::RPC::run> is actually a code
…		…
378		730
379	The other thing that can be done with the RPC object is to destroy it. In	731	The other thing that can be done with the RPC object is to destroy it. In
380	this case, the child process will execute all remaining RPC calls, report	732	this case, the child process will execute all remaining RPC calls, report
381	their results, and then exit.	733	their results, and then exit.
382		734
		735	See the examples section earlier in this document for some actual
		736	examples.
		737
383	=back	738	=back
384		739
385	=head1 CHILD PROCESS USAGE	740	=head1 CHILD PROCESS USAGE
386		741
387	The following function is not available in this module. They are only	742	The following function is not available in this module. They are only
…		…
395		750
396	Send an event to the parent. Events are a bit like RPC calls made by the	751	Send an event to the parent. Events are a bit like RPC calls made by the
397	child process to the parent, except that there is no notion of return	752	child process to the parent, except that there is no notion of return
398	values.	753	values.
399		754
		755	See the examples section earlier in this document for some actual
		756	examples.
		757
400	=back	758	=back
401		759
		760	=head2 PROCESS EXIT
		761
		762	If and when the child process exits depends on the backend and
		763	configuration. Apart from explicit exits (e.g. by calling C<exit>) or
		764	runtime conditions (uncaught exceptions, signals etc.), the backends exit
		765	under these conditions:
		766
		767	=over 4
		768
		769	=item Synchronous Backend
		770
		771	The synchronous backend is very simple: when the process waits for another
		772	request to arrive and the writing side (usually in the parent) is closed,
		773	it will exit normally, i.e. as if your main program reached the end of the
		774	file.
		775
		776	That means that if your parent process exits, the RPC process will usually
		777	exit as well, either because it is idle anyway, or because it executes a
		778	request. In the latter case, you will likely get an error when the RPc
		779	process tries to send the results to the parent (because agruably, you
		780	shouldn't exit your parent while there are still outstanding requests).
		781
		782	The process is usually quiescent when it happens, so it should rarely be a
		783	problem, and C<END> handlers can be used to clean up.
		784
		785	=item Asynchronous Backend
		786
		787	For the asynchronous backend, things are more complicated: Whenever it
		788	listens for another request by the parent, it might detect that the socket
		789	was closed (e.g. because the parent exited). It will sotp listening for
		790	new requests and instead try to write out any remaining data (if any) or
		791	simply check whether the socket can be written to. After this, the RPC
		792	process is effectively done - no new requests are incoming, no outstanding
		793	request data can be written back.
		794
		795	Since chances are high that there are event watchers that the RPC server
		796	knows nothing about (why else would one use the async backend if not for
		797	the ability to register watchers?), the event loop would often happily
		798	continue.
		799
		800	This is why the asynchronous backend explicitly calls C<CORE::exit> when
		801	it is done (under other circumstances, such as when there is an I/O error
		802	and there is outstanding data to write, it will log a fatal message via
		803	L<AnyEvent::Log>, also causing the program to exit).
		804
		805	You can override this by specifying a function name to call via the C<done>
		806	parameter instead.
		807
		808	=back
		809
		810	=head1 ADVANCED TOPICS
		811
		812	=head2 Choosing a backend
		813
		814	So how do you decide which backend to use? Well, that's your problem to
		815	solve, but here are some thoughts on the matter:
		816
		817	=over 4
		818
		819	=item Synchronous
		820
		821	The synchronous backend does not rely on any external modules (well,
		822	except L<common::sense>, which works around a bug in how perl's warning
		823	system works). This keeps the process very small, for example, on my
		824	system, an empty perl interpreter uses 1492kB RSS, which becomes 2020kB
		825	after C<use warnings; use strict> (for people who grew up with C64s around
		826	them this is probably shocking every single time they see it). The worker
		827	process in the first example in this document uses 1792kB.
		828
		829	Since the calls are done synchronously, slow jobs will keep newer jobs
		830	from executing.
		831
		832	The synchronous backend also has no overhead due to running an event loop
		833	- reading requests is therefore very efficient, while writing responses is
		834	less so, as every response results in a write syscall.
		835
		836	If the parent process is busy and a bit slow reading responses, the child
		837	waits instead of processing further requests. This also limits the amount
		838	of memory needed for buffering, as never more than one response has to be
		839	buffered.
		840
		841	The API in the child is simple - you just have to define a function that
		842	does something and returns something.
		843
		844	It's hard to use modules or code that relies on an event loop, as the
		845	child cannot execute anything while it waits for more input.
		846
		847	=item Asynchronous
		848
		849	The asynchronous backend relies on L<AnyEvent>, which tries to be small,
		850	but still comes at a price: On my system, the worker from example 1a uses
		851	3420kB RSS (for L<AnyEvent>, which loads L<EV>, which needs L<XSLoader>
		852	which in turn loads a lot of other modules such as L<warnings>, L<strict>,
		853	L<vars>, L<Exporter>...).
		854
		855	It batches requests and responses reasonably efficiently, doing only as
		856	few reads and writes as needed, but needs to poll for events via the event
		857	loop.
		858
		859	Responses are queued when the parent process is busy. This means the child
		860	can continue to execute any queued requests. It also means that a child
		861	might queue a lot of responses in memory when it generates them and the
		862	parent process is slow accepting them.
		863
		864	The API is not a straightforward RPC pattern - you have to call a
		865	"done" callback to pass return values and signal completion. Also, more
		866	importantly, the API starts jobs as fast as possible - when 1000 jobs
		867	are queued and the jobs are slow, they will all run concurrently. The
		868	child must implement some queueing/limiting mechanism if this causes
		869	problems. Alternatively, the parent could limit the amount of rpc calls
		870	that are outstanding.
		871
		872	Blocking use of condvars is not supported.
		873
		874	Using event-based modules such as L<IO::AIO>, L<Gtk2>, L<Tk> and so on is
		875	easy.
		876
		877	=back
		878
		879	=head2 Passing file descriptors
		880
		881	Unlike L<AnyEvent::Fork>, this module has no in-built file handle or file
		882	descriptor passing abilities.
		883
		884	The reason is that passing file descriptors is extraordinary tricky
		885	business, and conflicts with efficient batching of messages.
		886
		887	There still is a method you can use: Create a
		888	C<AnyEvent::Util::portable_socketpair> and C<send_fh> one half of it to
		889	the process before you pass control to C<AnyEvent::Fork::RPC::run>.
		890
		891	Whenever you want to pass a file descriptor, send an rpc request to the
		892	child process (so it expects the descriptor), then send it over the other
		893	half of the socketpair. The child should fetch the descriptor from the
		894	half it has passed earlier.
		895
		896	Here is some (untested) pseudocode to that effect:
		897
		898	use AnyEvent::Util;
		899	use AnyEvent::Fork;
		900	use AnyEvent::Fork::RPC;
		901	use IO::FDPass;
		902
		903	my ($s1, $s2) = AnyEvent::Util::portable_socketpair;
		904
		905	my $rpc = AnyEvent::Fork
		906	->new
		907	->send_fh ($s2)
		908	->require ("MyWorker")
		909	->AnyEvent::Fork::RPC::run ("MyWorker::run"
		910	init => "MyWorker::init",
		911	);
		912
		913	undef $s2; # no need to keep it around
		914
		915	# pass an fd
		916	$rpc->("i'll send some fd now, please expect it!", my $cv = AE::cv);
		917
		918	IO::FDPass fileno $s1, fileno $handle_to_pass;
		919
		920	$cv->recv;
		921
		922	The MyWorker module could look like this:
		923
		924	package MyWorker;
		925
		926	use IO::FDPass;
		927
		928	my $s2;
		929
		930	sub init {
		931	$s2 = $_[0];
		932	}
		933
		934	sub run {
		935	if ($_[0] eq "i'll send some fd now, please expect it!") {
		936	my $fd = IO::FDPass::recv fileno $s2;
		937	...
		938	}
		939	}
		940
		941	Of course, this might be blocking if you pass a lot of file descriptors,
		942	so you might want to look into L<AnyEvent::FDpasser> which can handle the
		943	gory details.
		944
		945	=head1 EXCEPTIONS
		946
		947	There are no provisions whatsoever for catching exceptions at this time -
		948	in the child, exeptions might kill the process, causing calls to be lost
		949	and the parent encountering a fatal error. In the parent, exceptions in
		950	the result callback will not be caught and cause undefined behaviour.
		951
402	=head1 SEE ALSO	952	=head1 SEE ALSO
403		953
404	L<AnyEvent::Fork> (to create the processes in the first place),	954	L<AnyEvent::Fork>, to create the processes in the first place.
		955
		956	L<AnyEvent::Fork::Remote>, likewise, but helpful for remote processes.
		957
405	L<AnyEvent::Fork::Pool> (to manage whole pools of processes).	958	L<AnyEvent::Fork::Pool>, to manage whole pools of processes.
406		959
407	=head1 AUTHOR AND CONTACT INFORMATION	960	=head1 AUTHOR AND CONTACT INFORMATION
408		961
409	Marc Lehmann <schmorp@schmorp.de>	962	Marc Lehmann <schmorp@schmorp.de>
410	http://software.schmorp.de/pkg/AnyEvent-Fork-RPC	963	http://software.schmorp.de/pkg/AnyEvent-Fork-RPC

Diff Legend

-–
+Removed lines
-+
+Added lines
-<
+Changed lines
->
+Changed lines

Comparing AnyEvent-Fork-RPC/RPC.pm (file contents): Revision 1.4 by root, Wed Apr 17 19:38:25 2013 UTC vs. Revision 1.34 by root, Wed Nov 20 15:26:56 2013 UTC

Diff Legend

Comparing AnyEvent-Fork-RPC/RPC.pm (file contents):
Revision 1.4 by root, Wed Apr 17 19:38:25 2013 UTC vs.
Revision 1.34 by root, Wed Nov 20 15:26:56 2013 UTC