[ViewVC] Diff of: cvs/AnyEvent-Fork-RPC/RPC.pm

Comparing AnyEvent-Fork-RPC/RPC.pm (file contents):
Revision 1.2 by root, Wed Apr 17 17:08:16 2013 UTC vs.
Revision 1.30 by root, Sun Aug 25 22:21:15 2013 UTC

…		…
12	->require ("MyModule")	12	->require ("MyModule")
13	->AnyEvent::Fork::RPC::run (	13	->AnyEvent::Fork::RPC::run (
14	"MyModule::server",	14	"MyModule::server",
15	);	15	);
16		16
		17	use AnyEvent;
		18
17	my $cv = AE::cv;	19	my $cv = AE::cv;
18		20
19	$rpc->(1, 2, 3, sub {	21	$rpc->(1, 2, 3, sub {
20	print "MyModule::server returned @_\n";	22	print "MyModule::server returned @_\n";
21	$cv->send;	23	$cv->send;
…		…
24	$cv->recv;	26	$cv->recv;
25		27
26	=head1 DESCRIPTION	28	=head1 DESCRIPTION
27		29
28	This module implements a simple RPC protocol and backend for processes	30	This module implements a simple RPC protocol and backend for processes
29	created via L<AnyEvent::Fork>, allowing you to call a function in the	31	created via L<AnyEvent::Fork> or L<AnyEvent::Fork::Remote>, allowing you
30	child process and receive its return values (up to 4GB serialised).	32	to call a function in the child process and receive its return values (up
		33	to 4GB serialised).
31		34
32	It implements two different backends: a synchronous one that works like a	35	It implements two different backends: a synchronous one that works like a
33	normal function call, and an asynchronous one that can run multiple jobs	36	normal function call, and an asynchronous one that can run multiple jobs
34	concurrently in the child, using AnyEvent.	37	concurrently in the child, using AnyEvent.
35		38
36	It also implements an asynchronous event mechanism from the child to the	39	It also implements an asynchronous event mechanism from the child to the
37	parent, that could be used for progress indications or other information.	40	parent, that could be used for progress indications or other information.
38		41
		42	=head1 EXAMPLES
		43
		44	=head2 Example 1: Synchronous Backend
		45
		46	Here is a simple example that implements a backend that executes C<unlink>
		47	and C<rmdir> calls, and reports their status back. It also reports the
		48	number of requests it has processed every three requests, which is clearly
		49	silly, but illustrates the use of events.
		50
		51	First the parent process:
		52
		53	use AnyEvent;
		54	use AnyEvent::Fork;
		55	use AnyEvent::Fork::RPC;
		56
		57	my $done = AE::cv;
		58
		59	my $rpc = AnyEvent::Fork
		60	->new
		61	->require ("MyWorker")
		62	->AnyEvent::Fork::RPC::run ("MyWorker::run",
		63	on_error => sub { warn "ERROR: $_[0]"; exit 1 },
		64	on_event => sub { warn "$_[0] requests handled\n" },
		65	on_destroy => $done,
		66	);
		67
		68	for my $id (1..6) {
		69	$rpc->(rmdir => "/tmp/somepath/$id", sub {
		70	$_[0]
		71	or warn "/tmp/somepath/$id: $_[1]\n";
		72	});
		73	}
		74
		75	undef $rpc;
		76
		77	$done->recv;
		78
		79	The parent creates the process, queues a few rmdir's. It then forgets
		80	about the C<$rpc> object, so that the child exits after it has handled the
		81	requests, and then it waits till the requests have been handled.
		82
		83	The child is implemented using a separate module, C<MyWorker>, shown here:
		84
		85	package MyWorker;
		86
		87	my $count;
		88
		89	sub run {
		90	my ($cmd, $path) = @_;
		91
		92	AnyEvent::Fork::RPC::event ($count)
		93	unless ++$count % 3;
		94
		95	my $status = $cmd eq "rmdir" ? rmdir $path
		96	: $cmd eq "unlink" ? unlink $path
		97	: die "fatal error, illegal command '$cmd'";
		98
		99	$status or (0, "$!")
		100	}
		101
		102	1
		103
		104	The C<run> function first sends a "progress" event every three calls, and
		105	then executes C<rmdir> or C<unlink>, depending on the first parameter (or
		106	dies with a fatal error - obviously, you must never let this happen :).
		107
		108	Eventually it returns the status value true if the command was successful,
		109	or the status value 0 and the stringified error message.
		110
		111	On my system, running the first code fragment with the given
		112	F<MyWorker.pm> in the current directory yields:
		113
		114	/tmp/somepath/1: No such file or directory
		115	/tmp/somepath/2: No such file or directory
		116	3 requests handled
		117	/tmp/somepath/3: No such file or directory
		118	/tmp/somepath/4: No such file or directory
		119	/tmp/somepath/5: No such file or directory
		120	6 requests handled
		121	/tmp/somepath/6: No such file or directory
		122
		123	Obviously, none of the directories I am trying to delete even exist. Also,
		124	the events and responses are processed in exactly the same order as
		125	they were created in the child, which is true for both synchronous and
		126	asynchronous backends.
		127
		128	Note that the parentheses in the call to C<AnyEvent::Fork::RPC::event> are
		129	not optional. That is because the function isn't defined when the code is
		130	compiled. You can make sure it is visible by pre-loading the correct
		131	backend module in the call to C<require>:
		132
		133	->require ("AnyEvent::Fork::RPC::Sync", "MyWorker")
		134
		135	Since the backend module declares the C<event> function, loading it first
		136	ensures that perl will correctly interpret calls to it.
		137
		138	And as a final remark, there is a fine module on CPAN that can
		139	asynchronously C<rmdir> and C<unlink> and a lot more, and more efficiently
		140	than this example, namely L<IO::AIO>.
		141
		142	=head3 Example 1a: the same with the asynchronous backend
		143
		144	This example only shows what needs to be changed to use the async backend
		145	instead. Doing this is not very useful, the purpose of this example is
		146	to show the minimum amount of change that is required to go from the
		147	synchronous to the asynchronous backend.
		148
		149	To use the async backend in the previous example, you need to add the
		150	C<async> parameter to the C<AnyEvent::Fork::RPC::run> call:
		151
		152	->AnyEvent::Fork::RPC::run ("MyWorker::run",
		153	async => 1,
		154	...
		155
		156	And since the function call protocol is now changed, you need to adopt
		157	C<MyWorker::run> to the async API.
		158
		159	First, you need to accept the extra initial C<$done> callback:
		160
		161	sub run {
		162	my ($done, $cmd, $path) = @_;
		163
		164	And since a response is now generated when C<$done> is called, as opposed
		165	to when the function returns, we need to call the C<$done> function with
		166	the status:
		167
		168	$done->($status or (0, "$!"));
		169
		170	A few remarks are in order. First, it's quite pointless to use the async
		171	backend for this example - but it I<is> possible. Second, you can call
		172	C<$done> before or after returning from the function. Third, having both
		173	returned from the function and having called the C<$done> callback, the
		174	child process may exit at any time, so you should call C<$done> only when
		175	you really I<are> done.
		176
		177	=head2 Example 2: Asynchronous Backend
		178
		179	This example implements multiple count-downs in the child, using
		180	L<AnyEvent> timers. While this is a bit silly (one could use timers in te
		181	parent just as well), it illustrates the ability to use AnyEvent in the
		182	child and the fact that responses can arrive in a different order then the
		183	requests.
		184
		185	It also shows how to embed the actual child code into a C<__DATA__>
		186	section, so it doesn't need any external files at all.
		187
		188	And when your parent process is often busy, and you have stricter timing
		189	requirements, then running timers in a child process suddenly doesn't look
		190	so silly anymore.
		191
		192	Without further ado, here is the code:
		193
		194	use AnyEvent;
		195	use AnyEvent::Fork;
		196	use AnyEvent::Fork::RPC;
		197
		198	my $done = AE::cv;
		199
		200	my $rpc = AnyEvent::Fork
		201	->new
		202	->require ("AnyEvent::Fork::RPC::Async")
		203	->eval (do { local $/; <DATA> })
		204	->AnyEvent::Fork::RPC::run ("run",
		205	async => 1,
		206	on_error => sub { warn "ERROR: $_[0]"; exit 1 },
		207	on_event => sub { print $_[0] },
		208	on_destroy => $done,
		209	);
		210
		211	for my $count (3, 2, 1) {
		212	$rpc->($count, sub {
		213	warn "job $count finished\n";
		214	});
		215	}
		216
		217	undef $rpc;
		218
		219	$done->recv;
		220
		221	__DATA__
		222
		223	# this ends up in main, as we don't use a package declaration
		224
		225	use AnyEvent;
		226
		227	sub run {
		228	my ($done, $count) = @_;
		229
		230	my $n;
		231
		232	AnyEvent::Fork::RPC::event "starting to count up to $count\n";
		233
		234	my $w; $w = AE::timer 1, 1, sub {
		235	++$n;
		236
		237	AnyEvent::Fork::RPC::event "count $n of $count\n";
		238
		239	if ($n == $count) {
		240	undef $w;
		241	$done->();
		242	}
		243	};
		244	}
		245
		246	The parent part (the one before the C<__DATA__> section) isn't very
		247	different from the earlier examples. It sets async mode, preloads
		248	the backend module (so the C<AnyEvent::Fork::RPC::event> function is
		249	declared), uses a slightly different C<on_event> handler (which we use
		250	simply for logging purposes) and then, instead of loading a module with
		251	the actual worker code, it C<eval>'s the code from the data section in the
		252	child process.
		253
		254	It then starts three countdowns, from 3 to 1 seconds downwards, destroys
		255	the rpc object so the example finishes eventually, and then just waits for
		256	the stuff to trickle in.
		257
		258	The worker code uses the event function to log some progress messages, but
		259	mostly just creates a recurring one-second timer.
		260
		261	The timer callback increments a counter, logs a message, and eventually,
		262	when the count has been reached, calls the finish callback.
		263
		264	On my system, this results in the following output. Since all timers fire
		265	at roughly the same time, the actual order isn't guaranteed, but the order
		266	shown is very likely what you would get, too.
		267
		268	starting to count up to 3
		269	starting to count up to 2
		270	starting to count up to 1
		271	count 1 of 3
		272	count 1 of 2
		273	count 1 of 1
		274	job 1 finished
		275	count 2 of 2
		276	job 2 finished
		277	count 2 of 3
		278	count 3 of 3
		279	job 3 finished
		280
		281	While the overall ordering isn't guaranteed, the async backend still
		282	guarantees that events and responses are delivered to the parent process
		283	in the exact same ordering as they were generated in the child process.
		284
		285	And unless your system is I<very> busy, it should clearly show that the
		286	job started last will finish first, as it has the lowest count.
		287
		288	This concludes the async example. Since L<AnyEvent::Fork> does not
		289	actually fork, you are free to use about any module in the child, not just
		290	L<AnyEvent>, but also L<IO::AIO>, or L<Tk> for example.
		291
		292	=head2 Example 3: Asynchronous backend with Coro
		293
		294	With L<Coro> you can create a nice asynchronous backend implementation by
		295	defining an rpc server function that creates a new Coro thread for every
		296	request that calls a function "normally", i.e. the parameters from the
		297	parent process are passed to it, and any return values are returned to the
		298	parent process, e.g.:
		299
		300	package My::Arith;
		301
		302	sub add {
		303	return $_[0] + $_[1];
		304	}
		305
		306	sub mul {
		307	return $_[0] * $_[1];
		308	}
		309
		310	sub run {
		311	my ($done, $func, @arg) = @_;
		312
		313	Coro::async_pool {
		314	$done->($func->(@arg));
		315	};
		316	}
		317
		318	The C<run> function creates a new thread for every invocation, using the
		319	first argument as function name, and calls the C<$done> callback on it's
		320	return values. This makes it quite natural to define the C<add> and C<mul>
		321	functions to add or multiply two numbers and return the result.
		322
		323	Since this is the asynchronous backend, it's quite possible to define RPC
		324	function that do I/O or wait for external events - their execution will
		325	overlap as needed.
		326
		327	The above could be used like this:
		328
		329	my $rpc = AnyEvent::Fork
		330	->new
		331	->require ("MyWorker")
		332	->AnyEvent::Fork::RPC::run ("My::Arith::run",
		333	on_error => ..., on_event => ..., on_destroy => ...,
		334	);
		335
		336	$rpc->(add => 1, 3, Coro::rouse_cb); say Coro::rouse_wait;
		337	$rpc->(mul => 3, 2, Coro::rouse_cb); say Coro::rouse_wait;
		338
		339	The C<say>'s will print C<4> and C<6>.
		340
		341	=head2 Example 4: Forward AnyEvent::Log messages using C<on_event>
		342
		343	This partial example shows how to use the C<event> function to forward
		344	L<AnyEvent::Log> messages to the parent.
		345
		346	For this, the parent needs to provide a suitable C<on_event>:
		347
		348	->AnyEvent::Fork::RPC::run (
		349	on_event => sub {
		350	if ($_[0] eq "ae_log") {
		351	my (undef, $level, $message) = @_;
		352	AE::log $level, $message;
		353	} else {
		354	# other event types
		355	}
		356	},
		357	)
		358
		359	In the child, as early as possible, the following code should reconfigure
		360	L<AnyEvent::Log> to log via C<AnyEvent::Fork::RPC::event>:
		361
		362	$AnyEvent::Log::LOG->log_cb (sub {
		363	my ($timestamp, $orig_ctx, $level, $message) = @{+shift};
		364
		365	if (defined &AnyEvent::Fork::RPC::event) {
		366	AnyEvent::Fork::RPC::event (ae_log => $level, $message);
		367	} else {
		368	warn "[$$ before init] $message\n";
		369	}
		370	});
		371
		372	There is an important twist - the C<AnyEvent::Fork::RPC::event> function
		373	is only defined when the child is fully initialised. If you redirect the
		374	log messages in your C<init> function for example, then the C<event>
		375	function might not yet be available. This is why the log callback checks
		376	whether the fucntion is there using C<defined>, and only then uses it to
		377	log the message.
		378
39	=head1 PARENT PROCESS USAGE	379	=head1 PARENT PROCESS USAGE
40		380
41	This module exports nothing, and only implements a single function:	381	This module exports nothing, and only implements a single function:
42		382
43	=over 4	383	=over 4
…		…
50		390
51	use Errno ();	391	use Errno ();
52	use Guard ();	392	use Guard ();
53		393
54	use AnyEvent;	394	use AnyEvent;
55	#use AnyEvent::Fork;
56		395
57	our $VERSION = 0.1;	396	our $VERSION = 1.1;
58		397
59	=item my $rpc = AnyEvent::Fork::RPC::run $fork, $function, [key => value...]	398	=item my $rpc = AnyEvent::Fork::RPC::run $fork, $function, [key => value...]
60		399
61	The traditional way to call it. But it is way cooler to call it in the	400	The traditional way to call it. But it is way cooler to call it in the
62	following way:	401	following way:
…		…
82	Called on (fatal) errors, with a descriptive (hopefully) message. If	421	Called on (fatal) errors, with a descriptive (hopefully) message. If
83	this callback is not provided, but C<on_event> is, then the C<on_event>	422	this callback is not provided, but C<on_event> is, then the C<on_event>
84	callback is called with the first argument being the string C<error>,	423	callback is called with the first argument being the string C<error>,
85	followed by the error message.	424	followed by the error message.
86		425
87	If neither handler is provided it prints the error to STDERR and will	426	If neither handler is provided, then the error is reported with loglevel
88	start failing badly.	427	C<error> via C<AE::log>.
89		428
90	=item on_event => $cb->(...)	429	=item on_event => $cb->(...)
91		430
92	Called for every call to the C<AnyEvent::Fork::RPC::event> function in the	431	Called for every call to the C<AnyEvent::Fork::RPC::event> function in the
93	child, with the arguments of that function passed to the callback.	432	child, with the arguments of that function passed to the callback.
94		433
95	Also called on errors when no C<on_error> handler is provided.	434	Also called on errors when no C<on_error> handler is provided.
		435
		436	=item on_destroy => $cb->()
		437
		438	Called when the C<$rpc> object has been destroyed and all requests have
		439	been successfully handled. This is useful when you queue some requests and
		440	want the child to go away after it has handled them. The problem is that
		441	the parent must not exit either until all requests have been handled, and
		442	this can be accomplished by waiting for this callback.
96		443
97	=item init => $function (default none)	444	=item init => $function (default none)
98		445
99	When specified (by name), this function is called in the child as the very	446	When specified (by name), this function is called in the child as the very
100	first thing when taking over the process, with all the arguments normally	447	first thing when taking over the process, with all the arguments normally
…		…
102	socket.	449	socket.
103		450
104	It can be used to do one-time things in the child such as storing passed	451	It can be used to do one-time things in the child such as storing passed
105	parameters or opening database connections.	452	parameters or opening database connections.
106		453
		454	It is called very early - before the serialisers are created or the
		455	C<$function> name is resolved into a function reference, so it could be
		456	used to load any modules that provide the serialiser or function. It can
		457	not, however, create events.
		458
107	=item async => $boolean (default: 0)	459	=item async => $boolean (default: 0)
108		460
109	The default server used in the child does all I/O blockingly, and only	461	The default server used in the child does all I/O blockingly, and only
110	allows a single RPC call to execute concurrently.	462	allows a single RPC call to execute concurrently.
111		463
112	Setting C<async> to a true value switches to another implementation that	464	Setting C<async> to a true value switches to another implementation that
113	uses L<AnyEvent> in the child and allows multiple concurrent RPC calls.	465	uses L<AnyEvent> in the child and allows multiple concurrent RPC calls (it
		466	does not support recursion in the event loop however, blocking condvar
		467	calls will fail).
114		468
115	The actual API in the child is documented in the section that describes	469	The actual API in the child is documented in the section that describes
116	the calling semantics of the returned C<$rpc> function.	470	the calling semantics of the returned C<$rpc> function.
117		471
118	If you want to pre-load the actual back-end modules to enable memory	472	If you want to pre-load the actual back-end modules to enable memory
119	sharing, then you should load C<AnyEvent::Fork::RPC::Sync> for	473	sharing, then you should load C<AnyEvent::Fork::RPC::Sync> for
120	synchronous, and C<AnyEvent::Fork::RPC::Async> for asynchronous mode.	474	synchronous, and C<AnyEvent::Fork::RPC::Async> for asynchronous mode.
121		475
122	=item serialiser => $string (default: '(sub { pack "(w/a)", @_ }, sub { unpack "(w/a)", shift })')	476	If you use a template process and want to fork both sync and async
		477	children, then it is permissible to load both modules.
		478
		479	=item serialiser => $string (default: $AnyEvent::Fork::RPC::STRING_SERIALISER)
123		480
124	All arguments, result data and event data have to be serialised to be	481	All arguments, result data and event data have to be serialised to be
125	transferred between the processes. For this, they have to be frozen and	482	transferred between the processes. For this, they have to be frozen and
126	thawed in both parent and child processes.	483	thawed in both parent and child processes.
127		484
128	By default, only octet strings can be passed between the processes, which	485	By default, only octet strings can be passed between the processes, which
129	is reasonably fast and efficient.	486	is reasonably fast and efficient and requires no extra modules.
130		487
131	For more complicated use cases, you can provide your own freeze and thaw	488	For more complicated use cases, you can provide your own freeze and thaw
132	functions, by specifying a string with perl source code. It's supposed to	489	functions, by specifying a string with perl source code. It's supposed to
133	return two code references when evaluated: the first receives a list of	490	return two code references when evaluated: the first receives a list of
134	perl values and must return an octet string. The second receives the octet	491	perl values and must return an octet string. The second receives the octet
…		…
136		493
137	If you need an external module for serialisation, then you can either	494	If you need an external module for serialisation, then you can either
138	pre-load it into your L<AnyEvent::Fork> process, or you can add a C<use>	495	pre-load it into your L<AnyEvent::Fork> process, or you can add a C<use>
139	or C<require> statement into the serialiser string. Or both.	496	or C<require> statement into the serialiser string. Or both.
140		497
		498	Here are some examples - some of them are also available as global
		499	variables that make them easier to use.
		500
		501	=over 4
		502
		503	=item octet strings - C<$AnyEvent::Fork::RPC::STRING_SERIALISER>
		504
		505	This serialiser concatenates length-prefixes octet strings, and is the
		506	default. That means you can only pass (and return) strings containing
		507	character codes 0-255.
		508
		509	Implementation:
		510
		511	(
		512	sub { pack "(w/a)", @_ },
		513	sub { unpack "(w/a)", shift }
		514	)
		515
		516	=item json - C<$AnyEvent::Fork::RPC::JSON_SERIALISER>
		517
		518	This serialiser creates JSON arrays - you have to make sure the L<JSON>
		519	module is installed for this serialiser to work. It can be beneficial for
		520	sharing when you preload the L<JSON> module in a template process.
		521
		522	L<JSON> (with L<JSON::XS> installed) is slower than the octet string
		523	serialiser, but usually much faster than L<Storable>, unless big chunks of
		524	binary data need to be transferred.
		525
		526	Implementation:
		527
		528	use JSON ();
		529	(
		530	sub { JSON::encode_json \@_ },
		531	sub { @{ JSON::decode_json shift } }
		532	)
		533
		534	=item storable - C<$AnyEvent::Fork::RPC::STORABLE_SERIALISER>
		535
		536	This serialiser uses L<Storable>, which means it has high chance of
		537	serialising just about anything you throw at it, at the cost of having
		538	very high overhead per operation. It also comes with perl. It should be
		539	used when you need to serialise complex data structures.
		540
		541	Implementation:
		542
		543	use Storable ();
		544	(
		545	sub { Storable::freeze \@_ },
		546	sub { @{ Storable::thaw shift } }
		547	)
		548
		549	=item portable storable - C<$AnyEvent::Fork::RPC::NSTORABLE_SERIALISER>
		550
		551	This serialiser also uses L<Storable>, but uses it's "network" format
		552	to serialise data, which makes it possible to talk to different
		553	perl binaries (for example, when talking to a process created with
		554	L<AnyEvent::Fork::Remote>).
		555
		556	Implementation:
		557
		558	use Storable ();
		559	(
		560	sub { Storable::nfreeze \@_ },
		561	sub { @{ Storable::thaw shift } }
		562	)
		563
141	=back	564	=back
142		565
		566	=back
		567
		568	See the examples section earlier in this document for some actual
		569	examples.
		570
143	=cut	571	=cut
144		572
145	our $STRING_SERIALISER = '(sub { pack "(w/a)", @_ }, sub { unpack "(w/a)", shift })';	573	our $STRING_SERIALISER = '(sub { pack "(w/a)", @_ }, sub { unpack "(w/a)", shift })';
146		574	our $JSON_SERIALISER = 'use JSON (); (sub { JSON::encode_json \@_ }, sub { @{ JSON::decode_json shift } })';
147	# ideally, we want (SvLEN - SvCUR) \|\| 1024 or somesuch...	575	our $STORABLE_SERIALISER = 'use Storable (); (sub { Storable::freeze \@_ }, sub { @{ Storable::thaw shift } })';
148	sub rlen($) { ($_[0] < 384 ? 512 + 16 : 2 << int +(log $_[0] + 512) / log 2) - $_[0] - 16 }	576	our $NSTORABLE_SERIALISER = 'use Storable (); (sub { Storable::nfreeze \@_ }, sub { @{ Storable::thaw shift } })';
149		577
150	sub run {	578	sub run {
151	my ($self, $function, %arg) = @_;	579	my ($self, $function, %arg) = @_;
152		580
153	my $serialiser = delete $arg{serialiser} \|\| $STRING_SERIALISER;	581	my $serialiser = delete $arg{serialiser} \|\| $STRING_SERIALISER;
154	my $on_event = delete $arg{on_event};	582	my $on_event = delete $arg{on_event};
155	my $on_error = delete $arg{on_error};	583	my $on_error = delete $arg{on_error};
		584	my $on_destroy = delete $arg{on_destroy};
156		585
157	# default for on_error is to on_event, if specified	586	# default for on_error is to on_event, if specified
158	$on_error \|\|= $on_event	587	$on_error \|\|= $on_event
159	? sub { $on_event->(error => shift) }	588	? sub { $on_event->(error => shift) }
160	: sub { die "AnyEvent::Fork::RPC: uncaught error: $_[0].\n" };	589	: sub { AE::log die => "AnyEvent::Fork::RPC: uncaught error: $_[0]." };
161		590
162	# default for on_event is to raise an error	591	# default for on_event is to raise an error
163	$on_event \|\|= sub { $on_error->("event received, but no on_event handler") };	592	$on_event \|\|= sub { $on_error->("event received, but no on_event handler") };
164		593
165	my ($f, $t) = eval $serialiser; die $@ if $@;	594	my ($f, $t) = eval $serialiser; die $@ if $@;
166		595
167	my (@rcb, $fh, $shutdown, $wbuf, $ww, $rbuf, $rw);	596	my (@rcb, %rcb, $fh, $shutdown, $wbuf, $ww);
		597	my ($rlen, $rbuf, $rw) = 512 - 16;
168		598
169	my $wcb = sub {	599	my $wcb = sub {
170	my $len = syswrite $fh, $wbuf;	600	my $len = syswrite $fh, $wbuf;
171		601
172	if (!defined $len) {	602	unless (defined $len) {
173	if ($! != Errno::EAGAIN && $! != Errno::EWOULDBLOCK) {	603	if ($! != Errno::EAGAIN && $! != Errno::EWOULDBLOCK) {
174	undef $rw; undef $ww; # it ends here	604	undef $rw; undef $ww; # it ends here
175	$on_error->("$!");	605	$on_error->("$!");
176	}	606	}
177	}	607	}
…		…
188		618
189	$self->require ($module)	619	$self->require ($module)
190	->send_arg ($function, $arg{init}, $serialiser)	620	->send_arg ($function, $arg{init}, $serialiser)
191	->run ("$module\::run", sub {	621	->run ("$module\::run", sub {
192	$fh = shift;	622	$fh = shift;
		623
		624	my ($id, $len);
193	$rw = AE::io $fh, 0, sub {	625	$rw = AE::io $fh, 0, sub {
		626	$rlen = $rlen * 2 + 16 if $rlen - 128 < length $rbuf;
194	my $len = sysread $fh, $rbuf, rlen length $rbuf, length $rbuf;	627	$len = sysread $fh, $rbuf, $rlen - length $rbuf, length $rbuf;
195		628
196	if ($len) {	629	if ($len) {
197	while (5 <= length $rbuf) {	630	while (8 <= length $rbuf) {
198	$len = unpack "L", $rbuf;	631	($id, $len) = unpack "NN", $rbuf;
199	4 + $len <= length $rbuf	632	8 + $len <= length $rbuf
200	or last;	633	or last;
201		634
202	my @r = $t->(substr $rbuf, 4, $len);	635	my @r = $t->(substr $rbuf, 8, $len);
203	substr $rbuf, 0, $len + 4, "";	636	substr $rbuf, 0, 8 + $len, "";
		637
		638	if ($id) {
		639	if (@rcb) {
		640	(shift @rcb)->(@r);
		641	} elsif (my $cb = delete $rcb{$id}) {
		642	$cb->(@r);
		643	} else {
		644	undef $rw; undef $ww;
		645	$on_error->("unexpected data from child");
204		646	}
205	if (pop @r) {	647	} else {
206	$on_event->(@r);	648	$on_event->(@r);
207	} elsif (@rcb) {
208	(shift @rcb)->(@r);
209	} else {
210	undef $rw; undef $ww;
211	$on_error->("unexpected data from child");
212	}	649	}
213	}	650	}
214	} elsif (defined $len) {	651	} elsif (defined $len) {
215	undef $rw; undef $ww; # it ends here	652	undef $rw; undef $ww; # it ends here
		653
		654	if (@rcb \|\| %rcb) {
216	$on_error->("unexpected eof")	655	$on_error->("unexpected eof");
217	if @rcb;	656	} else {
		657	$on_destroy->()
		658	if $on_destroy;
		659	}
218	} elsif ($! != Errno::EAGAIN && $! != Errno::EWOULDBLOCK) {	660	} elsif ($! != Errno::EAGAIN && $! != Errno::EWOULDBLOCK) {
219	undef $rw; undef $ww; # it ends here	661	undef $rw; undef $ww; # it ends here
220	$on_error->("read: $!");	662	$on_error->("read: $!");
221	}	663	}
222	};	664	};
…		…
224	$ww \|\|= AE::io $fh, 1, $wcb;	666	$ww \|\|= AE::io $fh, 1, $wcb;
225	});	667	});
226		668
227	my $guard = Guard::guard {	669	my $guard = Guard::guard {
228	$shutdown = 1;	670	$shutdown = 1;
229	$ww \|\|= $fh && AE::io $fh, 1, $wcb;	671
		672	shutdown $fh, 1 if $fh && !$ww;
230	};	673	};
231		674
		675	my $id;
		676
		677	$arg{async}
232	sub {	678	? sub {
233	push @rcb, pop;	679	$id = ($id == 0xffffffff ? 0 : $id) + 1;
		680	$id = ($id == 0xffffffff ? 0 : $id) + 1 while exists $rcb{$id}; # rarely loops
234		681
		682	$rcb{$id} = pop;
		683
235	$guard; # keep it alive	684	$guard if 0; # keep it alive
236		685
237	$wbuf .= pack "L/a*", &$f;	686	$wbuf .= pack "NN/a*", $id, &$f;
238	$ww \|\|= $fh && AE::io $fh, 1, $wcb;	687	$ww \|\|= $fh && AE::io $fh, 1, $wcb;
239	}	688	}
		689	: sub {
		690	push @rcb, pop;
		691
		692	$guard; # keep it alive
		693
		694	$wbuf .= pack "N/a*", &$f;
		695	$ww \|\|= $fh && AE::io $fh, 1, $wcb;
		696	}
240	}	697	}
241		698
		699	=item $rpc->(..., $cb->(...))
		700
		701	The RPC object returned by C<AnyEvent::Fork::RPC::run> is actually a code
		702	reference. There are two things you can do with it: call it, and let it go
		703	out of scope (let it get destroyed).
		704
		705	If C<async> was false when C<$rpc> was created (the default), then, if you
		706	call C<$rpc>, the C<$function> is invoked with all arguments passed to
		707	C<$rpc> except the last one (the callback). When the function returns, the
		708	callback will be invoked with all the return values.
		709
		710	If C<async> was true, then the C<$function> receives an additional
		711	initial argument, the result callback. In this case, returning from
		712	C<$function> does nothing - the function only counts as "done" when the
		713	result callback is called, and any arguments passed to it are considered
		714	the return values. This makes it possible to "return" from event handlers
		715	or e.g. Coro threads.
		716
		717	The other thing that can be done with the RPC object is to destroy it. In
		718	this case, the child process will execute all remaining RPC calls, report
		719	their results, and then exit.
		720
		721	See the examples section earlier in this document for some actual
		722	examples.
		723
242	=back	724	=back
243		725
244	=head1 CHILD PROCESS USAGE	726	=head1 CHILD PROCESS USAGE
245		727
246	These functions are not available in this module. They are only available	728	The following function is not available in this module. They are only
247	in the namespace of this module when the child is running, without	729	available in the namespace of this module when the child is running,
248	having to load any extra module. They are part of the child-side API of	730	without having to load any extra modules. They are part of the child-side
249	L<AnyEvent::Fork::RPC>.	731	API of L<AnyEvent::Fork::RPC>.
250		732
251	=over 4	733	=over 4
252
253	=item AnyEvent::Fork::RPC::quit
254
255	This function can be called to gracefully stop the child process when it
256	is idle.
257
258	After this function is called, the process stops handling incoming RPC
259	requests, but outstanding events and function return values will be sent
260	to the parent. When all data has been sent, the process calls C<exit>.
261
262	Since the parent might not expect the child to exit at random points in
263	time, it is often better to signal the parent by sending an C<event> and
264	letting the parent close down the child process.
265		734
266	=item AnyEvent::Fork::RPC::event ...	735	=item AnyEvent::Fork::RPC::event ...
267		736
268	Send an event to the parent. Events are a bit like RPC calls made by the	737	Send an event to the parent. Events are a bit like RPC calls made by the
269	child process to the parent, except that there is no notion of return	738	child process to the parent, except that there is no notion of return
270	values.	739	values.
271		740
		741	See the examples section earlier in this document for some actual
		742	examples.
		743
272	=back	744	=back
273		745
		746	=head1 ADVANCED TOPICS
		747
		748	=head2 Choosing a backend
		749
		750	So how do you decide which backend to use? Well, that's your problem to
		751	solve, but here are some thoughts on the matter:
		752
		753	=over 4
		754
		755	=item Synchronous
		756
		757	The synchronous backend does not rely on any external modules (well,
		758	except L<common::sense>, which works around a bug in how perl's warning
		759	system works). This keeps the process very small, for example, on my
		760	system, an empty perl interpreter uses 1492kB RSS, which becomes 2020kB
		761	after C<use warnings; use strict> (for people who grew up with C64s around
		762	them this is probably shocking every single time they see it). The worker
		763	process in the first example in this document uses 1792kB.
		764
		765	Since the calls are done synchronously, slow jobs will keep newer jobs
		766	from executing.
		767
		768	The synchronous backend also has no overhead due to running an event loop
		769	- reading requests is therefore very efficient, while writing responses is
		770	less so, as every response results in a write syscall.
		771
		772	If the parent process is busy and a bit slow reading responses, the child
		773	waits instead of processing further requests. This also limits the amount
		774	of memory needed for buffering, as never more than one response has to be
		775	buffered.
		776
		777	The API in the child is simple - you just have to define a function that
		778	does something and returns something.
		779
		780	It's hard to use modules or code that relies on an event loop, as the
		781	child cannot execute anything while it waits for more input.
		782
		783	=item Asynchronous
		784
		785	The asynchronous backend relies on L<AnyEvent>, which tries to be small,
		786	but still comes at a price: On my system, the worker from example 1a uses
		787	3420kB RSS (for L<AnyEvent>, which loads L<EV>, which needs L<XSLoader>
		788	which in turn loads a lot of other modules such as L<warnings>, L<strict>,
		789	L<vars>, L<Exporter>...).
		790
		791	It batches requests and responses reasonably efficiently, doing only as
		792	few reads and writes as needed, but needs to poll for events via the event
		793	loop.
		794
		795	Responses are queued when the parent process is busy. This means the child
		796	can continue to execute any queued requests. It also means that a child
		797	might queue a lot of responses in memory when it generates them and the
		798	parent process is slow accepting them.
		799
		800	The API is not a straightforward RPC pattern - you have to call a
		801	"done" callback to pass return values and signal completion. Also, more
		802	importantly, the API starts jobs as fast as possible - when 1000 jobs
		803	are queued and the jobs are slow, they will all run concurrently. The
		804	child must implement some queueing/limiting mechanism if this causes
		805	problems. Alternatively, the parent could limit the amount of rpc calls
		806	that are outstanding.
		807
		808	Blocking use of condvars is not supported.
		809
		810	Using event-based modules such as L<IO::AIO>, L<Gtk2>, L<Tk> and so on is
		811	easy.
		812
		813	=back
		814
		815	=head2 Passing file descriptors
		816
		817	Unlike L<AnyEvent::Fork>, this module has no in-built file handle or file
		818	descriptor passing abilities.
		819
		820	The reason is that passing file descriptors is extraordinary tricky
		821	business, and conflicts with efficient batching of messages.
		822
		823	There still is a method you can use: Create a
		824	C<AnyEvent::Util::portable_socketpair> and C<send_fh> one half of it to
		825	the process before you pass control to C<AnyEvent::Fork::RPC::run>.
		826
		827	Whenever you want to pass a file descriptor, send an rpc request to the
		828	child process (so it expects the descriptor), then send it over the other
		829	half of the socketpair. The child should fetch the descriptor from the
		830	half it has passed earlier.
		831
		832	Here is some (untested) pseudocode to that effect:
		833
		834	use AnyEvent::Util;
		835	use AnyEvent::Fork;
		836	use AnyEvent::Fork::RPC;
		837	use IO::FDPass;
		838
		839	my ($s1, $s2) = AnyEvent::Util::portable_socketpair;
		840
		841	my $rpc = AnyEvent::Fork
		842	->new
		843	->send_fh ($s2)
		844	->require ("MyWorker")
		845	->AnyEvent::Fork::RPC::run ("MyWorker::run"
		846	init => "MyWorker::init",
		847	);
		848
		849	undef $s2; # no need to keep it around
		850
		851	# pass an fd
		852	$rpc->("i'll send some fd now, please expect it!", my $cv = AE::cv);
		853
		854	IO::FDPass fileno $s1, fileno $handle_to_pass;
		855
		856	$cv->recv;
		857
		858	The MyWorker module could look like this:
		859
		860	package MyWorker;
		861
		862	use IO::FDPass;
		863
		864	my $s2;
		865
		866	sub init {
		867	$s2 = $_[0];
		868	}
		869
		870	sub run {
		871	if ($_[0] eq "i'll send some fd now, please expect it!") {
		872	my $fd = IO::FDPass::recv fileno $s2;
		873	...
		874	}
		875	}
		876
		877	Of course, this might be blocking if you pass a lot of file descriptors,
		878	so you might want to look into L<AnyEvent::FDpasser> which can handle the
		879	gory details.
		880
		881	=head1 EXCEPTIONS
		882
		883	There are no provisions whatsoever for catching exceptions at this time -
		884	in the child, exeptions might kill the process, causing calls to be lost
		885	and the parent encountering a fatal error. In the parent, exceptions in
		886	the result callback will not be caught and cause undefined behaviour.
		887
274	=head1 SEE ALSO	888	=head1 SEE ALSO
275		889
276	L<AnyEvent::Fork> (to create the processes in the first place),	890	L<AnyEvent::Fork>, to create the processes in the first place.
		891
		892	L<AnyEvent::Fork::Remote>, likewise, but helpful for remote processes.
		893
277	L<AnyEvent::Fork::Pool> (to manage whole pools of processes).	894	L<AnyEvent::Fork::Pool>, to manage whole pools of processes.
278		895
279	=head1 AUTHOR AND CONTACT INFORMATION	896	=head1 AUTHOR AND CONTACT INFORMATION
280		897
281	Marc Lehmann <schmorp@schmorp.de>	898	Marc Lehmann <schmorp@schmorp.de>
282	http://software.schmorp.de/pkg/AnyEvent-Fork-RPC	899	http://software.schmorp.de/pkg/AnyEvent-Fork-RPC

Diff Legend

-–
+Removed lines
-+
+Added lines
-<
+Changed lines
->
+Changed lines

Comparing AnyEvent-Fork-RPC/RPC.pm (file contents): Revision 1.2 by root, Wed Apr 17 17:08:16 2013 UTC vs. Revision 1.30 by root, Sun Aug 25 22:21:15 2013 UTC

Diff Legend

Comparing AnyEvent-Fork-RPC/RPC.pm (file contents):
Revision 1.2 by root, Wed Apr 17 17:08:16 2013 UTC vs.
Revision 1.30 by root, Sun Aug 25 22:21:15 2013 UTC