--- AnyEvent-Fork-RPC/README 2013/04/21 12:27:03 1.3 +++ AnyEvent-Fork-RPC/README 2013/09/25 11:06:11 1.4 @@ -1,11 +1,9 @@ NAME AnyEvent::Fork::RPC - simple RPC extension for AnyEvent::Fork - THE API IS NOT FINISHED, CONSIDER THIS A BETA RELEASE - SYNOPSIS + use AnyEvent::Fork; use AnyEvent::Fork::RPC; - # use AnyEvent::Fork is not needed my $rpc = AnyEvent::Fork ->new @@ -27,8 +25,9 @@ DESCRIPTION This module implements a simple RPC protocol and backend for processes - created via AnyEvent::Fork, allowing you to call a function in the child - process and receive its return values (up to 4GB serialised). + created via AnyEvent::Fork or AnyEvent::Fork::Remote, allowing you to + call a function in the child process and receive its return values (up + to 4GB serialised). It implements two different backends: a synchronous one that works like a normal function call, and an asynchronous one that can run multiple @@ -38,9 +37,6 @@ parent, that could be used for progress indications or other information. - Loading this module also always loads AnyEvent::Fork, so you can make a - separate "use AnyEvent::Fork" if you wish, but you don't have to. - EXAMPLES Example 1: Synchronous Backend Here is a simple example that implements a backend that executes @@ -51,6 +47,7 @@ First the parent process: use AnyEvent; + use AnyEvent::Fork; use AnyEvent::Fork::RPC; my $done = AE::cv; @@ -59,7 +56,7 @@ ->new ->require ("MyWorker") ->AnyEvent::Fork::RPC::run ("MyWorker::run", - on_error => sub { warn "FATAL: $_[0]"; exit 1 }, + on_error => sub { warn "ERROR: $_[0]"; exit 1 }, on_event => sub { warn "$_[0] requests handled\n" }, on_destroy => $done, ); @@ -190,6 +187,7 @@ Without further ado, here is the code: use AnyEvent; + use AnyEvent::Fork; use AnyEvent::Fork::RPC; my $done = AE::cv; @@ -200,7 +198,7 @@ ->eval (do { local $/; }) ->AnyEvent::Fork::RPC::run ("run", async => 1, - on_error => sub { warn "FATAL: $_[0]"; exit 1 }, + on_error => sub { warn "ERROR: $_[0]"; exit 1 }, on_event => sub { print $_[0] }, on_destroy => $done, ); @@ -286,6 +284,91 @@ fork, you are free to use about any module in the child, not just AnyEvent, but also IO::AIO, or Tk for example. + Example 3: Asynchronous backend with Coro + With Coro you can create a nice asynchronous backend implementation by + defining an rpc server function that creates a new Coro thread for every + request that calls a function "normally", i.e. the parameters from the + parent process are passed to it, and any return values are returned to + the parent process, e.g.: + + package My::Arith; + + sub add { + return $_[0] + $_[1]; + } + + sub mul { + return $_[0] * $_[1]; + } + + sub run { + my ($done, $func, @arg) = @_; + + Coro::async_pool { + $done->($func->(@arg)); + }; + } + + The "run" function creates a new thread for every invocation, using the + first argument as function name, and calls the $done callback on it's + return values. This makes it quite natural to define the "add" and "mul" + functions to add or multiply two numbers and return the result. + + Since this is the asynchronous backend, it's quite possible to define + RPC function that do I/O or wait for external events - their execution + will overlap as needed. + + The above could be used like this: + + my $rpc = AnyEvent::Fork + ->new + ->require ("MyWorker") + ->AnyEvent::Fork::RPC::run ("My::Arith::run", + on_error => ..., on_event => ..., on_destroy => ..., + ); + + $rpc->(add => 1, 3, Coro::rouse_cb); say Coro::rouse_wait; + $rpc->(mul => 3, 2, Coro::rouse_cb); say Coro::rouse_wait; + + The "say"'s will print 4 and 6. + + Example 4: Forward AnyEvent::Log messages using "on_event" + This partial example shows how to use the "event" function to forward + AnyEvent::Log messages to the parent. + + For this, the parent needs to provide a suitable "on_event": + + ->AnyEvent::Fork::RPC::run ( + on_event => sub { + if ($_[0] eq "ae_log") { + my (undef, $level, $message) = @_; + AE::log $level, $message; + } else { + # other event types + } + }, + ) + + In the child, as early as possible, the following code should + reconfigure AnyEvent::Log to log via "AnyEvent::Fork::RPC::event": + + $AnyEvent::Log::LOG->log_cb (sub { + my ($timestamp, $orig_ctx, $level, $message) = @{+shift}; + + if (defined &AnyEvent::Fork::RPC::event) { + AnyEvent::Fork::RPC::event (ae_log => $level, $message); + } else { + warn "[$$ before init] $message\n"; + } + }); + + There is an important twist - the "AnyEvent::Fork::RPC::event" function + is only defined when the child is fully initialised. If you redirect the + log messages in your "init" function for example, then the "event" + function might not yet be available. This is why the log callback checks + whether the fucntion is there using "defined", and only then uses it to + log the message. + PARENT PROCESS USAGE This module exports nothing, and only implements a single function: @@ -313,8 +396,8 @@ then the "on_event" callback is called with the first argument being the string "error", followed by the error message. - If neither handler is provided it prints the error to STDERR and - will start failing badly. + If neither handler is provided, then the error is reported with + loglevel "error" via "AE::log". on_event => $cb->(...) Called for every call to the "AnyEvent::Fork::RPC::event" @@ -345,6 +428,21 @@ could be used to load any modules that provide the serialiser or function. It can not, however, create events. + done => $function (default "CORE::exit") + The function to call when the asynchronous backend detects an + end of file condition when reading from the communications + socket *and* there are no outstanding requests. It's ignored by + the synchronous backend. + + By overriding this you can prolong the life of a RPC process + after e.g. the parent has exited by running the event loop in + the provided function (or simply calling it, for example, when + your child process uses EV you could provide EV::loop as "done" + function). + + Of course, in that case you are responsible for exiting at the + appropriate time and not returning from + async => $boolean (default: 0) The default server used in the child does all I/O blockingly, and only allows a single RPC call to execute concurrently. @@ -392,7 +490,8 @@ octet strings - $AnyEvent::Fork::RPC::STRING_SERIALISER This serialiser concatenates length-prefixes octet strings, - and is the default. + and is the default. That means you can only pass (and + return) strings containing character codes 0-255. Implementation: @@ -423,7 +522,8 @@ This serialiser uses Storable, which means it has high chance of serialising just about anything you throw at it, at the cost of having very high overhead per operation. It - also comes with perl. + also comes with perl. It should be used when you need to + serialise complex data structures. Implementation: @@ -433,6 +533,20 @@ sub { @{ Storable::thaw shift } } ) + portable storable - $AnyEvent::Fork::RPC::NSTORABLE_SERIALISER + This serialiser also uses Storable, but uses it's "network" + format to serialise data, which makes it possible to talk to + different perl binaries (for example, when talking to a + process created with AnyEvent::Fork::Remote). + + Implementation: + + use Storable (); + ( + sub { Storable::nfreeze \@_ }, + sub { @{ Storable::thaw shift } } + ) + See the examples section earlier in this document for some actual examples. @@ -474,6 +588,51 @@ See the examples section earlier in this document for some actual examples. + PROCESS EXIT + If and when the child process exits depends on the backend and + configuration. Apart from explicit exits (e.g. by calling "exit") or + runtime conditions (uncaught exceptions, signals etc.), the backends + exit under these conditions: + + Synchronous Backend + The synchronous backend is very simple: when the process waits for + another request to arrive and the writing side (usually in the + parent) is closed, it will exit normally, i.e. as if your main + program reached the end of the file. + + That means that if your parent process exits, the RPC process will + usually exit as well, either because it is idle anyway, or because + it executes a request. In the latter case, you will likely get an + error when the RPc process tries to send the results to the parent + (because agruably, you shouldn't exit your parent while there are + still outstanding requests). + + The process is usually quiescent when it happens, so it should + rarely be a problem, and "END" handlers can be used to clean up. + + Asynchronous Backend + For the asynchronous backend, things are more complicated: Whenever + it listens for another request by the parent, it might detect that + the socket was closed (e.g. because the parent exited). It will sotp + listening for new requests and instead try to write out any + remaining data (if any) or simply check whether the socket cna be + written to. After this, the RPC process is effectively done - no new + requests are incoming, no outstanding request data can be written + back. + + Since chances are high that there are event watchers that the RPC + server knows nothing about (why else would one use the async backend + if not for the ability to register watchers?), the event loop would + often happily continue. + + This is why the asynchronous backend explicitly calls "CORE::exit" + when it is done (under other circumstances, such as when there is an + I/O error and there is outstanding data to write, it will log a + fatal message via AnyEvent::Log, also causing the program to exit). + + You can override this by specifying a function name to call via the + "done" parameter instead. + ADVANCED TOPICS Choosing a backend So how do you decide which backend to use? Well, that's your problem to @@ -555,6 +714,7 @@ Here is some (untested) pseudocode to that effect: use AnyEvent::Util; + use AnyEvent::Fork; use AnyEvent::Fork::RPC; use IO::FDPass; @@ -610,6 +770,8 @@ SEE ALSO AnyEvent::Fork, to create the processes in the first place. + AnyEvent::Fork::Remote, likewise, but helpful for remote processes. + AnyEvent::Fork::Pool, to manage whole pools of processes. AUTHOR AND CONTACT INFORMATION