ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-MP/MP.pm
(Generate patch)

Comparing AnyEvent-MP/MP.pm (file contents):
Revision 1.124 by root, Sat Mar 3 11:38:43 2012 UTC vs.
Revision 1.143 by root, Fri Mar 23 17:54:36 2012 UTC

35 # destroy a port again 35 # destroy a port again
36 kil $port; # "normal" kill 36 kil $port; # "normal" kill
37 kil $port, my_error => "everything is broken"; # error kill 37 kil $port, my_error => "everything is broken"; # error kill
38 38
39 # monitoring 39 # monitoring
40 mon $localport, $cb->(@msg) # callback is invoked on death 40 mon $port, $cb->(@msg) # callback is invoked on death
41 mon $localport, $otherport # kill otherport on abnormal death 41 mon $port, $localport # kill localport on abnormal death
42 mon $localport, $otherport, @msg # send message on death 42 mon $port, $localport, @msg # send message on death
43 43
44 # temporarily execute code in port context 44 # temporarily execute code in port context
45 peval $port, sub { die "kill the port!" }; 45 peval $port, sub { die "kill the port!" };
46 46
47 # execute callbacks in $SELF port context 47 # execute callbacks in $SELF port context
48 my $timer = AE::timer 1, 0, psub { 48 my $timer = AE::timer 1, 0, psub {
49 die "kill the port, delayed"; 49 die "kill the port, delayed";
50 }; 50 };
51 51
52=head1 CURRENT STATUS 52 # distributed database - modification
53 db_set $family => $subkey [=> $value] # add a subkey
54 db_del $family => $subkey... # delete one or more subkeys
55 db_reg $family => $port [=> $value] # register a port
53 56
54 bin/aemp - stable. 57 # distributed database - queries
55 AnyEvent::MP - stable API, should work. 58 db_family $family => $cb->(\%familyhash)
56 AnyEvent::MP::Intro - explains most concepts. 59 db_keys $family => $cb->(\@keys)
57 AnyEvent::MP::Kernel - mostly stable API. 60 db_values $family => $cb->(\@values)
58 AnyEvent::MP::Global - stable API. 61
62 # distributed database - monitoring a family
63 db_mon $family => $cb->(\%familyhash, \@added, \@changed, \@deleted)
59 64
60=head1 DESCRIPTION 65=head1 DESCRIPTION
61 66
62This module (-family) implements a simple message passing framework. 67This module (-family) implements a simple message passing framework.
63 68
184 189
185use common::sense; 190use common::sense;
186 191
187use Carp (); 192use Carp ();
188 193
189use AE (); 194use AnyEvent ();
190use Guard (); 195use Guard ();
191 196
192use base "Exporter"; 197use base "Exporter";
193 198
194our $VERSION = $AnyEvent::MP::Config::VERSION; 199our $VERSION = $AnyEvent::MP::Config::VERSION;
197 NODE $NODE *SELF node_of after 202 NODE $NODE *SELF node_of after
198 configure 203 configure
199 snd rcv mon mon_guard kil psub peval spawn cal 204 snd rcv mon mon_guard kil psub peval spawn cal
200 port 205 port
201 db_set db_del db_reg 206 db_set db_del db_reg
207 db_mon db_family db_keys db_values
202); 208);
203 209
204our $SELF; 210our $SELF;
205 211
206sub _self_die() { 212sub _self_die() {
230 236
231This function configures a node - it must be called exactly once (or 237This function configures a node - it must be called exactly once (or
232never) before calling other AnyEvent::MP functions. 238never) before calling other AnyEvent::MP functions.
233 239
234The key/value pairs are basically the same ones as documented for the 240The key/value pairs are basically the same ones as documented for the
235F<aemp> command line utility (sans the set/del prefix), with two additions: 241F<aemp> command line utility (sans the set/del prefix), with these additions:
236 242
237=over 4 243=over 4
238 244
239=item norc => $boolean (default false) 245=item norc => $boolean (default false)
240 246
271and the values specified directly via C<configure> have lowest priority, 277and the values specified directly via C<configure> have lowest priority,
272and can only be used to specify defaults. 278and can only be used to specify defaults.
273 279
274If the profile specifies a node ID, then this will become the node ID of 280If the profile specifies a node ID, then this will become the node ID of
275this process. If not, then the profile name will be used as node ID, with 281this process. If not, then the profile name will be used as node ID, with
276a slash (C</>) attached. 282a unique randoms tring (C</%u>) appended.
277 283
278If the node ID (or profile name) ends with a slash (C</>), then a random 284The node ID can contain some C<%> sequences that are expanded: C<%n>
279string is appended to make it unique. 285is expanded to the local nodename, C<%u> is replaced by a random
286strign to make the node unique. For example, the F<aemp> commandline
287utility uses C<aemp/%n/%u> as nodename, which might expand to
288C<aemp/cerebro/ZQDGSIkRhEZQDGSIkRhE>.
280 289
281=item step 2, bind listener sockets 290=item step 2, bind listener sockets
282 291
283The next step is to look up the binds in the profile, followed by binding 292The next step is to look up the binds in the profile, followed by binding
284aemp protocol listeners on all binds specified (it is possible and valid 293aemp protocol listeners on all binds specified (it is possible and valid
301Example: become a distributed node using the local node name as profile. 310Example: become a distributed node using the local node name as profile.
302This should be the most common form of invocation for "daemon"-type nodes. 311This should be the most common form of invocation for "daemon"-type nodes.
303 312
304 configure 313 configure
305 314
306Example: become an anonymous node. This form is often used for commandline 315Example: become a semi-anonymous node. This form is often used for
307clients. 316commandline clients.
308 317
309 configure nodeid => "anon/"; 318 configure nodeid => "myscript/%n/%u";
310 319
311Example: configure a node using a profile called seed, which is suitable 320Example: configure a node using a profile called seed, which is suitable
312for a seed node as it binds on all local addresses on a fixed port (4040, 321for a seed node as it binds on all local addresses on a fixed port (4040,
313customary for aemp). 322customary for aemp).
314 323
386 395
387=cut 396=cut
388 397
389sub rcv($@); 398sub rcv($@);
390 399
391sub _kilme { 400my $KILME = sub {
392 die "received message on port without callback"; 401 (my $tag = substr $_[0], 0, 30) =~ s/([\x20-\x7e])/./g;
393} 402 kil $SELF, unhandled_message => "no callback found for message '$tag'";
403};
394 404
395sub port(;&) { 405sub port(;&) {
396 my $id = $UNIQ . ++$ID; 406 my $id = $UNIQ . ++$ID;
397 my $port = "$NODE#$id"; 407 my $port = "$NODE#$id";
398 408
399 rcv $port, shift || \&_kilme; 409 rcv $port, shift || $KILME;
400 410
401 $port 411 $port
402} 412}
403 413
404=item rcv $local_port, $callback->(@msg) 414=item rcv $local_port, $callback->(@msg)
409 419
410The global C<$SELF> (exported by this module) contains C<$port> while 420The global C<$SELF> (exported by this module) contains C<$port> while
411executing the callback. Runtime errors during callback execution will 421executing the callback. Runtime errors during callback execution will
412result in the port being C<kil>ed. 422result in the port being C<kil>ed.
413 423
414The default callback received all messages not matched by a more specific 424The default callback receives all messages not matched by a more specific
415C<tag> match. 425C<tag> match.
416 426
417=item rcv $local_port, tag => $callback->(@msg_without_tag), ... 427=item rcv $local_port, tag => $callback->(@msg_without_tag), ...
418 428
419Register (or replace) callbacks to be called on messages starting with the 429Register (or replace) callbacks to be called on messages starting with the
583 $res 593 $res
584 } 594 }
585 } 595 }
586} 596}
587 597
598=item $guard = mon $port, $rcvport # kill $rcvport when $port dies
599
600=item $guard = mon $port # kill $SELF when $port dies
601
588=item $guard = mon $port, $cb->(@reason) # call $cb when $port dies 602=item $guard = mon $port, $cb->(@reason) # call $cb when $port dies
589
590=item $guard = mon $port, $rcvport # kill $rcvport when $port dies
591
592=item $guard = mon $port # kill $SELF when $port dies
593 603
594=item $guard = mon $port, $rcvport, @msg # send a message when $port dies 604=item $guard = mon $port, $rcvport, @msg # send a message when $port dies
595 605
596Monitor the given port and do something when the port is killed or 606Monitor the given port and do something when the port is killed or
597messages to it were lost, and optionally return a guard that can be used 607messages to it were lost, and optionally return a guard that can be used
598to stop monitoring again. 608to stop monitoring again.
599 609
610The first two forms distinguish between "normal" and "abnormal" kil's:
611
612In the first form (another port given), if the C<$port> is C<kil>'ed with
613a non-empty reason, the other port (C<$rcvport>) will be kil'ed with the
614same reason. That is, on "normal" kil's nothing happens, while under all
615other conditions, the other port is killed with the same reason.
616
617The second form (kill self) is the same as the first form, except that
618C<$rvport> defaults to C<$SELF>.
619
620The remaining forms don't distinguish between "normal" and "abnormal" kil's
621- it's up to the callback or receiver to check whether the C<@reason> is
622empty and act accordingly.
623
600In the first form (callback), the callback is simply called with any 624In the third form (callback), the callback is simply called with any
601number of C<@reason> elements (no @reason means that the port was deleted 625number of C<@reason> elements (empty @reason means that the port was deleted
602"normally"). Note also that I<< the callback B<must> never die >>, so use 626"normally"). Note also that I<< the callback B<must> never die >>, so use
603C<eval> if unsure. 627C<eval> if unsure.
604 628
605In the second form (another port given), the other port (C<$rcvport>)
606will be C<kil>'ed with C<@reason>, if a @reason was specified, i.e. on
607"normal" kils nothing happens, while under all other conditions, the other
608port is killed with the same reason.
609
610The third form (kill self) is the same as the second form, except that
611C<$rvport> defaults to C<$SELF>.
612
613In the last form (message), a message of the form C<@msg, @reason> will be 629In the last form (message), a message of the form C<$rcvport, @msg,
614C<snd>. 630@reason> will be C<snd>.
615 631
616Monitoring-actions are one-shot: once messages are lost (and a monitoring 632Monitoring-actions are one-shot: once messages are lost (and a monitoring
617alert was raised), they are removed and will not trigger again. 633alert was raised), they are removed and will not trigger again, even if it
634turns out that the port is still alive.
618 635
619As a rule of thumb, monitoring requests should always monitor a port from 636As a rule of thumb, monitoring requests should always monitor a remote
620a local port (or callback). The reason is that kill messages might get 637port locally (using a local C<$rcvport> or a callback). The reason is that
621lost, just like any other message. Another less obvious reason is that 638kill messages might get lost, just like any other message. Another less
622even monitoring requests can get lost (for example, when the connection 639obvious reason is that even monitoring requests can get lost (for example,
623to the other node goes down permanently). When monitoring a port locally 640when the connection to the other node goes down permanently). When
624these problems do not exist. 641monitoring a port locally these problems do not exist.
625 642
626C<mon> effectively guarantees that, in the absence of hardware failures, 643C<mon> effectively guarantees that, in the absence of hardware failures,
627after starting the monitor, either all messages sent to the port will 644after starting the monitor, either all messages sent to the port will
628arrive, or the monitoring action will be invoked after possible message 645arrive, or the monitoring action will be invoked after possible message
629loss has been detected. No messages will be lost "in between" (after 646loss has been detected. No messages will be lost "in between" (after
720will be reported as reason C<< die => $@ >>. 737will be reported as reason C<< die => $@ >>.
721 738
722Transport/communication errors are reported as C<< transport_error => 739Transport/communication errors are reported as C<< transport_error =>
723$message >>. 740$message >>.
724 741
725=cut 742Common idioms:
743
744 # silently remove yourself, do not kill linked ports
745 kil $SELF;
746
747 # report a failure in some detail
748 kil $SELF, failure_mode_1 => "it failed with too high temperature";
749
750 # do not waste much time with killing, just die when something goes wrong
751 open my $fh, "<file"
752 or die "file: $!";
726 753
727=item $port = spawn $node, $initfunc[, @initdata] 754=item $port = spawn $node, $initfunc[, @initdata]
728 755
729Creates a port on the node C<$node> (which can also be a port ID, in which 756Creates a port on the node C<$node> (which can also be a port ID, in which
730case it's the node where that port resides). 757case it's the node where that port resides).
822 ref $action[0] 849 ref $action[0]
823 ? $action[0]() 850 ? $action[0]()
824 : snd @action; 851 : snd @action;
825 }; 852 };
826} 853}
854
855#=item $cb2 = timeout $seconds, $cb[, @args]
827 856
828=item cal $port, @msg, $callback[, $timeout] 857=item cal $port, @msg, $callback[, $timeout]
829 858
830A simple form of RPC - sends a message to the given C<$port> with the 859A simple form of RPC - sends a message to the given C<$port> with the
831given contents (C<@msg>), but adds a reply port to the message. 860given contents (C<@msg>), but adds a reply port to the message.
880=back 909=back
881 910
882=head1 DISTRIBUTED DATABASE 911=head1 DISTRIBUTED DATABASE
883 912
884AnyEvent::MP comes with a simple distributed database. The database will 913AnyEvent::MP comes with a simple distributed database. The database will
885be mirrored asynchronously at all global nodes. Other nodes bind to one of 914be mirrored asynchronously on all global nodes. Other nodes bind to one
886the global nodes for their needs. 915of the global nodes for their needs. Every node has a "local database"
916which contains all the values that are set locally. All local databases
917are merged together to form the global database, which can be queried.
887 918
888The database consists of a two-level hash - a hash contains a hash which 919The database structure is that of a two-level hash - the database hash
889contains values. 920contains hashes which contain values, similarly to a perl hash of hashes,
921i.e.:
922
923 $DATABASE{$family}{$subkey} = $value
890 924
891The top level hash key is called "family", and the second-level hash key 925The top level hash key is called "family", and the second-level hash key
892is simply called "key". 926is called "subkey" or simply "key".
893 927
894The family and key must be alphanumeric ASCII strings, i.e. start 928The family must be alphanumeric, i.e. start with a letter and consist
895with a letter and consist of letters, digits, underscores and colons 929of letters, digits, underscores and colons (C<[A-Za-z][A-Za-z0-9_:]*>,
896(C<[A-Za-z][A-Za-z0-9_:]*>, pretty much like Perl module names. 930pretty much like Perl module names.
897 931
898As the family namespaceis global, it is recommended to prefix family names 932As the family namespace is global, it is recommended to prefix family names
899with the name of the application or module using it. 933with the name of the application or module using it.
900 934
935The subkeys must be non-empty strings, with no further restrictions.
936
901The values should preferably be strings, but other perl scalars should 937The values should preferably be strings, but other perl scalars should
902work as well (such as arrays and hashes). 938work as well (such as C<undef>, arrays and hashes).
903 939
904Every database entry is owned by one node - adding the same family/key 940Every database entry is owned by one node - adding the same family/subkey
905combination on multiple nodes will not cause discomfort for AnyEvent::MP, 941combination on multiple nodes will not cause discomfort for AnyEvent::MP,
906but the result might be nondeterministic, i.e. the key might have 942but the result might be nondeterministic, i.e. the key might have
907different values on different nodes. 943different values on different nodes.
908 944
909=item db_set $family => $key => $value 945Different subkeys in the same family can be owned by different nodes
946without problems, and in fact, this is the common method to create worker
947pools. For example, a worker port for image scaling might do this:
910 948
911Sets (or replaces) a key to the database. 949 db_set my_image_scalers => $port;
912 950
913=item db_del $family => $key 951And clients looking for an image scaler will want to get the
952C<my_image_scalers> keys from time to time:
914 953
915Deletes a key from the database. 954 db_keys my_image_scalers => sub {
955 @ports = @{ $_[0] };
956 };
916 957
958Or better yet, they want to monitor the database family, so they always
959have a reasonable up-to-date copy:
960
961 db_mon my_image_scalers => sub {
962 @ports = keys %{ $_[0] };
963 };
964
965In general, you can set or delete single subkeys, but query and monitor
966whole families only.
967
968If you feel the need to monitor or query a single subkey, try giving it
969it's own family.
970
971=over
972
917=item $guard = db_reg $family => $key [=> $value] 973=item $guard = db_set $family => $subkey [=> $value]
918 974
919Sets the key on the database and returns a guard. When the guard is 975Sets (or replaces) a key to the database - if C<$value> is omitted,
920destroyed, the key is deleted from the database. If C<$value> is missing, 976C<undef> is used instead.
921then C<undef> is used. 977
978When called in non-void context, C<db_set> returns a guard that
979automatically calls C<db_del> when it is destroyed.
980
981=item db_del $family => $subkey...
982
983Deletes one or more subkeys from the database family.
984
985=item $guard = db_reg $family => $port => $value
986
987=item $guard = db_reg $family => $port
988
989=item $guard = db_reg $family
990
991Registers a port in the given family and optionally returns a guard to
992remove it.
993
994This function basically does the same as:
995
996 db_set $family => $port => $value
997
998Except that the port is monitored and automatically removed from the
999database family when it is kil'ed.
1000
1001If C<$value> is missing, C<undef> is used. If C<$port> is missing, then
1002C<$SELF> is used.
1003
1004This function is most useful to register a port in some port group (which
1005is just another name for a database family), and have it removed when the
1006port is gone. This works best when the port is a local port.
1007
1008=cut
1009
1010sub db_reg($$;$) {
1011 my $family = shift;
1012 my $port = @_ ? shift : $SELF;
1013
1014 my $clr = sub { db_del $family => $port };
1015 mon $port, $clr;
1016
1017 db_set $family => $port => $_[0];
1018
1019 defined wantarray
1020 and &Guard::guard ($clr)
1021}
1022
1023=item db_family $family => $cb->(\%familyhash)
1024
1025Queries the named database C<$family> and call the callback with the
1026family represented as a hash. You can keep and freely modify the hash.
1027
1028=item db_keys $family => $cb->(\@keys)
1029
1030Same as C<db_family>, except it only queries the family I<subkeys> and passes
1031them as array reference to the callback.
1032
1033=item db_values $family => $cb->(\@values)
1034
1035Same as C<db_family>, except it only queries the family I<values> and passes them
1036as array reference to the callback.
1037
1038=item $guard = db_mon $family => $cb->(\%familyhash, \@added, \@changed, \@deleted)
1039
1040Creates a monitor on the given database family. Each time a key is set
1041or or is deleted the callback is called with a hash containing the
1042database family and three lists of added, changed and deleted subkeys,
1043respectively. If no keys have changed then the array reference might be
1044C<undef> or even missing.
1045
1046If not called in void context, a guard object is returned that, when
1047destroyed, stops the monitor.
1048
1049The family hash reference and the key arrays belong to AnyEvent::MP and
1050B<must not be modified or stored> by the callback. When in doubt, make a
1051copy.
1052
1053As soon as possible after the monitoring starts, the callback will be
1054called with the intiial contents of the family, even if it is empty,
1055i.e. there will always be a timely call to the callback with the current
1056contents.
1057
1058It is possible that the callback is called with a change event even though
1059the subkey is already present and the value has not changed.
1060
1061The monitoring stops when the guard object is destroyed.
1062
1063Example: on every change to the family "mygroup", print out all keys.
1064
1065 my $guard = db_mon mygroup => sub {
1066 my ($family, $a, $c, $d) = @_;
1067 print "mygroup members: ", (join " ", keys %$family), "\n";
1068 };
1069
1070Exmaple: wait until the family "My::Module::workers" is non-empty.
1071
1072 my $guard; $guard = db_mon My::Module::workers => sub {
1073 my ($family, $a, $c, $d) = @_;
1074 return unless %$family;
1075 undef $guard;
1076 print "My::Module::workers now nonempty\n";
1077 };
1078
1079Example: print all changes to the family "AnyRvent::Fantasy::Module".
1080
1081 my $guard = db_mon AnyRvent::Fantasy::Module => sub {
1082 my ($family, $a, $c, $d) = @_;
1083
1084 print "+$_=$family->{$_}\n" for @$a;
1085 print "*$_=$family->{$_}\n" for @$c;
1086 print "-$_=$family->{$_}\n" for @$d;
1087 };
922 1088
923=cut 1089=cut
924 1090
925=back 1091=back
926 1092
1085Keeping your messages simple, concentrating on data structures rather than 1251Keeping your messages simple, concentrating on data structures rather than
1086objects, will keep your messages clean, tidy and efficient. 1252objects, will keep your messages clean, tidy and efficient.
1087 1253
1088=back 1254=back
1089 1255
1256=head1 PORTING FROM AnyEvent::MP VERSION 1.X
1257
1258AEMP version 2 has a few major incompatible changes compared to version 1:
1259
1260=over 4
1261
1262=item AnyEvent::MP::Global no longer has group management functions.
1263
1264At least not officially - the grp_* functions are still exported and might
1265work, but they will be removed in some later release.
1266
1267AnyEvent::MP now comes with a distributed database that is more
1268powerful. Its database families map closely to port groups, but the API
1269has changed (the functions are also now exported by AnyEvent::MP). Here is
1270a rough porting guide:
1271
1272 grp_reg $group, $port # old
1273 db_reg $group, $port # new
1274
1275 $list = grp_get $group # old
1276 db_keys $group, sub { my $list = shift } # new
1277
1278 grp_mon $group, $cb->(\@ports, $add, $del) # old
1279 db_mon $group, $cb->(\%ports, $add, $change, $del) # new
1280
1281C<grp_reg> is a no-brainer (just replace by C<db_reg>), but C<grp_get> is
1282no longer instant, because the local node might not have a copy of the
1283group. You can either modify your code to allow for a callback, or use
1284C<db_mon> to keep an updated copy of the group:
1285
1286 my $local_group_copy;
1287 db_mon $group => sub { $local_group_copy = $_[0] };
1288
1289 # now "keys %$local_group_copy" always returns the most up-to-date
1290 # list of ports in the group.
1291
1292C<grp_mon> can be replaced by C<db_mon> with minor changes - C<db_mon>
1293passes a hash as first argument, and an extra C<$chg> argument that can be
1294ignored:
1295
1296 db_mon $group => sub {
1297 my ($ports, $add, $chg, $lde) = @_;
1298 $ports = [keys %$ports];
1299
1300 # now $ports, $add and $del are the same as
1301 # were originally passed by grp_mon.
1302 ...
1303 };
1304
1305=item Nodes not longer connect to all other nodes.
1306
1307In AEMP 1.x, every node automatically loads the L<AnyEvent::MP::Global>
1308module, which in turn would create connections to all other nodes in the
1309network (helped by the seed nodes).
1310
1311In version 2.x, global nodes still connect to all other global nodes, but
1312other nodes don't - now every node either is a global node itself, or
1313attaches itself to another global node.
1314
1315If a node isn't a global node itself, then it attaches itself to one
1316of its seed nodes. If that seed node isn't a global node yet, it will
1317automatically be upgraded to a global node.
1318
1319So in many cases, nothing needs to be changed - one just has to make sure
1320that all seed nodes are meshed together with the other seed nodes (as with
1321AEMP 1.x), and other nodes specify them as seed nodes. This is most easily
1322achieved by specifying the same set of seed nodes for all nodes in the
1323network.
1324
1325Not opening a connection to every other node is usually an advantage,
1326except when you need the lower latency of an already established
1327connection. To ensure a node establishes a connection to another node,
1328you can monitor the node port (C<mon $node, ...>), which will attempt to
1329create the connection (and notify you when the connection fails).
1330
1331=item Listener-less nodes (nodes without binds) are gone.
1332
1333And are not coming back, at least not in their old form. If no C<binds>
1334are specified for a node, AnyEvent::MP assumes a default of C<*:*>.
1335
1336There are vague plans to implement some form of routing domains, which
1337might or might not bring back listener-less nodes, but don't count on it.
1338
1339The fact that most connections are now optional somewhat mitigates this,
1340as a node can be effectively unreachable from the outside without any
1341problems, as long as it isn't a global node and only reaches out to other
1342nodes (as opposed to being contacted from other nodes).
1343
1344=item $AnyEvent::MP::Kernel::WARN has gone.
1345
1346AnyEvent has acquired a logging framework (L<AnyEvent::Log>), and AEMP now
1347uses this, and so should your programs.
1348
1349Every module now documents what kinds of messages it generates, with
1350AnyEvent::MP acting as a catch all.
1351
1352On the positive side, this means that instead of setting
1353C<PERL_ANYEVENT_MP_WARNLEVEL>, you can get away by setting C<AE_VERBOSE> -
1354much less to type.
1355
1356=back
1357
1358=head1 LOGGING
1359
1360AnyEvent::MP does not normally log anything by itself, but sinc eit is the
1361root of the contetx hierarchy for AnyEvent::MP modules, it will receive
1362all log messages by submodules.
1363
1090=head1 SEE ALSO 1364=head1 SEE ALSO
1091 1365
1092L<AnyEvent::MP::Intro> - a gentle introduction. 1366L<AnyEvent::MP::Intro> - a gentle introduction.
1093 1367
1094L<AnyEvent::MP::Kernel> - more, lower-level, stuff. 1368L<AnyEvent::MP::Kernel> - more, lower-level, stuff.

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines