ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-MP/MP/Intro.pod
Revision: 1.23
Committed: Sat Aug 29 14:36:10 2009 UTC (14 years, 9 months ago) by root
Branch: MAIN
Changes since 1.22: +48 -37 lines
Log Message:
*** empty log message ***

File Contents

# User Rev Content
1 root 1.4 =head1 Message Passing for the Non-Blocked Mind
2 elmex 1.1
3 root 1.8 =head1 Introduction and Terminology
4 elmex 1.1
5 root 1.4 This is a tutorial about how to get the swing of the new L<AnyEvent::MP>
6 root 1.23 module, which allows programs to transparently pass messages within the
7     process and to other processes on the same or a different host.
8 elmex 1.1
9 root 1.23 What kind of messages? Basically a message here means a list of Perl
10 root 1.15 strings, numbers, hashes and arrays, anything that can be expressed as a
11 root 1.23 L<JSON> text (as JSON is used by default in the protocol). Here are two
12     examples:
13 elmex 1.1
14 root 1.23 write_log => 1251555874, "action was successful.\n"
15     123, ["a", "b", "c"], { foo => "bar" }
16 elmex 1.21
17 root 1.23 When using L<AnyEvent::MP> it is customary to use a descriptive string as
18     first element of a message, that indictes the type of the message. This
19     element is called a I<tag> in L<AnyEvent::MP>, as some API functions
20     (C<rcv>) support matching it directly.
21    
22     Supposedly you want to send a ping message with your current time to
23     somewhere, this is how such a message might look like (in Perl syntax):
24    
25     ping => 1251381636
26    
27     Now that we know what a message is, to which entities are those
28     messages being I<passed>? They are I<passed> to I<ports>. A I<port> is
29     a destination for messages but also a context to execute code: when
30     a runtime error occurs while executing code belonging to a port, the
31     exception will be raised on the port and can even travel to interested
32     parties on other nodes, which makes supervision of distributed processes
33     easy.
34    
35     How do these ports relate to things you know? Each I<port> belongs
36     to a I<node>, and a I<node> is just the UNIX process that runs your
37     L<AnyEvent::MP> application.
38    
39     Each I<node> is distinguished from other I<nodes> running on the same or
40     another host in a network by its I<node ID>. A I<node ID> is simply a
41     unique string chosen manually or assigned by L<AnyEvent::MP> in some way
42     (UNIX nodename, random string...).
43    
44     Here is a diagram about how I<nodes>, I<ports> and UNIX processes relate
45     to each other. The setup consists of two nodes (more are of course
46     possible): Node C<A> (in UNIX process 7066) with the ports C<ABC> and
47     C<DEF>. And the node C<B> (in UNIX process 8321) with the ports C<FOO> and
48     C<BAR>.
49 elmex 1.17
50    
51     |- PID: 7066 -| |- PID: 8321 -|
52     | | | |
53     | Node ID: A | | Node ID: B |
54     | | | |
55     | Port ABC =|= <----\ /-----> =|= Port FOO |
56     | | X | |
57     | Port DEF =|= <----/ \-----> =|= Port BAR |
58     | | | |
59     |-------------| |-------------|
60    
61 root 1.23 The strings for the I<port IDs> here are just for illustrative
62     purposes: Even though I<ports> in L<AnyEvent::MP> are also identified by
63     strings, they can't be choosen manually and are assigned by the system
64     dynamically. These I<port IDs> are unique within a network and can also be
65     used to identify senders or as message tags for instance.
66    
67     The next sections will explain the API of L<AnyEvent::MP> by going through
68     a few simple examples. Later some more complex idioms are introduced,
69     which are hopefully useful to solve some real world problems.
70 root 1.8
71 elmex 1.16 =head1 Passing Your First Message
72    
73     As start lets have a look at the messaging API. The next example is just a
74     demo to show the basic elements of message passing with L<AnyEvent::MP>.
75     It shout just print: "Ending with: 123". So here the code:
76    
77     use AnyEvent;
78     use AnyEvent::MP;
79    
80     my $end_cv = AnyEvent->condvar;
81    
82     my $port = port;
83    
84     rcv $port, test => sub {
85     my ($data) = @_;
86     $end_cv->send ($data);
87     };
88    
89     snd $port, test => 123;
90    
91     print "Ending with: " . $end_cv->recv . "\n";
92    
93 elmex 1.17 It already contains most functions of the essential L<AnyEvent::MP> API.
94    
95     First there is the C<port> function which will create a I<port> and will return
96     it's I<port id>.
97    
98     That I<port id> can be used to send and receive messages. That I<port id> is a
99     simple string and can be safely passed to other I<nodes> in the network to
100     refer to that specific port (usually used for RPC, where you need to
101     tell the other end which I<port> to send the reply to).
102    
103     Next function is C<rcv>:
104 elmex 1.16
105 elmex 1.17 rcv $port, test => sub { ... };
106 elmex 1.16
107 elmex 1.17 It sets up a receiver callback on a specific I<port> which needs to be
108     specified as the first argument. The next argument, in this example C<test>, is
109     a I<tag> match. This means that whenever a message, with the first element
110     being the string C<tag>, is received the callback is called with the remaining
111     parts of that message.
112    
113     Messages can be send with the C<snd> function, which looks like this in the
114     example above:
115    
116     snd $port, test => 123;
117    
118     This will send the message C<['test', 123]> to the I<port> with the I<port id>
119     in C<$port>. The receiver got a I<tag> match on C<test> and will call the
120     callback with the first argument being the number C<123>.
121    
122     That callback then just passes that number on to the I<condition variable>
123     C<$end_cv> which will then pass the value to the print. But I<condition
124     variables> are out of the scope of this tutorial. So please consult the
125     L<AnyEvent::Intro> about them.
126    
127     But passing messages inside one process is boring, but before we can continue
128     and take the next step to interprocess message passing we first have to make
129     sure some things have been setup.
130    
131     =head1 System Requirements and System Setup
132    
133     Before we can start with real IPC we have to make sure some things work on your
134     system.
135    
136     First we have to setup a I<shared secret>: for two L<AnyEvent::MP> I<nodes> to
137     be able to communicate with each other and authenticate each other it is
138     necessary to setup the same I<shared secret> for both of them (or use TLS
139     certificates).
140    
141     The easiest way is to set this up is to use the F<aemp> utility:
142    
143     aemp gensecret
144    
145     This creates a F<$HOME/.perl-anyevent-mp> config file and generates a random
146     shared secret. You can copy this file to any other system and then communicate
147     over the network (via TCP) with it. You can also select your own shared secret
148     (F<aemp setsecret>) and for increased security requirements you can even create
149     a TLS certificate (F<aemp gencert>), causing connections to not just be
150     authenticated, but also to be encrypted.
151    
152     Connections will only be successful when the I<nodes> that want to connect to
153     each other have the same I<shared secret> (or successfully verify the TLS
154     certificate of the other side).
155    
156     B<If something does not work as expected, and for example tcpdump shows
157     that the connections are closed almost immediately, you should make sure
158     that F<~/.perl-anyevent-mp> is the same on all hosts/user accounts that
159     you try to connect with each other!>
160 elmex 1.16
161 elmex 1.18 Thats all for now, there is more fiddling around with the C<aemp> utility
162     later.
163    
164     =head1 Passing Messages Between Processes
165    
166     =head2 The Receiver
167    
168     Lets split the previous example up into two small programs. First the
169     receiver application:
170    
171     #!/opt/perl/bin/perl
172     use AnyEvent;
173     use AnyEvent::MP;
174     use AnyEvent::MP::Global;
175    
176     initialise_node "eg_simple_receiver";
177    
178     my $port = port;
179    
180     AnyEvent::MP::Global::register $port, "eg_receivers";
181    
182     rcv $port, test => sub {
183     my ($data, $reply_port) = @_;
184    
185     print "Received data: " . $data . "\n";
186     };
187    
188     AnyEvent->condvar->recv;
189    
190     =head3 AnyEvent::MP::Global
191    
192     Now, that wasn't too bad, was it? Ok, lets step through the new functions
193     and modules that have been used. For starters there is now an additional
194     module loaded: L<AnyEvent::MP::Global>.
195    
196     That module provides us with a I<global registry>, which lets us share data
197     among all I<nodes> in a network. Why do we need it you might ask?
198    
199 elmex 1.20 The thing is, that the I<port ids> are just random strings, assigned by
200     L<AnyEvent::MP>. We can't know those I<port ids> in advance, so we don't know
201     which I<port id> to send messages to if the message is to be passed between
202     I<nodes> (or UNIX processes). To find the right I<port> of another I<node> in
203     the network we will need to communicate that somehow to the sender. And
204     exactly that is what L<AnyEvent::MP::Global> provides.
205 elmex 1.18
206     =head3 initialise_node And The Network
207    
208     Now, lets have a look at the next new thing, the C<initialise_node>:
209    
210     initialise_node "eg_simple_receiver";
211    
212     Before we are able to send messages to other nodes we have to initialise
213     ourself. The first argument, the string C<"eg_simple_receiver">, is called the
214     I<profile> of this node. A profile holds some information about the application
215     that is going to be a node in an L<AnyEvent::MP> network.
216    
217     Most importantly the profile allows you to set the I<node id> that your
218     application will use. You can also set I<binds> in the profile, meaning that
219     you can define TCP ports that the application will listen on for incoming
220     connections from other nodes of the network.
221    
222     Next you can configure I<seeds> in profile. A I<seed> is just a TCP endpoint
223     which tells the application where to find other nodes of it's network. To
224     explain this a bit more detailed we have to look at the topology of an
225     L<AnyEvent::MP> network. The topology is called a I<fully connected mesh>, here
226     an example with 4 nodes:
227    
228     N1--N2
229     | \/ |
230     | /\ |
231     N3--N4
232    
233     Now imagine another I<node> C<N5>. wants to connect itself to that network:
234    
235     N1--N2
236     | \/ | N5
237     | /\ |
238     N3--N4
239    
240     The new node needs to know the I<binds> of all of those 4 already connected
241     nodes. And exactly this is what the I<seeds> are for. Now lets assume that
242     the new node C<N5> has as I<seed> the TCP endpoint of the node C<N2>.
243     It then connects to C<N2>:
244    
245     N1--N2____
246     | \/ | N5
247     | /\ |
248     N3--N4
249    
250     C<N2> then tells C<N5> the I<binds> of the other nodes it is connected to,
251     and C<N5> builds up the rest of the connections:
252    
253     /--------\
254     N1--N2____|
255     | \/ | N5
256     | /\ | /|
257     N3--N4--- |
258     \________/
259    
260     Finished. C<N5> is now happily connected to the rest of the network.
261    
262 elmex 1.19 =head3 Setting Up The Profiles
263    
264     Ok, so much to the profile. Now lets setup the C<eg_simple_receiver> I<profile>
265     for later. For the receiver we just give the receiver a I<bind>:
266    
267     aemp profile eg_simple_receiver setbinds localhost:12266
268    
269     And while we are at it, just setup the I<profile> for the sender in the second
270     part of this example too. We will call the sender I<profile>
271     C<eg_simple_sender>. For the sender we will just setup a I<seed> to the
272     receiver:
273    
274     aemp profile eg_simple_sender setseeds localhost:12266
275 elmex 1.22 aemp profile eg_simple_sender setbinds
276 elmex 1.19
277 elmex 1.22 You might wonder why we setup I<binds> to be empty here. Well, there can be
278 elmex 1.19 exceptions to the I<fully> in the I<fully connected mesh> in L<AnyEvent::MP>.
279     If you don't configure a I<bind> for a node's profile it won't bind itself
280     somewhere. These kinds of I<nodes> will not be able to send messages to other
281     I<nodes> that also didn't I<bind> them self to some TCP address. For this
282     example, as well as some cases in the real world, we can live with this
283     limitation.
284    
285     =head3 Registering The Receiver
286    
287     Ok, where were we. We now discussed the basic purpose of L<AnyEvent::MP::Global>
288     and initialise_node with it's relations to profiles. We also setup our profiles
289     for later use and now have to continue talking about the receiver example.
290    
291     Lets look at the next undiscussed line(s) of code:
292    
293     my $port = port;
294     AnyEvent::MP::Global::register $port, "eg_receivers";
295    
296     The C<port> function already has been discussed. It just creates a new I<port>
297     and gives us the I<port id>. Now to the C<register> function of
298     L<AnyEvent::MP::Global>: The first argument is a I<port id> that we want to add
299     to a I<global group>, and it's second argument is the name of that I<global
300     group>.
301    
302     You can choose that name of such a I<global group> freely, and it's purpose is
303     to store a set of I<port ids>. That set is made available throughout the whole
304     L<AnyEvent::MP> network, so that each node can see which ports belong to that
305     group.
306    
307     The sender will later look for the ports in that I<global group> and send
308     messages to them.
309    
310     Last step in the example is to setup a receiver callback for those messages
311     like we have discussed in the first example. We again match for the I<tag>
312     C<test>. The difference is just that we don't end the application after
313     receiving the first message. We just infinitely continue to look out for new
314     messages.
315    
316 elmex 1.20 =head2 The Sender
317 root 1.8
318 elmex 1.20 Ok, now lets take a look at the sender:
319 root 1.4
320 elmex 1.20 #!/opt/perl/bin/perl
321 elmex 1.1 use AnyEvent;
322     use AnyEvent::MP;
323 elmex 1.20 use AnyEvent::MP::Global;
324 elmex 1.1
325 elmex 1.20 initialise_node "eg_simple_sender";
326 elmex 1.1
327 elmex 1.20 my $find_timer =
328     AnyEvent->timer (after => 0, interval => 1, cb => sub {
329     my $ports = AnyEvent::MP::Global::find "eg_receivers"
330     or return;
331    
332     snd $_, test => time
333     for @$ports;
334     });
335 elmex 1.1
336     AnyEvent->condvar->recv;
337    
338 elmex 1.20 It's even less code. The C<initialise_node> is known now from the receiver
339     above. As discussed in the section where we setup the profiles we configure
340     this application to use the I<profile> C<eg_simple_sender>.
341 root 1.10
342 elmex 1.20 Next we setup a timer that repeatedly calls this chunk of code:
343 elmex 1.1
344 elmex 1.20 my $ports = AnyEvent::MP::Global::find "eg_receivers"
345     or return;
346 elmex 1.2
347 elmex 1.20 snd $_, test => time
348     for @$ports;
349 elmex 1.1
350 elmex 1.20 The new function here is the C<find> function of L<AnyEvent::MP::Global>. It
351     searches in the I<global group> named C<eg_receivers> for ports. If none are
352     found C<undef> is returned and we wait for the next time the timer fires.
353 elmex 1.1
354 elmex 1.20 In case the receiver application has been connected and the newly added port by
355     the receiver has propagated to the sender C<find> returns an array reference
356     that contains the I<port id> of the receiver I<port(s)>.
357 elmex 1.1
358 elmex 1.20 We then just send to every I<port> in the I<global group> a message consisting
359     of the I<tag> C<test> and the current time in form of a UNIX timestamp.
360 elmex 1.7
361 elmex 1.20 And thats all.
362 elmex 1.7
363 elmex 1.1 =head1 SEE ALSO
364    
365     L<AnyEvent>
366    
367     L<AnyEvent::Handle>
368    
369     L<AnyEvent::MP>
370    
371 elmex 1.20 L<AnyEvent::MP::Global>
372    
373 elmex 1.1 =head1 AUTHOR
374    
375     Robin Redeker <elmex@ta-sa.org>
376 root 1.4