ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-MP/MP/Intro.pod
Revision: 1.23
Committed: Sat Aug 29 14:36:10 2009 UTC (14 years, 9 months ago) by root
Branch: MAIN
Changes since 1.22: +48 -37 lines
Log Message:
*** empty log message ***

File Contents

# Content
1 =head1 Message Passing for the Non-Blocked Mind
2
3 =head1 Introduction and Terminology
4
5 This is a tutorial about how to get the swing of the new L<AnyEvent::MP>
6 module, which allows programs to transparently pass messages within the
7 process and to other processes on the same or a different host.
8
9 What kind of messages? Basically a message here means a list of Perl
10 strings, numbers, hashes and arrays, anything that can be expressed as a
11 L<JSON> text (as JSON is used by default in the protocol). Here are two
12 examples:
13
14 write_log => 1251555874, "action was successful.\n"
15 123, ["a", "b", "c"], { foo => "bar" }
16
17 When using L<AnyEvent::MP> it is customary to use a descriptive string as
18 first element of a message, that indictes the type of the message. This
19 element is called a I<tag> in L<AnyEvent::MP>, as some API functions
20 (C<rcv>) support matching it directly.
21
22 Supposedly you want to send a ping message with your current time to
23 somewhere, this is how such a message might look like (in Perl syntax):
24
25 ping => 1251381636
26
27 Now that we know what a message is, to which entities are those
28 messages being I<passed>? They are I<passed> to I<ports>. A I<port> is
29 a destination for messages but also a context to execute code: when
30 a runtime error occurs while executing code belonging to a port, the
31 exception will be raised on the port and can even travel to interested
32 parties on other nodes, which makes supervision of distributed processes
33 easy.
34
35 How do these ports relate to things you know? Each I<port> belongs
36 to a I<node>, and a I<node> is just the UNIX process that runs your
37 L<AnyEvent::MP> application.
38
39 Each I<node> is distinguished from other I<nodes> running on the same or
40 another host in a network by its I<node ID>. A I<node ID> is simply a
41 unique string chosen manually or assigned by L<AnyEvent::MP> in some way
42 (UNIX nodename, random string...).
43
44 Here is a diagram about how I<nodes>, I<ports> and UNIX processes relate
45 to each other. The setup consists of two nodes (more are of course
46 possible): Node C<A> (in UNIX process 7066) with the ports C<ABC> and
47 C<DEF>. And the node C<B> (in UNIX process 8321) with the ports C<FOO> and
48 C<BAR>.
49
50
51 |- PID: 7066 -| |- PID: 8321 -|
52 | | | |
53 | Node ID: A | | Node ID: B |
54 | | | |
55 | Port ABC =|= <----\ /-----> =|= Port FOO |
56 | | X | |
57 | Port DEF =|= <----/ \-----> =|= Port BAR |
58 | | | |
59 |-------------| |-------------|
60
61 The strings for the I<port IDs> here are just for illustrative
62 purposes: Even though I<ports> in L<AnyEvent::MP> are also identified by
63 strings, they can't be choosen manually and are assigned by the system
64 dynamically. These I<port IDs> are unique within a network and can also be
65 used to identify senders or as message tags for instance.
66
67 The next sections will explain the API of L<AnyEvent::MP> by going through
68 a few simple examples. Later some more complex idioms are introduced,
69 which are hopefully useful to solve some real world problems.
70
71 =head1 Passing Your First Message
72
73 As start lets have a look at the messaging API. The next example is just a
74 demo to show the basic elements of message passing with L<AnyEvent::MP>.
75 It shout just print: "Ending with: 123". So here the code:
76
77 use AnyEvent;
78 use AnyEvent::MP;
79
80 my $end_cv = AnyEvent->condvar;
81
82 my $port = port;
83
84 rcv $port, test => sub {
85 my ($data) = @_;
86 $end_cv->send ($data);
87 };
88
89 snd $port, test => 123;
90
91 print "Ending with: " . $end_cv->recv . "\n";
92
93 It already contains most functions of the essential L<AnyEvent::MP> API.
94
95 First there is the C<port> function which will create a I<port> and will return
96 it's I<port id>.
97
98 That I<port id> can be used to send and receive messages. That I<port id> is a
99 simple string and can be safely passed to other I<nodes> in the network to
100 refer to that specific port (usually used for RPC, where you need to
101 tell the other end which I<port> to send the reply to).
102
103 Next function is C<rcv>:
104
105 rcv $port, test => sub { ... };
106
107 It sets up a receiver callback on a specific I<port> which needs to be
108 specified as the first argument. The next argument, in this example C<test>, is
109 a I<tag> match. This means that whenever a message, with the first element
110 being the string C<tag>, is received the callback is called with the remaining
111 parts of that message.
112
113 Messages can be send with the C<snd> function, which looks like this in the
114 example above:
115
116 snd $port, test => 123;
117
118 This will send the message C<['test', 123]> to the I<port> with the I<port id>
119 in C<$port>. The receiver got a I<tag> match on C<test> and will call the
120 callback with the first argument being the number C<123>.
121
122 That callback then just passes that number on to the I<condition variable>
123 C<$end_cv> which will then pass the value to the print. But I<condition
124 variables> are out of the scope of this tutorial. So please consult the
125 L<AnyEvent::Intro> about them.
126
127 But passing messages inside one process is boring, but before we can continue
128 and take the next step to interprocess message passing we first have to make
129 sure some things have been setup.
130
131 =head1 System Requirements and System Setup
132
133 Before we can start with real IPC we have to make sure some things work on your
134 system.
135
136 First we have to setup a I<shared secret>: for two L<AnyEvent::MP> I<nodes> to
137 be able to communicate with each other and authenticate each other it is
138 necessary to setup the same I<shared secret> for both of them (or use TLS
139 certificates).
140
141 The easiest way is to set this up is to use the F<aemp> utility:
142
143 aemp gensecret
144
145 This creates a F<$HOME/.perl-anyevent-mp> config file and generates a random
146 shared secret. You can copy this file to any other system and then communicate
147 over the network (via TCP) with it. You can also select your own shared secret
148 (F<aemp setsecret>) and for increased security requirements you can even create
149 a TLS certificate (F<aemp gencert>), causing connections to not just be
150 authenticated, but also to be encrypted.
151
152 Connections will only be successful when the I<nodes> that want to connect to
153 each other have the same I<shared secret> (or successfully verify the TLS
154 certificate of the other side).
155
156 B<If something does not work as expected, and for example tcpdump shows
157 that the connections are closed almost immediately, you should make sure
158 that F<~/.perl-anyevent-mp> is the same on all hosts/user accounts that
159 you try to connect with each other!>
160
161 Thats all for now, there is more fiddling around with the C<aemp> utility
162 later.
163
164 =head1 Passing Messages Between Processes
165
166 =head2 The Receiver
167
168 Lets split the previous example up into two small programs. First the
169 receiver application:
170
171 #!/opt/perl/bin/perl
172 use AnyEvent;
173 use AnyEvent::MP;
174 use AnyEvent::MP::Global;
175
176 initialise_node "eg_simple_receiver";
177
178 my $port = port;
179
180 AnyEvent::MP::Global::register $port, "eg_receivers";
181
182 rcv $port, test => sub {
183 my ($data, $reply_port) = @_;
184
185 print "Received data: " . $data . "\n";
186 };
187
188 AnyEvent->condvar->recv;
189
190 =head3 AnyEvent::MP::Global
191
192 Now, that wasn't too bad, was it? Ok, lets step through the new functions
193 and modules that have been used. For starters there is now an additional
194 module loaded: L<AnyEvent::MP::Global>.
195
196 That module provides us with a I<global registry>, which lets us share data
197 among all I<nodes> in a network. Why do we need it you might ask?
198
199 The thing is, that the I<port ids> are just random strings, assigned by
200 L<AnyEvent::MP>. We can't know those I<port ids> in advance, so we don't know
201 which I<port id> to send messages to if the message is to be passed between
202 I<nodes> (or UNIX processes). To find the right I<port> of another I<node> in
203 the network we will need to communicate that somehow to the sender. And
204 exactly that is what L<AnyEvent::MP::Global> provides.
205
206 =head3 initialise_node And The Network
207
208 Now, lets have a look at the next new thing, the C<initialise_node>:
209
210 initialise_node "eg_simple_receiver";
211
212 Before we are able to send messages to other nodes we have to initialise
213 ourself. The first argument, the string C<"eg_simple_receiver">, is called the
214 I<profile> of this node. A profile holds some information about the application
215 that is going to be a node in an L<AnyEvent::MP> network.
216
217 Most importantly the profile allows you to set the I<node id> that your
218 application will use. You can also set I<binds> in the profile, meaning that
219 you can define TCP ports that the application will listen on for incoming
220 connections from other nodes of the network.
221
222 Next you can configure I<seeds> in profile. A I<seed> is just a TCP endpoint
223 which tells the application where to find other nodes of it's network. To
224 explain this a bit more detailed we have to look at the topology of an
225 L<AnyEvent::MP> network. The topology is called a I<fully connected mesh>, here
226 an example with 4 nodes:
227
228 N1--N2
229 | \/ |
230 | /\ |
231 N3--N4
232
233 Now imagine another I<node> C<N5>. wants to connect itself to that network:
234
235 N1--N2
236 | \/ | N5
237 | /\ |
238 N3--N4
239
240 The new node needs to know the I<binds> of all of those 4 already connected
241 nodes. And exactly this is what the I<seeds> are for. Now lets assume that
242 the new node C<N5> has as I<seed> the TCP endpoint of the node C<N2>.
243 It then connects to C<N2>:
244
245 N1--N2____
246 | \/ | N5
247 | /\ |
248 N3--N4
249
250 C<N2> then tells C<N5> the I<binds> of the other nodes it is connected to,
251 and C<N5> builds up the rest of the connections:
252
253 /--------\
254 N1--N2____|
255 | \/ | N5
256 | /\ | /|
257 N3--N4--- |
258 \________/
259
260 Finished. C<N5> is now happily connected to the rest of the network.
261
262 =head3 Setting Up The Profiles
263
264 Ok, so much to the profile. Now lets setup the C<eg_simple_receiver> I<profile>
265 for later. For the receiver we just give the receiver a I<bind>:
266
267 aemp profile eg_simple_receiver setbinds localhost:12266
268
269 And while we are at it, just setup the I<profile> for the sender in the second
270 part of this example too. We will call the sender I<profile>
271 C<eg_simple_sender>. For the sender we will just setup a I<seed> to the
272 receiver:
273
274 aemp profile eg_simple_sender setseeds localhost:12266
275 aemp profile eg_simple_sender setbinds
276
277 You might wonder why we setup I<binds> to be empty here. Well, there can be
278 exceptions to the I<fully> in the I<fully connected mesh> in L<AnyEvent::MP>.
279 If you don't configure a I<bind> for a node's profile it won't bind itself
280 somewhere. These kinds of I<nodes> will not be able to send messages to other
281 I<nodes> that also didn't I<bind> them self to some TCP address. For this
282 example, as well as some cases in the real world, we can live with this
283 limitation.
284
285 =head3 Registering The Receiver
286
287 Ok, where were we. We now discussed the basic purpose of L<AnyEvent::MP::Global>
288 and initialise_node with it's relations to profiles. We also setup our profiles
289 for later use and now have to continue talking about the receiver example.
290
291 Lets look at the next undiscussed line(s) of code:
292
293 my $port = port;
294 AnyEvent::MP::Global::register $port, "eg_receivers";
295
296 The C<port> function already has been discussed. It just creates a new I<port>
297 and gives us the I<port id>. Now to the C<register> function of
298 L<AnyEvent::MP::Global>: The first argument is a I<port id> that we want to add
299 to a I<global group>, and it's second argument is the name of that I<global
300 group>.
301
302 You can choose that name of such a I<global group> freely, and it's purpose is
303 to store a set of I<port ids>. That set is made available throughout the whole
304 L<AnyEvent::MP> network, so that each node can see which ports belong to that
305 group.
306
307 The sender will later look for the ports in that I<global group> and send
308 messages to them.
309
310 Last step in the example is to setup a receiver callback for those messages
311 like we have discussed in the first example. We again match for the I<tag>
312 C<test>. The difference is just that we don't end the application after
313 receiving the first message. We just infinitely continue to look out for new
314 messages.
315
316 =head2 The Sender
317
318 Ok, now lets take a look at the sender:
319
320 #!/opt/perl/bin/perl
321 use AnyEvent;
322 use AnyEvent::MP;
323 use AnyEvent::MP::Global;
324
325 initialise_node "eg_simple_sender";
326
327 my $find_timer =
328 AnyEvent->timer (after => 0, interval => 1, cb => sub {
329 my $ports = AnyEvent::MP::Global::find "eg_receivers"
330 or return;
331
332 snd $_, test => time
333 for @$ports;
334 });
335
336 AnyEvent->condvar->recv;
337
338 It's even less code. The C<initialise_node> is known now from the receiver
339 above. As discussed in the section where we setup the profiles we configure
340 this application to use the I<profile> C<eg_simple_sender>.
341
342 Next we setup a timer that repeatedly calls this chunk of code:
343
344 my $ports = AnyEvent::MP::Global::find "eg_receivers"
345 or return;
346
347 snd $_, test => time
348 for @$ports;
349
350 The new function here is the C<find> function of L<AnyEvent::MP::Global>. It
351 searches in the I<global group> named C<eg_receivers> for ports. If none are
352 found C<undef> is returned and we wait for the next time the timer fires.
353
354 In case the receiver application has been connected and the newly added port by
355 the receiver has propagated to the sender C<find> returns an array reference
356 that contains the I<port id> of the receiver I<port(s)>.
357
358 We then just send to every I<port> in the I<global group> a message consisting
359 of the I<tag> C<test> and the current time in form of a UNIX timestamp.
360
361 And thats all.
362
363 =head1 SEE ALSO
364
365 L<AnyEvent>
366
367 L<AnyEvent::Handle>
368
369 L<AnyEvent::MP>
370
371 L<AnyEvent::MP::Global>
372
373 =head1 AUTHOR
374
375 Robin Redeker <elmex@ta-sa.org>
376