ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-MP/MP.pm
(Generate patch)

Comparing AnyEvent-MP/MP.pm (file contents):
Revision 1.64 by root, Fri Aug 28 00:58:44 2009 UTC vs.
Revision 1.69 by root, Sun Aug 30 18:51:49 2009 UTC

40 40
41=head1 CURRENT STATUS 41=head1 CURRENT STATUS
42 42
43 AnyEvent::MP - stable API, should work 43 AnyEvent::MP - stable API, should work
44 AnyEvent::MP::Intro - outdated 44 AnyEvent::MP::Intro - outdated
45 AnyEvent::MP::Kernel - WIP
46 AnyEvent::MP::Transport - mostly stable 45 AnyEvent::MP::Kernel - mostly stable
46 AnyEvent::MP::Global - mostly stable
47 AnyEvent::MP::Node - mostly stable, but internal anyways
48 AnyEvent::MP::Transport - mostly stable, but internal anyways
47 49
48 stay tuned. 50 stay tuned.
49 51
50=head1 DESCRIPTION 52=head1 DESCRIPTION
51 53
52This module (-family) implements a simple message passing framework. 54This module (-family) implements a simple message passing framework.
53 55
54Despite its simplicity, you can securely message other processes running 56Despite its simplicity, you can securely message other processes running
55on the same or other hosts. 57on the same or other hosts, and you can supervise entities remotely.
56 58
57For an introduction to this module family, see the L<AnyEvent::MP::Intro> 59For an introduction to this module family, see the L<AnyEvent::MP::Intro>
58manual page. 60manual page and the examples under F<eg/>.
59 61
60At the moment, this module family is severly broken and underdocumented, 62At the moment, this module family is a bit underdocumented.
61so do not use. This was uploaded mainly to reserve the CPAN namespace -
62stay tuned!
63 63
64=head1 CONCEPTS 64=head1 CONCEPTS
65 65
66=over 4 66=over 4
67 67
71 71
72Ports allow you to register C<rcv> handlers that can match all or just 72Ports allow you to register C<rcv> handlers that can match all or just
73some messages. Messages send to ports will not be queued, regardless of 73some messages. Messages send to ports will not be queued, regardless of
74anything was listening for them or not. 74anything was listening for them or not.
75 75
76=item port ID - C<noderef#portname> 76=item port ID - C<nodeid#portname>
77 77
78A port ID is the concatenation of a noderef, a hash-mark (C<#>) as 78A port ID is the concatenation of a node ID, a hash-mark (C<#>) as
79separator, and a port name (a printable string of unspecified format). An 79separator, and a port name (a printable string of unspecified format).
80exception is the the node port, whose ID is identical to its node
81reference.
82 80
83=item node 81=item node
84 82
85A node is a single process containing at least one port - the node port, 83A node is a single process containing at least one port - the node port,
86which provides nodes to manage each other remotely, and to create new 84which enables nodes to manage each other remotely, and to create new
87ports. 85ports.
88 86
89Nodes are either private (single-process only), slaves (can only talk to 87Nodes are either public (have one or more listening ports) or private
90public nodes, but do not need an open port) or public nodes (connectable 88(no listening ports). Private nodes cannot talk to other private nodes
91from any other node). 89currently.
92 90
93=item node ID - C<[a-za-Z0-9_\-.:]+> 91=item node ID - C<[a-za-Z0-9_\-.:]+>
94 92
95A node ID is a string that uniquely identifies the node within a 93A node ID is a string that uniquely identifies the node within a
96network. Depending on the configuration used, node IDs can look like a 94network. Depending on the configuration used, node IDs can look like a
114are expected to be long-running, and at least one of those should always 112are expected to be long-running, and at least one of those should always
115be available. When nodes run out of connections (e.g. due to a network 113be available. When nodes run out of connections (e.g. due to a network
116error), they try to re-establish connections to some seednodes again to 114error), they try to re-establish connections to some seednodes again to
117join the network. 115join the network.
118 116
117Apart from being sued for seeding, seednodes are not special in any way -
118every public node can be a seednode.
119
119=back 120=back
120 121
121=head1 VARIABLES/FUNCTIONS 122=head1 VARIABLES/FUNCTIONS
122 123
123=over 4 124=over 4
138 139
139our $VERSION = $AnyEvent::MP::Kernel::VERSION; 140our $VERSION = $AnyEvent::MP::Kernel::VERSION;
140 141
141our @EXPORT = qw( 142our @EXPORT = qw(
142 NODE $NODE *SELF node_of after 143 NODE $NODE *SELF node_of after
143 resolve_node initialise_node 144 initialise_node
144 snd rcv mon mon_guard kil reg psub spawn 145 snd rcv mon mon_guard kil reg psub spawn
145 port 146 port
146); 147);
147 148
148our $SELF; 149our $SELF;
153 kil $SELF, die => $msg; 154 kil $SELF, die => $msg;
154} 155}
155 156
156=item $thisnode = NODE / $NODE 157=item $thisnode = NODE / $NODE
157 158
158The C<NODE> function returns, and the C<$NODE> variable contains the node 159The C<NODE> function returns, and the C<$NODE> variable contains, the node
159ID of the node running in the current process. This value is initialised by 160ID of the node running in the current process. This value is initialised by
160a call to C<initialise_node>. 161a call to C<initialise_node>.
161 162
162=item $nodeid = node_of $port 163=item $nodeid = node_of $port
163 164
164Extracts and returns the node ID part from a port ID or a node ID. 165Extracts and returns the node ID from a port ID or a node ID.
165 166
166=item initialise_node $profile_name 167=item initialise_node $profile_name, key => value...
167 168
168Before a node can talk to other nodes on the network (i.e. enter 169Before a node can talk to other nodes on the network (i.e. enter
169"distributed mode") it has to initialise itself - the minimum a node needs 170"distributed mode") it has to initialise itself - the minimum a node needs
170to know is its own name, and optionally it should know the addresses of 171to know is its own name, and optionally it should know the addresses of
171some other nodes in the network to discover other nodes. 172some other nodes in the network to discover other nodes.
174never) before calling other AnyEvent::MP functions. 175never) before calling other AnyEvent::MP functions.
175 176
176The first argument is a profile name. If it is C<undef> or missing, then 177The first argument is a profile name. If it is C<undef> or missing, then
177the current nodename will be used instead (i.e. F<uname -n>). 178the current nodename will be used instead (i.e. F<uname -n>).
178 179
179The function then looks up the profile in the aemp configuration (see the 180The function first looks up the profile in the aemp configuration (see the
180L<aemp> commandline utility). 181L<aemp> commandline utility). the profile is calculated as follows:
182
183First, all remaining key => value pairs will be used. Then they will be
184overwritten by any values specified in the global default configuration
185(see the F<aemp> utility), then the chain of profiles selected, if
186any. That means that the values specified in the profile have highest
187priority and the values specified via C<initialise_node> have lowest
188priority.
181 189
182If the profile specifies a node ID, then this will become the node ID of 190If the profile specifies a node ID, then this will become the node ID of
183this process. If not, then the profile name will be used as node ID. The 191this process. If not, then the profile name will be used as node ID. The
184special node ID of C<anon/> will be replaced by a random node ID. 192special node ID of C<anon/> will be replaced by a random node ID.
185 193
217=item $SELF 225=item $SELF
218 226
219Contains the current port id while executing C<rcv> callbacks or C<psub> 227Contains the current port id while executing C<rcv> callbacks or C<psub>
220blocks. 228blocks.
221 229
222=item SELF, %SELF, @SELF... 230=item *SELF, SELF, %SELF, @SELF...
223 231
224Due to some quirks in how perl exports variables, it is impossible to 232Due to some quirks in how perl exports variables, it is impossible to
225just export C<$SELF>, all the symbols called C<SELF> are exported by this 233just export C<$SELF>, all the symbols named C<SELF> are exported by this
226module, but only C<$SELF> is currently used. 234module, but only C<$SELF> is currently used.
227 235
228=item snd $port, type => @data 236=item snd $port, type => @data
229 237
230=item snd $port, @msg 238=item snd $port, @msg
231 239
232Send the given message to the given port ID, which can identify either 240Send the given message to the given port, which can identify either a
233a local or a remote port, and must be a port ID. 241local or a remote port, and must be a port ID.
234 242
235While the message can be about anything, it is highly recommended to use a 243While the message can be almost anything, it is highly recommended to
236string as first element (a port ID, or some word that indicates a request 244use a string as first element (a port ID, or some word that indicates a
237type etc.). 245request type etc.) and to consist if only simple perl values (scalars,
246arrays, hashes) - if you think you need to pass an object, think again.
238 247
239The message data effectively becomes read-only after a call to this 248The message data logically becomes read-only after a call to this
240function: modifying any argument is not allowed and can cause many 249function: modifying any argument (or values referenced by them) is
241problems. 250forbidden, as there can be considerable time between the call to C<snd>
251and the time the message is actually being serialised - in fact, it might
252never be copied as within the same process it is simply handed to the
253receiving port.
242 254
243The type of data you can transfer depends on the transport protocol: when 255The type of data you can transfer depends on the transport protocol: when
244JSON is used, then only strings, numbers and arrays and hashes consisting 256JSON is used, then only strings, numbers and arrays and hashes consisting
245of those are allowed (no objects). When Storable is used, then anything 257of those are allowed (no objects). When Storable is used, then anything
246that Storable can serialise and deserialise is allowed, and for the local 258that Storable can serialise and deserialise is allowed, and for the local
247node, anything can be passed. 259node, anything can be passed. Best rely only on the common denominator of
260these.
248 261
249=item $local_port = port 262=item $local_port = port
250 263
251Create a new local port object and returns its port ID. Initially it has 264Create a new local port object and returns its port ID. Initially it has
252no callbacks set and will throw an error when it receives messages. 265no callbacks set and will throw an error when it receives messages.
428 $res 441 $res
429 } 442 }
430 } 443 }
431} 444}
432 445
433=item $guard = mon $port, $cb->(@reason) 446=item $guard = mon $port, $cb->(@reason) # call $cb when $port dies
434 447
435=item $guard = mon $port, $rcvport 448=item $guard = mon $port, $rcvport # kill $rcvport when $port dies
436 449
437=item $guard = mon $port 450=item $guard = mon $port # kill $SELF when $port dies
438 451
439=item $guard = mon $port, $rcvport, @msg 452=item $guard = mon $port, $rcvport, @msg # send a message when $port dies
440 453
441Monitor the given port and do something when the port is killed or 454Monitor the given port and do something when the port is killed or
442messages to it were lost, and optionally return a guard that can be used 455messages to it were lost, and optionally return a guard that can be used
443to stop monitoring again. 456to stop monitoring again.
444 457
445C<mon> effectively guarantees that, in the absence of hardware failures, 458C<mon> effectively guarantees that, in the absence of hardware failures,
446that after starting the monitor, either all messages sent to the port 459after starting the monitor, either all messages sent to the port will
447will arrive, or the monitoring action will be invoked after possible 460arrive, or the monitoring action will be invoked after possible message
448message loss has been detected. No messages will be lost "in between" 461loss has been detected. No messages will be lost "in between" (after
449(after the first lost message no further messages will be received by the 462the first lost message no further messages will be received by the
450port). After the monitoring action was invoked, further messages might get 463port). After the monitoring action was invoked, further messages might get
451delivered again. 464delivered again.
452 465
453Note that monitoring-actions are one-shot: once released, they are removed 466Note that monitoring-actions are one-shot: once messages are lost (and a
454and will not trigger again. 467monitoring alert was raised), they are removed and will not trigger again.
455 468
456In the first form (callback), the callback is simply called with any 469In the first form (callback), the callback is simply called with any
457number of C<@reason> elements (no @reason means that the port was deleted 470number of C<@reason> elements (no @reason means that the port was deleted
458"normally"). Note also that I<< the callback B<must> never die >>, so use 471"normally"). Note also that I<< the callback B<must> never die >>, so use
459C<eval> if unsure. 472C<eval> if unsure.
521is killed, the references will be freed. 534is killed, the references will be freed.
522 535
523Optionally returns a guard that will stop the monitoring. 536Optionally returns a guard that will stop the monitoring.
524 537
525This function is useful when you create e.g. timers or other watchers and 538This function is useful when you create e.g. timers or other watchers and
526want to free them when the port gets killed: 539want to free them when the port gets killed (note the use of C<psub>):
527 540
528 $port->rcv (start => sub { 541 $port->rcv (start => sub {
529 my $timer; $timer = mon_guard $port, AE::timer 1, 1, sub { 542 my $timer; $timer = mon_guard $port, AE::timer 1, 1, psub {
530 undef $timer if 0.9 < rand; 543 undef $timer if 0.9 < rand;
531 }); 544 });
532 }); 545 });
533 546
534=cut 547=cut
543 556
544=item kil $port[, @reason] 557=item kil $port[, @reason]
545 558
546Kill the specified port with the given C<@reason>. 559Kill the specified port with the given C<@reason>.
547 560
548If no C<@reason> is specified, then the port is killed "normally" (linked 561If no C<@reason> is specified, then the port is killed "normally" (ports
549ports will not be kileld, or even notified). 562monitoring other ports will not necessarily die because a port dies
563"normally").
550 564
551Otherwise, linked ports get killed with the same reason (second form of 565Otherwise, linked ports get killed with the same reason (second form of
552C<mon>, see below). 566C<mon>, see above).
553 567
554Runtime errors while evaluating C<rcv> callbacks or inside C<psub> blocks 568Runtime errors while evaluating C<rcv> callbacks or inside C<psub> blocks
555will be reported as reason C<< die => $@ >>. 569will be reported as reason C<< die => $@ >>.
556 570
557Transport/communication errors are reported as C<< transport_error => 571Transport/communication errors are reported as C<< transport_error =>
562=item $port = spawn $node, $initfunc[, @initdata] 576=item $port = spawn $node, $initfunc[, @initdata]
563 577
564Creates a port on the node C<$node> (which can also be a port ID, in which 578Creates a port on the node C<$node> (which can also be a port ID, in which
565case it's the node where that port resides). 579case it's the node where that port resides).
566 580
567The port ID of the newly created port is return immediately, and it is 581The port ID of the newly created port is returned immediately, and it is
568permissible to immediately start sending messages or monitor the port. 582possible to immediately start sending messages or to monitor the port.
569 583
570After the port has been created, the init function is 584After the port has been created, the init function is called on the remote
571called. This function must be a fully-qualified function name 585node, in the same context as a C<rcv> callback. This function must be a
572(e.g. C<MyApp::Chat::Server::init>). To specify a function in the main 586fully-qualified function name (e.g. C<MyApp::Chat::Server::init>). To
573program, use C<::name>. 587specify a function in the main program, use C<::name>.
574 588
575If the function doesn't exist, then the node tries to C<require> 589If the function doesn't exist, then the node tries to C<require>
576the package, then the package above the package and so on (e.g. 590the package, then the package above the package and so on (e.g.
577C<MyApp::Chat::Server>, C<MyApp::Chat>, C<MyApp>) until the function 591C<MyApp::Chat::Server>, C<MyApp::Chat>, C<MyApp>) until the function
578exists or it runs out of package names. 592exists or it runs out of package names.
579 593
580The init function is then called with the newly-created port as context 594The init function is then called with the newly-created port as context
581object (C<$SELF>) and the C<@initdata> values as arguments. 595object (C<$SELF>) and the C<@initdata> values as arguments.
582 596
583A common idiom is to pass your own port, monitor the spawned port, and 597A common idiom is to pass a local port, immediately monitor the spawned
584in the init function, monitor the original port. This two-way monitoring 598port, and in the remote init function, immediately monitor the passed
585ensures that both ports get cleaned up when there is a problem. 599local port. This two-way monitoring ensures that both ports get cleaned up
600when there is a problem.
586 601
587Example: spawn a chat server port on C<$othernode>. 602Example: spawn a chat server port on C<$othernode>.
588 603
589 # this node, executed from within a port context: 604 # this node, executed from within a port context:
590 my $server = spawn $othernode, "MyApp::Chat::Server::connect", $SELF; 605 my $server = spawn $othernode, "MyApp::Chat::Server::connect", $SELF;
632=item after $timeout, $callback 647=item after $timeout, $callback
633 648
634Either sends the given message, or call the given callback, after the 649Either sends the given message, or call the given callback, after the
635specified number of seconds. 650specified number of seconds.
636 651
637This is simply a utility function that come sin handy at times. 652This is simply a utility function that comes in handy at times - the
653AnyEvent::MP author is not convinced of the wisdom of having it, though,
654so it may go away in the future.
638 655
639=cut 656=cut
640 657
641sub after($@) { 658sub after($@) {
642 my ($timeout, @action) = @_; 659 my ($timeout, @action) = @_;
665 682
666Despite the similarities, there are also some important differences: 683Despite the similarities, there are also some important differences:
667 684
668=over 4 685=over 4
669 686
670=item * Node references contain the recipe on how to contact them. 687=item * Node IDs are arbitrary strings in AEMP.
671 688
672Erlang relies on special naming and DNS to work everywhere in the 689Erlang relies on special naming and DNS to work everywhere in the same
673same way. AEMP relies on each node knowing it's own address(es), with 690way. AEMP relies on each node somehow knowing its own address(es) (e.g. by
674convenience functionality. 691configuraiton or DNS), but will otherwise discover other odes itself.
675
676This means that AEMP requires a less tightly controlled environment at the
677cost of longer node references and a slightly higher management overhead.
678 692
679=item * Erlang has a "remote ports are like local ports" philosophy, AEMP 693=item * Erlang has a "remote ports are like local ports" philosophy, AEMP
680uses "local ports are like remote ports". 694uses "local ports are like remote ports".
681 695
682The failure modes for local ports are quite different (runtime errors 696The failure modes for local ports are quite different (runtime errors
711 725
712Erlang makes few guarantees on messages delivery - messages can get lost 726Erlang makes few guarantees on messages delivery - messages can get lost
713without any of the processes realising it (i.e. you send messages a, b, 727without any of the processes realising it (i.e. you send messages a, b,
714and c, and the other side only receives messages a and c). 728and c, and the other side only receives messages a and c).
715 729
716AEMP guarantees correct ordering, and the guarantee that there are no 730AEMP guarantees correct ordering, and the guarantee that after one message
717holes in the message sequence. 731is lost, all following ones sent to the same port are lost as well, until
718 732monitoring raises an error, so there are no silent "holes" in the message
719=item * In Erlang, processes can be declared dead and later be found to be 733sequence.
720alive.
721
722In Erlang it can happen that a monitored process is declared dead and
723linked processes get killed, but later it turns out that the process is
724still alive - and can receive messages.
725
726In AEMP, when port monitoring detects a port as dead, then that port will
727eventually be killed - it cannot happen that a node detects a port as dead
728and then later sends messages to it, finding it is still alive.
729 734
730=item * Erlang can send messages to the wrong port, AEMP does not. 735=item * Erlang can send messages to the wrong port, AEMP does not.
731 736
732In Erlang it is quite likely that a node that restarts reuses a process ID 737In Erlang it is quite likely that a node that restarts reuses a process ID
733known to other nodes for a completely different process, causing messages 738known to other nodes for a completely different process, causing messages
737around in the network will not be sent to an unrelated port. 742around in the network will not be sent to an unrelated port.
738 743
739=item * Erlang uses unprotected connections, AEMP uses secure 744=item * Erlang uses unprotected connections, AEMP uses secure
740authentication and can use TLS. 745authentication and can use TLS.
741 746
742AEMP can use a proven protocol - SSL/TLS - to protect connections and 747AEMP can use a proven protocol - TLS - to protect connections and
743securely authenticate nodes. 748securely authenticate nodes.
744 749
745=item * The AEMP protocol is optimised for both text-based and binary 750=item * The AEMP protocol is optimised for both text-based and binary
746communications. 751communications.
747 752
748The AEMP protocol, unlike the Erlang protocol, supports both 753The AEMP protocol, unlike the Erlang protocol, supports both programming
749language-independent text-only protocols (good for debugging) and binary, 754language independent text-only protocols (good for debugging) and binary,
750language-specific serialisers (e.g. Storable). 755language-specific serialisers (e.g. Storable). By default, unless TLS is
756used, the protocol is actually completely text-based.
751 757
752It has also been carefully designed to be implementable in other languages 758It has also been carefully designed to be implementable in other languages
753with a minimum of work while gracefully degrading fucntionality to make the 759with a minimum of work while gracefully degrading functionality to make the
754protocol simple. 760protocol simple.
755 761
756=item * AEMP has more flexible monitoring options than Erlang. 762=item * AEMP has more flexible monitoring options than Erlang.
757 763
758In Erlang, you can chose to receive I<all> exit signals as messages 764In Erlang, you can chose to receive I<all> exit signals as messages
761Erlang, as one can choose between automatic kill, exit message or callback 767Erlang, as one can choose between automatic kill, exit message or callback
762on a per-process basis. 768on a per-process basis.
763 769
764=item * Erlang tries to hide remote/local connections, AEMP does not. 770=item * Erlang tries to hide remote/local connections, AEMP does not.
765 771
766Monitoring in Erlang is not an indicator of process death/crashes, 772Monitoring in Erlang is not an indicator of process death/crashes, in the
767as linking is (except linking is unreliable in Erlang). 773same way as linking is (except linking is unreliable in Erlang).
768 774
769In AEMP, you don't "look up" registered port names or send to named ports 775In AEMP, you don't "look up" registered port names or send to named ports
770that might or might not be persistent. Instead, you normally spawn a port 776that might or might not be persistent. Instead, you normally spawn a port
771on the remote node. The init function monitors the you, and you monitor 777on the remote node. The init function monitors you, and you monitor the
772the remote port. Since both monitors are local to the node, they are much 778remote port. Since both monitors are local to the node, they are much more
773more reliable. 779reliable (no need for C<spawn_link>).
774 780
775This also saves round-trips and avoids sending messages to the wrong port 781This also saves round-trips and avoids sending messages to the wrong port
776(hard to do in Erlang). 782(hard to do in Erlang).
777 783
778=back 784=back
779 785
780=head1 RATIONALE 786=head1 RATIONALE
781 787
782=over 4 788=over 4
783 789
784=item Why strings for ports and noderefs, why not objects? 790=item Why strings for port and node IDs, why not objects?
785 791
786We considered "objects", but found that the actual number of methods 792We considered "objects", but found that the actual number of methods
787thatc an be called are very low. Since port IDs and noderefs travel over 793that can be called are quite low. Since port and node IDs travel over
788the network frequently, the serialising/deserialising would add lots of 794the network frequently, the serialising/deserialising would add lots of
789overhead, as well as having to keep a proxy object. 795overhead, as well as having to keep a proxy object everywhere.
790 796
791Strings can easily be printed, easily serialised etc. and need no special 797Strings can easily be printed, easily serialised etc. and need no special
792procedures to be "valid". 798procedures to be "valid".
793 799
794And a a miniport consists of a single closure stored in a global hash - it 800And as a result, a miniport consists of a single closure stored in a
795can't become much cheaper. 801global hash - it can't become much cheaper.
796 802
797=item Why favour JSON, why not real serialising format such as Storable? 803=item Why favour JSON, why not a real serialising format such as Storable?
798 804
799In fact, any AnyEvent::MP node will happily accept Storable as framing 805In fact, any AnyEvent::MP node will happily accept Storable as framing
800format, but currently there is no way to make a node use Storable by 806format, but currently there is no way to make a node use Storable by
801default. 807default (although all nodes will accept it).
802 808
803The default framing protocol is JSON because a) JSON::XS is many times 809The default framing protocol is JSON because a) JSON::XS is many times
804faster for small messages and b) most importantly, after years of 810faster for small messages and b) most importantly, after years of
805experience we found that object serialisation is causing more problems 811experience we found that object serialisation is causing more problems
806than it gains: Just like function calls, objects simply do not travel 812than it solves: Just like function calls, objects simply do not travel
807easily over the network, mostly because they will always be a copy, so you 813easily over the network, mostly because they will always be a copy, so you
808always have to re-think your design. 814always have to re-think your design.
809 815
810Keeping your messages simple, concentrating on data structures rather than 816Keeping your messages simple, concentrating on data structures rather than
811objects, will keep your messages clean, tidy and efficient. 817objects, will keep your messages clean, tidy and efficient.
812 818
813=back 819=back
814 820
815=head1 SEE ALSO 821=head1 SEE ALSO
816 822
823L<AnyEvent::MP::Intro> - a gentle introduction.
824
825L<AnyEvent::MP::Kernel> - more, lower-level, stuff.
826
827L<AnyEvent::MP::Global> - network maintainance and port groups, to find
828your applications.
829
817L<AnyEvent>. 830L<AnyEvent>.
818 831
819=head1 AUTHOR 832=head1 AUTHOR
820 833
821 Marc Lehmann <schmorp@schmorp.de> 834 Marc Lehmann <schmorp@schmorp.de>

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines