AnyEvent-FastPing/FastPing.pm

=head1 NAME

AnyEvent::FastPing - quickly ping a large number of hosts

=head1 SYNOPSIS

 use AnyEvent::FastPing;

=head1 DESCRIPTION

This module was written for a single purpose only: sending ICMP ECHO
REQUEST packets as quickly as possible to a large number of hosts
(thousands to millions).

It employs a separate thread and is fully event-driven (using AnyEvent),
so you have to run an event model supported by AnyEvent to use this
module.

=head1 FUNCTIONS

=over 4

=cut

package AnyEvent::FastPing;

use common::sense;

use AnyEvent;

BEGIN {
   our $VERSION = '2.0';
   our @ISA = qw(Exporter);

   require Exporter;
   #Exporter::export_ok_tags (keys %EXPORT_TAGS);

   require XSLoader;
   XSLoader::load (__PACKAGE__, $VERSION);
}

our ($THR_RES_FD, $ICMP4_FD, $ICMP6_FD);

our $THR_RES_FH; open $THR_RES_FH, "<&=$THR_RES_FD" or die "FATAL: cannot fdopen";

our $ICMP4_FH; our $ICMP4_W = $ICMP4_FD >= 0 && (open $ICMP4_FH, "<&=$ICMP4_FD") && AE::io $ICMP4_FH, 0, \&_recv_icmp4;
our $ICMP6_FH; our $ICMP6_W = $ICMP6_FD >= 0 && (open $ICMP6_FH, "<&=$ICMP6_FD") && AE::io $ICMP6_FH, 0, \&_recv_icmp6;

=item AnyEvent::FastPing::ipv4_supported

Returns true iff IPv4 is supported in this module and on this system.

=item AnyEvent::FastPing::ipv6_supported

Returns true iff IPv6 is supported in this module and on this system.

=item AnyEvent::FastPing::icmp4_pktsize

Returns the number of octets per IPv4 ping packet (the whole IP packet
including headers, excluding lower-level headers or trailers such as
Ethernet).

Can be used to calculate e.g. octets/s from rate ...

   my $octets_per_second = $packets_per_second * AnyEvent::FastPing::icmp4_pktsize;

... or convert kilobit/second to packet rate ...

   my $packets_per_second = $kilobit_per_second
                            * (1000 / 8 / AnyEvent::FastPing::icmp4_pktsize);

etc.

=item AnyEvent::FastPing::icmp6_pktsize

Like AnyEvent::FastPing::icmp4_pktsize, but for IPv6.

=back

=head1 THE AnyEvent::FastPing CLASS

The AnyEvent::FastPing class represents a single "pinger". A "pinger"
comes with its own thread to send packets in the background, a rate-limit
machinery and separate idle/receive callbacks.

The recommended workflow (there are others) is this: 1. create a new
AnyEvent::FastPing object 2. configure the address lists and ranges to
ping, also configure an idle callback and optionally a receive callback
3. C<start> the pinger.

When the pinger has finished pinging all the configured addresses it will
call the idle callback.

The pinging process works like this: every range has a minimum interval
between sends, which is used to limit the rate at which hosts in that
range are being pinged. Distinct ranges are independent of each other,
which is why there is a per-pinger "global" minimum interval as well.

The pinger sends pings as fats as possible, while both obeying the pinger
rate limit as well as range limits.

When a range is exhausted, it is removed. When all ranges are exhausted,
the pinger waits another C<max_rtt> seconds and then exits, causing the
idle callback to trigger.

Performance: On my 2 GHz Opteron system with a pretty average nvidia
gigabit network card I can ping around 60k to 200k addresses per second,
depending on routing decisions.

Example: ping 10.0.0.1-10.0.0.15 with at most 100 packets/s, and
11.0.0.1-11.0.255.255 with at most 1000 packets/s. Also ping the IPv6
loopback address 5 times as fast as possible. Do not, however, exceed 1000
packets/s overall. Also dump each received reply.

   use AnyEvent::Socket;
   use AnyEvent::FastPing;

   my $done = AnyEvent->condvar;

   my $pinger = new AnyEvent::FastPing;

   $pinger->interval (1/1000);
   $pinger->max_rtt (0.1); # reasonably fast/reliable network

   $pinger->add_range (v10.0.0.1, v10.0.0.15, 1/100);
   $pinger->add_range (v11.0.0.1, v11.0.255.255, 1/1000);
   $pinger->add_hosts ([ (v0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1) x 5 ]);

   $pinger->on_recv (sub {
      for (@{ $_[0] }) {
         printf "%s %g\n", (AnyEvent::Socket::format_address $_->[0]), $_->[1];
      }
   });

   $pinger->on_idle (sub {
      print "done\n";
      undef $pinger;
   });

   $pinger->start;
   $done->wait;

=head2 METHODS

=over 4

=item $pinger = new AnyEvent::FastPing

Creates a new pinger - right now there can be at most C<65536> pingers in
a process, although that limit might change to something drastically lower
- you should be stingy with your pinger objects.

=cut

sub new {
   my ($klass) = @_;

   _new $klass, (rand 65536), (rand 65536), (rand 65536)
}

our @IDLE_CB;

sub DESTROY {
   undef $IDLE_CB[ &id ];
   &_free;
}

=item $pinger->on_recv ($callback->([[$host, $rtt], ...]))

Registers a callback to be called for ping replies. If no callback has
been registered than ping replies will be ignored, otherwise this module
calculates the round trip time, in seconds, for each reply and calls this
callback.

The callback receives a single argument, which is an array reference
with an entry for each reply packet (the replies will be batched for
efficiency). Each member in the array reference is again an array
reference with exactly two members: the binary host address (4 octets for
IPv4, 16 for IPv6) and the approximate round trip time, in seconds.

The replies will be passed to the callback as soon as they arrive, and
this callback can be called many times with batches of replies.

The receive callback will be called whenever a suitable reply arrives,
whether generated by this pinger or not, whether this pinger is started
or not. The packets will have a unique 64 bit ID to distinguish them from
other pinger objects and other generators, but this doesn't help against
malicious replies.

Note that very high packet rates can overwhelm your process, causing
replies to be dropped (configure your kernel with long receive queues for
raw sockets if this is a problem).

Example: register a callback which simply dumps the received data.

   use AnyEvent::Socket;

   $pinger->on_recv (sub {
      for (@{ $_[0] }) {
         printf "%s %g\n", (AnyEvent::Socket::format_address $_->[0]), $_->[1];
      }
   });

Example: a single ping reply with payload of 1 from C<::1> gets passed
like this:

   [
      [ "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1", 0.000280141830444336 ]
   ]

Example: ping replies for C<127.0.0.1> and C<127.0.0.2>:

   [
      [ "\177\0\0\1", 0.00015711784362793 ],
      [ "\177\0\0\2", 0.00090184211731 ]
   ]

=item $pinger->on_idle ($callback->())

Registers a callback to be called when the pinger becomes I<idle>, that
is, it has been started, has exhausted all ping ranges and waited for
the C<max_rtt> time. An idle pinger is also stopped, so the callback can
instantly add new ranges, if it so desires.

=cut

sub on_idle {
   $IDLE_CB[ &id ] = $_[1];
}

our $THR_RES_W = AE::io $THR_RES_FH, 0, sub {
   sysread $THR_RES_FH, my $buf, 8;

   for my $id (unpack "S*", $buf) {
      _stop_id $id;
      ($IDLE_CB[$id] || sub { })->();
   }
};

=item $pinger->interval ($seconds)

Configures the minimum interval between packet sends for this pinger - the
pinger will not send packets faster than this rate (or actually 1 / rate),
even if individual ranges have a lower interval.

A value of C<0> selects the fastest possible speed (currently no faster
than 1_000_000 packets/s).

=item $pinger->max_rtt ($seconds)

If your idle callback were called instantly after all ranges were
exhausted and you destroyed the object inside (which is common), then
there would be no chance to receive some replies, as there would be no
time of the packet to travel over the network.

This can be fixed by starting a timer in the idle callback, or more simply
by selecting a suitable C<max_rtt> value, which should be the maximum time
you allow a ping packet to travel to its destination and back.

The pinger thread automatically waits for this amount of time before becoming idle.

The default is currently C<0.5> seconds, which is usually plenty.

=item $pinger->add_range ($lo, $hi[, $interval])

Ping the IPv4 (or IPv6, but see below) address range, starting at binary
address C<$lo> and ending at C<$hi> (both C<$lo> and C<$hi> will be
pinged), generating no more than one ping per C<$interval> seconds (or as
fast as possible if omitted).

You can convert IP addresses from text to binary form by
using C<AnyEvent::Util::parse_address>, C<Socket::inet_aton>,
C<Socket6::inet_pton> or any other method that you like :)

The algorithm to select the next address is O(log n) on the number of
ranges, so even a large number of ranges (many thousands) is manageable.

No storage is allocated per address.

Note that, while IPv6 addresses are currently supported, the usefulness of
this option is extremely limited and might be gone in future versions - if
you want to ping a number of IPv6 hosts, better specify them individually
using the C<add_hosts> method.

=item $pinger->add_hosts ([$host...], $interval, $interleave)

Similar to C<add_range>, but uses a list of single addresses instead. The
list is specified as an array reference as first argument. Each entry in
the array should be a binary host address, either IPv4 or IPv6. If all
addresses are IPv4 addresses, then a compact IPv4-only format will be used
to store the list internally.

Minimum C<$interval> is the same as for C<add_range> and can be left out.

C<$interlave> specifies an increment between addresses: often address
lists are generated in a way that results in clustering - first all
addresses from one subnet, then from the next, and so on. To avoid this,
you can specify an interleave factor. If it is C<1> (the default), then
every address is pinged in the order specified. If it is C<2>, then only
every second address will be pinged in the first round, followed by a
second round with the others. Higher factors will create C<$interleave>
runs of addresses spaced C<$interleave> indices in the list.

The special value C<0> selects a (hopefully) suitable interleave factor
automatically - currently C<256> for lists with less than 65536 addresses,
and the square root of the list length otherwise.

=item $pinger->start

Start the pinger, unless it is running already. While a pinger is running
you must not modify the pinger. If you want to change a parameter, you
have to C<stop> the pinger first.

The pinger will automatically stop when destroyed.

=item $pinger->stop

Stop the pinger, if it is running. A pinger can be stopped at any time,
after which it's current state is preserved - starting it again will
continue where it left off.

=cut

1;

=back

=head1 AUTHOR

   Marc Lehmann <schmorp@schmorp.de>
   http://home.schmorp.de/

=head1 LICENSE

   This software is distributed under the GENERAL PUBLIC LICENSE, version 2
   or any later version or, at your option, the Artistic License.

=cut

Revision:	1.15
Committed:	Sat Feb 5 23:37:21 2011 UTC (13 years, 3 months ago) by root
Branch:	MAIN
Changes since 1.14:	+3 -3 lines
Log Message:	* empty log message *
#	User	Rev	Content
1	root	1.1	=head1 NAME
2
3			AnyEvent::FastPing - quickly ping a large number of hosts
4
5			=head1 SYNOPSIS
6
7			use AnyEvent::FastPing;
8
9			=head1 DESCRIPTION
10
11	root	1.2	This module was written for a single purpose only: sending ICMP ECHO
12	root	1.1	REQUEST packets as quickly as possible to a large number of hosts
13			(thousands to millions).
14
15	root	1.14	It employs a separate thread and is fully event-driven (using AnyEvent),
16			so you have to run an event model supported by AnyEvent to use this
17			module.
18	root	1.1
19			=head1 FUNCTIONS
20
21			=over 4
22
23			=cut
24
25			package AnyEvent::FastPing;
26
27	root	1.9	use common::sense;
28	root	1.1
29			use AnyEvent;
30
31			BEGIN {
32	root	1.11	our $VERSION = '2.0';
33	root	1.1	our @ISA = qw(Exporter);
34
35			require Exporter;
36			#Exporter::export_ok_tags (keys %EXPORT_TAGS);
37
38			require XSLoader;
39			XSLoader::load (__PACKAGE__, $VERSION);
40			}
41
42	root	1.11	our ($THR_RES_FD, $ICMP4_FD, $ICMP6_FD);
43	root	1.1
44			our $THR_RES_FH; open $THR_RES_FH, "<&=$THR_RES_FD" or die "FATAL: cannot fdopen";
45
46	root	1.11	our $ICMP4_FH; our $ICMP4_W = $ICMP4_FD >= 0 && (open $ICMP4_FH, "<&=$ICMP4_FD") && AE::io $ICMP4_FH, 0, \&_recv_icmp4;
47			our $ICMP6_FH; our $ICMP6_W = $ICMP6_FD >= 0 && (open $ICMP6_FH, "<&=$ICMP6_FD") && AE::io $ICMP6_FH, 0, \&_recv_icmp6;
48	root	1.1
49			=item AnyEvent::FastPing::ipv4_supported
50
51	root	1.13	Returns true iff IPv4 is supported in this module and on this system.
52	root	1.1
53			=item AnyEvent::FastPing::ipv6_supported
54
55	root	1.13	Returns true iff IPv6 is supported in this module and on this system.
56	root	1.1
57			=item AnyEvent::FastPing::icmp4_pktsize
58
59	root	1.13	Returns the number of octets per IPv4 ping packet (the whole IP packet
60			including headers, excluding lower-level headers or trailers such as
61			Ethernet).
62
63			Can be used to calculate e.g. octets/s from rate ...
64
65			my $octets_per_second = $packets_per_second * AnyEvent::FastPing::icmp4_pktsize;
66
67			... or convert kilobit/second to packet rate ...
68
69			my $packets_per_second = $kilobit_per_second
70			* (1000 / 8 / AnyEvent::FastPing::icmp4_pktsize);
71
72			etc.
73	root	1.1
74			=item AnyEvent::FastPing::icmp6_pktsize
75
76	root	1.13	Like AnyEvent::FastPing::icmp4_pktsize, but for IPv6.
77
78			=back
79
80			=head1 THE AnyEvent::FastPing CLASS
81
82			The AnyEvent::FastPing class represents a single "pinger". A "pinger"
83			comes with its own thread to send packets in the background, a rate-limit
84			machinery and separate idle/receive callbacks.
85
86			The recommended workflow (there are others) is this: 1. create a new
87			AnyEvent::FastPing object 2. configure the address lists and ranges to
88			ping, also configure an idle callback and optionally a receive callback
89			3. C<start> the pinger.
90
91			When the pinger has finished pinging all the configured addresses it will
92			call the idle callback.
93
94			The pinging process works like this: every range has a minimum interval
95			between sends, which is used to limit the rate at which hosts in that
96			range are being pinged. Distinct ranges are independent of each other,
97			which is why there is a per-pinger "global" minimum interval as well.
98
99			The pinger sends pings as fats as possible, while both obeying the pinger
100			rate limit as well as range limits.
101
102			When a range is exhausted, it is removed. When all ranges are exhausted,
103			the pinger waits another C<max_rtt> seconds and then exits, causing the
104			idle callback to trigger.
105
106			Performance: On my 2 GHz Opteron system with a pretty average nvidia
107	root	1.14	gigabit network card I can ping around 60k to 200k addresses per second,
108	root	1.13	depending on routing decisions.
109
110			Example: ping 10.0.0.1-10.0.0.15 with at most 100 packets/s, and
111			11.0.0.1-11.0.255.255 with at most 1000 packets/s. Also ping the IPv6
112			loopback address 5 times as fast as possible. Do not, however, exceed 1000
113	root	1.14	packets/s overall. Also dump each received reply.
114	root	1.13
115			use AnyEvent::Socket;
116			use AnyEvent::FastPing;
117
118			my $done = AnyEvent->condvar;
119
120			my $pinger = new AnyEvent::FastPing;
121
122			$pinger->interval (1/1000);
123			$pinger->max_rtt (0.1); # reasonably fast/reliable network
124
125			$pinger->add_range (v10.0.0.1, v10.0.0.15, 1/100);
126			$pinger->add_range (v11.0.0.1, v11.0.255.255, 1/1000);
127			$pinger->add_hosts ([ (v0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1) x 5 ]);
128
129			$pinger->on_recv (sub {
130			for (@{ $_[0] }) {
131			printf "%s %g\n", (AnyEvent::Socket::format_address $_->[0]), $_->[1];
132			}
133			});
134
135			$pinger->on_idle (sub {
136			print "done\n";
137			undef $pinger;
138			});
139
140			$pinger->start;
141			$done->wait;
142
143			=head2 METHODS
144
145			=over 4
146
147			=item $pinger = new AnyEvent::FastPing
148
149			Creates a new pinger - right now there can be at most C<65536> pingers in
150	root	1.14	a process, although that limit might change to something drastically lower
151			- you should be stingy with your pinger objects.
152	root	1.1
153	root	1.11	=cut
154
155			sub new {
156			my ($klass) = @_;
157
158			_new $klass, (rand 65536), (rand 65536), (rand 65536)
159			}
160
161			our @IDLE_CB;
162
163			sub DESTROY {
164			undef $IDLE_CB[ &id ];
165			&_free;
166			}
167
168	root	1.13	=item $pinger->on_recv ($callback->([[$host, $rtt], ...]))
169
170	root	1.14	Registers a callback to be called for ping replies. If no callback has
171	root	1.13	been registered than ping replies will be ignored, otherwise this module
172			calculates the round trip time, in seconds, for each reply and calls this
173			callback.
174
175			The callback receives a single argument, which is an array reference
176			with an entry for each reply packet (the replies will be batched for
177			efficiency). Each member in the array reference is again an array
178	root	1.14	reference with exactly two members: the binary host address (4 octets for
179	root	1.13	IPv4, 16 for IPv6) and the approximate round trip time, in seconds.
180
181			The replies will be passed to the callback as soon as they arrive, and
182			this callback can be called many times with batches of replies.
183
184			The receive callback will be called whenever a suitable reply arrives,
185			whether generated by this pinger or not, whether this pinger is started
186			or not. The packets will have a unique 64 bit ID to distinguish them from
187			other pinger objects and other generators, but this doesn't help against
188			malicious replies.
189
190			Note that very high packet rates can overwhelm your process, causing
191			replies to be dropped (configure your kernel with long receive queues for
192			raw sockets if this is a problem).
193
194			Example: register a callback which simply dumps the received data.
195
196			use AnyEvent::Socket;
197
198			$pinger->on_recv (sub {
199			for (@{ $_[0] }) {
200			printf "%s %g\n", (AnyEvent::Socket::format_address $_->[0]), $_->[1];
201			}
202	root	1.14	});
203	root	1.13
204			Example: a single ping reply with payload of 1 from C<::1> gets passed
205			like this:
206
207			[
208			[ "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1", 0.000280141830444336 ]
209			]
210
211			Example: ping replies for C<127.0.0.1> and C<127.0.0.2>:
212
213			[
214			[ "\177\0\0\1", 0.00015711784362793 ],
215			[ "\177\0\0\2", 0.00090184211731 ]
216			]
217
218			=item $pinger->on_idle ($callback->())
219
220			Registers a callback to be called when the pinger becomes I<idle>, that
221			is, it has been started, has exhausted all ping ranges and waited for
222			the C<max_rtt> time. An idle pinger is also stopped, so the callback can
223			instantly add new ranges, if it so desires.
224
225			=cut
226
227	root	1.11	sub on_idle {
228			$IDLE_CB[ &id ] = $_[1];
229			}
230
231			our $THR_RES_W = AE::io $THR_RES_FH, 0, sub {
232			sysread $THR_RES_FH, my $buf, 8;
233
234			for my $id (unpack "S*", $buf) {
235			_stop_id $id;
236			($IDLE_CB[$id] \|\| sub { })->();
237			}
238			};
239
240	root	1.13	=item $pinger->interval ($seconds)
241	root	1.12
242	root	1.13	Configures the minimum interval between packet sends for this pinger - the
243	root	1.14	pinger will not send packets faster than this rate (or actually 1 / rate),
244	root	1.13	even if individual ranges have a lower interval.
245	root	1.11
246	root	1.14	A value of C<0> selects the fastest possible speed (currently no faster
247	root	1.13	than 1_000_000 packets/s).
248	root	1.1
249	root	1.13	=item $pinger->max_rtt ($seconds)
250	root	1.1
251	root	1.13	If your idle callback were called instantly after all ranges were
252			exhausted and you destroyed the object inside (which is common), then
253			there would be no chance to receive some replies, as there would be no
254	root	1.14	time of the packet to travel over the network.
255	root	1.1
256	root	1.13	This can be fixed by starting a timer in the idle callback, or more simply
257			by selecting a suitable C<max_rtt> value, which should be the maximum time
258	root	1.14	you allow a ping packet to travel to its destination and back.
259	root	1.1
260	root	1.13	The pinger thread automatically waits for this amount of time before becoming idle.
261	root	1.1
262	root	1.13	The default is currently C<0.5> seconds, which is usually plenty.
263	root	1.1
264	root	1.13	=item $pinger->add_range ($lo, $hi[, $interval])
265	root	1.1
266	root	1.13	Ping the IPv4 (or IPv6, but see below) address range, starting at binary
267			address C<$lo> and ending at C<$hi> (both C<$lo> and C<$hi> will be
268			pinged), generating no more than one ping per C<$interval> seconds (or as
269			fast as possible if omitted).
270	root	1.1
271	root	1.13	You can convert IP addresses from text to binary form by
272			using C<AnyEvent::Util::parse_address>, C<Socket::inet_aton>,
273			C<Socket6::inet_pton> or any other method that you like :)
274	root	1.1
275	root	1.13	The algorithm to select the next address is O(log n) on the number of
276	root	1.14	ranges, so even a large number of ranges (many thousands) is manageable.
277	root	1.1
278	root	1.13	No storage is allocated per address.
279	root	1.1
280	root	1.13	Note that, while IPv6 addresses are currently supported, the usefulness of
281			this option is extremely limited and might be gone in future versions - if
282			you want to ping a number of IPv6 hosts, better specify them individually
283			using the C<add_hosts> method.
284
285			=item $pinger->add_hosts ([$host...], $interval, $interleave)
286
287			Similar to C<add_range>, but uses a list of single addresses instead. The
288			list is specified as an array reference as first argument. Each entry in
289	root	1.15	the array should be a binary host address, either IPv4 or IPv6. If all
290			addresses are IPv4 addresses, then a compact IPv4-only format will be used
291			to store the list internally.
292	root	1.13
293			Minimum C<$interval> is the same as for C<add_range> and can be left out.
294
295			C<$interlave> specifies an increment between addresses: often address
296			lists are generated in a way that results in clustering - first all
297			addresses from one subnet, then from the next, and so on. To avoid this,
298			you can specify an interleave factor. If it is C<1> (the default), then
299			every address is pinged in the order specified. If it is C<2>, then only
300			every second address will be pinged in the first round, followed by a
301			second round with the others. Higher factors will create C<$interleave>
302			runs of addresses spaced C<$interleave> indices in the list.
303
304			The special value C<0> selects a (hopefully) suitable interleave factor
305			automatically - currently C<256> for lists with less than 65536 addresses,
306			and the square root of the list length otherwise.
307
308			=item $pinger->start
309
310			Start the pinger, unless it is running already. While a pinger is running
311			you must not modify the pinger. If you want to change a parameter, you
312			have to C<stop> the pinger first.
313
314			The pinger will automatically stop when destroyed.
315
316			=item $pinger->stop
317
318			Stop the pinger, if it is running. A pinger can be stopped at any time,
319			after which it's current state is preserved - starting it again will
320			continue where it left off.
321	root	1.1
322			=cut
323
324			1;
325
326			=back
327
328			=head1 AUTHOR
329
330	root	1.4	Marc Lehmann <schmorp@schmorp.de>
331			http://home.schmorp.de/
332	root	1.1
333	root	1.4	=head1 LICENSE
334	root	1.1
335	root	1.4	This software is distributed under the GENERAL PUBLIC LICENSE, version 2
336			or any later version or, at your option, the Artistic License.
337	root	1.1
338			=cut
339