AnyEvent-FastPing/FastPing.pm

=head1 NAME

AnyEvent::FastPing - quickly ping a large number of hosts

=head1 SYNOPSIS

 use AnyEvent::FastPing;

=head1 DESCRIPTION

This module was written for a single purpose only: sending ICMP ECHO
REQUEST packets as quickly as possible to a large number of hosts
(thousands to millions).

It employs a separate thread and is fully event-driven (using AnyEvent),
so you have to run an event model supported by AnyEvent to use this
module.

=head1 FUNCTIONS

=over 4

=cut

package AnyEvent::FastPing;

use common::sense;

use AnyEvent;

BEGIN {
   our $VERSION = 2.03;
   our @ISA = qw(Exporter);

   require Exporter;
   #Exporter::export_ok_tags (keys %EXPORT_TAGS);

   require XSLoader;
   XSLoader::load (__PACKAGE__, $VERSION);
}

our ($THR_RES_FD, $ICMP4_FD, $ICMP6_FD);

our $THR_RES_FH; open $THR_RES_FH, "<&=$THR_RES_FD" or die "FATAL: cannot fdopen";

our $ICMP4_FH;
our $ICMP6_FH;

our @IDLE_CB;

AnyEvent::post_detect {
   our $ICMP4_W = $ICMP4_FD >= 0 && (open $ICMP4_FH, "<&=$ICMP4_FD") && AE::io $ICMP4_FH, 0, \&_recv_icmp4;
   our $ICMP6_W = $ICMP6_FD >= 0 && (open $ICMP6_FH, "<&=$ICMP6_FD") && AE::io $ICMP6_FH, 0, \&_recv_icmp6;

   our $THR_RES_W = AE::io $THR_RES_FH, 0, sub {
      sysread $THR_RES_FH, my $buf, 8;

      for my $id (unpack "S*", $buf) {
         _stop_id $id;
         ($IDLE_CB[$id] || sub { })->();
      }
   };
};

=item AnyEvent::FastPing::ipv4_supported

Returns true iff IPv4 is supported in this module and on this system.

=item AnyEvent::FastPing::ipv6_supported

Returns true iff IPv6 is supported in this module and on this system.

=item AnyEvent::FastPing::icmp4_pktsize

Returns the number of octets per IPv4 ping packet (the whole IP packet
including headers, excluding lower-level headers or trailers such as
Ethernet).

Can be used to calculate e.g. octets/s from rate ...

   my $octets_per_second = $packets_per_second * AnyEvent::FastPing::icmp4_pktsize;

... or convert kilobit/second to packet rate ...

   my $packets_per_second = $kilobit_per_second
                            * (1000 / 8 / AnyEvent::FastPing::icmp4_pktsize);

etc.

=item AnyEvent::FastPing::icmp6_pktsize

Like AnyEvent::FastPing::icmp4_pktsize, but for IPv6.

=back

=head1 THE AnyEvent::FastPing CLASS

The AnyEvent::FastPing class represents a single "pinger". A "pinger"
comes with its own thread to send packets in the background, a rate-limit
machinery and separate idle/receive callbacks.

The recommended workflow (there are others) is this: 1. create a new
AnyEvent::FastPing object 2. configure the address lists and ranges to
ping, also configure an idle callback and optionally a receive callback
3. C<start> the pinger.

When the pinger has finished pinging all the configured addresses it will
call the idle callback.

The pinging process works like this: every range has a minimum interval
between sends, which is used to limit the rate at which hosts in that
range are being pinged. Distinct ranges are independent of each other,
which is why there is a per-pinger "global" minimum interval as well.

The pinger sends pings as fats as possible, while both obeying the pinger
rate limit as well as range limits.

When a range is exhausted, it is removed. When all ranges are exhausted,
the pinger waits another C<max_rtt> seconds and then exits, causing the
idle callback to trigger.

Performance: On my 2 GHz Opteron system with a pretty average nvidia
gigabit network card I can ping around 60k to 200k addresses per second,
depending on routing decisions.

Example: ping 10.0.0.1-10.0.0.15 with at most 100 packets/s, and
11.0.0.1-11.0.255.255 with at most 1000 packets/s. Also ping the IPv6
loopback address 5 times as fast as possible. Do not, however, exceed 1000
packets/s overall. Also dump each received reply.

   use AnyEvent::Socket;
   use AnyEvent::FastPing;

   my $done = AnyEvent->condvar;

   my $pinger = new AnyEvent::FastPing;

   $pinger->interval (1/1000);
   $pinger->max_rtt (0.1); # reasonably fast/reliable network

   $pinger->add_range (v10.0.0.1, v10.0.0.15, 1/100);
   $pinger->add_range (v11.0.0.1, v11.0.255.255, 1/1000);
   $pinger->add_hosts ([ (v0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1) x 5 ]);

   $pinger->on_recv (sub {
      for (@{ $_[0] }) {
         printf "%s %g\n", (AnyEvent::Socket::format_address $_->[0]), $_->[1];
      }
   });

   $pinger->on_idle (sub {
      print "done\n";
      undef $pinger;
   });

   $pinger->start;
   $done->wait;

=head2 METHODS

=over 4

=item $pinger = new AnyEvent::FastPing

Creates a new pinger - right now there can be at most C<65536> pingers in
a process, although that limit might change to something drastically lower
- you should be stingy with your pinger objects.

=cut

sub new {
   my ($klass) = @_;

   AnyEvent::detect unless defined $AnyEvent::MODEL;

   _new $klass, (rand 65536), (rand 65536), (rand 65536)
}

sub DESTROY {
   undef $IDLE_CB[ &id ];
   &_free;
}

=item $pinger->on_recv ($callback->([[$host, $rtt], ...]))

Registers a callback to be called for ping replies. If no callback has
been registered than ping replies will be ignored, otherwise this module
calculates the round trip time, in seconds, for each reply and calls this
callback.

The callback receives a single argument, which is an array reference
with an entry for each reply packet (the replies will be batched for
efficiency). Each member in the array reference is again an array
reference with exactly two members: the binary host address (4 octets for
IPv4, 16 for IPv6) and the approximate round trip time, in seconds.

The replies will be passed to the callback as soon as they arrive, and
this callback can be called many times with batches of replies.

The receive callback will be called whenever a suitable reply arrives,
whether generated by this pinger or not, whether this pinger is started
or not. The packets will have a unique 64 bit ID to distinguish them from
other pinger objects and other generators, but this doesn't help against
malicious replies.

Note that very high packet rates can overwhelm your process, causing
replies to be dropped (configure your kernel with long receive queues for
raw sockets if this is a problem).

Example: register a callback which simply dumps the received data.

   use AnyEvent::Socket;

   $pinger->on_recv (sub {
      for (@{ $_[0] }) {
         printf "%s %g\n", (AnyEvent::Socket::format_address $_->[0]), $_->[1];
      }
   });

Example: a single ping reply with payload of 1 from C<::1> gets passed
like this:

   [
      [ "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1", 0.000280141830444336 ]
   ]

Example: ping replies for C<127.0.0.1> and C<127.0.0.2>:

   [
      [ "\177\0\0\1", 0.00015711784362793 ],
      [ "\177\0\0\2", 0.00090184211731 ]
   ]

=item $pinger->on_idle ($callback->())

Registers a callback to be called when the pinger becomes I<idle>, that
is, it has been started, has exhausted all ping ranges and waited for
the C<max_rtt> time. An idle pinger is also stopped, so the callback can
instantly add new ranges, if it so desires.

=cut

sub on_idle {
   $IDLE_CB[ &id ] = $_[1];
}

=item $pinger->interval ($seconds)

Configures the minimum interval between packet sends for this pinger - the
pinger will not send packets faster than this rate (or actually 1 / rate),
even if individual ranges have a lower interval.

A value of C<0> selects the fastest possible speed (currently no faster
than 1_000_000 packets/s).

=item $pinger->max_rtt ($seconds)

If your idle callback were called instantly after all ranges were
exhausted and you destroyed the object inside (which is common), then
there would be no chance to receive some replies, as there would be no
time of the packet to travel over the network.

This can be fixed by starting a timer in the idle callback, or more simply
by selecting a suitable C<max_rtt> value, which should be the maximum time
you allow a ping packet to travel to its destination and back.

The pinger thread automatically waits for this amount of time before becoming idle.

The default is currently C<0.5> seconds, which is usually plenty.

=item $pinger->add_range ($lo, $hi[, $interval])

Ping the IPv4 (or IPv6, but see below) address range, starting at binary
address C<$lo> and ending at C<$hi> (both C<$lo> and C<$hi> will be
pinged), generating no more than one ping per C<$interval> seconds (or as
fast as possible if omitted).

You can convert IP addresses from text to binary form by
using C<AnyEvent::Util::parse_address>, C<Socket::inet_aton>,
C<Socket6::inet_pton> or any other method that you like :)

The algorithm to select the next address is O(log n) on the number of
ranges, so even a large number of ranges (many thousands) is manageable.

No storage is allocated per address.

Note that, while IPv6 addresses are currently supported, the usefulness of
this option is extremely limited and might be gone in future versions - if
you want to ping a number of IPv6 hosts, better specify them individually
using the C<add_hosts> method.

=item $pinger->add_hosts ([$host...], $interval, $interleave)

Similar to C<add_range>, but uses a list of single addresses instead. The
list is specified as an array reference as first argument. Each entry in
the array should be a binary host address, either IPv4 or IPv6. If all
addresses are IPv4 addresses, then a compact IPv4-only format will be used
to store the list internally.

Minimum C<$interval> is the same as for C<add_range> and can be left out.

C<$interlave> specifies an increment between addresses: often address
lists are generated in a way that results in clustering - first all
addresses from one subnet, then from the next, and so on. To avoid this,
you can specify an interleave factor. If it is C<1> (the default), then
every address is pinged in the order specified. If it is C<2>, then only
every second address will be pinged in the first round, followed by a
second round with the others. Higher factors will create C<$interleave>
runs of addresses spaced C<$interleave> indices in the list.

The special value C<0> selects a (hopefully) suitable interleave factor
automatically - currently C<256> for lists with less than 65536 addresses,
and the square root of the list length otherwise.

=item $pinger->start

Start the pinger, unless it is running already. While a pinger is running
you must not modify the pinger. If you want to change a parameter, you
have to C<stop> the pinger first.

The pinger will automatically stop when destroyed.

=item $pinger->stop

Stop the pinger, if it is running. A pinger can be stopped at any time,
after which it's current state is preserved - starting it again will
continue where it left off.

=cut

1;

=back

=head1 AUTHOR

   Marc Lehmann <schmorp@schmorp.de>
   http://home.schmorp.de/

=head1 LICENSE

   This software is distributed under the GENERAL PUBLIC LICENSE, version 2
   or any later version or, at your option, the Artistic License.

=cut

Revision:	1.18
Committed:	Fri Nov 11 01:18:56 2016 UTC (7 years, 6 months ago) by root
Branch:	MAIN
CVS Tags:	rel-2_03
Changes since 1.17:	+21 -14 lines
Log Message:	rel-2_03
#	User	Rev	Content
1	root	1.1	=head1 NAME
2
3			AnyEvent::FastPing - quickly ping a large number of hosts
4
5			=head1 SYNOPSIS
6
7			use AnyEvent::FastPing;
8
9			=head1 DESCRIPTION
10
11	root	1.2	This module was written for a single purpose only: sending ICMP ECHO
12	root	1.1	REQUEST packets as quickly as possible to a large number of hosts
13			(thousands to millions).
14
15	root	1.14	It employs a separate thread and is fully event-driven (using AnyEvent),
16			so you have to run an event model supported by AnyEvent to use this
17			module.
18	root	1.1
19			=head1 FUNCTIONS
20
21			=over 4
22
23			=cut
24
25			package AnyEvent::FastPing;
26
27	root	1.9	use common::sense;
28	root	1.1
29			use AnyEvent;
30
31			BEGIN {
32	root	1.18	our $VERSION = 2.03;
33	root	1.1	our @ISA = qw(Exporter);
34
35			require Exporter;
36			#Exporter::export_ok_tags (keys %EXPORT_TAGS);
37
38			require XSLoader;
39			XSLoader::load (__PACKAGE__, $VERSION);
40			}
41
42	root	1.11	our ($THR_RES_FD, $ICMP4_FD, $ICMP6_FD);
43	root	1.1
44			our $THR_RES_FH; open $THR_RES_FH, "<&=$THR_RES_FD" or die "FATAL: cannot fdopen";
45
46	root	1.18	our $ICMP4_FH;
47			our $ICMP6_FH;
48
49			our @IDLE_CB;
50
51			AnyEvent::post_detect {
52			our $ICMP4_W = $ICMP4_FD >= 0 && (open $ICMP4_FH, "<&=$ICMP4_FD") && AE::io $ICMP4_FH, 0, \&_recv_icmp4;
53			our $ICMP6_W = $ICMP6_FD >= 0 && (open $ICMP6_FH, "<&=$ICMP6_FD") && AE::io $ICMP6_FH, 0, \&_recv_icmp6;
54
55			our $THR_RES_W = AE::io $THR_RES_FH, 0, sub {
56			sysread $THR_RES_FH, my $buf, 8;
57
58			for my $id (unpack "S*", $buf) {
59			_stop_id $id;
60			($IDLE_CB[$id] \|\| sub { })->();
61			}
62			};
63			};
64	root	1.1
65			=item AnyEvent::FastPing::ipv4_supported
66
67	root	1.13	Returns true iff IPv4 is supported in this module and on this system.
68	root	1.1
69			=item AnyEvent::FastPing::ipv6_supported
70
71	root	1.13	Returns true iff IPv6 is supported in this module and on this system.
72	root	1.1
73			=item AnyEvent::FastPing::icmp4_pktsize
74
75	root	1.13	Returns the number of octets per IPv4 ping packet (the whole IP packet
76			including headers, excluding lower-level headers or trailers such as
77			Ethernet).
78
79			Can be used to calculate e.g. octets/s from rate ...
80
81			my $octets_per_second = $packets_per_second * AnyEvent::FastPing::icmp4_pktsize;
82
83			... or convert kilobit/second to packet rate ...
84
85			my $packets_per_second = $kilobit_per_second
86			* (1000 / 8 / AnyEvent::FastPing::icmp4_pktsize);
87
88			etc.
89	root	1.1
90			=item AnyEvent::FastPing::icmp6_pktsize
91
92	root	1.13	Like AnyEvent::FastPing::icmp4_pktsize, but for IPv6.
93
94			=back
95
96			=head1 THE AnyEvent::FastPing CLASS
97
98			The AnyEvent::FastPing class represents a single "pinger". A "pinger"
99			comes with its own thread to send packets in the background, a rate-limit
100			machinery and separate idle/receive callbacks.
101
102			The recommended workflow (there are others) is this: 1. create a new
103			AnyEvent::FastPing object 2. configure the address lists and ranges to
104			ping, also configure an idle callback and optionally a receive callback
105			3. C<start> the pinger.
106
107			When the pinger has finished pinging all the configured addresses it will
108			call the idle callback.
109
110			The pinging process works like this: every range has a minimum interval
111			between sends, which is used to limit the rate at which hosts in that
112			range are being pinged. Distinct ranges are independent of each other,
113			which is why there is a per-pinger "global" minimum interval as well.
114
115			The pinger sends pings as fats as possible, while both obeying the pinger
116			rate limit as well as range limits.
117
118			When a range is exhausted, it is removed. When all ranges are exhausted,
119			the pinger waits another C<max_rtt> seconds and then exits, causing the
120			idle callback to trigger.
121
122			Performance: On my 2 GHz Opteron system with a pretty average nvidia
123	root	1.14	gigabit network card I can ping around 60k to 200k addresses per second,
124	root	1.13	depending on routing decisions.
125
126			Example: ping 10.0.0.1-10.0.0.15 with at most 100 packets/s, and
127			11.0.0.1-11.0.255.255 with at most 1000 packets/s. Also ping the IPv6
128			loopback address 5 times as fast as possible. Do not, however, exceed 1000
129	root	1.14	packets/s overall. Also dump each received reply.
130	root	1.13
131			use AnyEvent::Socket;
132			use AnyEvent::FastPing;
133
134			my $done = AnyEvent->condvar;
135
136			my $pinger = new AnyEvent::FastPing;
137
138			$pinger->interval (1/1000);
139			$pinger->max_rtt (0.1); # reasonably fast/reliable network
140
141			$pinger->add_range (v10.0.0.1, v10.0.0.15, 1/100);
142			$pinger->add_range (v11.0.0.1, v11.0.255.255, 1/1000);
143			$pinger->add_hosts ([ (v0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1) x 5 ]);
144
145			$pinger->on_recv (sub {
146			for (@{ $_[0] }) {
147			printf "%s %g\n", (AnyEvent::Socket::format_address $_->[0]), $_->[1];
148			}
149			});
150
151			$pinger->on_idle (sub {
152			print "done\n";
153			undef $pinger;
154			});
155
156			$pinger->start;
157			$done->wait;
158
159			=head2 METHODS
160
161			=over 4
162
163			=item $pinger = new AnyEvent::FastPing
164
165			Creates a new pinger - right now there can be at most C<65536> pingers in
166	root	1.14	a process, although that limit might change to something drastically lower
167			- you should be stingy with your pinger objects.
168	root	1.1
169	root	1.11	=cut
170
171			sub new {
172			my ($klass) = @_;
173
174	root	1.18	AnyEvent::detect unless defined $AnyEvent::MODEL;
175
176	root	1.11	_new $klass, (rand 65536), (rand 65536), (rand 65536)
177			}
178
179			sub DESTROY {
180			undef $IDLE_CB[ &id ];
181			&_free;
182			}
183
184	root	1.13	=item $pinger->on_recv ($callback->([[$host, $rtt], ...]))
185
186	root	1.14	Registers a callback to be called for ping replies. If no callback has
187	root	1.13	been registered than ping replies will be ignored, otherwise this module
188			calculates the round trip time, in seconds, for each reply and calls this
189			callback.
190
191			The callback receives a single argument, which is an array reference
192			with an entry for each reply packet (the replies will be batched for
193			efficiency). Each member in the array reference is again an array
194	root	1.14	reference with exactly two members: the binary host address (4 octets for
195	root	1.13	IPv4, 16 for IPv6) and the approximate round trip time, in seconds.
196
197			The replies will be passed to the callback as soon as they arrive, and
198			this callback can be called many times with batches of replies.
199
200			The receive callback will be called whenever a suitable reply arrives,
201			whether generated by this pinger or not, whether this pinger is started
202			or not. The packets will have a unique 64 bit ID to distinguish them from
203			other pinger objects and other generators, but this doesn't help against
204			malicious replies.
205
206			Note that very high packet rates can overwhelm your process, causing
207			replies to be dropped (configure your kernel with long receive queues for
208			raw sockets if this is a problem).
209
210			Example: register a callback which simply dumps the received data.
211
212			use AnyEvent::Socket;
213
214			$pinger->on_recv (sub {
215			for (@{ $_[0] }) {
216			printf "%s %g\n", (AnyEvent::Socket::format_address $_->[0]), $_->[1];
217			}
218	root	1.14	});
219	root	1.13
220			Example: a single ping reply with payload of 1 from C<::1> gets passed
221			like this:
222
223			[
224			[ "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1", 0.000280141830444336 ]
225			]
226
227			Example: ping replies for C<127.0.0.1> and C<127.0.0.2>:
228
229			[
230			[ "\177\0\0\1", 0.00015711784362793 ],
231			[ "\177\0\0\2", 0.00090184211731 ]
232			]
233
234			=item $pinger->on_idle ($callback->())
235
236			Registers a callback to be called when the pinger becomes I<idle>, that
237			is, it has been started, has exhausted all ping ranges and waited for
238			the C<max_rtt> time. An idle pinger is also stopped, so the callback can
239			instantly add new ranges, if it so desires.
240
241			=cut
242
243	root	1.11	sub on_idle {
244			$IDLE_CB[ &id ] = $_[1];
245			}
246
247	root	1.13	=item $pinger->interval ($seconds)
248	root	1.12
249	root	1.13	Configures the minimum interval between packet sends for this pinger - the
250	root	1.14	pinger will not send packets faster than this rate (or actually 1 / rate),
251	root	1.13	even if individual ranges have a lower interval.
252	root	1.11
253	root	1.14	A value of C<0> selects the fastest possible speed (currently no faster
254	root	1.13	than 1_000_000 packets/s).
255	root	1.1
256	root	1.13	=item $pinger->max_rtt ($seconds)
257	root	1.1
258	root	1.13	If your idle callback were called instantly after all ranges were
259			exhausted and you destroyed the object inside (which is common), then
260			there would be no chance to receive some replies, as there would be no
261	root	1.14	time of the packet to travel over the network.
262	root	1.1
263	root	1.13	This can be fixed by starting a timer in the idle callback, or more simply
264			by selecting a suitable C<max_rtt> value, which should be the maximum time
265	root	1.14	you allow a ping packet to travel to its destination and back.
266	root	1.1
267	root	1.13	The pinger thread automatically waits for this amount of time before becoming idle.
268	root	1.1
269	root	1.13	The default is currently C<0.5> seconds, which is usually plenty.
270	root	1.1
271	root	1.13	=item $pinger->add_range ($lo, $hi[, $interval])
272	root	1.1
273	root	1.13	Ping the IPv4 (or IPv6, but see below) address range, starting at binary
274			address C<$lo> and ending at C<$hi> (both C<$lo> and C<$hi> will be
275			pinged), generating no more than one ping per C<$interval> seconds (or as
276			fast as possible if omitted).
277	root	1.1
278	root	1.13	You can convert IP addresses from text to binary form by
279			using C<AnyEvent::Util::parse_address>, C<Socket::inet_aton>,
280			C<Socket6::inet_pton> or any other method that you like :)
281	root	1.1
282	root	1.13	The algorithm to select the next address is O(log n) on the number of
283	root	1.14	ranges, so even a large number of ranges (many thousands) is manageable.
284	root	1.1
285	root	1.13	No storage is allocated per address.
286	root	1.1
287	root	1.13	Note that, while IPv6 addresses are currently supported, the usefulness of
288			this option is extremely limited and might be gone in future versions - if
289			you want to ping a number of IPv6 hosts, better specify them individually
290			using the C<add_hosts> method.
291
292			=item $pinger->add_hosts ([$host...], $interval, $interleave)
293
294			Similar to C<add_range>, but uses a list of single addresses instead. The
295			list is specified as an array reference as first argument. Each entry in
296	root	1.15	the array should be a binary host address, either IPv4 or IPv6. If all
297			addresses are IPv4 addresses, then a compact IPv4-only format will be used
298			to store the list internally.
299	root	1.13
300			Minimum C<$interval> is the same as for C<add_range> and can be left out.
301
302			C<$interlave> specifies an increment between addresses: often address
303			lists are generated in a way that results in clustering - first all
304			addresses from one subnet, then from the next, and so on. To avoid this,
305			you can specify an interleave factor. If it is C<1> (the default), then
306			every address is pinged in the order specified. If it is C<2>, then only
307			every second address will be pinged in the first round, followed by a
308			second round with the others. Higher factors will create C<$interleave>
309			runs of addresses spaced C<$interleave> indices in the list.
310
311			The special value C<0> selects a (hopefully) suitable interleave factor
312			automatically - currently C<256> for lists with less than 65536 addresses,
313			and the square root of the list length otherwise.
314
315			=item $pinger->start
316
317			Start the pinger, unless it is running already. While a pinger is running
318			you must not modify the pinger. If you want to change a parameter, you
319			have to C<stop> the pinger first.
320
321			The pinger will automatically stop when destroyed.
322
323			=item $pinger->stop
324
325			Stop the pinger, if it is running. A pinger can be stopped at any time,
326			after which it's current state is preserved - starting it again will
327			continue where it left off.
328	root	1.1
329			=cut
330
331			1;
332
333			=back
334
335			=head1 AUTHOR
336
337	root	1.4	Marc Lehmann <schmorp@schmorp.de>
338			http://home.schmorp.de/
339	root	1.1
340	root	1.4	=head1 LICENSE
341	root	1.1
342	root	1.4	This software is distributed under the GENERAL PUBLIC LICENSE, version 2
343			or any later version or, at your option, the Artistic License.
344	root	1.1
345			=cut
346