AnyEvent-FastPing/FastPing.pm

=head1 NAME

AnyEvent::FastPing - quickly ping a large number of hosts

=head1 SYNOPSIS

 use AnyEvent::FastPing;

=head1 DESCRIPTION

This module was written for a single purpose only: sending ICMP ECHO
REQUEST packets as quickly as possible to a large number of hosts
(thousands to millions).

It employs a separate thread and is fully event-driven (using AnyEvent),
so you have to run an event model supported by AnyEvent to use this
module.

=head1 FUNCTIONS

=over 4

=cut

package AnyEvent::FastPing;

use common::sense;

use AnyEvent;

BEGIN {
   our $VERSION = 2.1;
   our @ISA = qw(Exporter);

   require Exporter;
   #Exporter::export_ok_tags (keys %EXPORT_TAGS);

   require XSLoader;
   XSLoader::load (__PACKAGE__, $VERSION);
}

our ($THR_RES_FD, $ICMP4_FD, $ICMP6_FD);

our $THR_RES_FH;

our $ICMP4_FH;
our $ICMP6_FH;

our @IDLE_CB;

=item AnyEvent::FastPing::ipv4_supported

Returns true iff IPv4 is supported in this module and on this system.

=item AnyEvent::FastPing::ipv6_supported

Returns true iff IPv6 is supported in this module and on this system.

=item AnyEvent::FastPing::icmp4_pktsize

Returns the number of octets per IPv4 ping packet (the whole IP packet
including headers, excluding lower-level headers or trailers such as
Ethernet).

Can be used to calculate e.g. octets/s from rate ...

   my $octets_per_second = $packets_per_second * AnyEvent::FastPing::icmp4_pktsize;

... or convert kilobit/second to packet rate ...

   my $packets_per_second = $kilobit_per_second
                            * (1000 / 8 / AnyEvent::FastPing::icmp4_pktsize);

etc.

=item AnyEvent::FastPing::icmp6_pktsize

Like AnyEvent::FastPing::icmp4_pktsize, but for IPv6.

=back

=head1 THE AnyEvent::FastPing CLASS

The AnyEvent::FastPing class represents a single "pinger". A "pinger"
comes with its own thread to send packets in the background, a rate-limit
machinery and separate idle/receive callbacks.

The recommended workflow (there are others) is this: 1. create a new
AnyEvent::FastPing object 2. configure the address lists and ranges to
ping, also configure an idle callback and optionally a receive callback
3. C<start> the pinger.

When the pinger has finished pinging all the configured addresses it will
call the idle callback.

The pinging process works like this: every range has a minimum interval
between sends, which is used to limit the rate at which hosts in that
range are being pinged. Distinct ranges are independent of each other,
which is why there is a per-pinger "global" minimum interval as well.

The pinger sends pings as fats as possible, while both obeying the pinger
rate limit as well as range limits.

When a range is exhausted, it is removed. When all ranges are exhausted,
the pinger waits another C<max_rtt> seconds and then exits, causing the
idle callback to trigger.

Performance: On my 2 GHz Opteron system with a pretty average nvidia
gigabit network card I can ping around 60k to 200k addresses per second,
depending on routing decisions.

Example: ping 10.0.0.1-10.0.0.15 with at most 100 packets/s, and
11.0.0.1-11.0.255.255 with at most 1000 packets/s. Also ping the IPv6
loopback address 5 times as fast as possible. Do not, however, exceed 1000
packets/s overall. Also dump each received reply.

   use AnyEvent::Socket;
   use AnyEvent::FastPing;

   my $done = AnyEvent->condvar;

   my $pinger = new AnyEvent::FastPing;

   $pinger->interval (1/1000);
   $pinger->max_rtt (0.1); # reasonably fast/reliable network

   $pinger->add_range (v10.0.0.1, v10.0.0.15, 1/100);
   $pinger->add_range (v11.0.0.1, v11.0.255.255, 1/1000);
   $pinger->add_hosts ([ (v0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1) x 5 ]);

   $pinger->on_recv (sub {
      for (@{ $_[0] }) {
         printf "%s %g\n", (AnyEvent::Socket::format_address $_->[0]), $_->[1];
      }
   });

   $pinger->on_idle (sub {
      print "done\n";
      undef $pinger;
   });

   $pinger->start;
   $done->wait;

=head2 METHODS

=over 4

=item $pinger = new AnyEvent::FastPing

Creates a new pinger - right now there can be at most C<65536> pingers in
a process, although that limit might change to something drastically lower
- you should be stingy with your pinger objects.

=cut

sub new {
   _boot;

   our $ICMP4_W = $ICMP4_FD >= 0 && (open $ICMP4_FH, "<&=$ICMP4_FD") && AE::io $ICMP4_FH, 0, \&_recv_icmp4;
   our $ICMP6_W = $ICMP6_FD >= 0 && (open $ICMP6_FH, "<&=$ICMP6_FD") && AE::io $ICMP6_FH, 0, \&_recv_icmp6;

   open $THR_RES_FH, "<&=$THR_RES_FD" or die "AnyEvent::FastPing: FATAL: cannot fdopen thread result fd";

   our $THR_RES_W = AE::io $THR_RES_FH, 0, sub {
      sysread $THR_RES_FH, my $buf, 8;

      for my $id (unpack "S*", $buf) {
         _stop_id $id;
         ($IDLE_CB[$id] || sub { })->();
      }
   };

   *new = sub {
      _new shift, (rand 65536), (rand 65536), (rand 65536)
   };

   goto &new;
}

sub DESTROY {
   undef $IDLE_CB[ &id ];
   &_free;
}

=item $pinger->on_recv ($callback->([[$host, $rtt], ...]))

Registers a callback to be called for ping replies. If no callback has
been registered than ping replies will be ignored, otherwise this module
calculates the round trip time, in seconds, for each reply and calls this
callback.

The callback receives a single argument, which is an array reference
with an entry for each reply packet (the replies will be batched for
efficiency). Each member in the array reference is again an array
reference with exactly two members: the binary host address (4 octets for
IPv4, 16 for IPv6) and the approximate round trip time, in seconds.

The replies will be passed to the callback as soon as they arrive, and
this callback can be called many times with batches of replies.

The receive callback will be called whenever a suitable reply arrives,
whether generated by this pinger or not, whether this pinger is started
or not. The packets will have a unique 64 bit ID to distinguish them from
other pinger objects and other generators, but this doesn't help against
malicious replies.

Note that very high packet rates can overwhelm your process, causing
replies to be dropped (configure your kernel with long receive queues for
raw sockets if this is a problem).

Example: register a callback which simply dumps the received data.

   use AnyEvent::Socket;

   $pinger->on_recv (sub {
      for (@{ $_[0] }) {
         printf "%s %g\n", (AnyEvent::Socket::format_address $_->[0]), $_->[1];
      }
   });

Example: a single ping reply with payload of 1 from C<::1> gets passed
like this:

   [
      [ "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1", 0.000280141830444336 ]
   ]

Example: ping replies for C<127.0.0.1> and C<127.0.0.2>:

   [
      [ "\177\0\0\1", 0.00015711784362793 ],
      [ "\177\0\0\2", 0.00090184211731 ]
   ]

=item $pinger->on_idle ($callback->())

Registers a callback to be called when the pinger becomes I<idle>, that
is, it has been started, has exhausted all ping ranges and waited for
the C<max_rtt> time. An idle pinger is also stopped, so the callback can
instantly add new ranges, if it so desires.

=cut

sub on_idle {
   $IDLE_CB[ &id ] = $_[1];
}

=item $pinger->interval ($seconds)

Configures the minimum interval between packet sends for this pinger - the
pinger will not send packets faster than this rate (or actually 1 / rate),
even if individual ranges have a lower interval.

A value of C<0> selects the fastest possible speed (currently no faster
than 1_000_000 packets/s).

=item $pinger->max_rtt ($seconds)

If your idle callback were called instantly after all ranges were
exhausted and you destroyed the object inside (which is common), then
there would be no chance to receive some replies, as there would be no
time of the packet to travel over the network.

This can be fixed by starting a timer in the idle callback, or more simply
by selecting a suitable C<max_rtt> value, which should be the maximum time
you allow a ping packet to travel to its destination and back.

The pinger thread automatically waits for this amount of time before becoming idle.

The default is currently C<0.5> seconds, which is usually plenty.

=item $pinger->add_range ($lo, $hi[, $interval])

Ping the IPv4 (or IPv6, but see below) address range, starting at binary
address C<$lo> and ending at C<$hi> (both C<$lo> and C<$hi> will be
pinged), generating no more than one ping per C<$interval> seconds (or as
fast as possible if omitted).

You can convert IP addresses from text to binary form by
using C<AnyEvent::Util::parse_address>, C<Socket::inet_aton>,
C<Socket6::inet_pton> or any other method that you like :)

The algorithm to select the next address is O(log n) on the number of
ranges, so even a large number of ranges (many thousands) is manageable.

No storage is allocated per address.

Note that, while IPv6 addresses are currently supported, the usefulness of
this option is extremely limited and might be gone in future versions - if
you want to ping a number of IPv6 hosts, better specify them individually
using the C<add_hosts> method.

=item $pinger->add_hosts ([$host...], $interval, $interleave)

Similar to C<add_range>, but uses a list of single addresses instead. The
list is specified as an array reference as first argument. Each entry in
the array should be a binary host address, either IPv4 or IPv6. If all
addresses are IPv4 addresses, then a compact IPv4-only format will be used
to store the list internally.

Minimum C<$interval> is the same as for C<add_range> and can be left out.

C<$interlave> specifies an increment between addresses: often address
lists are generated in a way that results in clustering - first all
addresses from one subnet, then from the next, and so on. To avoid this,
you can specify an interleave factor. If it is C<1> (the default), then
every address is pinged in the order specified. If it is C<2>, then only
every second address will be pinged in the first round, followed by a
second round with the others. Higher factors will create C<$interleave>
runs of addresses spaced C<$interleave> indices in the list.

The special value C<0> selects a (hopefully) suitable interleave factor
automatically - currently C<256> for lists with less than 65536 addresses,
and the square root of the list length otherwise.

=item $pinger->start

Start the pinger, unless it is running already. While a pinger is running
you must not modify the pinger. If you want to change a parameter, you
have to C<stop> the pinger first.

The pinger will automatically stop when destroyed.

=item $pinger->stop

Stop the pinger, if it is running. A pinger can be stopped at any time,
after which it's current state is preserved - starting it again will
continue where it left off.

=cut

1;

=back

=head1 AUTHOR

   Marc Lehmann <schmorp@schmorp.de>
   http://home.schmorp.de/

=head1 LICENSE

   This software is distributed under the GENERAL PUBLIC LICENSE, version 2
   or any later version or, at your option, the Artistic License.

=cut

Revision:	1.19
Committed:	Sat Nov 12 01:20:46 2016 UTC (7 years, 6 months ago) by root
Branch:	MAIN
CVS Tags:	rel-2_1, HEAD
Changes since 1.18:	+21 -19 lines
Log Message:	2.1
#	User	Rev	Content
1	root	1.1	=head1 NAME
2
3			AnyEvent::FastPing - quickly ping a large number of hosts
4
5			=head1 SYNOPSIS
6
7			use AnyEvent::FastPing;
8
9			=head1 DESCRIPTION
10
11	root	1.2	This module was written for a single purpose only: sending ICMP ECHO
12	root	1.1	REQUEST packets as quickly as possible to a large number of hosts
13			(thousands to millions).
14
15	root	1.14	It employs a separate thread and is fully event-driven (using AnyEvent),
16			so you have to run an event model supported by AnyEvent to use this
17			module.
18	root	1.1
19			=head1 FUNCTIONS
20
21			=over 4
22
23			=cut
24
25			package AnyEvent::FastPing;
26
27	root	1.9	use common::sense;
28	root	1.1
29			use AnyEvent;
30
31			BEGIN {
32	root	1.19	our $VERSION = 2.1;
33	root	1.1	our @ISA = qw(Exporter);
34
35			require Exporter;
36			#Exporter::export_ok_tags (keys %EXPORT_TAGS);
37
38			require XSLoader;
39			XSLoader::load (__PACKAGE__, $VERSION);
40			}
41
42	root	1.11	our ($THR_RES_FD, $ICMP4_FD, $ICMP6_FD);
43	root	1.1
44	root	1.19	our $THR_RES_FH;
45	root	1.1
46	root	1.18	our $ICMP4_FH;
47			our $ICMP6_FH;
48
49			our @IDLE_CB;
50
51	root	1.1	=item AnyEvent::FastPing::ipv4_supported
52
53	root	1.13	Returns true iff IPv4 is supported in this module and on this system.
54	root	1.1
55			=item AnyEvent::FastPing::ipv6_supported
56
57	root	1.13	Returns true iff IPv6 is supported in this module and on this system.
58	root	1.1
59			=item AnyEvent::FastPing::icmp4_pktsize
60
61	root	1.13	Returns the number of octets per IPv4 ping packet (the whole IP packet
62			including headers, excluding lower-level headers or trailers such as
63			Ethernet).
64
65			Can be used to calculate e.g. octets/s from rate ...
66
67			my $octets_per_second = $packets_per_second * AnyEvent::FastPing::icmp4_pktsize;
68
69			... or convert kilobit/second to packet rate ...
70
71			my $packets_per_second = $kilobit_per_second
72			* (1000 / 8 / AnyEvent::FastPing::icmp4_pktsize);
73
74			etc.
75	root	1.1
76			=item AnyEvent::FastPing::icmp6_pktsize
77
78	root	1.13	Like AnyEvent::FastPing::icmp4_pktsize, but for IPv6.
79
80			=back
81
82			=head1 THE AnyEvent::FastPing CLASS
83
84			The AnyEvent::FastPing class represents a single "pinger". A "pinger"
85			comes with its own thread to send packets in the background, a rate-limit
86			machinery and separate idle/receive callbacks.
87
88			The recommended workflow (there are others) is this: 1. create a new
89			AnyEvent::FastPing object 2. configure the address lists and ranges to
90			ping, also configure an idle callback and optionally a receive callback
91			3. C<start> the pinger.
92
93			When the pinger has finished pinging all the configured addresses it will
94			call the idle callback.
95
96			The pinging process works like this: every range has a minimum interval
97			between sends, which is used to limit the rate at which hosts in that
98			range are being pinged. Distinct ranges are independent of each other,
99			which is why there is a per-pinger "global" minimum interval as well.
100
101			The pinger sends pings as fats as possible, while both obeying the pinger
102			rate limit as well as range limits.
103
104			When a range is exhausted, it is removed. When all ranges are exhausted,
105			the pinger waits another C<max_rtt> seconds and then exits, causing the
106			idle callback to trigger.
107
108			Performance: On my 2 GHz Opteron system with a pretty average nvidia
109	root	1.14	gigabit network card I can ping around 60k to 200k addresses per second,
110	root	1.13	depending on routing decisions.
111
112			Example: ping 10.0.0.1-10.0.0.15 with at most 100 packets/s, and
113			11.0.0.1-11.0.255.255 with at most 1000 packets/s. Also ping the IPv6
114			loopback address 5 times as fast as possible. Do not, however, exceed 1000
115	root	1.14	packets/s overall. Also dump each received reply.
116	root	1.13
117			use AnyEvent::Socket;
118			use AnyEvent::FastPing;
119
120			my $done = AnyEvent->condvar;
121
122			my $pinger = new AnyEvent::FastPing;
123
124			$pinger->interval (1/1000);
125			$pinger->max_rtt (0.1); # reasonably fast/reliable network
126
127			$pinger->add_range (v10.0.0.1, v10.0.0.15, 1/100);
128			$pinger->add_range (v11.0.0.1, v11.0.255.255, 1/1000);
129			$pinger->add_hosts ([ (v0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1) x 5 ]);
130
131			$pinger->on_recv (sub {
132			for (@{ $_[0] }) {
133			printf "%s %g\n", (AnyEvent::Socket::format_address $_->[0]), $_->[1];
134			}
135			});
136
137			$pinger->on_idle (sub {
138			print "done\n";
139			undef $pinger;
140			});
141
142			$pinger->start;
143			$done->wait;
144
145			=head2 METHODS
146
147			=over 4
148
149			=item $pinger = new AnyEvent::FastPing
150
151			Creates a new pinger - right now there can be at most C<65536> pingers in
152	root	1.14	a process, although that limit might change to something drastically lower
153			- you should be stingy with your pinger objects.
154	root	1.1
155	root	1.11	=cut
156
157			sub new {
158	root	1.19	_boot;
159
160			our $ICMP4_W = $ICMP4_FD >= 0 && (open $ICMP4_FH, "<&=$ICMP4_FD") && AE::io $ICMP4_FH, 0, \&_recv_icmp4;
161			our $ICMP6_W = $ICMP6_FD >= 0 && (open $ICMP6_FH, "<&=$ICMP6_FD") && AE::io $ICMP6_FH, 0, \&_recv_icmp6;
162	root	1.11
163	root	1.19	open $THR_RES_FH, "<&=$THR_RES_FD" or die "AnyEvent::FastPing: FATAL: cannot fdopen thread result fd";
164
165			our $THR_RES_W = AE::io $THR_RES_FH, 0, sub {
166			sysread $THR_RES_FH, my $buf, 8;
167
168			for my $id (unpack "S*", $buf) {
169			_stop_id $id;
170			($IDLE_CB[$id] \|\| sub { })->();
171			}
172			};
173
174			*new = sub {
175			_new shift, (rand 65536), (rand 65536), (rand 65536)
176			};
177	root	1.18
178	root	1.19	goto &new;
179	root	1.11	}
180
181			sub DESTROY {
182			undef $IDLE_CB[ &id ];
183			&_free;
184			}
185
186	root	1.13	=item $pinger->on_recv ($callback->([[$host, $rtt], ...]))
187
188	root	1.14	Registers a callback to be called for ping replies. If no callback has
189	root	1.13	been registered than ping replies will be ignored, otherwise this module
190			calculates the round trip time, in seconds, for each reply and calls this
191			callback.
192
193			The callback receives a single argument, which is an array reference
194			with an entry for each reply packet (the replies will be batched for
195			efficiency). Each member in the array reference is again an array
196	root	1.14	reference with exactly two members: the binary host address (4 octets for
197	root	1.13	IPv4, 16 for IPv6) and the approximate round trip time, in seconds.
198
199			The replies will be passed to the callback as soon as they arrive, and
200			this callback can be called many times with batches of replies.
201
202			The receive callback will be called whenever a suitable reply arrives,
203			whether generated by this pinger or not, whether this pinger is started
204			or not. The packets will have a unique 64 bit ID to distinguish them from
205			other pinger objects and other generators, but this doesn't help against
206			malicious replies.
207
208			Note that very high packet rates can overwhelm your process, causing
209			replies to be dropped (configure your kernel with long receive queues for
210			raw sockets if this is a problem).
211
212			Example: register a callback which simply dumps the received data.
213
214			use AnyEvent::Socket;
215
216			$pinger->on_recv (sub {
217			for (@{ $_[0] }) {
218			printf "%s %g\n", (AnyEvent::Socket::format_address $_->[0]), $_->[1];
219			}
220	root	1.14	});
221	root	1.13
222			Example: a single ping reply with payload of 1 from C<::1> gets passed
223			like this:
224
225			[
226			[ "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1", 0.000280141830444336 ]
227			]
228
229			Example: ping replies for C<127.0.0.1> and C<127.0.0.2>:
230
231			[
232			[ "\177\0\0\1", 0.00015711784362793 ],
233			[ "\177\0\0\2", 0.00090184211731 ]
234			]
235
236			=item $pinger->on_idle ($callback->())
237
238			Registers a callback to be called when the pinger becomes I<idle>, that
239			is, it has been started, has exhausted all ping ranges and waited for
240			the C<max_rtt> time. An idle pinger is also stopped, so the callback can
241			instantly add new ranges, if it so desires.
242
243			=cut
244
245	root	1.11	sub on_idle {
246			$IDLE_CB[ &id ] = $_[1];
247			}
248
249	root	1.13	=item $pinger->interval ($seconds)
250	root	1.12
251	root	1.13	Configures the minimum interval between packet sends for this pinger - the
252	root	1.14	pinger will not send packets faster than this rate (or actually 1 / rate),
253	root	1.13	even if individual ranges have a lower interval.
254	root	1.11
255	root	1.14	A value of C<0> selects the fastest possible speed (currently no faster
256	root	1.13	than 1_000_000 packets/s).
257	root	1.1
258	root	1.13	=item $pinger->max_rtt ($seconds)
259	root	1.1
260	root	1.13	If your idle callback were called instantly after all ranges were
261			exhausted and you destroyed the object inside (which is common), then
262			there would be no chance to receive some replies, as there would be no
263	root	1.14	time of the packet to travel over the network.
264	root	1.1
265	root	1.13	This can be fixed by starting a timer in the idle callback, or more simply
266			by selecting a suitable C<max_rtt> value, which should be the maximum time
267	root	1.14	you allow a ping packet to travel to its destination and back.
268	root	1.1
269	root	1.13	The pinger thread automatically waits for this amount of time before becoming idle.
270	root	1.1
271	root	1.13	The default is currently C<0.5> seconds, which is usually plenty.
272	root	1.1
273	root	1.13	=item $pinger->add_range ($lo, $hi[, $interval])
274	root	1.1
275	root	1.13	Ping the IPv4 (or IPv6, but see below) address range, starting at binary
276			address C<$lo> and ending at C<$hi> (both C<$lo> and C<$hi> will be
277			pinged), generating no more than one ping per C<$interval> seconds (or as
278			fast as possible if omitted).
279	root	1.1
280	root	1.13	You can convert IP addresses from text to binary form by
281			using C<AnyEvent::Util::parse_address>, C<Socket::inet_aton>,
282			C<Socket6::inet_pton> or any other method that you like :)
283	root	1.1
284	root	1.13	The algorithm to select the next address is O(log n) on the number of
285	root	1.14	ranges, so even a large number of ranges (many thousands) is manageable.
286	root	1.1
287	root	1.13	No storage is allocated per address.
288	root	1.1
289	root	1.13	Note that, while IPv6 addresses are currently supported, the usefulness of
290			this option is extremely limited and might be gone in future versions - if
291			you want to ping a number of IPv6 hosts, better specify them individually
292			using the C<add_hosts> method.
293
294			=item $pinger->add_hosts ([$host...], $interval, $interleave)
295
296			Similar to C<add_range>, but uses a list of single addresses instead. The
297			list is specified as an array reference as first argument. Each entry in
298	root	1.15	the array should be a binary host address, either IPv4 or IPv6. If all
299			addresses are IPv4 addresses, then a compact IPv4-only format will be used
300			to store the list internally.
301	root	1.13
302			Minimum C<$interval> is the same as for C<add_range> and can be left out.
303
304			C<$interlave> specifies an increment between addresses: often address
305			lists are generated in a way that results in clustering - first all
306			addresses from one subnet, then from the next, and so on. To avoid this,
307			you can specify an interleave factor. If it is C<1> (the default), then
308			every address is pinged in the order specified. If it is C<2>, then only
309			every second address will be pinged in the first round, followed by a
310			second round with the others. Higher factors will create C<$interleave>
311			runs of addresses spaced C<$interleave> indices in the list.
312
313			The special value C<0> selects a (hopefully) suitable interleave factor
314			automatically - currently C<256> for lists with less than 65536 addresses,
315			and the square root of the list length otherwise.
316
317			=item $pinger->start
318
319			Start the pinger, unless it is running already. While a pinger is running
320			you must not modify the pinger. If you want to change a parameter, you
321			have to C<stop> the pinger first.
322
323			The pinger will automatically stop when destroyed.
324
325			=item $pinger->stop
326
327			Stop the pinger, if it is running. A pinger can be stopped at any time,
328			after which it's current state is preserved - starting it again will
329			continue where it left off.
330	root	1.1
331			=cut
332
333			1;
334
335			=back
336
337			=head1 AUTHOR
338
339	root	1.4	Marc Lehmann <schmorp@schmorp.de>
340			http://home.schmorp.de/
341	root	1.1
342	root	1.4	=head1 LICENSE
343	root	1.1
344	root	1.4	This software is distributed under the GENERAL PUBLIC LICENSE, version 2
345			or any later version or, at your option, the Artistic License.
346	root	1.1
347			=cut
348