ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-SNMP/SNMP.pm
(Generate patch)

Comparing AnyEvent-SNMP/SNMP.pm (file contents):
Revision 1.4 by root, Sun Apr 19 11:06:21 2009 UTC vs.
Revision 1.13 by root, Mon Nov 24 18:16:54 2014 UTC

1=head1 NAME 1=head1 NAME
2 2
3AnyEvent::SNMP - adaptor to integrate Net::SNMP into Anyevent. 3AnyEvent::SNMP - adaptor to integrate Net::SNMP into AnyEvent.
4 4
5=head1 SYNOPSIS 5=head1 SYNOPSIS
6 6
7 use AnyEvent::SNMP; 7 use AnyEvent::SNMP;
8 use Net::SNMP; 8 use Net::SNMP;
… …
23 my @result = $cv->wait; 23 my @result = $cv->wait;
24 24
25=head1 DESCRIPTION 25=head1 DESCRIPTION
26 26
27This module implements an alternative "event dispatcher" for Net::SNMP, 27This module implements an alternative "event dispatcher" for Net::SNMP,
28using AnyEvent as a backend. 28using AnyEvent as a backend. This integrates Net::SNMP into AnyEvent. That
29 29means you can make non-blocking Net::SNMP calls and as long as other
30This integrates Net::SNMP into AnyEvent: You can make non-blocking 30parts of your program also use AnyEvent (or some event loop supported by
31Net::SNMP calls and as long as other parts of your program also use 31AnyEvent), they will run in parallel.
32AnyEvent (or some event loop supported by AnyEvent), they will run in
33parallel.
34 32
35Also, the Net::SNMP scheduler is very inefficient with respect to both CPU 33Also, the Net::SNMP scheduler is very inefficient with respect to both CPU
36and memory usage. Most AnyEvent backends (including the pure-perl backend) 34and memory usage. Most AnyEvent backends (including the pure-perl backend)
37fare much better than the Net::SNMP dispatcher. 35fare much better than the Net::SNMP dispatcher.
38 36
37Another major added feature of this module over Net::SNMP is automatic
38rate-adjustments: Net::SNMP is so slow that firing a few thousand
39requests can cause many timeouts simply because Net::SNMP cannot process
40the replies in time. This module automatically adapts the send rate to
41avoid false timeouts caused by slow reply processing.
42
39A potential disadvantage is that replacing the dispatcher is not at all 43A potential disadvantage of this module is that replacing the dispatcher
40a documented thing to do, so future changes in Net::SNP might break this 44is not at all a documented thing to do, so future changes in Net::SNMP
41module (or the many similar ones). 45might break this module (or the many similar ones).
42 46
43This module does not export anything and does not require you to do 47This module does not export anything and does not require you to do
44anything special apart from loading it I<before doing any non-blocking 48anything special apart from loading it I<before doing any non-blocking
45requests with Net::SNMP>. It is recommended but not required to load this 49requests with Net::SNMP>. It is recommended but not required to load this
46module before C<Net::SNMP>. 50module before C<Net::SNMP>.
… …
49 53
50=over 4 54=over 4
51 55
52=item $AnyEvent::SNMP::MAX_OUTSTANDING (default: C<50>, dynamic) 56=item $AnyEvent::SNMP::MAX_OUTSTANDING (default: C<50>, dynamic)
53 57
58=item AnyEvent::SNMP::set_max_outstanding $new_value
59
54Use this package variable to restrict the number of outstanding SNMP 60Use this package variable to restrict the number of outstanding SNMP
55requests at any point in time. 61requests at any point in time.
56 62
57Net::SNMP is very fast at creating and sending SNMP requests, but much 63Net::SNMP is very fast at creating and sending SNMP requests, but much
58slower at parsing (big, bulk) responses. This makes it easy to request a 64slower at parsing (big, bulk) responses. This makes it easy to request a
… …
61In the best case, this can lead to unnecessary delays (and even time-outs, 67In the best case, this can lead to unnecessary delays (and even time-outs,
62as the data has been received but not yet processed) and in the worst 68as the data has been received but not yet processed) and in the worst
63case, this can lead to packet loss, when the receive queue overflows and 69case, this can lead to packet loss, when the receive queue overflows and
64the kernel can no longer accept new packets. 70the kernel can no longer accept new packets.
65 71
66To avoid this, you can (and should) limit the number of outstanding requests 72To avoid this, you can (and should) limit the number of outstanding
67to a number low enough so that parsing time doesn't introduce noticable delays. 73requests to a number low enough so that parsing time doesn't introduce
74noticeable delays.
68 75
69Unfortunately, this number depends not only on processing speed and load 76Unfortunately, this number depends not only on processing speed and load
70of the machine running Net::SNMP, but also on the network latency and the 77of the machine running Net::SNMP, but also on the network latency and the
71speed of your SNMP agents. 78speed of your SNMP agents.
72 79
73AnyEvent::SNMP tries to dynamically adjust this number dynamically upwards 80AnyEvent::SNMP tries to dynamically adjust this number upwards and
74and downwards. 81downwards.
82
83Increasing C<$MAX_OUTSTANDING> will not automatically use the
84extra request slots. To increase C<$MAX_OUTSTANDING> and make
85C<AnyEvent::SNMP> make use of the extra parallelity, call
86C<AnyEvent::SNMP::set_max_outstanding> with the new value, e.g.:
87
88 AnyEvent::SNMP::set_max_outstanding 500;
89
90Although due to the dynamic adjustment, this might have little lasting
91effect.
75 92
76Note that you can use L<Net::SNMP::XS> to speed up parsing of responses 93Note that you can use L<Net::SNMP::XS> to speed up parsing of responses
77considerably. 94considerably.
78 95
79=item $AnyEvent::SNMP::MIN_RECVQUEUE (default: C<4>) 96=item $AnyEvent::SNMP::MIN_RECVQUEUE (default: C<8>)
80 97
81=item $AnyEvent::SNMP::MAX_RECVQUEUE (default: C<64>) 98=item $AnyEvent::SNMP::MAX_RECVQUEUE (default: C<64>)
82 99
83These values specify the minimum and maximum receive queue length (in 100These values specify the minimum and maximum receive queue length (in
84units of one response packet). 101units of one response packet).
… …
86When AnyEvent::SNMP handles $MAX_RECVQUEUE or more packets per iteration 103When AnyEvent::SNMP handles $MAX_RECVQUEUE or more packets per iteration
87it will reduce $MAX_OUTSTANDING. If it handles less than $MIN_RECVQUEUE, 104it will reduce $MAX_OUTSTANDING. If it handles less than $MIN_RECVQUEUE,
88it increases $MAX_OUTSTANDING. 105it increases $MAX_OUTSTANDING.
89 106
90This has the result of adjusting the number of outstanding requests so that 107This has the result of adjusting the number of outstanding requests so that
91the recv queue is between the minimum and maximu, usually. 108the recv queue is between the minimum and maximum, usually.
92 109
93This algorithm works reasonably well as long as the responses, response 110This algorithm works reasonably well as long as the responses, response
94latencies and processing times are the same size per packet on average. 111latencies and processing times are the same per packet on average.
95 112
96=back 113=back
97 114
98=head1 COMPATIBILITY 115=head1 COMPATIBILITY
99 116
… …
118 135
119=cut 136=cut
120 137
121package AnyEvent::SNMP; 138package AnyEvent::SNMP;
122 139
123no warnings; 140use common::sense;
124use strict qw(subs vars);
125 141
126# it is possible to do this without loading 142# it is possible to do this without loading
127# Net::SNMP::Dispatcher, but much more awkward. 143# Net::SNMP::Dispatcher, but much more awkward.
128use Net::SNMP::Dispatcher; 144use Net::SNMP::Dispatcher;
129 145
146# we could inherit fro Net:SNMP::Dispatcher, but since this is undocumented,
147# I'd rather see it die (and reported) than silenty and subtly fail.
148*msg_handle_alloc = \&Net::SNMP::Dispatcher::msg_handle_alloc;
149
130sub Net::SNMP::Dispatcher::instance { 150sub Net::SNMP::Dispatcher::instance {
131 AnyEvent::SNMP:: 151 AnyEvent::SNMP::
132} 152}
133 153
134use Net::SNMP (); 154use Net::SNMP ();
135use AnyEvent (); 155use AnyEvent ();
136 156
137our $VERSION = '0.2'; 157our $VERSION = '6.0';
138 158
139$Net::SNMP::DISPATCHER = instance Net::SNMP::Dispatcher; 159$Net::SNMP::DISPATCHER = instance Net::SNMP::Dispatcher;
140 160
141our $MESSAGE_PROCESSING = $Net::SNMP::Dispatcher::MESSAGE_PROCESSING; 161our $MESSAGE_PROCESSING = $Net::SNMP::Dispatcher::MESSAGE_PROCESSING;
142 162
143# avoid the method call
144my $timer = sub { shift->timer (@_) };
145AnyEvent::post_detect { $timer = AnyEvent->can ("timer") };
146
147our $BUSY; 163our $BUSY;
164our $DONE; # finished all jobs
148our %TRANSPORT; # address => [count, watcher] 165our @TRANSPORT; # fileno => [count, watcher]
149our @QUEUE; 166our @QUEUE;
150our $MAX_OUTSTANDING = 50; 167our $MAX_OUTSTANDING = 50;
151our $MIN_RECVQUEUE = 4; 168our $MIN_RECVQUEUE = 8;
152our $MAX_RECVQUEUE = 64; 169our $MAX_RECVQUEUE = 64;
153 170
154sub kick_job; 171sub kick_job;
155 172
156sub _send_pdu { 173sub _send_pdu {
… …
175 $MESSAGE_PROCESSING->msg_handle_delete ($pdu->msg_id) 192 $MESSAGE_PROCESSING->msg_handle_delete ($pdu->msg_id)
176 if $pdu->expect_response; 193 if $pdu->expect_response;
177 194
178 # A crude attempt to recover from temporary failures. 195 # A crude attempt to recover from temporary failures.
179 if ($retries-- > 0 && ($!{EAGAIN} || $!{EWOULDBLOCK} || $!{ENOSPC})) { 196 if ($retries-- > 0 && ($!{EAGAIN} || $!{EWOULDBLOCK} || $!{ENOSPC})) {
180 my $retry_w; $retry_w = AnyEvent->$timer (after => $pdu->timeout, cb => sub { 197 my $retry_w; $retry_w = AE::timer $pdu->timeout, 0, sub {
181 undef $retry_w; 198 undef $retry_w;
182 _send_pdu ($pdu, $retries); 199 _send_pdu ($pdu, $retries);
183 }); 200 };
184 } else { 201 } else {
185 --$BUSY; 202 --$BUSY;
186 kick_job; 203 kick_job;
187 } 204 }
188 205
… …
192 } 209 }
193 210
194 # Schedule the timeout handler if the message expects a response. 211 # Schedule the timeout handler if the message expects a response.
195 if ($pdu->expect_response) { 212 if ($pdu->expect_response) {
196 my $transport = $msg->transport; 213 my $transport = $msg->transport;
214 my $fileno = $transport->fileno;
197 215
198 # register the transport 216 # register the transport
199 unless ($TRANSPORT{$transport+0}[0]++) { 217 unless ($TRANSPORT[$fileno][0]++) {
200 $TRANSPORT{$transport+0}[1] = AnyEvent->io (fh => $transport->socket, poll => 'r', cb => sub { 218 $TRANSPORT[$fileno][1] = AE::io $transport->socket, 0, sub {
201 for my $count (1..$MAX_RECVQUEUE) { # handle up to this many requests in one go 219 for my $count (1..$MAX_RECVQUEUE) { # handle up to this many requests in one go
202 # Create a new Message object to receive the response 220 # Create a new Message object to receive the response
203 my ($msg, $error) = Net::SNMP::Message->new (-transport => $transport); 221 my ($msg, $error) = Net::SNMP::Message->new (-transport => $transport);
204 222
205 if (!defined $msg) { 223 if (!defined $msg) {
… …
215 ++$MAX_OUTSTANDING; 233 ++$MAX_OUTSTANDING;
216 kick_job; 234 kick_job;
217 } 235 }
218 } else { 236 } else {
219 # for some reason, connected-oriented transports seem to need this 237 # for some reason, connected-oriented transports seem to need this
220 delete $TRANSPORT{$transport+0} 238 delete $TRANSPORT[$fileno]
221 unless --$TRANSPORT{$transport+0}[0]; 239 unless --$TRANSPORT[$fileno][0];
222 } 240 }
223 241
224 $msg->error; 242 $msg->error;
225 return; 243 return;
226 } 244 }
… …
249 undef $$rtimeout_w; 267 undef $$rtimeout_w;
250 268
251 --$BUSY; 269 --$BUSY;
252 kick_job; 270 kick_job;
253 271
254 unless (--$TRANSPORT{$transport+0}[0]) { 272 unless (--$TRANSPORT[$fileno][0]) {
255 delete $TRANSPORT{$transport+0}; 273 delete $TRANSPORT[$fileno];
256 return; 274 return;
257 } 275 }
258 } 276 }
259 } 277 }
260 278
261 # when we end up here, we successfully handled $MAX_RECVQUEUE 279 # when we end up here, we successfully handled $MAX_RECVQUEUE
262 # replies in one iteration, so assume we are overloaded 280 # replies in one iteration, so assume we are overloaded
263 # and reduce the amount of parallelity. 281 # and reduce the amount of parallelity.
264 $MAX_OUTSTANDING = (int $MAX_OUTSTANDING * 0.9) || 1; 282 $MAX_OUTSTANDING = (int $MAX_OUTSTANDING * 0.95) || 1;
265 }); 283 };
266 } 284 }
267 285
268 $msg->timeout_id (\(my $rtimeout_w = 286 $msg->timeout_id (\(my $rtimeout_w =
269 AnyEvent->$timer (after => $pdu->timeout, cb => sub { 287 AE::timer $pdu->timeout, 0, sub {
270 my $rtimeout_w = $msg->timeout_id; 288 my $rtimeout_w = $msg->timeout_id;
271 if ($$rtimeout_w) { 289 if ($$rtimeout_w) {
272 undef $$rtimeout_w; 290 undef $$rtimeout_w;
273 delete $TRANSPORT{$transport+0} 291 delete $TRANSPORT[$fileno]
274 unless --$TRANSPORT{$transport+0}[0]; 292 unless --$TRANSPORT[$fileno][0];
275 } 293 }
276 294
277 if ($retries--) { 295 if ($retries--) {
278 _send_pdu ($pdu, $retries); 296 _send_pdu ($pdu, $retries);
279 } else { 297 } else {
… …
282 300
283 --$BUSY; 301 --$BUSY;
284 kick_job; 302 kick_job;
285 } 303 }
286 }) 304 })
287 )); 305 );
288 } else { 306 } else {
289 --$BUSY; 307 --$BUSY;
290 kick_job; 308 kick_job;
291 } 309 }
292} 310}
… …
295 while ($BUSY < $MAX_OUTSTANDING) { 313 while ($BUSY < $MAX_OUTSTANDING) {
296 my $pdu = shift @QUEUE 314 my $pdu = shift @QUEUE
297 or last; 315 or last;
298 316
299 ++$BUSY; 317 ++$BUSY;
300
301 _send_pdu $pdu, $pdu->retries; 318 _send_pdu $pdu, $pdu->retries;
302 } 319 }
320
321 $DONE and $DONE->() unless $BUSY;
303} 322}
323
304sub send_pdu($$$) { 324sub send_pdu($$$) {
305 my (undef, $pdu, $delay) = @_; 325 my (undef, $pdu, $delay) = @_;
306 326
307 # $delay is not very sensibly implemented by AnyEvent::SNMP, 327 # $delay is not very sensibly implemented by AnyEvent::SNMP,
308 # but apparently it is not a very sensible feature. 328 # but apparently it is not a very sensible feature.
309 if ($delay > 0) { 329 if ($delay > 0) {
310 ++$BUSY; 330 ++$BUSY;
311 my $delay_w; $delay_w = AnyEvent->$timer (after => $delay, cb => sub { 331 my $delay_w; $delay_w = AE::timer $delay, 0, sub {
312 undef $delay_w; 332 undef $delay_w;
333 push @QUEUE, $pdu;
313 --$BUSY; 334 --$BUSY;
314 push @QUEUE, $pdu;
315 kick_job; 335 kick_job;
316 }); 336 };
317 return 1; 337 return 1;
318 } 338 }
319 339
320 push @QUEUE, $pdu; 340 push @QUEUE, $pdu;
321 kick_job; 341 kick_job;
322 342
323 1 343 1
324} 344}
325 345
326sub activate($) { 346sub loop($) {
327 AnyEvent->one_event while $BUSY; 347 while ($BUSY) {
348 $DONE = AE::cv;
349 $DONE->recv;
350 undef $DONE;
351 }
328} 352}
353
354*activate = \&loop; # 5.x compatibility?
355*listen = \&loop; # 5.x compatibility?
329 356
330sub one_event($) { 357sub one_event($) {
358 # should not ever be used
331 AnyEvent->one_event; 359 AnyEvent->one_event; #d# todo
332} 360}
361
362sub set_max_outstanding($) {
363 $MAX_OUTSTANDING = $_[0];
364 kick_job;
365}
366
367# not provided yet:
368# schedule # apparently only used by Net::SNMP::Dispatcher itself
369# register # apparently only used by Net::SNMP::Dispatcher itself
370# deregister # apparently only used by Net::SNMP::Dispatcher itself
371# cancel # apparently only used by Net::SNMP::Dispatcher itself
372# return_response_pdu # apparently not used at all?
373# error # only used by Net::SNMP::Dispatcher itself?
374# debug # only used by Net::SNMP::Dispatcher itself?
333 375
334=head1 SEE ALSO 376=head1 SEE ALSO
335 377
336L<AnyEvent>, L<Net::SNMP>, L<Net::SNMP::XS>, L<Net::SNMP::EV>. 378L<AnyEvent>, L<Net::SNMP>, L<Net::SNMP::XS>, L<Net::SNMP::EV>.
337 379

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines