ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-SNMP/SNMP.pm
(Generate patch)

Comparing AnyEvent-SNMP/SNMP.pm (file contents):
Revision 1.1 by root, Tue Mar 31 21:55:18 2009 UTC vs.
Revision 1.14 by root, Wed Oct 9 18:22:59 2019 UTC

1=head1 NAME 1=head1 NAME
2 2
3AnyEvent::SNMP - adaptor to integrate Net::SNMP into Anyevent. 3AnyEvent::SNMP - adaptor to integrate Net::SNMP into AnyEvent.
4 4
5=head1 SYNOPSIS 5=head1 SYNOPSIS
6 6
7 use AnyEvent::SNMP; 7 use AnyEvent::SNMP;
8 use Net::SNMP; 8 use Net::SNMP;
… …
23 my @result = $cv->wait; 23 my @result = $cv->wait;
24 24
25=head1 DESCRIPTION 25=head1 DESCRIPTION
26 26
27This module implements an alternative "event dispatcher" for Net::SNMP, 27This module implements an alternative "event dispatcher" for Net::SNMP,
28using AnyEvent as a backend. 28using AnyEvent as a backend. This integrates Net::SNMP into AnyEvent. That
29 29means you can make non-blocking Net::SNMP calls and as long as other
30This integrates Net::SNMP into AnyEvent: You can make non-blocking 30parts of your program also use AnyEvent (or some event loop supported by
31Net::SNMP calls and as long as other parts of your program also use 31AnyEvent), they will run in parallel.
32AnyEvent (or some event loop supported by AnyEvent), they will run in
33parallel.
34 32
35Also, the Net::SNMP scheduler is very inefficient with respect to both CPU 33Also, the Net::SNMP scheduler is very inefficient with respect to both CPU
36and memory usage. Most AnyEvent backends (including the pure-perl backend) 34and memory usage. Most AnyEvent backends (including the pure-perl backend)
37fare much better than the Net::SNMP dispatcher. 35fare much better than the Net::SNMP dispatcher.
38 36
37Another major added feature of this module over Net::SNMP is automatic
38rate-adjustments: Net::SNMP is so slow that firing a few thousand
39requests can cause many timeouts simply because Net::SNMP cannot process
40the replies in time. This module automatically adapts the send rate to
41avoid false timeouts caused by slow reply processing.
42
39A potential disadvantage is that replacing the dispatcher is not at all 43A potential disadvantage of this module is that replacing the dispatcher
40a documented thing to do, so future changes in Net::SNP might break this 44is not at all a documented thing to do, so future changes in Net::SNMP
41module (or the many similar ones). 45might break this module (or the many similar ones).
42 46
43This module does not export anything and does not require you to do 47This module does not export anything and does not require you to do
44anything special apart from loading it I<before doing any non-blocking 48anything special apart from loading it I<before doing any non-blocking
45requests with Net::SNMP>. It is recommended but not required to load this 49requests with Net::SNMP>. It is recommended but not required to load this
46module before C<Net::SNMP>. 50module before C<Net::SNMP>.
47 51
52=head1 GLOBAL VARIABLES
53
54=over 4
55
56=item $AnyEvent::SNMP::MAX_OUTSTANDING (default: C<50>, dynamic)
57
58=item AnyEvent::SNMP::set_max_outstanding $new_value
59
60Use this package variable to restrict the number of outstanding SNMP
61requests at any point in time.
62
63Net::SNMP is very fast at creating and sending SNMP requests, but much
64slower at parsing (big, bulk) responses. This makes it easy to request a
65lot of data that can take many seconds to parse.
66
67In the best case, this can lead to unnecessary delays (and even time-outs,
68as the data has been received but not yet processed) and in the worst
69case, this can lead to packet loss, when the receive queue overflows and
70the kernel can no longer accept new packets.
71
72To avoid this, you can (and should) limit the number of outstanding
73requests to a number low enough so that parsing time doesn't introduce
74noticeable delays.
75
76Unfortunately, this number depends not only on processing speed and load
77of the machine running Net::SNMP, but also on the network latency and the
78speed of your SNMP agents.
79
80AnyEvent::SNMP tries to dynamically adjust this number upwards and
81downwards.
82
83Increasing C<$MAX_OUTSTANDING> will not automatically use the
84extra request slots. To increase C<$MAX_OUTSTANDING> and make
85C<AnyEvent::SNMP> make use of the extra parallelity, call
86C<AnyEvent::SNMP::set_max_outstanding> with the new value, e.g.:
87
88 AnyEvent::SNMP::set_max_outstanding 500;
89
90Although due to the dynamic adjustment, this might have little lasting
91effect.
92
93Note that you can use L<Net::SNMP::XS> to speed up parsing of responses
94considerably.
95
96=item $AnyEvent::SNMP::MIN_RECVQUEUE (default: C<8>)
97
98=item $AnyEvent::SNMP::MAX_RECVQUEUE (default: C<64>)
99
100These values specify the minimum and maximum receive queue length (in
101units of one response packet).
102
103When AnyEvent::SNMP handles $MAX_RECVQUEUE or more packets per iteration
104it will reduce $MAX_OUTSTANDING. If it handles less than $MIN_RECVQUEUE,
105it increases $MAX_OUTSTANDING.
106
107This has the result of adjusting the number of outstanding requests so that
108the recv queue is between the minimum and maximum, usually.
109
110This algorithm works reasonably well as long as the responses, response
111latencies and processing times are the same per packet on average.
112
113=back
114
115=head1 COMPATIBILITY
116
117This module may be used as a drop in replacement for the
118Net::SNMP::Dispatcher in existing programs. You can still call
119C<snmp_dispatcher> to start the event-loop, but then you loose the benefit
120of mixing Net::SNMP events with other events.
121
122 use AnyEvent::SNMP;
123 use Net::SNMP;
124
125 # just use Net::SNMP as before
126
127 # ... start non-blocking snmp request(s)...
128 Net::SNMP->session (
129 -hostname => "127.0.0.1",
130 -community => "public",
131 -nonblocking => 1,
132 )->get_request (-callback => sub { ... });
133
134 snmp_dispatcher;
135
48=cut 136=cut
49 137
50package AnyEvent::SNMP; 138package AnyEvent::SNMP;
51 139
52no warnings; 140use common::sense;
53use strict qw(subs vars);
54 141
55# it is possible to do this without loading 142# it is possible to do this without loading
56# Net::SNMP::Dispatcher, but much more awkward. 143# Net::SNMP::Dispatcher, but much more awkward.
57use Net::SNMP::Dispatcher; 144use Net::SNMP::Dispatcher;
58 145
146# we could inherit fro Net:SNMP::Dispatcher, but since this is undocumented,
147# I'd rather see it die (and reported) than silenty and subtly fail.
148*msg_handle_alloc = \&Net::SNMP::Dispatcher::msg_handle_alloc;
149
59sub Net::SNMP::Dispatcher::instance { 150sub Net::SNMP::Dispatcher::instance {
60 AnyEvent::SNMP:: 151 AnyEvent::SNMP::
61} 152}
62 153
63use Net::SNMP (); 154use Net::SNMP ();
64use AnyEvent (); 155use AnyEvent ();
65 156
66our $VERSION = '0.1'; 157our $VERSION = '6.02';
67 158
68$Net::SNMP::DISPATCHER = instance Net::SNMP::Dispatcher; 159$Net::SNMP::DISPATCHER = instance Net::SNMP::Dispatcher;
69 160
70our $MESSAGE_PROCESSING = $Net::SNMP::Dispatcher::MESSAGE_PROCESSING; 161our $MESSAGE_PROCESSING = $Net::SNMP::Dispatcher::MESSAGE_PROCESSING;
71 162
72# avoid the method call
73my $timer = sub { shift->timer (@_) };
74AnyEvent::post_detect { $timer = AnyEvent->can ("timer") };
75
76our $BUSY; 163our $BUSY;
164our $DONE; # finished all jobs
77our %TRANSPORT; # address => [count, watcher] 165our @TRANSPORT; # fileno => [count, watcher]
166our @QUEUE;
167our $MAX_OUTSTANDING = 50;
168our $MIN_RECVQUEUE = 8;
169our $MAX_RECVQUEUE = 64;
170
171sub kick_job;
78 172
79sub _send_pdu { 173sub _send_pdu {
80 my ($pdu, $retries) = @_; 174 my ($pdu, $retries) = @_;
81 175
82 # mostly copied from Net::SNMP::Dispatch 176 # mostly copied from Net::SNMP::Dispatch
… …
85 # create the new outgoing message. 179 # create the new outgoing message.
86 my $msg = $MESSAGE_PROCESSING->prepare_outgoing_msg ($pdu); 180 my $msg = $MESSAGE_PROCESSING->prepare_outgoing_msg ($pdu);
87 181
88 if (!defined $msg) { 182 if (!defined $msg) {
89 --$BUSY; 183 --$BUSY;
184 kick_job;
90 # Inform the command generator about the Message Processing error. 185 # Inform the command generator about the Message Processing error.
91 $pdu->status_information ($MESSAGE_PROCESSING->error); 186 $pdu->status_information ($MESSAGE_PROCESSING->error);
92 return; 187 return;
93 } 188 }
94 189
… …
97 $MESSAGE_PROCESSING->msg_handle_delete ($pdu->msg_id) 192 $MESSAGE_PROCESSING->msg_handle_delete ($pdu->msg_id)
98 if $pdu->expect_response; 193 if $pdu->expect_response;
99 194
100 # A crude attempt to recover from temporary failures. 195 # A crude attempt to recover from temporary failures.
101 if ($retries-- > 0 && ($!{EAGAIN} || $!{EWOULDBLOCK} || $!{ENOSPC})) { 196 if ($retries-- > 0 && ($!{EAGAIN} || $!{EWOULDBLOCK} || $!{ENOSPC})) {
102 my $retry_w; $retry_w = AnyEvent->$timer (after => $pdu->timeout, cb => sub { 197 my $retry_w; $retry_w = AE::timer $pdu->timeout, 0, sub {
103 undef $retry_w; 198 undef $retry_w;
104 _send_pdu ($pdu, $retries); 199 _send_pdu ($pdu, $retries);
105 }); 200 };
106 } else { 201 } else {
107 --$BUSY; 202 --$BUSY;
203 kick_job;
108 } 204 }
109 205
110 # Inform the command generator about the send() error. 206 # Inform the command generator about the send() error.
111 $pdu->status_information ($msg->error); 207 $pdu->status_information ($msg->error);
112 return; 208 return;
113 } 209 }
114 210
115 # Schedule the timeout handler if the message expects a response. 211 # Schedule the timeout handler if the message expects a response.
116 if ($pdu->expect_response) { 212 if ($pdu->expect_response) {
117 my $transport = $msg->transport; 213 my $transport = $msg->transport;
214 my $fileno = $transport->fileno;
118 215
119 # register the transport 216 # register the transport
120 unless ($TRANSPORT{$transport+0}[0]++) { 217 unless ($TRANSPORT[$fileno][0]++) {
121 $TRANSPORT{$transport+0}[1] = AnyEvent->io (fh => $transport->socket, poll => 'r', cb => sub { 218 $TRANSPORT[$fileno][1] = AE::io $transport->socket, 0, sub {
219 for my $count (1..$MAX_RECVQUEUE) { # handle up to this many requests in one go
122 # Create a new Message object to receive the response 220 # Create a new Message object to receive the response
123 my ($msg, $error) = Net::SNMP::Message->new (-transport => $transport); 221 my ($msg, $error) = Net::SNMP::Message->new (-transport => $transport);
124 222
125 if (!defined $msg) { 223 if (!defined $msg) {
126 die sprintf 'Failed to create Message object [%s]', $error; 224 die sprintf 'Failed to create Message object [%s]', $error;
225 }
226
227 # Read the message from the Transport Layer
228 if (!defined $msg->recv) {
229 if ($transport->connectionless) {
230 # if we handled very few replies and we have queued work, try
231 # to increase the parallelity as we probably can handle more.
232 if ($count < $MIN_RECVQUEUE && @QUEUE) {
233 ++$MAX_OUTSTANDING;
234 kick_job;
235 }
236 } else {
237 # for some reason, connected-oriented transports seem to need this
238 delete $TRANSPORT[$fileno]
239 unless --$TRANSPORT[$fileno][0];
240 }
241
242 $msg->error;
243 return;
244 }
245
246 # For connection-oriented Transport Domains, it is possible to
247 # "recv" an empty buffer if reassembly is required.
248 if (!$msg->length) {
249 return;
250 }
251
252 # Hand the message over to Message Processing.
253 if (!defined $MESSAGE_PROCESSING->prepare_data_elements ($msg)) {
254 $MESSAGE_PROCESSING->error;
255 return;
256 }
257
258 # Set the error if applicable.
259 $msg->error ($MESSAGE_PROCESSING->error) if $MESSAGE_PROCESSING->error;
260
261 # Notify the command generator to process the response.
262 # Net::SNMP calls process_response_pdu, which simply calls callback_execute,
263 # but some errors cause $msg to be of type Net::SNMP::Message, not Net::SMMP::PDU,
264 # so we call the underlying callback_execute method which exists on both and
265 # seems to do the right thing.
266 $msg->callback_execute;
267
268 # Cancel the timeout.
269 my $rtimeout_w = $msg->timeout_id;
270 if ($$rtimeout_w) {
271 undef $$rtimeout_w;
272
273 --$BUSY;
274 kick_job;
275
276 unless (--$TRANSPORT[$fileno][0]) {
277 delete $TRANSPORT[$fileno];
278 return;
279 }
280 }
127 } 281 }
128 282
129 # Read the message from the Transport Layer 283 # when we end up here, we successfully handled $MAX_RECVQUEUE
130 if (!defined $msg->recv) { 284 # replies in one iteration, so assume we are overloaded
131 # for some reason, connected-oriented transports seem to need this 285 # and reduce the amount of parallelity.
132 unless ($transport->connectionless) { 286 $MAX_OUTSTANDING = (int $MAX_OUTSTANDING * 0.95) || 1;
133 delete $TRANSPORT{$transport+0}
134 unless --$TRANSPORT{$transport+0}[0];
135 }
136
137 $msg->error;
138 return;
139 } 287 };
288 }
140 289
141 # For connection-oriented Transport Domains, it is possible to 290 $msg->timeout_id (\(my $rtimeout_w =
142 # "recv" an empty buffer if reassembly is required. 291 AE::timer $pdu->timeout, 0, sub {
143 if (!$msg->length) {
144 return;
145 }
146
147 # Hand the message over to Message Processing.
148 if (!defined $MESSAGE_PROCESSING->prepare_data_elements ($msg)) {
149 $MESSAGE_PROCESSING->error;
150 return;
151 }
152
153 # Set the error if applicable.
154 $msg->error ($MESSAGE_PROCESSING->error) if $MESSAGE_PROCESSING->error;
155
156 # Cancel the timeout.
157 my $rtimeout_w = $msg->timeout_id; 292 my $rtimeout_w = $msg->timeout_id;
158 if ($$rtimeout_w) { 293 if ($$rtimeout_w) {
159 undef $$rtimeout_w; 294 undef $$rtimeout_w;
160 delete $TRANSPORT{$transport+0} 295 delete $TRANSPORT[$fileno]
161 unless --$TRANSPORT{$transport+0}[0]; 296 unless --$TRANSPORT[$fileno][0];
162
163 --$BUSY;
164 }
165
166 # Notify the command generator to process the response.
167 $msg->process_response_pdu;
168 });
169 }
170
171 #####d# timeout_id, wtf?
172 $msg->timeout_id (\(my $rtimeout_w =
173 AnyEvent->$timer (after => $pdu->timeout, cb => sub {
174 my $rtimeout_w = $msg->timeout_id;
175 if ($$rtimeout_w) {
176 undef $$rtimeout_w;
177 delete $TRANSPORT{$transport+0}
178 unless --$TRANSPORT{$transport+0}[0];
179 } 297 }
180 298
181 if ($retries--) { 299 if ($retries--) {
182 _send_pdu ($pdu, $retries); 300 _send_pdu ($pdu, $retries);
183 } else { 301 } else {
184 --$BUSY;
185 $MESSAGE_PROCESSING->msg_handle_delete ($pdu->msg_id); 302 $MESSAGE_PROCESSING->msg_handle_delete ($pdu->msg_id);
186 $pdu->status_information ("No response from remote host '%s'", $pdu->hostname); 303 $pdu->status_information ("No response from remote host '%s'", $pdu->hostname);
304
305 --$BUSY;
306 kick_job;
187 } 307 }
188 }) 308 })
189 )); 309 );
190 } else { 310 } else {
191 --$BUSY; 311 --$BUSY;
312 kick_job;
192 } 313 }
314}
315
316sub kick_job {
317 while ($BUSY < $MAX_OUTSTANDING) {
318 my $pdu = shift @QUEUE
319 or last;
320
321 ++$BUSY;
322 _send_pdu $pdu, $pdu->retries;
323 }
324
325 $DONE and $DONE->() unless $BUSY;
193} 326}
194 327
195sub send_pdu($$$) { 328sub send_pdu($$$) {
196 my (undef, $pdu, $delay) = @_; 329 my (undef, $pdu, $delay) = @_;
197 330
198 ++$BUSY; 331 # $delay is not very sensibly implemented by AnyEvent::SNMP,
199 332 # but apparently it is not a very sensible feature.
200 if ($delay > 0) { 333 if ($delay > 0) {
334 ++$BUSY;
201 my $delay_w; $delay_w = AnyEvent->$timer (after => $delay, cb => sub { 335 my $delay_w; $delay_w = AE::timer $delay, 0, sub {
202 undef $delay_w; 336 undef $delay_w;
203 _send_pdu ($pdu, $pdu->retries); 337 push @QUEUE, $pdu;
338 --$BUSY;
339 kick_job;
204 }); 340 };
205 return 1; 341 return 1;
206 } 342 }
207 343
208 _send_pdu $pdu, $pdu->retries; 344 push @QUEUE, $pdu;
345 kick_job;
346
209 1 347 1
210} 348}
211 349
212sub activate($) { 350sub loop($) {
213 AnyEvent->one_event while $BUSY; 351 while ($BUSY) {
352 $DONE = AE::cv;
353 $DONE->recv;
354 undef $DONE;
355 }
214} 356}
357
358*activate = \&loop; # 5.x compatibility?
359*listen = \&loop; # 5.x compatibility?
215 360
216sub one_event($) { 361sub one_event($) {
217 die; 362 # should not ever be used
363 AnyEvent->one_event; #d# todo
218} 364}
365
366sub set_max_outstanding($) {
367 $MAX_OUTSTANDING = $_[0];
368 kick_job;
369}
370
371# not provided yet:
372# schedule # apparently only used by Net::SNMP::Dispatcher itself
373# register # apparently only used by Net::SNMP::Dispatcher itself
374# deregister # apparently only used by Net::SNMP::Dispatcher itself
375# cancel # apparently only used by Net::SNMP::Dispatcher itself
376# return_response_pdu # apparently not used at all?
377# error # only used by Net::SNMP::Dispatcher itself?
378# debug # only used by Net::SNMP::Dispatcher itself?
219 379
220=head1 SEE ALSO 380=head1 SEE ALSO
221 381
222L<AnyEvent>, L<Net::SNMP>, L<Net::SNMP::EV>. 382L<AnyEvent>, L<Net::SNMP>, L<Net::SNMP::XS>, L<Net::SNMP::EV>.
223 383
224=head1 AUTHOR 384=head1 AUTHOR
225 385
226 Marc Lehmann <schmorp@schmorp.de> 386 Marc Lehmann <schmorp@schmorp.de>
227 http://home.schmorp.de/ 387 http://home.schmorp.de/

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines