ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/BDB/BDB.pm
(Generate patch)

Comparing BDB/BDB.pm (file contents):
Revision 1.1 by root, Mon Feb 5 18:40:55 2007 UTC vs.
Revision 1.2 by root, Mon Feb 5 20:21:38 2007 UTC

1=head1 NAME 1=head1 NAME
2 2
3BDB::AIO - Asynchronous Berkeley DB access 3BDB - Asynchronous Berkeley DB access
4 4
5=head1 SYNOPSIS 5=head1 SYNOPSIS
6 6
7 use BDB::AIO; 7 use BDB;
8 8
9=head1 DESCRIPTION 9=head1 DESCRIPTION
10 10
11=head2 EXAMPLE 11=head2 EXAMPLE
12 12
… …
55 55
56=back 56=back
57 57
58=cut 58=cut
59 59
60package BDB::AIO; 60package BDB;
61 61
62no warnings; 62no warnings;
63use strict 'vars'; 63use strict 'vars';
64 64
65use base 'Exporter'; 65use base 'Exporter';
… …
72 min_parallel max_parallel max_idle 72 min_parallel max_parallel max_idle
73 nreqs nready npending nthreads 73 nreqs nready npending nthreads
74 max_poll_time max_poll_reqs); 74 max_poll_time max_poll_reqs);
75 75
76 require XSLoader; 76 require XSLoader;
77 XSLoader::load ("BDB::AIO", $VERSION); 77 XSLoader::load ("BDB", $VERSION);
78} 78}
79 79
80=head2 SUPPORT FUNCTIONS 80=head2 SUPPORT FUNCTIONS
81 81
82=head3 EVENT PROCESSING AND EVENT LOOP INTEGRATION 82=head3 EVENT PROCESSING AND EVENT LOOP INTEGRATION
83 83
84=over 4 84=over 4
85 85
86=item $fileno = BDB::AIO::poll_fileno 86=item $fileno = BDB::poll_fileno
87 87
88Return the I<request result pipe file descriptor>. This filehandle must be 88Return the I<request result pipe file descriptor>. This filehandle must be
89polled for reading by some mechanism outside this module (e.g. Event or 89polled for reading by some mechanism outside this module (e.g. Event or
90select, see below or the SYNOPSIS). If the pipe becomes readable you have 90select, see below or the SYNOPSIS). If the pipe becomes readable you have
91to call C<poll_cb> to check the results. 91to call C<poll_cb> to check the results.
92 92
93See C<poll_cb> for an example. 93See C<poll_cb> for an example.
94 94
95=item BDB::AIO::poll_cb 95=item BDB::poll_cb
96 96
97Process some outstanding events on the result pipe. You have to call this 97Process some outstanding events on the result pipe. You have to call this
98regularly. Returns the number of events processed. Returns immediately 98regularly. Returns the number of events processed. Returns immediately
99when no events are outstanding. The amount of events processed depends on 99when no events are outstanding. The amount of events processed depends on
100the settings of C<BDB::AIO::max_poll_req> and C<BDB::AIO::max_poll_time>. 100the settings of C<BDB::max_poll_req> and C<BDB::max_poll_time>.
101 101
102If not all requests were processed for whatever reason, the filehandle 102If not all requests were processed for whatever reason, the filehandle
103will still be ready when C<poll_cb> returns. 103will still be ready when C<poll_cb> returns.
104 104
105Example: Install an Event watcher that automatically calls 105Example: Install an Event watcher that automatically calls
106BDB::AIO::poll_cb with high priority: 106BDB::poll_cb with high priority:
107 107
108 Event->io (fd => BDB::AIO::poll_fileno, 108 Event->io (fd => BDB::poll_fileno,
109 poll => 'r', async => 1, 109 poll => 'r', async => 1,
110 cb => \&BDB::AIO::poll_cb); 110 cb => \&BDB::poll_cb);
111 111
112=item BDB::AIO::max_poll_reqs $nreqs 112=item BDB::max_poll_reqs $nreqs
113 113
114=item BDB::AIO::max_poll_time $seconds 114=item BDB::max_poll_time $seconds
115 115
116These set the maximum number of requests (default C<0>, meaning infinity) 116These set the maximum number of requests (default C<0>, meaning infinity)
117that are being processed by C<BDB::AIO::poll_cb> in one call, respectively 117that are being processed by C<BDB::poll_cb> in one call, respectively
118the maximum amount of time (default C<0>, meaning infinity) spent in 118the maximum amount of time (default C<0>, meaning infinity) spent in
119C<BDB::AIO::poll_cb> to process requests (more correctly the mininum amount 119C<BDB::poll_cb> to process requests (more correctly the mininum amount
120of time C<poll_cb> is allowed to use). 120of time C<poll_cb> is allowed to use).
121 121
122Setting C<max_poll_time> to a non-zero value creates an overhead of one 122Setting C<max_poll_time> to a non-zero value creates an overhead of one
123syscall per request processed, which is not normally a problem unless your 123syscall per request processed, which is not normally a problem unless your
124callbacks are really really fast or your OS is really really slow (I am 124callbacks are really really fast or your OS is really really slow (I am
… …
129time. 129time.
130 130
131For interactive programs, values such as C<0.01> to C<0.1> should be fine. 131For interactive programs, values such as C<0.01> to C<0.1> should be fine.
132 132
133Example: Install an Event watcher that automatically calls 133Example: Install an Event watcher that automatically calls
134BDB::AIO::poll_cb with low priority, to ensure that other parts of the 134BDB::poll_cb with low priority, to ensure that other parts of the
135program get the CPU sometimes even under high AIO load. 135program get the CPU sometimes even under high AIO load.
136 136
137 # try not to spend much more than 0.1s in poll_cb 137 # try not to spend much more than 0.1s in poll_cb
138 BDB::AIO::max_poll_time 0.1; 138 BDB::max_poll_time 0.1;
139 139
140 # use a low priority so other tasks have priority 140 # use a low priority so other tasks have priority
141 Event->io (fd => BDB::AIO::poll_fileno, 141 Event->io (fd => BDB::poll_fileno,
142 poll => 'r', nice => 1, 142 poll => 'r', nice => 1,
143 cb => &BDB::AIO::poll_cb); 143 cb => &BDB::poll_cb);
144 144
145=item BDB::AIO::poll_wait 145=item BDB::poll_wait
146 146
147If there are any outstanding requests and none of them in the result 147If there are any outstanding requests and none of them in the result
148phase, wait till the result filehandle becomes ready for reading (simply 148phase, wait till the result filehandle becomes ready for reading (simply
149does a C<select> on the filehandle. This is useful if you want to 149does a C<select> on the filehandle. This is useful if you want to
150synchronously wait for some requests to finish). 150synchronously wait for some requests to finish).
151 151
152See C<nreqs> for an example. 152See C<nreqs> for an example.
153 153
154=item BDB::AIO::poll 154=item BDB::poll
155 155
156Waits until some requests have been handled. 156Waits until some requests have been handled.
157 157
158Returns the number of requests processed, but is otherwise strictly 158Returns the number of requests processed, but is otherwise strictly
159equivalent to: 159equivalent to:
160 160
161 BDB::AIO::poll_wait, BDB::AIO::poll_cb 161 BDB::poll_wait, BDB::poll_cb
162 162
163=item BDB::AIO::flush 163=item BDB::flush
164 164
165Wait till all outstanding AIO requests have been handled. 165Wait till all outstanding AIO requests have been handled.
166 166
167Strictly equivalent to: 167Strictly equivalent to:
168 168
169 BDB::AIO::poll_wait, BDB::AIO::poll_cb 169 BDB::poll_wait, BDB::poll_cb
170 while BDB::AIO::nreqs; 170 while BDB::nreqs;
171 171
172=head3 CONTROLLING THE NUMBER OF THREADS 172=head3 CONTROLLING THE NUMBER OF THREADS
173 173
174=item BDB::AIO::min_parallel $nthreads 174=item BDB::min_parallel $nthreads
175 175
176Set the minimum number of AIO threads to C<$nthreads>. The current 176Set the minimum number of AIO threads to C<$nthreads>. The current
177default is C<8>, which means eight asynchronous operations can execute 177default is C<8>, which means eight asynchronous operations can execute
178concurrently at any one time (the number of outstanding requests, 178concurrently at any one time (the number of outstanding requests,
179however, is unlimited). 179however, is unlimited).
180 180
181BDB::AIO starts threads only on demand, when an AIO request is queued and 181BDB starts threads only on demand, when an AIO request is queued and
182no free thread exists. Please note that queueing up a hundred requests can 182no free thread exists. Please note that queueing up a hundred requests can
183create demand for a hundred threads, even if it turns out that everything 183create demand for a hundred threads, even if it turns out that everything
184is in the cache and could have been processed faster by a single thread. 184is in the cache and could have been processed faster by a single thread.
185 185
186It is recommended to keep the number of threads relatively low, as some 186It is recommended to keep the number of threads relatively low, as some
… …
189versions, 4-32 threads should be fine. 189versions, 4-32 threads should be fine.
190 190
191Under most circumstances you don't need to call this function, as the 191Under most circumstances you don't need to call this function, as the
192module selects a default that is suitable for low to moderate load. 192module selects a default that is suitable for low to moderate load.
193 193
194=item BDB::AIO::max_parallel $nthreads 194=item BDB::max_parallel $nthreads
195 195
196Sets the maximum number of AIO threads to C<$nthreads>. If more than the 196Sets the maximum number of AIO threads to C<$nthreads>. If more than the
197specified number of threads are currently running, this function kills 197specified number of threads are currently running, this function kills
198them. This function blocks until the limit is reached. 198them. This function blocks until the limit is reached.
199 199
… …
203This module automatically runs C<max_parallel 0> at program end, to ensure 203This module automatically runs C<max_parallel 0> at program end, to ensure
204that all threads are killed and that there are no outstanding requests. 204that all threads are killed and that there are no outstanding requests.
205 205
206Under normal circumstances you don't need to call this function. 206Under normal circumstances you don't need to call this function.
207 207
208=item BDB::AIO::max_idle $nthreads 208=item BDB::max_idle $nthreads
209 209
210Limit the number of threads (default: 4) that are allowed to idle (i.e., 210Limit the number of threads (default: 4) that are allowed to idle (i.e.,
211threads that did not get a request to process within 10 seconds). That 211threads that did not get a request to process within 10 seconds). That
212means if a thread becomes idle while C<$nthreads> other threads are also 212means if a thread becomes idle while C<$nthreads> other threads are also
213idle, it will free its resources and exit. 213idle, it will free its resources and exit.
… …
218 218
219The default is probably ok in most situations, especially if thread 219The default is probably ok in most situations, especially if thread
220creation is fast. If thread creation is very slow on your system you might 220creation is fast. If thread creation is very slow on your system you might
221want to use larger values. 221want to use larger values.
222 222
223=item $oldmaxreqs = BDB::AIO::max_outstanding $maxreqs 223=item $oldmaxreqs = BDB::max_outstanding $maxreqs
224 224
225This is a very bad function to use in interactive programs because it 225This is a very bad function to use in interactive programs because it
226blocks, and a bad way to reduce concurrency because it is inexact: Better 226blocks, and a bad way to reduce concurrency because it is inexact: Better
227use an C<aio_group> together with a feed callback. 227use an C<aio_group> together with a feed callback.
228 228
… …
238C<max_oustsanding> is mainly useful in simple scripts (with low values) or 238C<max_oustsanding> is mainly useful in simple scripts (with low values) or
239as a stop gap to shield against fatal memory overflow (with large values). 239as a stop gap to shield against fatal memory overflow (with large values).
240 240
241=head3 STATISTICAL INFORMATION 241=head3 STATISTICAL INFORMATION
242 242
243=item BDB::AIO::nreqs 243=item BDB::nreqs
244 244
245Returns the number of requests currently in the ready, execute or pending 245Returns the number of requests currently in the ready, execute or pending
246states (i.e. for which their callback has not been invoked yet). 246states (i.e. for which their callback has not been invoked yet).
247 247
248Example: wait till there are no outstanding requests anymore: 248Example: wait till there are no outstanding requests anymore:
249 249
250 BDB::AIO::poll_wait, BDB::AIO::poll_cb 250 BDB::poll_wait, BDB::poll_cb
251 while BDB::AIO::nreqs; 251 while BDB::nreqs;
252 252
253=item BDB::AIO::nready 253=item BDB::nready
254 254
255Returns the number of requests currently in the ready state (not yet 255Returns the number of requests currently in the ready state (not yet
256executed). 256executed).
257 257
258=item BDB::AIO::npending 258=item BDB::npending
259 259
260Returns the number of requests currently in the pending state (executed, 260Returns the number of requests currently in the pending state (executed,
261but not yet processed by poll_cb). 261but not yet processed by poll_cb).
262 262
263=back 263=back
264 264
265=cut 265=cut
266
267# support function to convert a fd into a perl filehandle
268sub _fd2fh {
269 return undef if $_[0] < 0;
270
271 # try to generate nice filehandles
272 my $sym = "BDB::AIO::fd#$_[0]";
273 local *$sym;
274
275 open *$sym, "+<&=$_[0]" # usually works under any unix
276 or open *$sym, "<&=$_[0]" # cygwin needs this
277 or open *$sym, ">&=$_[0]" # or this
278 or return undef;
279
280 *$sym
281}
282 266
283min_parallel 8; 267min_parallel 8;
284 268
285END { flush } 269END { flush }
286 270

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines