1 |
root |
1.1 |
=head1 NAME |
2 |
|
|
|
3 |
root |
1.2 |
BDB - Asynchronous Berkeley DB access |
4 |
root |
1.1 |
|
5 |
|
|
=head1 SYNOPSIS |
6 |
|
|
|
7 |
root |
1.2 |
use BDB; |
8 |
root |
1.1 |
|
9 |
|
|
=head1 DESCRIPTION |
10 |
|
|
|
11 |
root |
1.7 |
See the eg/ directory in the distribution and the berkeleydb C |
12 |
|
|
documentation. This is inadequate, but the only sources of documentation |
13 |
|
|
known for this module so far. |
14 |
|
|
|
15 |
root |
1.1 |
=head2 EXAMPLE |
16 |
|
|
|
17 |
|
|
=head1 REQUEST ANATOMY AND LIFETIME |
18 |
|
|
|
19 |
|
|
Every request method creates a request. which is a C data structure not |
20 |
|
|
directly visible to Perl. |
21 |
|
|
|
22 |
|
|
During their existance, bdb requests travel through the following states, |
23 |
|
|
in order: |
24 |
|
|
|
25 |
|
|
=over 4 |
26 |
|
|
|
27 |
|
|
=item ready |
28 |
|
|
|
29 |
|
|
Immediately after a request is created it is put into the ready state, |
30 |
|
|
waiting for a thread to execute it. |
31 |
|
|
|
32 |
|
|
=item execute |
33 |
|
|
|
34 |
|
|
A thread has accepted the request for processing and is currently |
35 |
|
|
executing it (e.g. blocking in read). |
36 |
|
|
|
37 |
|
|
=item pending |
38 |
|
|
|
39 |
|
|
The request has been executed and is waiting for result processing. |
40 |
|
|
|
41 |
|
|
While request submission and execution is fully asynchronous, result |
42 |
|
|
processing is not and relies on the perl interpreter calling C<poll_cb> |
43 |
|
|
(or another function with the same effect). |
44 |
|
|
|
45 |
|
|
=item result |
46 |
|
|
|
47 |
|
|
The request results are processed synchronously by C<poll_cb>. |
48 |
|
|
|
49 |
|
|
The C<poll_cb> function will process all outstanding aio requests by |
50 |
|
|
calling their callbacks, freeing memory associated with them and managing |
51 |
|
|
any groups they are contained in. |
52 |
|
|
|
53 |
|
|
=item done |
54 |
|
|
|
55 |
|
|
Request has reached the end of its lifetime and holds no resources anymore |
56 |
|
|
(except possibly for the Perl object, but its connection to the actual |
57 |
|
|
aio request is severed and calling its methods will either do nothing or |
58 |
|
|
result in a runtime error). |
59 |
|
|
|
60 |
|
|
=back |
61 |
|
|
|
62 |
|
|
=cut |
63 |
|
|
|
64 |
root |
1.2 |
package BDB; |
65 |
root |
1.1 |
|
66 |
|
|
no warnings; |
67 |
|
|
use strict 'vars'; |
68 |
|
|
|
69 |
|
|
use base 'Exporter'; |
70 |
|
|
|
71 |
|
|
BEGIN { |
72 |
|
|
our $VERSION = '0.1'; |
73 |
|
|
|
74 |
root |
1.3 |
our @BDB_REQ = qw( |
75 |
root |
1.6 |
db_env_open db_env_close db_env_txn_checkpoint db_env_lock_detect |
76 |
|
|
db_env_memp_sync db_env_memp_trickle |
77 |
|
|
db_open db_close db_compact db_sync db_put db_get db_pget db_del db_key_range |
78 |
root |
1.4 |
db_txn_commit db_txn_abort |
79 |
root |
1.5 |
db_c_close db_c_count db_c_put db_c_get db_c_pget db_c_del |
80 |
root |
1.6 |
db_sequence_open db_sequence_close |
81 |
|
|
db_sequence_get db_sequence_remove |
82 |
|
|
); |
83 |
|
|
our @EXPORT = (@BDB_REQ, qw(dbreq_pri dbreq_nice db_env_create db_create)); |
84 |
|
|
our @EXPORT_OK = qw( |
85 |
|
|
poll_fileno poll_cb poll_wait flush |
86 |
|
|
min_parallel max_parallel max_idle |
87 |
|
|
nreqs nready npending nthreads |
88 |
|
|
max_poll_time max_poll_reqs |
89 |
root |
1.3 |
); |
90 |
root |
1.1 |
|
91 |
|
|
require XSLoader; |
92 |
root |
1.2 |
XSLoader::load ("BDB", $VERSION); |
93 |
root |
1.1 |
} |
94 |
|
|
|
95 |
|
|
=head2 SUPPORT FUNCTIONS |
96 |
|
|
|
97 |
|
|
=head3 EVENT PROCESSING AND EVENT LOOP INTEGRATION |
98 |
|
|
|
99 |
|
|
=over 4 |
100 |
|
|
|
101 |
root |
1.2 |
=item $fileno = BDB::poll_fileno |
102 |
root |
1.1 |
|
103 |
|
|
Return the I<request result pipe file descriptor>. This filehandle must be |
104 |
|
|
polled for reading by some mechanism outside this module (e.g. Event or |
105 |
|
|
select, see below or the SYNOPSIS). If the pipe becomes readable you have |
106 |
|
|
to call C<poll_cb> to check the results. |
107 |
|
|
|
108 |
|
|
See C<poll_cb> for an example. |
109 |
|
|
|
110 |
root |
1.2 |
=item BDB::poll_cb |
111 |
root |
1.1 |
|
112 |
|
|
Process some outstanding events on the result pipe. You have to call this |
113 |
|
|
regularly. Returns the number of events processed. Returns immediately |
114 |
|
|
when no events are outstanding. The amount of events processed depends on |
115 |
root |
1.2 |
the settings of C<BDB::max_poll_req> and C<BDB::max_poll_time>. |
116 |
root |
1.1 |
|
117 |
|
|
If not all requests were processed for whatever reason, the filehandle |
118 |
|
|
will still be ready when C<poll_cb> returns. |
119 |
|
|
|
120 |
|
|
Example: Install an Event watcher that automatically calls |
121 |
root |
1.2 |
BDB::poll_cb with high priority: |
122 |
root |
1.1 |
|
123 |
root |
1.2 |
Event->io (fd => BDB::poll_fileno, |
124 |
root |
1.1 |
poll => 'r', async => 1, |
125 |
root |
1.2 |
cb => \&BDB::poll_cb); |
126 |
root |
1.1 |
|
127 |
root |
1.2 |
=item BDB::max_poll_reqs $nreqs |
128 |
root |
1.1 |
|
129 |
root |
1.2 |
=item BDB::max_poll_time $seconds |
130 |
root |
1.1 |
|
131 |
|
|
These set the maximum number of requests (default C<0>, meaning infinity) |
132 |
root |
1.2 |
that are being processed by C<BDB::poll_cb> in one call, respectively |
133 |
root |
1.1 |
the maximum amount of time (default C<0>, meaning infinity) spent in |
134 |
root |
1.2 |
C<BDB::poll_cb> to process requests (more correctly the mininum amount |
135 |
root |
1.1 |
of time C<poll_cb> is allowed to use). |
136 |
|
|
|
137 |
|
|
Setting C<max_poll_time> to a non-zero value creates an overhead of one |
138 |
|
|
syscall per request processed, which is not normally a problem unless your |
139 |
|
|
callbacks are really really fast or your OS is really really slow (I am |
140 |
|
|
not mentioning Solaris here). Using C<max_poll_reqs> incurs no overhead. |
141 |
|
|
|
142 |
|
|
Setting these is useful if you want to ensure some level of |
143 |
|
|
interactiveness when perl is not fast enough to process all requests in |
144 |
|
|
time. |
145 |
|
|
|
146 |
|
|
For interactive programs, values such as C<0.01> to C<0.1> should be fine. |
147 |
|
|
|
148 |
|
|
Example: Install an Event watcher that automatically calls |
149 |
root |
1.2 |
BDB::poll_cb with low priority, to ensure that other parts of the |
150 |
root |
1.1 |
program get the CPU sometimes even under high AIO load. |
151 |
|
|
|
152 |
|
|
# try not to spend much more than 0.1s in poll_cb |
153 |
root |
1.2 |
BDB::max_poll_time 0.1; |
154 |
root |
1.1 |
|
155 |
|
|
# use a low priority so other tasks have priority |
156 |
root |
1.2 |
Event->io (fd => BDB::poll_fileno, |
157 |
root |
1.1 |
poll => 'r', nice => 1, |
158 |
root |
1.2 |
cb => &BDB::poll_cb); |
159 |
root |
1.1 |
|
160 |
root |
1.2 |
=item BDB::poll_wait |
161 |
root |
1.1 |
|
162 |
|
|
If there are any outstanding requests and none of them in the result |
163 |
|
|
phase, wait till the result filehandle becomes ready for reading (simply |
164 |
|
|
does a C<select> on the filehandle. This is useful if you want to |
165 |
|
|
synchronously wait for some requests to finish). |
166 |
|
|
|
167 |
|
|
See C<nreqs> for an example. |
168 |
|
|
|
169 |
root |
1.2 |
=item BDB::poll |
170 |
root |
1.1 |
|
171 |
|
|
Waits until some requests have been handled. |
172 |
|
|
|
173 |
|
|
Returns the number of requests processed, but is otherwise strictly |
174 |
|
|
equivalent to: |
175 |
|
|
|
176 |
root |
1.2 |
BDB::poll_wait, BDB::poll_cb |
177 |
root |
1.1 |
|
178 |
root |
1.2 |
=item BDB::flush |
179 |
root |
1.1 |
|
180 |
|
|
Wait till all outstanding AIO requests have been handled. |
181 |
|
|
|
182 |
|
|
Strictly equivalent to: |
183 |
|
|
|
184 |
root |
1.2 |
BDB::poll_wait, BDB::poll_cb |
185 |
|
|
while BDB::nreqs; |
186 |
root |
1.1 |
|
187 |
|
|
=head3 CONTROLLING THE NUMBER OF THREADS |
188 |
|
|
|
189 |
root |
1.2 |
=item BDB::min_parallel $nthreads |
190 |
root |
1.1 |
|
191 |
|
|
Set the minimum number of AIO threads to C<$nthreads>. The current |
192 |
|
|
default is C<8>, which means eight asynchronous operations can execute |
193 |
|
|
concurrently at any one time (the number of outstanding requests, |
194 |
|
|
however, is unlimited). |
195 |
|
|
|
196 |
root |
1.2 |
BDB starts threads only on demand, when an AIO request is queued and |
197 |
root |
1.1 |
no free thread exists. Please note that queueing up a hundred requests can |
198 |
|
|
create demand for a hundred threads, even if it turns out that everything |
199 |
|
|
is in the cache and could have been processed faster by a single thread. |
200 |
|
|
|
201 |
|
|
It is recommended to keep the number of threads relatively low, as some |
202 |
|
|
Linux kernel versions will scale negatively with the number of threads |
203 |
|
|
(higher parallelity => MUCH higher latency). With current Linux 2.6 |
204 |
|
|
versions, 4-32 threads should be fine. |
205 |
|
|
|
206 |
|
|
Under most circumstances you don't need to call this function, as the |
207 |
|
|
module selects a default that is suitable for low to moderate load. |
208 |
|
|
|
209 |
root |
1.2 |
=item BDB::max_parallel $nthreads |
210 |
root |
1.1 |
|
211 |
|
|
Sets the maximum number of AIO threads to C<$nthreads>. If more than the |
212 |
|
|
specified number of threads are currently running, this function kills |
213 |
|
|
them. This function blocks until the limit is reached. |
214 |
|
|
|
215 |
|
|
While C<$nthreads> are zero, aio requests get queued but not executed |
216 |
|
|
until the number of threads has been increased again. |
217 |
|
|
|
218 |
|
|
This module automatically runs C<max_parallel 0> at program end, to ensure |
219 |
|
|
that all threads are killed and that there are no outstanding requests. |
220 |
|
|
|
221 |
|
|
Under normal circumstances you don't need to call this function. |
222 |
|
|
|
223 |
root |
1.2 |
=item BDB::max_idle $nthreads |
224 |
root |
1.1 |
|
225 |
|
|
Limit the number of threads (default: 4) that are allowed to idle (i.e., |
226 |
|
|
threads that did not get a request to process within 10 seconds). That |
227 |
|
|
means if a thread becomes idle while C<$nthreads> other threads are also |
228 |
|
|
idle, it will free its resources and exit. |
229 |
|
|
|
230 |
|
|
This is useful when you allow a large number of threads (e.g. 100 or 1000) |
231 |
|
|
to allow for extremely high load situations, but want to free resources |
232 |
|
|
under normal circumstances (1000 threads can easily consume 30MB of RAM). |
233 |
|
|
|
234 |
|
|
The default is probably ok in most situations, especially if thread |
235 |
|
|
creation is fast. If thread creation is very slow on your system you might |
236 |
|
|
want to use larger values. |
237 |
|
|
|
238 |
root |
1.2 |
=item $oldmaxreqs = BDB::max_outstanding $maxreqs |
239 |
root |
1.1 |
|
240 |
|
|
This is a very bad function to use in interactive programs because it |
241 |
|
|
blocks, and a bad way to reduce concurrency because it is inexact: Better |
242 |
|
|
use an C<aio_group> together with a feed callback. |
243 |
|
|
|
244 |
|
|
Sets the maximum number of outstanding requests to C<$nreqs>. If you |
245 |
|
|
to queue up more than this number of requests, the next call to the |
246 |
|
|
C<poll_cb> (and C<poll_some> and other functions calling C<poll_cb>) |
247 |
|
|
function will block until the limit is no longer exceeded. |
248 |
|
|
|
249 |
|
|
The default value is very large, so there is no practical limit on the |
250 |
|
|
number of outstanding requests. |
251 |
|
|
|
252 |
|
|
You can still queue as many requests as you want. Therefore, |
253 |
|
|
C<max_oustsanding> is mainly useful in simple scripts (with low values) or |
254 |
|
|
as a stop gap to shield against fatal memory overflow (with large values). |
255 |
|
|
|
256 |
root |
1.3 |
=item BDB::set_sync_prepare $cb |
257 |
|
|
|
258 |
|
|
Sets a callback that is called whenever a request is created without an |
259 |
|
|
explicit callback. It has to return two code references. The first is used |
260 |
|
|
as the request callback, and the second is called to wait until the first |
261 |
|
|
callback has been called. The default implementation works like this: |
262 |
|
|
|
263 |
|
|
sub { |
264 |
|
|
my $status; |
265 |
|
|
( |
266 |
|
|
sub { $status = $! }, |
267 |
|
|
sub { BDB::poll while !defined $status; $! = $status }, |
268 |
|
|
) |
269 |
|
|
} |
270 |
|
|
|
271 |
|
|
=back |
272 |
|
|
|
273 |
root |
1.1 |
=head3 STATISTICAL INFORMATION |
274 |
|
|
|
275 |
root |
1.3 |
=over 4 |
276 |
|
|
|
277 |
root |
1.2 |
=item BDB::nreqs |
278 |
root |
1.1 |
|
279 |
|
|
Returns the number of requests currently in the ready, execute or pending |
280 |
|
|
states (i.e. for which their callback has not been invoked yet). |
281 |
|
|
|
282 |
|
|
Example: wait till there are no outstanding requests anymore: |
283 |
|
|
|
284 |
root |
1.2 |
BDB::poll_wait, BDB::poll_cb |
285 |
|
|
while BDB::nreqs; |
286 |
root |
1.1 |
|
287 |
root |
1.2 |
=item BDB::nready |
288 |
root |
1.1 |
|
289 |
|
|
Returns the number of requests currently in the ready state (not yet |
290 |
|
|
executed). |
291 |
|
|
|
292 |
root |
1.2 |
=item BDB::npending |
293 |
root |
1.1 |
|
294 |
|
|
Returns the number of requests currently in the pending state (executed, |
295 |
|
|
but not yet processed by poll_cb). |
296 |
|
|
|
297 |
|
|
=back |
298 |
|
|
|
299 |
|
|
=cut |
300 |
|
|
|
301 |
root |
1.3 |
set_sync_prepare { |
302 |
|
|
my $status; |
303 |
|
|
( |
304 |
|
|
sub { |
305 |
|
|
$status = $!; |
306 |
|
|
}, |
307 |
|
|
sub { |
308 |
|
|
BDB::poll while !defined $status; |
309 |
|
|
$! = $status; |
310 |
|
|
}, |
311 |
|
|
) |
312 |
|
|
}; |
313 |
|
|
|
314 |
root |
1.1 |
min_parallel 8; |
315 |
|
|
|
316 |
|
|
END { flush } |
317 |
|
|
|
318 |
|
|
1; |
319 |
|
|
|
320 |
|
|
=head2 FORK BEHAVIOUR |
321 |
|
|
|
322 |
|
|
This module should do "the right thing" when the process using it forks: |
323 |
|
|
|
324 |
|
|
Before the fork, IO::AIO enters a quiescent state where no requests |
325 |
|
|
can be added in other threads and no results will be processed. After |
326 |
|
|
the fork the parent simply leaves the quiescent state and continues |
327 |
|
|
request/result processing, while the child frees the request/result queue |
328 |
|
|
(so that the requests started before the fork will only be handled in the |
329 |
|
|
parent). Threads will be started on demand until the limit set in the |
330 |
|
|
parent process has been reached again. |
331 |
|
|
|
332 |
|
|
In short: the parent will, after a short pause, continue as if fork had |
333 |
|
|
not been called, while the child will act as if IO::AIO has not been used |
334 |
|
|
yet. |
335 |
|
|
|
336 |
|
|
=head2 MEMORY USAGE |
337 |
|
|
|
338 |
|
|
Per-request usage: |
339 |
|
|
|
340 |
|
|
Each aio request uses - depending on your architecture - around 100-200 |
341 |
|
|
bytes of memory. In addition, stat requests need a stat buffer (possibly |
342 |
|
|
a few hundred bytes), readdir requires a result buffer and so on. Perl |
343 |
|
|
scalars and other data passed into aio requests will also be locked and |
344 |
|
|
will consume memory till the request has entered the done state. |
345 |
|
|
|
346 |
|
|
This is now awfully much, so queuing lots of requests is not usually a |
347 |
|
|
problem. |
348 |
|
|
|
349 |
|
|
Per-thread usage: |
350 |
|
|
|
351 |
|
|
In the execution phase, some aio requests require more memory for |
352 |
|
|
temporary buffers, and each thread requires a stack and other data |
353 |
|
|
structures (usually around 16k-128k, depending on the OS). |
354 |
|
|
|
355 |
|
|
=head1 KNOWN BUGS |
356 |
|
|
|
357 |
|
|
Known bugs will be fixed in the next release. |
358 |
|
|
|
359 |
|
|
=head1 SEE ALSO |
360 |
|
|
|
361 |
|
|
L<Coro::AIO>. |
362 |
|
|
|
363 |
|
|
=head1 AUTHOR |
364 |
|
|
|
365 |
|
|
Marc Lehmann <schmorp@schmorp.de> |
366 |
|
|
http://home.schmorp.de/ |
367 |
|
|
|
368 |
|
|
=cut |
369 |
|
|
|