ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/BDB/BDB.pm
(Generate patch)

Comparing BDB/BDB.pm (file contents):
Revision 1.1 by root, Mon Feb 5 18:40:55 2007 UTC vs.
Revision 1.3 by root, Mon Feb 5 22:19:07 2007 UTC

1=head1 NAME 1=head1 NAME
2 2
3BDB::AIO - Asynchronous Berkeley DB access 3BDB - Asynchronous Berkeley DB access
4 4
5=head1 SYNOPSIS 5=head1 SYNOPSIS
6 6
7 use BDB::AIO; 7 use BDB;
8 8
9=head1 DESCRIPTION 9=head1 DESCRIPTION
10 10
11=head2 EXAMPLE 11=head2 EXAMPLE
12 12
… …
55 55
56=back 56=back
57 57
58=cut 58=cut
59 59
60package BDB::AIO; 60package BDB;
61 61
62no warnings; 62no warnings;
63use strict 'vars'; 63use strict 'vars';
64 64
65use base 'Exporter'; 65use base 'Exporter';
66 66
67BEGIN { 67BEGIN {
68 our $VERSION = '0.1'; 68 our $VERSION = '0.1';
69 69
70 our @BDB_REQ = qw(); 70 our @BDB_REQ = qw(
71 db_env_create db_env_open db_env_close
72 db_create db_open db_close db_compact db_sync db_put
73 );
74 our @EXPORT = (@BDB_REQ, qw(dbreq_pri dbreq_nice));
71 our @EXPORT_OK = qw(poll_fileno poll_cb poll_wait flush 75 our @EXPORT_OK = qw(poll_fileno poll_cb poll_wait flush
72 min_parallel max_parallel max_idle 76 min_parallel max_parallel max_idle
73 nreqs nready npending nthreads 77 nreqs nready npending nthreads
74 max_poll_time max_poll_reqs); 78 max_poll_time max_poll_reqs);
75 79
76 require XSLoader; 80 require XSLoader;
77 XSLoader::load ("BDB::AIO", $VERSION); 81 XSLoader::load ("BDB", $VERSION);
78} 82}
79 83
80=head2 SUPPORT FUNCTIONS 84=head2 SUPPORT FUNCTIONS
81 85
82=head3 EVENT PROCESSING AND EVENT LOOP INTEGRATION 86=head3 EVENT PROCESSING AND EVENT LOOP INTEGRATION
83 87
84=over 4 88=over 4
85 89
86=item $fileno = BDB::AIO::poll_fileno 90=item $fileno = BDB::poll_fileno
87 91
88Return the I<request result pipe file descriptor>. This filehandle must be 92Return the I<request result pipe file descriptor>. This filehandle must be
89polled for reading by some mechanism outside this module (e.g. Event or 93polled for reading by some mechanism outside this module (e.g. Event or
90select, see below or the SYNOPSIS). If the pipe becomes readable you have 94select, see below or the SYNOPSIS). If the pipe becomes readable you have
91to call C<poll_cb> to check the results. 95to call C<poll_cb> to check the results.
92 96
93See C<poll_cb> for an example. 97See C<poll_cb> for an example.
94 98
95=item BDB::AIO::poll_cb 99=item BDB::poll_cb
96 100
97Process some outstanding events on the result pipe. You have to call this 101Process some outstanding events on the result pipe. You have to call this
98regularly. Returns the number of events processed. Returns immediately 102regularly. Returns the number of events processed. Returns immediately
99when no events are outstanding. The amount of events processed depends on 103when no events are outstanding. The amount of events processed depends on
100the settings of C<BDB::AIO::max_poll_req> and C<BDB::AIO::max_poll_time>. 104the settings of C<BDB::max_poll_req> and C<BDB::max_poll_time>.
101 105
102If not all requests were processed for whatever reason, the filehandle 106If not all requests were processed for whatever reason, the filehandle
103will still be ready when C<poll_cb> returns. 107will still be ready when C<poll_cb> returns.
104 108
105Example: Install an Event watcher that automatically calls 109Example: Install an Event watcher that automatically calls
106BDB::AIO::poll_cb with high priority: 110BDB::poll_cb with high priority:
107 111
108 Event->io (fd => BDB::AIO::poll_fileno, 112 Event->io (fd => BDB::poll_fileno,
109 poll => 'r', async => 1, 113 poll => 'r', async => 1,
110 cb => \&BDB::AIO::poll_cb); 114 cb => \&BDB::poll_cb);
111 115
112=item BDB::AIO::max_poll_reqs $nreqs 116=item BDB::max_poll_reqs $nreqs
113 117
114=item BDB::AIO::max_poll_time $seconds 118=item BDB::max_poll_time $seconds
115 119
116These set the maximum number of requests (default C<0>, meaning infinity) 120These set the maximum number of requests (default C<0>, meaning infinity)
117that are being processed by C<BDB::AIO::poll_cb> in one call, respectively 121that are being processed by C<BDB::poll_cb> in one call, respectively
118the maximum amount of time (default C<0>, meaning infinity) spent in 122the maximum amount of time (default C<0>, meaning infinity) spent in
119C<BDB::AIO::poll_cb> to process requests (more correctly the mininum amount 123C<BDB::poll_cb> to process requests (more correctly the mininum amount
120of time C<poll_cb> is allowed to use). 124of time C<poll_cb> is allowed to use).
121 125
122Setting C<max_poll_time> to a non-zero value creates an overhead of one 126Setting C<max_poll_time> to a non-zero value creates an overhead of one
123syscall per request processed, which is not normally a problem unless your 127syscall per request processed, which is not normally a problem unless your
124callbacks are really really fast or your OS is really really slow (I am 128callbacks are really really fast or your OS is really really slow (I am
… …
129time. 133time.
130 134
131For interactive programs, values such as C<0.01> to C<0.1> should be fine. 135For interactive programs, values such as C<0.01> to C<0.1> should be fine.
132 136
133Example: Install an Event watcher that automatically calls 137Example: Install an Event watcher that automatically calls
134BDB::AIO::poll_cb with low priority, to ensure that other parts of the 138BDB::poll_cb with low priority, to ensure that other parts of the
135program get the CPU sometimes even under high AIO load. 139program get the CPU sometimes even under high AIO load.
136 140
137 # try not to spend much more than 0.1s in poll_cb 141 # try not to spend much more than 0.1s in poll_cb
138 BDB::AIO::max_poll_time 0.1; 142 BDB::max_poll_time 0.1;
139 143
140 # use a low priority so other tasks have priority 144 # use a low priority so other tasks have priority
141 Event->io (fd => BDB::AIO::poll_fileno, 145 Event->io (fd => BDB::poll_fileno,
142 poll => 'r', nice => 1, 146 poll => 'r', nice => 1,
143 cb => &BDB::AIO::poll_cb); 147 cb => &BDB::poll_cb);
144 148
145=item BDB::AIO::poll_wait 149=item BDB::poll_wait
146 150
147If there are any outstanding requests and none of them in the result 151If there are any outstanding requests and none of them in the result
148phase, wait till the result filehandle becomes ready for reading (simply 152phase, wait till the result filehandle becomes ready for reading (simply
149does a C<select> on the filehandle. This is useful if you want to 153does a C<select> on the filehandle. This is useful if you want to
150synchronously wait for some requests to finish). 154synchronously wait for some requests to finish).
151 155
152See C<nreqs> for an example. 156See C<nreqs> for an example.
153 157
154=item BDB::AIO::poll 158=item BDB::poll
155 159
156Waits until some requests have been handled. 160Waits until some requests have been handled.
157 161
158Returns the number of requests processed, but is otherwise strictly 162Returns the number of requests processed, but is otherwise strictly
159equivalent to: 163equivalent to:
160 164
161 BDB::AIO::poll_wait, BDB::AIO::poll_cb 165 BDB::poll_wait, BDB::poll_cb
162 166
163=item BDB::AIO::flush 167=item BDB::flush
164 168
165Wait till all outstanding AIO requests have been handled. 169Wait till all outstanding AIO requests have been handled.
166 170
167Strictly equivalent to: 171Strictly equivalent to:
168 172
169 BDB::AIO::poll_wait, BDB::AIO::poll_cb 173 BDB::poll_wait, BDB::poll_cb
170 while BDB::AIO::nreqs; 174 while BDB::nreqs;
171 175
172=head3 CONTROLLING THE NUMBER OF THREADS 176=head3 CONTROLLING THE NUMBER OF THREADS
173 177
174=item BDB::AIO::min_parallel $nthreads 178=item BDB::min_parallel $nthreads
175 179
176Set the minimum number of AIO threads to C<$nthreads>. The current 180Set the minimum number of AIO threads to C<$nthreads>. The current
177default is C<8>, which means eight asynchronous operations can execute 181default is C<8>, which means eight asynchronous operations can execute
178concurrently at any one time (the number of outstanding requests, 182concurrently at any one time (the number of outstanding requests,
179however, is unlimited). 183however, is unlimited).
180 184
181BDB::AIO starts threads only on demand, when an AIO request is queued and 185BDB starts threads only on demand, when an AIO request is queued and
182no free thread exists. Please note that queueing up a hundred requests can 186no free thread exists. Please note that queueing up a hundred requests can
183create demand for a hundred threads, even if it turns out that everything 187create demand for a hundred threads, even if it turns out that everything
184is in the cache and could have been processed faster by a single thread. 188is in the cache and could have been processed faster by a single thread.
185 189
186It is recommended to keep the number of threads relatively low, as some 190It is recommended to keep the number of threads relatively low, as some
… …
189versions, 4-32 threads should be fine. 193versions, 4-32 threads should be fine.
190 194
191Under most circumstances you don't need to call this function, as the 195Under most circumstances you don't need to call this function, as the
192module selects a default that is suitable for low to moderate load. 196module selects a default that is suitable for low to moderate load.
193 197
194=item BDB::AIO::max_parallel $nthreads 198=item BDB::max_parallel $nthreads
195 199
196Sets the maximum number of AIO threads to C<$nthreads>. If more than the 200Sets the maximum number of AIO threads to C<$nthreads>. If more than the
197specified number of threads are currently running, this function kills 201specified number of threads are currently running, this function kills
198them. This function blocks until the limit is reached. 202them. This function blocks until the limit is reached.
199 203
… …
203This module automatically runs C<max_parallel 0> at program end, to ensure 207This module automatically runs C<max_parallel 0> at program end, to ensure
204that all threads are killed and that there are no outstanding requests. 208that all threads are killed and that there are no outstanding requests.
205 209
206Under normal circumstances you don't need to call this function. 210Under normal circumstances you don't need to call this function.
207 211
208=item BDB::AIO::max_idle $nthreads 212=item BDB::max_idle $nthreads
209 213
210Limit the number of threads (default: 4) that are allowed to idle (i.e., 214Limit the number of threads (default: 4) that are allowed to idle (i.e.,
211threads that did not get a request to process within 10 seconds). That 215threads that did not get a request to process within 10 seconds). That
212means if a thread becomes idle while C<$nthreads> other threads are also 216means if a thread becomes idle while C<$nthreads> other threads are also
213idle, it will free its resources and exit. 217idle, it will free its resources and exit.
… …
218 222
219The default is probably ok in most situations, especially if thread 223The default is probably ok in most situations, especially if thread
220creation is fast. If thread creation is very slow on your system you might 224creation is fast. If thread creation is very slow on your system you might
221want to use larger values. 225want to use larger values.
222 226
223=item $oldmaxreqs = BDB::AIO::max_outstanding $maxreqs 227=item $oldmaxreqs = BDB::max_outstanding $maxreqs
224 228
225This is a very bad function to use in interactive programs because it 229This is a very bad function to use in interactive programs because it
226blocks, and a bad way to reduce concurrency because it is inexact: Better 230blocks, and a bad way to reduce concurrency because it is inexact: Better
227use an C<aio_group> together with a feed callback. 231use an C<aio_group> together with a feed callback.
228 232
… …
236 240
237You can still queue as many requests as you want. Therefore, 241You can still queue as many requests as you want. Therefore,
238C<max_oustsanding> is mainly useful in simple scripts (with low values) or 242C<max_oustsanding> is mainly useful in simple scripts (with low values) or
239as a stop gap to shield against fatal memory overflow (with large values). 243as a stop gap to shield against fatal memory overflow (with large values).
240 244
245=item BDB::set_sync_prepare $cb
246
247Sets a callback that is called whenever a request is created without an
248explicit callback. It has to return two code references. The first is used
249as the request callback, and the second is called to wait until the first
250callback has been called. The default implementation works like this:
251
252 sub {
253 my $status;
254 (
255 sub { $status = $! },
256 sub { BDB::poll while !defined $status; $! = $status },
257 )
258 }
259
260=back
261
241=head3 STATISTICAL INFORMATION 262=head3 STATISTICAL INFORMATION
242 263
264=over 4
265
243=item BDB::AIO::nreqs 266=item BDB::nreqs
244 267
245Returns the number of requests currently in the ready, execute or pending 268Returns the number of requests currently in the ready, execute or pending
246states (i.e. for which their callback has not been invoked yet). 269states (i.e. for which their callback has not been invoked yet).
247 270
248Example: wait till there are no outstanding requests anymore: 271Example: wait till there are no outstanding requests anymore:
249 272
250 BDB::AIO::poll_wait, BDB::AIO::poll_cb 273 BDB::poll_wait, BDB::poll_cb
251 while BDB::AIO::nreqs; 274 while BDB::nreqs;
252 275
253=item BDB::AIO::nready 276=item BDB::nready
254 277
255Returns the number of requests currently in the ready state (not yet 278Returns the number of requests currently in the ready state (not yet
256executed). 279executed).
257 280
258=item BDB::AIO::npending 281=item BDB::npending
259 282
260Returns the number of requests currently in the pending state (executed, 283Returns the number of requests currently in the pending state (executed,
261but not yet processed by poll_cb). 284but not yet processed by poll_cb).
262 285
263=back 286=back
264 287
265=cut 288=cut
266 289
267# support function to convert a fd into a perl filehandle 290set_sync_prepare {
268sub _fd2fh { 291 my $status;
269 return undef if $_[0] < 0; 292 (
270 293 sub {
271 # try to generate nice filehandles 294 $status = $!;
272 my $sym = "BDB::AIO::fd#$_[0]"; 295 },
273 local *$sym; 296 sub {
274 297 BDB::poll while !defined $status;
275 open *$sym, "+<&=$_[0]" # usually works under any unix 298 $! = $status;
276 or open *$sym, "<&=$_[0]" # cygwin needs this 299 },
277 or open *$sym, ">&=$_[0]" # or this 300 )
278 or return undef; 301};
279
280 *$sym
281}
282 302
283min_parallel 8; 303min_parallel 8;
284 304
285END { flush } 305END { flush }
286 306

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines