ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/BDB/BDB.pm
(Generate patch)

Comparing BDB/BDB.pm (file contents):
Revision 1.1 by root, Mon Feb 5 18:40:55 2007 UTC vs.
Revision 1.4 by root, Mon Feb 5 23:46:15 2007 UTC

1=head1 NAME 1=head1 NAME
2 2
3BDB::AIO - Asynchronous Berkeley DB access 3BDB - Asynchronous Berkeley DB access
4 4
5=head1 SYNOPSIS 5=head1 SYNOPSIS
6 6
7 use BDB::AIO; 7 use BDB;
8 8
9=head1 DESCRIPTION 9=head1 DESCRIPTION
10 10
11=head2 EXAMPLE 11=head2 EXAMPLE
12 12
… …
55 55
56=back 56=back
57 57
58=cut 58=cut
59 59
60package BDB::AIO; 60package BDB;
61 61
62no warnings; 62no warnings;
63use strict 'vars'; 63use strict 'vars';
64 64
65use base 'Exporter'; 65use base 'Exporter';
66 66
67BEGIN { 67BEGIN {
68 our $VERSION = '0.1'; 68 our $VERSION = '0.1';
69 69
70 our @BDB_REQ = qw(); 70 our @BDB_REQ = qw(
71 db_env_create db_env_open db_env_close
72 db_create db_open db_close db_compact db_sync db_put db_get db_pget
73 db_txn_commit db_txn_abort
74 );
75 our @EXPORT = (@BDB_REQ, qw(dbreq_pri dbreq_nice));
71 our @EXPORT_OK = qw(poll_fileno poll_cb poll_wait flush 76 our @EXPORT_OK = qw(poll_fileno poll_cb poll_wait flush
72 min_parallel max_parallel max_idle 77 min_parallel max_parallel max_idle
73 nreqs nready npending nthreads 78 nreqs nready npending nthreads
74 max_poll_time max_poll_reqs); 79 max_poll_time max_poll_reqs);
75 80
76 require XSLoader; 81 require XSLoader;
77 XSLoader::load ("BDB::AIO", $VERSION); 82 XSLoader::load ("BDB", $VERSION);
78} 83}
79 84
80=head2 SUPPORT FUNCTIONS 85=head2 SUPPORT FUNCTIONS
81 86
82=head3 EVENT PROCESSING AND EVENT LOOP INTEGRATION 87=head3 EVENT PROCESSING AND EVENT LOOP INTEGRATION
83 88
84=over 4 89=over 4
85 90
86=item $fileno = BDB::AIO::poll_fileno 91=item $fileno = BDB::poll_fileno
87 92
88Return the I<request result pipe file descriptor>. This filehandle must be 93Return the I<request result pipe file descriptor>. This filehandle must be
89polled for reading by some mechanism outside this module (e.g. Event or 94polled for reading by some mechanism outside this module (e.g. Event or
90select, see below or the SYNOPSIS). If the pipe becomes readable you have 95select, see below or the SYNOPSIS). If the pipe becomes readable you have
91to call C<poll_cb> to check the results. 96to call C<poll_cb> to check the results.
92 97
93See C<poll_cb> for an example. 98See C<poll_cb> for an example.
94 99
95=item BDB::AIO::poll_cb 100=item BDB::poll_cb
96 101
97Process some outstanding events on the result pipe. You have to call this 102Process some outstanding events on the result pipe. You have to call this
98regularly. Returns the number of events processed. Returns immediately 103regularly. Returns the number of events processed. Returns immediately
99when no events are outstanding. The amount of events processed depends on 104when no events are outstanding. The amount of events processed depends on
100the settings of C<BDB::AIO::max_poll_req> and C<BDB::AIO::max_poll_time>. 105the settings of C<BDB::max_poll_req> and C<BDB::max_poll_time>.
101 106
102If not all requests were processed for whatever reason, the filehandle 107If not all requests were processed for whatever reason, the filehandle
103will still be ready when C<poll_cb> returns. 108will still be ready when C<poll_cb> returns.
104 109
105Example: Install an Event watcher that automatically calls 110Example: Install an Event watcher that automatically calls
106BDB::AIO::poll_cb with high priority: 111BDB::poll_cb with high priority:
107 112
108 Event->io (fd => BDB::AIO::poll_fileno, 113 Event->io (fd => BDB::poll_fileno,
109 poll => 'r', async => 1, 114 poll => 'r', async => 1,
110 cb => \&BDB::AIO::poll_cb); 115 cb => \&BDB::poll_cb);
111 116
112=item BDB::AIO::max_poll_reqs $nreqs 117=item BDB::max_poll_reqs $nreqs
113 118
114=item BDB::AIO::max_poll_time $seconds 119=item BDB::max_poll_time $seconds
115 120
116These set the maximum number of requests (default C<0>, meaning infinity) 121These set the maximum number of requests (default C<0>, meaning infinity)
117that are being processed by C<BDB::AIO::poll_cb> in one call, respectively 122that are being processed by C<BDB::poll_cb> in one call, respectively
118the maximum amount of time (default C<0>, meaning infinity) spent in 123the maximum amount of time (default C<0>, meaning infinity) spent in
119C<BDB::AIO::poll_cb> to process requests (more correctly the mininum amount 124C<BDB::poll_cb> to process requests (more correctly the mininum amount
120of time C<poll_cb> is allowed to use). 125of time C<poll_cb> is allowed to use).
121 126
122Setting C<max_poll_time> to a non-zero value creates an overhead of one 127Setting C<max_poll_time> to a non-zero value creates an overhead of one
123syscall per request processed, which is not normally a problem unless your 128syscall per request processed, which is not normally a problem unless your
124callbacks are really really fast or your OS is really really slow (I am 129callbacks are really really fast or your OS is really really slow (I am
… …
129time. 134time.
130 135
131For interactive programs, values such as C<0.01> to C<0.1> should be fine. 136For interactive programs, values such as C<0.01> to C<0.1> should be fine.
132 137
133Example: Install an Event watcher that automatically calls 138Example: Install an Event watcher that automatically calls
134BDB::AIO::poll_cb with low priority, to ensure that other parts of the 139BDB::poll_cb with low priority, to ensure that other parts of the
135program get the CPU sometimes even under high AIO load. 140program get the CPU sometimes even under high AIO load.
136 141
137 # try not to spend much more than 0.1s in poll_cb 142 # try not to spend much more than 0.1s in poll_cb
138 BDB::AIO::max_poll_time 0.1; 143 BDB::max_poll_time 0.1;
139 144
140 # use a low priority so other tasks have priority 145 # use a low priority so other tasks have priority
141 Event->io (fd => BDB::AIO::poll_fileno, 146 Event->io (fd => BDB::poll_fileno,
142 poll => 'r', nice => 1, 147 poll => 'r', nice => 1,
143 cb => &BDB::AIO::poll_cb); 148 cb => &BDB::poll_cb);
144 149
145=item BDB::AIO::poll_wait 150=item BDB::poll_wait
146 151
147If there are any outstanding requests and none of them in the result 152If there are any outstanding requests and none of them in the result
148phase, wait till the result filehandle becomes ready for reading (simply 153phase, wait till the result filehandle becomes ready for reading (simply
149does a C<select> on the filehandle. This is useful if you want to 154does a C<select> on the filehandle. This is useful if you want to
150synchronously wait for some requests to finish). 155synchronously wait for some requests to finish).
151 156
152See C<nreqs> for an example. 157See C<nreqs> for an example.
153 158
154=item BDB::AIO::poll 159=item BDB::poll
155 160
156Waits until some requests have been handled. 161Waits until some requests have been handled.
157 162
158Returns the number of requests processed, but is otherwise strictly 163Returns the number of requests processed, but is otherwise strictly
159equivalent to: 164equivalent to:
160 165
161 BDB::AIO::poll_wait, BDB::AIO::poll_cb 166 BDB::poll_wait, BDB::poll_cb
162 167
163=item BDB::AIO::flush 168=item BDB::flush
164 169
165Wait till all outstanding AIO requests have been handled. 170Wait till all outstanding AIO requests have been handled.
166 171
167Strictly equivalent to: 172Strictly equivalent to:
168 173
169 BDB::AIO::poll_wait, BDB::AIO::poll_cb 174 BDB::poll_wait, BDB::poll_cb
170 while BDB::AIO::nreqs; 175 while BDB::nreqs;
171 176
172=head3 CONTROLLING THE NUMBER OF THREADS 177=head3 CONTROLLING THE NUMBER OF THREADS
173 178
174=item BDB::AIO::min_parallel $nthreads 179=item BDB::min_parallel $nthreads
175 180
176Set the minimum number of AIO threads to C<$nthreads>. The current 181Set the minimum number of AIO threads to C<$nthreads>. The current
177default is C<8>, which means eight asynchronous operations can execute 182default is C<8>, which means eight asynchronous operations can execute
178concurrently at any one time (the number of outstanding requests, 183concurrently at any one time (the number of outstanding requests,
179however, is unlimited). 184however, is unlimited).
180 185
181BDB::AIO starts threads only on demand, when an AIO request is queued and 186BDB starts threads only on demand, when an AIO request is queued and
182no free thread exists. Please note that queueing up a hundred requests can 187no free thread exists. Please note that queueing up a hundred requests can
183create demand for a hundred threads, even if it turns out that everything 188create demand for a hundred threads, even if it turns out that everything
184is in the cache and could have been processed faster by a single thread. 189is in the cache and could have been processed faster by a single thread.
185 190
186It is recommended to keep the number of threads relatively low, as some 191It is recommended to keep the number of threads relatively low, as some
… …
189versions, 4-32 threads should be fine. 194versions, 4-32 threads should be fine.
190 195
191Under most circumstances you don't need to call this function, as the 196Under most circumstances you don't need to call this function, as the
192module selects a default that is suitable for low to moderate load. 197module selects a default that is suitable for low to moderate load.
193 198
194=item BDB::AIO::max_parallel $nthreads 199=item BDB::max_parallel $nthreads
195 200
196Sets the maximum number of AIO threads to C<$nthreads>. If more than the 201Sets the maximum number of AIO threads to C<$nthreads>. If more than the
197specified number of threads are currently running, this function kills 202specified number of threads are currently running, this function kills
198them. This function blocks until the limit is reached. 203them. This function blocks until the limit is reached.
199 204
… …
203This module automatically runs C<max_parallel 0> at program end, to ensure 208This module automatically runs C<max_parallel 0> at program end, to ensure
204that all threads are killed and that there are no outstanding requests. 209that all threads are killed and that there are no outstanding requests.
205 210
206Under normal circumstances you don't need to call this function. 211Under normal circumstances you don't need to call this function.
207 212
208=item BDB::AIO::max_idle $nthreads 213=item BDB::max_idle $nthreads
209 214
210Limit the number of threads (default: 4) that are allowed to idle (i.e., 215Limit the number of threads (default: 4) that are allowed to idle (i.e.,
211threads that did not get a request to process within 10 seconds). That 216threads that did not get a request to process within 10 seconds). That
212means if a thread becomes idle while C<$nthreads> other threads are also 217means if a thread becomes idle while C<$nthreads> other threads are also
213idle, it will free its resources and exit. 218idle, it will free its resources and exit.
… …
218 223
219The default is probably ok in most situations, especially if thread 224The default is probably ok in most situations, especially if thread
220creation is fast. If thread creation is very slow on your system you might 225creation is fast. If thread creation is very slow on your system you might
221want to use larger values. 226want to use larger values.
222 227
223=item $oldmaxreqs = BDB::AIO::max_outstanding $maxreqs 228=item $oldmaxreqs = BDB::max_outstanding $maxreqs
224 229
225This is a very bad function to use in interactive programs because it 230This is a very bad function to use in interactive programs because it
226blocks, and a bad way to reduce concurrency because it is inexact: Better 231blocks, and a bad way to reduce concurrency because it is inexact: Better
227use an C<aio_group> together with a feed callback. 232use an C<aio_group> together with a feed callback.
228 233
… …
236 241
237You can still queue as many requests as you want. Therefore, 242You can still queue as many requests as you want. Therefore,
238C<max_oustsanding> is mainly useful in simple scripts (with low values) or 243C<max_oustsanding> is mainly useful in simple scripts (with low values) or
239as a stop gap to shield against fatal memory overflow (with large values). 244as a stop gap to shield against fatal memory overflow (with large values).
240 245
246=item BDB::set_sync_prepare $cb
247
248Sets a callback that is called whenever a request is created without an
249explicit callback. It has to return two code references. The first is used
250as the request callback, and the second is called to wait until the first
251callback has been called. The default implementation works like this:
252
253 sub {
254 my $status;
255 (
256 sub { $status = $! },
257 sub { BDB::poll while !defined $status; $! = $status },
258 )
259 }
260
261=back
262
241=head3 STATISTICAL INFORMATION 263=head3 STATISTICAL INFORMATION
242 264
265=over 4
266
243=item BDB::AIO::nreqs 267=item BDB::nreqs
244 268
245Returns the number of requests currently in the ready, execute or pending 269Returns the number of requests currently in the ready, execute or pending
246states (i.e. for which their callback has not been invoked yet). 270states (i.e. for which their callback has not been invoked yet).
247 271
248Example: wait till there are no outstanding requests anymore: 272Example: wait till there are no outstanding requests anymore:
249 273
250 BDB::AIO::poll_wait, BDB::AIO::poll_cb 274 BDB::poll_wait, BDB::poll_cb
251 while BDB::AIO::nreqs; 275 while BDB::nreqs;
252 276
253=item BDB::AIO::nready 277=item BDB::nready
254 278
255Returns the number of requests currently in the ready state (not yet 279Returns the number of requests currently in the ready state (not yet
256executed). 280executed).
257 281
258=item BDB::AIO::npending 282=item BDB::npending
259 283
260Returns the number of requests currently in the pending state (executed, 284Returns the number of requests currently in the pending state (executed,
261but not yet processed by poll_cb). 285but not yet processed by poll_cb).
262 286
263=back 287=back
264 288
265=cut 289=cut
266 290
267# support function to convert a fd into a perl filehandle 291set_sync_prepare {
268sub _fd2fh { 292 my $status;
269 return undef if $_[0] < 0; 293 (
270 294 sub {
271 # try to generate nice filehandles 295 $status = $!;
272 my $sym = "BDB::AIO::fd#$_[0]"; 296 },
273 local *$sym; 297 sub {
274 298 BDB::poll while !defined $status;
275 open *$sym, "+<&=$_[0]" # usually works under any unix 299 $! = $status;
276 or open *$sym, "<&=$_[0]" # cygwin needs this 300 },
277 or open *$sym, ">&=$_[0]" # or this 301 )
278 or return undef; 302};
279
280 *$sym
281}
282 303
283min_parallel 8; 304min_parallel 8;
284 305
285END { flush } 306END { flush }
286 307

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines