ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/BDB/BDB.pm
(Generate patch)

Comparing BDB/BDB.pm (file contents):
Revision 1.1 by root, Mon Feb 5 18:40:55 2007 UTC vs.
Revision 1.6 by root, Sun Feb 11 22:07:23 2007 UTC

1=head1 NAME 1=head1 NAME
2 2
3BDB::AIO - Asynchronous Berkeley DB access 3BDB - Asynchronous Berkeley DB access
4 4
5=head1 SYNOPSIS 5=head1 SYNOPSIS
6 6
7 use BDB::AIO; 7 use BDB;
8 8
9=head1 DESCRIPTION 9=head1 DESCRIPTION
10 10
11=head2 EXAMPLE 11=head2 EXAMPLE
12 12
… …
55 55
56=back 56=back
57 57
58=cut 58=cut
59 59
60package BDB::AIO; 60package BDB;
61 61
62no warnings; 62no warnings;
63use strict 'vars'; 63use strict 'vars';
64 64
65use base 'Exporter'; 65use base 'Exporter';
66 66
67BEGIN { 67BEGIN {
68 our $VERSION = '0.1'; 68 our $VERSION = '0.1';
69 69
70 our @BDB_REQ = qw(); 70 our @BDB_REQ = qw(
71 db_env_open db_env_close db_env_txn_checkpoint db_env_lock_detect
72 db_env_memp_sync db_env_memp_trickle
73 db_open db_close db_compact db_sync db_put db_get db_pget db_del db_key_range
74 db_txn_commit db_txn_abort
75 db_c_close db_c_count db_c_put db_c_get db_c_pget db_c_del
76 db_sequence_open db_sequence_close
77 db_sequence_get db_sequence_remove
78 );
79 our @EXPORT = (@BDB_REQ, qw(dbreq_pri dbreq_nice db_env_create db_create));
80 our @EXPORT_OK = qw(
71 our @EXPORT_OK = qw(poll_fileno poll_cb poll_wait flush 81 poll_fileno poll_cb poll_wait flush
72 min_parallel max_parallel max_idle 82 min_parallel max_parallel max_idle
73 nreqs nready npending nthreads 83 nreqs nready npending nthreads
74 max_poll_time max_poll_reqs); 84 max_poll_time max_poll_reqs
85 );
75 86
76 require XSLoader; 87 require XSLoader;
77 XSLoader::load ("BDB::AIO", $VERSION); 88 XSLoader::load ("BDB", $VERSION);
78} 89}
79 90
80=head2 SUPPORT FUNCTIONS 91=head2 SUPPORT FUNCTIONS
81 92
82=head3 EVENT PROCESSING AND EVENT LOOP INTEGRATION 93=head3 EVENT PROCESSING AND EVENT LOOP INTEGRATION
83 94
84=over 4 95=over 4
85 96
86=item $fileno = BDB::AIO::poll_fileno 97=item $fileno = BDB::poll_fileno
87 98
88Return the I<request result pipe file descriptor>. This filehandle must be 99Return the I<request result pipe file descriptor>. This filehandle must be
89polled for reading by some mechanism outside this module (e.g. Event or 100polled for reading by some mechanism outside this module (e.g. Event or
90select, see below or the SYNOPSIS). If the pipe becomes readable you have 101select, see below or the SYNOPSIS). If the pipe becomes readable you have
91to call C<poll_cb> to check the results. 102to call C<poll_cb> to check the results.
92 103
93See C<poll_cb> for an example. 104See C<poll_cb> for an example.
94 105
95=item BDB::AIO::poll_cb 106=item BDB::poll_cb
96 107
97Process some outstanding events on the result pipe. You have to call this 108Process some outstanding events on the result pipe. You have to call this
98regularly. Returns the number of events processed. Returns immediately 109regularly. Returns the number of events processed. Returns immediately
99when no events are outstanding. The amount of events processed depends on 110when no events are outstanding. The amount of events processed depends on
100the settings of C<BDB::AIO::max_poll_req> and C<BDB::AIO::max_poll_time>. 111the settings of C<BDB::max_poll_req> and C<BDB::max_poll_time>.
101 112
102If not all requests were processed for whatever reason, the filehandle 113If not all requests were processed for whatever reason, the filehandle
103will still be ready when C<poll_cb> returns. 114will still be ready when C<poll_cb> returns.
104 115
105Example: Install an Event watcher that automatically calls 116Example: Install an Event watcher that automatically calls
106BDB::AIO::poll_cb with high priority: 117BDB::poll_cb with high priority:
107 118
108 Event->io (fd => BDB::AIO::poll_fileno, 119 Event->io (fd => BDB::poll_fileno,
109 poll => 'r', async => 1, 120 poll => 'r', async => 1,
110 cb => \&BDB::AIO::poll_cb); 121 cb => \&BDB::poll_cb);
111 122
112=item BDB::AIO::max_poll_reqs $nreqs 123=item BDB::max_poll_reqs $nreqs
113 124
114=item BDB::AIO::max_poll_time $seconds 125=item BDB::max_poll_time $seconds
115 126
116These set the maximum number of requests (default C<0>, meaning infinity) 127These set the maximum number of requests (default C<0>, meaning infinity)
117that are being processed by C<BDB::AIO::poll_cb> in one call, respectively 128that are being processed by C<BDB::poll_cb> in one call, respectively
118the maximum amount of time (default C<0>, meaning infinity) spent in 129the maximum amount of time (default C<0>, meaning infinity) spent in
119C<BDB::AIO::poll_cb> to process requests (more correctly the mininum amount 130C<BDB::poll_cb> to process requests (more correctly the mininum amount
120of time C<poll_cb> is allowed to use). 131of time C<poll_cb> is allowed to use).
121 132
122Setting C<max_poll_time> to a non-zero value creates an overhead of one 133Setting C<max_poll_time> to a non-zero value creates an overhead of one
123syscall per request processed, which is not normally a problem unless your 134syscall per request processed, which is not normally a problem unless your
124callbacks are really really fast or your OS is really really slow (I am 135callbacks are really really fast or your OS is really really slow (I am
… …
129time. 140time.
130 141
131For interactive programs, values such as C<0.01> to C<0.1> should be fine. 142For interactive programs, values such as C<0.01> to C<0.1> should be fine.
132 143
133Example: Install an Event watcher that automatically calls 144Example: Install an Event watcher that automatically calls
134BDB::AIO::poll_cb with low priority, to ensure that other parts of the 145BDB::poll_cb with low priority, to ensure that other parts of the
135program get the CPU sometimes even under high AIO load. 146program get the CPU sometimes even under high AIO load.
136 147
137 # try not to spend much more than 0.1s in poll_cb 148 # try not to spend much more than 0.1s in poll_cb
138 BDB::AIO::max_poll_time 0.1; 149 BDB::max_poll_time 0.1;
139 150
140 # use a low priority so other tasks have priority 151 # use a low priority so other tasks have priority
141 Event->io (fd => BDB::AIO::poll_fileno, 152 Event->io (fd => BDB::poll_fileno,
142 poll => 'r', nice => 1, 153 poll => 'r', nice => 1,
143 cb => &BDB::AIO::poll_cb); 154 cb => &BDB::poll_cb);
144 155
145=item BDB::AIO::poll_wait 156=item BDB::poll_wait
146 157
147If there are any outstanding requests and none of them in the result 158If there are any outstanding requests and none of them in the result
148phase, wait till the result filehandle becomes ready for reading (simply 159phase, wait till the result filehandle becomes ready for reading (simply
149does a C<select> on the filehandle. This is useful if you want to 160does a C<select> on the filehandle. This is useful if you want to
150synchronously wait for some requests to finish). 161synchronously wait for some requests to finish).
151 162
152See C<nreqs> for an example. 163See C<nreqs> for an example.
153 164
154=item BDB::AIO::poll 165=item BDB::poll
155 166
156Waits until some requests have been handled. 167Waits until some requests have been handled.
157 168
158Returns the number of requests processed, but is otherwise strictly 169Returns the number of requests processed, but is otherwise strictly
159equivalent to: 170equivalent to:
160 171
161 BDB::AIO::poll_wait, BDB::AIO::poll_cb 172 BDB::poll_wait, BDB::poll_cb
162 173
163=item BDB::AIO::flush 174=item BDB::flush
164 175
165Wait till all outstanding AIO requests have been handled. 176Wait till all outstanding AIO requests have been handled.
166 177
167Strictly equivalent to: 178Strictly equivalent to:
168 179
169 BDB::AIO::poll_wait, BDB::AIO::poll_cb 180 BDB::poll_wait, BDB::poll_cb
170 while BDB::AIO::nreqs; 181 while BDB::nreqs;
171 182
172=head3 CONTROLLING THE NUMBER OF THREADS 183=head3 CONTROLLING THE NUMBER OF THREADS
173 184
174=item BDB::AIO::min_parallel $nthreads 185=item BDB::min_parallel $nthreads
175 186
176Set the minimum number of AIO threads to C<$nthreads>. The current 187Set the minimum number of AIO threads to C<$nthreads>. The current
177default is C<8>, which means eight asynchronous operations can execute 188default is C<8>, which means eight asynchronous operations can execute
178concurrently at any one time (the number of outstanding requests, 189concurrently at any one time (the number of outstanding requests,
179however, is unlimited). 190however, is unlimited).
180 191
181BDB::AIO starts threads only on demand, when an AIO request is queued and 192BDB starts threads only on demand, when an AIO request is queued and
182no free thread exists. Please note that queueing up a hundred requests can 193no free thread exists. Please note that queueing up a hundred requests can
183create demand for a hundred threads, even if it turns out that everything 194create demand for a hundred threads, even if it turns out that everything
184is in the cache and could have been processed faster by a single thread. 195is in the cache and could have been processed faster by a single thread.
185 196
186It is recommended to keep the number of threads relatively low, as some 197It is recommended to keep the number of threads relatively low, as some
… …
189versions, 4-32 threads should be fine. 200versions, 4-32 threads should be fine.
190 201
191Under most circumstances you don't need to call this function, as the 202Under most circumstances you don't need to call this function, as the
192module selects a default that is suitable for low to moderate load. 203module selects a default that is suitable for low to moderate load.
193 204
194=item BDB::AIO::max_parallel $nthreads 205=item BDB::max_parallel $nthreads
195 206
196Sets the maximum number of AIO threads to C<$nthreads>. If more than the 207Sets the maximum number of AIO threads to C<$nthreads>. If more than the
197specified number of threads are currently running, this function kills 208specified number of threads are currently running, this function kills
198them. This function blocks until the limit is reached. 209them. This function blocks until the limit is reached.
199 210
… …
203This module automatically runs C<max_parallel 0> at program end, to ensure 214This module automatically runs C<max_parallel 0> at program end, to ensure
204that all threads are killed and that there are no outstanding requests. 215that all threads are killed and that there are no outstanding requests.
205 216
206Under normal circumstances you don't need to call this function. 217Under normal circumstances you don't need to call this function.
207 218
208=item BDB::AIO::max_idle $nthreads 219=item BDB::max_idle $nthreads
209 220
210Limit the number of threads (default: 4) that are allowed to idle (i.e., 221Limit the number of threads (default: 4) that are allowed to idle (i.e.,
211threads that did not get a request to process within 10 seconds). That 222threads that did not get a request to process within 10 seconds). That
212means if a thread becomes idle while C<$nthreads> other threads are also 223means if a thread becomes idle while C<$nthreads> other threads are also
213idle, it will free its resources and exit. 224idle, it will free its resources and exit.
… …
218 229
219The default is probably ok in most situations, especially if thread 230The default is probably ok in most situations, especially if thread
220creation is fast. If thread creation is very slow on your system you might 231creation is fast. If thread creation is very slow on your system you might
221want to use larger values. 232want to use larger values.
222 233
223=item $oldmaxreqs = BDB::AIO::max_outstanding $maxreqs 234=item $oldmaxreqs = BDB::max_outstanding $maxreqs
224 235
225This is a very bad function to use in interactive programs because it 236This is a very bad function to use in interactive programs because it
226blocks, and a bad way to reduce concurrency because it is inexact: Better 237blocks, and a bad way to reduce concurrency because it is inexact: Better
227use an C<aio_group> together with a feed callback. 238use an C<aio_group> together with a feed callback.
228 239
… …
236 247
237You can still queue as many requests as you want. Therefore, 248You can still queue as many requests as you want. Therefore,
238C<max_oustsanding> is mainly useful in simple scripts (with low values) or 249C<max_oustsanding> is mainly useful in simple scripts (with low values) or
239as a stop gap to shield against fatal memory overflow (with large values). 250as a stop gap to shield against fatal memory overflow (with large values).
240 251
252=item BDB::set_sync_prepare $cb
253
254Sets a callback that is called whenever a request is created without an
255explicit callback. It has to return two code references. The first is used
256as the request callback, and the second is called to wait until the first
257callback has been called. The default implementation works like this:
258
259 sub {
260 my $status;
261 (
262 sub { $status = $! },
263 sub { BDB::poll while !defined $status; $! = $status },
264 )
265 }
266
267=back
268
241=head3 STATISTICAL INFORMATION 269=head3 STATISTICAL INFORMATION
242 270
271=over 4
272
243=item BDB::AIO::nreqs 273=item BDB::nreqs
244 274
245Returns the number of requests currently in the ready, execute or pending 275Returns the number of requests currently in the ready, execute or pending
246states (i.e. for which their callback has not been invoked yet). 276states (i.e. for which their callback has not been invoked yet).
247 277
248Example: wait till there are no outstanding requests anymore: 278Example: wait till there are no outstanding requests anymore:
249 279
250 BDB::AIO::poll_wait, BDB::AIO::poll_cb 280 BDB::poll_wait, BDB::poll_cb
251 while BDB::AIO::nreqs; 281 while BDB::nreqs;
252 282
253=item BDB::AIO::nready 283=item BDB::nready
254 284
255Returns the number of requests currently in the ready state (not yet 285Returns the number of requests currently in the ready state (not yet
256executed). 286executed).
257 287
258=item BDB::AIO::npending 288=item BDB::npending
259 289
260Returns the number of requests currently in the pending state (executed, 290Returns the number of requests currently in the pending state (executed,
261but not yet processed by poll_cb). 291but not yet processed by poll_cb).
262 292
263=back 293=back
264 294
265=cut 295=cut
266 296
267# support function to convert a fd into a perl filehandle 297set_sync_prepare {
268sub _fd2fh { 298 my $status;
269 return undef if $_[0] < 0; 299 (
270 300 sub {
271 # try to generate nice filehandles 301 $status = $!;
272 my $sym = "BDB::AIO::fd#$_[0]"; 302 },
273 local *$sym; 303 sub {
274 304 BDB::poll while !defined $status;
275 open *$sym, "+<&=$_[0]" # usually works under any unix 305 $! = $status;
276 or open *$sym, "<&=$_[0]" # cygwin needs this 306 },
277 or open *$sym, ">&=$_[0]" # or this 307 )
278 or return undef; 308};
279
280 *$sym
281}
282 309
283min_parallel 8; 310min_parallel 8;
284 311
285END { flush } 312END { flush }
286 313

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines