ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/BDB/BDB.pm
Revision: 1.18
Committed: Tue Dec 4 11:07:39 2007 UTC (16 years, 5 months ago) by root
Branch: MAIN
Changes since 1.17: +10 -13 lines
Log Message:
*** empty log message ***

File Contents

# User Rev Content
1 root 1.1 =head1 NAME
2    
3 root 1.2 BDB - Asynchronous Berkeley DB access
4 root 1.1
5     =head1 SYNOPSIS
6    
7 root 1.2 use BDB;
8 root 1.1
9     =head1 DESCRIPTION
10    
11 root 1.10 See the BerkeleyDB documentation (L<http://www.oracle.com/technology/documentation/berkeley-db/db/index.html>).
12 root 1.12 The BDB API is very similar to the C API (the translation has been very faithful).
13 root 1.10
14     See also the example sections in the document below and possibly the eg/
15     subdirectory of the BDB distribution. Last not least see the IO::AIO
16     documentation, as that module uses almost the same asynchronous request
17     model as this module.
18    
19     I know this is woefully inadequate documentation. Send a patch!
20 root 1.7
21 root 1.1
22     =head1 REQUEST ANATOMY AND LIFETIME
23    
24     Every request method creates a request. which is a C data structure not
25     directly visible to Perl.
26    
27     During their existance, bdb requests travel through the following states,
28     in order:
29    
30     =over 4
31    
32     =item ready
33    
34     Immediately after a request is created it is put into the ready state,
35     waiting for a thread to execute it.
36    
37     =item execute
38    
39     A thread has accepted the request for processing and is currently
40     executing it (e.g. blocking in read).
41    
42     =item pending
43    
44     The request has been executed and is waiting for result processing.
45    
46     While request submission and execution is fully asynchronous, result
47     processing is not and relies on the perl interpreter calling C<poll_cb>
48     (or another function with the same effect).
49    
50     =item result
51    
52     The request results are processed synchronously by C<poll_cb>.
53    
54     The C<poll_cb> function will process all outstanding aio requests by
55     calling their callbacks, freeing memory associated with them and managing
56     any groups they are contained in.
57    
58     =item done
59    
60     Request has reached the end of its lifetime and holds no resources anymore
61     (except possibly for the Perl object, but its connection to the actual
62     aio request is severed and calling its methods will either do nothing or
63     result in a runtime error).
64    
65     =back
66    
67     =cut
68    
69 root 1.2 package BDB;
70 root 1.1
71     no warnings;
72     use strict 'vars';
73    
74     use base 'Exporter';
75    
76     BEGIN {
77 root 1.17 our $VERSION = '1.2';
78 root 1.1
79 root 1.3 our @BDB_REQ = qw(
80 root 1.6 db_env_open db_env_close db_env_txn_checkpoint db_env_lock_detect
81     db_env_memp_sync db_env_memp_trickle
82     db_open db_close db_compact db_sync db_put db_get db_pget db_del db_key_range
83 root 1.15 db_txn_commit db_txn_abort db_txn_finish
84 root 1.5 db_c_close db_c_count db_c_put db_c_get db_c_pget db_c_del
85 root 1.6 db_sequence_open db_sequence_close
86     db_sequence_get db_sequence_remove
87     );
88     our @EXPORT = (@BDB_REQ, qw(dbreq_pri dbreq_nice db_env_create db_create));
89     our @EXPORT_OK = qw(
90     poll_fileno poll_cb poll_wait flush
91     min_parallel max_parallel max_idle
92     nreqs nready npending nthreads
93     max_poll_time max_poll_reqs
94 root 1.3 );
95 root 1.1
96     require XSLoader;
97 root 1.2 XSLoader::load ("BDB", $VERSION);
98 root 1.1 }
99    
100 root 1.10 =head2 BERKELEYDB FUNCTIONS
101    
102     All of these are functions. The create functions simply return a new
103     object and never block. All the remaining functions all take an optional
104     callback as last argument. If it is missing, then the fucntion will be
105     executed synchronously.
106    
107     BDB functions that cannot block (mostly functions that manipulate
108     settings) are method calls on the relevant objects, so the rule of thumb
109     is: if its a method, its not blocking, if its a function, it takes a
110     callback as last argument.
111    
112     In the following, C<$int> signifies an integer return value,
113     C<octetstring> is a "binary string" (i.e. a perl string with no character
114     indices >255), C<U32> is an unsigned 32 bit integer, C<int> is some
115     integer, C<NV> is a floating point value.
116    
117     The C<SV *> types are generic perl scalars (for input and output of data
118     values), and the C<SV *callback> is the optional callback function to call
119     when the request is completed.
120    
121 root 1.11 The various C<DB_ENV> etc. arguments are handles return by
122     C<db_env_create>, C<db_create>, C<txn_begin> and so on. If they have an
123     appended C<_ornull> this means they are optional and you can pass C<undef>
124     for them, resulting a NULL pointer on the C level.
125 root 1.10
126     =head3 BDB functions
127    
128     Functions in the BDB namespace, exported by default:
129    
130     $env = db_env_create (U32 env_flags = 0)
131 root 1.14 flags: RPCCLIENT
132 root 1.10
133     db_env_open (DB_ENV *env, octetstring db_home, U32 open_flags, int mode, SV *callback = &PL_sv_undef)
134 root 1.14 open_flags: INIT_CDB INIT_LOCK INIT_LOG INIT_MPOOL INIT_REP INIT_TXN RECOVER RECOVER_FATAL USE_ENVIRON USE_ENVIRON_ROOT CREATE LOCKDOWN PRIVATE REGISTER SYSTEM_MEM
135 root 1.10 db_env_close (DB_ENV *env, U32 flags = 0, SV *callback = &PL_sv_undef)
136     db_env_txn_checkpoint (DB_ENV *env, U32 kbyte = 0, U32 min = 0, U32 flags = 0, SV *callback = &PL_sv_undef)
137 root 1.14 flags: FORCE
138 root 1.10 db_env_lock_detect (DB_ENV *env, U32 flags = 0, U32 atype = DB_LOCK_DEFAULT, SV *dummy = 0, SV *callback = &PL_sv_undef)
139 root 1.14 atype: LOCK_DEFAULT LOCK_EXPIRE LOCK_MAXLOCKS LOCK_MAXWRITE LOCK_MINLOCKS LOCK_MINWRITE LOCK_OLDEST LOCK_RANDOM LOCK_YOUNGEST
140 root 1.10 db_env_memp_sync (DB_ENV *env, SV *dummy = 0, SV *callback = &PL_sv_undef)
141     db_env_memp_trickle (DB_ENV *env, int percent, SV *dummy = 0, SV *callback = &PL_sv_undef)
142    
143     $db = db_create (DB_ENV *env = 0, U32 flags = 0)
144 root 1.14 flags: XA_CREATE
145 root 1.10
146     db_open (DB *db, DB_TXN_ornull *txnid, octetstring file, octetstring database, int type, U32 flags, int mode, SV *callback = &PL_sv_undef)
147 root 1.14 flags: AUTO_COMMIT CREATE EXCL MULTIVERSION NOMMAP RDONLY READ_UNCOMMITTED THREAD TRUNCATE
148 root 1.10 db_close (DB *db, U32 flags = 0, SV *callback = &PL_sv_undef)
149 root 1.14 flags: DB_NOSYNC
150 root 1.12 db_compact (DB *db, DB_TXN_ornull *txn = 0, SV *start = 0, SV *stop = 0, SV *unused1 = 0, U32 flags = DB_FREE_SPACE, SV *unused2 = 0, SV *callback = &PL_sv_undef)
151 root 1.14 flags: FREELIST_ONLY FREE_SPACE
152 root 1.10 db_sync (DB *db, U32 flags = 0, SV *callback = &PL_sv_undef)
153     db_key_range (DB *db, DB_TXN_ornull *txn, SV *key, SV *key_range, U32 flags = 0, SV *callback = &PL_sv_undef)
154     db_put (DB *db, DB_TXN_ornull *txn, SV *key, SV *data, U32 flags = 0, SV *callback = &PL_sv_undef)
155 root 1.14 flags: APPEND NODUPDATA NOOVERWRITE
156 root 1.10 db_get (DB *db, DB_TXN_ornull *txn, SV *key, SV *data, U32 flags = 0, SV *callback = &PL_sv_undef)
157 root 1.14 flags: CONSUME CONSUME_WAIT GET_BOTH SET_RECNO MULTIPLE READ_COMMITTED READ_UNCOMMITTED RMW
158 root 1.10 db_pget (DB *db, DB_TXN_ornull *txn, SV *key, SV *pkey, SV *data, U32 flags = 0, SV *callback = &PL_sv_undef)
159 root 1.14 flags: CONSUME CONSUME_WAIT GET_BOTH SET_RECNO MULTIPLE READ_COMMITTED READ_UNCOMMITTED RMW
160 root 1.10 db_del (DB *db, DB_TXN_ornull *txn, SV *key, U32 flags = 0, SV *callback = &PL_sv_undef)
161     db_txn_commit (DB_TXN *txn, U32 flags = 0, SV *callback = &PL_sv_undef)
162 root 1.14 flags: TXN_NOSYNC TXN_SYNC
163 root 1.10 db_txn_abort (DB_TXN *txn, SV *callback = &PL_sv_undef)
164 root 1.14
165 root 1.10 db_c_close (DBC *dbc, SV *callback = &PL_sv_undef)
166     db_c_count (DBC *dbc, SV *count, U32 flags = 0, SV *callback = &PL_sv_undef)
167     db_c_put (DBC *dbc, SV *key, SV *data, U32 flags = 0, SV *callback = &PL_sv_undef)
168 root 1.14 flags: AFTER BEFORE CURRENT KEYFIRST KEYLAST NODUPDATA
169 root 1.10 db_c_get (DBC *dbc, SV *key, SV *data, U32 flags = 0, SV *callback = &PL_sv_undef)
170 root 1.14 flags: CURRENT FIRST GET_BOTH GET_BOTH_RANGE GET_RECNO JOIN_ITEM LAST NEXT NEXT_DUP NEXT_NODUP PREV PREV_DUP PREV_NODUP SET SET_RANGE SET_RECNO READ_UNCOMMITTED MULTIPLE MULTIPLE_KEY RMW
171 root 1.10 db_c_pget (DBC *dbc, SV *key, SV *pkey, SV *data, U32 flags = 0, SV *callback = &PL_sv_undef)
172     db_c_del (DBC *dbc, U32 flags = 0, SV *callback = &PL_sv_undef)
173    
174     db_sequence_open (DB_SEQUENCE *seq, DB_TXN_ornull *txnid, SV *key, U32 flags = 0, SV *callback = &PL_sv_undef)
175 root 1.14 flags: CREATE EXCL
176 root 1.10 db_sequence_close (DB_SEQUENCE *seq, U32 flags = 0, SV *callback = &PL_sv_undef)
177     db_sequence_get (DB_SEQUENCE *seq, DB_TXN_ornull *txnid, int delta, SV *seq_value, U32 flags = DB_TXN_NOSYNC, SV *callback = &PL_sv_undef)
178 root 1.14 flags: TXN_NOSYNC
179 root 1.10 db_sequence_remove (DB_SEQUENCE *seq, DB_TXN_ornull *txnid = 0, U32 flags = 0, SV *callback = &PL_sv_undef)
180 root 1.14 flags: TXN_NOSYNC
181 root 1.10
182 root 1.15 =head4 db_txn_finish (DB_TXN *txn, U32 flags = 0, SV *callback = &PL_sv_undef)
183    
184 root 1.16 This is not actually a Berkeley DB function but a BDB module
185     extension. The background for this exytension is: It is very annoying to
186     have to check every single BDB function for error returns and provide a
187     codepath out of your transaction. While the BDB module still makes this
188     possible, it contains the following extensions:
189 root 1.15
190     When a transaction-protected function returns any operating system
191     error (errno > 0), BDB will set the C<TXN_DEADLOCK> flag on the
192 root 1.16 transaction. This flag is also set by Berkeley DB functions themselves
193 root 1.15 when an operation fails with LOCK_DEADLOCK, and it causes all further
194     operations on that transaction (including C<db_txn_commit>) to fail.
195    
196     The C<db_txn_finish> request will look at this flag, and, if it is set,
197     will automatically call C<db_txn_abort> (setting errno to C<LOCK_DEADLOCK>
198 root 1.16 if it isn't set to something else yet). If it isn't set, it will call
199     C<db_txn_commit> and return the error normally.
200 root 1.15
201     How to use this? Easy: just write your transaction normally:
202    
203     my $txn = $db_env->txn_begin;
204     db_get $db, $txn, "key", my $data;
205     db_put $db, $txn, "key", $data + 1 unless $! == BDB::NOTFOUND;
206     db_txn_finish $txn;
207     die "transaction failed" if $!;
208    
209     That is, handle only the expected errors. If something unexpected happens
210     (EIO, LOCK_NOTGRANTED or a deadlock in either db_get or db_put), then the remaining
211     requests (db_put in this case) will simply be skipped (they will fail with
212     LOCK_DEADLOCK) and the transaction will be aborted.
213    
214 root 1.16 You can use the C<< $txn->failed >> method to check wether a transaction
215 root 1.15 has failed in this way and abort further processing (excluding
216     C<db_txn_finish>).
217    
218 root 1.10 =head3 DB_ENV/database environment methods
219    
220     Methods available on DB_ENV/$env handles:
221    
222     DESTROY (DB_ENV_ornull *env)
223     CODE:
224     if (env)
225     env->close (env, 0);
226    
227     $int = $env->set_data_dir (const char *dir)
228     $int = $env->set_tmp_dir (const char *dir)
229     $int = $env->set_lg_dir (const char *dir)
230     $int = $env->set_shm_key (long shm_key)
231     $int = $env->set_cachesize (U32 gbytes, U32 bytes, int ncache = 0)
232     $int = $env->set_flags (U32 flags, int onoff)
233     $env->set_errfile (FILE *errfile = 0)
234     $env->set_msgfile (FILE *msgfile = 0)
235     $int = $env->set_verbose (U32 which, int onoff = 1)
236     $int = $env->set_encrypt (const char *password, U32 flags = 0)
237 root 1.15 $int = $env->set_timeout (NV timeout_seconds, U32 flags = SET_TXN_TIMEOUT)
238 root 1.10 $int = $env->set_mp_max_openfd (int maxopenfd);
239     $int = $env->set_mp_max_write (int maxwrite, int maxwrite_sleep);
240     $int = $env->set_mp_mmapsize (int mmapsize_mb)
241     $int = $env->set_lk_detect (U32 detect = DB_LOCK_DEFAULT)
242     $int = $env->set_lk_max_lockers (U32 max)
243     $int = $env->set_lk_max_locks (U32 max)
244     $int = $env->set_lk_max_objects (U32 max)
245     $int = $env->set_lg_bsize (U32 max)
246     $int = $env->set_lg_max (U32 max)
247    
248     $txn = $env->txn_begin (DB_TXN_ornull *parent = 0, U32 flags = 0)
249 root 1.14 flags: READ_COMMITTED READ_UNCOMMITTED TXN_NOSYNC TXN_NOWAIT TXN_SNAPSHOT TXN_SYNC TXN_WAIT TXN_WRITE_NOSYNC
250 root 1.10
251 root 1.12 =head4 Example:
252 root 1.10
253     use AnyEvent;
254     use BDB;
255    
256     our $FH; open $FH, "<&=" . BDB::poll_fileno;
257     our $WATCHER = AnyEvent->io (fh => $FH, poll => 'r', cb => \&BDB::poll_cb);
258    
259     BDB::min_parallel 8;
260    
261     my $env = db_env_create;
262    
263     mkdir "bdtest", 0700;
264     db_env_open
265     $env,
266     "bdtest",
267     BDB::INIT_LOCK | BDB::INIT_LOG | BDB::INIT_MPOOL | BDB::INIT_TXN | BDB::RECOVER | BDB::USE_ENVIRON | BDB::CREATE,
268     0600;
269    
270     $env->set_flags (BDB::AUTO_COMMIT | BDB::TXN_NOSYNC, 1);
271    
272    
273     =head3 DB/database methods
274    
275     Methods available on DB/$db handles:
276    
277     DESTROY (DB_ornull *db)
278     CODE:
279     if (db)
280     {
281     SV *env = (SV *)db->app_private;
282     db->close (db, 0);
283     SvREFCNT_dec (env);
284     }
285    
286     $int = $db->set_cachesize (U32 gbytes, U32 bytes, int ncache = 0)
287     $int = $db->set_flags (U32 flags)
288 root 1.14 flags: CHKSUM ENCRYPT TXN_NOT_DURABLE
289     Btree: DUP DUPSORT RECNUM REVSPLITOFF
290     Hash: DUP DUPSORT
291     Queue: INORDER
292     Recno: RENUMBER SNAPSHOT
293    
294 root 1.10 $int = $db->set_encrypt (const char *password, U32 flags)
295     $int = $db->set_lorder (int lorder)
296     $int = $db->set_bt_minkey (U32 minkey)
297     $int = $db->set_re_delim (int delim)
298     $int = $db->set_re_pad (int re_pad)
299     $int = $db->set_re_source (char *source)
300     $int = $db->set_re_len (U32 re_len)
301     $int = $db->set_h_ffactor (U32 h_ffactor)
302     $int = $db->set_h_nelem (U32 h_nelem)
303     $int = $db->set_q_extentsize (U32 extentsize)
304    
305     $dbc = $db->cursor (DB_TXN_ornull *txn = 0, U32 flags = 0)
306 root 1.14 flags: READ_COMMITTED READ_UNCOMMITTED WRITECURSOR TXN_SNAPSHOT
307 root 1.10 $seq = $db->sequence (U32 flags = 0)
308    
309 root 1.12 =head4 Example:
310 root 1.10
311     my $db = db_create $env;
312     db_open $db, undef, "table", undef, BDB::BTREE, BDB::AUTO_COMMIT | BDB::CREATE | BDB::READ_UNCOMMITTED, 0600;
313    
314     for (1..1000) {
315     db_put $db, undef, "key $_", "data $_";
316    
317     db_key_range $db, undef, "key $_", my $keyrange;
318     my ($lt, $eq, $gt) = @$keyrange;
319     }
320    
321     db_del $db, undef, "key $_" for 1..1000;
322    
323     db_sync $db;
324    
325    
326     =head3 DB_TXN/transaction methods
327    
328     Methods available on DB_TXN/$txn handles:
329    
330     DESTROY (DB_TXN_ornull *txn)
331     CODE:
332     if (txn)
333     txn->abort (txn);
334    
335 root 1.15 $int = $txn->set_timeout (NV timeout_seconds, U32 flags = SET_TXN_TIMEOUT)
336 root 1.14 flags: SET_LOCK_TIMEOUT SET_TXN_TIMEOUT
337 root 1.10
338 root 1.15 $bool = $txn->failed
339     # see db_txn_finish documentation, above
340    
341 root 1.10
342     =head3 DBC/cursor methods
343    
344     Methods available on DBC/$dbc handles:
345    
346     DESTROY (DBC_ornull *dbc)
347     CODE:
348     if (dbc)
349     dbc->c_close (dbc);
350    
351 root 1.12 =head4 Example:
352 root 1.10
353     my $c = $db->cursor;
354    
355     for (;;) {
356     db_c_get $c, my $key, my $data, BDB::NEXT;
357     warn "<$!,$key,$data>";
358     last if $!;
359     }
360    
361     db_c_close $c;
362    
363 root 1.12
364 root 1.10 =head3 DB_SEQUENCE/sequence methods
365    
366     Methods available on DB_SEQUENCE/$seq handles:
367    
368     DESTROY (DB_SEQUENCE_ornull *seq)
369     CODE:
370     if (seq)
371     seq->close (seq, 0);
372    
373     $int = $seq->initial_value (db_seq_t value)
374     $int = $seq->set_cachesize (U32 size)
375     $int = $seq->set_flags (U32 flags)
376 root 1.14 flags: SEQ_DEC SEQ_INC SEQ_WRAP
377 root 1.10 $int = $seq->set_range (db_seq_t min, db_seq_t max)
378    
379 root 1.12 =head4 Example:
380 root 1.10
381     my $seq = $db->sequence;
382    
383     db_sequence_open $seq, undef, "seq", BDB::CREATE;
384     db_sequence_get $seq, undef, 1, my $value;
385    
386    
387 root 1.1 =head2 SUPPORT FUNCTIONS
388    
389     =head3 EVENT PROCESSING AND EVENT LOOP INTEGRATION
390    
391     =over 4
392    
393 root 1.2 =item $fileno = BDB::poll_fileno
394 root 1.1
395     Return the I<request result pipe file descriptor>. This filehandle must be
396     polled for reading by some mechanism outside this module (e.g. Event or
397     select, see below or the SYNOPSIS). If the pipe becomes readable you have
398     to call C<poll_cb> to check the results.
399    
400     See C<poll_cb> for an example.
401    
402 root 1.2 =item BDB::poll_cb
403 root 1.1
404     Process some outstanding events on the result pipe. You have to call this
405     regularly. Returns the number of events processed. Returns immediately
406     when no events are outstanding. The amount of events processed depends on
407 root 1.2 the settings of C<BDB::max_poll_req> and C<BDB::max_poll_time>.
408 root 1.1
409     If not all requests were processed for whatever reason, the filehandle
410     will still be ready when C<poll_cb> returns.
411    
412     Example: Install an Event watcher that automatically calls
413 root 1.2 BDB::poll_cb with high priority:
414 root 1.1
415 root 1.2 Event->io (fd => BDB::poll_fileno,
416 root 1.1 poll => 'r', async => 1,
417 root 1.2 cb => \&BDB::poll_cb);
418 root 1.1
419 root 1.2 =item BDB::max_poll_reqs $nreqs
420 root 1.1
421 root 1.2 =item BDB::max_poll_time $seconds
422 root 1.1
423     These set the maximum number of requests (default C<0>, meaning infinity)
424 root 1.2 that are being processed by C<BDB::poll_cb> in one call, respectively
425 root 1.1 the maximum amount of time (default C<0>, meaning infinity) spent in
426 root 1.2 C<BDB::poll_cb> to process requests (more correctly the mininum amount
427 root 1.1 of time C<poll_cb> is allowed to use).
428    
429     Setting C<max_poll_time> to a non-zero value creates an overhead of one
430     syscall per request processed, which is not normally a problem unless your
431     callbacks are really really fast or your OS is really really slow (I am
432     not mentioning Solaris here). Using C<max_poll_reqs> incurs no overhead.
433    
434     Setting these is useful if you want to ensure some level of
435     interactiveness when perl is not fast enough to process all requests in
436     time.
437    
438     For interactive programs, values such as C<0.01> to C<0.1> should be fine.
439    
440 root 1.18 Example: Install an EV watcher that automatically calls
441 root 1.2 BDB::poll_cb with low priority, to ensure that other parts of the
442 root 1.18 program get the CPU sometimes even under high load.
443 root 1.1
444     # try not to spend much more than 0.1s in poll_cb
445 root 1.2 BDB::max_poll_time 0.1;
446 root 1.1
447 root 1.18 my $bdb_poll = EV::io BDB::poll_fileno, EV::READ, \&BDB::poll_cb);
448 root 1.1
449 root 1.2 =item BDB::poll_wait
450 root 1.1
451     If there are any outstanding requests and none of them in the result
452     phase, wait till the result filehandle becomes ready for reading (simply
453     does a C<select> on the filehandle. This is useful if you want to
454     synchronously wait for some requests to finish).
455    
456     See C<nreqs> for an example.
457    
458 root 1.2 =item BDB::poll
459 root 1.1
460     Waits until some requests have been handled.
461    
462     Returns the number of requests processed, but is otherwise strictly
463     equivalent to:
464    
465 root 1.2 BDB::poll_wait, BDB::poll_cb
466 root 1.1
467 root 1.2 =item BDB::flush
468 root 1.1
469 root 1.18 Wait till all outstanding BDB requests have been handled.
470 root 1.1
471     Strictly equivalent to:
472    
473 root 1.2 BDB::poll_wait, BDB::poll_cb
474     while BDB::nreqs;
475 root 1.1
476 root 1.8 =back
477    
478 root 1.1 =head3 CONTROLLING THE NUMBER OF THREADS
479    
480 root 1.8 =over 4
481    
482 root 1.2 =item BDB::min_parallel $nthreads
483 root 1.1
484 root 1.18 Set the minimum number of BDB threads to C<$nthreads>. The current
485 root 1.1 default is C<8>, which means eight asynchronous operations can execute
486     concurrently at any one time (the number of outstanding requests,
487     however, is unlimited).
488    
489 root 1.18 BDB starts threads only on demand, when an BDB request is queued and
490 root 1.1 no free thread exists. Please note that queueing up a hundred requests can
491     create demand for a hundred threads, even if it turns out that everything
492     is in the cache and could have been processed faster by a single thread.
493    
494     It is recommended to keep the number of threads relatively low, as some
495     Linux kernel versions will scale negatively with the number of threads
496     (higher parallelity => MUCH higher latency). With current Linux 2.6
497     versions, 4-32 threads should be fine.
498    
499     Under most circumstances you don't need to call this function, as the
500     module selects a default that is suitable for low to moderate load.
501    
502 root 1.2 =item BDB::max_parallel $nthreads
503 root 1.1
504 root 1.18 Sets the maximum number of BDB threads to C<$nthreads>. If more than the
505 root 1.1 specified number of threads are currently running, this function kills
506     them. This function blocks until the limit is reached.
507    
508     While C<$nthreads> are zero, aio requests get queued but not executed
509     until the number of threads has been increased again.
510    
511     This module automatically runs C<max_parallel 0> at program end, to ensure
512     that all threads are killed and that there are no outstanding requests.
513    
514     Under normal circumstances you don't need to call this function.
515    
516 root 1.2 =item BDB::max_idle $nthreads
517 root 1.1
518     Limit the number of threads (default: 4) that are allowed to idle (i.e.,
519     threads that did not get a request to process within 10 seconds). That
520     means if a thread becomes idle while C<$nthreads> other threads are also
521     idle, it will free its resources and exit.
522    
523     This is useful when you allow a large number of threads (e.g. 100 or 1000)
524     to allow for extremely high load situations, but want to free resources
525     under normal circumstances (1000 threads can easily consume 30MB of RAM).
526    
527     The default is probably ok in most situations, especially if thread
528     creation is fast. If thread creation is very slow on your system you might
529     want to use larger values.
530    
531 root 1.2 =item $oldmaxreqs = BDB::max_outstanding $maxreqs
532 root 1.1
533     This is a very bad function to use in interactive programs because it
534     blocks, and a bad way to reduce concurrency because it is inexact: Better
535     use an C<aio_group> together with a feed callback.
536    
537     Sets the maximum number of outstanding requests to C<$nreqs>. If you
538     to queue up more than this number of requests, the next call to the
539     C<poll_cb> (and C<poll_some> and other functions calling C<poll_cb>)
540     function will block until the limit is no longer exceeded.
541    
542     The default value is very large, so there is no practical limit on the
543     number of outstanding requests.
544    
545     You can still queue as many requests as you want. Therefore,
546     C<max_oustsanding> is mainly useful in simple scripts (with low values) or
547     as a stop gap to shield against fatal memory overflow (with large values).
548    
549 root 1.3 =item BDB::set_sync_prepare $cb
550    
551     Sets a callback that is called whenever a request is created without an
552     explicit callback. It has to return two code references. The first is used
553     as the request callback, and the second is called to wait until the first
554     callback has been called. The default implementation works like this:
555    
556     sub {
557     my $status;
558     (
559     sub { $status = $! },
560     sub { BDB::poll while !defined $status; $! = $status },
561     )
562     }
563    
564     =back
565    
566 root 1.1 =head3 STATISTICAL INFORMATION
567    
568 root 1.3 =over 4
569    
570 root 1.2 =item BDB::nreqs
571 root 1.1
572     Returns the number of requests currently in the ready, execute or pending
573     states (i.e. for which their callback has not been invoked yet).
574    
575     Example: wait till there are no outstanding requests anymore:
576    
577 root 1.2 BDB::poll_wait, BDB::poll_cb
578     while BDB::nreqs;
579 root 1.1
580 root 1.2 =item BDB::nready
581 root 1.1
582     Returns the number of requests currently in the ready state (not yet
583     executed).
584    
585 root 1.2 =item BDB::npending
586 root 1.1
587     Returns the number of requests currently in the pending state (executed,
588     but not yet processed by poll_cb).
589    
590     =back
591    
592     =cut
593    
594 root 1.3 set_sync_prepare {
595     my $status;
596     (
597     sub {
598     $status = $!;
599     },
600     sub {
601     BDB::poll while !defined $status;
602     $! = $status;
603     },
604     )
605     };
606    
607 root 1.1 min_parallel 8;
608    
609     END { flush }
610    
611     1;
612    
613     =head2 FORK BEHAVIOUR
614    
615     This module should do "the right thing" when the process using it forks:
616    
617 root 1.18 Before the fork, BDB enters a quiescent state where no requests
618 root 1.1 can be added in other threads and no results will be processed. After
619     the fork the parent simply leaves the quiescent state and continues
620     request/result processing, while the child frees the request/result queue
621     (so that the requests started before the fork will only be handled in the
622     parent). Threads will be started on demand until the limit set in the
623     parent process has been reached again.
624    
625     In short: the parent will, after a short pause, continue as if fork had
626 root 1.18 not been called, while the child will act as if BDB has not been used
627 root 1.1 yet.
628    
629     =head2 MEMORY USAGE
630    
631     Per-request usage:
632    
633     Each aio request uses - depending on your architecture - around 100-200
634     bytes of memory. In addition, stat requests need a stat buffer (possibly
635     a few hundred bytes), readdir requires a result buffer and so on. Perl
636     scalars and other data passed into aio requests will also be locked and
637     will consume memory till the request has entered the done state.
638    
639 root 1.13 This is not awfully much, so queuing lots of requests is not usually a
640 root 1.1 problem.
641    
642     Per-thread usage:
643    
644     In the execution phase, some aio requests require more memory for
645     temporary buffers, and each thread requires a stack and other data
646     structures (usually around 16k-128k, depending on the OS).
647    
648     =head1 KNOWN BUGS
649    
650 root 1.15 Known bugs will be fixed in the next release, except:
651    
652     If you use a transaction in any request, and the request returns
653     with an operating system error or DB_LOCK_NOTGRANTED, the internal
654     TXN_DEADLOCK flag will be set on the transaction. See C<db_txn_finish>,
655     above.
656 root 1.1
657     =head1 SEE ALSO
658    
659 root 1.18 L<Coro::BDB>, L<IO::AIO>.
660 root 1.1
661     =head1 AUTHOR
662    
663     Marc Lehmann <schmorp@schmorp.de>
664     http://home.schmorp.de/
665    
666     =cut
667