ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/BDB/README
(Generate patch)

Comparing BDB/README (file contents):
Revision 1.1 by root, Mon Feb 5 18:40:55 2007 UTC vs.
Revision 1.9 by root, Sat May 10 20:23:19 2008 UTC

1NAME 1NAME
2 BDB::AIO - Asynchronous Berkeley DB access 2 BDB - Asynchronous Berkeley DB access
3 3
4SYNOPSIS 4SYNOPSIS
5 use BDB::AIO; 5 use BDB;
6
7 my $env = db_env_create;
8
9 mkdir "bdtest", 0700;
10 db_env_open
11 $env,
12 "bdtest",
13 BDB::INIT_LOCK | BDB::INIT_LOG | BDB::INIT_MPOOL
14 | BDB::INIT_TXN | BDB::RECOVER | BDB::USE_ENVIRON | BDB::CREATE,
15 0600;
16
17 $env->set_flags (BDB::AUTO_COMMIT | BDB::TXN_NOSYNC, 1);
18
19 my $db = db_create $env;
20 db_open $db, undef, "table", undef, BDB::BTREE, BDB::AUTO_COMMIT | BDB::CREATE
21 | BDB::READ_UNCOMMITTED, 0600;
22 db_put $db, undef, "key", "data", 0, sub {
23 db_del $db, undef, "key";
24 };
25 db_sync $db;
26
27 # when you also use Coro, management is easy:
28 use Coro::BDB;
29
30 # automatic event loop intergration with AnyEvent:
31 use AnyEvent::BDB;
32
33 # automatic result processing with EV:
34 my $WATCHER = EV::io BDB::poll_fileno, EV::READ, \&BDB::poll_cb;
35
36 # with Glib:
37 add_watch Glib::IO BDB::poll_fileno,
38 in => sub { BDB::poll_cb; 1 };
39
40 # or simply flush manually
41 BDB::flush;
6 42
7DESCRIPTION 43DESCRIPTION
8 EXAMPLE 44 See the BerkeleyDB documentation
45 (<http://www.oracle.com/technology/documentation/berkeley-db/db/index.ht
46 ml>). The BDB API is very similar to the C API (the translation has been
47 very faithful).
48
49 See also the example sections in the document below and possibly the eg/
50 subdirectory of the BDB distribution. Last not least see the IO::AIO
51 documentation, as that module uses almost the same asynchronous request
52 model as this module.
53
54 I know this is woefully inadequate documentation. Send a patch!
55
9REQUEST ANATOMY AND LIFETIME 56REQUEST ANATOMY AND LIFETIME
10 Every request method creates a request. which is a C data structure not 57 Every request method creates a request. which is a C data structure not
11 directly visible to Perl. 58 directly visible to Perl.
12 59
13 During their existance, bdb requests travel through the following 60 During their existance, bdb requests travel through the following
… …
39 Request has reached the end of its lifetime and holds no resources 86 Request has reached the end of its lifetime and holds no resources
40 anymore (except possibly for the Perl object, but its connection to 87 anymore (except possibly for the Perl object, but its connection to
41 the actual aio request is severed and calling its methods will 88 the actual aio request is severed and calling its methods will
42 either do nothing or result in a runtime error). 89 either do nothing or result in a runtime error).
43 90
91 WIN32 FILENAMES/DATABASE NAME MESS
92 Perl on Win32 supports only ASCII filenames (the reason is that it
93 abuses an internal flag to store wether a filename is Unicode or ANSI,
94 but that flag is used for somethign else in the perl core, so there is
95 no way to detect wether a filename is ANSI or Unicode-encoded). The BDB
96 module tries to work around this issue by assuming that the filename is
97 an ANSI filename and BDB was built for unicode support.
98
99 BERKELEYDB FUNCTIONS
100 All of these are functions. The create functions simply return a new
101 object and never block. All the remaining functions take an optional
102 callback as last argument. If it is missing, then the function will be
103 executed synchronously. In both cases, $! will reflect the return value
104 of the function.
105
106 BDB functions that cannot block (mostly functions that manipulate
107 settings) are method calls on the relevant objects, so the rule of thumb
108 is: if it's a method, it's not blocking, if it's a function, it takes a
109 callback as last argument.
110
111 In the following, $int signifies an integer return value, "octetstring"
112 is a "binary string" (i.e. a perl string with no character indices
113 >255), "U32" is an unsigned 32 bit integer, "int" is some integer, "NV"
114 is a floating point value.
115
116 The "SV *" types are generic perl scalars (for input and output of data
117 values), and the "SV *callback" is the optional callback function to
118 call when the request is completed.
119
120 The various "DB_ENV" etc. arguments are handles return by
121 "db_env_create", "db_create", "txn_begin" and so on. If they have an
122 appended "_ornull" this means they are optional and you can pass "undef"
123 for them, resulting a NULL pointer on the C level.
124
125 BDB functions
126 Functions in the BDB namespace, exported by default:
127
128 $env = db_env_create (U32 env_flags = 0)
129 flags: RPCCLIENT
130
131 db_env_open (DB_ENV *env, octetstring db_home, U32 open_flags, int mode, SV *callback = &PL_sv_undef)
132 open_flags: INIT_CDB INIT_LOCK INIT_LOG INIT_MPOOL INIT_REP INIT_TXN RECOVER RECOVER_FATAL USE_ENVIRON USE_ENVIRON_ROOT CREATE LOCKDOWN PRIVATE REGISTER SYSTEM_MEM
133 db_env_close (DB_ENV *env, U32 flags = 0, SV *callback = &PL_sv_undef)
134 db_env_txn_checkpoint (DB_ENV *env, U32 kbyte = 0, U32 min = 0, U32 flags = 0, SV *callback = &PL_sv_undef)
135 flags: FORCE
136 db_env_lock_detect (DB_ENV *env, U32 flags = 0, U32 atype = DB_LOCK_DEFAULT, SV *dummy = 0, SV *callback = &PL_sv_undef)
137 atype: LOCK_DEFAULT LOCK_EXPIRE LOCK_MAXLOCKS LOCK_MAXWRITE LOCK_MINLOCKS LOCK_MINWRITE LOCK_OLDEST LOCK_RANDOM LOCK_YOUNGEST
138 db_env_memp_sync (DB_ENV *env, SV *dummy = 0, SV *callback = &PL_sv_undef)
139 db_env_memp_trickle (DB_ENV *env, int percent, SV *dummy = 0, SV *callback = &PL_sv_undef)
140
141 $db = db_create (DB_ENV *env = 0, U32 flags = 0)
142 flags: XA_CREATE
143
144 db_open (DB *db, DB_TXN_ornull *txnid, octetstring file, octetstring database, int type, U32 flags, int mode, SV *callback = &PL_sv_undef)
145 flags: AUTO_COMMIT CREATE EXCL MULTIVERSION NOMMAP RDONLY READ_UNCOMMITTED THREAD TRUNCATE
146 db_close (DB *db, U32 flags = 0, SV *callback = &PL_sv_undef)
147 flags: DB_NOSYNC
148 db_upgrade (DB *db, octetstring file, U32 flags = 0, SV *callback = &PL_sv_undef)
149 db_compact (DB *db, DB_TXN_ornull *txn = 0, SV *start = 0, SV *stop = 0, SV *unused1 = 0, U32 flags = DB_FREE_SPACE, SV *unused2 = 0, SV *callback = &PL_sv_undef)
150 flags: FREELIST_ONLY FREE_SPACE
151 db_sync (DB *db, U32 flags = 0, SV *callback = &PL_sv_undef)
152 db_key_range (DB *db, DB_TXN_ornull *txn, SV *key, SV *key_range, U32 flags = 0, SV *callback = &PL_sv_undef)
153 db_put (DB *db, DB_TXN_ornull *txn, SV *key, SV *data, U32 flags = 0, SV *callback = &PL_sv_undef)
154 flags: APPEND NODUPDATA NOOVERWRITE
155 db_get (DB *db, DB_TXN_ornull *txn, SV *key, SV *data, U32 flags = 0, SV *callback = &PL_sv_undef)
156 flags: CONSUME CONSUME_WAIT GET_BOTH SET_RECNO MULTIPLE READ_COMMITTED READ_UNCOMMITTED RMW
157 db_pget (DB *db, DB_TXN_ornull *txn, SV *key, SV *pkey, SV *data, U32 flags = 0, SV *callback = &PL_sv_undef)
158 flags: CONSUME CONSUME_WAIT GET_BOTH SET_RECNO MULTIPLE READ_COMMITTED READ_UNCOMMITTED RMW
159 db_del (DB *db, DB_TXN_ornull *txn, SV *key, U32 flags = 0, SV *callback = &PL_sv_undef)
160 db_txn_commit (DB_TXN *txn, U32 flags = 0, SV *callback = &PL_sv_undef)
161 flags: TXN_NOSYNC TXN_SYNC
162 db_txn_abort (DB_TXN *txn, SV *callback = &PL_sv_undef)
163
164 db_c_close (DBC *dbc, SV *callback = &PL_sv_undef)
165 db_c_count (DBC *dbc, SV *count, U32 flags = 0, SV *callback = &PL_sv_undef)
166 db_c_put (DBC *dbc, SV *key, SV *data, U32 flags = 0, SV *callback = &PL_sv_undef)
167 flags: AFTER BEFORE CURRENT KEYFIRST KEYLAST NODUPDATA
168 db_c_get (DBC *dbc, SV *key, SV *data, U32 flags = 0, SV *callback = &PL_sv_undef)
169 flags: CURRENT FIRST GET_BOTH GET_BOTH_RANGE GET_RECNO JOIN_ITEM LAST NEXT NEXT_DUP NEXT_NODUP PREV PREV_DUP PREV_NODUP SET SET_RANGE SET_RECNO READ_UNCOMMITTED MULTIPLE MULTIPLE_KEY RMW
170 db_c_pget (DBC *dbc, SV *key, SV *pkey, SV *data, U32 flags = 0, SV *callback = &PL_sv_undef)
171 db_c_del (DBC *dbc, U32 flags = 0, SV *callback = &PL_sv_undef)
172
173 db_sequence_open (DB_SEQUENCE *seq, DB_TXN_ornull *txnid, SV *key, U32 flags = 0, SV *callback = &PL_sv_undef)
174 flags: CREATE EXCL
175 db_sequence_close (DB_SEQUENCE *seq, U32 flags = 0, SV *callback = &PL_sv_undef)
176 db_sequence_get (DB_SEQUENCE *seq, DB_TXN_ornull *txnid, int delta, SV *seq_value, U32 flags = DB_TXN_NOSYNC, SV *callback = &PL_sv_undef)
177 flags: TXN_NOSYNC
178 db_sequence_remove (DB_SEQUENCE *seq, DB_TXN_ornull *txnid = 0, U32 flags = 0, SV *callback = &PL_sv_undef)
179 flags: TXN_NOSYNC
180
181 db_txn_finish (DB_TXN *txn, U32 flags = 0, SV *callback = &PL_sv_undef)
182 This is not actually a Berkeley DB function but a BDB module extension.
183 The background for this exytension is: It is very annoying to have to
184 check every single BDB function for error returns and provide a codepath
185 out of your transaction. While the BDB module still makes this possible,
186 it contains the following extensions:
187
188 When a transaction-protected function returns any operating system error
189 (errno > 0), BDB will set the "TXN_DEADLOCK" flag on the transaction.
190 This flag is also set by Berkeley DB functions themselves when an
191 operation fails with LOCK_DEADLOCK, and it causes all further operations
192 on that transaction (including "db_txn_commit") to fail.
193
194 The "db_txn_finish" request will look at this flag, and, if it is set,
195 will automatically call "db_txn_abort" (setting errno to "LOCK_DEADLOCK"
196 if it isn't set to something else yet). If it isn't set, it will call
197 "db_txn_commit" and return the error normally.
198
199 How to use this? Easy: just write your transaction normally:
200
201 my $txn = $db_env->txn_begin;
202 db_get $db, $txn, "key", my $data;
203 db_put $db, $txn, "key", $data + 1 unless $! == BDB::NOTFOUND;
204 db_txn_finish $txn;
205 die "transaction failed" if $!;
206
207 That is, handle only the expected errors. If something unexpected
208 happens (EIO, LOCK_NOTGRANTED or a deadlock in either db_get or db_put),
209 then the remaining requests (db_put in this case) will simply be skipped
210 (they will fail with LOCK_DEADLOCK) and the transaction will be aborted.
211
212 You can use the "$txn->failed" method to check wether a transaction has
213 failed in this way and abort further processing (excluding
214 "db_txn_finish").
215
216 DB_ENV/database environment methods
217 Methods available on DB_ENV/$env handles:
218
219 DESTROY (DB_ENV_ornull *env)
220 CODE:
221 if (env)
222 env->close (env, 0);
223
224 $int = $env->set_data_dir (const char *dir)
225 $int = $env->set_tmp_dir (const char *dir)
226 $int = $env->set_lg_dir (const char *dir)
227 $int = $env->set_shm_key (long shm_key)
228 $int = $env->set_cachesize (U32 gbytes, U32 bytes, int ncache = 0)
229 $int = $env->set_flags (U32 flags, int onoff)
230 $env->set_errfile (FILE *errfile = 0)
231 $env->set_msgfile (FILE *msgfile = 0)
232 $int = $env->set_verbose (U32 which, int onoff = 1)
233 $int = $env->set_encrypt (const char *password, U32 flags = 0)
234 $int = $env->set_timeout (NV timeout_seconds, U32 flags = SET_TXN_TIMEOUT)
235 $int = $env->set_mp_max_openfd (int maxopenfd);
236 $int = $env->set_mp_max_write (int maxwrite, int maxwrite_sleep);
237 $int = $env->set_mp_mmapsize (int mmapsize_mb)
238 $int = $env->set_lk_detect (U32 detect = DB_LOCK_DEFAULT)
239 $int = $env->set_lk_max_lockers (U32 max)
240 $int = $env->set_lk_max_locks (U32 max)
241 $int = $env->set_lk_max_objects (U32 max)
242 $int = $env->set_lg_bsize (U32 max)
243 $int = $env->set_lg_max (U32 max)
244 $int = $env->mutex_set_increment (U32 increment)
245 $int = $env->mutex_set_tas_spins (U32 tas_spins)
246 $int = $env->mutex_set_max (U32 max)
247 $int = $env->mutex_set_align (U32 align)
248
249 $txn = $env->txn_begin (DB_TXN_ornull *parent = 0, U32 flags = 0)
250 flags: READ_COMMITTED READ_UNCOMMITTED TXN_NOSYNC TXN_NOWAIT TXN_SNAPSHOT TXN_SYNC TXN_WAIT TXN_WRITE_NOSYNC
251
252 Example:
253 use AnyEvent;
254 use BDB;
255
256 our $FH; open $FH, "<&=" . BDB::poll_fileno;
257 our $WATCHER = AnyEvent->io (fh => $FH, poll => 'r', cb => \&BDB::poll_cb);
258
259 BDB::min_parallel 8;
260
261 my $env = db_env_create;
262
263 mkdir "bdtest", 0700;
264 db_env_open
265 $env,
266 "bdtest",
267 BDB::INIT_LOCK | BDB::INIT_LOG | BDB::INIT_MPOOL | BDB::INIT_TXN | BDB::RECOVER | BDB::USE_ENVIRON | BDB::CREATE,
268 0600;
269
270 $env->set_flags (BDB::AUTO_COMMIT | BDB::TXN_NOSYNC, 1);
271
272 DB/database methods
273 Methods available on DB/$db handles:
274
275 DESTROY (DB_ornull *db)
276 CODE:
277 if (db)
278 {
279 SV *env = (SV *)db->app_private;
280 db->close (db, 0);
281 SvREFCNT_dec (env);
282 }
283
284 $int = $db->set_cachesize (U32 gbytes, U32 bytes, int ncache = 0)
285 $int = $db->set_flags (U32 flags)
286 flags: CHKSUM ENCRYPT TXN_NOT_DURABLE
287 Btree: DUP DUPSORT RECNUM REVSPLITOFF
288 Hash: DUP DUPSORT
289 Queue: INORDER
290 Recno: RENUMBER SNAPSHOT
291
292 $int = $db->set_encrypt (const char *password, U32 flags)
293 $int = $db->set_lorder (int lorder)
294 $int = $db->set_bt_minkey (U32 minkey)
295 $int = $db->set_re_delim (int delim)
296 $int = $db->set_re_pad (int re_pad)
297 $int = $db->set_re_source (char *source)
298 $int = $db->set_re_len (U32 re_len)
299 $int = $db->set_h_ffactor (U32 h_ffactor)
300 $int = $db->set_h_nelem (U32 h_nelem)
301 $int = $db->set_q_extentsize (U32 extentsize)
302
303 $dbc = $db->cursor (DB_TXN_ornull *txn = 0, U32 flags = 0)
304 flags: READ_COMMITTED READ_UNCOMMITTED WRITECURSOR TXN_SNAPSHOT
305 $seq = $db->sequence (U32 flags = 0)
306
307 Example:
308 my $db = db_create $env;
309 db_open $db, undef, "table", undef, BDB::BTREE, BDB::AUTO_COMMIT | BDB::CREATE | BDB::READ_UNCOMMITTED, 0600;
310
311 for (1..1000) {
312 db_put $db, undef, "key $_", "data $_";
313
314 db_key_range $db, undef, "key $_", my $keyrange;
315 my ($lt, $eq, $gt) = @$keyrange;
316 }
317
318 db_del $db, undef, "key $_" for 1..1000;
319
320 db_sync $db;
321
322 DB_TXN/transaction methods
323 Methods available on DB_TXN/$txn handles:
324
325 DESTROY (DB_TXN_ornull *txn)
326 CODE:
327 if (txn)
328 txn->abort (txn);
329
330 $int = $txn->set_timeout (NV timeout_seconds, U32 flags = SET_TXN_TIMEOUT)
331 flags: SET_LOCK_TIMEOUT SET_TXN_TIMEOUT
332
333 $bool = $txn->failed
334 # see db_txn_finish documentation, above
335
336 DBC/cursor methods
337 Methods available on DBC/$dbc handles:
338
339 DESTROY (DBC_ornull *dbc)
340 CODE:
341 if (dbc)
342 dbc->c_close (dbc);
343
344 $int = $cursor->set_priority ($priority = PRIORITY_*)
345
346 Example:
347 my $c = $db->cursor;
348
349 for (;;) {
350 db_c_get $c, my $key, my $data, BDB::NEXT;
351 warn "<$!,$key,$data>";
352 last if $!;
353 }
354
355 db_c_close $c;
356
357 DB_SEQUENCE/sequence methods
358 Methods available on DB_SEQUENCE/$seq handles:
359
360 DESTROY (DB_SEQUENCE_ornull *seq)
361 CODE:
362 if (seq)
363 seq->close (seq, 0);
364
365 $int = $seq->initial_value (db_seq_t value)
366 $int = $seq->set_cachesize (U32 size)
367 $int = $seq->set_flags (U32 flags)
368 flags: SEQ_DEC SEQ_INC SEQ_WRAP
369 $int = $seq->set_range (db_seq_t min, db_seq_t max)
370
371 Example:
372 my $seq = $db->sequence;
373
374 db_sequence_open $seq, undef, "seq", BDB::CREATE;
375 db_sequence_get $seq, undef, 1, my $value;
376
44 SUPPORT FUNCTIONS 377 SUPPORT FUNCTIONS
45 EVENT PROCESSING AND EVENT LOOP INTEGRATION 378 EVENT PROCESSING AND EVENT LOOP INTEGRATION
379 $msg = BDB::strerror [$errno]
380 Returns the string corresponding to the given errno value. If no
381 argument is given, use $!.
382
383 Note that the BDB module also patches the $! variable directly, so
384 you should be able to get a bdb error string by simply stringifying
385 $!.
386
46 $fileno = BDB::AIO::poll_fileno 387 $fileno = BDB::poll_fileno
47 Return the *request result pipe file descriptor*. This filehandle 388 Return the *request result pipe file descriptor*. This filehandle
48 must be polled for reading by some mechanism outside this module 389 must be polled for reading by some mechanism outside this module
49 (e.g. Event or select, see below or the SYNOPSIS). If the pipe 390 (e.g. Event or select, see below or the SYNOPSIS). If the pipe
50 becomes readable you have to call "poll_cb" to check the results. 391 becomes readable you have to call "poll_cb" to check the results.
51 392
52 See "poll_cb" for an example. 393 See "poll_cb" for an example.
53 394
54 BDB::AIO::poll_cb 395 BDB::poll_cb
55 Process some outstanding events on the result pipe. You have to call 396 Process some outstanding events on the result pipe. You have to call
56 this regularly. Returns the number of events processed. Returns 397 this regularly. Returns the number of events processed. Returns
57 immediately when no events are outstanding. The amount of events 398 immediately when no events are outstanding. The amount of events
58 processed depends on the settings of "BDB::AIO::max_poll_req" and 399 processed depends on the settings of "BDB::max_poll_req" and
59 "BDB::AIO::max_poll_time". 400 "BDB::max_poll_time".
60 401
61 If not all requests were processed for whatever reason, the 402 If not all requests were processed for whatever reason, the
62 filehandle will still be ready when "poll_cb" returns. 403 filehandle will still be ready when "poll_cb" returns.
63 404
64 Example: Install an Event watcher that automatically calls 405 Example: Install an Event watcher that automatically calls
65 BDB::AIO::poll_cb with high priority: 406 BDB::poll_cb with high priority:
66 407
67 Event->io (fd => BDB::AIO::poll_fileno, 408 Event->io (fd => BDB::poll_fileno,
68 poll => 'r', async => 1, 409 poll => 'r', async => 1,
69 cb => \&BDB::AIO::poll_cb); 410 cb => \&BDB::poll_cb);
70 411
71 BDB::AIO::max_poll_reqs $nreqs 412 BDB::max_poll_reqs $nreqs
72 BDB::AIO::max_poll_time $seconds 413 BDB::max_poll_time $seconds
73 These set the maximum number of requests (default 0, meaning 414 These set the maximum number of requests (default 0, meaning
74 infinity) that are being processed by "BDB::AIO::poll_cb" in one 415 infinity) that are being processed by "BDB::poll_cb" in one call,
75 call, respectively the maximum amount of time (default 0, meaning 416 respectively the maximum amount of time (default 0, meaning
76 infinity) spent in "BDB::AIO::poll_cb" to process requests (more 417 infinity) spent in "BDB::poll_cb" to process requests (more
77 correctly the mininum amount of time "poll_cb" is allowed to use). 418 correctly the mininum amount of time "poll_cb" is allowed to use).
78 419
79 Setting "max_poll_time" to a non-zero value creates an overhead of 420 Setting "max_poll_time" to a non-zero value creates an overhead of
80 one syscall per request processed, which is not normally a problem 421 one syscall per request processed, which is not normally a problem
81 unless your callbacks are really really fast or your OS is really 422 unless your callbacks are really really fast or your OS is really
… …
86 interactiveness when perl is not fast enough to process all requests 427 interactiveness when perl is not fast enough to process all requests
87 in time. 428 in time.
88 429
89 For interactive programs, values such as 0.01 to 0.1 should be fine. 430 For interactive programs, values such as 0.01 to 0.1 should be fine.
90 431
91 Example: Install an Event watcher that automatically calls 432 Example: Install an EV watcher that automatically calls BDB::poll_cb
92 BDB::AIO::poll_cb with low priority, to ensure that other parts of 433 with low priority, to ensure that other parts of the program get the
93 the program get the CPU sometimes even under high AIO load. 434 CPU sometimes even under high load.
94 435
95 # try not to spend much more than 0.1s in poll_cb 436 # try not to spend much more than 0.1s in poll_cb
96 BDB::AIO::max_poll_time 0.1; 437 BDB::max_poll_time 0.1;
97 438
98 # use a low priority so other tasks have priority 439 my $bdb_poll = EV::io BDB::poll_fileno, EV::READ, \&BDB::poll_cb);
99 Event->io (fd => BDB::AIO::poll_fileno,
100 poll => 'r', nice => 1,
101 cb => &BDB::AIO::poll_cb);
102 440
103 BDB::AIO::poll_wait 441 BDB::poll_wait
104 If there are any outstanding requests and none of them in the result 442 If there are any outstanding requests and none of them in the result
105 phase, wait till the result filehandle becomes ready for reading 443 phase, wait till the result filehandle becomes ready for reading
106 (simply does a "select" on the filehandle. This is useful if you 444 (simply does a "select" on the filehandle. This is useful if you
107 want to synchronously wait for some requests to finish). 445 want to synchronously wait for some requests to finish).
108 446
109 See "nreqs" for an example. 447 See "nreqs" for an example.
110 448
111 BDB::AIO::poll 449 BDB::poll
112 Waits until some requests have been handled. 450 Waits until some requests have been handled.
113 451
114 Returns the number of requests processed, but is otherwise strictly 452 Returns the number of requests processed, but is otherwise strictly
115 equivalent to: 453 equivalent to:
116 454
117 BDB::AIO::poll_wait, BDB::AIO::poll_cb 455 BDB::poll_wait, BDB::poll_cb
118 456
119 BDB::AIO::flush 457 BDB::flush
120 Wait till all outstanding AIO requests have been handled. 458 Wait till all outstanding BDB requests have been handled.
121 459
122 Strictly equivalent to: 460 Strictly equivalent to:
123 461
124 BDB::AIO::poll_wait, BDB::AIO::poll_cb 462 BDB::poll_wait, BDB::poll_cb
125 while BDB::AIO::nreqs; 463 while BDB::nreqs;
126 464
127 CONTROLLING THE NUMBER OF THREADS 465 CONTROLLING THE NUMBER OF THREADS
128 BDB::AIO::min_parallel $nthreads 466 BDB::min_parallel $nthreads
129 Set the minimum number of AIO threads to $nthreads. The current 467 Set the minimum number of BDB threads to $nthreads. The current
130 default is 8, which means eight asynchronous operations can execute 468 default is 8, which means eight asynchronous operations can execute
131 concurrently at any one time (the number of outstanding requests, 469 concurrently at any one time (the number of outstanding requests,
132 however, is unlimited). 470 however, is unlimited).
133 471
134 BDB::AIO starts threads only on demand, when an AIO request is 472 BDB starts threads only on demand, when an BDB request is queued and
135 queued and no free thread exists. Please note that queueing up a 473 no free thread exists. Please note that queueing up a hundred
136 hundred requests can create demand for a hundred threads, even if it 474 requests can create demand for a hundred threads, even if it turns
137 turns out that everything is in the cache and could have been 475 out that everything is in the cache and could have been processed
138 processed faster by a single thread. 476 faster by a single thread.
139 477
140 It is recommended to keep the number of threads relatively low, as 478 It is recommended to keep the number of threads relatively low, as
141 some Linux kernel versions will scale negatively with the number of 479 some Linux kernel versions will scale negatively with the number of
142 threads (higher parallelity => MUCH higher latency). With current 480 threads (higher parallelity => MUCH higher latency). With current
143 Linux 2.6 versions, 4-32 threads should be fine. 481 Linux 2.6 versions, 4-32 threads should be fine.
144 482
145 Under most circumstances you don't need to call this function, as 483 Under most circumstances you don't need to call this function, as
146 the module selects a default that is suitable for low to moderate 484 the module selects a default that is suitable for low to moderate
147 load. 485 load.
148 486
149 BDB::AIO::max_parallel $nthreads 487 BDB::max_parallel $nthreads
150 Sets the maximum number of AIO threads to $nthreads. If more than 488 Sets the maximum number of BDB threads to $nthreads. If more than
151 the specified number of threads are currently running, this function 489 the specified number of threads are currently running, this function
152 kills them. This function blocks until the limit is reached. 490 kills them. This function blocks until the limit is reached.
153 491
154 While $nthreads are zero, aio requests get queued but not executed 492 While $nthreads are zero, aio requests get queued but not executed
155 until the number of threads has been increased again. 493 until the number of threads has been increased again.
… …
158 ensure that all threads are killed and that there are no outstanding 496 ensure that all threads are killed and that there are no outstanding
159 requests. 497 requests.
160 498
161 Under normal circumstances you don't need to call this function. 499 Under normal circumstances you don't need to call this function.
162 500
163 BDB::AIO::max_idle $nthreads 501 BDB::max_idle $nthreads
164 Limit the number of threads (default: 4) that are allowed to idle 502 Limit the number of threads (default: 4) that are allowed to idle
165 (i.e., threads that did not get a request to process within 10 503 (i.e., threads that did not get a request to process within 10
166 seconds). That means if a thread becomes idle while $nthreads other 504 seconds). That means if a thread becomes idle while $nthreads other
167 threads are also idle, it will free its resources and exit. 505 threads are also idle, it will free its resources and exit.
168 506
… …
173 511
174 The default is probably ok in most situations, especially if thread 512 The default is probably ok in most situations, especially if thread
175 creation is fast. If thread creation is very slow on your system you 513 creation is fast. If thread creation is very slow on your system you
176 might want to use larger values. 514 might want to use larger values.
177 515
178 $oldmaxreqs = BDB::AIO::max_outstanding $maxreqs 516 $oldmaxreqs = BDB::max_outstanding $maxreqs
179 This is a very bad function to use in interactive programs because 517 This is a very bad function to use in interactive programs because
180 it blocks, and a bad way to reduce concurrency because it is 518 it blocks, and a bad way to reduce concurrency because it is
181 inexact: Better use an "aio_group" together with a feed callback. 519 inexact: Better use an "aio_group" together with a feed callback.
182 520
183 Sets the maximum number of outstanding requests to $nreqs. If you to 521 Sets the maximum number of outstanding requests to $nreqs. If you to
… …
191 You can still queue as many requests as you want. Therefore, 529 You can still queue as many requests as you want. Therefore,
192 "max_oustsanding" is mainly useful in simple scripts (with low 530 "max_oustsanding" is mainly useful in simple scripts (with low
193 values) or as a stop gap to shield against fatal memory overflow 531 values) or as a stop gap to shield against fatal memory overflow
194 (with large values). 532 (with large values).
195 533
534 BDB::set_sync_prepare $cb
535 Sets a callback that is called whenever a request is created without
536 an explicit callback. It has to return two code references. The
537 first is used as the request callback, and the second is called to
538 wait until the first callback has been called. The default
539 implementation works like this:
540
541 sub {
542 my $status;
543 (
544 sub { $status = $! },
545 sub { BDB::poll while !defined $status; $! = $status },
546 )
547 }
548
196 STATISTICAL INFORMATION 549 STATISTICAL INFORMATION
197 BDB::AIO::nreqs 550 BDB::nreqs
198 Returns the number of requests currently in the ready, execute or 551 Returns the number of requests currently in the ready, execute or
199 pending states (i.e. for which their callback has not been invoked 552 pending states (i.e. for which their callback has not been invoked
200 yet). 553 yet).
201 554
202 Example: wait till there are no outstanding requests anymore: 555 Example: wait till there are no outstanding requests anymore:
203 556
204 BDB::AIO::poll_wait, BDB::AIO::poll_cb 557 BDB::poll_wait, BDB::poll_cb
205 while BDB::AIO::nreqs; 558 while BDB::nreqs;
206 559
207 BDB::AIO::nready 560 BDB::nready
208 Returns the number of requests currently in the ready state (not yet 561 Returns the number of requests currently in the ready state (not yet
209 executed). 562 executed).
210 563
211 BDB::AIO::npending 564 BDB::npending
212 Returns the number of requests currently in the pending state 565 Returns the number of requests currently in the pending state
213 (executed, but not yet processed by poll_cb). 566 (executed, but not yet processed by poll_cb).
214 567
215 FORK BEHAVIOUR 568 FORK BEHAVIOUR
216 This module should do "the right thing" when the process using it forks: 569 This module should do "the right thing" when the process using it forks:
217 570
218 Before the fork, IO::AIO enters a quiescent state where no requests can 571 Before the fork, BDB enters a quiescent state where no requests can be
219 be added in other threads and no results will be processed. After the 572 added in other threads and no results will be processed. After the fork
220 fork the parent simply leaves the quiescent state and continues 573 the parent simply leaves the quiescent state and continues
221 request/result processing, while the child frees the request/result 574 request/result processing, while the child frees the request/result
222 queue (so that the requests started before the fork will only be handled 575 queue (so that the requests started before the fork will only be handled
223 in the parent). Threads will be started on demand until the limit set in 576 in the parent). Threads will be started on demand until the limit set in
224 the parent process has been reached again. 577 the parent process has been reached again.
225 578
226 In short: the parent will, after a short pause, continue as if fork had 579 In short: the parent will, after a short pause, continue as if fork had
227 not been called, while the child will act as if IO::AIO has not been 580 not been called, while the child will act as if BDB has not been used
228 used yet. 581 yet.
582
583 Win32 note: there is no fork on win32, and perls emulation of it is too
584 broken to be supported, so do not use BDB in a windows pseudo-fork,
585 better yet, switch to a more capable platform.
229 586
230 MEMORY USAGE 587 MEMORY USAGE
231 Per-request usage: 588 Per-request usage:
232 589
233 Each aio request uses - depending on your architecture - around 100-200 590 Each aio request uses - depending on your architecture - around 100-200
234 bytes of memory. In addition, stat requests need a stat buffer (possibly 591 bytes of memory. In addition, stat requests need a stat buffer (possibly
235 a few hundred bytes), readdir requires a result buffer and so on. Perl 592 a few hundred bytes), readdir requires a result buffer and so on. Perl
236 scalars and other data passed into aio requests will also be locked and 593 scalars and other data passed into aio requests will also be locked and
237 will consume memory till the request has entered the done state. 594 will consume memory till the request has entered the done state.
238 595
239 This is now awfully much, so queuing lots of requests is not usually a 596 This is not awfully much, so queuing lots of requests is not usually a
240 problem. 597 problem.
241 598
242 Per-thread usage: 599 Per-thread usage:
243 600
244 In the execution phase, some aio requests require more memory for 601 In the execution phase, some aio requests require more memory for
245 temporary buffers, and each thread requires a stack and other data 602 temporary buffers, and each thread requires a stack and other data
246 structures (usually around 16k-128k, depending on the OS). 603 structures (usually around 16k-128k, depending on the OS).
247 604
248KNOWN BUGS 605KNOWN BUGS
249 Known bugs will be fixed in the next release. 606 Known bugs will be fixed in the next release, except:
607
608 If you use a transaction in any request, and the request returns
609 with an operating system error or DB_LOCK_NOTGRANTED, the internal
610 TXN_DEADLOCK flag will be set on the transaction. See C<db_txn_finish>,
611 above.
250 612
251SEE ALSO 613SEE ALSO
252 Coro::AIO. 614 AnyEvent::BDB (event loop integration), Coro::BDB (more natural syntax),
615 IO::AIO (nice to have).
253 616
254AUTHOR 617AUTHOR
255 Marc Lehmann <schmorp@schmorp.de> 618 Marc Lehmann <schmorp@schmorp.de>
256 http://home.schmorp.de/ 619 http://home.schmorp.de/
257 620

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines