ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/BDB/README
(Generate patch)

Comparing BDB/README (file contents):
Revision 1.1 by root, Mon Feb 5 18:40:55 2007 UTC vs.
Revision 1.5 by root, Wed Dec 12 01:20:54 2007 UTC

1NAME 1NAME
2 BDB::AIO - Asynchronous Berkeley DB access 2 BDB - Asynchronous Berkeley DB access
3 3
4SYNOPSIS 4SYNOPSIS
5 use BDB::AIO; 5 use BDB;
6
7 my $env = db_env_create;
8
9 mkdir "bdtest", 0700;
10 db_env_open
11 $env,
12 "bdtest",
13 BDB::INIT_LOCK | BDB::INIT_LOG | BDB::INIT_MPOOL
14 | BDB::INIT_TXN | BDB::RECOVER | BDB::USE_ENVIRON | BDB::CREATE,
15 0600;
16
17 $env->set_flags (BDB::AUTO_COMMIT | BDB::TXN_NOSYNC, 1);
18
19 my $db = db_create $env;
20 db_open $db, undef, "table", undef, BDB::BTREE, BDB::AUTO_COMMIT | BDB::CREATE
21 | BDB::READ_UNCOMMITTED, 0600;
22 db_put $db, undef, "key", "data", 0, sub {
23 db_del $db, undef, "key";
24 };
25 db_sync $db;
26
27 # automatic result processing with AnyEvent:
28 our $FH; open $FH, "<&=" . BDB::poll_fileno;
29 our $WATCHER = AnyEvent->io (fh => $FH, poll => 'r', cb => \&BDB::poll_cb);
30
31 # automatic result processing with EV:
32 my $WATCHER = EV::io BDB::poll_fileno, EV::READ, \&BDB::poll_cb;
33
34 # with Glib:
35 add_watch Glib::IO BDB::poll_fileno,
36 in => sub { BDB::poll_cb; 1 };
37
38 # or simply flush manually
39 BDB::flush;
6 40
7DESCRIPTION 41DESCRIPTION
8 EXAMPLE 42 See the BerkeleyDB documentation
43 (<http://www.oracle.com/technology/documentation/berkeley-db/db/index.ht
44 ml>). The BDB API is very similar to the C API (the translation has been
45 very faithful).
46
47 See also the example sections in the document below and possibly the eg/
48 subdirectory of the BDB distribution. Last not least see the IO::AIO
49 documentation, as that module uses almost the same asynchronous request
50 model as this module.
51
52 I know this is woefully inadequate documentation. Send a patch!
53
9REQUEST ANATOMY AND LIFETIME 54REQUEST ANATOMY AND LIFETIME
10 Every request method creates a request. which is a C data structure not 55 Every request method creates a request. which is a C data structure not
11 directly visible to Perl. 56 directly visible to Perl.
12 57
13 During their existance, bdb requests travel through the following 58 During their existance, bdb requests travel through the following
… …
39 Request has reached the end of its lifetime and holds no resources 84 Request has reached the end of its lifetime and holds no resources
40 anymore (except possibly for the Perl object, but its connection to 85 anymore (except possibly for the Perl object, but its connection to
41 the actual aio request is severed and calling its methods will 86 the actual aio request is severed and calling its methods will
42 either do nothing or result in a runtime error). 87 either do nothing or result in a runtime error).
43 88
89 BERKELEYDB FUNCTIONS
90 All of these are functions. The create functions simply return a new
91 object and never block. All the remaining functions all take an optional
92 callback as last argument. If it is missing, then the function will be
93 executed synchronously. In both cases, $! will reflect the return value
94 of the function.
95
96 BDB functions that cannot block (mostly functions that manipulate
97 settings) are method calls on the relevant objects, so the rule of thumb
98 is: if its a method, its not blocking, if its a function, it takes a
99 callback as last argument.
100
101 In the following, $int signifies an integer return value, "octetstring"
102 is a "binary string" (i.e. a perl string with no character indices
103 >255), "U32" is an unsigned 32 bit integer, "int" is some integer, "NV"
104 is a floating point value.
105
106 The "SV *" types are generic perl scalars (for input and output of data
107 values), and the "SV *callback" is the optional callback function to
108 call when the request is completed.
109
110 The various "DB_ENV" etc. arguments are handles return by
111 "db_env_create", "db_create", "txn_begin" and so on. If they have an
112 appended "_ornull" this means they are optional and you can pass "undef"
113 for them, resulting a NULL pointer on the C level.
114
115 BDB functions
116 Functions in the BDB namespace, exported by default:
117
118 $env = db_env_create (U32 env_flags = 0)
119 flags: RPCCLIENT
120
121 db_env_open (DB_ENV *env, octetstring db_home, U32 open_flags, int mode, SV *callback = &PL_sv_undef)
122 open_flags: INIT_CDB INIT_LOCK INIT_LOG INIT_MPOOL INIT_REP INIT_TXN RECOVER RECOVER_FATAL USE_ENVIRON USE_ENVIRON_ROOT CREATE LOCKDOWN PRIVATE REGISTER SYSTEM_MEM
123 db_env_close (DB_ENV *env, U32 flags = 0, SV *callback = &PL_sv_undef)
124 db_env_txn_checkpoint (DB_ENV *env, U32 kbyte = 0, U32 min = 0, U32 flags = 0, SV *callback = &PL_sv_undef)
125 flags: FORCE
126 db_env_lock_detect (DB_ENV *env, U32 flags = 0, U32 atype = DB_LOCK_DEFAULT, SV *dummy = 0, SV *callback = &PL_sv_undef)
127 atype: LOCK_DEFAULT LOCK_EXPIRE LOCK_MAXLOCKS LOCK_MAXWRITE LOCK_MINLOCKS LOCK_MINWRITE LOCK_OLDEST LOCK_RANDOM LOCK_YOUNGEST
128 db_env_memp_sync (DB_ENV *env, SV *dummy = 0, SV *callback = &PL_sv_undef)
129 db_env_memp_trickle (DB_ENV *env, int percent, SV *dummy = 0, SV *callback = &PL_sv_undef)
130
131 $db = db_create (DB_ENV *env = 0, U32 flags = 0)
132 flags: XA_CREATE
133
134 db_open (DB *db, DB_TXN_ornull *txnid, octetstring file, octetstring database, int type, U32 flags, int mode, SV *callback = &PL_sv_undef)
135 flags: AUTO_COMMIT CREATE EXCL MULTIVERSION NOMMAP RDONLY READ_UNCOMMITTED THREAD TRUNCATE
136 db_close (DB *db, U32 flags = 0, SV *callback = &PL_sv_undef)
137 flags: DB_NOSYNC
138 db_upgrade (DB *db, octetstring file, U32 flags = 0, SV *callback = &PL_sv_undef)
139 db_compact (DB *db, DB_TXN_ornull *txn = 0, SV *start = 0, SV *stop = 0, SV *unused1 = 0, U32 flags = DB_FREE_SPACE, SV *unused2 = 0, SV *callback = &PL_sv_undef)
140 flags: FREELIST_ONLY FREE_SPACE
141 db_sync (DB *db, U32 flags = 0, SV *callback = &PL_sv_undef)
142 db_key_range (DB *db, DB_TXN_ornull *txn, SV *key, SV *key_range, U32 flags = 0, SV *callback = &PL_sv_undef)
143 db_put (DB *db, DB_TXN_ornull *txn, SV *key, SV *data, U32 flags = 0, SV *callback = &PL_sv_undef)
144 flags: APPEND NODUPDATA NOOVERWRITE
145 db_get (DB *db, DB_TXN_ornull *txn, SV *key, SV *data, U32 flags = 0, SV *callback = &PL_sv_undef)
146 flags: CONSUME CONSUME_WAIT GET_BOTH SET_RECNO MULTIPLE READ_COMMITTED READ_UNCOMMITTED RMW
147 db_pget (DB *db, DB_TXN_ornull *txn, SV *key, SV *pkey, SV *data, U32 flags = 0, SV *callback = &PL_sv_undef)
148 flags: CONSUME CONSUME_WAIT GET_BOTH SET_RECNO MULTIPLE READ_COMMITTED READ_UNCOMMITTED RMW
149 db_del (DB *db, DB_TXN_ornull *txn, SV *key, U32 flags = 0, SV *callback = &PL_sv_undef)
150 db_txn_commit (DB_TXN *txn, U32 flags = 0, SV *callback = &PL_sv_undef)
151 flags: TXN_NOSYNC TXN_SYNC
152 db_txn_abort (DB_TXN *txn, SV *callback = &PL_sv_undef)
153
154 db_c_close (DBC *dbc, SV *callback = &PL_sv_undef)
155 db_c_count (DBC *dbc, SV *count, U32 flags = 0, SV *callback = &PL_sv_undef)
156 db_c_put (DBC *dbc, SV *key, SV *data, U32 flags = 0, SV *callback = &PL_sv_undef)
157 flags: AFTER BEFORE CURRENT KEYFIRST KEYLAST NODUPDATA
158 db_c_get (DBC *dbc, SV *key, SV *data, U32 flags = 0, SV *callback = &PL_sv_undef)
159 flags: CURRENT FIRST GET_BOTH GET_BOTH_RANGE GET_RECNO JOIN_ITEM LAST NEXT NEXT_DUP NEXT_NODUP PREV PREV_DUP PREV_NODUP SET SET_RANGE SET_RECNO READ_UNCOMMITTED MULTIPLE MULTIPLE_KEY RMW
160 db_c_pget (DBC *dbc, SV *key, SV *pkey, SV *data, U32 flags = 0, SV *callback = &PL_sv_undef)
161 db_c_del (DBC *dbc, U32 flags = 0, SV *callback = &PL_sv_undef)
162
163 db_sequence_open (DB_SEQUENCE *seq, DB_TXN_ornull *txnid, SV *key, U32 flags = 0, SV *callback = &PL_sv_undef)
164 flags: CREATE EXCL
165 db_sequence_close (DB_SEQUENCE *seq, U32 flags = 0, SV *callback = &PL_sv_undef)
166 db_sequence_get (DB_SEQUENCE *seq, DB_TXN_ornull *txnid, int delta, SV *seq_value, U32 flags = DB_TXN_NOSYNC, SV *callback = &PL_sv_undef)
167 flags: TXN_NOSYNC
168 db_sequence_remove (DB_SEQUENCE *seq, DB_TXN_ornull *txnid = 0, U32 flags = 0, SV *callback = &PL_sv_undef)
169 flags: TXN_NOSYNC
170
171 db_txn_finish (DB_TXN *txn, U32 flags = 0, SV *callback = &PL_sv_undef)
172 This is not actually a Berkeley DB function but a BDB module extension.
173 The background for this exytension is: It is very annoying to have to
174 check every single BDB function for error returns and provide a codepath
175 out of your transaction. While the BDB module still makes this possible,
176 it contains the following extensions:
177
178 When a transaction-protected function returns any operating system error
179 (errno > 0), BDB will set the "TXN_DEADLOCK" flag on the transaction.
180 This flag is also set by Berkeley DB functions themselves when an
181 operation fails with LOCK_DEADLOCK, and it causes all further operations
182 on that transaction (including "db_txn_commit") to fail.
183
184 The "db_txn_finish" request will look at this flag, and, if it is set,
185 will automatically call "db_txn_abort" (setting errno to "LOCK_DEADLOCK"
186 if it isn't set to something else yet). If it isn't set, it will call
187 "db_txn_commit" and return the error normally.
188
189 How to use this? Easy: just write your transaction normally:
190
191 my $txn = $db_env->txn_begin;
192 db_get $db, $txn, "key", my $data;
193 db_put $db, $txn, "key", $data + 1 unless $! == BDB::NOTFOUND;
194 db_txn_finish $txn;
195 die "transaction failed" if $!;
196
197 That is, handle only the expected errors. If something unexpected
198 happens (EIO, LOCK_NOTGRANTED or a deadlock in either db_get or db_put),
199 then the remaining requests (db_put in this case) will simply be skipped
200 (they will fail with LOCK_DEADLOCK) and the transaction will be aborted.
201
202 You can use the "$txn->failed" method to check wether a transaction has
203 failed in this way and abort further processing (excluding
204 "db_txn_finish").
205
206 DB_ENV/database environment methods
207 Methods available on DB_ENV/$env handles:
208
209 DESTROY (DB_ENV_ornull *env)
210 CODE:
211 if (env)
212 env->close (env, 0);
213
214 $int = $env->set_data_dir (const char *dir)
215 $int = $env->set_tmp_dir (const char *dir)
216 $int = $env->set_lg_dir (const char *dir)
217 $int = $env->set_shm_key (long shm_key)
218 $int = $env->set_cachesize (U32 gbytes, U32 bytes, int ncache = 0)
219 $int = $env->set_flags (U32 flags, int onoff)
220 $env->set_errfile (FILE *errfile = 0)
221 $env->set_msgfile (FILE *msgfile = 0)
222 $int = $env->set_verbose (U32 which, int onoff = 1)
223 $int = $env->set_encrypt (const char *password, U32 flags = 0)
224 $int = $env->set_timeout (NV timeout_seconds, U32 flags = SET_TXN_TIMEOUT)
225 $int = $env->set_mp_max_openfd (int maxopenfd);
226 $int = $env->set_mp_max_write (int maxwrite, int maxwrite_sleep);
227 $int = $env->set_mp_mmapsize (int mmapsize_mb)
228 $int = $env->set_lk_detect (U32 detect = DB_LOCK_DEFAULT)
229 $int = $env->set_lk_max_lockers (U32 max)
230 $int = $env->set_lk_max_locks (U32 max)
231 $int = $env->set_lk_max_objects (U32 max)
232 $int = $env->set_lg_bsize (U32 max)
233 $int = $env->set_lg_max (U32 max)
234 $int = $env->mutex_set_increment (U32 increment)
235 $int = $env->mutex_set_tas_spins (U32 tas_spins)
236 $int = $env->mutex_set_max (U32 max)
237 $int = $env->mutex_set_align (U32 align)
238
239 $txn = $env->txn_begin (DB_TXN_ornull *parent = 0, U32 flags = 0)
240 flags: READ_COMMITTED READ_UNCOMMITTED TXN_NOSYNC TXN_NOWAIT TXN_SNAPSHOT TXN_SYNC TXN_WAIT TXN_WRITE_NOSYNC
241
242 Example:
243 use AnyEvent;
244 use BDB;
245
246 our $FH; open $FH, "<&=" . BDB::poll_fileno;
247 our $WATCHER = AnyEvent->io (fh => $FH, poll => 'r', cb => \&BDB::poll_cb);
248
249 BDB::min_parallel 8;
250
251 my $env = db_env_create;
252
253 mkdir "bdtest", 0700;
254 db_env_open
255 $env,
256 "bdtest",
257 BDB::INIT_LOCK | BDB::INIT_LOG | BDB::INIT_MPOOL | BDB::INIT_TXN | BDB::RECOVER | BDB::USE_ENVIRON | BDB::CREATE,
258 0600;
259
260 $env->set_flags (BDB::AUTO_COMMIT | BDB::TXN_NOSYNC, 1);
261
262 DB/database methods
263 Methods available on DB/$db handles:
264
265 DESTROY (DB_ornull *db)
266 CODE:
267 if (db)
268 {
269 SV *env = (SV *)db->app_private;
270 db->close (db, 0);
271 SvREFCNT_dec (env);
272 }
273
274 $int = $db->set_cachesize (U32 gbytes, U32 bytes, int ncache = 0)
275 $int = $db->set_flags (U32 flags)
276 flags: CHKSUM ENCRYPT TXN_NOT_DURABLE
277 Btree: DUP DUPSORT RECNUM REVSPLITOFF
278 Hash: DUP DUPSORT
279 Queue: INORDER
280 Recno: RENUMBER SNAPSHOT
281
282 $int = $db->set_encrypt (const char *password, U32 flags)
283 $int = $db->set_lorder (int lorder)
284 $int = $db->set_bt_minkey (U32 minkey)
285 $int = $db->set_re_delim (int delim)
286 $int = $db->set_re_pad (int re_pad)
287 $int = $db->set_re_source (char *source)
288 $int = $db->set_re_len (U32 re_len)
289 $int = $db->set_h_ffactor (U32 h_ffactor)
290 $int = $db->set_h_nelem (U32 h_nelem)
291 $int = $db->set_q_extentsize (U32 extentsize)
292
293 $dbc = $db->cursor (DB_TXN_ornull *txn = 0, U32 flags = 0)
294 flags: READ_COMMITTED READ_UNCOMMITTED WRITECURSOR TXN_SNAPSHOT
295 $seq = $db->sequence (U32 flags = 0)
296
297 Example:
298 my $db = db_create $env;
299 db_open $db, undef, "table", undef, BDB::BTREE, BDB::AUTO_COMMIT | BDB::CREATE | BDB::READ_UNCOMMITTED, 0600;
300
301 for (1..1000) {
302 db_put $db, undef, "key $_", "data $_";
303
304 db_key_range $db, undef, "key $_", my $keyrange;
305 my ($lt, $eq, $gt) = @$keyrange;
306 }
307
308 db_del $db, undef, "key $_" for 1..1000;
309
310 db_sync $db;
311
312 DB_TXN/transaction methods
313 Methods available on DB_TXN/$txn handles:
314
315 DESTROY (DB_TXN_ornull *txn)
316 CODE:
317 if (txn)
318 txn->abort (txn);
319
320 $int = $txn->set_timeout (NV timeout_seconds, U32 flags = SET_TXN_TIMEOUT)
321 flags: SET_LOCK_TIMEOUT SET_TXN_TIMEOUT
322
323 $bool = $txn->failed
324 # see db_txn_finish documentation, above
325
326 DBC/cursor methods
327 Methods available on DBC/$dbc handles:
328
329 DESTROY (DBC_ornull *dbc)
330 CODE:
331 if (dbc)
332 dbc->c_close (dbc);
333
334 Example:
335 my $c = $db->cursor;
336
337 for (;;) {
338 db_c_get $c, my $key, my $data, BDB::NEXT;
339 warn "<$!,$key,$data>";
340 last if $!;
341 }
342
343 db_c_close $c;
344
345 DB_SEQUENCE/sequence methods
346 Methods available on DB_SEQUENCE/$seq handles:
347
348 DESTROY (DB_SEQUENCE_ornull *seq)
349 CODE:
350 if (seq)
351 seq->close (seq, 0);
352
353 $int = $seq->initial_value (db_seq_t value)
354 $int = $seq->set_cachesize (U32 size)
355 $int = $seq->set_flags (U32 flags)
356 flags: SEQ_DEC SEQ_INC SEQ_WRAP
357 $int = $seq->set_range (db_seq_t min, db_seq_t max)
358
359 Example:
360 my $seq = $db->sequence;
361
362 db_sequence_open $seq, undef, "seq", BDB::CREATE;
363 db_sequence_get $seq, undef, 1, my $value;
364
44 SUPPORT FUNCTIONS 365 SUPPORT FUNCTIONS
45 EVENT PROCESSING AND EVENT LOOP INTEGRATION 366 EVENT PROCESSING AND EVENT LOOP INTEGRATION
367 $msg = BDB::strerror [$errno]
368 Returns the string corresponding to the given errno value. If no
369 argument is given, use $!.
370
46 $fileno = BDB::AIO::poll_fileno 371 $fileno = BDB::poll_fileno
47 Return the *request result pipe file descriptor*. This filehandle 372 Return the *request result pipe file descriptor*. This filehandle
48 must be polled for reading by some mechanism outside this module 373 must be polled for reading by some mechanism outside this module
49 (e.g. Event or select, see below or the SYNOPSIS). If the pipe 374 (e.g. Event or select, see below or the SYNOPSIS). If the pipe
50 becomes readable you have to call "poll_cb" to check the results. 375 becomes readable you have to call "poll_cb" to check the results.
51 376
52 See "poll_cb" for an example. 377 See "poll_cb" for an example.
53 378
54 BDB::AIO::poll_cb 379 BDB::poll_cb
55 Process some outstanding events on the result pipe. You have to call 380 Process some outstanding events on the result pipe. You have to call
56 this regularly. Returns the number of events processed. Returns 381 this regularly. Returns the number of events processed. Returns
57 immediately when no events are outstanding. The amount of events 382 immediately when no events are outstanding. The amount of events
58 processed depends on the settings of "BDB::AIO::max_poll_req" and 383 processed depends on the settings of "BDB::max_poll_req" and
59 "BDB::AIO::max_poll_time". 384 "BDB::max_poll_time".
60 385
61 If not all requests were processed for whatever reason, the 386 If not all requests were processed for whatever reason, the
62 filehandle will still be ready when "poll_cb" returns. 387 filehandle will still be ready when "poll_cb" returns.
63 388
64 Example: Install an Event watcher that automatically calls 389 Example: Install an Event watcher that automatically calls
65 BDB::AIO::poll_cb with high priority: 390 BDB::poll_cb with high priority:
66 391
67 Event->io (fd => BDB::AIO::poll_fileno, 392 Event->io (fd => BDB::poll_fileno,
68 poll => 'r', async => 1, 393 poll => 'r', async => 1,
69 cb => \&BDB::AIO::poll_cb); 394 cb => \&BDB::poll_cb);
70 395
71 BDB::AIO::max_poll_reqs $nreqs 396 BDB::max_poll_reqs $nreqs
72 BDB::AIO::max_poll_time $seconds 397 BDB::max_poll_time $seconds
73 These set the maximum number of requests (default 0, meaning 398 These set the maximum number of requests (default 0, meaning
74 infinity) that are being processed by "BDB::AIO::poll_cb" in one 399 infinity) that are being processed by "BDB::poll_cb" in one call,
75 call, respectively the maximum amount of time (default 0, meaning 400 respectively the maximum amount of time (default 0, meaning
76 infinity) spent in "BDB::AIO::poll_cb" to process requests (more 401 infinity) spent in "BDB::poll_cb" to process requests (more
77 correctly the mininum amount of time "poll_cb" is allowed to use). 402 correctly the mininum amount of time "poll_cb" is allowed to use).
78 403
79 Setting "max_poll_time" to a non-zero value creates an overhead of 404 Setting "max_poll_time" to a non-zero value creates an overhead of
80 one syscall per request processed, which is not normally a problem 405 one syscall per request processed, which is not normally a problem
81 unless your callbacks are really really fast or your OS is really 406 unless your callbacks are really really fast or your OS is really
… …
86 interactiveness when perl is not fast enough to process all requests 411 interactiveness when perl is not fast enough to process all requests
87 in time. 412 in time.
88 413
89 For interactive programs, values such as 0.01 to 0.1 should be fine. 414 For interactive programs, values such as 0.01 to 0.1 should be fine.
90 415
91 Example: Install an Event watcher that automatically calls 416 Example: Install an EV watcher that automatically calls BDB::poll_cb
92 BDB::AIO::poll_cb with low priority, to ensure that other parts of 417 with low priority, to ensure that other parts of the program get the
93 the program get the CPU sometimes even under high AIO load. 418 CPU sometimes even under high load.
94 419
95 # try not to spend much more than 0.1s in poll_cb 420 # try not to spend much more than 0.1s in poll_cb
96 BDB::AIO::max_poll_time 0.1; 421 BDB::max_poll_time 0.1;
97 422
98 # use a low priority so other tasks have priority 423 my $bdb_poll = EV::io BDB::poll_fileno, EV::READ, \&BDB::poll_cb);
99 Event->io (fd => BDB::AIO::poll_fileno,
100 poll => 'r', nice => 1,
101 cb => &BDB::AIO::poll_cb);
102 424
103 BDB::AIO::poll_wait 425 BDB::poll_wait
104 If there are any outstanding requests and none of them in the result 426 If there are any outstanding requests and none of them in the result
105 phase, wait till the result filehandle becomes ready for reading 427 phase, wait till the result filehandle becomes ready for reading
106 (simply does a "select" on the filehandle. This is useful if you 428 (simply does a "select" on the filehandle. This is useful if you
107 want to synchronously wait for some requests to finish). 429 want to synchronously wait for some requests to finish).
108 430
109 See "nreqs" for an example. 431 See "nreqs" for an example.
110 432
111 BDB::AIO::poll 433 BDB::poll
112 Waits until some requests have been handled. 434 Waits until some requests have been handled.
113 435
114 Returns the number of requests processed, but is otherwise strictly 436 Returns the number of requests processed, but is otherwise strictly
115 equivalent to: 437 equivalent to:
116 438
117 BDB::AIO::poll_wait, BDB::AIO::poll_cb 439 BDB::poll_wait, BDB::poll_cb
118 440
119 BDB::AIO::flush 441 BDB::flush
120 Wait till all outstanding AIO requests have been handled. 442 Wait till all outstanding BDB requests have been handled.
121 443
122 Strictly equivalent to: 444 Strictly equivalent to:
123 445
124 BDB::AIO::poll_wait, BDB::AIO::poll_cb 446 BDB::poll_wait, BDB::poll_cb
125 while BDB::AIO::nreqs; 447 while BDB::nreqs;
126 448
127 CONTROLLING THE NUMBER OF THREADS 449 CONTROLLING THE NUMBER OF THREADS
128 BDB::AIO::min_parallel $nthreads 450 BDB::min_parallel $nthreads
129 Set the minimum number of AIO threads to $nthreads. The current 451 Set the minimum number of BDB threads to $nthreads. The current
130 default is 8, which means eight asynchronous operations can execute 452 default is 8, which means eight asynchronous operations can execute
131 concurrently at any one time (the number of outstanding requests, 453 concurrently at any one time (the number of outstanding requests,
132 however, is unlimited). 454 however, is unlimited).
133 455
134 BDB::AIO starts threads only on demand, when an AIO request is 456 BDB starts threads only on demand, when an BDB request is queued and
135 queued and no free thread exists. Please note that queueing up a 457 no free thread exists. Please note that queueing up a hundred
136 hundred requests can create demand for a hundred threads, even if it 458 requests can create demand for a hundred threads, even if it turns
137 turns out that everything is in the cache and could have been 459 out that everything is in the cache and could have been processed
138 processed faster by a single thread. 460 faster by a single thread.
139 461
140 It is recommended to keep the number of threads relatively low, as 462 It is recommended to keep the number of threads relatively low, as
141 some Linux kernel versions will scale negatively with the number of 463 some Linux kernel versions will scale negatively with the number of
142 threads (higher parallelity => MUCH higher latency). With current 464 threads (higher parallelity => MUCH higher latency). With current
143 Linux 2.6 versions, 4-32 threads should be fine. 465 Linux 2.6 versions, 4-32 threads should be fine.
144 466
145 Under most circumstances you don't need to call this function, as 467 Under most circumstances you don't need to call this function, as
146 the module selects a default that is suitable for low to moderate 468 the module selects a default that is suitable for low to moderate
147 load. 469 load.
148 470
149 BDB::AIO::max_parallel $nthreads 471 BDB::max_parallel $nthreads
150 Sets the maximum number of AIO threads to $nthreads. If more than 472 Sets the maximum number of BDB threads to $nthreads. If more than
151 the specified number of threads are currently running, this function 473 the specified number of threads are currently running, this function
152 kills them. This function blocks until the limit is reached. 474 kills them. This function blocks until the limit is reached.
153 475
154 While $nthreads are zero, aio requests get queued but not executed 476 While $nthreads are zero, aio requests get queued but not executed
155 until the number of threads has been increased again. 477 until the number of threads has been increased again.
… …
158 ensure that all threads are killed and that there are no outstanding 480 ensure that all threads are killed and that there are no outstanding
159 requests. 481 requests.
160 482
161 Under normal circumstances you don't need to call this function. 483 Under normal circumstances you don't need to call this function.
162 484
163 BDB::AIO::max_idle $nthreads 485 BDB::max_idle $nthreads
164 Limit the number of threads (default: 4) that are allowed to idle 486 Limit the number of threads (default: 4) that are allowed to idle
165 (i.e., threads that did not get a request to process within 10 487 (i.e., threads that did not get a request to process within 10
166 seconds). That means if a thread becomes idle while $nthreads other 488 seconds). That means if a thread becomes idle while $nthreads other
167 threads are also idle, it will free its resources and exit. 489 threads are also idle, it will free its resources and exit.
168 490
… …
173 495
174 The default is probably ok in most situations, especially if thread 496 The default is probably ok in most situations, especially if thread
175 creation is fast. If thread creation is very slow on your system you 497 creation is fast. If thread creation is very slow on your system you
176 might want to use larger values. 498 might want to use larger values.
177 499
178 $oldmaxreqs = BDB::AIO::max_outstanding $maxreqs 500 $oldmaxreqs = BDB::max_outstanding $maxreqs
179 This is a very bad function to use in interactive programs because 501 This is a very bad function to use in interactive programs because
180 it blocks, and a bad way to reduce concurrency because it is 502 it blocks, and a bad way to reduce concurrency because it is
181 inexact: Better use an "aio_group" together with a feed callback. 503 inexact: Better use an "aio_group" together with a feed callback.
182 504
183 Sets the maximum number of outstanding requests to $nreqs. If you to 505 Sets the maximum number of outstanding requests to $nreqs. If you to
… …
191 You can still queue as many requests as you want. Therefore, 513 You can still queue as many requests as you want. Therefore,
192 "max_oustsanding" is mainly useful in simple scripts (with low 514 "max_oustsanding" is mainly useful in simple scripts (with low
193 values) or as a stop gap to shield against fatal memory overflow 515 values) or as a stop gap to shield against fatal memory overflow
194 (with large values). 516 (with large values).
195 517
518 BDB::set_sync_prepare $cb
519 Sets a callback that is called whenever a request is created without
520 an explicit callback. It has to return two code references. The
521 first is used as the request callback, and the second is called to
522 wait until the first callback has been called. The default
523 implementation works like this:
524
525 sub {
526 my $status;
527 (
528 sub { $status = $! },
529 sub { BDB::poll while !defined $status; $! = $status },
530 )
531 }
532
196 STATISTICAL INFORMATION 533 STATISTICAL INFORMATION
197 BDB::AIO::nreqs 534 BDB::nreqs
198 Returns the number of requests currently in the ready, execute or 535 Returns the number of requests currently in the ready, execute or
199 pending states (i.e. for which their callback has not been invoked 536 pending states (i.e. for which their callback has not been invoked
200 yet). 537 yet).
201 538
202 Example: wait till there are no outstanding requests anymore: 539 Example: wait till there are no outstanding requests anymore:
203 540
204 BDB::AIO::poll_wait, BDB::AIO::poll_cb 541 BDB::poll_wait, BDB::poll_cb
205 while BDB::AIO::nreqs; 542 while BDB::nreqs;
206 543
207 BDB::AIO::nready 544 BDB::nready
208 Returns the number of requests currently in the ready state (not yet 545 Returns the number of requests currently in the ready state (not yet
209 executed). 546 executed).
210 547
211 BDB::AIO::npending 548 BDB::npending
212 Returns the number of requests currently in the pending state 549 Returns the number of requests currently in the pending state
213 (executed, but not yet processed by poll_cb). 550 (executed, but not yet processed by poll_cb).
214 551
215 FORK BEHAVIOUR 552 FORK BEHAVIOUR
216 This module should do "the right thing" when the process using it forks: 553 This module should do "the right thing" when the process using it forks:
217 554
218 Before the fork, IO::AIO enters a quiescent state where no requests can 555 Before the fork, BDB enters a quiescent state where no requests can be
219 be added in other threads and no results will be processed. After the 556 added in other threads and no results will be processed. After the fork
220 fork the parent simply leaves the quiescent state and continues 557 the parent simply leaves the quiescent state and continues
221 request/result processing, while the child frees the request/result 558 request/result processing, while the child frees the request/result
222 queue (so that the requests started before the fork will only be handled 559 queue (so that the requests started before the fork will only be handled
223 in the parent). Threads will be started on demand until the limit set in 560 in the parent). Threads will be started on demand until the limit set in
224 the parent process has been reached again. 561 the parent process has been reached again.
225 562
226 In short: the parent will, after a short pause, continue as if fork had 563 In short: the parent will, after a short pause, continue as if fork had
227 not been called, while the child will act as if IO::AIO has not been 564 not been called, while the child will act as if BDB has not been used
228 used yet. 565 yet.
566
567 Win32 note: there is no fork on win32, and perls emulation of it is too
568 broken to be supported, so do not use BDB in a windows pseudo-fork,
569 better yet, switch to a more capable platform.
229 570
230 MEMORY USAGE 571 MEMORY USAGE
231 Per-request usage: 572 Per-request usage:
232 573
233 Each aio request uses - depending on your architecture - around 100-200 574 Each aio request uses - depending on your architecture - around 100-200
234 bytes of memory. In addition, stat requests need a stat buffer (possibly 575 bytes of memory. In addition, stat requests need a stat buffer (possibly
235 a few hundred bytes), readdir requires a result buffer and so on. Perl 576 a few hundred bytes), readdir requires a result buffer and so on. Perl
236 scalars and other data passed into aio requests will also be locked and 577 scalars and other data passed into aio requests will also be locked and
237 will consume memory till the request has entered the done state. 578 will consume memory till the request has entered the done state.
238 579
239 This is now awfully much, so queuing lots of requests is not usually a 580 This is not awfully much, so queuing lots of requests is not usually a
240 problem. 581 problem.
241 582
242 Per-thread usage: 583 Per-thread usage:
243 584
244 In the execution phase, some aio requests require more memory for 585 In the execution phase, some aio requests require more memory for
245 temporary buffers, and each thread requires a stack and other data 586 temporary buffers, and each thread requires a stack and other data
246 structures (usually around 16k-128k, depending on the OS). 587 structures (usually around 16k-128k, depending on the OS).
247 588
248KNOWN BUGS 589KNOWN BUGS
249 Known bugs will be fixed in the next release. 590 Known bugs will be fixed in the next release, except:
591
592 If you use a transaction in any request, and the request returns
593 with an operating system error or DB_LOCK_NOTGRANTED, the internal
594 TXN_DEADLOCK flag will be set on the transaction. See C<db_txn_finish>,
595 above.
250 596
251SEE ALSO 597SEE ALSO
252 Coro::AIO. 598 Coro::BDB, IO::AIO.
253 599
254AUTHOR 600AUTHOR
255 Marc Lehmann <schmorp@schmorp.de> 601 Marc Lehmann <schmorp@schmorp.de>
256 http://home.schmorp.de/ 602 http://home.schmorp.de/
257 603

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines