BDB/BDB.pm

=head1 NAME

BDB - Asynchronous Berkeley DB access

=head1 SYNOPSIS

 use BDB;

=head1 DESCRIPTION

See the eg/ directory in the distribution and the berkeleydb C
documentation. This is inadequate, but the only sources of documentation
known for this module so far.

=head2 EXAMPLE

=head1 REQUEST ANATOMY AND LIFETIME

Every request method creates a request. which is a C data structure not
directly visible to Perl.

During their existance, bdb requests travel through the following states,
in order:

=over 4

=item ready

Immediately after a request is created it is put into the ready state,
waiting for a thread to execute it.

=item execute

A thread has accepted the request for processing and is currently
executing it (e.g. blocking in read).

=item pending

The request has been executed and is waiting for result processing.

While request submission and execution is fully asynchronous, result
processing is not and relies on the perl interpreter calling C<poll_cb>
(or another function with the same effect).

=item result

The request results are processed synchronously by C<poll_cb>.

The C<poll_cb> function will process all outstanding aio requests by
calling their callbacks, freeing memory associated with them and managing
any groups they are contained in.

=item done

Request has reached the end of its lifetime and holds no resources anymore
(except possibly for the Perl object, but its connection to the actual
aio request is severed and calling its methods will either do nothing or
result in a runtime error).

=back

=cut

package BDB;

no warnings;
use strict 'vars';

use base 'Exporter';

BEGIN {
   our $VERSION = '0.5';

   our @BDB_REQ = qw(
      db_env_open db_env_close db_env_txn_checkpoint db_env_lock_detect
      db_env_memp_sync db_env_memp_trickle
      db_open db_close db_compact db_sync db_put db_get db_pget db_del db_key_range
      db_txn_commit db_txn_abort
      db_c_close db_c_count db_c_put db_c_get db_c_pget db_c_del
      db_sequence_open db_sequence_close
      db_sequence_get db_sequence_remove
   );
   our @EXPORT = (@BDB_REQ, qw(dbreq_pri dbreq_nice db_env_create db_create));
   our @EXPORT_OK = qw(
      poll_fileno poll_cb poll_wait flush
      min_parallel max_parallel max_idle
      nreqs nready npending nthreads
      max_poll_time max_poll_reqs
   );

   require XSLoader;
   XSLoader::load ("BDB", $VERSION);
}

=head2 SUPPORT FUNCTIONS

=head3 EVENT PROCESSING AND EVENT LOOP INTEGRATION

=over 4

=item $fileno = BDB::poll_fileno

Return the I<request result pipe file descriptor>. This filehandle must be
polled for reading by some mechanism outside this module (e.g. Event or
select, see below or the SYNOPSIS). If the pipe becomes readable you have
to call C<poll_cb> to check the results.

See C<poll_cb> for an example.

=item BDB::poll_cb

Process some outstanding events on the result pipe. You have to call this
regularly. Returns the number of events processed. Returns immediately
when no events are outstanding. The amount of events processed depends on
the settings of C<BDB::max_poll_req> and C<BDB::max_poll_time>.

If not all requests were processed for whatever reason, the filehandle
will still be ready when C<poll_cb> returns.

Example: Install an Event watcher that automatically calls
BDB::poll_cb with high priority:

   Event->io (fd => BDB::poll_fileno,
              poll => 'r', async => 1,
              cb => \&BDB::poll_cb);

=item BDB::max_poll_reqs $nreqs

=item BDB::max_poll_time $seconds

These set the maximum number of requests (default C<0>, meaning infinity)
that are being processed by C<BDB::poll_cb> in one call, respectively
the maximum amount of time (default C<0>, meaning infinity) spent in
C<BDB::poll_cb> to process requests (more correctly the mininum amount
of time C<poll_cb> is allowed to use).

Setting C<max_poll_time> to a non-zero value creates an overhead of one
syscall per request processed, which is not normally a problem unless your
callbacks are really really fast or your OS is really really slow (I am
not mentioning Solaris here). Using C<max_poll_reqs> incurs no overhead.

Setting these is useful if you want to ensure some level of
interactiveness when perl is not fast enough to process all requests in
time.

For interactive programs, values such as C<0.01> to C<0.1> should be fine.

Example: Install an Event watcher that automatically calls
BDB::poll_cb with low priority, to ensure that other parts of the
program get the CPU sometimes even under high AIO load.

   # try not to spend much more than 0.1s in poll_cb
   BDB::max_poll_time 0.1;

   # use a low priority so other tasks have priority
   Event->io (fd => BDB::poll_fileno,
              poll => 'r', nice => 1,
              cb => &BDB::poll_cb);

=item BDB::poll_wait

If there are any outstanding requests and none of them in the result
phase, wait till the result filehandle becomes ready for reading (simply
does a C<select> on the filehandle. This is useful if you want to
synchronously wait for some requests to finish).

See C<nreqs> for an example.

=item BDB::poll

Waits until some requests have been handled.

Returns the number of requests processed, but is otherwise strictly
equivalent to:

   BDB::poll_wait, BDB::poll_cb

=item BDB::flush

Wait till all outstanding AIO requests have been handled.

Strictly equivalent to:

   BDB::poll_wait, BDB::poll_cb
      while BDB::nreqs;

=back

=head3 CONTROLLING THE NUMBER OF THREADS

=over 4

=item BDB::min_parallel $nthreads

Set the minimum number of AIO threads to C<$nthreads>. The current
default is C<8>, which means eight asynchronous operations can execute
concurrently at any one time (the number of outstanding requests,
however, is unlimited).

BDB starts threads only on demand, when an AIO request is queued and
no free thread exists. Please note that queueing up a hundred requests can
create demand for a hundred threads, even if it turns out that everything
is in the cache and could have been processed faster by a single thread.

It is recommended to keep the number of threads relatively low, as some
Linux kernel versions will scale negatively with the number of threads
(higher parallelity => MUCH higher latency). With current Linux 2.6
versions, 4-32 threads should be fine.

Under most circumstances you don't need to call this function, as the
module selects a default that is suitable for low to moderate load.

=item BDB::max_parallel $nthreads

Sets the maximum number of AIO threads to C<$nthreads>. If more than the
specified number of threads are currently running, this function kills
them. This function blocks until the limit is reached.

While C<$nthreads> are zero, aio requests get queued but not executed
until the number of threads has been increased again.

This module automatically runs C<max_parallel 0> at program end, to ensure
that all threads are killed and that there are no outstanding requests.

Under normal circumstances you don't need to call this function.

=item BDB::max_idle $nthreads

Limit the number of threads (default: 4) that are allowed to idle (i.e.,
threads that did not get a request to process within 10 seconds). That
means if a thread becomes idle while C<$nthreads> other threads are also
idle, it will free its resources and exit.

This is useful when you allow a large number of threads (e.g. 100 or 1000)
to allow for extremely high load situations, but want to free resources
under normal circumstances (1000 threads can easily consume 30MB of RAM).

The default is probably ok in most situations, especially if thread
creation is fast. If thread creation is very slow on your system you might
want to use larger values.

=item $oldmaxreqs = BDB::max_outstanding $maxreqs

This is a very bad function to use in interactive programs because it
blocks, and a bad way to reduce concurrency because it is inexact: Better
use an C<aio_group> together with a feed callback.

Sets the maximum number of outstanding requests to C<$nreqs>. If you
to queue up more than this number of requests, the next call to the
C<poll_cb> (and C<poll_some> and other functions calling C<poll_cb>)
function will block until the limit is no longer exceeded.

The default value is very large, so there is no practical limit on the
number of outstanding requests.

You can still queue as many requests as you want. Therefore,
C<max_oustsanding> is mainly useful in simple scripts (with low values) or
as a stop gap to shield against fatal memory overflow (with large values).

=item BDB::set_sync_prepare $cb

Sets a callback that is called whenever a request is created without an
explicit callback. It has to return two code references. The first is used
as the request callback, and the second is called to wait until the first
callback has been called. The default implementation works like this:

   sub {
      my $status;
      (
         sub { $status = $! },
         sub { BDB::poll while !defined $status; $! = $status },
      )
   }

=back

=head3 STATISTICAL INFORMATION

=over 4

=item BDB::nreqs

Returns the number of requests currently in the ready, execute or pending
states (i.e. for which their callback has not been invoked yet).

Example: wait till there are no outstanding requests anymore:

   BDB::poll_wait, BDB::poll_cb
      while BDB::nreqs;

=item BDB::nready

Returns the number of requests currently in the ready state (not yet
executed).

=item BDB::npending

Returns the number of requests currently in the pending state (executed,
but not yet processed by poll_cb).

=back

=cut

set_sync_prepare {
   my $status;
   (
      sub {
         $status = $!;
      },
      sub {
         BDB::poll while !defined $status;
         $! = $status;
      },
   )
};

min_parallel 8;

END { flush }

1;

=head2 FORK BEHAVIOUR

This module should do "the right thing" when the process using it forks:

Before the fork, IO::AIO enters a quiescent state where no requests
can be added in other threads and no results will be processed. After
the fork the parent simply leaves the quiescent state and continues
request/result processing, while the child frees the request/result queue
(so that the requests started before the fork will only be handled in the
parent). Threads will be started on demand until the limit set in the
parent process has been reached again.

In short: the parent will, after a short pause, continue as if fork had
not been called, while the child will act as if IO::AIO has not been used
yet.

=head2 MEMORY USAGE

Per-request usage:

Each aio request uses - depending on your architecture - around 100-200
bytes of memory. In addition, stat requests need a stat buffer (possibly
a few hundred bytes), readdir requires a result buffer and so on. Perl
scalars and other data passed into aio requests will also be locked and
will consume memory till the request has entered the done state.

This is now awfully much, so queuing lots of requests is not usually a
problem.

Per-thread usage:

In the execution phase, some aio requests require more memory for
temporary buffers, and each thread requires a stack and other data
structures (usually around 16k-128k, depending on the OS).

=head1 KNOWN BUGS

Known bugs will be fixed in the next release.

=head1 SEE ALSO

L<Coro::AIO>.

=head1 AUTHOR

 Marc Lehmann <schmorp@schmorp.de>
 http://home.schmorp.de/

=cut

Revision:	1.9
Committed:	Sun Jul 8 14:57:59 2007 UTC (16 years, 10 months ago) by root
Branch:	MAIN
CVS Tags:	rel-0_5
Changes since 1.8:	+1 -1 lines
Log Message:	* empty log message *
#	User	Rev	Content
1	root	1.1	=head1 NAME
2
3	root	1.2	BDB - Asynchronous Berkeley DB access
4	root	1.1
5			=head1 SYNOPSIS
6
7	root	1.2	use BDB;
8	root	1.1
9			=head1 DESCRIPTION
10
11	root	1.7	See the eg/ directory in the distribution and the berkeleydb C
12			documentation. This is inadequate, but the only sources of documentation
13			known for this module so far.
14
15	root	1.1	=head2 EXAMPLE
16
17			=head1 REQUEST ANATOMY AND LIFETIME
18
19			Every request method creates a request. which is a C data structure not
20			directly visible to Perl.
21
22			During their existance, bdb requests travel through the following states,
23			in order:
24
25			=over 4
26
27			=item ready
28
29			Immediately after a request is created it is put into the ready state,
30			waiting for a thread to execute it.
31
32			=item execute
33
34			A thread has accepted the request for processing and is currently
35			executing it (e.g. blocking in read).
36
37			=item pending
38
39			The request has been executed and is waiting for result processing.
40
41			While request submission and execution is fully asynchronous, result
42			processing is not and relies on the perl interpreter calling C<poll_cb>
43			(or another function with the same effect).
44
45			=item result
46
47			The request results are processed synchronously by C<poll_cb>.
48
49			The C<poll_cb> function will process all outstanding aio requests by
50			calling their callbacks, freeing memory associated with them and managing
51			any groups they are contained in.
52
53			=item done
54
55			Request has reached the end of its lifetime and holds no resources anymore
56			(except possibly for the Perl object, but its connection to the actual
57			aio request is severed and calling its methods will either do nothing or
58			result in a runtime error).
59
60			=back
61
62			=cut
63
64	root	1.2	package BDB;
65	root	1.1
66			no warnings;
67			use strict 'vars';
68
69			use base 'Exporter';
70
71			BEGIN {
72	root	1.9	our $VERSION = '0.5';
73	root	1.1
74	root	1.3	our @BDB_REQ = qw(
75	root	1.6	db_env_open db_env_close db_env_txn_checkpoint db_env_lock_detect
76			db_env_memp_sync db_env_memp_trickle
77			db_open db_close db_compact db_sync db_put db_get db_pget db_del db_key_range
78	root	1.4	db_txn_commit db_txn_abort
79	root	1.5	db_c_close db_c_count db_c_put db_c_get db_c_pget db_c_del
80	root	1.6	db_sequence_open db_sequence_close
81			db_sequence_get db_sequence_remove
82			);
83			our @EXPORT = (@BDB_REQ, qw(dbreq_pri dbreq_nice db_env_create db_create));
84			our @EXPORT_OK = qw(
85			poll_fileno poll_cb poll_wait flush
86			min_parallel max_parallel max_idle
87			nreqs nready npending nthreads
88			max_poll_time max_poll_reqs
89	root	1.3	);
90	root	1.1
91			require XSLoader;
92	root	1.2	XSLoader::load ("BDB", $VERSION);
93	root	1.1	}
94
95			=head2 SUPPORT FUNCTIONS
96
97			=head3 EVENT PROCESSING AND EVENT LOOP INTEGRATION
98
99			=over 4
100
101	root	1.2	=item $fileno = BDB::poll_fileno
102	root	1.1
103			Return the I<request result pipe file descriptor>. This filehandle must be
104			polled for reading by some mechanism outside this module (e.g. Event or
105			select, see below or the SYNOPSIS). If the pipe becomes readable you have
106			to call C<poll_cb> to check the results.
107
108			See C<poll_cb> for an example.
109
110	root	1.2	=item BDB::poll_cb
111	root	1.1
112			Process some outstanding events on the result pipe. You have to call this
113			regularly. Returns the number of events processed. Returns immediately
114			when no events are outstanding. The amount of events processed depends on
115	root	1.2	the settings of C<BDB::max_poll_req> and C<BDB::max_poll_time>.
116	root	1.1
117			If not all requests were processed for whatever reason, the filehandle
118			will still be ready when C<poll_cb> returns.
119
120			Example: Install an Event watcher that automatically calls
121	root	1.2	BDB::poll_cb with high priority:
122	root	1.1
123	root	1.2	Event->io (fd => BDB::poll_fileno,
124	root	1.1	poll => 'r', async => 1,
125	root	1.2	cb => \&BDB::poll_cb);
126	root	1.1
127	root	1.2	=item BDB::max_poll_reqs $nreqs
128	root	1.1
129	root	1.2	=item BDB::max_poll_time $seconds
130	root	1.1
131			These set the maximum number of requests (default C<0>, meaning infinity)
132	root	1.2	that are being processed by C<BDB::poll_cb> in one call, respectively
133	root	1.1	the maximum amount of time (default C<0>, meaning infinity) spent in
134	root	1.2	C<BDB::poll_cb> to process requests (more correctly the mininum amount
135	root	1.1	of time C<poll_cb> is allowed to use).
136
137			Setting C<max_poll_time> to a non-zero value creates an overhead of one
138			syscall per request processed, which is not normally a problem unless your
139			callbacks are really really fast or your OS is really really slow (I am
140			not mentioning Solaris here). Using C<max_poll_reqs> incurs no overhead.
141
142			Setting these is useful if you want to ensure some level of
143			interactiveness when perl is not fast enough to process all requests in
144			time.
145
146			For interactive programs, values such as C<0.01> to C<0.1> should be fine.
147
148			Example: Install an Event watcher that automatically calls
149	root	1.2	BDB::poll_cb with low priority, to ensure that other parts of the
150	root	1.1	program get the CPU sometimes even under high AIO load.
151
152			# try not to spend much more than 0.1s in poll_cb
153	root	1.2	BDB::max_poll_time 0.1;
154	root	1.1
155			# use a low priority so other tasks have priority
156	root	1.2	Event->io (fd => BDB::poll_fileno,
157	root	1.1	poll => 'r', nice => 1,
158	root	1.2	cb => &BDB::poll_cb);
159	root	1.1
160	root	1.2	=item BDB::poll_wait
161	root	1.1
162			If there are any outstanding requests and none of them in the result
163			phase, wait till the result filehandle becomes ready for reading (simply
164			does a C<select> on the filehandle. This is useful if you want to
165			synchronously wait for some requests to finish).
166
167			See C<nreqs> for an example.
168
169	root	1.2	=item BDB::poll
170	root	1.1
171			Waits until some requests have been handled.
172
173			Returns the number of requests processed, but is otherwise strictly
174			equivalent to:
175
176	root	1.2	BDB::poll_wait, BDB::poll_cb
177	root	1.1
178	root	1.2	=item BDB::flush
179	root	1.1
180			Wait till all outstanding AIO requests have been handled.
181
182			Strictly equivalent to:
183
184	root	1.2	BDB::poll_wait, BDB::poll_cb
185			while BDB::nreqs;
186	root	1.1
187	root	1.8	=back
188
189	root	1.1	=head3 CONTROLLING THE NUMBER OF THREADS
190
191	root	1.8	=over 4
192
193	root	1.2	=item BDB::min_parallel $nthreads
194	root	1.1
195			Set the minimum number of AIO threads to C<$nthreads>. The current
196			default is C<8>, which means eight asynchronous operations can execute
197			concurrently at any one time (the number of outstanding requests,
198			however, is unlimited).
199
200	root	1.2	BDB starts threads only on demand, when an AIO request is queued and
201	root	1.1	no free thread exists. Please note that queueing up a hundred requests can
202			create demand for a hundred threads, even if it turns out that everything
203			is in the cache and could have been processed faster by a single thread.
204
205			It is recommended to keep the number of threads relatively low, as some
206			Linux kernel versions will scale negatively with the number of threads
207			(higher parallelity => MUCH higher latency). With current Linux 2.6
208			versions, 4-32 threads should be fine.
209
210			Under most circumstances you don't need to call this function, as the
211			module selects a default that is suitable for low to moderate load.
212
213	root	1.2	=item BDB::max_parallel $nthreads
214	root	1.1
215			Sets the maximum number of AIO threads to C<$nthreads>. If more than the
216			specified number of threads are currently running, this function kills
217			them. This function blocks until the limit is reached.
218
219			While C<$nthreads> are zero, aio requests get queued but not executed
220			until the number of threads has been increased again.
221
222			This module automatically runs C<max_parallel 0> at program end, to ensure
223			that all threads are killed and that there are no outstanding requests.
224
225			Under normal circumstances you don't need to call this function.
226
227	root	1.2	=item BDB::max_idle $nthreads
228	root	1.1
229			Limit the number of threads (default: 4) that are allowed to idle (i.e.,
230			threads that did not get a request to process within 10 seconds). That
231			means if a thread becomes idle while C<$nthreads> other threads are also
232			idle, it will free its resources and exit.
233
234			This is useful when you allow a large number of threads (e.g. 100 or 1000)
235			to allow for extremely high load situations, but want to free resources
236			under normal circumstances (1000 threads can easily consume 30MB of RAM).
237
238			The default is probably ok in most situations, especially if thread
239			creation is fast. If thread creation is very slow on your system you might
240			want to use larger values.
241
242	root	1.2	=item $oldmaxreqs = BDB::max_outstanding $maxreqs
243	root	1.1
244			This is a very bad function to use in interactive programs because it
245			blocks, and a bad way to reduce concurrency because it is inexact: Better
246			use an C<aio_group> together with a feed callback.
247
248			Sets the maximum number of outstanding requests to C<$nreqs>. If you
249			to queue up more than this number of requests, the next call to the
250			C<poll_cb> (and C<poll_some> and other functions calling C<poll_cb>)
251			function will block until the limit is no longer exceeded.
252
253			The default value is very large, so there is no practical limit on the
254			number of outstanding requests.
255
256			You can still queue as many requests as you want. Therefore,
257			C<max_oustsanding> is mainly useful in simple scripts (with low values) or
258			as a stop gap to shield against fatal memory overflow (with large values).
259
260	root	1.3	=item BDB::set_sync_prepare $cb
261
262			Sets a callback that is called whenever a request is created without an
263			explicit callback. It has to return two code references. The first is used
264			as the request callback, and the second is called to wait until the first
265			callback has been called. The default implementation works like this:
266
267			sub {
268			my $status;
269			(
270			sub { $status = $! },
271			sub { BDB::poll while !defined $status; $! = $status },
272			)
273			}
274
275			=back
276
277	root	1.1	=head3 STATISTICAL INFORMATION
278
279	root	1.3	=over 4
280
281	root	1.2	=item BDB::nreqs
282	root	1.1
283			Returns the number of requests currently in the ready, execute or pending
284			states (i.e. for which their callback has not been invoked yet).
285
286			Example: wait till there are no outstanding requests anymore:
287
288	root	1.2	BDB::poll_wait, BDB::poll_cb
289			while BDB::nreqs;
290	root	1.1
291	root	1.2	=item BDB::nready
292	root	1.1
293			Returns the number of requests currently in the ready state (not yet
294			executed).
295
296	root	1.2	=item BDB::npending
297	root	1.1
298			Returns the number of requests currently in the pending state (executed,
299			but not yet processed by poll_cb).
300
301			=back
302
303			=cut
304
305	root	1.3	set_sync_prepare {
306			my $status;
307			(
308			sub {
309			$status = $!;
310			},
311			sub {
312			BDB::poll while !defined $status;
313			$! = $status;
314			},
315			)
316			};
317
318	root	1.1	min_parallel 8;
319
320			END { flush }
321
322			1;
323
324			=head2 FORK BEHAVIOUR
325
326			This module should do "the right thing" when the process using it forks:
327
328			Before the fork, IO::AIO enters a quiescent state where no requests
329			can be added in other threads and no results will be processed. After
330			the fork the parent simply leaves the quiescent state and continues
331			request/result processing, while the child frees the request/result queue
332			(so that the requests started before the fork will only be handled in the
333			parent). Threads will be started on demand until the limit set in the
334			parent process has been reached again.
335
336			In short: the parent will, after a short pause, continue as if fork had
337			not been called, while the child will act as if IO::AIO has not been used
338			yet.
339
340			=head2 MEMORY USAGE
341
342			Per-request usage:
343
344			Each aio request uses - depending on your architecture - around 100-200
345			bytes of memory. In addition, stat requests need a stat buffer (possibly
346			a few hundred bytes), readdir requires a result buffer and so on. Perl
347			scalars and other data passed into aio requests will also be locked and
348			will consume memory till the request has entered the done state.
349
350			This is now awfully much, so queuing lots of requests is not usually a
351			problem.
352
353			Per-thread usage:
354
355			In the execution phase, some aio requests require more memory for
356			temporary buffers, and each thread requires a stack and other data
357			structures (usually around 16k-128k, depending on the OS).
358
359			=head1 KNOWN BUGS
360
361			Known bugs will be fixed in the next release.
362
363			=head1 SEE ALSO
364
365			L<Coro::AIO>.
366
367			=head1 AUTHOR
368
369			Marc Lehmann <schmorp@schmorp.de>
370			http://home.schmorp.de/
371
372			=cut
373