[ViewVC] Diff of: cvs/BDB/BDB.pm

Comparing BDB/BDB.pm (file contents):
Revision 1.1 by root, Mon Feb 5 18:40:55 2007 UTC vs.
Revision 1.8 by root, Wed May 9 06:42:23 2007 UTC

…		…
1	=head1 NAME	1	=head1 NAME
2		2
3	BDB::AIO - Asynchronous Berkeley DB access	3	BDB - Asynchronous Berkeley DB access
4		4
5	=head1 SYNOPSIS	5	=head1 SYNOPSIS
6		6
7	use BDB::AIO;	7	use BDB;
8		8
9	=head1 DESCRIPTION	9	=head1 DESCRIPTION
		10
		11	See the eg/ directory in the distribution and the berkeleydb C
		12	documentation. This is inadequate, but the only sources of documentation
		13	known for this module so far.
10		14
11	=head2 EXAMPLE	15	=head2 EXAMPLE
12		16
13	=head1 REQUEST ANATOMY AND LIFETIME	17	=head1 REQUEST ANATOMY AND LIFETIME
14		18
55		59
56	=back	60	=back
57		61
58	=cut	62	=cut
59		63
60	package BDB::AIO;	64	package BDB;
61		65
62	no warnings;	66	no warnings;
63	use strict 'vars';	67	use strict 'vars';
64		68
65	use base 'Exporter';	69	use base 'Exporter';
66		70
67	BEGIN {	71	BEGIN {
68	our $VERSION = '0.1';	72	our $VERSION = '0.1';
69		73
70	our @BDB_REQ = qw();	74	our @BDB_REQ = qw(
		75	db_env_open db_env_close db_env_txn_checkpoint db_env_lock_detect
		76	db_env_memp_sync db_env_memp_trickle
		77	db_open db_close db_compact db_sync db_put db_get db_pget db_del db_key_range
		78	db_txn_commit db_txn_abort
		79	db_c_close db_c_count db_c_put db_c_get db_c_pget db_c_del
		80	db_sequence_open db_sequence_close
		81	db_sequence_get db_sequence_remove
		82	);
		83	our @EXPORT = (@BDB_REQ, qw(dbreq_pri dbreq_nice db_env_create db_create));
		84	our @EXPORT_OK = qw(
71	our @EXPORT_OK = qw(poll_fileno poll_cb poll_wait flush	85	poll_fileno poll_cb poll_wait flush
72	min_parallel max_parallel max_idle	86	min_parallel max_parallel max_idle
73	nreqs nready npending nthreads	87	nreqs nready npending nthreads
74	max_poll_time max_poll_reqs);	88	max_poll_time max_poll_reqs
		89	);
75		90
76	require XSLoader;	91	require XSLoader;
77	XSLoader::load ("BDB::AIO", $VERSION);	92	XSLoader::load ("BDB", $VERSION);
78	}	93	}
79		94
80	=head2 SUPPORT FUNCTIONS	95	=head2 SUPPORT FUNCTIONS
81		96
82	=head3 EVENT PROCESSING AND EVENT LOOP INTEGRATION	97	=head3 EVENT PROCESSING AND EVENT LOOP INTEGRATION
83		98
84	=over 4	99	=over 4
85		100
86	=item $fileno = BDB::AIO::poll_fileno	101	=item $fileno = BDB::poll_fileno
87		102
88	Return the I<request result pipe file descriptor>. This filehandle must be	103	Return the I<request result pipe file descriptor>. This filehandle must be
89	polled for reading by some mechanism outside this module (e.g. Event or	104	polled for reading by some mechanism outside this module (e.g. Event or
90	select, see below or the SYNOPSIS). If the pipe becomes readable you have	105	select, see below or the SYNOPSIS). If the pipe becomes readable you have
91	to call C<poll_cb> to check the results.	106	to call C<poll_cb> to check the results.
92		107
93	See C<poll_cb> for an example.	108	See C<poll_cb> for an example.
94		109
95	=item BDB::AIO::poll_cb	110	=item BDB::poll_cb
96		111
97	Process some outstanding events on the result pipe. You have to call this	112	Process some outstanding events on the result pipe. You have to call this
98	regularly. Returns the number of events processed. Returns immediately	113	regularly. Returns the number of events processed. Returns immediately
99	when no events are outstanding. The amount of events processed depends on	114	when no events are outstanding. The amount of events processed depends on
100	the settings of C<BDB::AIO::max_poll_req> and C<BDB::AIO::max_poll_time>.	115	the settings of C<BDB::max_poll_req> and C<BDB::max_poll_time>.
101		116
102	If not all requests were processed for whatever reason, the filehandle	117	If not all requests were processed for whatever reason, the filehandle
103	will still be ready when C<poll_cb> returns.	118	will still be ready when C<poll_cb> returns.
104		119
105	Example: Install an Event watcher that automatically calls	120	Example: Install an Event watcher that automatically calls
106	BDB::AIO::poll_cb with high priority:	121	BDB::poll_cb with high priority:
107		122
108	Event->io (fd => BDB::AIO::poll_fileno,	123	Event->io (fd => BDB::poll_fileno,
109	poll => 'r', async => 1,	124	poll => 'r', async => 1,
110	cb => \&BDB::AIO::poll_cb);	125	cb => \&BDB::poll_cb);
111		126
112	=item BDB::AIO::max_poll_reqs $nreqs	127	=item BDB::max_poll_reqs $nreqs
113		128
114	=item BDB::AIO::max_poll_time $seconds	129	=item BDB::max_poll_time $seconds
115		130
116	These set the maximum number of requests (default C<0>, meaning infinity)	131	These set the maximum number of requests (default C<0>, meaning infinity)
117	that are being processed by C<BDB::AIO::poll_cb> in one call, respectively	132	that are being processed by C<BDB::poll_cb> in one call, respectively
118	the maximum amount of time (default C<0>, meaning infinity) spent in	133	the maximum amount of time (default C<0>, meaning infinity) spent in
119	C<BDB::AIO::poll_cb> to process requests (more correctly the mininum amount	134	C<BDB::poll_cb> to process requests (more correctly the mininum amount
120	of time C<poll_cb> is allowed to use).	135	of time C<poll_cb> is allowed to use).
121		136
122	Setting C<max_poll_time> to a non-zero value creates an overhead of one	137	Setting C<max_poll_time> to a non-zero value creates an overhead of one
123	syscall per request processed, which is not normally a problem unless your	138	syscall per request processed, which is not normally a problem unless your
124	callbacks are really really fast or your OS is really really slow (I am	139	callbacks are really really fast or your OS is really really slow (I am
…		…
129	time.	144	time.
130		145
131	For interactive programs, values such as C<0.01> to C<0.1> should be fine.	146	For interactive programs, values such as C<0.01> to C<0.1> should be fine.
132		147
133	Example: Install an Event watcher that automatically calls	148	Example: Install an Event watcher that automatically calls
134	BDB::AIO::poll_cb with low priority, to ensure that other parts of the	149	BDB::poll_cb with low priority, to ensure that other parts of the
135	program get the CPU sometimes even under high AIO load.	150	program get the CPU sometimes even under high AIO load.
136		151
137	# try not to spend much more than 0.1s in poll_cb	152	# try not to spend much more than 0.1s in poll_cb
138	BDB::AIO::max_poll_time 0.1;	153	BDB::max_poll_time 0.1;
139		154
140	# use a low priority so other tasks have priority	155	# use a low priority so other tasks have priority
141	Event->io (fd => BDB::AIO::poll_fileno,	156	Event->io (fd => BDB::poll_fileno,
142	poll => 'r', nice => 1,	157	poll => 'r', nice => 1,
143	cb => &BDB::AIO::poll_cb);	158	cb => &BDB::poll_cb);
144		159
145	=item BDB::AIO::poll_wait	160	=item BDB::poll_wait
146		161
147	If there are any outstanding requests and none of them in the result	162	If there are any outstanding requests and none of them in the result
148	phase, wait till the result filehandle becomes ready for reading (simply	163	phase, wait till the result filehandle becomes ready for reading (simply
149	does a C<select> on the filehandle. This is useful if you want to	164	does a C<select> on the filehandle. This is useful if you want to
150	synchronously wait for some requests to finish).	165	synchronously wait for some requests to finish).
151		166
152	See C<nreqs> for an example.	167	See C<nreqs> for an example.
153		168
154	=item BDB::AIO::poll	169	=item BDB::poll
155		170
156	Waits until some requests have been handled.	171	Waits until some requests have been handled.
157		172
158	Returns the number of requests processed, but is otherwise strictly	173	Returns the number of requests processed, but is otherwise strictly
159	equivalent to:	174	equivalent to:
160		175
161	BDB::AIO::poll_wait, BDB::AIO::poll_cb	176	BDB::poll_wait, BDB::poll_cb
162		177
163	=item BDB::AIO::flush	178	=item BDB::flush
164		179
165	Wait till all outstanding AIO requests have been handled.	180	Wait till all outstanding AIO requests have been handled.
166		181
167	Strictly equivalent to:	182	Strictly equivalent to:
168		183
169	BDB::AIO::poll_wait, BDB::AIO::poll_cb	184	BDB::poll_wait, BDB::poll_cb
170	while BDB::AIO::nreqs;	185	while BDB::nreqs;
		186
		187	=back
171		188
172	=head3 CONTROLLING THE NUMBER OF THREADS	189	=head3 CONTROLLING THE NUMBER OF THREADS
173		190
		191	=over 4
		192
174	=item BDB::AIO::min_parallel $nthreads	193	=item BDB::min_parallel $nthreads
175		194
176	Set the minimum number of AIO threads to C<$nthreads>. The current	195	Set the minimum number of AIO threads to C<$nthreads>. The current
177	default is C<8>, which means eight asynchronous operations can execute	196	default is C<8>, which means eight asynchronous operations can execute
178	concurrently at any one time (the number of outstanding requests,	197	concurrently at any one time (the number of outstanding requests,
179	however, is unlimited).	198	however, is unlimited).
180		199
181	BDB::AIO starts threads only on demand, when an AIO request is queued and	200	BDB starts threads only on demand, when an AIO request is queued and
182	no free thread exists. Please note that queueing up a hundred requests can	201	no free thread exists. Please note that queueing up a hundred requests can
183	create demand for a hundred threads, even if it turns out that everything	202	create demand for a hundred threads, even if it turns out that everything
184	is in the cache and could have been processed faster by a single thread.	203	is in the cache and could have been processed faster by a single thread.
185		204
186	It is recommended to keep the number of threads relatively low, as some	205	It is recommended to keep the number of threads relatively low, as some
…		…
189	versions, 4-32 threads should be fine.	208	versions, 4-32 threads should be fine.
190		209
191	Under most circumstances you don't need to call this function, as the	210	Under most circumstances you don't need to call this function, as the
192	module selects a default that is suitable for low to moderate load.	211	module selects a default that is suitable for low to moderate load.
193		212
194	=item BDB::AIO::max_parallel $nthreads	213	=item BDB::max_parallel $nthreads
195		214
196	Sets the maximum number of AIO threads to C<$nthreads>. If more than the	215	Sets the maximum number of AIO threads to C<$nthreads>. If more than the
197	specified number of threads are currently running, this function kills	216	specified number of threads are currently running, this function kills
198	them. This function blocks until the limit is reached.	217	them. This function blocks until the limit is reached.
199		218
…		…
203	This module automatically runs C<max_parallel 0> at program end, to ensure	222	This module automatically runs C<max_parallel 0> at program end, to ensure
204	that all threads are killed and that there are no outstanding requests.	223	that all threads are killed and that there are no outstanding requests.
205		224
206	Under normal circumstances you don't need to call this function.	225	Under normal circumstances you don't need to call this function.
207		226
208	=item BDB::AIO::max_idle $nthreads	227	=item BDB::max_idle $nthreads
209		228
210	Limit the number of threads (default: 4) that are allowed to idle (i.e.,	229	Limit the number of threads (default: 4) that are allowed to idle (i.e.,
211	threads that did not get a request to process within 10 seconds). That	230	threads that did not get a request to process within 10 seconds). That
212	means if a thread becomes idle while C<$nthreads> other threads are also	231	means if a thread becomes idle while C<$nthreads> other threads are also
213	idle, it will free its resources and exit.	232	idle, it will free its resources and exit.
…		…
218		237
219	The default is probably ok in most situations, especially if thread	238	The default is probably ok in most situations, especially if thread
220	creation is fast. If thread creation is very slow on your system you might	239	creation is fast. If thread creation is very slow on your system you might
221	want to use larger values.	240	want to use larger values.
222		241
223	=item $oldmaxreqs = BDB::AIO::max_outstanding $maxreqs	242	=item $oldmaxreqs = BDB::max_outstanding $maxreqs
224		243
225	This is a very bad function to use in interactive programs because it	244	This is a very bad function to use in interactive programs because it
226	blocks, and a bad way to reduce concurrency because it is inexact: Better	245	blocks, and a bad way to reduce concurrency because it is inexact: Better
227	use an C<aio_group> together with a feed callback.	246	use an C<aio_group> together with a feed callback.
228		247
…		…
236		255
237	You can still queue as many requests as you want. Therefore,	256	You can still queue as many requests as you want. Therefore,
238	C<max_oustsanding> is mainly useful in simple scripts (with low values) or	257	C<max_oustsanding> is mainly useful in simple scripts (with low values) or
239	as a stop gap to shield against fatal memory overflow (with large values).	258	as a stop gap to shield against fatal memory overflow (with large values).
240		259
		260	=item BDB::set_sync_prepare $cb
		261
		262	Sets a callback that is called whenever a request is created without an
		263	explicit callback. It has to return two code references. The first is used
		264	as the request callback, and the second is called to wait until the first
		265	callback has been called. The default implementation works like this:
		266
		267	sub {
		268	my $status;
		269	(
		270	sub { $status = $! },
		271	sub { BDB::poll while !defined $status; $! = $status },
		272	)
		273	}
		274
		275	=back
		276
241	=head3 STATISTICAL INFORMATION	277	=head3 STATISTICAL INFORMATION
242		278
		279	=over 4
		280
243	=item BDB::AIO::nreqs	281	=item BDB::nreqs
244		282
245	Returns the number of requests currently in the ready, execute or pending	283	Returns the number of requests currently in the ready, execute or pending
246	states (i.e. for which their callback has not been invoked yet).	284	states (i.e. for which their callback has not been invoked yet).
247		285
248	Example: wait till there are no outstanding requests anymore:	286	Example: wait till there are no outstanding requests anymore:
249		287
250	BDB::AIO::poll_wait, BDB::AIO::poll_cb	288	BDB::poll_wait, BDB::poll_cb
251	while BDB::AIO::nreqs;	289	while BDB::nreqs;
252		290
253	=item BDB::AIO::nready	291	=item BDB::nready
254		292
255	Returns the number of requests currently in the ready state (not yet	293	Returns the number of requests currently in the ready state (not yet
256	executed).	294	executed).
257		295
258	=item BDB::AIO::npending	296	=item BDB::npending
259		297
260	Returns the number of requests currently in the pending state (executed,	298	Returns the number of requests currently in the pending state (executed,
261	but not yet processed by poll_cb).	299	but not yet processed by poll_cb).
262		300
263	=back	301	=back
264		302
265	=cut	303	=cut
266		304
267	# support function to convert a fd into a perl filehandle	305	set_sync_prepare {
268	sub _fd2fh {	306	my $status;
269	return undef if $_[0] < 0;	307	(
270		308	sub {
271	# try to generate nice filehandles	309	$status = $!;
272	my $sym = "BDB::AIO::fd#$_[0]";	310	},
273	local *$sym;	311	sub {
274		312	BDB::poll while !defined $status;
275	open *$sym, "+<&=$_[0]" # usually works under any unix	313	$! = $status;
276	or open *$sym, "<&=$_[0]" # cygwin needs this	314	},
277	or open *$sym, ">&=$_[0]" # or this	315	)
278	or return undef;	316	};
279
280	*$sym
281	}
282		317
283	min_parallel 8;	318	min_parallel 8;
284		319
285	END { flush }	320	END { flush }
286		321

Diff Legend

-–
+Removed lines
-+
+Added lines
-<
+Changed lines
->
+Changed lines

Comparing BDB/BDB.pm (file contents): Revision 1.1 by root, Mon Feb 5 18:40:55 2007 UTC vs. Revision 1.8 by root, Wed May 9 06:42:23 2007 UTC

Diff Legend

Comparing BDB/BDB.pm (file contents):
Revision 1.1 by root, Mon Feb 5 18:40:55 2007 UTC vs.
Revision 1.8 by root, Wed May 9 06:42:23 2007 UTC