cvsroot/TDB_FileX/TDB_FileX.pm

package TDB_FileX;

use common::sense;

use Exporter ();
use XSLoader ();

our @ISA = qw(Exporter);

# Items to export into callers namespace by default. Note: do not export
# names by default without a very good reason. Use EXPORT_OK instead.
# Do not simply export all your public functions/methods/constants.

our %EXPORT_TAGS = (
   flags => [qw(
      ALLOW_NESTING
      BIGENDIAN
      CLEAR_IF_FIRST
      CONVERT
      DEFAULT
      DISALLOW_NESTING
      INCOMPATIBLE_HASH
      INTERNAL
      MUTEX_LOCKING
      NOLOCK
      NOMMAP
      NOSYNC
      SEQNUM
      VOLATILE
   )],
   insert => [qw(
      INSERT
      MODIFY
      REPLACE
   )],
   error => [qw(
      SUCCESS
      ERR_CORRUPT
      ERR_EXISTS
      ERR_IO
      ERR_LOCK
      ERR_LOCK_TIMEOUT
      ERR_NOEXIST
      ERR_NOLOCK
      ERR_OOM
      ERR_EINVAL
      ERR_RDONLY
   )],
   debug => [qw(
      DEBUG_FATAL
      DEBUG_ERROR
      DEBUG_WARNING
      DEBUG_TRACE}
   )],
);

our @EXPORT_OK;

Exporter::export_ok_tags qw(flags insert error debug);

$EXPORT_TAGS{all} = \@EXPORT_OK;

our $VERSION = '0.97';

XSLoader::load __PACKAGE__, $VERSION;

1;
__END__

=head1 NAME

TDB_FileX - Perl access to the trivial database library

=head1 SYNOPSIS

  use TDB_FileX;

  # tie interface
  tie %hash, TDB_FileX => $filename,
     hash_size => 8000,
     mutex     => 1,
  ;
  $hash{key} = 'value';
  while (my ($k, $v) = each %hash) { print "$k -> $v\n" }

  # OO interface
  my $tdb = TDB_FileX->open ($filename, flags => TDB_FileX::CLEAR_IF_FIRST)
     or die $!;

  $tdb->store (key => 'value') or die $tdb->errorstr;
  $tdb->traverse (sub { print "$_[0] -> $_[1]\n" });

=head1 DESCRIPTION

TDB is a simple database similar to GDBM, but allows multiple simultaneous
writers. It's main drawback is the need to manually configure a hash table
size in advance - see the C<hash_size> option for C<open>.

TDB_FileX provides a simple C<tie> interface, similar to DB_File and
friends; and an object-oriented interface, which provides access to all of
the functions in the tdb library.

=head2 FUNCTIONS

=over 4

=item TDB_FileX->open ($path[, key => value...])

TDB_FileX constructor. Opens $path and returns a TDB_FileX object. The
same arguments may be passed to the C<tie> function. On error, C<$!>
is set and C<undef> is returned.

A number of key-value pairs are accepted, with hopefully sensible defaults
for most of these (but you should consider at least setting C<hash_size>
if you want to store more than a thousand or so pairs, And C<mutex> if you
want to take advantage of much faster locking.

=over

=item tdb_flags => $flags (default: C<TDB_FileX::DEFAULT>)

A set of flags that influence the behaviour and format of the database.

See tdb_open(3) for the meanings and possible values (note that the
C<TDB_> prefix has to be removed, see the L<EXPORTS> section for a list.

=item open_flags => $flags (default: C<Fcntl::O_RDWR | Fcntl::O_CREAT>)

Standard open flags, as used in C<sysopen>.

=item mode => $mode (default: C<0666>)

Standard file open mode, as use din C<sysopen>. Only used when creating
the database file.

=item hash_size => $int (default: internal to libtdb, but normally C<131>)

The size of the internal hash table - only used when creating the
database. The default is usually very low and only good for a few thousand
keys. As a rule of thumb, it should be at least one percent of the number
of keys you plan to store, e.g. for C<800000> keys you should use around a
size of C<8000>.

To give you an idea of the performance, I inserted about 600000 records, 3GB of data,
into a TDB file:

    size time
     131 160s
     256  85s
    1024  12s
    4096   5s
    8192   4s

As you can see, the default hash table size for this case caused it to use
40 times the time to insert than a larger hash table, the optimum being
around 75 ke4ys per hash entry).

=item log_cb => $cb->($level, $msg)

Setsb a code reference that is called with a message and a log level
(lower means more important, there are C<DEBUG_FATAL>, C<DEBUG_ERROR>,
C<DEBUG_WARNING> and C<DEBUG_TRACE}>). Unlike the "debug" in the name
might indicate, if you want to find out why, for instance, you could not
open a database, you need to use a logging callback.

=item hash => $hash (default: C<undef>)

Selects a hash function to use.

=over

=item C<"default"> or C<undef>

Use autodetection and use either C<"jenkins"> or the original tdb hash function.

=item C<"jenkins">

The jenkins hash function. Relatively fast, and recommended for modern tdb databases
that need to be interoperable between implementations.

=item C<"fnv1ax">

The FNV1-A hash with 32 bit post-mixing. Pretty good and verxy fast for
keys up to 10-20 octets.

=item C<"xxh3">

The XXH3 hash. Very good and fast especially for long keys.

=item C<1> .. C<4>

Specify one of up to four custom hash functions (see L<set_hash_function>).

=back

=item mutex => $bool (default: C<0>)

TDB can take advantage of fast interprocess mutexes, which
can be orders of magnitude faster than the syscall-based
locking used by default. Normally, you need to call
C<TDB_FileX::runtime_check_for_robust_mutexes> and set the
C<MUTEX_LOCKING> flag if support is indicated.

This option, when enabled, enables C<MUTEX_LOCKING> if it is supported
by the platform - which involves a fork when opening the database. When
disabled, it will remove the flag.

It is recommended to keep this one, but enabling this changes the format
of the database, so might not be an option if interoperability with other
programs is required.

=back

There is no explicit close function. The database is closed implicitly
when there are no remaining references.

=item $tdb->store ($key, $value[, $flag=REPLACE])

Store $value in the database with key w$key. The $flag defaults to
C<REPLACE>, but can also be C<INSERT> or C<MODIFY>, see tdb_store(3) for
details.

On failure, a perl false is returned. See L</$tdb->error> and
L</$tdb->errorstr> for the reason.

=item $tdb->fetch ($key)

Fetch the value associated with $key, or C<undef> if it is not found.

=item $tdb->delete ($key)

Delete the value associated with $key.

On failure, a perl false is returned. See L</$tdb->error> and
L</$tdb->errorstr> for the reason.

=item $tdb->exists ($key)

Return true if the $key is found, false otherwise.

=item $tdb->firstkey

Return the key of the first value in the database. Returns C<undef> on
failure or if there are no keys in the database. See tdb_firstkey(3)
for details.

=item $tdb->nextkey ($lastkey)

Return the next key in the database after $lastkey. Returns C<undef> on
failure or if there are no more keys in the database. See tdb_nextkey(3)
for details.

=item $tdb->error

Returns the current error state of the C<$tdb> object. See the list of
error codes given in L<EXPORTS>.

=item $tdb->errorstr

Returns a printable string that describes the error state of the
database.

=item $tdb->reopen

Closes and reopens the database. Required after a
L<fork|perlfunc/fork>, if both processes wish to use the database.

B<NB:> If C<reopen> fails (returns false), then it is unsafe to call
any further methods on C<$tdb>. Thus, the only way to find out I<why>
C<reopen> failed is to use a logging function.

=item TDB_FileX::reopen_all

Closes and reopens all open databases. See L</$tdb->reopen>.

B<NB:> If C<reopen_all> fails (returns false), there is no indication
of I<which> C<$tdb> objects failed or why. If you have to survive
failures, you may wish to do your own C<reopen> loop instead.

=item $tdb->traverse ($cb->()) #TODO

Call $cb for each entry in the database.

The callback is called with the key and value as arguments and should return a
true value if you wish to continue traversal.

C<traverse> returns the number of elements traversed or C<undef> on
error. If SUB is C<undef>, then this function simply counts the number
of elements.

=item $tdb->set_logging_function(SUB)

Set the logging function to use when this database object encounters
errors. SUB should be a coderef or a string giving the name of a
function.

SUB is called with the severity level (an integer) and the message (a
string).

=item $tdb->lockall

Lock an entire database, returning false on error.

=item $tdb->unlockall

Unlock an entire database previously locked with
L</$tdb->lockall>.

=item $tdb->dump_all

Dump the records and freelist to STDOUT in an almost human readable
form.

=item $tdb->printfreelist

Dump the freelist to STDOUT.

=back

=head2 EXPORTS

Nothing constants are exported by default.

The tag C<:all> exports allpo of the constants.

Individually or with the tag C<:flags>:

  DEFAULT
  CLEAR_IF_FIRST
  INTERNAL
  NOLOCK
  NOMMAP
  CONVERT
  BIGENDIAN
  NOSYNC
  SEQNUM
  VOLATILE
  ALLOW_NESTING
  DISALLOW_NESTING
  INCOMPATIBLE_HASH
  MUTEX_LOCKING

Individually or with the tag C<:insert>:

  REPLACE
  INSERT
  MODIFY

Individually or with the tag C<:error>:

  SUCCESS
  ERR_CORRUPT
  ERR_IO
  ERR_LOCK
  ERR_OOM
  ERR_EXISTS
  ERR_NOLOCK
  ERR_LOCK_TIMEOUT
  ERR_NOEXIST
  ERR_EINVAL
  ERR_RDONLY

Individually or with the tag C<:debug>:

  DEBUG_FATAL
  DEBUG_ERROR
  DEBUG_WARNING
  DEBUG_TRACE

=head1 BUGS

There is no way to survive an error during C<reopen_all>.
Unfortunately this is a limitation in the TDB C API.

Currently the hash functions are set globally. This is due to a limitation
in the TDB C API.

=head1 SEE ALSO

tdb(3), L<perltie>.

=head1 AUTHOR

Angus Lees, E<lt>gus@inodes.org>

Currently maintained by Marc A. Lehmann <schmorp@schmorp.de>
http://home.schmorp.de/

=cut
Revision:	1.1
Committed:	Fri Apr 25 16:43:02 2025 UTC (16 months, 3 weeks ago) by root
Branch:	MAIN
Log Message:	* empty log message *
#	Content
1	package TDB_FileX;
2
3	use common::sense;
4
5	use Exporter ();
6	use XSLoader ();
7
8	our @ISA = qw(Exporter);
9
10	# Items to export into callers namespace by default. Note: do not export
11	# names by default without a very good reason. Use EXPORT_OK instead.
12	# Do not simply export all your public functions/methods/constants.
13
14	our %EXPORT_TAGS = (
15	flags => [qw(
16	ALLOW_NESTING
17	BIGENDIAN
18	CLEAR_IF_FIRST
19	CONVERT
20	DEFAULT
21	DISALLOW_NESTING
22	INCOMPATIBLE_HASH
23	INTERNAL
24	MUTEX_LOCKING
25	NOLOCK
26	NOMMAP
27	NOSYNC
28	SEQNUM
29	VOLATILE
30	)],
31	insert => [qw(
32	INSERT
33	MODIFY
34	REPLACE
35	)],
36	error => [qw(
37	SUCCESS
38	ERR_CORRUPT
39	ERR_EXISTS
40	ERR_IO
41	ERR_LOCK
42	ERR_LOCK_TIMEOUT
43	ERR_NOEXIST
44	ERR_NOLOCK
45	ERR_OOM
46	ERR_EINVAL
47	ERR_RDONLY
48	)],
49	debug => [qw(
50	DEBUG_FATAL
51	DEBUG_ERROR
52	DEBUG_WARNING
53	DEBUG_TRACE}
54	)],
55	);
56
57	our @EXPORT_OK;
58
59	Exporter::export_ok_tags qw(flags insert error debug);
60
61	$EXPORT_TAGS{all} = \@EXPORT_OK;
62
63	our $VERSION = '0.97';
64
65	XSLoader::load __PACKAGE__, $VERSION;
66
67	1;
68	__END__
69
70	=head1 NAME
71
72	TDB_FileX - Perl access to the trivial database library
73
74	=head1 SYNOPSIS
75
76	use TDB_FileX;
77
78	# tie interface
79	tie %hash, TDB_FileX => $filename,
80	hash_size => 8000,
81	mutex => 1,
82	;
83	$hash{key} = 'value';
84	while (my ($k, $v) = each %hash) { print "$k -> $v\n" }
85
86	# OO interface
87	my $tdb = TDB_FileX->open ($filename, flags => TDB_FileX::CLEAR_IF_FIRST)
88	or die $!;
89
90	$tdb->store (key => 'value') or die $tdb->errorstr;
91	$tdb->traverse (sub { print "$_[0] -> $_[1]\n" });
92
93	=head1 DESCRIPTION
94
95	TDB is a simple database similar to GDBM, but allows multiple simultaneous
96	writers. It's main drawback is the need to manually configure a hash table
97	size in advance - see the C<hash_size> option for C<open>.
98
99	TDB_FileX provides a simple C<tie> interface, similar to DB_File and
100	friends; and an object-oriented interface, which provides access to all of
101	the functions in the tdb library.
102
103	=head2 FUNCTIONS
104
105	=over 4
106
107	=item TDB_FileX->open ($path[, key => value...])
108
109	TDB_FileX constructor. Opens $path and returns a TDB_FileX object. The
110	same arguments may be passed to the C<tie> function. On error, C<$!>
111	is set and C<undef> is returned.
112
113	A number of key-value pairs are accepted, with hopefully sensible defaults
114	for most of these (but you should consider at least setting C<hash_size>
115	if you want to store more than a thousand or so pairs, And C<mutex> if you
116	want to take advantage of much faster locking.
117
118	=over
119
120	=item tdb_flags => $flags (default: C<TDB_FileX::DEFAULT>)
121
122	A set of flags that influence the behaviour and format of the database.
123
124	See tdb_open(3) for the meanings and possible values (note that the
125	C<TDB_> prefix has to be removed, see the L<EXPORTS> section for a list.
126
127	=item open_flags => $flags (default: C<Fcntl::O_RDWR \| Fcntl::O_CREAT>)
128
129	Standard open flags, as used in C<sysopen>.
130
131	=item mode => $mode (default: C<0666>)
132
133	Standard file open mode, as use din C<sysopen>. Only used when creating
134	the database file.
135
136	=item hash_size => $int (default: internal to libtdb, but normally C<131>)
137
138	The size of the internal hash table - only used when creating the
139	database. The default is usually very low and only good for a few thousand
140	keys. As a rule of thumb, it should be at least one percent of the number
141	of keys you plan to store, e.g. for C<800000> keys you should use around a
142	size of C<8000>.
143
144	To give you an idea of the performance, I inserted about 600000 records, 3GB of data,
145	into a TDB file:
146
147	size time
148	131 160s
149	256 85s
150	1024 12s
151	4096 5s
152	8192 4s
153
154	As you can see, the default hash table size for this case caused it to use
155	40 times the time to insert than a larger hash table, the optimum being
156	around 75 ke4ys per hash entry).
157
158	=item log_cb => $cb->($level, $msg)
159
160	Setsb a code reference that is called with a message and a log level
161	(lower means more important, there are C<DEBUG_FATAL>, C<DEBUG_ERROR>,
162	C<DEBUG_WARNING> and C<DEBUG_TRACE}>). Unlike the "debug" in the name
163	might indicate, if you want to find out why, for instance, you could not
164	open a database, you need to use a logging callback.
165
166	=item hash => $hash (default: C<undef>)
167
168	Selects a hash function to use.
169
170	=over
171
172	=item C<"default"> or C<undef>
173
174	Use autodetection and use either C<"jenkins"> or the original tdb hash function.
175
176	=item C<"jenkins">
177
178	The jenkins hash function. Relatively fast, and recommended for modern tdb databases
179	that need to be interoperable between implementations.
180
181	=item C<"fnv1ax">
182
183	The FNV1-A hash with 32 bit post-mixing. Pretty good and verxy fast for
184	keys up to 10-20 octets.
185
186	=item C<"xxh3">
187
188	The XXH3 hash. Very good and fast especially for long keys.
189
190	=item C<1> .. C<4>
191
192	Specify one of up to four custom hash functions (see L<set_hash_function>).
193
194	=back
195
196	=item mutex => $bool (default: C<0>)
197
198	TDB can take advantage of fast interprocess mutexes, which
199	can be orders of magnitude faster than the syscall-based
200	locking used by default. Normally, you need to call
201	C<TDB_FileX::runtime_check_for_robust_mutexes> and set the
202	C<MUTEX_LOCKING> flag if support is indicated.
203
204	This option, when enabled, enables C<MUTEX_LOCKING> if it is supported
205	by the platform - which involves a fork when opening the database. When
206	disabled, it will remove the flag.
207
208	It is recommended to keep this one, but enabling this changes the format
209	of the database, so might not be an option if interoperability with other
210	programs is required.
211
212	=back
213
214	There is no explicit close function. The database is closed implicitly
215	when there are no remaining references.
216
217	=item $tdb->store ($key, $value[, $flag=REPLACE])
218
219	Store $value in the database with key w$key. The $flag defaults to
220	C<REPLACE>, but can also be C<INSERT> or C<MODIFY>, see tdb_store(3) for
221	details.
222
223	On failure, a perl false is returned. See L</$tdb->error> and
224	L</$tdb->errorstr> for the reason.
225
226	=item $tdb->fetch ($key)
227
228	Fetch the value associated with $key, or C<undef> if it is not found.
229
230	=item $tdb->delete ($key)
231
232	Delete the value associated with $key.
233
234	On failure, a perl false is returned. See L</$tdb->error> and
235	L</$tdb->errorstr> for the reason.
236
237	=item $tdb->exists ($key)
238
239	Return true if the $key is found, false otherwise.
240
241	=item $tdb->firstkey
242
243	Return the key of the first value in the database. Returns C<undef> on
244	failure or if there are no keys in the database. See tdb_firstkey(3)
245	for details.
246
247	=item $tdb->nextkey ($lastkey)
248
249	Return the next key in the database after $lastkey. Returns C<undef> on
250	failure or if there are no more keys in the database. See tdb_nextkey(3)
251	for details.
252
253	=item $tdb->error
254
255	Returns the current error state of the C<$tdb> object. See the list of
256	error codes given in L<EXPORTS>.
257
258	=item $tdb->errorstr
259
260	Returns a printable string that describes the error state of the
261	database.
262
263	=item $tdb->reopen
264
265	Closes and reopens the database. Required after a
266	L<fork\|perlfunc/fork>, if both processes wish to use the database.
267
268	B<NB:> If C<reopen> fails (returns false), then it is unsafe to call
269	any further methods on C<$tdb>. Thus, the only way to find out I<why>
270	C<reopen> failed is to use a logging function.
271
272	=item TDB_FileX::reopen_all
273
274	Closes and reopens all open databases. See L</$tdb->reopen>.
275
276	B<NB:> If C<reopen_all> fails (returns false), there is no indication
277	of I<which> C<$tdb> objects failed or why. If you have to survive
278	failures, you may wish to do your own C<reopen> loop instead.
279
280	=item $tdb->traverse ($cb->()) #TODO
281
282	Call $cb for each entry in the database.
283
284	The callback is called with the key and value as arguments and should return a
285	true value if you wish to continue traversal.
286
287	C<traverse> returns the number of elements traversed or C<undef> on
288	error. If SUB is C<undef>, then this function simply counts the number
289	of elements.
290
291	=item $tdb->set_logging_function(SUB)
292
293	Set the logging function to use when this database object encounters
294	errors. SUB should be a coderef or a string giving the name of a
295	function.
296
297	SUB is called with the severity level (an integer) and the message (a
298	string).
299
300	=item $tdb->lockall
301
302	Lock an entire database, returning false on error.
303
304	=item $tdb->unlockall
305
306	Unlock an entire database previously locked with
307	L</$tdb->lockall>.
308
309	=item $tdb->dump_all
310
311	Dump the records and freelist to STDOUT in an almost human readable
312	form.
313
314	=item $tdb->printfreelist
315
316	Dump the freelist to STDOUT.
317
318	=back
319
320	=head2 EXPORTS
321
322	Nothing constants are exported by default.
323
324	The tag C<:all> exports allpo of the constants.
325
326	Individually or with the tag C<:flags>:
327
328	DEFAULT
329	CLEAR_IF_FIRST
330	INTERNAL
331	NOLOCK
332	NOMMAP
333	CONVERT
334	BIGENDIAN
335	NOSYNC
336	SEQNUM
337	VOLATILE
338	ALLOW_NESTING
339	DISALLOW_NESTING
340	INCOMPATIBLE_HASH
341	MUTEX_LOCKING
342
343	Individually or with the tag C<:insert>:
344
345	REPLACE
346	INSERT
347	MODIFY
348
349	Individually or with the tag C<:error>:
350
351	SUCCESS
352	ERR_CORRUPT
353	ERR_IO
354	ERR_LOCK
355	ERR_OOM
356	ERR_EXISTS
357	ERR_NOLOCK
358	ERR_LOCK_TIMEOUT
359	ERR_NOEXIST
360	ERR_EINVAL
361	ERR_RDONLY
362
363	Individually or with the tag C<:debug>:
364
365	DEBUG_FATAL
366	DEBUG_ERROR
367	DEBUG_WARNING
368	DEBUG_TRACE
369
370	=head1 BUGS
371
372	There is no way to survive an error during C<reopen_all>.
373	Unfortunately this is a limitation in the TDB C API.
374
375	Currently the hash functions are set globally. This is due to a limitation
376	in the TDB C API.
377
378	=head1 SEE ALSO
379
380	tdb(3), L<perltie>.
381
382	=head1 AUTHOR
383
384	Angus Lees, E<lt>gus@inodes.org>
385
386	Currently maintained by Marc A. Lehmann <schmorp@schmorp.de>
387	http://home.schmorp.de/
388
389	=cut