cvsroot/TDB_FileX/TDB_FileX.pm

package TDB_FileX;

use common::sense;

use Exporter ();
use XSLoader ();

our @ISA = qw(Exporter);

# Items to export into callers namespace by default. Note: do not export
# names by default without a very good reason. Use EXPORT_OK instead.
# Do not simply export all your public functions/methods/constants.

our %EXPORT_TAGS = (
   flags => [qw(
      ALLOW_NESTING
      BIGENDIAN
      CLEAR_IF_FIRST
      CONVERT
      DEFAULT
      DISALLOW_NESTING
      INCOMPATIBLE_HASH
      INTERNAL
      MUTEX_LOCKING
      NOLOCK
      NOMMAP
      NOSYNC
      SEQNUM
      VOLATILE
   )],
   insert => [qw(
      INSERT
      MODIFY
      REPLACE
   )],
   error => [qw(
      SUCCESS
      ERR_CORRUPT
      ERR_EXISTS
      ERR_IO
      ERR_LOCK
      ERR_LOCK_TIMEOUT
      ERR_NOEXIST
      ERR_NOLOCK
      ERR_OOM
      ERR_EINVAL
      ERR_RDONLY
   )],
   debug => [qw(
      DEBUG_FATAL
      DEBUG_ERROR
      DEBUG_WARNING
      DEBUG_TRACE}
   )],
);

our @EXPORT_OK;

Exporter::export_ok_tags qw(flags insert error debug);

$EXPORT_TAGS{all} = \@EXPORT_OK;

our $VERSION = '0.97';

XSLoader::load __PACKAGE__, $VERSION;

1;
__END__

=head1 NAME

TDB_FileX - Perl access to the trivial database library

=head1 SYNOPSIS

  use TDB_FileX;

  # tie interface
  tie %hash, TDB_FileX => $filename,
     hash_size => 8000,
     mutex     => 1,
  ;
  $hash{key} = 'value';
  while (my ($k, $v) = each %hash) { print "$k -> $v\n" }

  # OO interface
  my $tdb = TDB_FileX->open ($filename, flags => TDB_FileX::CLEAR_IF_FIRST)
     or die $!;

  $tdb->store (key => 'value') or die $tdb->errorstr;
  $tdb->traverse (sub { print "$_[0] -> $_[1]\n" });

=head1 DESCRIPTION

TDB is a simple database similar to GDBM, but allows multiple simultaneous
writers. It's main drawback is the need to manually configure a hash table
size in advance - see the C<hash_size> option for C<open>.

TDB_FileX provides a simple C<tie> interface, similar to DB_File and
friends; and an object-oriented interface, which provides access to all of
the functions in the tdb library.

=head2 FUNCTIONS

=over 4

=item TDB_FileX->open ($path[, key => value...])

TDB_FileX constructor. Opens $path and returns a TDB_FileX object. The
same arguments may be passed to the C<tie> function. On error, C<$!>
is set and C<undef> is returned.

A number of key-value pairs are accepted, with hopefully sensible defaults
for most of these (but you should consider at least setting C<hash_size>
if you want to store more than a thousand or so pairs, And C<mutex> if you
want to take advantage of much faster locking.

=over

=item tdb_flags => $flags (default: C<TDB_FileX::DEFAULT>)

A set of flags that influence the behaviour and format of the database.

See tdb_open(3) for the meanings and possible values (note that the
C<TDB_> prefix has to be removed, see the L<EXPORTS> section for a list.

=item open_flags => $flags (default: C<Fcntl::O_RDWR | Fcntl::O_CREAT>)

Standard open flags, as used in C<sysopen>.

=item mode => $mode (default: C<0666>)

Standard file open mode, as use din C<sysopen>. Only used when creating
the database file.

=item hash_size => $int (default: internal to libtdb, but normally C<131>)

The size of the internal hash table - only used when creating the
database. The default is usually very low and only good for a few thousand
keys. As a rule of thumb, it should be at least one percent of the number
of keys you plan to store, e.g. for C<800000> keys you should use around a
size of C<8000>.

To give you an idea of the performance, I inserted about 600000 records, 3GB of data,
into a TDB file:

    size time
     131 160s
     256  85s
    1024  12s
    4096   5s
    8192   4s

As you can see, the default hash table size for this case caused it to use
40 times the time to insert than a larger hash table, the optimum being
around 75 ke4ys per hash entry).

=item log_cb => $cb->($level, $msg)

Setsb a code reference that is called with a message and a log level
(lower means more important, there are C<DEBUG_FATAL>, C<DEBUG_ERROR>,
C<DEBUG_WARNING> and C<DEBUG_TRACE}>). Unlike the "debug" in the name
might indicate, if you want to find out why, for instance, you could not
open a database, you need to use a logging callback.

=item hash => $hash (default: C<undef>)

Selects a hash function to use.

=over

=item C<"default"> or C<undef>

Use autodetection and use either C<"jenkins"> or the original tdb hash function.

=item C<"jenkins">

The jenkins hash function. Relatively fast, and recommended for modern tdb databases
that need to be interoperable between implementations.

=item C<"fnv1ax">

The FNV1-A hash with 32 bit post-mixing. Pretty good and verxy fast for
keys up to 10-20 octets.

=item C<"xxh3">

The XXH3 hash. Very good and fast especially for long keys.

=item C<1> .. C<4>

Specify one of up to four custom hash functions (see L<set_hash_function>).

=back

=item mutex => $bool (default: C<0>)

TDB can take advantage of fast interprocess mutexes, which
can be orders of magnitude faster than the syscall-based
locking used by default. Normally, you need to call
C<TDB_FileX::runtime_check_for_robust_mutexes> and set the
C<MUTEX_LOCKING> flag if support is indicated.

This option, when enabled, enables C<MUTEX_LOCKING> if it is supported
by the platform - which involves a fork when opening the database. When
disabled, it will remove the flag.

It is recommended to keep this one, but enabling this changes the format
of the database, so might not be an option if interoperability with other
programs is required.

=back

There is no explicit close function. The database is closed implicitly
when there are no remaining references.

=item $tdb->store ($key, $value[, $flag=REPLACE])

Store $value in the database with key w$key. The $flag defaults to
C<REPLACE>, but can also be C<INSERT> or C<MODIFY>, see tdb_store(3) for
details.

On failure, a perl false is returned. See L</$tdb->error> and
L</$tdb->errorstr> for the reason.

=item $tdb->fetch ($key)

Fetch the value associated with $key, or C<undef> if it is not found.

=item $tdb->delete ($key)

Delete the value associated with $key.

On failure, a perl false is returned. See L</$tdb->error> and
L</$tdb->errorstr> for the reason.

=item $tdb->exists ($key)

Return true if the $key is found, false otherwise.

=item $tdb->firstkey

Return the key of the first value in the database. Returns C<undef> on
failure or if there are no keys in the database. See tdb_firstkey(3)
for details.

=item $tdb->nextkey ($lastkey)

Return the next key in the database after $lastkey. Returns C<undef> on
failure or if there are no more keys in the database. See tdb_nextkey(3)
for details.

=item $tdb->error

Returns the current error state of the C<$tdb> object. See the list of
error codes given in L<EXPORTS>.

=item $tdb->errorstr

Returns a printable string that describes the error state of the
database.

=item $tdb->reopen

Closes and reopens the database. Required after a
L<fork|perlfunc/fork>, if both processes wish to use the database.

B<NB:> If C<reopen> fails (returns false), then it is unsafe to call
any further methods on C<$tdb>. Thus, the only way to find out I<why>
C<reopen> failed is to use a logging function.

=item TDB_FileX::reopen_all

Closes and reopens all open databases. See L</$tdb->reopen>.

B<NB:> If C<reopen_all> fails (returns false), there is no indication
of I<which> C<$tdb> objects failed or why. If you have to survive
failures, you may wish to do your own C<reopen> loop instead.

=item $tdb->traverse ($cb->()) #TODO

Call $cb for each entry in the database.

The callback is called with the key and value as arguments and should return a
true value if you wish to continue traversal.

C<traverse> returns the number of elements traversed or C<undef> on
error. If SUB is C<undef>, then this function simply counts the number
of elements.

=item $tdb->set_logging_function(SUB)

Set the logging function to use when this database object encounters
errors. SUB should be a coderef or a string giving the name of a
function.

SUB is called with the severity level (an integer) and the message (a
string).

=item $tdb->lockall

Lock an entire database, returning false on error.

=item $tdb->unlockall

Unlock an entire database previously locked with
L</$tdb->lockall>.

=item $tdb->dump_all

Dump the records and freelist to STDOUT in an almost human readable
form.

=item $tdb->printfreelist

Dump the freelist to STDOUT.

=back

=head2 EXPORTS

Nothing constants are exported by default.

The tag C<:all> exports allpo of the constants.

Individually or with the tag C<:flags>:

  DEFAULT
  CLEAR_IF_FIRST
  INTERNAL
  NOLOCK
  NOMMAP
  CONVERT
  BIGENDIAN
  NOSYNC
  SEQNUM
  VOLATILE
  ALLOW_NESTING
  DISALLOW_NESTING
  INCOMPATIBLE_HASH
  MUTEX_LOCKING

Individually or with the tag C<:insert>:

  REPLACE
  INSERT
  MODIFY

Individually or with the tag C<:error>:

  SUCCESS
  ERR_CORRUPT
  ERR_IO
  ERR_LOCK
  ERR_OOM
  ERR_EXISTS
  ERR_NOLOCK
  ERR_LOCK_TIMEOUT
  ERR_NOEXIST
  ERR_EINVAL
  ERR_RDONLY

Individually or with the tag C<:debug>:

  DEBUG_FATAL
  DEBUG_ERROR
  DEBUG_WARNING
  DEBUG_TRACE

=head1 BUGS

There is no way to survive an error during C<reopen_all>.
Unfortunately this is a limitation in the TDB C API.

Currently the hash functions are set globally. This is due to a limitation
in the TDB C API.

=head1 SEE ALSO

tdb(3), L<perltie>.

=head1 AUTHOR

Angus Lees, E<lt>gus@inodes.org>

Currently maintained by Marc A. Lehmann <schmorp@schmorp.de>
http://home.schmorp.de/

=cut
Revision:	1.1
Committed:	Fri Apr 25 16:43:02 2025 UTC (16 months, 4 weeks ago) by root
Branch:	MAIN
Log Message:	* empty log message *
#	User	Rev	Content
1	root	1.1	package TDB_FileX;
2
3			use common::sense;
4
5			use Exporter ();
6			use XSLoader ();
7
8			our @ISA = qw(Exporter);
9
10			# Items to export into callers namespace by default. Note: do not export
11			# names by default without a very good reason. Use EXPORT_OK instead.
12			# Do not simply export all your public functions/methods/constants.
13
14			our %EXPORT_TAGS = (
15			flags => [qw(
16			ALLOW_NESTING
17			BIGENDIAN
18			CLEAR_IF_FIRST
19			CONVERT
20			DEFAULT
21			DISALLOW_NESTING
22			INCOMPATIBLE_HASH
23			INTERNAL
24			MUTEX_LOCKING
25			NOLOCK
26			NOMMAP
27			NOSYNC
28			SEQNUM
29			VOLATILE
30			)],
31			insert => [qw(
32			INSERT
33			MODIFY
34			REPLACE
35			)],
36			error => [qw(
37			SUCCESS
38			ERR_CORRUPT
39			ERR_EXISTS
40			ERR_IO
41			ERR_LOCK
42			ERR_LOCK_TIMEOUT
43			ERR_NOEXIST
44			ERR_NOLOCK
45			ERR_OOM
46			ERR_EINVAL
47			ERR_RDONLY
48			)],
49			debug => [qw(
50			DEBUG_FATAL
51			DEBUG_ERROR
52			DEBUG_WARNING
53			DEBUG_TRACE}
54			)],
55			);
56
57			our @EXPORT_OK;
58
59			Exporter::export_ok_tags qw(flags insert error debug);
60
61			$EXPORT_TAGS{all} = \@EXPORT_OK;
62
63			our $VERSION = '0.97';
64
65			XSLoader::load __PACKAGE__, $VERSION;
66
67			1;
68			__END__
69
70			=head1 NAME
71
72			TDB_FileX - Perl access to the trivial database library
73
74			=head1 SYNOPSIS
75
76			use TDB_FileX;
77
78			# tie interface
79			tie %hash, TDB_FileX => $filename,
80			hash_size => 8000,
81			mutex => 1,
82			;
83			$hash{key} = 'value';
84			while (my ($k, $v) = each %hash) { print "$k -> $v\n" }
85
86			# OO interface
87			my $tdb = TDB_FileX->open ($filename, flags => TDB_FileX::CLEAR_IF_FIRST)
88			or die $!;
89
90			$tdb->store (key => 'value') or die $tdb->errorstr;
91			$tdb->traverse (sub { print "$_[0] -> $_[1]\n" });
92
93			=head1 DESCRIPTION
94
95			TDB is a simple database similar to GDBM, but allows multiple simultaneous
96			writers. It's main drawback is the need to manually configure a hash table
97			size in advance - see the C<hash_size> option for C<open>.
98
99			TDB_FileX provides a simple C<tie> interface, similar to DB_File and
100			friends; and an object-oriented interface, which provides access to all of
101			the functions in the tdb library.
102
103			=head2 FUNCTIONS
104
105			=over 4
106
107			=item TDB_FileX->open ($path[, key => value...])
108
109			TDB_FileX constructor. Opens $path and returns a TDB_FileX object. The
110			same arguments may be passed to the C<tie> function. On error, C<$!>
111			is set and C<undef> is returned.
112
113			A number of key-value pairs are accepted, with hopefully sensible defaults
114			for most of these (but you should consider at least setting C<hash_size>
115			if you want to store more than a thousand or so pairs, And C<mutex> if you
116			want to take advantage of much faster locking.
117
118			=over
119
120			=item tdb_flags => $flags (default: C<TDB_FileX::DEFAULT>)
121
122			A set of flags that influence the behaviour and format of the database.
123
124			See tdb_open(3) for the meanings and possible values (note that the
125			C<TDB_> prefix has to be removed, see the L<EXPORTS> section for a list.
126
127			=item open_flags => $flags (default: C<Fcntl::O_RDWR \| Fcntl::O_CREAT>)
128
129			Standard open flags, as used in C<sysopen>.
130
131			=item mode => $mode (default: C<0666>)
132
133			Standard file open mode, as use din C<sysopen>. Only used when creating
134			the database file.
135
136			=item hash_size => $int (default: internal to libtdb, but normally C<131>)
137
138			The size of the internal hash table - only used when creating the
139			database. The default is usually very low and only good for a few thousand
140			keys. As a rule of thumb, it should be at least one percent of the number
141			of keys you plan to store, e.g. for C<800000> keys you should use around a
142			size of C<8000>.
143
144			To give you an idea of the performance, I inserted about 600000 records, 3GB of data,
145			into a TDB file:
146
147			size time
148			131 160s
149			256 85s
150			1024 12s
151			4096 5s
152			8192 4s
153
154			As you can see, the default hash table size for this case caused it to use
155			40 times the time to insert than a larger hash table, the optimum being
156			around 75 ke4ys per hash entry).
157
158			=item log_cb => $cb->($level, $msg)
159
160			Setsb a code reference that is called with a message and a log level
161			(lower means more important, there are C<DEBUG_FATAL>, C<DEBUG_ERROR>,
162			C<DEBUG_WARNING> and C<DEBUG_TRACE}>). Unlike the "debug" in the name
163			might indicate, if you want to find out why, for instance, you could not
164			open a database, you need to use a logging callback.
165
166			=item hash => $hash (default: C<undef>)
167
168			Selects a hash function to use.
169
170			=over
171
172			=item C<"default"> or C<undef>
173
174			Use autodetection and use either C<"jenkins"> or the original tdb hash function.
175
176			=item C<"jenkins">
177
178			The jenkins hash function. Relatively fast, and recommended for modern tdb databases
179			that need to be interoperable between implementations.
180
181			=item C<"fnv1ax">
182
183			The FNV1-A hash with 32 bit post-mixing. Pretty good and verxy fast for
184			keys up to 10-20 octets.
185
186			=item C<"xxh3">
187
188			The XXH3 hash. Very good and fast especially for long keys.
189
190			=item C<1> .. C<4>
191
192			Specify one of up to four custom hash functions (see L<set_hash_function>).
193
194			=back
195
196			=item mutex => $bool (default: C<0>)
197
198			TDB can take advantage of fast interprocess mutexes, which
199			can be orders of magnitude faster than the syscall-based
200			locking used by default. Normally, you need to call
201			C<TDB_FileX::runtime_check_for_robust_mutexes> and set the
202			C<MUTEX_LOCKING> flag if support is indicated.
203
204			This option, when enabled, enables C<MUTEX_LOCKING> if it is supported
205			by the platform - which involves a fork when opening the database. When
206			disabled, it will remove the flag.
207
208			It is recommended to keep this one, but enabling this changes the format
209			of the database, so might not be an option if interoperability with other
210			programs is required.
211
212			=back
213
214			There is no explicit close function. The database is closed implicitly
215			when there are no remaining references.
216
217			=item $tdb->store ($key, $value[, $flag=REPLACE])
218
219			Store $value in the database with key w$key. The $flag defaults to
220			C<REPLACE>, but can also be C<INSERT> or C<MODIFY>, see tdb_store(3) for
221			details.
222
223			On failure, a perl false is returned. See L</$tdb->error> and
224			L</$tdb->errorstr> for the reason.
225
226			=item $tdb->fetch ($key)
227
228			Fetch the value associated with $key, or C<undef> if it is not found.
229
230			=item $tdb->delete ($key)
231
232			Delete the value associated with $key.
233
234			On failure, a perl false is returned. See L</$tdb->error> and
235			L</$tdb->errorstr> for the reason.
236
237			=item $tdb->exists ($key)
238
239			Return true if the $key is found, false otherwise.
240
241			=item $tdb->firstkey
242
243			Return the key of the first value in the database. Returns C<undef> on
244			failure or if there are no keys in the database. See tdb_firstkey(3)
245			for details.
246
247			=item $tdb->nextkey ($lastkey)
248
249			Return the next key in the database after $lastkey. Returns C<undef> on
250			failure or if there are no more keys in the database. See tdb_nextkey(3)
251			for details.
252
253			=item $tdb->error
254
255			Returns the current error state of the C<$tdb> object. See the list of
256			error codes given in L<EXPORTS>.
257
258			=item $tdb->errorstr
259
260			Returns a printable string that describes the error state of the
261			database.
262
263			=item $tdb->reopen
264
265			Closes and reopens the database. Required after a
266			L<fork\|perlfunc/fork>, if both processes wish to use the database.
267
268			B<NB:> If C<reopen> fails (returns false), then it is unsafe to call
269			any further methods on C<$tdb>. Thus, the only way to find out I<why>
270			C<reopen> failed is to use a logging function.
271
272			=item TDB_FileX::reopen_all
273
274			Closes and reopens all open databases. See L</$tdb->reopen>.
275
276			B<NB:> If C<reopen_all> fails (returns false), there is no indication
277			of I<which> C<$tdb> objects failed or why. If you have to survive
278			failures, you may wish to do your own C<reopen> loop instead.
279
280			=item $tdb->traverse ($cb->()) #TODO
281
282			Call $cb for each entry in the database.
283
284			The callback is called with the key and value as arguments and should return a
285			true value if you wish to continue traversal.
286
287			C<traverse> returns the number of elements traversed or C<undef> on
288			error. If SUB is C<undef>, then this function simply counts the number
289			of elements.
290
291			=item $tdb->set_logging_function(SUB)
292
293			Set the logging function to use when this database object encounters
294			errors. SUB should be a coderef or a string giving the name of a
295			function.
296
297			SUB is called with the severity level (an integer) and the message (a
298			string).
299
300			=item $tdb->lockall
301
302			Lock an entire database, returning false on error.
303
304			=item $tdb->unlockall
305
306			Unlock an entire database previously locked with
307			L</$tdb->lockall>.
308
309			=item $tdb->dump_all
310
311			Dump the records and freelist to STDOUT in an almost human readable
312			form.
313
314			=item $tdb->printfreelist
315
316			Dump the freelist to STDOUT.
317
318			=back
319
320			=head2 EXPORTS
321
322			Nothing constants are exported by default.
323
324			The tag C<:all> exports allpo of the constants.
325
326			Individually or with the tag C<:flags>:
327
328			DEFAULT
329			CLEAR_IF_FIRST
330			INTERNAL
331			NOLOCK
332			NOMMAP
333			CONVERT
334			BIGENDIAN
335			NOSYNC
336			SEQNUM
337			VOLATILE
338			ALLOW_NESTING
339			DISALLOW_NESTING
340			INCOMPATIBLE_HASH
341			MUTEX_LOCKING
342
343			Individually or with the tag C<:insert>:
344
345			REPLACE
346			INSERT
347			MODIFY
348
349			Individually or with the tag C<:error>:
350
351			SUCCESS
352			ERR_CORRUPT
353			ERR_IO
354			ERR_LOCK
355			ERR_OOM
356			ERR_EXISTS
357			ERR_NOLOCK
358			ERR_LOCK_TIMEOUT
359			ERR_NOEXIST
360			ERR_EINVAL
361			ERR_RDONLY
362
363			Individually or with the tag C<:debug>:
364
365			DEBUG_FATAL
366			DEBUG_ERROR
367			DEBUG_WARNING
368			DEBUG_TRACE
369
370			=head1 BUGS
371
372			There is no way to survive an error during C<reopen_all>.
373			Unfortunately this is a limitation in the TDB C API.
374
375			Currently the hash functions are set globally. This is due to a limitation
376			in the TDB C API.
377
378			=head1 SEE ALSO
379
380			tdb(3), L<perltie>.
381
382			=head1 AUTHOR
383
384			Angus Lees, E<lt>gus@inodes.org>
385
386			Currently maintained by Marc A. Lehmann <schmorp@schmorp.de>
387			http://home.schmorp.de/
388
389			=cut