cvsroot/TDB_FileX/TDB_FileX.pm

package TDB_FileX;

use common::sense;

use Exporter ();
use XSLoader ();

our @ISA = qw(Exporter);

# Items to export into callers namespace by default. Note: do not export
# names by default without a very good reason. Use EXPORT_OK instead.
# Do not simply export all your public functions/methods/constants.

our %EXPORT_TAGS = (
   flags => [qw(
      ALLOW_NESTING
      BIGENDIAN
      CLEAR_IF_FIRST
      CONVERT
      DEFAULT
      DISALLOW_NESTING
      INCOMPATIBLE_HASH
      INTERNAL
      MUTEX_LOCKING
      NOLOCK
      NOMMAP
      NOSYNC
      SEQNUM
      VOLATILE
   )],
   insert => [qw(
      INSERT
      MODIFY
      REPLACE
   )],
   error => [qw(
      SUCCESS
      ERR_CORRUPT
      ERR_EXISTS
      ERR_IO
      ERR_LOCK
      ERR_LOCK_TIMEOUT
      ERR_NOEXIST
      ERR_NOLOCK
      ERR_OOM
      ERR_EINVAL
      ERR_RDONLY
   )],
   debug => [qw(
      DEBUG_FATAL
      DEBUG_ERROR
      DEBUG_WARNING
      DEBUG_TRACE}
   )],
);

our @EXPORT_OK;

Exporter::export_ok_tags qw(flags insert error debug);

$EXPORT_TAGS{all} = \@EXPORT_OK;

our $VERSION = '0.97';

XSLoader::load __PACKAGE__, $VERSION;

1;
__END__

=head1 NAME

TDB_FileX - Perl access to the trivial database library

=head1 SYNOPSIS

  use TDB_FileX;

  # tie interface
  tie %hash, TDB_FileX => $filename,
     hash_size => 8000,
     mutex     => 1,
  ;
  $hash{key} = 'value';
  while (my ($k, $v) = each %hash) { print "$k -> $v\n" }

  # OO interface
  my $tdb = TDB_FileX->open ($filename, flags => TDB_FileX::CLEAR_IF_FIRST)
     or die $!;

  $tdb->store (key => 'value') or die $tdb->errorstr;
  $tdb->traverse (sub { print "$_[0] -> $_[1]\n" });

=head1 DESCRIPTION

TDB is a simple database similar to GDBM, but allows multiple simultaneous
writers. It's main drawback is the need to manually configure a hash table
size in advance - see the C<hash_size> option for C<open>.

TDB_FileX provides a simple C<tie> interface, similar to DB_File and
friends; and an object-oriented interface, which provides access to most
of the functions in the TDB library.

=head1 ERROR HANDLKING

The TDB C API is not well designed - among other things, error handling is
a bit erratic. Many functions that normally should just work are marked
with a [CROAK] - these will throw an exception on error, the exception
string being the error message. C<$!> is set to the numerical error value
in that case (and will stringify into the wrong OS-error string, so don't
do that!).

This is so that you can concentrate on the important parts, while there
are still no silent unexpected errors.

The other functions will not generally croak - check their description for
details on how errors are handled.

=head2 FUNCTIONS

=over 4

=item $tdb = tie %hash, TDB_FileX => $path[ , key => value...]

=item $tdb = TDB_FileX->open ($path[, key => value...])

TDB_FileX constructor (same as C<TIE>). Opens $path and returns a
TDB_FileX object. The same arguments may be passed to the C<tie>
function. On error, C<undef> is returned.

To open a tdb file that is created and used by another application
(maximising compatibility):

   my $tdb = TDB_FileX->open ($path, log_cb => sub { warn $_[1] })
      or die "$path: failed to open\n";

To open a tdb file with good performance for many thousands of large keys,
maybe not compatible to others:

   my $tdb = TDB_FileX->open ($path,
      hash_size => 10000,
      hash      => "xxh3",
      mutex     => 1,
      nocow     => 1,
      log_cb    => sub { warn $_[1] },
   ) or die "$path: failed to open\n";

A number of key-value pairs are accepted, with hopefully sensible defaults
for most of these (but you should consider at least setting C<hash_size>
if you want to store more than a thousand or so pairs, and C<mutex> if you
want to take advantage of much faster locking.

=over

=item tdb_flags => $flags (default: C<TDB_FileX::DEFAULT>)

A set of flags that influence the behaviour and format of the database.

See tdb_open(3) for the meanings and possible values (note that the
C<TDB_> prefix has to be removed, see the L<EXPORTS> section for a list.

=item open_flags => $flags (default: C<Fcntl::O_RDWR | Fcntl::O_CREAT>)

Standard open flags, as used in C<sysopen>.

=item mode => $mode (default: C<0666>)

Standard file open mode, as use din C<sysopen>. Only used when creating
the database file.

=item hash_size => $int (default: internal to libtdb, but normally C<131>)

The size of the internal hash table - only used when creating the
database. The default is usually very low and only good for a few thousand
keys. As a rule of thumb, it should be at least one percent of the number
of keys you plan to store, e.g. for C<800000> keys you should use around a
size of C<8000>.

Every hash entry is only 4 octets, os usually it isn't an issue to make the
hash table too large.

To give you an idea of the performance, I inserted about 600000 records,
3GB of data, into a TDB file, using different hash sizes.

    size time
     131 160s
     256  85s
    1024  12s
    4096   5s
    8192   4s

As you can see, the default hash table size for this case caused it to
use 40 times the time to insert than a larger hash table, the optimum
being around 75 keys per hash entry. But the databse easily was cached in
memory. If that is not the case, you might want to consider a hash table
that is larger than the number of keys you want to store - each hash slot
only uses 4 octets.

=item log_cb => $cb->($level, $msg)

Sets a code reference that is called with a message and a log level
(lower means more important, there are C<DEBUG_FATAL>, C<DEBUG_ERROR>,
C<DEBUG_WARNING> and C<DEBUG_TRACE}>). Unlike the "debug" in the name
might indicate, if you want to find out why, for instance, you could not
open a database, you need to use a logging callback.

=item hash => $hash (default: C<undef>)

Selects a hash function to use.

=over

=item C<"default"> or C<undef>

Use autodetection and use either C<"jenkins"> or the original tdb hash function.

=item C<"jenkins">

The jenkins hash function. Relatively fast, and recommended for modern tdb databases
that need to be interoperable between implementations.

=item C<"fnv1ax">

The FNV1-A hash with 32 bit post-mixing. Pretty good and verxy fast for
keys up to 10-20 octets.

=item C<"xxh3">

The XXH3 hash. Very good and fast especially for long keys.

=item C<1> .. C<4>

Specify one of up to four custom hash functions (see L<set_hash_function>).

=back

=item mutex => $bool (default: C<0>)

TDB can take advantage of fast interprocess mutexes, which can be
orders of magnitude faster than the syscall-based locking used by
default, but only works on the same machine.

Normally, you need to call C<TDB_FileX::runtime_check_for_robust_mutexes>
and set the C<MUTEX_LOCKING> flag if support is indicated.

This option, when enabled, enables C<MUTEX_LOCKING> if it is supported
by the platform - which involves a fork when opening the database. When
disabled, it will remove the flag.

It is recommended to keep this one, but enabling this changes the format
of the database, so might not be an option if interoperability with other
programs is required.

=item nocow => $bool (default: C<0>)

Set the no-copy-on-write flag I<iff> this flag is true, C<open_flags>
contains C<O_CREAT>, C<tdb_flags> do not contain C<INTERNAL> and this is
supported on the platform and filesystem. Copy-on-write filesystems have
to make a copy of every block written to, which can both be costly and can
cause massive fragmentation of the database file.

Setting the no-copy-on-write flag (same as C<chattr +C>) disables this,
usually at the expense of data protection (checksumming), reducing the
safety to the level of a normal filesystem such as ext4.

This is best effort and usually only takes effect when the
database is initially created. If it fails, TDB_FileX will simply
continue. Correctness should not be affected either way.

=back

There is no explicit close function. The database is closed implicitly
when there are no remaining references.

=item $tdb->store ($key, $value[, $flag=REPLACE]) [CROAK]

Store $value in the database with key w$key. The $flag defaults to
C<REPLACE>, but can also be C<INSERT> or C<MODIFY>, see tdb_store(3) for
details.

=item $tdb->append ($key, $value) [CROAK]

Appends $data to the data already stored for $key, or creates a new entry with it.

=item $tdb->fetch ($key)

Fetch the value associated with $key, or C<undef> if it is not found (or
on any error).

=item $tdb->delete ($key) [CROAK]

Delete the value associated with $key.

On failure, a perl false is returned. See L</$tdb->error> and
L</$tdb->errorstr> for the reason.

=item $bool = $tdb->exists ($key)

Return true if the $key is found, false otherwise.

=item $tdb->firstkey

Return the key of the first value in the database. Returns C<undef> on
failure or if there are no keys in the database. See tdb_firstkey(3)
for details.

=item $tdb->nextkey ($lastkey)

Return the next key in the database after $lastkey. Returns C<undef> on
failure or if there are no more keys in the database. See tdb_nextkey(3)
for details.

=item $tdb->error

Returns the current error state of the C<$tdb> object. See the list of
error codes given in L<EXPORTS>.

=item $tdb->errorstr

Returns a printable string that describes the error state of the
database.

=item $tdb->reopen [CROAK]

Closes and reopens the database. Required after a
L<fork|perlfunc/fork>, if both processes wish to use the database.

B<NB:> If C<reopen> fails, then it is unsafe to call any further methods
on C<$tdb>. Thus, the only way to find out I<why> C<reopen> failed is to
use a logging function.

=item TDB_FileX::reopen_all [CROAK]

Closes and reopens all open databases. See L</$tdb->reopen>.

B<NB:> If C<reopen_all> fails, there is no indication of I<which> C<$tdb>
objects failed or why. If you have to survive failures, you may wish to do
your own C<reopen> loop instead.

=item $tdb->traverse ($cb->($key, $data)) [CROAK]

Call $cb for each entry in the database. The callback should return false
to continue, or a true value to abort the traversal.

The callback is called with the key and value as arguments and should
return a false value if you wish to continue traversal, and a true value
if the traversal should be aborted.

C<traverse> returns the number of elements traversed. If $cb is C<undef>,
then this function simply counts the number of elements.

=item $tdb->traverse_read ($cb->($key, $data)) [CROAK]

Like C<traverse>, but only acquires a read lock.

=item $tdb->set_logging_function ($cb->($level, $msg))

Set the logging function to use when this database object encounters
errors.

$cb is called with the severity level (an integer) and the message (a
string).

=item $tdb->lockall [CROAK]

Lock an entire database with an exclusive write lock, returning false on
error. The purpose of this call is to avoid locking overhead for many
operations, but the database has to be unlocked manually when done.

=item $tdb->unlockall [CROAK]

Unlock an entire database previously locked with
L</$tdb->lockall>.

=item $tdb->lockall_read [CROAK]

Same as L</$tdb->lockall>, but uses a shared read lock instead of a writer lock.

=item $tdb->unlockall_read [CROAK]

Opposite of L</$tdb->lockall_read>.

=item $tdb->lockall_mark [CROAK]

=item $tdb->lockall_unmark [CROAK]

These apparently mark and unmark locks internally, but do not actually do
locking. Probably you should not use this, but feel free to tell me when
these are useful.

=item $tdb->lockall_nonblock [CROAK]

=item $success = $tdb->lockall_read_nonblock [CROAK]

Try to lock, but instead of waiting, fail if the lock could not
be acquired. Returns true if the lock could be acquired, false
otherwise. Croaks on all other errors.

=item $tdb->transaction_start [CROAK]

Starts a transaction - all operations will be queued, but not applied
to the database, until the transaction is either committed L</$tdb->transaction_commit>
or aborted/thrown away, with L</$tdb->transaction_cancel>.

=item $tdb->transaction_start_nonblock [CROAK]

Tries to start a transaction, but instead of waiting, fail if the lock
could not be acquired. Returns true if the transaction could be started,
false otherwise. Croaks on all other errors.

Please tell me what this does.

=item $tdb->transaction_commit [CROAK]

Applies all changes in the transaction.

=item $tdb->transaction_cancel [CROAK]

Throws away all changes in the transaction.

=item $tdb->transaction_prepare_commit [CROAK]

Instead of calling L</$tdb->transaction_commit> you can do the commit in
two phases by calling this method before commit, which does the expensive
steps first.

=item $bool = $tdb->transaction_active

Returns true if a transaction is currently active.

=item $tdb->enable_seqnum

Enables sequence number generatiuon supporet for the database.

=item $seq = $tdb->get_seqnum

Returns the current sequence number. Internally, this is a 32 bit unsigned
integer, but the API converts it into a native integer, so the same
internal sequence number might be represented differently on different
machines.

=item $tdb->increment_seqnum_nonblock

Increments the sequence number. Note that the sequence number is also
incremented by TDB itself oon many operations.

=item $size = $tdb->hash_size

Returns the size of the hash table. The size cannot be changed other than
by recreating the database.

=item $octets = $tdb->map_size

Returns the current mmap size for the database.

=item $flags = $tdb->get_flags

Returns the flags for the database (the same as the C<tdb_flags> in C<open>).

=item $tdb->add_flags ($flag)

Tried to add flags to the database - yes, the parameter says C<$flag> (singular)
but the documentation and the code say I<flags> (plural).

=item $tdb->remove_flags ($flag)

Attempts to remove flags from the database.

=item $fileno = $tdb->fd

Returns the file descriptor (not file handle) for the underlying database file.

=item $path = $tdb->name

Returns the path used to open the database.

=item $tdb->wipe_all [CROAK]

Efficientlly deletes all entries in the database. This does not shrink the
file itself.

=item $tdb->repack [CROAK]

Tries to improve layout of the database by copying all items into a
temporary in-memory database, wiping the database, and copying all items
back. Yes, everything must fit into memory.

=item $bool = $tdb->check ($cb->($key, $value))

Does extensive checks on the database, optionally (if not C<undef>)
calling a check function for each pair, which must return true if the data
is valid.

Returns a boolean indicating whether the database was found healthy and
all the calls to the callback returned true.

=item $bool = $tdb->rescue ($cb->($key, $value))

Tries to recover some or all key-value pairs from a potentially damanged
database file. For each recovered pair it calls the given callback.

Returns a boolean indicating whether the database was found healthy and
all the calls to the callback returned true.

=item $bool = TDB_FileX::runtime_check_for_robust_mutexes

Tests whether robust mutexes are available for locking. This involves
forking the process, so it can be costly and problematic. This function
needs to be called before using the C<MUTEX_LOCKING> flag. But see the
C<mutex> parametrer to C<open> for an alternative.

=item $tdb->dump_all

Dump the records and freelist to STDOUT in an almost human readable
form.

=item $tdb->printfreelist

Dump the freelist to STDOUT.

=back

=head2 EXPORTS

Nothing constants are exported by default.

The tag C<:all> exports allpo of the constants.

Individually or with the tag C<:flags>:

  DEFAULT
  CLEAR_IF_FIRST
  INTERNAL
  NOLOCK
  NOMMAP
  CONVERT
  BIGENDIAN
  NOSYNC
  SEQNUM
  VOLATILE
  ALLOW_NESTING
  DISALLOW_NESTING
  INCOMPATIBLE_HASH
  MUTEX_LOCKING

Individually or with the tag C<:insert>:

  REPLACE
  INSERT
  MODIFY

Individually or with the tag C<:error>:

  SUCCESS
  ERR_CORRUPT
  ERR_IO
  ERR_LOCK
  ERR_OOM
  ERR_EXISTS
  ERR_NOLOCK
  ERR_LOCK_TIMEOUT
  ERR_NOEXIST
  ERR_EINVAL
  ERR_RDONLY

Individually or with the tag C<:debug>:

  DEBUG_FATAL
  DEBUG_ERROR
  DEBUG_WARNING
  DEBUG_TRACE

=head1 UNICODE HANDLING

TDB databases can only store octet strings. Unlike most other database
interfaces, TDB_FileX will safely handle Perl strings by downgrading
them. Perl will warn about strings that cnanot be downgraded.

=head1 DISK USAGE

TDB databses use 24 octets for every key-value pair, plus the octet size
of the key and sata, e.g. the pair "key" => "value" takes up 24+3+5 octets
on disk.

The database header is 168 octets (if I haven't miscounted).

Each hashtable entry is 4 octets, and one more than the hash size is
allocated, so the hahs table size is (hash_size + 1) * 4.

=head1 LIMITATIONS

=head2 Database Size

As far as I can see, TDB dastabases are limited to 4 GB.

=head2 Hash Functions

Hash functioons need to be set globally - they are limited to a maximum of
4, but this can be easily extended, but requires source code editing. This
is due to a limitation of the TDB C API.

=head2 No recoivery after failed C<reopoen_all>

There is no way to survive an error during C<reopen_all>.
Unfortunately this is a limitation in the TDB C API.

=head1 SEE ALSO

tdb(3), L<perltie>.

=head1 AUTHOR

Angus Lees, E<lt>gus@inodes.org>

Currently maintained by Marc A. Lehmann <schmorp@schmorp.de>
http://home.schmorp.de/

=cut
Revision:	1.8
Committed:	Fri May 2 20:47:43 2025 UTC (16 months, 3 weeks ago) by root
Branch:	MAIN
Changes since 1.7:	+11 -3 lines
Log Message:	* empty log message *
#	User	Rev	Content
1	root	1.1	package TDB_FileX;
2
3			use common::sense;
4
5			use Exporter ();
6			use XSLoader ();
7
8			our @ISA = qw(Exporter);
9
10			# Items to export into callers namespace by default. Note: do not export
11			# names by default without a very good reason. Use EXPORT_OK instead.
12			# Do not simply export all your public functions/methods/constants.
13
14			our %EXPORT_TAGS = (
15			flags => [qw(
16			ALLOW_NESTING
17			BIGENDIAN
18			CLEAR_IF_FIRST
19			CONVERT
20			DEFAULT
21			DISALLOW_NESTING
22			INCOMPATIBLE_HASH
23			INTERNAL
24			MUTEX_LOCKING
25			NOLOCK
26			NOMMAP
27			NOSYNC
28			SEQNUM
29			VOLATILE
30			)],
31			insert => [qw(
32			INSERT
33			MODIFY
34			REPLACE
35			)],
36			error => [qw(
37			SUCCESS
38			ERR_CORRUPT
39			ERR_EXISTS
40			ERR_IO
41			ERR_LOCK
42			ERR_LOCK_TIMEOUT
43			ERR_NOEXIST
44			ERR_NOLOCK
45			ERR_OOM
46			ERR_EINVAL
47			ERR_RDONLY
48			)],
49			debug => [qw(
50			DEBUG_FATAL
51			DEBUG_ERROR
52			DEBUG_WARNING
53			DEBUG_TRACE}
54			)],
55			);
56
57			our @EXPORT_OK;
58
59			Exporter::export_ok_tags qw(flags insert error debug);
60
61			$EXPORT_TAGS{all} = \@EXPORT_OK;
62
63			our $VERSION = '0.97';
64
65			XSLoader::load __PACKAGE__, $VERSION;
66
67			1;
68			__END__
69
70			=head1 NAME
71
72			TDB_FileX - Perl access to the trivial database library
73
74			=head1 SYNOPSIS
75
76			use TDB_FileX;
77
78			# tie interface
79			tie %hash, TDB_FileX => $filename,
80			hash_size => 8000,
81			mutex => 1,
82			;
83			$hash{key} = 'value';
84			while (my ($k, $v) = each %hash) { print "$k -> $v\n" }
85
86			# OO interface
87			my $tdb = TDB_FileX->open ($filename, flags => TDB_FileX::CLEAR_IF_FIRST)
88			or die $!;
89
90			$tdb->store (key => 'value') or die $tdb->errorstr;
91			$tdb->traverse (sub { print "$_[0] -> $_[1]\n" });
92
93			=head1 DESCRIPTION
94
95			TDB is a simple database similar to GDBM, but allows multiple simultaneous
96			writers. It's main drawback is the need to manually configure a hash table
97			size in advance - see the C<hash_size> option for C<open>.
98
99			TDB_FileX provides a simple C<tie> interface, similar to DB_File and
100	root	1.3	friends; and an object-oriented interface, which provides access to most
101			of the functions in the TDB library.
102
103			=head1 ERROR HANDLKING
104
105			The TDB C API is not well designed - among other things, error handling is
106			a bit erratic. Many functions that normally should just work are marked
107			with a [CROAK] - these will throw an exception on error, the exception
108			string being the error message. C<$!> is set to the numerical error value
109			in that case (and will stringify into the wrong OS-error string, so don't
110			do that!).
111
112			This is so that you can concentrate on the important parts, while there
113			are still no silent unexpected errors.
114
115			The other functions will not generally croak - check their description for
116			details on how errors are handled.
117	root	1.1
118			=head2 FUNCTIONS
119
120			=over 4
121
122	root	1.3	=item $tdb = tie %hash, TDB_FileX => $path[ , key => value...]
123
124			=item $tdb = TDB_FileX->open ($path[, key => value...])
125
126			TDB_FileX constructor (same as C<TIE>). Opens $path and returns a
127			TDB_FileX object. The same arguments may be passed to the C<tie>
128			function. On error, C<undef> is returned.
129	root	1.1
130	root	1.3	To open a tdb file that is created and used by another application
131	root	1.6	(maximising compatibility):
132	root	1.3
133			my $tdb = TDB_FileX->open ($path, log_cb => sub { warn $_[1] })
134			or die "$path: failed to open\n";
135
136			To open a tdb file with good performance for many thousands of large keys,
137			maybe not compatible to others:
138
139			my $tdb = TDB_FileX->open ($path,
140			hash_size => 10000,
141			hash => "xxh3",
142			mutex => 1,
143	root	1.6	nocow => 1,
144	root	1.3	log_cb => sub { warn $_[1] },
145			) or die "$path: failed to open\n";
146	root	1.1
147			A number of key-value pairs are accepted, with hopefully sensible defaults
148			for most of these (but you should consider at least setting C<hash_size>
149	root	1.6	if you want to store more than a thousand or so pairs, and C<mutex> if you
150	root	1.1	want to take advantage of much faster locking.
151
152			=over
153
154			=item tdb_flags => $flags (default: C<TDB_FileX::DEFAULT>)
155
156			A set of flags that influence the behaviour and format of the database.
157
158			See tdb_open(3) for the meanings and possible values (note that the
159			C<TDB_> prefix has to be removed, see the L<EXPORTS> section for a list.
160
161			=item open_flags => $flags (default: C<Fcntl::O_RDWR \| Fcntl::O_CREAT>)
162
163			Standard open flags, as used in C<sysopen>.
164
165			=item mode => $mode (default: C<0666>)
166
167			Standard file open mode, as use din C<sysopen>. Only used when creating
168			the database file.
169
170			=item hash_size => $int (default: internal to libtdb, but normally C<131>)
171
172			The size of the internal hash table - only used when creating the
173			database. The default is usually very low and only good for a few thousand
174			keys. As a rule of thumb, it should be at least one percent of the number
175			of keys you plan to store, e.g. for C<800000> keys you should use around a
176			size of C<8000>.
177
178	root	1.5	Every hash entry is only 4 octets, os usually it isn't an issue to make the
179	root	1.2	hash table too large.
180
181			To give you an idea of the performance, I inserted about 600000 records,
182			3GB of data, into a TDB file, using different hash sizes.
183	root	1.1
184			size time
185			131 160s
186			256 85s
187			1024 12s
188			4096 5s
189			8192 4s
190
191	root	1.3	As you can see, the default hash table size for this case caused it to
192			use 40 times the time to insert than a larger hash table, the optimum
193	root	1.5	being around 75 keys per hash entry. But the databse easily was cached in
194	root	1.3	memory. If that is not the case, you might want to consider a hash table
195			that is larger than the number of keys you want to store - each hash slot
196			only uses 4 octets.
197	root	1.1
198			=item log_cb => $cb->($level, $msg)
199
200	root	1.2	Sets a code reference that is called with a message and a log level
201	root	1.1	(lower means more important, there are C<DEBUG_FATAL>, C<DEBUG_ERROR>,
202			C<DEBUG_WARNING> and C<DEBUG_TRACE}>). Unlike the "debug" in the name
203			might indicate, if you want to find out why, for instance, you could not
204			open a database, you need to use a logging callback.
205
206			=item hash => $hash (default: C<undef>)
207
208			Selects a hash function to use.
209
210			=over
211
212			=item C<"default"> or C<undef>
213
214			Use autodetection and use either C<"jenkins"> or the original tdb hash function.
215
216			=item C<"jenkins">
217
218			The jenkins hash function. Relatively fast, and recommended for modern tdb databases
219			that need to be interoperable between implementations.
220
221			=item C<"fnv1ax">
222
223			The FNV1-A hash with 32 bit post-mixing. Pretty good and verxy fast for
224			keys up to 10-20 octets.
225
226			=item C<"xxh3">
227
228			The XXH3 hash. Very good and fast especially for long keys.
229
230			=item C<1> .. C<4>
231
232			Specify one of up to four custom hash functions (see L<set_hash_function>).
233
234			=back
235
236			=item mutex => $bool (default: C<0>)
237
238	root	1.6	TDB can take advantage of fast interprocess mutexes, which can be
239			orders of magnitude faster than the syscall-based locking used by
240			default, but only works on the same machine.
241
242			Normally, you need to call C<TDB_FileX::runtime_check_for_robust_mutexes>
243			and set the C<MUTEX_LOCKING> flag if support is indicated.
244	root	1.1
245			This option, when enabled, enables C<MUTEX_LOCKING> if it is supported
246			by the platform - which involves a fork when opening the database. When
247			disabled, it will remove the flag.
248
249			It is recommended to keep this one, but enabling this changes the format
250			of the database, so might not be an option if interoperability with other
251			programs is required.
252
253	root	1.6	=item nocow => $bool (default: C<0>)
254
255			Set the no-copy-on-write flag I<iff> this flag is true, C<open_flags>
256	root	1.7	contains C<O_CREAT>, C<tdb_flags> do not contain C<INTERNAL> and this is
257			supported on the platform and filesystem. Copy-on-write filesystems have
258			to make a copy of every block written to, which can both be costly and can
259			cause massive fragmentation of the database file.
260	root	1.6
261			Setting the no-copy-on-write flag (same as C<chattr +C>) disables this,
262			usually at the expense of data protection (checksumming), reducing the
263			safety to the level of a normal filesystem such as ext4.
264
265			This is best effort and usually only takes effect when the
266			database is initially created. If it fails, TDB_FileX will simply
267			continue. Correctness should not be affected either way.
268
269	root	1.1	=back
270
271			There is no explicit close function. The database is closed implicitly
272			when there are no remaining references.
273
274	root	1.3	=item $tdb->store ($key, $value[, $flag=REPLACE]) [CROAK]
275	root	1.1
276			Store $value in the database with key w$key. The $flag defaults to
277			C<REPLACE>, but can also be C<INSERT> or C<MODIFY>, see tdb_store(3) for
278			details.
279
280	root	1.3	=item $tdb->append ($key, $value) [CROAK]
281
282			Appends $data to the data already stored for $key, or creates a new entry with it.
283	root	1.1
284			=item $tdb->fetch ($key)
285
286	root	1.3	Fetch the value associated with $key, or C<undef> if it is not found (or
287			on any error).
288	root	1.1
289	root	1.3	=item $tdb->delete ($key) [CROAK]
290	root	1.1
291			Delete the value associated with $key.
292
293			On failure, a perl false is returned. See L</$tdb->error> and
294			L</$tdb->errorstr> for the reason.
295
296	root	1.3	=item $bool = $tdb->exists ($key)
297	root	1.1
298			Return true if the $key is found, false otherwise.
299
300			=item $tdb->firstkey
301
302			Return the key of the first value in the database. Returns C<undef> on
303			failure or if there are no keys in the database. See tdb_firstkey(3)
304			for details.
305
306			=item $tdb->nextkey ($lastkey)
307
308			Return the next key in the database after $lastkey. Returns C<undef> on
309			failure or if there are no more keys in the database. See tdb_nextkey(3)
310			for details.
311
312			=item $tdb->error
313
314			Returns the current error state of the C<$tdb> object. See the list of
315			error codes given in L<EXPORTS>.
316
317			=item $tdb->errorstr
318
319			Returns a printable string that describes the error state of the
320			database.
321
322	root	1.3	=item $tdb->reopen [CROAK]
323	root	1.1
324			Closes and reopens the database. Required after a
325			L<fork\|perlfunc/fork>, if both processes wish to use the database.
326
327	root	1.3	B<NB:> If C<reopen> fails, then it is unsafe to call any further methods
328			on C<$tdb>. Thus, the only way to find out I<why> C<reopen> failed is to
329			use a logging function.
330	root	1.1
331	root	1.3	=item TDB_FileX::reopen_all [CROAK]
332	root	1.1
333			Closes and reopens all open databases. See L</$tdb->reopen>.
334
335	root	1.3	B<NB:> If C<reopen_all> fails, there is no indication of I<which> C<$tdb>
336			objects failed or why. If you have to survive failures, you may wish to do
337			your own C<reopen> loop instead.
338	root	1.1
339	root	1.4	=item $tdb->traverse ($cb->($key, $data)) [CROAK]
340	root	1.1
341	root	1.4	Call $cb for each entry in the database. The callback should return false
342			to continue, or a true value to abort the traversal.
343	root	1.1
344	root	1.3	The callback is called with the key and value as arguments and should
345	root	1.4	return a false value if you wish to continue traversal, and a true value
346			if the traversal should be aborted.
347	root	1.1
348	root	1.4	C<traverse> returns the number of elements traversed. If $cb is C<undef>,
349			then this function simply counts the number of elements.
350
351			=item $tdb->traverse_read ($cb->($key, $data)) [CROAK]
352
353			Like C<traverse>, but only acquires a read lock.
354	root	1.1
355	root	1.3	=item $tdb->set_logging_function ($cb->($level, $msg))
356	root	1.1
357			Set the logging function to use when this database object encounters
358	root	1.3	errors.
359	root	1.1
360	root	1.3	$cb is called with the severity level (an integer) and the message (a
361	root	1.1	string).
362
363	root	1.3	=item $tdb->lockall [CROAK]
364	root	1.1
365	root	1.2	Lock an entire database with an exclusive write lock, returning false on
366			error. The purpose of this call is to avoid locking overhead for many
367			operations, but the database has to be unlocked manually when done.
368	root	1.1
369	root	1.3	=item $tdb->unlockall [CROAK]
370	root	1.1
371			Unlock an entire database previously locked with
372			L</$tdb->lockall>.
373
374	root	1.3	=item $tdb->lockall_read [CROAK]
375	root	1.2
376			Same as L</$tdb->lockall>, but uses a shared read lock instead of a writer lock.
377
378	root	1.3	=item $tdb->unlockall_read [CROAK]
379	root	1.2
380			Opposite of L</$tdb->lockall_read>.
381
382	root	1.3	=item $tdb->lockall_mark [CROAK]
383	root	1.2
384	root	1.3	=item $tdb->lockall_unmark [CROAK]
385	root	1.2
386	root	1.8	These apparently mark and unmark locks internally, but do not actually do
387			locking. Probably you should not use this, but feel free to tell me when
388			these are useful.
389	root	1.2
390	root	1.3	=item $tdb->lockall_nonblock [CROAK]
391	root	1.2
392	root	1.8	=item $success = $tdb->lockall_read_nonblock [CROAK]
393	root	1.2
394	root	1.8	Try to lock, but instead of waiting, fail if the lock could not
395			be acquired. Returns true if the lock could be acquired, false
396			otherwise. Croaks on all other errors.
397	root	1.2
398	root	1.3	=item $tdb->transaction_start [CROAK]
399	root	1.2
400			Starts a transaction - all operations will be queued, but not applied
401			to the database, until the transaction is either committed L</$tdb->transaction_commit>
402			or aborted/thrown away, with L</$tdb->transaction_cancel>.
403
404	root	1.3	=item $tdb->transaction_start_nonblock [CROAK]
405	root	1.2
406	root	1.8	Tries to start a transaction, but instead of waiting, fail if the lock
407			could not be acquired. Returns true if the transaction could be started,
408			false otherwise. Croaks on all other errors.
409
410	root	1.2	Please tell me what this does.
411
412	root	1.3	=item $tdb->transaction_commit [CROAK]
413	root	1.2
414			Applies all changes in the transaction.
415
416	root	1.3	=item $tdb->transaction_cancel [CROAK]
417	root	1.2
418			Throws away all changes in the transaction.
419
420	root	1.3	=item $tdb->transaction_prepare_commit [CROAK]
421	root	1.2
422			Instead of calling L</$tdb->transaction_commit> you can do the commit in
423			two phases by calling this method before commit, which does the expensive
424			steps first.
425
426			=item $bool = $tdb->transaction_active
427
428			Returns true if a transaction is currently active.
429
430			=item $tdb->enable_seqnum
431
432			Enables sequence number generatiuon supporet for the database.
433
434			=item $seq = $tdb->get_seqnum
435
436			Returns the current sequence number. Internally, this is a 32 bit unsigned
437			integer, but the API converts it into a native integer, so the same
438			internal sequence number might be represented differently on different
439			machines.
440
441			=item $tdb->increment_seqnum_nonblock
442
443			Increments the sequence number. Note that the sequence number is also
444			incremented by TDB itself oon many operations.
445
446			=item $size = $tdb->hash_size
447
448			Returns the size of the hash table. The size cannot be changed other than
449			by recreating the database.
450
451			=item $octets = $tdb->map_size
452
453			Returns the current mmap size for the database.
454
455			=item $flags = $tdb->get_flags
456
457			Returns the flags for the database (the same as the C<tdb_flags> in C<open>).
458
459			=item $tdb->add_flags ($flag)
460
461			Tried to add flags to the database - yes, the parameter says C<$flag> (singular)
462			but the documentation and the code say I<flags> (plural).
463
464			=item $tdb->remove_flags ($flag)
465
466			Attempts to remove flags from the database.
467
468			=item $fileno = $tdb->fd
469
470			Returns the file descriptor (not file handle) for the underlying database file.
471
472			=item $path = $tdb->name
473
474			Returns the path used to open the database.
475
476	root	1.3	=item $tdb->wipe_all [CROAK]
477	root	1.2
478			Efficientlly deletes all entries in the database. This does not shrink the
479			file itself.
480
481	root	1.3	=item $tdb->repack [CROAK]
482	root	1.2
483			Tries to improve layout of the database by copying all items into a
484			temporary in-memory database, wiping the database, and copying all items
485	root	1.3	back. Yes, everything must fit into memory.
486	root	1.2
487	root	1.4	=item $bool = $tdb->check ($cb->($key, $value))
488
489			Does extensive checks on the database, optionally (if not C<undef>)
490			calling a check function for each pair, which must return true if the data
491			is valid.
492
493			Returns a boolean indicating whether the database was found healthy and
494			all the calls to the callback returned true.
495
496			=item $bool = $tdb->rescue ($cb->($key, $value))
497
498			Tries to recover some or all key-value pairs from a potentially damanged
499			database file. For each recovered pair it calls the given callback.
500
501			Returns a boolean indicating whether the database was found healthy and
502			all the calls to the callback returned true.
503
504	root	1.2	=item $bool = TDB_FileX::runtime_check_for_robust_mutexes
505
506			Tests whether robust mutexes are available for locking. This involves
507			forking the process, so it can be costly and problematic. This function
508			needs to be called before using the C<MUTEX_LOCKING> flag. But see the
509			C<mutex> parametrer to C<open> for an alternative.
510
511	root	1.1	=item $tdb->dump_all
512
513			Dump the records and freelist to STDOUT in an almost human readable
514			form.
515
516			=item $tdb->printfreelist
517
518			Dump the freelist to STDOUT.
519
520			=back
521
522			=head2 EXPORTS
523
524			Nothing constants are exported by default.
525
526			The tag C<:all> exports allpo of the constants.
527
528			Individually or with the tag C<:flags>:
529
530			DEFAULT
531			CLEAR_IF_FIRST
532			INTERNAL
533			NOLOCK
534			NOMMAP
535			CONVERT
536			BIGENDIAN
537			NOSYNC
538			SEQNUM
539			VOLATILE
540			ALLOW_NESTING
541			DISALLOW_NESTING
542			INCOMPATIBLE_HASH
543			MUTEX_LOCKING
544
545			Individually or with the tag C<:insert>:
546
547			REPLACE
548			INSERT
549			MODIFY
550
551			Individually or with the tag C<:error>:
552
553			SUCCESS
554			ERR_CORRUPT
555			ERR_IO
556			ERR_LOCK
557			ERR_OOM
558			ERR_EXISTS
559			ERR_NOLOCK
560			ERR_LOCK_TIMEOUT
561			ERR_NOEXIST
562			ERR_EINVAL
563			ERR_RDONLY
564
565			Individually or with the tag C<:debug>:
566
567			DEBUG_FATAL
568			DEBUG_ERROR
569			DEBUG_WARNING
570			DEBUG_TRACE
571
572	root	1.2	=head1 UNICODE HANDLING
573
574			TDB databases can only store octet strings. Unlike most other database
575			interfaces, TDB_FileX will safely handle Perl strings by downgrading
576			them. Perl will warn about strings that cnanot be downgraded.
577
578			=head1 DISK USAGE
579
580			TDB databses use 24 octets for every key-value pair, plus the octet size
581			of the key and sata, e.g. the pair "key" => "value" takes up 24+3+5 octets
582			on disk.
583
584			The database header is 168 octets (if I haven't miscounted).
585
586			Each hashtable entry is 4 octets, and one more than the hash size is
587			allocated, so the hahs table size is (hash_size + 1) * 4.
588
589			=head1 LIMITATIONS
590
591			=head2 Database Size
592
593			As far as I can see, TDB dastabases are limited to 4 GB.
594
595			=head2 Hash Functions
596
597			Hash functioons need to be set globally - they are limited to a maximum of
598			4, but this can be easily extended, but requires source code editing. This
599			is due to a limitation of the TDB C API.
600
601			=head2 No recoivery after failed C<reopoen_all>
602	root	1.1
603			There is no way to survive an error during C<reopen_all>.
604			Unfortunately this is a limitation in the TDB C API.
605
606			=head1 SEE ALSO
607
608			tdb(3), L<perltie>.
609
610			=head1 AUTHOR
611
612			Angus Lees, E<lt>gus@inodes.org>
613
614			Currently maintained by Marc A. Lehmann <schmorp@schmorp.de>
615			http://home.schmorp.de/
616
617			=cut