ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/cvsroot/TDB_FileX/TDB_FileX.pm
Revision: 1.3
Committed: Tue Apr 29 21:27:54 2025 UTC (17 months, 1 week ago) by root
Branch: MAIN
Changes since 1.2: +85 -48 lines
Log Message:
*** empty log message ***

File Contents

# Content
1 package TDB_FileX;
2
3 use common::sense;
4
5 use Exporter ();
6 use XSLoader ();
7
8 our @ISA = qw(Exporter);
9
10 # Items to export into callers namespace by default. Note: do not export
11 # names by default without a very good reason. Use EXPORT_OK instead.
12 # Do not simply export all your public functions/methods/constants.
13
14 our %EXPORT_TAGS = (
15 flags => [qw(
16 ALLOW_NESTING
17 BIGENDIAN
18 CLEAR_IF_FIRST
19 CONVERT
20 DEFAULT
21 DISALLOW_NESTING
22 INCOMPATIBLE_HASH
23 INTERNAL
24 MUTEX_LOCKING
25 NOLOCK
26 NOMMAP
27 NOSYNC
28 SEQNUM
29 VOLATILE
30 )],
31 insert => [qw(
32 INSERT
33 MODIFY
34 REPLACE
35 )],
36 error => [qw(
37 SUCCESS
38 ERR_CORRUPT
39 ERR_EXISTS
40 ERR_IO
41 ERR_LOCK
42 ERR_LOCK_TIMEOUT
43 ERR_NOEXIST
44 ERR_NOLOCK
45 ERR_OOM
46 ERR_EINVAL
47 ERR_RDONLY
48 )],
49 debug => [qw(
50 DEBUG_FATAL
51 DEBUG_ERROR
52 DEBUG_WARNING
53 DEBUG_TRACE}
54 )],
55 );
56
57 our @EXPORT_OK;
58
59 Exporter::export_ok_tags qw(flags insert error debug);
60
61 $EXPORT_TAGS{all} = \@EXPORT_OK;
62
63 our $VERSION = '0.97';
64
65 XSLoader::load __PACKAGE__, $VERSION;
66
67 1;
68 __END__
69
70 =head1 NAME
71
72 TDB_FileX - Perl access to the trivial database library
73
74 =head1 SYNOPSIS
75
76 use TDB_FileX;
77
78 # tie interface
79 tie %hash, TDB_FileX => $filename,
80 hash_size => 8000,
81 mutex => 1,
82 ;
83 $hash{key} = 'value';
84 while (my ($k, $v) = each %hash) { print "$k -> $v\n" }
85
86 # OO interface
87 my $tdb = TDB_FileX->open ($filename, flags => TDB_FileX::CLEAR_IF_FIRST)
88 or die $!;
89
90 $tdb->store (key => 'value') or die $tdb->errorstr;
91 $tdb->traverse (sub { print "$_[0] -> $_[1]\n" });
92
93 =head1 DESCRIPTION
94
95 TDB is a simple database similar to GDBM, but allows multiple simultaneous
96 writers. It's main drawback is the need to manually configure a hash table
97 size in advance - see the C<hash_size> option for C<open>.
98
99 TDB_FileX provides a simple C<tie> interface, similar to DB_File and
100 friends; and an object-oriented interface, which provides access to most
101 of the functions in the TDB library.
102
103 =head1 ERROR HANDLKING
104
105 The TDB C API is not well designed - among other things, error handling is
106 a bit erratic. Many functions that normally should just work are marked
107 with a [CROAK] - these will throw an exception on error, the exception
108 string being the error message. C<$!> is set to the numerical error value
109 in that case (and will stringify into the wrong OS-error string, so don't
110 do that!).
111
112 This is so that you can concentrate on the important parts, while there
113 are still no silent unexpected errors.
114
115 The other functions will not generally croak - check their description for
116 details on how errors are handled.
117
118 =head2 FUNCTIONS
119
120 =over 4
121
122 =item $tdb = tie %hash, TDB_FileX => $path[ , key => value...]
123
124 =item $tdb = TDB_FileX->open ($path[, key => value...])
125
126 TDB_FileX constructor (same as C<TIE>). Opens $path and returns a
127 TDB_FileX object. The same arguments may be passed to the C<tie>
128 function. On error, C<undef> is returned.
129
130 To open a tdb file that is created and used by another application
131 (maximize compatibility):
132
133 my $tdb = TDB_FileX->open ($path, log_cb => sub { warn $_[1] })
134 or die "$path: failed to open\n";
135
136 To open a tdb file with good performance for many thousands of large keys,
137 maybe not compatible to others:
138
139 my $tdb = TDB_FileX->open ($path,
140 hash_size => 10000,
141 hash => "xxh3",
142 mutex => 1,
143 log_cb => sub { warn $_[1] },
144 ) or die "$path: failed to open\n";
145
146 A number of key-value pairs are accepted, with hopefully sensible defaults
147 for most of these (but you should consider at least setting C<hash_size>
148 if you want to store more than a thousand or so pairs, And C<mutex> if you
149 want to take advantage of much faster locking.
150
151 =over
152
153 =item tdb_flags => $flags (default: C<TDB_FileX::DEFAULT>)
154
155 A set of flags that influence the behaviour and format of the database.
156
157 See tdb_open(3) for the meanings and possible values (note that the
158 C<TDB_> prefix has to be removed, see the L<EXPORTS> section for a list.
159
160 =item open_flags => $flags (default: C<Fcntl::O_RDWR | Fcntl::O_CREAT>)
161
162 Standard open flags, as used in C<sysopen>.
163
164 =item mode => $mode (default: C<0666>)
165
166 Standard file open mode, as use din C<sysopen>. Only used when creating
167 the database file.
168
169 =item hash_size => $int (default: internal to libtdb, but normally C<131>)
170
171 The size of the internal hash table - only used when creating the
172 database. The default is usually very low and only good for a few thousand
173 keys. As a rule of thumb, it should be at least one percent of the number
174 of keys you plan to store, e.g. for C<800000> keys you should use around a
175 size of C<8000>.
176
177 Every has entry is only 4 octets, os usually it isn't an issue to make the
178 hash table too large.
179
180 To give you an idea of the performance, I inserted about 600000 records,
181 3GB of data, into a TDB file, using different hash sizes.
182
183 size time
184 131 160s
185 256 85s
186 1024 12s
187 4096 5s
188 8192 4s
189
190 As you can see, the default hash table size for this case caused it to
191 use 40 times the time to insert than a larger hash table, the optimum
192 being around 75 keys per hash entry. Buit the databse easily was cached in
193 memory. If that is not the case, you might want to consider a hash table
194 that is larger than the number of keys you want to store - each hash slot
195 only uses 4 octets.
196
197 =item log_cb => $cb->($level, $msg)
198
199 Sets a code reference that is called with a message and a log level
200 (lower means more important, there are C<DEBUG_FATAL>, C<DEBUG_ERROR>,
201 C<DEBUG_WARNING> and C<DEBUG_TRACE}>). Unlike the "debug" in the name
202 might indicate, if you want to find out why, for instance, you could not
203 open a database, you need to use a logging callback.
204
205 =item hash => $hash (default: C<undef>)
206
207 Selects a hash function to use.
208
209 =over
210
211 =item C<"default"> or C<undef>
212
213 Use autodetection and use either C<"jenkins"> or the original tdb hash function.
214
215 =item C<"jenkins">
216
217 The jenkins hash function. Relatively fast, and recommended for modern tdb databases
218 that need to be interoperable between implementations.
219
220 =item C<"fnv1ax">
221
222 The FNV1-A hash with 32 bit post-mixing. Pretty good and verxy fast for
223 keys up to 10-20 octets.
224
225 =item C<"xxh3">
226
227 The XXH3 hash. Very good and fast especially for long keys.
228
229 =item C<1> .. C<4>
230
231 Specify one of up to four custom hash functions (see L<set_hash_function>).
232
233 =back
234
235 =item mutex => $bool (default: C<0>)
236
237 TDB can take advantage of fast interprocess mutexes, which
238 can be orders of magnitude faster than the syscall-based
239 locking used by default. Normally, you need to call
240 C<TDB_FileX::runtime_check_for_robust_mutexes> and set the
241 C<MUTEX_LOCKING> flag if support is indicated.
242
243 This option, when enabled, enables C<MUTEX_LOCKING> if it is supported
244 by the platform - which involves a fork when opening the database. When
245 disabled, it will remove the flag.
246
247 It is recommended to keep this one, but enabling this changes the format
248 of the database, so might not be an option if interoperability with other
249 programs is required.
250
251 =back
252
253 There is no explicit close function. The database is closed implicitly
254 when there are no remaining references.
255
256 =item $tdb->store ($key, $value[, $flag=REPLACE]) [CROAK]
257
258 Store $value in the database with key w$key. The $flag defaults to
259 C<REPLACE>, but can also be C<INSERT> or C<MODIFY>, see tdb_store(3) for
260 details.
261
262 =item $tdb->append ($key, $value) [CROAK]
263
264 Appends $data to the data already stored for $key, or creates a new entry with it.
265
266 =item $tdb->fetch ($key)
267
268 Fetch the value associated with $key, or C<undef> if it is not found (or
269 on any error).
270
271 =item $tdb->delete ($key) [CROAK]
272
273 Delete the value associated with $key.
274
275 On failure, a perl false is returned. See L</$tdb->error> and
276 L</$tdb->errorstr> for the reason.
277
278 =item $bool = $tdb->exists ($key)
279
280 Return true if the $key is found, false otherwise.
281
282 =item $tdb->firstkey
283
284 Return the key of the first value in the database. Returns C<undef> on
285 failure or if there are no keys in the database. See tdb_firstkey(3)
286 for details.
287
288 =item $tdb->nextkey ($lastkey)
289
290 Return the next key in the database after $lastkey. Returns C<undef> on
291 failure or if there are no more keys in the database. See tdb_nextkey(3)
292 for details.
293
294 =item $tdb->error
295
296 Returns the current error state of the C<$tdb> object. See the list of
297 error codes given in L<EXPORTS>.
298
299 =item $tdb->errorstr
300
301 Returns a printable string that describes the error state of the
302 database.
303
304 =item $tdb->reopen [CROAK]
305
306 Closes and reopens the database. Required after a
307 L<fork|perlfunc/fork>, if both processes wish to use the database.
308
309 B<NB:> If C<reopen> fails, then it is unsafe to call any further methods
310 on C<$tdb>. Thus, the only way to find out I<why> C<reopen> failed is to
311 use a logging function.
312
313 =item TDB_FileX::reopen_all [CROAK]
314
315 Closes and reopens all open databases. See L</$tdb->reopen>.
316
317 B<NB:> If C<reopen_all> fails, there is no indication of I<which> C<$tdb>
318 objects failed or why. If you have to survive failures, you may wish to do
319 your own C<reopen> loop instead.
320
321 =item $tdb->traverse ($cb->()) [CROAK] #TODO
322
323 Call $cb for each entry in the database.
324
325 The callback is called with the key and value as arguments and should
326 return a true value if you wish to continue traversal.
327
328 C<traverse> returns the number of elements traversed or C<undef> on
329 error. If $cb is C<undef>, then this function simply counts the number of
330 elements.
331
332 =item $tdb->set_logging_function ($cb->($level, $msg))
333
334 Set the logging function to use when this database object encounters
335 errors.
336
337 $cb is called with the severity level (an integer) and the message (a
338 string).
339
340 =item $tdb->lockall [CROAK]
341
342 Lock an entire database with an exclusive write lock, returning false on
343 error. The purpose of this call is to avoid locking overhead for many
344 operations, but the database has to be unlocked manually when done.
345
346 =item $tdb->unlockall [CROAK]
347
348 Unlock an entire database previously locked with
349 L</$tdb->lockall>.
350
351 =item $tdb->lockall_read [CROAK]
352
353 Same as L</$tdb->lockall>, but uses a shared read lock instead of a writer lock.
354
355 =item $tdb->unlockall_read [CROAK]
356
357 Opposite of L</$tdb->lockall_read>.
358
359 =item $tdb->lockall_mark [CROAK]
360
361 =item $tdb->lockall_unmark [CROAK]
362
363 Please tlel me exactly what these do.
364
365 =item $tdb->lockall_nonblock [CROAK]
366
367 =item $tdb->lockall_read_nonblock [CROAK]
368
369 Please tell me what exactly these do.
370
371 =item $tdb->transaction_start [CROAK]
372
373 Starts a transaction - all operations will be queued, but not applied
374 to the database, until the transaction is either committed L</$tdb->transaction_commit>
375 or aborted/thrown away, with L</$tdb->transaction_cancel>.
376
377 =item $tdb->transaction_start_nonblock [CROAK]
378
379 Please tell me what this does.
380
381 =item $tdb->transaction_commit [CROAK]
382
383 Applies all changes in the transaction.
384
385 =item $tdb->transaction_cancel [CROAK]
386
387 Throws away all changes in the transaction.
388
389 =item $tdb->transaction_prepare_commit [CROAK]
390
391 Instead of calling L</$tdb->transaction_commit> you can do the commit in
392 two phases by calling this method before commit, which does the expensive
393 steps first.
394
395 =item $bool = $tdb->transaction_active
396
397 Returns true if a transaction is currently active.
398
399 =item $tdb->enable_seqnum
400
401 Enables sequence number generatiuon supporet for the database.
402
403 =item $seq = $tdb->get_seqnum
404
405 Returns the current sequence number. Internally, this is a 32 bit unsigned
406 integer, but the API converts it into a native integer, so the same
407 internal sequence number might be represented differently on different
408 machines.
409
410 =item $tdb->increment_seqnum_nonblock
411
412 Increments the sequence number. Note that the sequence number is also
413 incremented by TDB itself oon many operations.
414
415 =item $size = $tdb->hash_size
416
417 Returns the size of the hash table. The size cannot be changed other than
418 by recreating the database.
419
420 =item $octets = $tdb->map_size
421
422 Returns the current mmap size for the database.
423
424 =item $flags = $tdb->get_flags
425
426 Returns the flags for the database (the same as the C<tdb_flags> in C<open>).
427
428 =item $tdb->add_flags ($flag)
429
430 Tried to add flags to the database - yes, the parameter says C<$flag> (singular)
431 but the documentation and the code say I<flags> (plural).
432
433 =item $tdb->remove_flags ($flag)
434
435 Attempts to remove flags from the database.
436
437 =item $fileno = $tdb->fd
438
439 Returns the file descriptor (not file handle) for the underlying database file.
440
441 =item $path = $tdb->name
442
443 Returns the path used to open the database.
444
445 =item $tdb->wipe_all [CROAK]
446
447 Efficientlly deletes all entries in the database. This does not shrink the
448 file itself.
449
450 =item $tdb->repack [CROAK]
451
452 Tries to improve layout of the database by copying all items into a
453 temporary in-memory database, wiping the database, and copying all items
454 back. Yes, everything must fit into memory.
455
456 =item $bool = TDB_FileX::runtime_check_for_robust_mutexes
457
458 Tests whether robust mutexes are available for locking. This involves
459 forking the process, so it can be costly and problematic. This function
460 needs to be called before using the C<MUTEX_LOCKING> flag. But see the
461 C<mutex> parametrer to C<open> for an alternative.
462
463 =item $tdb->dump_all
464
465 Dump the records and freelist to STDOUT in an almost human readable
466 form.
467
468 =item $tdb->printfreelist
469
470 Dump the freelist to STDOUT.
471
472 =back
473
474 =head2 EXPORTS
475
476 Nothing constants are exported by default.
477
478 The tag C<:all> exports allpo of the constants.
479
480 Individually or with the tag C<:flags>:
481
482 DEFAULT
483 CLEAR_IF_FIRST
484 INTERNAL
485 NOLOCK
486 NOMMAP
487 CONVERT
488 BIGENDIAN
489 NOSYNC
490 SEQNUM
491 VOLATILE
492 ALLOW_NESTING
493 DISALLOW_NESTING
494 INCOMPATIBLE_HASH
495 MUTEX_LOCKING
496
497 Individually or with the tag C<:insert>:
498
499 REPLACE
500 INSERT
501 MODIFY
502
503 Individually or with the tag C<:error>:
504
505 SUCCESS
506 ERR_CORRUPT
507 ERR_IO
508 ERR_LOCK
509 ERR_OOM
510 ERR_EXISTS
511 ERR_NOLOCK
512 ERR_LOCK_TIMEOUT
513 ERR_NOEXIST
514 ERR_EINVAL
515 ERR_RDONLY
516
517 Individually or with the tag C<:debug>:
518
519 DEBUG_FATAL
520 DEBUG_ERROR
521 DEBUG_WARNING
522 DEBUG_TRACE
523
524 =head1 UNICODE HANDLING
525
526 TDB databases can only store octet strings. Unlike most other database
527 interfaces, TDB_FileX will safely handle Perl strings by downgrading
528 them. Perl will warn about strings that cnanot be downgraded.
529
530 =head1 DISK USAGE
531
532 TDB databses use 24 octets for every key-value pair, plus the octet size
533 of the key and sata, e.g. the pair "key" => "value" takes up 24+3+5 octets
534 on disk.
535
536 The database header is 168 octets (if I haven't miscounted).
537
538 Each hashtable entry is 4 octets, and one more than the hash size is
539 allocated, so the hahs table size is (hash_size + 1) * 4.
540
541 =head1 LIMITATIONS
542
543 =head2 Database Size
544
545 As far as I can see, TDB dastabases are limited to 4 GB.
546
547 =head2 Hash Functions
548
549 Hash functioons need to be set globally - they are limited to a maximum of
550 4, but this can be easily extended, but requires source code editing. This
551 is due to a limitation of the TDB C API.
552
553 =head2 No recoivery after failed C<reopoen_all>
554
555 There is no way to survive an error during C<reopen_all>.
556 Unfortunately this is a limitation in the TDB C API.
557
558 =head1 SEE ALSO
559
560 tdb(3), L<perltie>.
561
562 =head1 AUTHOR
563
564 Angus Lees, E<lt>gus@inodes.org>
565
566 Currently maintained by Marc A. Lehmann <schmorp@schmorp.de>
567 http://home.schmorp.de/
568
569 =cut