[ViewVC] Diff of: cvs/BDB/README

Comparing BDB/README (file contents):
Revision 1.3 by root, Tue Dec 4 10:14:34 2007 UTC vs.
Revision 1.14 by root, Tue Oct 21 02:21:25 2008 UTC

 NAME
     BDB - Asynchronous Berkeley DB access
 SYNOPSIS
      use BDB;
+     my $env = db_env_create;
+     mkdir "bdtest", 0700;
+     db_env_open
+        $env,
+        "bdtest",
+        BDB::INIT_LOCK | BDB::INIT_LOG | BDB::INIT_MPOOL
+        | BDB::INIT_TXN | BDB::RECOVER | BDB::USE_ENVIRON | BDB::CREATE,
+        0600;
+     $env->set_flags (BDB::AUTO_COMMIT | BDB::TXN_NOSYNC, 1);
+     my $db = db_create $env;
+     db_open $db, undef, "table", undef, BDB::BTREE, BDB::AUTO_COMMIT | BDB::CREATE
+                                         | BDB::READ_UNCOMMITTED, 0600;
+     db_put $db, undef, "key", "data", 0, sub {
+        db_del $db, undef, "key";
+     };
+     db_sync $db;
+     # when you also use Coro, management is easy:
+     use Coro::BDB;
+     # automatic event loop intergration with AnyEvent:
+     use AnyEvent::BDB;
+     # automatic result processing with EV:
+     my $WATCHER = EV::io BDB::poll_fileno, EV::READ, \&BDB::poll_cb;
+     # with Glib:
+     add_watch Glib::IO BDB::poll_fileno,
+               in => sub { BDB::poll_cb; 1 };
+     # or simply flush manually
+     BDB::flush;
 DESCRIPTION
     See the BerkeleyDB documentation
     (<http://www.oracle.com/technology/documentation/berkeley-db/db/index.ht
     ml>). The BDB API is very similar to the C API (the translation has been
         Request has reached the end of its lifetime and holds no resources
         anymore (except possibly for the Perl object, but its connection to
         the actual aio request is severed and calling its methods will
         either do nothing or result in a runtime error).
-  BERKELEYDB FUNCTIONS
+BERKELEYDB FUNCTIONS
     All of these are functions. The create functions simply return a new
-    object and never block. All the remaining functions all take an optional
+    object and never block. All the remaining functions take an optional
-    callback as last argument. If it is missing, then the fucntion will be
+    callback as last argument. If it is missing, then the function will be
-    executed synchronously.
+    executed synchronously. In both cases, $! will reflect the return value
+    of the function.
     BDB functions that cannot block (mostly functions that manipulate
     settings) are method calls on the relevant objects, so the rule of thumb
-    is: if its a method, its not blocking, if its a function, it takes a
+    is: if it's a method, it's not blocking, if it's a function, it takes a
     callback as last argument.
-    In the following, $int signifies an integer return value, "octetstring"
+    In the following, $int signifies an integer return value, "bdb_filename"
-    is a "binary string" (i.e. a perl string with no character indices
+    is a "filename" (octets on unix, madness on windows), "U32" is an
-    >255), "U32" is an unsigned 32 bit integer, "int" is some integer, "NV"
+    unsigned 32 bit integer, "int" is some integer, "NV" is a floating point
-    is a floating point value.
+    value.
-    The "SV *" types are generic perl scalars (for input and output of data
+    Most "SV *" types are generic perl scalars (for input and output of data
-    values), and the "SV *callback" is the optional callback function to
+    values).
-    call when the request is completed.
     The various "DB_ENV" etc. arguments are handles return by
     "db_env_create", "db_create", "txn_begin" and so on. If they have an
     appended "_ornull" this means they are optional and you can pass "undef"
     for them, resulting a NULL pointer on the C level.
+    The "SV *callback" is the optional callback function to call when the
+    request is completed. This last callback argument is special: the
+    callback is simply the last argument passed. If there are "optional"
+    arguments before the callback they can be left out. The callback itself
+    can be left out or specified as "undef", in which case the function will
+    be executed synchronously.
+    For example, "db_env_txn_checkpoint" usually is called with all integer
+    arguments zero. These can be left out, so all of these specify a call to
+    "DB_ENV->txn_checkpoint", to be executed asynchronously with a callback
+    to be called:
+       db_env_txn_checkpoint $db_env, 0, 0, 0, sub { };
+       db_env_txn_checkpoint $db_env, 0, 0, sub { };
+       db_env_txn_checkpoint $db_env, sub { };
+    While these all specify a call to "DB_ENV->txn_checkpoint" to be
+    executed synchronously:
+       db_env_txn_checkpoint $db_env, 0, 0, 0, undef;
+       db_env_txn_checkpoint $db_env, 0, 0, 0;
+       db_env_txn_checkpoint $db_env, 0;
-   BDB functions
+  BDB functions
     Functions in the BDB namespace, exported by default:
        $env = db_env_create (U32 env_flags = 0)
           flags: RPCCLIENT
-       db_env_open (DB_ENV *env, octetstring db_home, U32 open_flags, int mode, SV *callback = &PL_sv_undef)
+       db_env_open (DB_ENV *env, bdb_filename db_home, U32 open_flags, int mode, SV *callback = &PL_sv_undef)
           open_flags: INIT_CDB INIT_LOCK INIT_LOG INIT_MPOOL INIT_REP INIT_TXN RECOVER RECOVER_FATAL USE_ENVIRON USE_ENVIRON_ROOT CREATE LOCKDOWN PRIVATE REGISTER SYSTEM_MEM
        db_env_close (DB_ENV *env, U32 flags = 0, SV *callback = &PL_sv_undef)
        db_env_txn_checkpoint (DB_ENV *env, U32 kbyte = 0, U32 min = 0, U32 flags = 0, SV *callback = &PL_sv_undef)
           flags: FORCE
        db_env_lock_detect (DB_ENV *env, U32 flags = 0, U32 atype = DB_LOCK_DEFAULT, SV *dummy = 0, SV *callback = &PL_sv_undef)
           atype: LOCK_DEFAULT LOCK_EXPIRE LOCK_MAXLOCKS LOCK_MAXWRITE LOCK_MINLOCKS LOCK_MINWRITE LOCK_OLDEST LOCK_RANDOM LOCK_YOUNGEST
        db_env_memp_sync (DB_ENV *env, SV *dummy = 0, SV *callback = &PL_sv_undef)
        db_env_memp_trickle (DB_ENV *env, int percent, SV *dummy = 0, SV *callback = &PL_sv_undef)
+       db_env_dbremove (DB_ENV *env, DB_TXN_ornull *txnid, bdb_filename file, bdb_filename database, U32 flags = 0, SV *callback = &PL_sv_undef)
+       db_env_dbrename (DB_ENV *env, DB_TXN_ornull *txnid, bdb_filename file, bdb_filename database, bdb_filename newname, U32 flags = 0, SV *callback = &PL_sv_undef)
+       db_env_log_archive (DB_ENV *env, SV *listp, U32 flags = 0, SV *callback = &PL_sv_undef)
        $db = db_create (DB_ENV *env = 0, U32 flags = 0)
           flags: XA_CREATE
-       db_open (DB *db, DB_TXN_ornull *txnid, octetstring file, octetstring database, int type, U32 flags, int mode, SV *callback = &PL_sv_undef)
+       db_open (DB *db, DB_TXN_ornull *txnid, bdb_filename file, bdb_filename database, int type, U32 flags, int mode, SV *callback = &PL_sv_undef)
           flags: AUTO_COMMIT CREATE EXCL MULTIVERSION NOMMAP RDONLY READ_UNCOMMITTED THREAD TRUNCATE
        db_close (DB *db, U32 flags = 0, SV *callback = &PL_sv_undef)
           flags: DB_NOSYNC
+       db_upgrade (DB *db, bdb_filename file, U32 flags = 0, SV *callback = &PL_sv_undef)
        db_compact (DB *db, DB_TXN_ornull *txn = 0, SV *start = 0, SV *stop = 0, SV *unused1 = 0, U32 flags = DB_FREE_SPACE, SV *unused2 = 0, SV *callback = &PL_sv_undef)
           flags: FREELIST_ONLY FREE_SPACE
        db_sync (DB *db, U32 flags = 0, SV *callback = &PL_sv_undef)
        db_key_range (DB *db, DB_TXN_ornull *txn, SV *key, SV *key_range, U32 flags = 0, SV *callback = &PL_sv_undef)
        db_put (DB *db, DB_TXN_ornull *txn, SV *key, SV *data, U32 flags = 0, SV *callback = &PL_sv_undef)
           flags: APPEND NODUPDATA NOOVERWRITE
+       db_exists (DB *db, DB_TXN_ornull *txn, SV *key, U32 flags = 0, SV *callback = 0) (v4.6)
        db_get (DB *db, DB_TXN_ornull *txn, SV *key, SV *data, U32 flags = 0, SV *callback = &PL_sv_undef)
           flags: CONSUME CONSUME_WAIT GET_BOTH SET_RECNO MULTIPLE READ_COMMITTED READ_UNCOMMITTED RMW
        db_pget (DB *db, DB_TXN_ornull *txn, SV *key, SV *pkey, SV *data, U32 flags = 0, SV *callback = &PL_sv_undef)
           flags: CONSUME CONSUME_WAIT GET_BOTH SET_RECNO MULTIPLE READ_COMMITTED READ_UNCOMMITTED RMW
        db_del (DB *db, DB_TXN_ornull *txn, SV *key, U32 flags = 0, SV *callback = &PL_sv_undef)
     You can use the "$txn->failed" method to check wether a transaction has
     failed in this way and abort further processing (excluding
     "db_txn_finish").
-   DB_ENV/database environment methods
+  DB_ENV/database environment methods
     Methods available on DB_ENV/$env handles:
        DESTROY (DB_ENV_ornull *env)
                CODE:
                if (env)
        $int = $env->set_data_dir (const char *dir)
        $int = $env->set_tmp_dir (const char *dir)
        $int = $env->set_lg_dir (const char *dir)
        $int = $env->set_shm_key (long shm_key)
        $int = $env->set_cachesize (U32 gbytes, U32 bytes, int ncache = 0)
-       $int = $env->set_flags (U32 flags, int onoff)
+       $int = $env->set_flags (U32 flags, int onoff = 1)
+       $int = $env->log_set_config (U32 flags, int onoff = 1) (v4.7)
+       $int = $env->set_intermediate_dir_mode (const char *modestring) (v4.7)
        $env->set_errfile (FILE *errfile = 0)
        $env->set_msgfile (FILE *msgfile = 0)
        $int = $env->set_verbose (U32 which, int onoff = 1)
        $int = $env->set_encrypt (const char *password, U32 flags = 0)
        $int = $env->set_timeout (NV timeout_seconds, U32 flags = SET_TXN_TIMEOUT)
        $int = $env->set_lk_max_lockers (U32 max)
        $int = $env->set_lk_max_locks (U32 max)
        $int = $env->set_lk_max_objects (U32 max)
        $int = $env->set_lg_bsize (U32 max)
        $int = $env->set_lg_max (U32 max)
+       $int = $env->mutex_set_increment (U32 increment)
+       $int = $env->mutex_set_tas_spins (U32 tas_spins)
+       $int = $env->mutex_set_max (U32 max)
+       $int = $env->mutex_set_align (U32 align)
        $txn = $env->txn_begin (DB_TXN_ornull *parent = 0, U32 flags = 0)
           flags: READ_COMMITTED READ_UNCOMMITTED TXN_NOSYNC TXN_NOWAIT TXN_SNAPSHOT TXN_SYNC TXN_WAIT TXN_WRITE_NOSYNC
+       $txn = $env->cdsgroup_begin; (v4.5)
    Example:
        use AnyEvent;
        use BDB;
           BDB::INIT_LOCK | BDB::INIT_LOG | BDB::INIT_MPOOL | BDB::INIT_TXN | BDB::RECOVER | BDB::USE_ENVIRON | BDB::CREATE,
           0600;
        $env->set_flags (BDB::AUTO_COMMIT | BDB::TXN_NOSYNC, 1);
-   DB/database methods
+  DB/database methods
     Methods available on DB/$db handles:
        DESTROY (DB_ornull *db)
                CODE:
                if (db)
        db_del $db, undef, "key $_" for 1..1000;
        db_sync $db;
-   DB_TXN/transaction methods
+  DB_TXN/transaction methods
     Methods available on DB_TXN/$txn handles:
        DESTROY (DB_TXN_ornull *txn)
                CODE:
                if (txn)
           flags: SET_LOCK_TIMEOUT SET_TXN_TIMEOUT
        $bool = $txn->failed
        # see db_txn_finish documentation, above
-   DBC/cursor methods
+  DBC/cursor methods
     Methods available on DBC/$dbc handles:
        DESTROY (DBC_ornull *dbc)
                CODE:
                if (dbc)
                  dbc->c_close (dbc);
+       $int = $cursor->set_priority ($priority = PRIORITY_*) (v4.6)
    Example:
        my $c = $db->cursor;
        for (;;) {
           last if $!;
        }
        db_c_close $c;
-   DB_SEQUENCE/sequence methods
+  DB_SEQUENCE/sequence methods
     Methods available on DB_SEQUENCE/$seq handles:
        DESTROY (DB_SEQUENCE_ornull *seq)
                CODE:
                if (seq)
           flags: SEQ_DEC SEQ_INC SEQ_WRAP
        $int = $seq->set_range (db_seq_t min, db_seq_t max)
    Example:
        my $seq = $db->sequence;
-       db_sequence_open $seq, undef, "seq", BDB::CREATE;
+   db_sequence_open $seq, undef, "seq", BDB::CREATE;
        db_sequence_get $seq, undef, 1, my $value;
-  SUPPORT FUNCTIONS
+SUPPORT FUNCTIONS
-   EVENT PROCESSING AND EVENT LOOP INTEGRATION
+  EVENT PROCESSING AND EVENT LOOP INTEGRATION
+    $msg = BDB::strerror [$errno]
+        Returns the string corresponding to the given errno value. If no
+        argument is given, use $!.
+        Note that the BDB module also patches the $! variable directly, so
+        you should be able to get a bdb error string by simply stringifying
+        $!.
     $fileno = BDB::poll_fileno
         Return the *request result pipe file descriptor*. This filehandle
         must be polled for reading by some mechanism outside this module
         (e.g. Event or select, see below or the SYNOPSIS). If the pipe
         becomes readable you have to call "poll_cb" to check the results.
         interactiveness when perl is not fast enough to process all requests
         in time.
         For interactive programs, values such as 0.01 to 0.1 should be fine.
-        Example: Install an Event watcher that automatically calls
+        Example: Install an EV watcher that automatically calls BDB::poll_cb
-        BDB::poll_cb with low priority, to ensure that other parts of the
+        with low priority, to ensure that other parts of the program get the
-        program get the CPU sometimes even under high AIO load.
+        CPU sometimes even under high load.
            # try not to spend much more than 0.1s in poll_cb
            BDB::max_poll_time 0.1;
-           # use a low priority so other tasks have priority
+           my $bdb_poll = EV::io BDB::poll_fileno, EV::READ, \&BDB::poll_cb);
-           Event->io (fd => BDB::poll_fileno,
-                      poll => 'r', nice => 1,
-                      cb => &BDB::poll_cb);
     BDB::poll_wait
         If there are any outstanding requests and none of them in the result
         phase, wait till the result filehandle becomes ready for reading
         (simply does a "select" on the filehandle. This is useful if you
         equivalent to:
            BDB::poll_wait, BDB::poll_cb
     BDB::flush
-        Wait till all outstanding AIO requests have been handled.
+        Wait till all outstanding BDB requests have been handled.
         Strictly equivalent to:
            BDB::poll_wait, BDB::poll_cb
               while BDB::nreqs;
+  VERSION CHECKING
+    BerkeleyDB comes in various versions, many of them have minor
+    incompatibilities. This means that traditional "at least version x.x"
+    checks are often not sufficient.
+    Example: set the log_autoremove option in a way compatible with <v4.7
+    and v4.7. Note the use of & on the constants to avoid triggering a
+    compiletime bug when the symbol isn't available.
+       $DB_ENV->set_flags      (&BDB::LOG_AUTOREMOVE ) if BDB::VERSION v0, v4.7;
+       $DB_ENV->log_set_config (&BDB::LOG_AUTO_REMOVE) if BDB::VERSION v4.7;
+    BDB::VERSION
+        The "BDB::VERSION" function, when called without arguments, returns
+        the Berkeley DB version as a v-string (usually with 3 components).
+        You should use "lt" and "ge" operators exclusively to make
+        comparisons.
+        Example: check for at least version 4.7.
+           BDB::VERSION ge v4.7 or die;
+    BDB::VERSION min-version
+        Returns true if the BDB version is at least the given version
+        (specified as a v-string), false otherwise.
+        Example: check for at least version 4.5.
+           BDB::VERSION v4.7 or die;
+    BDB::VERSION min-version, max-version
+        Returns true of the BDB version is at least version "min-version"
+        (specify "undef" or "v0" for any minimum version) and less then
+        "max-version".
+        Example: check wether version is strictly less then v4.7.
+           BDB::VERSION v0, v4.7
+              or die "version 4.7 is not yet supported";
-   CONTROLLING THE NUMBER OF THREADS
+  CONTROLLING THE NUMBER OF THREADS
     BDB::min_parallel $nthreads
-        Set the minimum number of AIO threads to $nthreads. The current
+        Set the minimum number of BDB threads to $nthreads. The current
         default is 8, which means eight asynchronous operations can execute
         concurrently at any one time (the number of outstanding requests,
         however, is unlimited).
-        BDB starts threads only on demand, when an AIO request is queued and
+        BDB starts threads only on demand, when an BDB request is queued and
         no free thread exists. Please note that queueing up a hundred
         requests can create demand for a hundred threads, even if it turns
         out that everything is in the cache and could have been processed
         faster by a single thread.
         Under most circumstances you don't need to call this function, as
         the module selects a default that is suitable for low to moderate
         load.
     BDB::max_parallel $nthreads
-        Sets the maximum number of AIO threads to $nthreads. If more than
+        Sets the maximum number of BDB threads to $nthreads. If more than
         the specified number of threads are currently running, this function
         kills them. This function blocks until the limit is reached.
         While $nthreads are zero, aio requests get queued but not executed
         until the number of threads has been increased again.
         You can still queue as many requests as you want. Therefore,
         "max_oustsanding" is mainly useful in simple scripts (with low
         values) or as a stop gap to shield against fatal memory overflow
         (with large values).
-    BDB::set_sync_prepare $cb
+    $old_cb = BDB::set_sync_prepare $cb
         Sets a callback that is called whenever a request is created without
         an explicit callback. It has to return two code references. The
-        first is used as the request callback, and the second is called to
+        first is used as the request callback (it should save the return
-        wait until the first callback has been called. The default
+        status), and the second is called to wait until the first callback
-        implementation works like this:
+        has been called (it must set $! to the return status).
+        This mechanism can be used to include BDB into other event
+        mechanisms, such as Coro::BDB.
+        To allow other, callback-based, events to be executed while
+        callback-less ones are run, you could use this sync prepare
+        function:
            sub {
               my $status;
               (
                  sub { $status = $! },
                  sub { BDB::poll while !defined $status; $! = $status },
               )
            }
+        It works by polling for results till the request has finished and
+        then sets $! to the return value. This means that if you don't use a
+        callback, BDB would simply fall back to synchronous operations.
+        By default, or if the sync prepare function is set to "undef", is to
+        execute callback-less BDB requests in the foreground thread, setting
+        $! to the return value, without polling for other events.
-   STATISTICAL INFORMATION
+  STATISTICAL INFORMATION
     BDB::nreqs
         Returns the number of requests currently in the ready, execute or
         pending states (i.e. for which their callback has not been invoked
         yet).
     BDB::npending
         Returns the number of requests currently in the pending state
         (executed, but not yet processed by poll_cb).
+COMMON PITFALLS
+  Unexpected Crashes
+    Remember that, by default, BDB will execute requests in parallel, in
+    somewhat random order. That means that it is easy to run a "db_get"
+    request on thesa me database as a concurrent "db_close" request, leading
+    to a crash, silent data corruption, eventually the next world war on
+    terrorism.
+    If you only ever use foreground requests (without a callback), this will
+    not be an issue.
+  Unexpected Freezes or Deadlocks
+    Remember that, by default, BDB will execute requests in parallel, which
+    easily leads to deadlocks (even concurrent put's on the same database
+    can deadlock).
+    You either need to run deadlock detection (and handle the resulting
+    errors), or make sure only one process ever updates the database, ine
+    one thread, e.g. by using only foreground requests (without a callback).
-  FORK BEHAVIOUR
+FORK BEHAVIOUR
     This module should do "the right thing" when the process using it forks:
-    Before the fork, IO::AIO enters a quiescent state where no requests can
+    Before the fork, BDB enters a quiescent state where no requests can be
-    be added in other threads and no results will be processed. After the
+    added in other threads and no results will be processed. After the fork
-    fork the parent simply leaves the quiescent state and continues
+    the parent simply leaves the quiescent state and continues
     request/result processing, while the child frees the request/result
     queue (so that the requests started before the fork will only be handled
     in the parent). Threads will be started on demand until the limit set in
     the parent process has been reached again.
     In short: the parent will, after a short pause, continue as if fork had
-    not been called, while the child will act as if IO::AIO has not been
+    not been called, while the child will act as if BDB has not been used
-    used yet.
+    yet.
+    Win32 note: there is no fork on win32, and perls emulation of it is too
+    broken to be supported, so do not use BDB in a windows pseudo-fork,
+    better yet, switch to a more capable platform.
-  MEMORY USAGE
+MEMORY USAGE
     Per-request usage:
     Each aio request uses - depending on your architecture - around 100-200
     bytes of memory. In addition, stat requests need a stat buffer (possibly
     a few hundred bytes), readdir requires a result buffer and so on. Perl
     In the execution phase, some aio requests require more memory for
     temporary buffers, and each thread requires a stack and other data
     structures (usually around 16k-128k, depending on the OS).
+WIN32 FILENAMES/DATABASE NAME MESS
+    Perl on Win32 supports only ASCII filenames (the reason is that it
+    abuses an internal flag to store wether a filename is Unicode or ANSI,
+    but that flag is used for somethign else in the perl core, so there is
+    no way to detect wether a filename is ANSI or Unicode-encoded). The BDB
+    module tries to work around this issue by assuming that the filename is
+    an ANSI filename and BDB was built for unicode support.
 KNOWN BUGS
     Known bugs will be fixed in the next release, except:
        If you use a transaction in any request, and the request returns
        with an operating system error or DB_LOCK_NOTGRANTED, the internal
        TXN_DEADLOCK flag will be set on the transaction. See C<db_txn_finish>,
        above.
 SEE ALSO
-    Coro::AIO.
+    AnyEvent::BDB (event loop integration), Coro::BDB (more natural syntax),
+    IO::AIO (nice to have).
 AUTHOR
      Marc Lehmann <schmorp@schmorp.de>
      http://home.schmorp.de/

Diff Legend

-–
+Removed lines
-+
+Added lines
-<
+Changed lines
->
+Changed lines

Comparing BDB/README (file contents): Revision 1.3 by root, Tue Dec 4 10:14:34 2007 UTC vs. Revision 1.14 by root, Tue Oct 21 02:21:25 2008 UTC

Diff Legend

Comparing BDB/README (file contents):
Revision 1.3 by root, Tue Dec 4 10:14:34 2007 UTC vs.
Revision 1.14 by root, Tue Oct 21 02:21:25 2008 UTC