[ViewVC] Diff of: cvs/IO-AIO/README

Comparing IO-AIO/README (file contents):
Revision 1.45 by root, Thu Dec 30 07:19:31 2010 UTC vs.
Revision 1.48 by root, Wed Jun 29 11:25:17 2011 UTC

        IO::AIO::max_poll_reqs $nreqs
        IO::AIO::max_poll_time $seconds
        IO::AIO::min_parallel $nthreads
        IO::AIO::max_parallel $nthreads
        IO::AIO::max_idle $nthreads
+       IO::AIO::idle_timeout $seconds
        IO::AIO::max_outstanding $maxreqs
        IO::AIO::nreqs
        IO::AIO::nready
        IO::AIO::npending
               } else {
                  die "open failed: $!\n";
               }
            };
+        In addition to all the common open modes/flags ("O_RDONLY",
+        "O_WRONLY", "O_RDWR", "O_CREAT", "O_TRUNC", "O_EXCL" and
+        "O_APPEND"), the following POSIX and non-POSIX constants are
+        available (missing ones on your system are, as usual, 0):
+        "O_ASYNC", "O_DIRECT", "O_NOATIME", "O_CLOEXEC", "O_NOCTTY",
+        "O_NOFOLLOW", "O_NONBLOCK", "O_EXEC", "O_SEARCH", "O_DIRECTORY",
+        "O_DSYNC", "O_RSYNC", "O_SYNC" and "O_TTY_INIT".
     aio_close $fh, $callback->($status)
         Asynchronously close a file and call the callback with the result
         code.
         Unfortunately, you can't do this to perl. Perl *insists* very
     aio_sendfile $out_fh, $in_fh, $in_offset, $length, $callback->($retval)
         Tries to copy $length bytes from $in_fh to $out_fh. It starts
         reading at byte offset $in_offset, and starts writing at the current
         file offset of $out_fh. Because of that, it is not safe to issue
         more than one "aio_sendfile" per $out_fh, as they will interfere
-        with each other.
+        with each other. The same $in_fh works fine though, as this function
+        does not move or use the file offset of $in_fh.
         Please note that "aio_sendfile" can read more bytes from $in_fh than
-        are written, and there is no way to find out how many bytes have
+        are written, and there is no way to find out how many more bytes
-        been read from "aio_sendfile" alone, as "aio_sendfile" only provides
+        have been read from "aio_sendfile" alone, as "aio_sendfile" only
-        the number of bytes written to $out_fh. Only if the result value
+        provides the number of bytes written to $out_fh. Only if the result
-        equals $length one can assume that $length bytes have been read.
+        value equals $length one can assume that $length bytes have been
+        read.
         Unlike with other "aio_" functions, it makes a lot of sense to use
         "aio_sendfile" on non-blocking sockets, as long as one end
         (typically the $in_fh) is a file - the file I/O will then be
         asynchronous, while the socket I/O will be non-blocking. Note,
         however, that you can run into a trap where "aio_sendfile" reads
         some data with readahead, then fails to write all data, and when the
         socket is ready the next time, the data in the cache is already
         lost, forcing "aio_sendfile" to again hit the disk. Explicit
-        "aio_read" + "aio_write" let's you control resource usage much
+        "aio_read" + "aio_write" let's you better control resource usage.
-        better.
-        This call tries to make use of a native "sendfile" syscall to
+        This call tries to make use of a native "sendfile"-like syscall to
         provide zero-copy operation. For this to work, $out_fh should refer
         to a socket, and $in_fh should refer to an mmap'able file.
         If a native sendfile cannot be found or it fails with "ENOSYS",
-        "ENOTSUP", "EOPNOTSUPP", "EAFNOSUPPORT", "EPROTOTYPE" or "ENOTSOCK",
+        "EINVAL", "ENOTSUP", "EOPNOTSUPP", "EAFNOSUPPORT", "EPROTOTYPE" or
-        it will be emulated, so you can call "aio_sendfile" on any type of
+        "ENOTSOCK", it will be emulated, so you can call "aio_sendfile" on
-        filehandle regardless of the limitations of the operating system.
+        any type of filehandle regardless of the limitations of the
+        operating system.
+        As native sendfile syscalls (as practically any non-POSIX interface
+        hacked together in a hurry to improve benchmark numbers) tend to be
+        rather buggy on many systems, this implementation tries to work
+        around some known bugs in Linux and FreeBSD kernels (probably
+        others, too), but that might fail, so you really really should check
+        the return value of "aio_sendfile" - fewre bytes than expected might
+        have been transferred.
     aio_readahead $fh,$offset,$length, $callback->($retval)
         "aio_readahead" populates the page cache with data from a file so
         that subsequent reads from that file will not block on disk I/O. The
         $offset argument specifies the starting point from which data is to
         Currently, the stats are always 64-bit-stats, i.e. instead of
         returning an error when stat'ing a large file, the results will be
         silently truncated unless perl itself is compiled with large file
         support.
+        To help interpret the mode and dev/rdev stat values, IO::AIO offers
+        the following constants and functions (if not implemented, the
+        constants will be 0 and the functions will either "croak" or fall
+        back on traditional behaviour).
+        "S_IFMT", "S_IFIFO", "S_IFCHR", "S_IFBLK", "S_IFLNK", "S_IFREG",
+        "S_IFDIR", "S_IFWHT", "S_IFSOCK", "IO::AIO::major $dev_t",
+        "IO::AIO::minor $dev_t", "IO::AIO::makedev $major, $minor".
         Example: Print the length of /etc/passwd:
            aio_stat "/etc/passwd", sub {
               $_[0] and die "stat failed: $!";
         The only (POSIX-) portable way of calling this function is:
            aio_mknod $path, IO::AIO::S_IFIFO | $mode, 0, sub { ...
+        See "aio_stat" for info about some potentially helpful extra
+        constants and functions.
     aio_link $srcpath, $dstpath, $callback->($status)
         Asynchronously create a new link to the existing object at $srcpath
         at the path $dstpath and call the callback with the result code.
     aio_symlink $srcpath, $dstpath, $callback->($status)
         The flags are a combination of the following constants, ORed
         together (the flags will also be passed to the callback, possibly
         modified):
         IO::AIO::READDIR_DENTS
-            When this flag is off, then the callback gets an arrayref with
+            When this flag is off, then the callback gets an arrayref
-            of names only (as with "aio_readdir"), otherwise it gets an
+            consisting of names only (as with "aio_readdir"), otherwise it
-            arrayref with "[$name, $type, $inode]" arrayrefs, each
+            gets an arrayref with "[$name, $type, $inode]" arrayrefs, each
             describing a single directory entry in more detail.
             $name is the name of the entry.
             $type is one of the "IO::AIO::DT_xxx" constants:
             unspecified content on systems that do not deliver the inode
             information.
         IO::AIO::READDIR_DIRS_FIRST
             When this flag is set, then the names will be returned in an
-            order where likely directories come first. This is useful when
+            order where likely directories come first, in optimal stat
-            you need to quickly find directories, or you want to find all
+            order. This is useful when you need to quickly find directories,
-            directories while avoiding to stat() each entry.
+            or you want to find all directories while avoiding to stat()
+            each entry.
             If the system returns type information in readdir, then this is
             used to find directories directly. Otherwise, likely directories
-            are files beginning with ".", or otherwise files with no dots,
+            are names beginning with ".", or otherwise names with no dots,
-            of which files with short names are tried first.
+            of which names with short names are tried first.
         IO::AIO::READDIR_STAT_ORDER
             When this flag is set, then the names will be returned in an
             order suitable for stat()'ing each one. That is, when you plan
             to stat() all files in the given directory, then the returned
         See "poll_cb" for an example.
     IO::AIO::poll_cb
         Process some outstanding events on the result pipe. You have to call
-        this regularly. Returns 0 if all events could be processed, or -1 if
+        this regularly. Returns 0 if all events could be processed (or there
-        it returned earlier for whatever reason. Returns immediately when no
+        were no events to process), or -1 if it returned earlier for
-        events are outstanding. The amount of events processed depends on
+        whatever reason. Returns immediately when no events are outstanding.
-        the settings of "IO::AIO::max_poll_req" and
+        The amount of events processed depends on the settings of
-        "IO::AIO::max_poll_time".
+        "IO::AIO::max_poll_req" and "IO::AIO::max_poll_time".
         If not all requests were processed for whatever reason, the
         filehandle will still be ready when "poll_cb" returns, so normally
         you don't have to do anything special to have it called later.
+        Apart from calling "IO::AIO::poll_cb" when the event filehandle
+        becomes ready, it can be beneficial to call this function from loops
+        which submit a lot of requests, to make sure the results get
+        processed when they become available and not just when the loop is
+        finished and the event loop takes over again. This function returns
+        very fast when there are no outstanding requests.
         Example: Install an Event watcher that automatically calls
         IO::AIO::poll_cb with high priority (more examples can be found in
         the SYNOPSIS section, at the top of this document):
         Under normal circumstances you don't need to call this function.
     IO::AIO::max_idle $nthreads
         Limit the number of threads (default: 4) that are allowed to idle
-        (i.e., threads that did not get a request to process within 10
+        (i.e., threads that did not get a request to process within the idle
-        seconds). That means if a thread becomes idle while $nthreads other
+        timeout (default: 10 seconds). That means if a thread becomes idle
-        threads are also idle, it will free its resources and exit.
+        while $nthreads other threads are also idle, it will free its
+        resources and exit.
         This is useful when you allow a large number of threads (e.g. 100 or
         1000) to allow for extremely high load situations, but want to free
         resources under normal circumstances (1000 threads can easily
         consume 30MB of RAM).
         The default is probably ok in most situations, especially if thread
         creation is fast. If thread creation is very slow on your system you
         might want to use larger values.
+    IO::AIO::idle_timeout $seconds
+        Sets the minimum idle timeout (default 10) after which worker
+        threads are allowed to exit. SEe "IO::AIO::max_idle".
     IO::AIO::max_outstanding $maxreqs
+        Sets the maximum number of outstanding requests to $nreqs. If you do
+        queue up more than this number of requests, the next call to
+        "IO::AIO::poll_cb" (and other functions calling "poll_cb", such as
+        "IO::AIO::flush" or "IO::AIO::poll") will block until the limit is
+        no longer exceeded.
+        In other words, this setting does not enforce a queue limit, but can
+        be used to make poll functions block if the limit is exceeded.
         This is a very bad function to use in interactive programs because
         it blocks, and a bad way to reduce concurrency because it is
         inexact: Better use an "aio_group" together with a feed callback.
-        Sets the maximum number of outstanding requests to $nreqs. If you do
+        It's main use is in scripts without an event loop - when you want to
-        queue up more than this number of requests, the next call to the
+        stat a lot of files, you can write somehting like this:
-        "poll_cb" (and "poll_some" and other functions calling "poll_cb")
-        function will block until the limit is no longer exceeded.
-        The default value is very large, so there is no practical limit on
+           IO::AIO::max_outstanding 32;
+           for my $path (...) {
+              aio_stat $path , ...;
+              IO::AIO::poll_cb;
+           }
+           IO::AIO::flush;
+        The call to "poll_cb" inside the loop will normally return
+        instantly, but as soon as more thna 32 reqeusts are in-flight, it
+        will block until some requests have been handled. This keeps the
+        loop from pushing a large number of "aio_stat" requests onto the
+        queue.
+        The default value for "max_outstanding" is very large, so there is
-        the number of outstanding requests.
+        no practical limit on the number of outstanding requests.
-        You can still queue as many requests as you want. Therefore,
-        "max_outstanding" is mainly useful in simple scripts (with low
-        values) or as a stop gap to shield against fatal memory overflow
-        (with large values).
    STATISTICAL INFORMATION
     IO::AIO::nreqs
         Returns the number of requests currently in the ready, execute or
         pending states (i.e. for which their callback has not been invoked
      # Danga::Socket integration
      Danga::Socket->AddOtherFds (IO::AIO::poll_fileno =>
                                  \&IO::AIO::poll_cb);
   FORK BEHAVIOUR
-    This module should do "the right thing" when the process using it forks:
+    Usage of pthreads in a program changes the semantics of fork
+    considerably. Specifically, only async-safe functions can be called
+    after fork. Perl doesn't know about this, so in general, you cannot call
+    fork with defined behaviour in perl. IO::AIO uses pthreads, so this
+    applies, but many other extensions and (for inexplicable reasons) perl
+    itself often is linked against pthreads, so this limitation applies.
-    Before the fork, IO::AIO enters a quiescent state where no requests can
+    Some operating systems have extensions that allow safe use of fork, and
-    be added in other threads and no results will be processed. After the
+    this module should do "the right thing" on those, and tries on others.
-    fork the parent simply leaves the quiescent state and continues
+    At the time of this writing (2011) only GNU/Linux supports these
-    request/result processing, while the child frees the request/result
+    extensions to POSIX.
-    queue (so that the requests started before the fork will only be handled
-    in the parent). Threads will be started on demand until the limit set in
-    the parent process has been reached again.
-    In short: the parent will, after a short pause, continue as if fork had
-    not been called, while the child will act as if IO::AIO has not been
-    used yet.
   MEMORY USAGE
     Per-request usage:
     Each aio request uses - depending on your architecture - around 100-200

Diff Legend

-–
+Removed lines
-+
+Added lines
-<
+Changed lines
->
+Changed lines

Comparing IO-AIO/README (file contents): Revision 1.45 by root, Thu Dec 30 07:19:31 2010 UTC vs. Revision 1.48 by root, Wed Jun 29 11:25:17 2011 UTC

Diff Legend

Comparing IO-AIO/README (file contents):
Revision 1.45 by root, Thu Dec 30 07:19:31 2010 UTC vs.
Revision 1.48 by root, Wed Jun 29 11:25:17 2011 UTC