[ViewVC] Diff of: cvs/IO-AIO/AIO.pm

Comparing IO-AIO/AIO.pm (file contents):
Revision 1.204 by root, Mon Jul 18 02:02:26 2011 UTC vs.
Revision 1.211 by root, Tue Sep 27 12:10:29 2011 UTC

 use common::sense;
 use base 'Exporter';
 BEGIN {
-   our $VERSION = '3.93';
+   our $VERSION = '4.0';
    our @AIO_REQ = qw(aio_sendfile aio_read aio_write aio_open aio_close
                      aio_stat aio_lstat aio_unlink aio_rmdir aio_readdir aio_readdirx
-                     aio_scandir aio_symlink aio_readlink aio_realpath aio_sync aio_fsync
+                     aio_scandir aio_symlink aio_readlink aio_realpath aio_sync
-                     aio_fdatasync aio_sync_file_range aio_fallocate
+                     aio_fsync aio_syncfs aio_fdatasync aio_sync_file_range aio_fallocate
                      aio_pathsync aio_readahead
                      aio_rename aio_link aio_move aio_copy aio_group
                      aio_nop aio_mknod aio_load aio_rmtree aio_mkdir aio_chown
                      aio_chmod aio_utime aio_truncate
                      aio_msync aio_mtouch aio_mlock aio_mlockall
-                     aio_statvfs);
+                     aio_statvfs
+                     aio_wd);
    our @EXPORT = (@AIO_REQ, qw(aioreq_pri aioreq_nice));
    our @EXPORT_OK = qw(poll_fileno poll_cb poll_wait flush
                        min_parallel max_parallel max_idle idle_timeout
                        nreqs nready npending nthreads
 This section simply lists the prototypes of the most important functions
 for quick reference. See the following sections for function-by-function
 documentation.
+   aio_wd $pathname, $callback->($wd)
    aio_open $pathname, $flags, $mode, $callback->($fh)
    aio_close $fh, $callback->($status)
    aio_read  $fh,$offset,$length, $data,$dataoffset, $callback->($retval)
    aio_write $fh,$offset,$length, $data,$dataoffset, $callback->($retval)
    aio_sendfile $out_fh, $in_fh, $in_offset, $length, $callback->($retval)
    aio_utime $fh_or_path, $atime, $mtime, $callback->($status)
    aio_chown $fh_or_path, $uid, $gid, $callback->($status)
    aio_truncate $fh_or_path, $offset, $callback->($status)
    aio_chmod $fh_or_path, $mode, $callback->($status)
    aio_unlink $pathname, $callback->($status)
-   aio_mknod $path, $mode, $dev, $callback->($status)
+   aio_mknod $pathname, $mode, $dev, $callback->($status)
    aio_link $srcpath, $dstpath, $callback->($status)
    aio_symlink $srcpath, $dstpath, $callback->($status)
-   aio_readlink $path, $callback->($link)
+   aio_readlink $pathname, $callback->($link)
-   aio_realpath $path, $callback->($link)
+   aio_realpath $pathname, $callback->($link)
    aio_rename $srcpath, $dstpath, $callback->($status)
    aio_mkdir $pathname, $mode, $callback->($status)
    aio_rmdir $pathname, $callback->($status)
    aio_readdir $pathname, $callback->($entries)
    aio_readdirx $pathname, $flags, $callback->($entries, $flags)
       IO::AIO::READDIR_DENTS IO::AIO::READDIR_DIRS_FIRST
       IO::AIO::READDIR_STAT_ORDER IO::AIO::READDIR_FOUND_UNKNOWN
-   aio_load $path, $data, $callback->($status)
+   aio_load $pathname, $data, $callback->($status)
    aio_copy $srcpath, $dstpath, $callback->($status)
    aio_move $srcpath, $dstpath, $callback->($status)
-   aio_scandir $path, $maxreq, $callback->($dirs, $nondirs)
+   aio_scandir $pathname, $maxreq, $callback->($dirs, $nondirs)
-   aio_rmtree $path, $callback->($status)
+   aio_rmtree $pathname, $callback->($status)
    aio_sync $callback->($status)
+   aio_syncfs $fh, $callback->($status)
    aio_fsync $fh, $callback->($status)
    aio_fdatasync $fh, $callback->($status)
    aio_sync_file_range $fh, $offset, $nbytes, $flags, $callback->($status)
-   aio_pathsync $path, $callback->($status)
+   aio_pathsync $pathname, $callback->($status)
    aio_msync $scalar, $offset = 0, $length = undef, flags = 0, $callback->($status)
    aio_mtouch $scalar, $offset = 0, $length = undef, flags = 0, $callback->($status)
    aio_mlock $scalar, $offset = 0, $length = undef, $callback->($status)
    aio_mlockall $flags, $callback->($status)
    aio_group $callback->(...)
 internally until the request has finished.
 All functions return request objects of type L<IO::AIO::REQ> that allow
 further manipulation of those requests while they are in-flight.
-The pathnames you pass to these routines I<must> be absolute and
+The pathnames you pass to these routines I<should> be absolute. The
-encoded as octets. The reason for the former is that at the time the
+reason for this is that at the time the request is being executed, the
-request is being executed, the current working directory could have
+current working directory could have changed. Alternatively, you can make
-changed. Alternatively, you can make sure that you never change the
+sure that you never change the current working directory anywhere in
-current working directory anywhere in the program and then use relative
+the program and then use relative paths. Lastly, you can take advantage
-paths.
+of IO::AIOs working directory abstraction - see the description of the
+C<IO::AIO::WD> class later in this document.
 To encode pathnames as octets, either make sure you either: a) always pass
 in filenames you got from outside (command line, readdir etc.) without
 tinkering, b) are ASCII or ISO 8859-1, c) use the Encode module and encode
 your pathnames to the locale (or other) encoding in effect in the user
 Asynchronously unlink (delete) a file and call the callback with the
 result code.
-=item aio_mknod $path, $mode, $dev, $callback->($status)
+=item aio_mknod $pathname, $mode, $dev, $callback->($status)
 [EXPERIMENTAL]
 Asynchronously create a device node (or fifo). See mknod(2).
 The only (POSIX-) portable way of calling this function is:
-   aio_mknod $path, IO::AIO::S_IFIFO | $mode, 0, sub { ...
+   aio_mknod $pathname, IO::AIO::S_IFIFO | $mode, 0, sub { ...
 See C<aio_stat> for info about some potentially helpful extra constants
 and functions.
 =item aio_link $srcpath, $dstpath, $callback->($status)
 Asynchronously create a new symbolic link to the existing object at C<$srcpath> at
 the path C<$dstpath> and call the callback with the result code.
-=item aio_readlink $path, $callback->($link)
+=item aio_readlink $pathname, $callback->($link)
 Asynchronously read the symlink specified by C<$path> and pass it to
 the callback. If an error occurs, nothing or undef gets passed to the
 callback.
-=item aio_realpath $path, $callback->($path)
+=item aio_realpath $pathname, $callback->($path)
 Asynchronously make the path absolute and resolve any symlinks in
 C<$path>. The resulting path only consists of directories (Same as
 L<Cwd::realpath>).
 array-ref with the filenames.
 =item aio_readdirx $pathname, $flags, $callback->($entries, $flags)
-Quite similar to C<aio_readdir>, but the C<$flags> argument allows to tune
+Quite similar to C<aio_readdir>, but the C<$flags> argument allows one to
-behaviour and output format. In case of an error, C<$entries> will be
+tune behaviour and output format. In case of an error, C<$entries> will be
 C<undef>.
 The flags are a combination of the following constants, ORed together (the
 flags will also be passed to the callback, possibly modified):
 =item IO::AIO::READDIR_FOUND_UNKNOWN
 This flag should not be set when calling C<aio_readdirx>. Instead, it
 is being set by C<aio_readdirx>, when any of the C<$type>'s found were
-C<IO::AIO::DT_UNKNOWN>. The absense of this flag therefore indicates that all
+C<IO::AIO::DT_UNKNOWN>. The absence of this flag therefore indicates that all
 C<$type>'s are known, which can be used to speed up some algorithms.
 =back
-=item aio_load $path, $data, $callback->($status)
+=item aio_load $pathname, $data, $callback->($status)
 This is a composite request that tries to fully load the given file into
 memory. Status is the same as with aio_read.
 =cut
    };
    $grp
 }
-=item aio_scandir $path, $maxreq, $callback->($dirs, $nondirs)
+=item aio_scandir $pathname, $maxreq, $callback->($dirs, $nondirs)
 Scans a directory (similar to C<aio_readdir>) but additionally tries to
 efficiently separate the entries of directory C<$path> into two sets of
 names, directories you can recurse into (directories), and ones you cannot
 recurse into (everything else, including symlinks to directories).
 Then entries will be sorted into likely directories a non-initial dot
 currently) and likely non-directories (see C<aio_readdirx>). Then every
 entry plus an appended C</.> will be C<stat>'ed, likely directories first,
 in order of their inode numbers. If that succeeds, it assumes that the
 entry is a directory or a symlink to directory (which will be checked
-seperately). This is often faster than stat'ing the entry itself because
+separately). This is often faster than stat'ing the entry itself because
 filesystems might detect the type of the entry without reading the inode
 data (e.g. ext2fs filetype feature), even on systems that cannot return
 the filetype information on readdir.
 If the known number of directories (link count - 2) has been reached, the
    my $grp = aio_group $cb;
    $maxreq = 4 if $maxreq <= 0;
-   # stat once
+   # get a wd object
    aioreq_pri $pri;
-   add $grp aio_stat $path, sub {
+   add $grp aio_wd $path, sub {
-      return $grp->result () if $_[0];
+      my $wd = [shift, "."];
-      my $now = time;
-      my $hash1 = join ":", (stat _)[0,1,3,7,9];
-      # read the directory entries
+      # stat once
       aioreq_pri $pri;
-      add $grp aio_readdirx $path, READDIR_DIRS_FIRST, sub {
+      add $grp aio_stat $wd, sub {
-         my $entries = shift
-            or return $grp->result ();
+         return $grp->result () if $_[0];
+         my $now = time;
+         my $hash1 = join ":", (stat _)[0,1,3,7,9];
-         # stat the dir another time
+         # read the directory entries
          aioreq_pri $pri;
+         add $grp aio_readdirx $wd, READDIR_DIRS_FIRST, sub {
+            my $entries = shift
+               or return $grp->result ();
+            # stat the dir another time
+            aioreq_pri $pri;
-         add $grp aio_stat $path, sub {
+            add $grp aio_stat $wd, sub {
-            my $hash2 = join ":", (stat _)[0,1,3,7,9];
+               my $hash2 = join ":", (stat _)[0,1,3,7,9];
-            my $ndirs;
+               my $ndirs;
-            # take the slow route if anything looks fishy
+               # take the slow route if anything looks fishy
-            if ($hash1 ne $hash2 or (stat _)[9] == $now) {
+               if ($hash1 ne $hash2 or (stat _)[9] == $now) {
-               $ndirs = -1;
+                  $ndirs = -1;
-            } else {
+               } else {
-               # if nlink == 2, we are finished
+                  # if nlink == 2, we are finished
-               # for non-posix-fs's, we rely on nlink < 2
+                  # for non-posix-fs's, we rely on nlink < 2
-               $ndirs = (stat _)[3] - 2
+                  $ndirs = (stat _)[3] - 2
-                  or return $grp->result ([], $entries);
+                     or return $grp->result ([], $entries);
-            }
+               }
-            my (@dirs, @nondirs);
+               my (@dirs, @nondirs);
-            my $statgrp = add $grp aio_group sub {
+               my $statgrp = add $grp aio_group sub {
-               $grp->result (\@dirs, \@nondirs);
+                  $grp->result (\@dirs, \@nondirs);
-            };
+               };
-            limit $statgrp $maxreq;
+               limit $statgrp $maxreq;
-            feed $statgrp sub {
+               feed $statgrp sub {
-               return unless @$entries;
+                  return unless @$entries;
-               my $entry = shift @$entries;
+                  my $entry = shift @$entries;
-               aioreq_pri $pri;
+                  aioreq_pri $pri;
+                  $wd->[1] = "$entry/.";
-               add $statgrp aio_stat "$path/$entry/.", sub {
+                  add $statgrp aio_stat $wd, sub {
-                  if ($_[0] < 0) {
+                     if ($_[0] < 0) {
-                     push @nondirs, $entry;
+                        push @nondirs, $entry;
-                  } else {
+                     } else {
-                     # need to check for real directory
+                        # need to check for real directory
-                     aioreq_pri $pri;
+                        aioreq_pri $pri;
+                        $wd->[1] = $entry;
-                     add $statgrp aio_lstat "$path/$entry", sub {
+                        add $statgrp aio_lstat $wd, sub {
-                        if (-d _) {
+                           if (-d _) {
-                           push @dirs, $entry;
+                              push @dirs, $entry;
-                           unless (--$ndirs) {
+                              unless (--$ndirs) {
-                              push @nondirs, @$entries;
+                                 push @nondirs, @$entries;
-                              feed $statgrp;
+                                 feed $statgrp;
+                              }
+                           } else {
+                              push @nondirs, $entry;
                            }
-                        } else {
-                           push @nondirs, $entry;
                         }
                      }
-                  }
+                  };
                };
             };
          };
       };
    };
    $grp
 }
-=item aio_rmtree $path, $callback->($status)
+=item aio_rmtree $pathname, $callback->($status)
 Delete a directory tree starting (and including) C<$path>, return the
 status of the final C<rmdir> only.  This is a composite request that
 uses C<aio_scandir> to recurse into and rmdir directories, and unlink
 everything else.
 callback with the fdatasync result code.
 If this call isn't available because your OS lacks it or it couldn't be
 detected, it will be emulated by calling C<fsync> instead.
+=item aio_syncfs $fh, $callback->($status)
+Asynchronously call the syncfs syscall to sync the filesystem associated
+to the given filehandle and call the callback with the syncfs result
+code. If syncfs is not available, calls sync(), but returns C<-1> and sets
+errno to C<ENOSYS> nevertheless.
 =item aio_sync_file_range $fh, $offset, $nbytes, $flags, $callback->($status)
 Sync the data portion of the file specified by C<$offset> and C<$length>
 to disk (but NOT the metadata), by calling the Linux-specific
 sync_file_range call. If sync_file_range is not available or it returns
 C<$flags> can be a combination of C<IO::AIO::SYNC_FILE_RANGE_WAIT_BEFORE>,
 C<IO::AIO::SYNC_FILE_RANGE_WRITE> and
 C<IO::AIO::SYNC_FILE_RANGE_WAIT_AFTER>: refer to the sync_file_range
 manpage for details.
-=item aio_pathsync $path, $callback->($status)
+=item aio_pathsync $pathname, $callback->($status)
 This request tries to open, fsync and close the given path. This is a
 composite request intended to sync directories after directory operations
 (E.g. rename). This might not work on all operating systems or have any
 specific effect, but usually it makes sure that directory changes get
 immense (it blocks a thread for a long time) so do not use this function
 except to put your application under artificial I/O pressure.
 =back
+=head2 IO::AIO::WD - multiple working directories
+Your process only has one current working directory, which is used by all
+threads. This makes it hard to use relative paths (some other component
+could call C<chdir> at any time, and it is hard to control when the path
+will be used by IO::AIO).
+One solution for this is to always use absolute paths. This usually works,
+but can be quite slow (the kernel has to walk the whole path on every
+access), and can also be a hassle to implement.
+Newer POSIX systems have a number of functions (openat, fdopendir,
+futimensat and so on) that make it possible to specify working directories
+per operation.
+For portability, and because the clowns who "designed", or shall I write,
+perpetrated this new interface were obviously half-drunk, this abstraction
+cannot be perfect, though.
+IO::AIO allows you to convert directory paths into a so-called IO::AIO::WD
+object. This object stores the canonicalised, absolute version of the
+path, and on systems that allow it, also a directory file descriptor.
+Everywhere where a pathname is accepted by IO::AIO (e.g. in C<aio_stat>
+or C<aio_unlink>), one can specify an array reference with an IO::AIO::WD
+object and a pathname instead. If the pathname is absolute, the
+IO::AIO::WD objetc is ignored, otherwise the pathname is resolved relative
+to that IO::AIO::WD object.
+For example, to get a wd object for F</etc> and then stat F<passwd>
+inside, you would write:
+   aio_wd "/etc", sub {
+      my $etcdir = shift;
+      # although $etcdir can be undef on error, there is generally no reason
+      # to check for errors here, as aio_stat will fail with ENOENT
+      # when $etcdir is undef.
+      aio_stat [$etcdir, "passwd"], sub {
+         # yay
+      };
+   };
+This shows that creating an IO::AIO::WD object is itself a potentially
+blocking operation, which is why it is done asynchronously.
+As with normal pathnames, IO::AIO keeps a copy of the working directory
+object and the pathname string, so you could write the following without
+causing any issues due to C<$path> getting reused:
+   my $path = [$wd, undef];
+   for my $name (qw(abc def ghi)) {
+      $path->[1] = $name;
+      aio_stat $path, sub {
+         # ...
+      };
+   }
+There are some caveats: when directories get renamed (or deleted), the
+pathname string doesn't change, so will point to the new directory (or
+nowhere at all), while the directory fd, if available on the system,
+will still point to the original directory. Most functions accepting a
+pathname will use the directory fd on newer systems, and the string on
+older systems. Some functions (such as realpath) will always rely on the
+string form of the pathname.
+So this fucntionality is mainly useful to get some protection against
+C<chdir>, to easily get an absolute path out of a relative path for future
+reference, and to speed up doing many operations in the same directory
+(e.g. when stat'ing all files in a directory).
+The following functions implement this working directory abstraction:
+=over 4
+=item aio_wd $pathname, $callback->($wd)
+Asynchonously canonicalise the given pathname and convert it to an
+IO::AIO::WD object representing it. If possible and supported on the
+system, also open a directory fd to speed up pathname resolution relative
+to this working directory.
+If something goes wrong, then C<undef> is passwd to the callback instead
+of a working directory object and C<$!> is set appropriately. Since
+passing C<undef> as working directory component of a pathname fails the
+request with C<ENOENT>, there is often no need for error checking in the
+C<aio_wd> callback, as future requests using the value will fail in the
+expected way.
+If this call isn't available because your OS lacks it or it couldn't be
+detected, it will be emulated by calling C<fsync> instead.
+=item IO::AIO::CWD
+This is a compiletime constant (object) that represents the process
+current working directory.
+Specifying this object as working directory object for a pathname is as
+if the pathname would be specified directly, without a directory object,
+e.g., these calls are functionally identical:
+   aio_stat "somefile", sub { ... };
+   aio_stat [IO::AIO::CWD, "somefile"], sub { ... };
+=back
 =head2 IO::AIO::REQ CLASS
 All non-aggregate C<aio_*> functions return an object of this class when
 called in non-void context.
 Sets a feeder/generator on this group: every group can have an attached
 generator that generates requests if idle. The idea behind this is that,
 although you could just queue as many requests as you want in a group,
 this might starve other requests for a potentially long time. For example,
-C<aio_scandir> might generate hundreds of thousands C<aio_stat> requests,
+C<aio_scandir> might generate hundreds of thousands of C<aio_stat>
-delaying any later requests for a long time.
+requests, delaying any later requests for a long time.
 To avoid this, and allow incremental generation of requests, you can
 instead a group and set a feeder on it that generates those requests. The
 feed callback will be called whenever there are few enough (see C<limit>,
 below) requests active in the group itself and is expected to queue more
 =item IO::AIO::fadvise $fh, $offset, $len, $advice
 Simply calls the C<posix_fadvise> function (see its
 manpage for details). The following advice constants are
-avaiable: C<IO::AIO::FADV_NORMAL>, C<IO::AIO::FADV_SEQUENTIAL>,
+available: C<IO::AIO::FADV_NORMAL>, C<IO::AIO::FADV_SEQUENTIAL>,
 C<IO::AIO::FADV_RANDOM>, C<IO::AIO::FADV_NOREUSE>,
 C<IO::AIO::FADV_WILLNEED>, C<IO::AIO::FADV_DONTNEED>.
 On systems that do not implement C<posix_fadvise>, this function returns
 ENOSYS, otherwise the return value of C<posix_fadvise>.
 =item IO::AIO::madvise $scalar, $offset, $len, $advice
 Simply calls the C<posix_madvise> function (see its
 manpage for details). The following advice constants are
-avaiable: C<IO::AIO::MADV_NORMAL>, C<IO::AIO::MADV_SEQUENTIAL>,
+available: C<IO::AIO::MADV_NORMAL>, C<IO::AIO::MADV_SEQUENTIAL>,
 C<IO::AIO::MADV_RANDOM>, C<IO::AIO::MADV_WILLNEED>, C<IO::AIO::MADV_DONTNEED>.
 On systems that do not implement C<posix_madvise>, this function returns
 ENOSYS, otherwise the return value of C<posix_madvise>.
 =item IO::AIO::mprotect $scalar, $offset, $len, $protect
 Simply calls the C<mprotect> function on the preferably AIO::mmap'ed
 $scalar (see its manpage for details). The following protect
-constants are avaiable: C<IO::AIO::PROT_NONE>, C<IO::AIO::PROT_READ>,
+constants are available: C<IO::AIO::PROT_NONE>, C<IO::AIO::PROT_READ>,
 C<IO::AIO::PROT_WRITE>, C<IO::AIO::PROT_EXEC>.
 On systems that do not implement C<mprotect>, this function returns
 ENOSYS, otherwise the return value of C<mprotect>.
 =over 4
 =item IO::AIO::reinit
-Abondons all current requests and I/O threads and simply reinitialises all
+Abandons all current requests and I/O threads and simply reinitialises all
-data structures. This is not an operation suppported by any standards, but
+data structures. This is not an operation supported by any standards, but
 happens to work on GNU/Linux and some newer BSD systems.
 The only reasonable use for this function is to call it after forking, if
 C<IO::AIO> was used in the parent. Calling it while IO::AIO is active in
 the process will result in undefined behaviour. Calling it at any time

Diff Legend

-–
+Removed lines
-+
+Added lines
-<
+Changed lines
->
+Changed lines

Comparing IO-AIO/AIO.pm (file contents): Revision 1.204 by root, Mon Jul 18 02:02:26 2011 UTC vs. Revision 1.211 by root, Tue Sep 27 12:10:29 2011 UTC

Diff Legend

Comparing IO-AIO/AIO.pm (file contents):
Revision 1.204 by root, Mon Jul 18 02:02:26 2011 UTC vs.
Revision 1.211 by root, Tue Sep 27 12:10:29 2011 UTC