--- IO-AIO/README 2007/10/06 14:05:37 1.27 +++ IO-AIO/README 2009/06/12 16:55:17 1.37 @@ -26,9 +26,11 @@ my $grp = aio_group sub { print "all stats done\n" }; add $grp aio_stat "..." for ...; - # AnyEvent integration - open my $fh, "<&=" . IO::AIO::poll_fileno or die "$!"; - my $w = AnyEvent->io (fh => $fh, poll => 'r', cb => sub { IO::AIO::poll_cb }); + # AnyEvent integration (EV, Event, Glib, Tk, POE, urxvt, pureperl...) + use AnyEvent::AIO; + + # EV integration + my $w = EV::io IO::AIO::poll_fileno, EV::READ, \&IO::AIO::poll_cb; # Event integration Event->io (fd => IO::AIO::poll_fileno, @@ -175,8 +177,8 @@ identical, and they all accept an additional (and optional) $callback argument which must be a code reference. This code reference will get called with the syscall return code (e.g. most syscalls return -1 on - error, unlike perl, which usually delivers "false") as it's sole - argument when the given syscall has been executed asynchronously. + error, unlike perl, which usually delivers "false") as its sole argument + after the given syscall has been executed asynchronously. All functions expecting a filehandle keep a copy of the filehandle internally until the request has finished. @@ -199,7 +201,7 @@ contents. This works, btw. independent of the internal UTF-8 bit, which IO::AIO - handles correctly wether it is set or not. + handles correctly whether it is set or not. $prev_pri = aioreq_pri [$pri] Returns the priority value that would be used for the next request @@ -264,41 +266,24 @@ Unfortunately, you can't do this to perl. Perl *insists* very strongly on closing the file descriptor associated with the - filehandle itself. Here is what aio_close will try: - - 1. dup()licate the fd - 2. asynchronously close() the duplicated fd - 3. dup()licate the fd once more - 4. let perl close() the filehandle - 5. asynchronously close the duplicated fd - - The idea is that the first close() flushes stuff to disk that - closing an fd will flush, so when perl closes the fd, nothing much - will need to be flushed. The second async. close() will then flush - stuff to disk that closing the last fd to the file will flush. - - Just FYI, SuSv3 has this to say on close: - - All outstanding record locks owned by the process on the file - associated with the file descriptor shall be removed. - - If fildes refers to a socket, close() shall cause the socket to be - destroyed. ... close() shall block for up to the current linger - interval until all data is transmitted. - [this actually sounds like a specification bug, but who knows] + filehandle itself. - And at least Linux additionally actually flushes stuff on every - close, even when the file itself is still open. + Therefore, "aio_close" will not close the filehandle - instead it + will use dup2 to overwrite the file descriptor with the write-end of + a pipe (the pipe fd will be created on demand and will be cached). - Sounds enourmously inefficient and complicated? Yes... please show - me how to nuke perl's fd out of existence... + Or in other words: the file descriptor will be closed, but it will + not be free for reuse until the perl filehandle is closed. aio_read $fh,$offset,$length, $data,$dataoffset, $callback->($retval) aio_write $fh,$offset,$length, $data,$dataoffset, $callback->($retval) - Reads or writes $length bytes from the specified $fh and $offset - into the scalar given by $data and offset $dataoffset and calls the - callback without the actual number of bytes read (or -1 on error, - just like the syscall). + Reads or writes $length bytes from or to the specified $fh and + $offset into the scalar given by $data and offset $dataoffset and + calls the callback without the actual number of bytes read (or -1 on + error, just like the syscall). + + "aio_read" will, like "sysread", shrink or grow the $data scalar to + offset plus the actual number of bytes read. If $offset is undefined, then the current file descriptor offset will be used (and updated), otherwise the file descriptor offset @@ -459,8 +444,68 @@ entire directory (i.e. opendir + readdir + closedir). The entries will not be sorted, and will NOT include the "." and ".." entries. - The callback a single argument which is either "undef" or an - array-ref with the filenames. + The callback is passed a single argument which is either "undef" or + an array-ref with the filenames. + + aio_readdirx $pathname, $flags, $callback->($entries, $flags) + Quite similar to "aio_readdir", but the $flags argument allows to + tune behaviour and output format. In case of an error, $entries will + be "undef". + + The flags are a combination of the following constants, ORed + together (the flags will also be passed to the callback, possibly + modified): + + IO::AIO::READDIR_DENTS + When this flag is off, then the callback gets an arrayref with + of names only (as with "aio_readdir"), otherwise it gets an + arrayref with "[$name, $type, $inode]" arrayrefs, each + describing a single directory entry in more detail. + + $name is the name of the entry. + + $type is one of the "IO::AIO::DT_xxx" constants: + + "IO::AIO::DT_UNKNOWN", "IO::AIO::DT_FIFO", "IO::AIO::DT_CHR", + "IO::AIO::DT_DIR", "IO::AIO::DT_BLK", "IO::AIO::DT_REG", + "IO::AIO::DT_LNK", "IO::AIO::DT_SOCK", "IO::AIO::DT_WHT". + + "IO::AIO::DT_UNKNOWN" means just that: readdir does not know. If + you need to know, you have to run stat yourself. Also, for speed + reasons, the $type scalars are read-only: you can not modify + them. + + $inode is the inode number (which might not be exact on systems + with 64 bit inode numbers and 32 bit perls). On systems that do + not deliver the inode information, this will always be zero. + + IO::AIO::READDIR_DIRS_FIRST + When this flag is set, then the names will be returned in an + order where likely directories come first. This is useful when + you need to quickly find directories, or you want to find all + directories while avoiding to stat() each entry. + + If the system returns type information in readdir, then this is + used to find directories directly. Otherwise, likely directories + are files beginning with ".", or otherwise files with no dots, + of which files with short names are tried first. + + IO::AIO::READDIR_STAT_ORDER + When this flag is set, then the names will be returned in an + order suitable for stat()'ing each one. That is, when you plan + to stat() all files in the given directory, then the returned + order will likely be fastest. + + If both this flag and "IO::AIO::READDIR_DIRS_FIRST" are + specified, then the likely dirs come first, resulting in a less + optimal stat order. + + IO::AIO::READDIR_FOUND_UNKNOWN + This flag should not be set when calling "aio_readdirx". + Instead, it is being set by "aio_readdirx", when any of the + $type's found were "IO::AIO::DT_UNKNOWN". The absense of this + flag therefore indicates that all $type's are known, which can + be used to speed up some algorithms. aio_load $path, $data, $callback->($status) This is a composite request that tries to fully load the given file @@ -471,10 +516,10 @@ or destination) from $srcpath to $dstpath and call the callback with the 0 (error) or -1 ok. - This is a composite request that it creates the destination file - with mode 0200 and copies the contents of the source file into it - using "aio_sendfile", followed by restoring atime, mtime, access - mode and uid/gid, in that order. + This is a composite request that creates the destination file with + mode 0200 and copies the contents of the source file into it using + "aio_sendfile", followed by restoring atime, mtime, access mode and + uid/gid, in that order. If an error occurs, the partial destination file will be unlinked, if possible, except when setting atime, mtime, access mode and @@ -485,9 +530,9 @@ or destination) from $srcpath to $dstpath and call the callback with the 0 (error) or -1 ok. - This is a composite request that tries to rename(2) the file first. - If rename files with "EXDEV", it copies the file with "aio_copy" - and, if that is successful, unlinking the $srcpath. + This is a composite request that tries to rename(2) the file first; + if rename fails with "EXDEV", it copies the file with "aio_copy" + and, if that is successful, unlinks the $srcpath. aio_scandir $path, $maxreq, $callback->($dirs, $nondirs) Scans a directory (similar to "aio_readdir") but additionally tries @@ -517,20 +562,26 @@ The "aio_readdir" cannot be avoided, but "stat()"'ing every entry can. - After reading the directory, the modification time, size etc. of the - directory before and after the readdir is checked, and if they match - (and isn't the current time), the link count will be used to decide - how many entries are directories (if >= 2). Otherwise, no knowledge - of the number of subdirectories will be assumed. - - Then entries will be sorted into likely directories (everything - without a non-initial dot currently) and likely non-directories - (everything else). Then every entry plus an appended "/." will be - "stat"'ed, likely directories first. If that succeeds, it assumes - that the entry is a directory or a symlink to directory (which will - be checked seperately). This is often faster than stat'ing the entry - itself because filesystems might detect the type of the entry - without reading the inode data (e.g. ext2fs filetype feature). + If readdir returns file type information, then this is used directly + to find directories. + + Otherwise, after reading the directory, the modification time, size + etc. of the directory before and after the readdir is checked, and + if they match (and isn't the current time), the link count will be + used to decide how many entries are directories (if >= 2). + Otherwise, no knowledge of the number of subdirectories will be + assumed. + + Then entries will be sorted into likely directories a non-initial + dot currently) and likely non-directories (see "aio_readdirx"). Then + every entry plus an appended "/." will be "stat"'ed, likely + directories first, in order of their inode numbers. If that + succeeds, it assumes that the entry is a directory or a symlink to + directory (which will be checked seperately). This is often faster + than stat'ing the entry itself because filesystems might detect the + type of the entry without reading the inode data (e.g. ext2fs + filetype feature), even on systems that cannot return the filetype + information on readdir. If the known number of directories (link count - 2) has been reached, the rest of the entries is assumed to be non-directories. @@ -548,6 +599,9 @@ uses "aio_scandir" to recurse into and rmdir directories, and unlink everything else. + aio_sync $callback->($status) + Asynchronously call sync and call the callback when finished. + aio_fsync $fh, $callback->($status) Asynchronously call fsync on the given filehandle and call the callback with the fsync result code. @@ -559,6 +613,28 @@ If this call isn't available because your OS lacks it or it couldn't be detected, it will be emulated by calling "fsync" instead. + aio_sync_file_range $fh, $offset, $nbytes, $flags, $callback->($status) + Sync the data portion of the file specified by $offset and $length + to disk (but NOT the metadata), by calling the Linux-specific + sync_file_range call. If sync_file_range is not available or it + returns ENOSYS, then fdatasync or fsync is being substituted. + + $flags can be a combination of + "IO::AIO::SYNC_FILE_RANGE_WAIT_BEFORE", + "IO::AIO::SYNC_FILE_RANGE_WRITE" and + "IO::AIO::SYNC_FILE_RANGE_WAIT_AFTER": refer to the sync_file_range + manpage for details. + + aio_pathsync $path, $callback->($status) + This request tries to open, fsync and close the given path. This is + a composite request intended to sync directories after directory + operations (E.g. rename). This might not work on all operating + systems or have any specific effect, but usually it makes sure that + directory changes get written to disc. It works for anything that + can be opened for read-only, not just directories. + + Passes 0 when everything went ok, and -1 on error. + aio_group $callback->(...) This is a very special aio request: Instead of doing something, it is a container for other aio requests, which is useful if you want @@ -610,9 +686,9 @@ Cancels the request, if possible. Has the effect of skipping execution when entering the execute state and skipping calling the callback when entering the the result state, but will leave the - request otherwise untouched. That means that requests that currently - execute will not be stopped and resources held by the request will - not be freed prematurely. + request otherwise untouched (with the exception of readdir). That + means that requests that currently execute will not be stopped and + resources held by the request will not be freed prematurely. cb $req $callback->(...) Replace (or simply set) the callback registered to the request. @@ -649,23 +725,27 @@ This makes it very easy to create composite requests (see the source of "aio_move" for an application) that work and feel like simple requests. - * The IO::AIO::GRP objects will be cleaned up during calls to - "IO::AIO::poll_cb", just like any other request. - * They can be canceled like any other request. Canceling will cancel not - only the request itself, but also all requests it contains. - * They can also can also be added to other IO::AIO::GRP objects. - * You must not add requests to a group from within the group callback - (or any later time). + * The IO::AIO::GRP objects will be cleaned up during calls to + "IO::AIO::poll_cb", just like any other request. + + * They can be canceled like any other request. Canceling will cancel + not only the request itself, but also all requests it contains. + + * They can also can also be added to other IO::AIO::GRP objects. + + * You must not add requests to a group from within the group callback + (or any later time). Their lifetime, simplified, looks like this: when they are empty, they will finish very quickly. If they contain only requests that are in the "done" state, they will also finish. Otherwise they will continue to exist. - That means after creating a group you have some time to add requests. - And in the callbacks of those requests, you can add further requests to - the group. And only when all those requests have finished will the the - group itself finish. + That means after creating a group you have some time to add requests + (precisely before the callback has been invoked, which is only done + within the "poll_cb"). And in the callbacks of those requests, you can + add further requests to the group. And only when all those requests have + finished will the the group itself finish. add $grp ... $grp->add (...) @@ -682,7 +762,7 @@ $grp->result (...) Set the result value(s) that will be passed to the group callback - when all subrequests have finished and set thre groups errno to the + when all subrequests have finished and set the groups errno to the current value of errno (just like calling "errno" without an error number). By default, no argument will be passed and errno is zero. @@ -718,7 +798,8 @@ If the feed does not queue more requests when called, it will be automatically removed from the group. - If the feed limit is 0, it will be set to 2 automatically. + If the feed limit is 0 when this method is called, it will be set to + 2 automatically. Example: @@ -739,6 +820,9 @@ Setting the limit to 0 will pause the feeding process. + The default value for the limit is 0, but note that setting a feeder + automatically bumps it up to 2. + SUPPORT FUNCTIONS EVENT PROCESSING AND EVENT LOOP INTEGRATION $fileno = IO::AIO::poll_fileno @@ -751,13 +835,15 @@ IO::AIO::poll_cb Process some outstanding events on the result pipe. You have to call - this regularly. Returns the number of events processed. Returns - immediately when no events are outstanding. The amount of events - processed depends on the settings of "IO::AIO::max_poll_req" and + this regularly. Returns 0 if all events could be processed, or -1 if + it returned earlier for whatever reason. Returns immediately when no + events are outstanding. The amount of events processed depends on + the settings of "IO::AIO::max_poll_req" and "IO::AIO::max_poll_time". If not all requests were processed for whatever reason, the - filehandle will still be ready when "poll_cb" returns. + filehandle will still be ready when "poll_cb" returns, so normally + you don't have to do anything special to have it called later. Example: Install an Event watcher that automatically calls IO::AIO::poll_cb with high priority: @@ -873,7 +959,7 @@ creation is fast. If thread creation is very slow on your system you might want to use larger values. - $oldmaxreqs = IO::AIO::max_outstanding $maxreqs + IO::AIO::max_outstanding $maxreqs This is a very bad function to use in interactive programs because it blocks, and a bad way to reduce concurrency because it is inexact: Better use an "aio_group" together with a feed callback. @@ -887,7 +973,7 @@ the number of outstanding requests. You can still queue as many requests as you want. Therefore, - "max_oustsanding" is mainly useful in simple scripts (with low + "max_outstanding" is mainly useful in simple scripts (with low values) or as a stop gap to shield against fatal memory overflow (with large values). @@ -947,7 +1033,8 @@ Known bugs will be fixed in the next release. SEE ALSO - Coro::AIO. + AnyEvent::AIO for easy integration into event loops, Coro::AIO for a + more natural syntax. AUTHOR Marc Lehmann