--- IO-AIO/README 2005/07/10 17:07:44 1.1 +++ IO-AIO/README 2005/08/16 23:33:38 1.9 @@ -4,108 +4,99 @@ SYNOPSIS use IO::AIO; + aio_open "/etc/passwd", O_RDONLY, 0, sub { + my ($fh) = @_; + ... + }; + + aio_unlink "/tmp/file", sub { }; + + aio_read $fh, 30000, 1024, $buffer, 0, sub { + $_[0] > 0 or die "read error: $!"; + }; + + # Event + Event->io (fd => IO::AIO::poll_fileno, + poll => 'r', + cb => \&IO::AIO::poll_cb); + + # Glib/Gtk2 + add_watch Glib::IO IO::AIO::poll_fileno, + in => sub { IO::AIO::poll_cb; 1 }; + + # Tk + Tk::Event::IO->fileevent (IO::AIO::poll_fileno, "", + readable => \&IO::AIO::poll_cb); + + # Danga::Socket + Danga::Socket->AddOtherFds (IO::AIO::poll_fileno => + \&IO::AIO::poll_cb); + DESCRIPTION This module implements asynchronous I/O using whatever means your - operating system supports. Currently, it falls back to Linux::AIO if - that module is available, or uses pthreads to emulato aio functionality. + operating system supports. - Currently, in this module a number of threads are started that execute - your read/writes and signal their completion. You don't need thread - support in your libc or perl, and the threads created by this module - will not be visible to the pthreads library. + Currently, a number of threads are started that execute your read/writes + and signal their completion. You don't need thread support in your libc + or perl, and the threads created by this module will not be visible to + the pthreads library. In the future, this module might make use of the + native aio functions available on many operating systems. However, they + are often not well-supported (Linux doesn't allow them on normal files + currently, for example), and they would only support aio_read and + aio_write, so the remaining functionality would have to be implemented + using threads anyway. Although the module will work with in the presence of other threads, it - is not reentrant, so use appropriate locking yourself. + is currently not reentrant, so use appropriate locking yourself, always + call "poll_cb" from within the same thread, or never call "poll_cb" (or + other "aio_" functions) recursively. - API NOTES +FUNCTIONS + AIO FUNCTIONS All the "aio_*" calls are more or less thin wrappers around the syscall with the same name (sans "aio_"). The arguments are similar or - identical, and they all accept an additional $callback argument which - must be a code reference. This code reference will get called with the - syscall return code (e.g. most syscalls return -1 on error, unlike perl, - which usually delivers "false") as it's sole argument when the given - syscall has been executed asynchronously. - - All functions that expect a filehandle will also accept a file - descriptor. - - The filenames you pass to these routines *must* be absolute. The reason - is that at the time the request is being executed, the current working - directory could have changed. Alternatively, you can make sure that you - never change the current working directory. - - IO::AIO::min_parallel $nthreads - Set the minimum number of AIO threads to $nthreads. The default is - 1, which means a single asynchronous operation can be done at one - time (the number of outstanding operations, however, is unlimited). - - It is recommended to keep the number of threads low, as some linux - kernel versions will scale negatively with the number of threads - (higher parallelity => MUCH higher latency). - - Under normal circumstances you don't need to call this function, as - this module automatically starts a single async thread. - - IO::AIO::max_parallel $nthreads - Sets the maximum number of AIO threads to $nthreads. If more than - the specified number of threads are currently running, kill them. - This function blocks until the limit is reached. - - This module automatically runs "max_parallel 0" at program end, to - ensure that all threads are killed and that there are no outstanding - requests. - - Under normal circumstances you don't need to call this function. - - $fileno = IO::AIO::poll_fileno - Return the *request result pipe filehandle*. This filehandle must be - polled for reading by some mechanism outside this module (e.g. Event - or select, see below). If the pipe becomes readable you have to call - "poll_cb" to check the results. - - See "poll_cb" for an example. - - IO::AIO::poll_cb - Process all outstanding events on the result pipe. You have to call - this regularly. Returns the number of events processed. Returns - immediately when no events are outstanding. - - You can use Event to multiplex, e.g.: - - Event->io (fd => IO::AIO::poll_fileno, - poll => 'r', async => 1, - cb => \&IO::AIO::poll_cb); - - IO::AIO::poll_wait - Wait till the result filehandle becomes ready for reading (simply - does a select on the filehandle. This is useful if you want to - synchronously wait for some requests to finish). - - See "nreqs" for an example. - - IO::AIO::nreqs - Returns the number of requests currently outstanding. + identical, and they all accept an additional (and optional) $callback + argument which must be a code reference. This code reference will get + called with the syscall return code (e.g. most syscalls return -1 on + error, unlike perl, which usually delivers "false") as it's sole + argument when the given syscall has been executed asynchronously. + + All functions expecting a filehandle keep a copy of the filehandle + internally until the request has finished. + + The pathnames you pass to these routines *must* be absolute and encoded + in byte form. The reason for the former is that at the time the request + is being executed, the current working directory could have changed. + Alternatively, you can make sure that you never change the current + working directory. + + To encode pathnames to byte form, either make sure you either: a) always + pass in filenames you got from outside (command line, readdir etc.), b) + are ASCII or ISO 8859-1, c) use the Encode module and encode your + pathnames to the locale (or other) encoding in effect in the user + environment, d) use Glib::filename_from_unicode on unicode filenames or + e) use something else. - Example: wait till there are no outstanding requests anymore: + aio_open $pathname, $flags, $mode, $callback + Asynchronously open or create a file and call the callback with a + newly created filehandle for the file. - IO::AIO::poll_wait, IO::AIO::poll_cb - while IO::AIO::nreqs; + The pathname passed to "aio_open" must be absolute. See API NOTES, + above, for an explanation. - aio_open $pathname, $flags, $mode, $callback - Asynchronously open or create a file and call the callback with the - filedescriptor (NOT a perl filehandle, sorry for that, but watch - out, this might change in the future). + The $flags argument is a bitmask. See the "Fcntl" module for a list. + They are the same as used by "sysopen". - The $mode argument is a bitmask. See the "Fcntl" module for a list. - They are the same as used in "sysopen". + Likewise, $mode specifies the mode of the newly created file, if it + didn't exist and "O_CREAT" has been given, just like perl's + "sysopen", except that it is mandatory (i.e. use 0 if you don't + create new files, and 0666 or 0777 if you do). Example: aio_open "/etc/passwd", O_RDONLY, 0, sub { - if ($_[0] >= 0) { - open my $fh, "<&$_[0]"; # create a copy for perl - aio_close $_[0], sub { }; # close the aio handle - print "open successful, fh is $fh\n"; + if ($_[0]) { + print "open successful, fh is $_[0]\n"; ... } else { die "open failed: $!\n"; @@ -114,7 +105,13 @@ aio_close $fh, $callback Asynchronously close a file and call the callback with the result - code. + code. *WARNING:* although accepted, you should not pass in a perl + filehandle here, as perl will likely close the file descriptor + another time when the filehandle is destroyed. Normally, you can + safely call perls "close" or just let filehandles go out of scope. + + This is supposed to be a bug in the API, so that might change. It's + therefore best to avoid this function. aio_read $fh,$offset,$length, $data,$dataoffset,$callback aio_write $fh,$offset,$length, $data,$dataoffset,$callback @@ -123,28 +120,29 @@ the callback without the actual number of bytes read (or -1 on error, just like the syscall). - Example: Read 15 bytes at offset 7 into scalar $buffer, strating at + Example: Read 15 bytes at offset 7 into scalar $buffer, starting at offset 0 within the scalar: aio_read $fh, 7, 15, $buffer, 0, sub { - $_[0] >= 0 or die "read error: $!"; - print "read <$buffer>\n"; + $_[0] > 0 or die "read error: $!"; + print "read $_[0] bytes: <$buffer>\n"; }; aio_readahead $fh,$offset,$length, $callback - Asynchronously reads the specified byte range into the page cache, - using the "readahead" syscall. - - readahead() populates the page cache with data from a file so that - subsequent reads from that file will not block on disk I/O. The + "aio_readahead" populates the page cache with data from a file so + that subsequent reads from that file will not block on disk I/O. The $offset argument specifies the starting point from which data is to be read and $length specifies the number of bytes to be read. I/O is performed in whole pages, so that offset is effectively rounded down to a page boundary and bytes are read up to the next page boundary - greater than or equal to (off-set+length). aio_readahead() does not + greater than or equal to (off-set+length). "aio_readahead" does not read beyond the end of the file. The current file offset of the file is left unchanged. + If that syscall doesn't exist (likely if your OS isn't Linux) it + will be emulated by simply reading the data, which would have a + similar effect. + aio_stat $fh_or_path, $callback aio_lstat $fh, $callback Works like perl's "stat" or "lstat" in void context. The callback @@ -170,6 +168,10 @@ Asynchronously unlink (delete) a file and call the callback with the result code. + aio_rmdir $pathname, $callback + Asynchronously rmdir (delete) a directory and call the callback with + the result code. + aio_fsync $fh, $callback Asynchronously call fsync on the given filehandle and call the callback with the fsync result code. @@ -178,14 +180,105 @@ Asynchronously call fdatasync on the given filehandle and call the callback with the fdatasync result code. -BUGS - This module has been extensively tested in a large and very busy - webserver for many years now. + If this call isn't available because your OS lacks it or it couldn't + be detected, it will be emulated by calling "fsync" instead. + + SUPPORT FUNCTIONS + $fileno = IO::AIO::poll_fileno + Return the *request result pipe file descriptor*. This filehandle + must be polled for reading by some mechanism outside this module + (e.g. Event or select, see below or the SYNOPSIS). If the pipe + becomes readable you have to call "poll_cb" to check the results. + + See "poll_cb" for an example. + + IO::AIO::poll_cb + Process all outstanding events on the result pipe. You have to call + this regularly. Returns the number of events processed. Returns + immediately when no events are outstanding. + + Example: Install an Event watcher that automatically calls + IO::AIO::poll_cb with high priority: + + Event->io (fd => IO::AIO::poll_fileno, + poll => 'r', async => 1, + cb => \&IO::AIO::poll_cb); + + IO::AIO::poll_wait + Wait till the result filehandle becomes ready for reading (simply + does a "select" on the filehandle. This is useful if you want to + synchronously wait for some requests to finish). + + See "nreqs" for an example. + + IO::AIO::nreqs + Returns the number of requests currently outstanding (i.e. for which + their callback has not been invoked yet). + + Example: wait till there are no outstanding requests anymore: + + IO::AIO::poll_wait, IO::AIO::poll_cb + while IO::AIO::nreqs; + + IO::AIO::flush + Wait till all outstanding AIO requests have been handled. + + Strictly equivalent to: + + IO::AIO::poll_wait, IO::AIO::poll_cb + while IO::AIO::nreqs; + + IO::AIO::poll + Waits until some requests have been handled. + + Strictly equivalent to: + + IO::AIO::poll_wait, IO::AIO::poll_cb + if IO::AIO::nreqs; + + IO::AIO::min_parallel $nthreads + Set the minimum number of AIO threads to $nthreads. The default is + 1, which means a single asynchronous operation can be done at one + time (the number of outstanding operations, however, is unlimited). + + It is recommended to keep the number of threads low, as some Linux + kernel versions will scale negatively with the number of threads + (higher parallelity => MUCH higher latency). With current Linux 2.6 + versions, 4-32 threads should be fine. + + Under normal circumstances you don't need to call this function, as + this module automatically starts some threads (the exact number + might change, and is currently 4). + + IO::AIO::max_parallel $nthreads + Sets the maximum number of AIO threads to $nthreads. If more than + the specified number of threads are currently running, kill them. + This function blocks until the limit is reached. + + This module automatically runs "max_parallel 0" at program end, to + ensure that all threads are killed and that there are no outstanding + requests. + + Under normal circumstances you don't need to call this function. + + $oldnreqs = IO::AIO::max_outstanding $nreqs + Sets the maximum number of outstanding requests to $nreqs. If you + try to queue up more than this number of requests, the caller will + block until some requests have been handled. + + The default is very large, so normally there is no practical limit. + If you queue up many requests in a loop it it often improves speed + if you set this to a relatively low number, such as 100. + + Under normal circumstances you don't need to call this function. - - aio_open gives a fd, but all other functions expect a perl filehandle. + FORK BEHAVIOUR + IO::AIO handles all outstanding AIO requests before the fork, destroys + all AIO threads, and recreates them in both the parent and the child + after the fork. SEE ALSO - Coro. + Coro, Linux::AIO. AUTHOR Marc Lehmann