ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/IO-AIO/AIO.pm
Revision: 1.37
Committed: Tue Aug 23 12:37:19 2005 UTC (18 years, 9 months ago) by root
Branch: MAIN
Changes since 1.36: +10 -1 lines
Log Message:
*** empty log message ***

File Contents

# User Rev Content
1 root 1.1 =head1 NAME
2    
3     IO::AIO - Asynchronous Input/Output
4    
5     =head1 SYNOPSIS
6    
7     use IO::AIO;
8    
9 root 1.6 aio_open "/etc/passwd", O_RDONLY, 0, sub {
10     my ($fh) = @_;
11     ...
12     };
13    
14     aio_unlink "/tmp/file", sub { };
15    
16     aio_read $fh, 30000, 1024, $buffer, 0, sub {
17 root 1.8 $_[0] > 0 or die "read error: $!";
18 root 1.6 };
19    
20     # Event
21     Event->io (fd => IO::AIO::poll_fileno,
22 root 1.7 poll => 'r',
23 root 1.6 cb => \&IO::AIO::poll_cb);
24    
25     # Glib/Gtk2
26     add_watch Glib::IO IO::AIO::poll_fileno,
27 root 1.22 in => sub { IO::AIO::poll_cb; 1 };
28 root 1.6
29     # Tk
30     Tk::Event::IO->fileevent (IO::AIO::poll_fileno, "",
31     readable => \&IO::AIO::poll_cb);
32    
33 root 1.11 # Danga::Socket
34     Danga::Socket->AddOtherFds (IO::AIO::poll_fileno =>
35     \&IO::AIO::poll_cb);
36    
37    
38 root 1.1 =head1 DESCRIPTION
39    
40     This module implements asynchronous I/O using whatever means your
41 root 1.2 operating system supports.
42 root 1.1
43 root 1.2 Currently, a number of threads are started that execute your read/writes
44     and signal their completion. You don't need thread support in your libc or
45     perl, and the threads created by this module will not be visible to the
46     pthreads library. In the future, this module might make use of the native
47     aio functions available on many operating systems. However, they are often
48     not well-supported (Linux doesn't allow them on normal files currently,
49     for example), and they would only support aio_read and aio_write, so the
50     remaining functionality would have to be implemented using threads anyway.
51 root 1.1
52     Although the module will work with in the presence of other threads, it is
53 root 1.22 currently not reentrant, so use appropriate locking yourself, always call
54     C<poll_cb> from within the same thread, or never call C<poll_cb> (or other
55     C<aio_> functions) recursively.
56 root 1.1
57     =cut
58    
59     package IO::AIO;
60    
61 root 1.23 no warnings;
62    
63 root 1.1 use base 'Exporter';
64    
65 root 1.2 use Fcntl ();
66    
67 root 1.1 BEGIN {
68 root 1.36 $VERSION = 1.6;
69 root 1.1
70     @EXPORT = qw(aio_read aio_write aio_open aio_close aio_stat aio_lstat aio_unlink
71 root 1.37 aio_rmdir aio_readdir aio_symlink aio_fsync aio_fdatasync aio_readahead);
72 root 1.4 @EXPORT_OK = qw(poll_fileno poll_cb min_parallel max_parallel max_outstanding nreqs);
73 root 1.1
74     require XSLoader;
75     XSLoader::load IO::AIO, $VERSION;
76     }
77    
78 root 1.5 =head1 FUNCTIONS
79 root 1.1
80 root 1.5 =head2 AIO FUNCTIONS
81 root 1.1
82 root 1.5 All the C<aio_*> calls are more or less thin wrappers around the syscall
83     with the same name (sans C<aio_>). The arguments are similar or identical,
84 root 1.14 and they all accept an additional (and optional) C<$callback> argument
85     which must be a code reference. This code reference will get called with
86     the syscall return code (e.g. most syscalls return C<-1> on error, unlike
87     perl, which usually delivers "false") as it's sole argument when the given
88     syscall has been executed asynchronously.
89 root 1.1
90 root 1.23 All functions expecting a filehandle keep a copy of the filehandle
91     internally until the request has finished.
92 root 1.1
93 root 1.28 The pathnames you pass to these routines I<must> be absolute and
94     encoded in byte form. The reason for the former is that at the time the
95     request is being executed, the current working directory could have
96     changed. Alternatively, you can make sure that you never change the
97     current working directory.
98    
99     To encode pathnames to byte form, either make sure you either: a)
100     always pass in filenames you got from outside (command line, readdir
101     etc.), b) are ASCII or ISO 8859-1, c) use the Encode module and encode
102     your pathnames to the locale (or other) encoding in effect in the user
103     environment, d) use Glib::filename_from_unicode on unicode filenames or e)
104     use something else.
105 root 1.1
106 root 1.5 =over 4
107 root 1.1
108     =item aio_open $pathname, $flags, $mode, $callback
109    
110 root 1.2 Asynchronously open or create a file and call the callback with a newly
111     created filehandle for the file.
112 root 1.1
113     The pathname passed to C<aio_open> must be absolute. See API NOTES, above,
114     for an explanation.
115    
116 root 1.20 The C<$flags> argument is a bitmask. See the C<Fcntl> module for a
117     list. They are the same as used by C<sysopen>.
118    
119     Likewise, C<$mode> specifies the mode of the newly created file, if it
120     didn't exist and C<O_CREAT> has been given, just like perl's C<sysopen>,
121     except that it is mandatory (i.e. use C<0> if you don't create new files,
122     and C<0666> or C<0777> if you do).
123 root 1.1
124     Example:
125    
126     aio_open "/etc/passwd", O_RDONLY, 0, sub {
127 root 1.2 if ($_[0]) {
128     print "open successful, fh is $_[0]\n";
129 root 1.1 ...
130     } else {
131     die "open failed: $!\n";
132     }
133     };
134    
135     =item aio_close $fh, $callback
136    
137 root 1.2 Asynchronously close a file and call the callback with the result
138     code. I<WARNING:> although accepted, you should not pass in a perl
139 root 1.20 filehandle here, as perl will likely close the file descriptor another
140     time when the filehandle is destroyed. Normally, you can safely call perls
141     C<close> or just let filehandles go out of scope.
142    
143     This is supposed to be a bug in the API, so that might change. It's
144     therefore best to avoid this function.
145 root 1.1
146     =item aio_read $fh,$offset,$length, $data,$dataoffset,$callback
147    
148     =item aio_write $fh,$offset,$length, $data,$dataoffset,$callback
149    
150     Reads or writes C<length> bytes from the specified C<fh> and C<offset>
151     into the scalar given by C<data> and offset C<dataoffset> and calls the
152     callback without the actual number of bytes read (or -1 on error, just
153     like the syscall).
154    
155 root 1.31 The C<$data> scalar I<MUST NOT> be modified in any way while the request
156     is outstanding. Modifying it can result in segfaults or WW3 (if the
157     necessary/optional hardware is installed).
158    
159 root 1.17 Example: Read 15 bytes at offset 7 into scalar C<$buffer>, starting at
160 root 1.1 offset C<0> within the scalar:
161    
162     aio_read $fh, 7, 15, $buffer, 0, sub {
163 root 1.9 $_[0] > 0 or die "read error: $!";
164     print "read $_[0] bytes: <$buffer>\n";
165 root 1.1 };
166    
167 root 1.35 =item aio_sendfile $out_fh, $in_fh, $in_offset, $length, $callback
168    
169     Tries to copy C<$length> bytes from C<$in_fh> to C<$out_fh>. It starts
170     reading at byte offset C<$in_offset>, and starts writing at the current
171     file offset of C<$out_fh>. Because of that, it is not safe to issue more
172     than one C<aio_sendfile> per C<$out_fh>, as they will interfere with each
173     other.
174    
175     This call tries to make use of a native C<sendfile> syscall to provide
176     zero-copy operation. For this to work, C<$out_fh> should refer to a
177     socket, and C<$in_fh> should refer to mmap'able file.
178    
179     If the native sendfile call fails or is not implemented, it will be
180 root 1.36 emulated, so you can call C<aio_sendfile> on any type of filehandle
181     regardless of the limitations of the operating system.
182 root 1.35
183     Please note, however, that C<aio_sendfile> can read more bytes from
184     C<$in_fh> than are written, and there is no way to find out how many
185 root 1.36 bytes have been read from C<aio_sendfile> alone, as C<aio_sendfile> only
186     provides the number of bytes written to C<$out_fh>. Only if the result
187     value equals C<$length> one can assume that C<$length> bytes have been
188     read.
189 root 1.35
190 root 1.1 =item aio_readahead $fh,$offset,$length, $callback
191    
192 root 1.20 C<aio_readahead> populates the page cache with data from a file so that
193 root 1.1 subsequent reads from that file will not block on disk I/O. The C<$offset>
194     argument specifies the starting point from which data is to be read and
195     C<$length> specifies the number of bytes to be read. I/O is performed in
196     whole pages, so that offset is effectively rounded down to a page boundary
197     and bytes are read up to the next page boundary greater than or equal to
198 root 1.20 (off-set+length). C<aio_readahead> does not read beyond the end of the
199 root 1.1 file. The current file offset of the file is left unchanged.
200    
201 root 1.26 If that syscall doesn't exist (likely if your OS isn't Linux) it will be
202     emulated by simply reading the data, which would have a similar effect.
203    
204 root 1.1 =item aio_stat $fh_or_path, $callback
205    
206     =item aio_lstat $fh, $callback
207    
208     Works like perl's C<stat> or C<lstat> in void context. The callback will
209     be called after the stat and the results will be available using C<stat _>
210     or C<-s _> etc...
211    
212     The pathname passed to C<aio_stat> must be absolute. See API NOTES, above,
213     for an explanation.
214    
215     Currently, the stats are always 64-bit-stats, i.e. instead of returning an
216     error when stat'ing a large file, the results will be silently truncated
217     unless perl itself is compiled with large file support.
218    
219     Example: Print the length of F</etc/passwd>:
220    
221     aio_stat "/etc/passwd", sub {
222     $_[0] and die "stat failed: $!";
223     print "size is ", -s _, "\n";
224     };
225    
226     =item aio_unlink $pathname, $callback
227    
228     Asynchronously unlink (delete) a file and call the callback with the
229     result code.
230    
231 root 1.27 =item aio_rmdir $pathname, $callback
232    
233     Asynchronously rmdir (delete) a directory and call the callback with the
234     result code.
235    
236 root 1.37 =item aio_readdir $pathname $callback
237    
238     Unlike the POSIX call of the same name, C<aio_readdir> reads an entire
239     directory (i.e. opendir + readdir + closedir). The entries will not be
240     sorted, and will B<NOT> include the C<.> and C<..> entries.
241    
242     The callback a single argument which is either C<undef> or an array-ref
243     with the filenames.
244    
245 root 1.1 =item aio_fsync $fh, $callback
246    
247     Asynchronously call fsync on the given filehandle and call the callback
248     with the fsync result code.
249    
250     =item aio_fdatasync $fh, $callback
251    
252     Asynchronously call fdatasync on the given filehandle and call the
253 root 1.26 callback with the fdatasync result code.
254    
255     If this call isn't available because your OS lacks it or it couldn't be
256     detected, it will be emulated by calling C<fsync> instead.
257 root 1.1
258 root 1.5 =back
259    
260     =head2 SUPPORT FUNCTIONS
261    
262     =over 4
263    
264     =item $fileno = IO::AIO::poll_fileno
265    
266 root 1.20 Return the I<request result pipe file descriptor>. This filehandle must be
267     polled for reading by some mechanism outside this module (e.g. Event or
268     select, see below or the SYNOPSIS). If the pipe becomes readable you have
269     to call C<poll_cb> to check the results.
270 root 1.5
271     See C<poll_cb> for an example.
272    
273     =item IO::AIO::poll_cb
274    
275     Process all outstanding events on the result pipe. You have to call this
276     regularly. Returns the number of events processed. Returns immediately
277     when no events are outstanding.
278    
279 root 1.20 Example: Install an Event watcher that automatically calls
280     IO::AIO::poll_cb with high priority:
281 root 1.5
282     Event->io (fd => IO::AIO::poll_fileno,
283     poll => 'r', async => 1,
284     cb => \&IO::AIO::poll_cb);
285    
286     =item IO::AIO::poll_wait
287    
288     Wait till the result filehandle becomes ready for reading (simply does a
289 root 1.20 C<select> on the filehandle. This is useful if you want to synchronously wait
290 root 1.5 for some requests to finish).
291    
292     See C<nreqs> for an example.
293    
294     =item IO::AIO::nreqs
295    
296 root 1.20 Returns the number of requests currently outstanding (i.e. for which their
297     callback has not been invoked yet).
298 root 1.5
299     Example: wait till there are no outstanding requests anymore:
300    
301     IO::AIO::poll_wait, IO::AIO::poll_cb
302     while IO::AIO::nreqs;
303    
304 root 1.12 =item IO::AIO::flush
305    
306     Wait till all outstanding AIO requests have been handled.
307    
308 root 1.13 Strictly equivalent to:
309    
310     IO::AIO::poll_wait, IO::AIO::poll_cb
311     while IO::AIO::nreqs;
312    
313     =item IO::AIO::poll
314    
315     Waits until some requests have been handled.
316    
317     Strictly equivalent to:
318    
319     IO::AIO::poll_wait, IO::AIO::poll_cb
320     if IO::AIO::nreqs;
321    
322 root 1.5 =item IO::AIO::min_parallel $nthreads
323    
324 root 1.34 Set the minimum number of AIO threads to C<$nthreads>. The current default
325     is C<4>, which means four asynchronous operations can be done at one time
326 root 1.5 (the number of outstanding operations, however, is unlimited).
327    
328 root 1.34 IO::AIO starts threads only on demand, when an AIO request is queued and
329     no free thread exists.
330    
331 root 1.5 It is recommended to keep the number of threads low, as some Linux
332     kernel versions will scale negatively with the number of threads (higher
333     parallelity => MUCH higher latency). With current Linux 2.6 versions, 4-32
334     threads should be fine.
335    
336 root 1.34 Under most circumstances you don't need to call this function, as the
337     module selects a default that is suitable for low to moderate load.
338 root 1.5
339     =item IO::AIO::max_parallel $nthreads
340    
341 root 1.34 Sets the maximum number of AIO threads to C<$nthreads>. If more than the
342     specified number of threads are currently running, this function kills
343     them. This function blocks until the limit is reached.
344    
345     While C<$nthreads> are zero, aio requests get queued but not executed
346     until the number of threads has been increased again.
347 root 1.5
348     This module automatically runs C<max_parallel 0> at program end, to ensure
349     that all threads are killed and that there are no outstanding requests.
350    
351     Under normal circumstances you don't need to call this function.
352    
353     =item $oldnreqs = IO::AIO::max_outstanding $nreqs
354    
355     Sets the maximum number of outstanding requests to C<$nreqs>. If you
356     try to queue up more than this number of requests, the caller will block until
357     some requests have been handled.
358    
359     The default is very large, so normally there is no practical limit. If you
360 root 1.34 queue up many requests in a loop it often improves speed if you set
361 root 1.5 this to a relatively low number, such as C<100>.
362    
363     Under normal circumstances you don't need to call this function.
364    
365     =back
366    
367 root 1.1 =cut
368    
369 root 1.2 # support function to convert a fd into a perl filehandle
370     sub _fd2fh {
371     return undef if $_[0] < 0;
372    
373 root 1.23 # try to generate nice filehandles
374     my $sym = "IO::AIO::fd#$_[0]";
375     local *$sym;
376 root 1.25
377 root 1.27 open *$sym, "+<&=$_[0]" # usually works under any unix
378     or open *$sym, "<&=$_[0]" # cygwin needs this
379     or open *$sym, ">&=$_[0]" # or this
380 root 1.2 or return undef;
381    
382 root 1.23 *$sym
383 root 1.2 }
384    
385 root 1.1 min_parallel 4;
386    
387     END {
388     max_parallel 0;
389     }
390    
391     1;
392    
393 root 1.27 =head2 FORK BEHAVIOUR
394    
395 root 1.34 Before the fork, IO::AIO enters a quiescent state where no requests
396     can be added in other threads and no results will be processed. After
397     the fork the parent simply leaves the quiescent state and continues
398     request/result processing, while the child clears the request/result
399     queue (so the requests started before the fork will only be handled in
400     the parent). Threats will be started on demand until the limit ste in the
401     parent process has been reached again.
402 root 1.27
403 root 1.1 =head1 SEE ALSO
404    
405     L<Coro>, L<Linux::AIO>.
406    
407     =head1 AUTHOR
408    
409     Marc Lehmann <schmorp@schmorp.de>
410     http://home.schmorp.de/
411    
412     =cut
413