ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/IO-AIO/AIO.pm
Revision: 1.35
Committed: Mon Aug 22 23:20:37 2005 UTC (18 years, 9 months ago) by root
Branch: MAIN
Changes since 1.34: +23 -0 lines
Log Message:
*** empty log message ***

File Contents

# User Rev Content
1 root 1.1 =head1 NAME
2    
3     IO::AIO - Asynchronous Input/Output
4    
5     =head1 SYNOPSIS
6    
7     use IO::AIO;
8    
9 root 1.6 aio_open "/etc/passwd", O_RDONLY, 0, sub {
10     my ($fh) = @_;
11     ...
12     };
13    
14     aio_unlink "/tmp/file", sub { };
15    
16     aio_read $fh, 30000, 1024, $buffer, 0, sub {
17 root 1.8 $_[0] > 0 or die "read error: $!";
18 root 1.6 };
19    
20     # Event
21     Event->io (fd => IO::AIO::poll_fileno,
22 root 1.7 poll => 'r',
23 root 1.6 cb => \&IO::AIO::poll_cb);
24    
25     # Glib/Gtk2
26     add_watch Glib::IO IO::AIO::poll_fileno,
27 root 1.22 in => sub { IO::AIO::poll_cb; 1 };
28 root 1.6
29     # Tk
30     Tk::Event::IO->fileevent (IO::AIO::poll_fileno, "",
31     readable => \&IO::AIO::poll_cb);
32    
33 root 1.11 # Danga::Socket
34     Danga::Socket->AddOtherFds (IO::AIO::poll_fileno =>
35     \&IO::AIO::poll_cb);
36    
37    
38 root 1.1 =head1 DESCRIPTION
39    
40     This module implements asynchronous I/O using whatever means your
41 root 1.2 operating system supports.
42 root 1.1
43 root 1.2 Currently, a number of threads are started that execute your read/writes
44     and signal their completion. You don't need thread support in your libc or
45     perl, and the threads created by this module will not be visible to the
46     pthreads library. In the future, this module might make use of the native
47     aio functions available on many operating systems. However, they are often
48     not well-supported (Linux doesn't allow them on normal files currently,
49     for example), and they would only support aio_read and aio_write, so the
50     remaining functionality would have to be implemented using threads anyway.
51 root 1.1
52     Although the module will work with in the presence of other threads, it is
53 root 1.22 currently not reentrant, so use appropriate locking yourself, always call
54     C<poll_cb> from within the same thread, or never call C<poll_cb> (or other
55     C<aio_> functions) recursively.
56 root 1.1
57     =cut
58    
59     package IO::AIO;
60    
61 root 1.23 no warnings;
62    
63 root 1.1 use base 'Exporter';
64    
65 root 1.2 use Fcntl ();
66    
67 root 1.1 BEGIN {
68 root 1.34 $VERSION = 1.5;
69 root 1.1
70     @EXPORT = qw(aio_read aio_write aio_open aio_close aio_stat aio_lstat aio_unlink
71 root 1.27 aio_rmdir aio_symlink aio_fsync aio_fdatasync aio_readahead);
72 root 1.4 @EXPORT_OK = qw(poll_fileno poll_cb min_parallel max_parallel max_outstanding nreqs);
73 root 1.1
74     require XSLoader;
75     XSLoader::load IO::AIO, $VERSION;
76     }
77    
78 root 1.5 =head1 FUNCTIONS
79 root 1.1
80 root 1.5 =head2 AIO FUNCTIONS
81 root 1.1
82 root 1.5 All the C<aio_*> calls are more or less thin wrappers around the syscall
83     with the same name (sans C<aio_>). The arguments are similar or identical,
84 root 1.14 and they all accept an additional (and optional) C<$callback> argument
85     which must be a code reference. This code reference will get called with
86     the syscall return code (e.g. most syscalls return C<-1> on error, unlike
87     perl, which usually delivers "false") as it's sole argument when the given
88     syscall has been executed asynchronously.
89 root 1.1
90 root 1.23 All functions expecting a filehandle keep a copy of the filehandle
91     internally until the request has finished.
92 root 1.1
93 root 1.28 The pathnames you pass to these routines I<must> be absolute and
94     encoded in byte form. The reason for the former is that at the time the
95     request is being executed, the current working directory could have
96     changed. Alternatively, you can make sure that you never change the
97     current working directory.
98    
99     To encode pathnames to byte form, either make sure you either: a)
100     always pass in filenames you got from outside (command line, readdir
101     etc.), b) are ASCII or ISO 8859-1, c) use the Encode module and encode
102     your pathnames to the locale (or other) encoding in effect in the user
103     environment, d) use Glib::filename_from_unicode on unicode filenames or e)
104     use something else.
105 root 1.1
106 root 1.5 =over 4
107 root 1.1
108     =item aio_open $pathname, $flags, $mode, $callback
109    
110 root 1.2 Asynchronously open or create a file and call the callback with a newly
111     created filehandle for the file.
112 root 1.1
113     The pathname passed to C<aio_open> must be absolute. See API NOTES, above,
114     for an explanation.
115    
116 root 1.20 The C<$flags> argument is a bitmask. See the C<Fcntl> module for a
117     list. They are the same as used by C<sysopen>.
118    
119     Likewise, C<$mode> specifies the mode of the newly created file, if it
120     didn't exist and C<O_CREAT> has been given, just like perl's C<sysopen>,
121     except that it is mandatory (i.e. use C<0> if you don't create new files,
122     and C<0666> or C<0777> if you do).
123 root 1.1
124     Example:
125    
126     aio_open "/etc/passwd", O_RDONLY, 0, sub {
127 root 1.2 if ($_[0]) {
128     print "open successful, fh is $_[0]\n";
129 root 1.1 ...
130     } else {
131     die "open failed: $!\n";
132     }
133     };
134    
135     =item aio_close $fh, $callback
136    
137 root 1.2 Asynchronously close a file and call the callback with the result
138     code. I<WARNING:> although accepted, you should not pass in a perl
139 root 1.20 filehandle here, as perl will likely close the file descriptor another
140     time when the filehandle is destroyed. Normally, you can safely call perls
141     C<close> or just let filehandles go out of scope.
142    
143     This is supposed to be a bug in the API, so that might change. It's
144     therefore best to avoid this function.
145 root 1.1
146     =item aio_read $fh,$offset,$length, $data,$dataoffset,$callback
147    
148     =item aio_write $fh,$offset,$length, $data,$dataoffset,$callback
149    
150     Reads or writes C<length> bytes from the specified C<fh> and C<offset>
151     into the scalar given by C<data> and offset C<dataoffset> and calls the
152     callback without the actual number of bytes read (or -1 on error, just
153     like the syscall).
154    
155 root 1.31 The C<$data> scalar I<MUST NOT> be modified in any way while the request
156     is outstanding. Modifying it can result in segfaults or WW3 (if the
157     necessary/optional hardware is installed).
158    
159 root 1.17 Example: Read 15 bytes at offset 7 into scalar C<$buffer>, starting at
160 root 1.1 offset C<0> within the scalar:
161    
162     aio_read $fh, 7, 15, $buffer, 0, sub {
163 root 1.9 $_[0] > 0 or die "read error: $!";
164     print "read $_[0] bytes: <$buffer>\n";
165 root 1.1 };
166    
167 root 1.35 =item aio_sendfile $out_fh, $in_fh, $in_offset, $length, $callback
168    
169     Tries to copy C<$length> bytes from C<$in_fh> to C<$out_fh>. It starts
170     reading at byte offset C<$in_offset>, and starts writing at the current
171     file offset of C<$out_fh>. Because of that, it is not safe to issue more
172     than one C<aio_sendfile> per C<$out_fh>, as they will interfere with each
173     other.
174    
175     This call tries to make use of a native C<sendfile> syscall to provide
176     zero-copy operation. For this to work, C<$out_fh> should refer to a
177     socket, and C<$in_fh> should refer to mmap'able file.
178    
179     If the native sendfile call fails or is not implemented, it will be
180     emulated, so you can call C<aio_sendfile> on any filehandles regardless of
181     the limitations of the OS.
182    
183     Please note, however, that C<aio_sendfile> can read more bytes from
184     C<$in_fh> than are written, and there is no way to find out how many
185     bytes have been read form C<aio_sendfile> alone, as C<aio_sendfile> only
186     provides the number of bytes written to C<$out_fh>. Only if the return
187     value (the value provided to the callback) equals C<$length> one can
188     assume that C<$length> bytes have been read.
189    
190 root 1.1 =item aio_readahead $fh,$offset,$length, $callback
191    
192 root 1.20 C<aio_readahead> populates the page cache with data from a file so that
193 root 1.1 subsequent reads from that file will not block on disk I/O. The C<$offset>
194     argument specifies the starting point from which data is to be read and
195     C<$length> specifies the number of bytes to be read. I/O is performed in
196     whole pages, so that offset is effectively rounded down to a page boundary
197     and bytes are read up to the next page boundary greater than or equal to
198 root 1.20 (off-set+length). C<aio_readahead> does not read beyond the end of the
199 root 1.1 file. The current file offset of the file is left unchanged.
200    
201 root 1.26 If that syscall doesn't exist (likely if your OS isn't Linux) it will be
202     emulated by simply reading the data, which would have a similar effect.
203    
204 root 1.1 =item aio_stat $fh_or_path, $callback
205    
206     =item aio_lstat $fh, $callback
207    
208     Works like perl's C<stat> or C<lstat> in void context. The callback will
209     be called after the stat and the results will be available using C<stat _>
210     or C<-s _> etc...
211    
212     The pathname passed to C<aio_stat> must be absolute. See API NOTES, above,
213     for an explanation.
214    
215     Currently, the stats are always 64-bit-stats, i.e. instead of returning an
216     error when stat'ing a large file, the results will be silently truncated
217     unless perl itself is compiled with large file support.
218    
219     Example: Print the length of F</etc/passwd>:
220    
221     aio_stat "/etc/passwd", sub {
222     $_[0] and die "stat failed: $!";
223     print "size is ", -s _, "\n";
224     };
225    
226     =item aio_unlink $pathname, $callback
227    
228     Asynchronously unlink (delete) a file and call the callback with the
229     result code.
230    
231 root 1.27 =item aio_rmdir $pathname, $callback
232    
233     Asynchronously rmdir (delete) a directory and call the callback with the
234     result code.
235    
236 root 1.1 =item aio_fsync $fh, $callback
237    
238     Asynchronously call fsync on the given filehandle and call the callback
239     with the fsync result code.
240    
241     =item aio_fdatasync $fh, $callback
242    
243     Asynchronously call fdatasync on the given filehandle and call the
244 root 1.26 callback with the fdatasync result code.
245    
246     If this call isn't available because your OS lacks it or it couldn't be
247     detected, it will be emulated by calling C<fsync> instead.
248 root 1.1
249 root 1.5 =back
250    
251     =head2 SUPPORT FUNCTIONS
252    
253     =over 4
254    
255     =item $fileno = IO::AIO::poll_fileno
256    
257 root 1.20 Return the I<request result pipe file descriptor>. This filehandle must be
258     polled for reading by some mechanism outside this module (e.g. Event or
259     select, see below or the SYNOPSIS). If the pipe becomes readable you have
260     to call C<poll_cb> to check the results.
261 root 1.5
262     See C<poll_cb> for an example.
263    
264     =item IO::AIO::poll_cb
265    
266     Process all outstanding events on the result pipe. You have to call this
267     regularly. Returns the number of events processed. Returns immediately
268     when no events are outstanding.
269    
270 root 1.20 Example: Install an Event watcher that automatically calls
271     IO::AIO::poll_cb with high priority:
272 root 1.5
273     Event->io (fd => IO::AIO::poll_fileno,
274     poll => 'r', async => 1,
275     cb => \&IO::AIO::poll_cb);
276    
277     =item IO::AIO::poll_wait
278    
279     Wait till the result filehandle becomes ready for reading (simply does a
280 root 1.20 C<select> on the filehandle. This is useful if you want to synchronously wait
281 root 1.5 for some requests to finish).
282    
283     See C<nreqs> for an example.
284    
285     =item IO::AIO::nreqs
286    
287 root 1.20 Returns the number of requests currently outstanding (i.e. for which their
288     callback has not been invoked yet).
289 root 1.5
290     Example: wait till there are no outstanding requests anymore:
291    
292     IO::AIO::poll_wait, IO::AIO::poll_cb
293     while IO::AIO::nreqs;
294    
295 root 1.12 =item IO::AIO::flush
296    
297     Wait till all outstanding AIO requests have been handled.
298    
299 root 1.13 Strictly equivalent to:
300    
301     IO::AIO::poll_wait, IO::AIO::poll_cb
302     while IO::AIO::nreqs;
303    
304     =item IO::AIO::poll
305    
306     Waits until some requests have been handled.
307    
308     Strictly equivalent to:
309    
310     IO::AIO::poll_wait, IO::AIO::poll_cb
311     if IO::AIO::nreqs;
312    
313 root 1.5 =item IO::AIO::min_parallel $nthreads
314    
315 root 1.34 Set the minimum number of AIO threads to C<$nthreads>. The current default
316     is C<4>, which means four asynchronous operations can be done at one time
317 root 1.5 (the number of outstanding operations, however, is unlimited).
318    
319 root 1.34 IO::AIO starts threads only on demand, when an AIO request is queued and
320     no free thread exists.
321    
322 root 1.5 It is recommended to keep the number of threads low, as some Linux
323     kernel versions will scale negatively with the number of threads (higher
324     parallelity => MUCH higher latency). With current Linux 2.6 versions, 4-32
325     threads should be fine.
326    
327 root 1.34 Under most circumstances you don't need to call this function, as the
328     module selects a default that is suitable for low to moderate load.
329 root 1.5
330     =item IO::AIO::max_parallel $nthreads
331    
332 root 1.34 Sets the maximum number of AIO threads to C<$nthreads>. If more than the
333     specified number of threads are currently running, this function kills
334     them. This function blocks until the limit is reached.
335    
336     While C<$nthreads> are zero, aio requests get queued but not executed
337     until the number of threads has been increased again.
338 root 1.5
339     This module automatically runs C<max_parallel 0> at program end, to ensure
340     that all threads are killed and that there are no outstanding requests.
341    
342     Under normal circumstances you don't need to call this function.
343    
344     =item $oldnreqs = IO::AIO::max_outstanding $nreqs
345    
346     Sets the maximum number of outstanding requests to C<$nreqs>. If you
347     try to queue up more than this number of requests, the caller will block until
348     some requests have been handled.
349    
350     The default is very large, so normally there is no practical limit. If you
351 root 1.34 queue up many requests in a loop it often improves speed if you set
352 root 1.5 this to a relatively low number, such as C<100>.
353    
354     Under normal circumstances you don't need to call this function.
355    
356     =back
357    
358 root 1.1 =cut
359    
360 root 1.2 # support function to convert a fd into a perl filehandle
361     sub _fd2fh {
362     return undef if $_[0] < 0;
363    
364 root 1.23 # try to generate nice filehandles
365     my $sym = "IO::AIO::fd#$_[0]";
366     local *$sym;
367 root 1.25
368 root 1.27 open *$sym, "+<&=$_[0]" # usually works under any unix
369     or open *$sym, "<&=$_[0]" # cygwin needs this
370     or open *$sym, ">&=$_[0]" # or this
371 root 1.2 or return undef;
372    
373 root 1.23 *$sym
374 root 1.2 }
375    
376 root 1.1 min_parallel 4;
377    
378     END {
379     max_parallel 0;
380     }
381    
382     1;
383    
384 root 1.27 =head2 FORK BEHAVIOUR
385    
386 root 1.34 Before the fork, IO::AIO enters a quiescent state where no requests
387     can be added in other threads and no results will be processed. After
388     the fork the parent simply leaves the quiescent state and continues
389     request/result processing, while the child clears the request/result
390     queue (so the requests started before the fork will only be handled in
391     the parent). Threats will be started on demand until the limit ste in the
392     parent process has been reached again.
393 root 1.27
394 root 1.1 =head1 SEE ALSO
395    
396     L<Coro>, L<Linux::AIO>.
397    
398     =head1 AUTHOR
399    
400     Marc Lehmann <schmorp@schmorp.de>
401     http://home.schmorp.de/
402    
403     =cut
404