ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/IO-AIO/AIO.pm
Revision: 1.34
Committed: Thu Aug 18 16:32:10 2005 UTC (18 years, 9 months ago) by root
Branch: MAIN
CVS Tags: rel-1_5
Changes since 1.33: +22 -15 lines
Log Message:
*** empty log message ***

File Contents

# User Rev Content
1 root 1.1 =head1 NAME
2    
3     IO::AIO - Asynchronous Input/Output
4    
5     =head1 SYNOPSIS
6    
7     use IO::AIO;
8    
9 root 1.6 aio_open "/etc/passwd", O_RDONLY, 0, sub {
10     my ($fh) = @_;
11     ...
12     };
13    
14     aio_unlink "/tmp/file", sub { };
15    
16     aio_read $fh, 30000, 1024, $buffer, 0, sub {
17 root 1.8 $_[0] > 0 or die "read error: $!";
18 root 1.6 };
19    
20     # Event
21     Event->io (fd => IO::AIO::poll_fileno,
22 root 1.7 poll => 'r',
23 root 1.6 cb => \&IO::AIO::poll_cb);
24    
25     # Glib/Gtk2
26     add_watch Glib::IO IO::AIO::poll_fileno,
27 root 1.22 in => sub { IO::AIO::poll_cb; 1 };
28 root 1.6
29     # Tk
30     Tk::Event::IO->fileevent (IO::AIO::poll_fileno, "",
31     readable => \&IO::AIO::poll_cb);
32    
33 root 1.11 # Danga::Socket
34     Danga::Socket->AddOtherFds (IO::AIO::poll_fileno =>
35     \&IO::AIO::poll_cb);
36    
37    
38 root 1.1 =head1 DESCRIPTION
39    
40     This module implements asynchronous I/O using whatever means your
41 root 1.2 operating system supports.
42 root 1.1
43 root 1.2 Currently, a number of threads are started that execute your read/writes
44     and signal their completion. You don't need thread support in your libc or
45     perl, and the threads created by this module will not be visible to the
46     pthreads library. In the future, this module might make use of the native
47     aio functions available on many operating systems. However, they are often
48     not well-supported (Linux doesn't allow them on normal files currently,
49     for example), and they would only support aio_read and aio_write, so the
50     remaining functionality would have to be implemented using threads anyway.
51 root 1.1
52     Although the module will work with in the presence of other threads, it is
53 root 1.22 currently not reentrant, so use appropriate locking yourself, always call
54     C<poll_cb> from within the same thread, or never call C<poll_cb> (or other
55     C<aio_> functions) recursively.
56 root 1.1
57     =cut
58    
59     package IO::AIO;
60    
61 root 1.23 no warnings;
62    
63 root 1.1 use base 'Exporter';
64    
65 root 1.2 use Fcntl ();
66    
67 root 1.1 BEGIN {
68 root 1.34 $VERSION = 1.5;
69 root 1.1
70     @EXPORT = qw(aio_read aio_write aio_open aio_close aio_stat aio_lstat aio_unlink
71 root 1.27 aio_rmdir aio_symlink aio_fsync aio_fdatasync aio_readahead);
72 root 1.4 @EXPORT_OK = qw(poll_fileno poll_cb min_parallel max_parallel max_outstanding nreqs);
73 root 1.1
74     require XSLoader;
75     XSLoader::load IO::AIO, $VERSION;
76     }
77    
78 root 1.5 =head1 FUNCTIONS
79 root 1.1
80 root 1.5 =head2 AIO FUNCTIONS
81 root 1.1
82 root 1.5 All the C<aio_*> calls are more or less thin wrappers around the syscall
83     with the same name (sans C<aio_>). The arguments are similar or identical,
84 root 1.14 and they all accept an additional (and optional) C<$callback> argument
85     which must be a code reference. This code reference will get called with
86     the syscall return code (e.g. most syscalls return C<-1> on error, unlike
87     perl, which usually delivers "false") as it's sole argument when the given
88     syscall has been executed asynchronously.
89 root 1.1
90 root 1.23 All functions expecting a filehandle keep a copy of the filehandle
91     internally until the request has finished.
92 root 1.1
93 root 1.28 The pathnames you pass to these routines I<must> be absolute and
94     encoded in byte form. The reason for the former is that at the time the
95     request is being executed, the current working directory could have
96     changed. Alternatively, you can make sure that you never change the
97     current working directory.
98    
99     To encode pathnames to byte form, either make sure you either: a)
100     always pass in filenames you got from outside (command line, readdir
101     etc.), b) are ASCII or ISO 8859-1, c) use the Encode module and encode
102     your pathnames to the locale (or other) encoding in effect in the user
103     environment, d) use Glib::filename_from_unicode on unicode filenames or e)
104     use something else.
105 root 1.1
106 root 1.5 =over 4
107 root 1.1
108     =item aio_open $pathname, $flags, $mode, $callback
109    
110 root 1.2 Asynchronously open or create a file and call the callback with a newly
111     created filehandle for the file.
112 root 1.1
113     The pathname passed to C<aio_open> must be absolute. See API NOTES, above,
114     for an explanation.
115    
116 root 1.20 The C<$flags> argument is a bitmask. See the C<Fcntl> module for a
117     list. They are the same as used by C<sysopen>.
118    
119     Likewise, C<$mode> specifies the mode of the newly created file, if it
120     didn't exist and C<O_CREAT> has been given, just like perl's C<sysopen>,
121     except that it is mandatory (i.e. use C<0> if you don't create new files,
122     and C<0666> or C<0777> if you do).
123 root 1.1
124     Example:
125    
126     aio_open "/etc/passwd", O_RDONLY, 0, sub {
127 root 1.2 if ($_[0]) {
128     print "open successful, fh is $_[0]\n";
129 root 1.1 ...
130     } else {
131     die "open failed: $!\n";
132     }
133     };
134    
135     =item aio_close $fh, $callback
136    
137 root 1.2 Asynchronously close a file and call the callback with the result
138     code. I<WARNING:> although accepted, you should not pass in a perl
139 root 1.20 filehandle here, as perl will likely close the file descriptor another
140     time when the filehandle is destroyed. Normally, you can safely call perls
141     C<close> or just let filehandles go out of scope.
142    
143     This is supposed to be a bug in the API, so that might change. It's
144     therefore best to avoid this function.
145 root 1.1
146     =item aio_read $fh,$offset,$length, $data,$dataoffset,$callback
147    
148     =item aio_write $fh,$offset,$length, $data,$dataoffset,$callback
149    
150     Reads or writes C<length> bytes from the specified C<fh> and C<offset>
151     into the scalar given by C<data> and offset C<dataoffset> and calls the
152     callback without the actual number of bytes read (or -1 on error, just
153     like the syscall).
154    
155 root 1.31 The C<$data> scalar I<MUST NOT> be modified in any way while the request
156     is outstanding. Modifying it can result in segfaults or WW3 (if the
157     necessary/optional hardware is installed).
158    
159 root 1.17 Example: Read 15 bytes at offset 7 into scalar C<$buffer>, starting at
160 root 1.1 offset C<0> within the scalar:
161    
162     aio_read $fh, 7, 15, $buffer, 0, sub {
163 root 1.9 $_[0] > 0 or die "read error: $!";
164     print "read $_[0] bytes: <$buffer>\n";
165 root 1.1 };
166    
167     =item aio_readahead $fh,$offset,$length, $callback
168    
169 root 1.20 C<aio_readahead> populates the page cache with data from a file so that
170 root 1.1 subsequent reads from that file will not block on disk I/O. The C<$offset>
171     argument specifies the starting point from which data is to be read and
172     C<$length> specifies the number of bytes to be read. I/O is performed in
173     whole pages, so that offset is effectively rounded down to a page boundary
174     and bytes are read up to the next page boundary greater than or equal to
175 root 1.20 (off-set+length). C<aio_readahead> does not read beyond the end of the
176 root 1.1 file. The current file offset of the file is left unchanged.
177    
178 root 1.26 If that syscall doesn't exist (likely if your OS isn't Linux) it will be
179     emulated by simply reading the data, which would have a similar effect.
180    
181 root 1.1 =item aio_stat $fh_or_path, $callback
182    
183     =item aio_lstat $fh, $callback
184    
185     Works like perl's C<stat> or C<lstat> in void context. The callback will
186     be called after the stat and the results will be available using C<stat _>
187     or C<-s _> etc...
188    
189     The pathname passed to C<aio_stat> must be absolute. See API NOTES, above,
190     for an explanation.
191    
192     Currently, the stats are always 64-bit-stats, i.e. instead of returning an
193     error when stat'ing a large file, the results will be silently truncated
194     unless perl itself is compiled with large file support.
195    
196     Example: Print the length of F</etc/passwd>:
197    
198     aio_stat "/etc/passwd", sub {
199     $_[0] and die "stat failed: $!";
200     print "size is ", -s _, "\n";
201     };
202    
203     =item aio_unlink $pathname, $callback
204    
205     Asynchronously unlink (delete) a file and call the callback with the
206     result code.
207    
208 root 1.27 =item aio_rmdir $pathname, $callback
209    
210     Asynchronously rmdir (delete) a directory and call the callback with the
211     result code.
212    
213 root 1.1 =item aio_fsync $fh, $callback
214    
215     Asynchronously call fsync on the given filehandle and call the callback
216     with the fsync result code.
217    
218     =item aio_fdatasync $fh, $callback
219    
220     Asynchronously call fdatasync on the given filehandle and call the
221 root 1.26 callback with the fdatasync result code.
222    
223     If this call isn't available because your OS lacks it or it couldn't be
224     detected, it will be emulated by calling C<fsync> instead.
225 root 1.1
226 root 1.5 =back
227    
228     =head2 SUPPORT FUNCTIONS
229    
230     =over 4
231    
232     =item $fileno = IO::AIO::poll_fileno
233    
234 root 1.20 Return the I<request result pipe file descriptor>. This filehandle must be
235     polled for reading by some mechanism outside this module (e.g. Event or
236     select, see below or the SYNOPSIS). If the pipe becomes readable you have
237     to call C<poll_cb> to check the results.
238 root 1.5
239     See C<poll_cb> for an example.
240    
241     =item IO::AIO::poll_cb
242    
243     Process all outstanding events on the result pipe. You have to call this
244     regularly. Returns the number of events processed. Returns immediately
245     when no events are outstanding.
246    
247 root 1.20 Example: Install an Event watcher that automatically calls
248     IO::AIO::poll_cb with high priority:
249 root 1.5
250     Event->io (fd => IO::AIO::poll_fileno,
251     poll => 'r', async => 1,
252     cb => \&IO::AIO::poll_cb);
253    
254     =item IO::AIO::poll_wait
255    
256     Wait till the result filehandle becomes ready for reading (simply does a
257 root 1.20 C<select> on the filehandle. This is useful if you want to synchronously wait
258 root 1.5 for some requests to finish).
259    
260     See C<nreqs> for an example.
261    
262     =item IO::AIO::nreqs
263    
264 root 1.20 Returns the number of requests currently outstanding (i.e. for which their
265     callback has not been invoked yet).
266 root 1.5
267     Example: wait till there are no outstanding requests anymore:
268    
269     IO::AIO::poll_wait, IO::AIO::poll_cb
270     while IO::AIO::nreqs;
271    
272 root 1.12 =item IO::AIO::flush
273    
274     Wait till all outstanding AIO requests have been handled.
275    
276 root 1.13 Strictly equivalent to:
277    
278     IO::AIO::poll_wait, IO::AIO::poll_cb
279     while IO::AIO::nreqs;
280    
281     =item IO::AIO::poll
282    
283     Waits until some requests have been handled.
284    
285     Strictly equivalent to:
286    
287     IO::AIO::poll_wait, IO::AIO::poll_cb
288     if IO::AIO::nreqs;
289    
290 root 1.5 =item IO::AIO::min_parallel $nthreads
291    
292 root 1.34 Set the minimum number of AIO threads to C<$nthreads>. The current default
293     is C<4>, which means four asynchronous operations can be done at one time
294 root 1.5 (the number of outstanding operations, however, is unlimited).
295    
296 root 1.34 IO::AIO starts threads only on demand, when an AIO request is queued and
297     no free thread exists.
298    
299 root 1.5 It is recommended to keep the number of threads low, as some Linux
300     kernel versions will scale negatively with the number of threads (higher
301     parallelity => MUCH higher latency). With current Linux 2.6 versions, 4-32
302     threads should be fine.
303    
304 root 1.34 Under most circumstances you don't need to call this function, as the
305     module selects a default that is suitable for low to moderate load.
306 root 1.5
307     =item IO::AIO::max_parallel $nthreads
308    
309 root 1.34 Sets the maximum number of AIO threads to C<$nthreads>. If more than the
310     specified number of threads are currently running, this function kills
311     them. This function blocks until the limit is reached.
312    
313     While C<$nthreads> are zero, aio requests get queued but not executed
314     until the number of threads has been increased again.
315 root 1.5
316     This module automatically runs C<max_parallel 0> at program end, to ensure
317     that all threads are killed and that there are no outstanding requests.
318    
319     Under normal circumstances you don't need to call this function.
320    
321     =item $oldnreqs = IO::AIO::max_outstanding $nreqs
322    
323     Sets the maximum number of outstanding requests to C<$nreqs>. If you
324     try to queue up more than this number of requests, the caller will block until
325     some requests have been handled.
326    
327     The default is very large, so normally there is no practical limit. If you
328 root 1.34 queue up many requests in a loop it often improves speed if you set
329 root 1.5 this to a relatively low number, such as C<100>.
330    
331     Under normal circumstances you don't need to call this function.
332    
333     =back
334    
335 root 1.1 =cut
336    
337 root 1.2 # support function to convert a fd into a perl filehandle
338     sub _fd2fh {
339     return undef if $_[0] < 0;
340    
341 root 1.23 # try to generate nice filehandles
342     my $sym = "IO::AIO::fd#$_[0]";
343     local *$sym;
344 root 1.25
345 root 1.27 open *$sym, "+<&=$_[0]" # usually works under any unix
346     or open *$sym, "<&=$_[0]" # cygwin needs this
347     or open *$sym, ">&=$_[0]" # or this
348 root 1.2 or return undef;
349    
350 root 1.23 *$sym
351 root 1.2 }
352    
353 root 1.1 min_parallel 4;
354    
355     END {
356     max_parallel 0;
357     }
358    
359     1;
360    
361 root 1.27 =head2 FORK BEHAVIOUR
362    
363 root 1.34 Before the fork, IO::AIO enters a quiescent state where no requests
364     can be added in other threads and no results will be processed. After
365     the fork the parent simply leaves the quiescent state and continues
366     request/result processing, while the child clears the request/result
367     queue (so the requests started before the fork will only be handled in
368     the parent). Threats will be started on demand until the limit ste in the
369     parent process has been reached again.
370 root 1.27
371 root 1.1 =head1 SEE ALSO
372    
373     L<Coro>, L<Linux::AIO>.
374    
375     =head1 AUTHOR
376    
377     Marc Lehmann <schmorp@schmorp.de>
378     http://home.schmorp.de/
379    
380     =cut
381