ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/IO-AIO/AIO.pm
Revision: 1.240
Committed: Mon Dec 17 06:18:39 2012 UTC (11 years, 5 months ago) by root
Branch: MAIN
Changes since 1.239: +18 -14 lines
Log Message:
*** empty log message ***

File Contents

# User Rev Content
1 root 1.1 =head1 NAME
2    
3     IO::AIO - Asynchronous Input/Output
4    
5     =head1 SYNOPSIS
6    
7     use IO::AIO;
8    
9 root 1.181 aio_open "/etc/passwd", IO::AIO::O_RDONLY, 0, sub {
10 root 1.94 my $fh = shift
11     or die "/etc/passwd: $!";
12 root 1.6 ...
13     };
14    
15     aio_unlink "/tmp/file", sub { };
16    
17     aio_read $fh, 30000, 1024, $buffer, 0, sub {
18 root 1.8 $_[0] > 0 or die "read error: $!";
19 root 1.6 };
20    
21 root 1.56 # version 2+ has request and group objects
22     use IO::AIO 2;
23 root 1.52
24 root 1.68 aioreq_pri 4; # give next request a very high priority
25 root 1.52 my $req = aio_unlink "/tmp/file", sub { };
26     $req->cancel; # cancel request if still in queue
27    
28 root 1.56 my $grp = aio_group sub { print "all stats done\n" };
29     add $grp aio_stat "..." for ...;
30    
31 root 1.1 =head1 DESCRIPTION
32    
33     This module implements asynchronous I/O using whatever means your
34 root 1.156 operating system supports. It is implemented as an interface to C<libeio>
35     (L<http://software.schmorp.de/pkg/libeio.html>).
36 root 1.1
37 root 1.85 Asynchronous means that operations that can normally block your program
38     (e.g. reading from disk) will be done asynchronously: the operation
39     will still block, but you can do something else in the meantime. This
40     is extremely useful for programs that need to stay interactive even
41     when doing heavy I/O (GUI programs, high performance network servers
42     etc.), but can also be used to easily do operations in parallel that are
43     normally done sequentially, e.g. stat'ing many files, which is much faster
44     on a RAID volume or over NFS when you do a number of stat operations
45     concurrently.
46    
47 root 1.108 While most of this works on all types of file descriptors (for
48     example sockets), using these functions on file descriptors that
49 root 1.156 support nonblocking operation (again, sockets, pipes etc.) is
50     very inefficient. Use an event loop for that (such as the L<EV>
51 root 1.108 module): IO::AIO will naturally fit into such an event loop itself.
52 root 1.85
53 root 1.72 In this version, a number of threads are started that execute your
54     requests and signal their completion. You don't need thread support
55     in perl, and the threads created by this module will not be visible
56     to perl. In the future, this module might make use of the native aio
57     functions available on many operating systems. However, they are often
58 root 1.85 not well-supported or restricted (GNU/Linux doesn't allow them on normal
59 root 1.72 files currently, for example), and they would only support aio_read and
60     aio_write, so the remaining functionality would have to be implemented
61     using threads anyway.
62    
63 root 1.108 Although the module will work in the presence of other (Perl-) threads,
64     it is currently not reentrant in any way, so use appropriate locking
65     yourself, always call C<poll_cb> from within the same thread, or never
66     call C<poll_cb> (or other C<aio_> functions) recursively.
67 root 1.72
68 root 1.86 =head2 EXAMPLE
69    
70 root 1.156 This is a simple example that uses the EV module and loads
71 root 1.86 F</etc/passwd> asynchronously:
72    
73     use Fcntl;
74 root 1.156 use EV;
75 root 1.86 use IO::AIO;
76    
77 root 1.156 # register the IO::AIO callback with EV
78     my $aio_w = EV::io IO::AIO::poll_fileno, EV::READ, \&IO::AIO::poll_cb;
79 root 1.86
80     # queue the request to open /etc/passwd
81 root 1.181 aio_open "/etc/passwd", IO::AIO::O_RDONLY, 0, sub {
82 root 1.94 my $fh = shift
83 root 1.86 or die "error while opening: $!";
84    
85     # stat'ing filehandles is generally non-blocking
86     my $size = -s $fh;
87    
88     # queue a request to read the file
89     my $contents;
90     aio_read $fh, 0, $size, $contents, 0, sub {
91     $_[0] == $size
92     or die "short read: $!";
93    
94     close $fh;
95    
96     # file contents now in $contents
97     print $contents;
98    
99     # exit event loop and program
100 root 1.156 EV::unloop;
101 root 1.86 };
102     };
103    
104     # possibly queue up other requests, or open GUI windows,
105     # check for sockets etc. etc.
106    
107     # process events as long as there are some:
108 root 1.156 EV::loop;
109 root 1.86
110 root 1.72 =head1 REQUEST ANATOMY AND LIFETIME
111    
112     Every C<aio_*> function creates a request. which is a C data structure not
113     directly visible to Perl.
114    
115     If called in non-void context, every request function returns a Perl
116     object representing the request. In void context, nothing is returned,
117     which saves a bit of memory.
118    
119     The perl object is a fairly standard ref-to-hash object. The hash contents
120     are not used by IO::AIO so you are free to store anything you like in it.
121    
122     During their existance, aio requests travel through the following states,
123     in order:
124    
125     =over 4
126    
127     =item ready
128    
129     Immediately after a request is created it is put into the ready state,
130     waiting for a thread to execute it.
131    
132     =item execute
133    
134     A thread has accepted the request for processing and is currently
135     executing it (e.g. blocking in read).
136    
137     =item pending
138    
139     The request has been executed and is waiting for result processing.
140    
141     While request submission and execution is fully asynchronous, result
142     processing is not and relies on the perl interpreter calling C<poll_cb>
143     (or another function with the same effect).
144    
145     =item result
146    
147     The request results are processed synchronously by C<poll_cb>.
148    
149     The C<poll_cb> function will process all outstanding aio requests by
150     calling their callbacks, freeing memory associated with them and managing
151     any groups they are contained in.
152    
153     =item done
154    
155     Request has reached the end of its lifetime and holds no resources anymore
156     (except possibly for the Perl object, but its connection to the actual
157     aio request is severed and calling its methods will either do nothing or
158     result in a runtime error).
159 root 1.1
160 root 1.88 =back
161    
162 root 1.1 =cut
163    
164     package IO::AIO;
165    
166 root 1.117 use Carp ();
167    
168 root 1.161 use common::sense;
169 root 1.23
170 root 1.1 use base 'Exporter';
171    
172     BEGIN {
173 root 1.238 our $VERSION = '4.18';
174 root 1.1
175 root 1.220 our @AIO_REQ = qw(aio_sendfile aio_seek aio_read aio_write aio_open aio_close
176 root 1.148 aio_stat aio_lstat aio_unlink aio_rmdir aio_readdir aio_readdirx
177 root 1.206 aio_scandir aio_symlink aio_readlink aio_realpath aio_sync
178 root 1.236 aio_fsync aio_syncfs aio_fdatasync aio_sync_file_range aio_allocate
179 root 1.222 aio_pathsync aio_readahead aio_fiemap
180 root 1.120 aio_rename aio_link aio_move aio_copy aio_group
181     aio_nop aio_mknod aio_load aio_rmtree aio_mkdir aio_chown
182 root 1.170 aio_chmod aio_utime aio_truncate
183 root 1.182 aio_msync aio_mtouch aio_mlock aio_mlockall
184 root 1.208 aio_statvfs
185     aio_wd);
186 root 1.120
187 root 1.123 our @EXPORT = (@AIO_REQ, qw(aioreq_pri aioreq_nice));
188 root 1.67 our @EXPORT_OK = qw(poll_fileno poll_cb poll_wait flush
189 root 1.188 min_parallel max_parallel max_idle idle_timeout
190 root 1.86 nreqs nready npending nthreads
191 root 1.157 max_poll_time max_poll_reqs
192 root 1.182 sendfile fadvise madvise
193     mmap munmap munlock munlockall);
194 root 1.1
195 root 1.143 push @AIO_REQ, qw(aio_busy); # not exported
196    
197 root 1.54 @IO::AIO::GRP::ISA = 'IO::AIO::REQ';
198    
199 root 1.1 require XSLoader;
200 root 1.51 XSLoader::load ("IO::AIO", $VERSION);
201 root 1.1 }
202    
203 root 1.5 =head1 FUNCTIONS
204 root 1.1
205 root 1.175 =head2 QUICK OVERVIEW
206    
207 root 1.230 This section simply lists the prototypes most of the functions for
208     quick reference. See the following sections for function-by-function
209 root 1.175 documentation.
210    
211 root 1.208 aio_wd $pathname, $callback->($wd)
212 root 1.175 aio_open $pathname, $flags, $mode, $callback->($fh)
213     aio_close $fh, $callback->($status)
214 root 1.220 aio_seek $fh,$offset,$whence, $callback->($offs)
215 root 1.175 aio_read $fh,$offset,$length, $data,$dataoffset, $callback->($retval)
216     aio_write $fh,$offset,$length, $data,$dataoffset, $callback->($retval)
217     aio_sendfile $out_fh, $in_fh, $in_offset, $length, $callback->($retval)
218     aio_readahead $fh,$offset,$length, $callback->($retval)
219     aio_stat $fh_or_path, $callback->($status)
220     aio_lstat $fh, $callback->($status)
221     aio_statvfs $fh_or_path, $callback->($statvfs)
222     aio_utime $fh_or_path, $atime, $mtime, $callback->($status)
223     aio_chown $fh_or_path, $uid, $gid, $callback->($status)
224 root 1.220 aio_chmod $fh_or_path, $mode, $callback->($status)
225 root 1.175 aio_truncate $fh_or_path, $offset, $callback->($status)
226 root 1.229 aio_allocate $fh, $mode, $offset, $len, $callback->($status)
227 root 1.230 aio_fiemap $fh, $start, $length, $flags, $count, $cb->(\@extents)
228 root 1.175 aio_unlink $pathname, $callback->($status)
229 root 1.209 aio_mknod $pathname, $mode, $dev, $callback->($status)
230 root 1.175 aio_link $srcpath, $dstpath, $callback->($status)
231     aio_symlink $srcpath, $dstpath, $callback->($status)
232 root 1.209 aio_readlink $pathname, $callback->($link)
233     aio_realpath $pathname, $callback->($link)
234 root 1.175 aio_rename $srcpath, $dstpath, $callback->($status)
235     aio_mkdir $pathname, $mode, $callback->($status)
236     aio_rmdir $pathname, $callback->($status)
237     aio_readdir $pathname, $callback->($entries)
238     aio_readdirx $pathname, $flags, $callback->($entries, $flags)
239     IO::AIO::READDIR_DENTS IO::AIO::READDIR_DIRS_FIRST
240     IO::AIO::READDIR_STAT_ORDER IO::AIO::READDIR_FOUND_UNKNOWN
241 root 1.215 aio_scandir $pathname, $maxreq, $callback->($dirs, $nondirs)
242 root 1.209 aio_load $pathname, $data, $callback->($status)
243 root 1.175 aio_copy $srcpath, $dstpath, $callback->($status)
244     aio_move $srcpath, $dstpath, $callback->($status)
245 root 1.209 aio_rmtree $pathname, $callback->($status)
246 root 1.175 aio_sync $callback->($status)
247 root 1.206 aio_syncfs $fh, $callback->($status)
248 root 1.175 aio_fsync $fh, $callback->($status)
249     aio_fdatasync $fh, $callback->($status)
250     aio_sync_file_range $fh, $offset, $nbytes, $flags, $callback->($status)
251 root 1.209 aio_pathsync $pathname, $callback->($status)
252 root 1.175 aio_msync $scalar, $offset = 0, $length = undef, flags = 0, $callback->($status)
253     aio_mtouch $scalar, $offset = 0, $length = undef, flags = 0, $callback->($status)
254 root 1.182 aio_mlock $scalar, $offset = 0, $length = undef, $callback->($status)
255     aio_mlockall $flags, $callback->($status)
256 root 1.175 aio_group $callback->(...)
257     aio_nop $callback->()
258    
259     $prev_pri = aioreq_pri [$pri]
260     aioreq_nice $pri_adjust
261    
262     IO::AIO::poll_wait
263     IO::AIO::poll_cb
264     IO::AIO::poll
265     IO::AIO::flush
266     IO::AIO::max_poll_reqs $nreqs
267     IO::AIO::max_poll_time $seconds
268     IO::AIO::min_parallel $nthreads
269     IO::AIO::max_parallel $nthreads
270     IO::AIO::max_idle $nthreads
271 root 1.188 IO::AIO::idle_timeout $seconds
272 root 1.175 IO::AIO::max_outstanding $maxreqs
273     IO::AIO::nreqs
274     IO::AIO::nready
275     IO::AIO::npending
276    
277     IO::AIO::sendfile $ofh, $ifh, $offset, $count
278     IO::AIO::fadvise $fh, $offset, $len, $advice
279 root 1.226 IO::AIO::mmap $scalar, $length, $prot, $flags[, $fh[, $offset]]
280     IO::AIO::munmap $scalar
281 root 1.184 IO::AIO::madvise $scalar, $offset, $length, $advice
282     IO::AIO::mprotect $scalar, $offset, $length, $protect
283 root 1.182 IO::AIO::munlock $scalar, $offset = 0, $length = undef
284 root 1.175 IO::AIO::munlockall
285    
286 root 1.219 =head2 API NOTES
287 root 1.1
288 root 1.5 All the C<aio_*> calls are more or less thin wrappers around the syscall
289     with the same name (sans C<aio_>). The arguments are similar or identical,
290 root 1.14 and they all accept an additional (and optional) C<$callback> argument
291 root 1.212 which must be a code reference. This code reference will be called after
292     the syscall has been executed in an asynchronous fashion. The results
293     of the request will be passed as arguments to the callback (and, if an
294     error occured, in C<$!>) - for most requests the syscall return code (e.g.
295     most syscalls return C<-1> on error, unlike perl, which usually delivers
296     "false").
297    
298     Some requests (such as C<aio_readdir>) pass the actual results and
299     communicate failures by passing C<undef>.
300 root 1.1
301 root 1.23 All functions expecting a filehandle keep a copy of the filehandle
302     internally until the request has finished.
303 root 1.1
304 root 1.87 All functions return request objects of type L<IO::AIO::REQ> that allow
305     further manipulation of those requests while they are in-flight.
306 root 1.52
307 root 1.209 The pathnames you pass to these routines I<should> be absolute. The
308     reason for this is that at the time the request is being executed, the
309 root 1.212 current working directory could have changed. Alternatively, you can
310     make sure that you never change the current working directory anywhere
311     in the program and then use relative paths. You can also take advantage
312     of IO::AIOs working directory abstraction, that lets you specify paths
313     relative to some previously-opened "working directory object" - see the
314     description of the C<IO::AIO::WD> class later in this document.
315 root 1.28
316 root 1.87 To encode pathnames as octets, either make sure you either: a) always pass
317     in filenames you got from outside (command line, readdir etc.) without
318 root 1.212 tinkering, b) are in your native filesystem encoding, c) use the Encode
319     module and encode your pathnames to the locale (or other) encoding in
320     effect in the user environment, d) use Glib::filename_from_unicode on
321     unicode filenames or e) use something else to ensure your scalar has the
322     correct contents.
323 root 1.87
324     This works, btw. independent of the internal UTF-8 bit, which IO::AIO
325 root 1.136 handles correctly whether it is set or not.
326 root 1.1
327 root 1.219 =head2 AIO REQUEST FUNCTIONS
328    
329 root 1.5 =over 4
330 root 1.1
331 root 1.80 =item $prev_pri = aioreq_pri [$pri]
332 root 1.68
333 root 1.80 Returns the priority value that would be used for the next request and, if
334     C<$pri> is given, sets the priority for the next aio request.
335 root 1.68
336 root 1.80 The default priority is C<0>, the minimum and maximum priorities are C<-4>
337     and C<4>, respectively. Requests with higher priority will be serviced
338     first.
339    
340     The priority will be reset to C<0> after each call to one of the C<aio_*>
341 root 1.68 functions.
342    
343 root 1.69 Example: open a file with low priority, then read something from it with
344     higher priority so the read request is serviced before other low priority
345     open requests (potentially spamming the cache):
346    
347     aioreq_pri -3;
348     aio_open ..., sub {
349     return unless $_[0];
350    
351     aioreq_pri -2;
352     aio_read $_[0], ..., sub {
353     ...
354     };
355     };
356    
357 root 1.106
358 root 1.69 =item aioreq_nice $pri_adjust
359    
360     Similar to C<aioreq_pri>, but subtracts the given value from the current
361 root 1.87 priority, so the effect is cumulative.
362 root 1.69
363 root 1.106
364 root 1.40 =item aio_open $pathname, $flags, $mode, $callback->($fh)
365 root 1.1
366 root 1.2 Asynchronously open or create a file and call the callback with a newly
367 root 1.233 created filehandle for the file (or C<undef> in case of an error).
368 root 1.1
369     The pathname passed to C<aio_open> must be absolute. See API NOTES, above,
370     for an explanation.
371    
372 root 1.20 The C<$flags> argument is a bitmask. See the C<Fcntl> module for a
373     list. They are the same as used by C<sysopen>.
374    
375     Likewise, C<$mode> specifies the mode of the newly created file, if it
376     didn't exist and C<O_CREAT> has been given, just like perl's C<sysopen>,
377     except that it is mandatory (i.e. use C<0> if you don't create new files,
378 root 1.101 and C<0666> or C<0777> if you do). Note that the C<$mode> will be modified
379     by the umask in effect then the request is being executed, so better never
380     change the umask.
381 root 1.1
382     Example:
383    
384 root 1.181 aio_open "/etc/passwd", IO::AIO::O_RDONLY, 0, sub {
385 root 1.2 if ($_[0]) {
386     print "open successful, fh is $_[0]\n";
387 root 1.1 ...
388     } else {
389     die "open failed: $!\n";
390     }
391     };
392    
393 root 1.194 In addition to all the common open modes/flags (C<O_RDONLY>, C<O_WRONLY>,
394     C<O_RDWR>, C<O_CREAT>, C<O_TRUNC>, C<O_EXCL> and C<O_APPEND>), the
395     following POSIX and non-POSIX constants are available (missing ones on
396     your system are, as usual, C<0>):
397    
398     C<O_ASYNC>, C<O_DIRECT>, C<O_NOATIME>, C<O_CLOEXEC>, C<O_NOCTTY>, C<O_NOFOLLOW>,
399     C<O_NONBLOCK>, C<O_EXEC>, C<O_SEARCH>, C<O_DIRECTORY>, C<O_DSYNC>,
400     C<O_RSYNC>, C<O_SYNC> and C<O_TTY_INIT>.
401    
402 root 1.106
403 root 1.40 =item aio_close $fh, $callback->($status)
404 root 1.1
405 root 1.2 Asynchronously close a file and call the callback with the result
406 root 1.116 code.
407    
408 root 1.117 Unfortunately, you can't do this to perl. Perl I<insists> very strongly on
409 root 1.121 closing the file descriptor associated with the filehandle itself.
410 root 1.117
411 root 1.121 Therefore, C<aio_close> will not close the filehandle - instead it will
412     use dup2 to overwrite the file descriptor with the write-end of a pipe
413     (the pipe fd will be created on demand and will be cached).
414 root 1.117
415 root 1.121 Or in other words: the file descriptor will be closed, but it will not be
416     free for reuse until the perl filehandle is closed.
417 root 1.117
418     =cut
419    
420 root 1.220 =item aio_seek $fh, $offset, $whence, $callback->($offs)
421    
422 root 1.221 Seeks the filehandle to the new C<$offset>, similarly to perl's
423 root 1.220 C<sysseek>. The C<$whence> can use the traditional values (C<0> for
424     C<IO::AIO::SEEK_SET>, C<1> for C<IO::AIO::SEEK_CUR> or C<2> for
425     C<IO::AIO::SEEK_END>).
426    
427     The resulting absolute offset will be passed to the callback, or C<-1> in
428     case of an error.
429    
430     In theory, the C<$whence> constants could be different than the
431     corresponding values from L<Fcntl>, but perl guarantees they are the same,
432     so don't panic.
433    
434 root 1.225 As a GNU/Linux (and maybe Solaris) extension, also the constants
435     C<IO::AIO::SEEK_DATA> and C<IO::AIO::SEEK_HOLE> are available, if they
436     could be found. No guarantees about suitability for use in C<aio_seek> or
437     Perl's C<sysseek> can be made though, although I would naively assume they
438     "just work".
439    
440 root 1.40 =item aio_read $fh,$offset,$length, $data,$dataoffset, $callback->($retval)
441 root 1.1
442 root 1.40 =item aio_write $fh,$offset,$length, $data,$dataoffset, $callback->($retval)
443 root 1.1
444 root 1.145 Reads or writes C<$length> bytes from or to the specified C<$fh> and
445     C<$offset> into the scalar given by C<$data> and offset C<$dataoffset>
446     and calls the callback without the actual number of bytes read (or -1 on
447     error, just like the syscall).
448 root 1.109
449 root 1.146 C<aio_read> will, like C<sysread>, shrink or grow the C<$data> scalar to
450     offset plus the actual number of bytes read.
451    
452 root 1.112 If C<$offset> is undefined, then the current file descriptor offset will
453     be used (and updated), otherwise the file descriptor offset will not be
454     changed by these calls.
455 root 1.109
456 root 1.145 If C<$length> is undefined in C<aio_write>, use the remaining length of
457     C<$data>.
458 root 1.109
459     If C<$dataoffset> is less than zero, it will be counted from the end of
460     C<$data>.
461 root 1.1
462 root 1.31 The C<$data> scalar I<MUST NOT> be modified in any way while the request
463 root 1.108 is outstanding. Modifying it can result in segfaults or World War III (if
464     the necessary/optional hardware is installed).
465 root 1.31
466 root 1.17 Example: Read 15 bytes at offset 7 into scalar C<$buffer>, starting at
467 root 1.1 offset C<0> within the scalar:
468    
469     aio_read $fh, 7, 15, $buffer, 0, sub {
470 root 1.9 $_[0] > 0 or die "read error: $!";
471     print "read $_[0] bytes: <$buffer>\n";
472 root 1.1 };
473    
474 root 1.106
475 root 1.40 =item aio_sendfile $out_fh, $in_fh, $in_offset, $length, $callback->($retval)
476 root 1.35
477     Tries to copy C<$length> bytes from C<$in_fh> to C<$out_fh>. It starts
478     reading at byte offset C<$in_offset>, and starts writing at the current
479     file offset of C<$out_fh>. Because of that, it is not safe to issue more
480     than one C<aio_sendfile> per C<$out_fh>, as they will interfere with each
481 root 1.196 other. The same C<$in_fh> works fine though, as this function does not
482     move or use the file offset of C<$in_fh>.
483 root 1.35
484 root 1.185 Please note that C<aio_sendfile> can read more bytes from C<$in_fh> than
485 root 1.196 are written, and there is no way to find out how many more bytes have been
486     read from C<aio_sendfile> alone, as C<aio_sendfile> only provides the
487     number of bytes written to C<$out_fh>. Only if the result value equals
488     C<$length> one can assume that C<$length> bytes have been read.
489 root 1.185
490     Unlike with other C<aio_> functions, it makes a lot of sense to use
491     C<aio_sendfile> on non-blocking sockets, as long as one end (typically
492     the C<$in_fh>) is a file - the file I/O will then be asynchronous, while
493 root 1.196 the socket I/O will be non-blocking. Note, however, that you can run
494     into a trap where C<aio_sendfile> reads some data with readahead, then
495     fails to write all data, and when the socket is ready the next time, the
496     data in the cache is already lost, forcing C<aio_sendfile> to again hit
497     the disk. Explicit C<aio_read> + C<aio_write> let's you better control
498     resource usage.
499    
500     This call tries to make use of a native C<sendfile>-like syscall to
501     provide zero-copy operation. For this to work, C<$out_fh> should refer to
502     a socket, and C<$in_fh> should refer to an mmap'able file.
503 root 1.35
504 root 1.170 If a native sendfile cannot be found or it fails with C<ENOSYS>,
505 root 1.196 C<EINVAL>, C<ENOTSUP>, C<EOPNOTSUPP>, C<EAFNOSUPPORT>, C<EPROTOTYPE> or
506     C<ENOTSOCK>, it will be emulated, so you can call C<aio_sendfile> on any
507     type of filehandle regardless of the limitations of the operating system.
508    
509     As native sendfile syscalls (as practically any non-POSIX interface hacked
510     together in a hurry to improve benchmark numbers) tend to be rather buggy
511     on many systems, this implementation tries to work around some known bugs
512     in Linux and FreeBSD kernels (probably others, too), but that might fail,
513     so you really really should check the return value of C<aio_sendfile> -
514     fewre bytes than expected might have been transferred.
515 root 1.35
516 root 1.106
517 root 1.40 =item aio_readahead $fh,$offset,$length, $callback->($retval)
518 root 1.1
519 root 1.20 C<aio_readahead> populates the page cache with data from a file so that
520 root 1.1 subsequent reads from that file will not block on disk I/O. The C<$offset>
521     argument specifies the starting point from which data is to be read and
522     C<$length> specifies the number of bytes to be read. I/O is performed in
523     whole pages, so that offset is effectively rounded down to a page boundary
524     and bytes are read up to the next page boundary greater than or equal to
525 root 1.20 (off-set+length). C<aio_readahead> does not read beyond the end of the
526 root 1.1 file. The current file offset of the file is left unchanged.
527    
528 root 1.26 If that syscall doesn't exist (likely if your OS isn't Linux) it will be
529     emulated by simply reading the data, which would have a similar effect.
530    
531 root 1.106
532 root 1.40 =item aio_stat $fh_or_path, $callback->($status)
533 root 1.1
534 root 1.40 =item aio_lstat $fh, $callback->($status)
535 root 1.1
536     Works like perl's C<stat> or C<lstat> in void context. The callback will
537     be called after the stat and the results will be available using C<stat _>
538     or C<-s _> etc...
539    
540     The pathname passed to C<aio_stat> must be absolute. See API NOTES, above,
541     for an explanation.
542    
543     Currently, the stats are always 64-bit-stats, i.e. instead of returning an
544     error when stat'ing a large file, the results will be silently truncated
545     unless perl itself is compiled with large file support.
546    
547 root 1.187 To help interpret the mode and dev/rdev stat values, IO::AIO offers the
548     following constants and functions (if not implemented, the constants will
549     be C<0> and the functions will either C<croak> or fall back on traditional
550     behaviour).
551    
552     C<S_IFMT>, C<S_IFIFO>, C<S_IFCHR>, C<S_IFBLK>, C<S_IFLNK>, C<S_IFREG>,
553     C<S_IFDIR>, C<S_IFWHT>, C<S_IFSOCK>, C<IO::AIO::major $dev_t>,
554     C<IO::AIO::minor $dev_t>, C<IO::AIO::makedev $major, $minor>.
555    
556 root 1.1 Example: Print the length of F</etc/passwd>:
557    
558     aio_stat "/etc/passwd", sub {
559     $_[0] and die "stat failed: $!";
560     print "size is ", -s _, "\n";
561     };
562    
563 root 1.106
564 root 1.175 =item aio_statvfs $fh_or_path, $callback->($statvfs)
565 root 1.172
566     Works like the POSIX C<statvfs> or C<fstatvfs> syscalls, depending on
567     whether a file handle or path was passed.
568    
569     On success, the callback is passed a hash reference with the following
570     members: C<bsize>, C<frsize>, C<blocks>, C<bfree>, C<bavail>, C<files>,
571     C<ffree>, C<favail>, C<fsid>, C<flag> and C<namemax>. On failure, C<undef>
572     is passed.
573    
574     The following POSIX IO::AIO::ST_* constants are defined: C<ST_RDONLY> and
575     C<ST_NOSUID>.
576    
577     The following non-POSIX IO::AIO::ST_* flag masks are defined to
578     their correct value when available, or to C<0> on systems that do
579     not support them: C<ST_NODEV>, C<ST_NOEXEC>, C<ST_SYNCHRONOUS>,
580     C<ST_MANDLOCK>, C<ST_WRITE>, C<ST_APPEND>, C<ST_IMMUTABLE>, C<ST_NOATIME>,
581     C<ST_NODIRATIME> and C<ST_RELATIME>.
582    
583     Example: stat C</wd> and dump out the data if successful.
584    
585     aio_statvfs "/wd", sub {
586     my $f = $_[0]
587     or die "statvfs: $!";
588    
589     use Data::Dumper;
590     say Dumper $f;
591     };
592    
593     # result:
594     {
595     bsize => 1024,
596     bfree => 4333064312,
597     blocks => 10253828096,
598     files => 2050765568,
599     flag => 4096,
600     favail => 2042092649,
601     bavail => 4333064312,
602     ffree => 2042092649,
603     namemax => 255,
604     frsize => 1024,
605     fsid => 1810
606     }
607    
608 root 1.235 Here is a (likely partial) list of fsid values used by Linux - it is safe
609     to hardcode these when the $^O is C<linux>:
610    
611     0x0000adf5 adfs
612     0x0000adff affs
613     0x5346414f afs
614     0x09041934 anon-inode filesystem
615     0x00000187 autofs
616     0x42465331 befs
617     0x1badface bfs
618     0x42494e4d binfmt_misc
619     0x9123683e btrfs
620     0x0027e0eb cgroupfs
621     0xff534d42 cifs
622     0x73757245 coda
623     0x012ff7b7 coh
624     0x28cd3d45 cramfs
625     0x453dcd28 cramfs-wend (wrong endianness)
626     0x64626720 debugfs
627     0x00001373 devfs
628     0x00001cd1 devpts
629     0x0000f15f ecryptfs
630     0x00414a53 efs
631     0x0000137d ext
632     0x0000ef53 ext2/ext3
633     0x0000ef51 ext2
634     0x00004006 fat
635     0x65735546 fuseblk
636     0x65735543 fusectl
637     0x0bad1dea futexfs
638     0x01161970 gfs2
639     0x47504653 gpfs
640     0x00004244 hfs
641     0xf995e849 hpfs
642     0x958458f6 hugetlbfs
643     0x2bad1dea inotifyfs
644     0x00009660 isofs
645     0x000072b6 jffs2
646     0x3153464a jfs
647     0x6b414653 k-afs
648     0x0bd00bd0 lustre
649     0x0000137f minix
650     0x0000138f minix 30 char names
651     0x00002468 minix v2
652     0x00002478 minix v2 30 char names
653     0x00004d5a minix v3
654     0x19800202 mqueue
655     0x00004d44 msdos
656     0x0000564c novell
657     0x00006969 nfs
658     0x6e667364 nfsd
659     0x00003434 nilfs
660     0x5346544e ntfs
661     0x00009fa1 openprom
662     0x7461636F ocfs2
663     0x00009fa0 proc
664     0x6165676c pstorefs
665     0x0000002f qnx4
666     0x858458f6 ramfs
667     0x52654973 reiserfs
668     0x00007275 romfs
669     0x67596969 rpc_pipefs
670     0x73636673 securityfs
671     0xf97cff8c selinux
672     0x0000517b smb
673     0x534f434b sockfs
674     0x73717368 squashfs
675     0x62656572 sysfs
676     0x012ff7b6 sysv2
677     0x012ff7b5 sysv4
678     0x01021994 tmpfs
679     0x15013346 udf
680     0x00011954 ufs
681     0x54190100 ufs byteswapped
682     0x00009fa2 usbdevfs
683     0x01021997 v9fs
684     0xa501fcf5 vxfs
685     0xabba1974 xenfs
686     0x012ff7b4 xenix
687     0x58465342 xfs
688     0x012fd16d xia
689 root 1.172
690 root 1.106 =item aio_utime $fh_or_path, $atime, $mtime, $callback->($status)
691    
692     Works like perl's C<utime> function (including the special case of $atime
693     and $mtime being undef). Fractional times are supported if the underlying
694     syscalls support them.
695    
696     When called with a pathname, uses utimes(2) if available, otherwise
697     utime(2). If called on a file descriptor, uses futimes(2) if available,
698     otherwise returns ENOSYS, so this is not portable.
699    
700     Examples:
701    
702 root 1.107 # set atime and mtime to current time (basically touch(1)):
703 root 1.106 aio_utime "path", undef, undef;
704     # set atime to current time and mtime to beginning of the epoch:
705     aio_utime "path", time, undef; # undef==0
706    
707    
708     =item aio_chown $fh_or_path, $uid, $gid, $callback->($status)
709    
710     Works like perl's C<chown> function, except that C<undef> for either $uid
711     or $gid is being interpreted as "do not change" (but -1 can also be used).
712    
713     Examples:
714    
715     # same as "chown root path" in the shell:
716     aio_chown "path", 0, -1;
717     # same as above:
718     aio_chown "path", 0, undef;
719    
720    
721 root 1.110 =item aio_truncate $fh_or_path, $offset, $callback->($status)
722    
723     Works like truncate(2) or ftruncate(2).
724    
725    
726 root 1.229 =item aio_allocate $fh, $mode, $offset, $len, $callback->($status)
727    
728     Allocates or freed disk space according to the C<$mode> argument. See the
729     linux C<fallocate> docuemntation for details.
730    
731     C<$mode> can currently be C<0> or C<IO::AIO::FALLOC_FL_KEEP_SIZE>
732     to allocate space, or C<IO::AIO::FALLOC_FL_PUNCH_HOLE |
733     IO::AIO::FALLOC_FL_KEEP_SIZE>, to deallocate a file range.
734    
735     The file system block size used by C<fallocate> is presumably the
736     C<f_bsize> returned by C<statvfs>.
737    
738     If C<fallocate> isn't available or cannot be emulated (currently no
739     emulation will be attempted), passes C<-1> and sets C<$!> to C<ENOSYS>.
740    
741    
742 root 1.106 =item aio_chmod $fh_or_path, $mode, $callback->($status)
743    
744     Works like perl's C<chmod> function.
745    
746    
747 root 1.40 =item aio_unlink $pathname, $callback->($status)
748 root 1.1
749     Asynchronously unlink (delete) a file and call the callback with the
750     result code.
751    
752 root 1.106
753 root 1.209 =item aio_mknod $pathname, $mode, $dev, $callback->($status)
754 root 1.82
755 root 1.86 [EXPERIMENTAL]
756    
757 root 1.83 Asynchronously create a device node (or fifo). See mknod(2).
758    
759 root 1.86 The only (POSIX-) portable way of calling this function is:
760 root 1.83
761 root 1.209 aio_mknod $pathname, IO::AIO::S_IFIFO | $mode, 0, sub { ...
762 root 1.82
763 root 1.187 See C<aio_stat> for info about some potentially helpful extra constants
764     and functions.
765 root 1.106
766 root 1.50 =item aio_link $srcpath, $dstpath, $callback->($status)
767    
768     Asynchronously create a new link to the existing object at C<$srcpath> at
769     the path C<$dstpath> and call the callback with the result code.
770    
771 root 1.106
772 root 1.50 =item aio_symlink $srcpath, $dstpath, $callback->($status)
773    
774     Asynchronously create a new symbolic link to the existing object at C<$srcpath> at
775     the path C<$dstpath> and call the callback with the result code.
776    
777 root 1.106
778 root 1.209 =item aio_readlink $pathname, $callback->($link)
779 root 1.90
780     Asynchronously read the symlink specified by C<$path> and pass it to
781     the callback. If an error occurs, nothing or undef gets passed to the
782     callback.
783    
784 root 1.106
785 root 1.209 =item aio_realpath $pathname, $callback->($path)
786 root 1.201
787     Asynchronously make the path absolute and resolve any symlinks in
788 root 1.239 C<$path>. The resulting path only consists of directories (same as
789 root 1.202 L<Cwd::realpath>).
790 root 1.201
791     This request can be used to get the absolute path of the current working
792     directory by passing it a path of F<.> (a single dot).
793    
794    
795 root 1.50 =item aio_rename $srcpath, $dstpath, $callback->($status)
796    
797     Asynchronously rename the object at C<$srcpath> to C<$dstpath>, just as
798     rename(2) and call the callback with the result code.
799    
800 root 1.106
801 root 1.101 =item aio_mkdir $pathname, $mode, $callback->($status)
802    
803     Asynchronously mkdir (create) a directory and call the callback with
804     the result code. C<$mode> will be modified by the umask at the time the
805     request is executed, so do not change your umask.
806    
807 root 1.106
808 root 1.40 =item aio_rmdir $pathname, $callback->($status)
809 root 1.27
810     Asynchronously rmdir (delete) a directory and call the callback with the
811     result code.
812    
813 root 1.106
814 root 1.46 =item aio_readdir $pathname, $callback->($entries)
815 root 1.37
816     Unlike the POSIX call of the same name, C<aio_readdir> reads an entire
817     directory (i.e. opendir + readdir + closedir). The entries will not be
818     sorted, and will B<NOT> include the C<.> and C<..> entries.
819    
820 root 1.148 The callback is passed a single argument which is either C<undef> or an
821     array-ref with the filenames.
822    
823    
824     =item aio_readdirx $pathname, $flags, $callback->($entries, $flags)
825    
826 root 1.207 Quite similar to C<aio_readdir>, but the C<$flags> argument allows one to
827     tune behaviour and output format. In case of an error, C<$entries> will be
828 root 1.148 C<undef>.
829    
830     The flags are a combination of the following constants, ORed together (the
831     flags will also be passed to the callback, possibly modified):
832    
833     =over 4
834    
835 root 1.150 =item IO::AIO::READDIR_DENTS
836 root 1.148
837 root 1.190 When this flag is off, then the callback gets an arrayref consisting of
838     names only (as with C<aio_readdir>), otherwise it gets an arrayref with
839 root 1.150 C<[$name, $type, $inode]> arrayrefs, each describing a single directory
840 root 1.148 entry in more detail.
841    
842     C<$name> is the name of the entry.
843    
844 root 1.150 C<$type> is one of the C<IO::AIO::DT_xxx> constants:
845 root 1.148
846 root 1.150 C<IO::AIO::DT_UNKNOWN>, C<IO::AIO::DT_FIFO>, C<IO::AIO::DT_CHR>, C<IO::AIO::DT_DIR>,
847     C<IO::AIO::DT_BLK>, C<IO::AIO::DT_REG>, C<IO::AIO::DT_LNK>, C<IO::AIO::DT_SOCK>,
848     C<IO::AIO::DT_WHT>.
849 root 1.148
850 root 1.150 C<IO::AIO::DT_UNKNOWN> means just that: readdir does not know. If you need to
851 root 1.148 know, you have to run stat yourself. Also, for speed reasons, the C<$type>
852     scalars are read-only: you can not modify them.
853    
854 root 1.150 C<$inode> is the inode number (which might not be exact on systems with 64
855 root 1.155 bit inode numbers and 32 bit perls). This field has unspecified content on
856     systems that do not deliver the inode information.
857 root 1.150
858     =item IO::AIO::READDIR_DIRS_FIRST
859 root 1.148
860     When this flag is set, then the names will be returned in an order where
861 root 1.193 likely directories come first, in optimal stat order. This is useful when
862     you need to quickly find directories, or you want to find all directories
863     while avoiding to stat() each entry.
864 root 1.148
865 root 1.149 If the system returns type information in readdir, then this is used
866 root 1.193 to find directories directly. Otherwise, likely directories are names
867     beginning with ".", or otherwise names with no dots, of which names with
868 root 1.149 short names are tried first.
869    
870 root 1.150 =item IO::AIO::READDIR_STAT_ORDER
871 root 1.148
872     When this flag is set, then the names will be returned in an order
873     suitable for stat()'ing each one. That is, when you plan to stat()
874     all files in the given directory, then the returned order will likely
875     be fastest.
876    
877 root 1.150 If both this flag and C<IO::AIO::READDIR_DIRS_FIRST> are specified, then
878     the likely dirs come first, resulting in a less optimal stat order.
879 root 1.148
880 root 1.150 =item IO::AIO::READDIR_FOUND_UNKNOWN
881 root 1.148
882     This flag should not be set when calling C<aio_readdirx>. Instead, it
883     is being set by C<aio_readdirx>, when any of the C<$type>'s found were
884 root 1.207 C<IO::AIO::DT_UNKNOWN>. The absence of this flag therefore indicates that all
885 root 1.148 C<$type>'s are known, which can be used to speed up some algorithms.
886    
887     =back
888 root 1.37
889 root 1.106
890 root 1.209 =item aio_load $pathname, $data, $callback->($status)
891 root 1.98
892     This is a composite request that tries to fully load the given file into
893     memory. Status is the same as with aio_read.
894    
895     =cut
896    
897     sub aio_load($$;$) {
898 root 1.123 my ($path, undef, $cb) = @_;
899     my $data = \$_[1];
900 root 1.98
901 root 1.123 my $pri = aioreq_pri;
902     my $grp = aio_group $cb;
903    
904     aioreq_pri $pri;
905     add $grp aio_open $path, O_RDONLY, 0, sub {
906     my $fh = shift
907     or return $grp->result (-1);
908 root 1.98
909     aioreq_pri $pri;
910 root 1.123 add $grp aio_read $fh, 0, (-s $fh), $$data, 0, sub {
911     $grp->result ($_[0]);
912 root 1.98 };
913 root 1.123 };
914 root 1.98
915 root 1.123 $grp
916 root 1.98 }
917    
918 root 1.82 =item aio_copy $srcpath, $dstpath, $callback->($status)
919    
920     Try to copy the I<file> (directories not supported as either source or
921     destination) from C<$srcpath> to C<$dstpath> and call the callback with
922 root 1.165 a status of C<0> (ok) or C<-1> (error, see C<$!>).
923 root 1.82
924 root 1.134 This is a composite request that creates the destination file with
925 root 1.82 mode 0200 and copies the contents of the source file into it using
926     C<aio_sendfile>, followed by restoring atime, mtime, access mode and
927     uid/gid, in that order.
928    
929     If an error occurs, the partial destination file will be unlinked, if
930     possible, except when setting atime, mtime, access mode and uid/gid, where
931     errors are being ignored.
932    
933     =cut
934    
935     sub aio_copy($$;$) {
936 root 1.123 my ($src, $dst, $cb) = @_;
937 root 1.82
938 root 1.123 my $pri = aioreq_pri;
939     my $grp = aio_group $cb;
940 root 1.82
941 root 1.123 aioreq_pri $pri;
942     add $grp aio_open $src, O_RDONLY, 0, sub {
943     if (my $src_fh = $_[0]) {
944 root 1.166 my @stat = stat $src_fh; # hmm, might block over nfs?
945 root 1.95
946 root 1.123 aioreq_pri $pri;
947     add $grp aio_open $dst, O_CREAT | O_WRONLY | O_TRUNC, 0200, sub {
948     if (my $dst_fh = $_[0]) {
949     aioreq_pri $pri;
950     add $grp aio_sendfile $dst_fh, $src_fh, 0, $stat[7], sub {
951     if ($_[0] == $stat[7]) {
952     $grp->result (0);
953     close $src_fh;
954    
955 root 1.147 my $ch = sub {
956     aioreq_pri $pri;
957     add $grp aio_chmod $dst_fh, $stat[2] & 07777, sub {
958     aioreq_pri $pri;
959     add $grp aio_chown $dst_fh, $stat[4], $stat[5], sub {
960     aioreq_pri $pri;
961     add $grp aio_close $dst_fh;
962     }
963     };
964     };
965 root 1.123
966     aioreq_pri $pri;
967 root 1.147 add $grp aio_utime $dst_fh, $stat[8], $stat[9], sub {
968     if ($_[0] < 0 && $! == ENOSYS) {
969     aioreq_pri $pri;
970     add $grp aio_utime $dst, $stat[8], $stat[9], $ch;
971     } else {
972     $ch->();
973     }
974     };
975 root 1.123 } else {
976     $grp->result (-1);
977     close $src_fh;
978     close $dst_fh;
979    
980     aioreq $pri;
981     add $grp aio_unlink $dst;
982     }
983     };
984     } else {
985     $grp->result (-1);
986     }
987     },
988 root 1.82
989 root 1.123 } else {
990     $grp->result (-1);
991     }
992     };
993 root 1.82
994 root 1.123 $grp
995 root 1.82 }
996    
997     =item aio_move $srcpath, $dstpath, $callback->($status)
998    
999     Try to move the I<file> (directories not supported as either source or
1000     destination) from C<$srcpath> to C<$dstpath> and call the callback with
1001 root 1.165 a status of C<0> (ok) or C<-1> (error, see C<$!>).
1002 root 1.82
1003 root 1.137 This is a composite request that tries to rename(2) the file first; if
1004     rename fails with C<EXDEV>, it copies the file with C<aio_copy> and, if
1005     that is successful, unlinks the C<$srcpath>.
1006 root 1.82
1007     =cut
1008    
1009     sub aio_move($$;$) {
1010 root 1.123 my ($src, $dst, $cb) = @_;
1011 root 1.82
1012 root 1.123 my $pri = aioreq_pri;
1013     my $grp = aio_group $cb;
1014 root 1.82
1015 root 1.123 aioreq_pri $pri;
1016     add $grp aio_rename $src, $dst, sub {
1017     if ($_[0] && $! == EXDEV) {
1018     aioreq_pri $pri;
1019     add $grp aio_copy $src, $dst, sub {
1020     $grp->result ($_[0]);
1021 root 1.95
1022 root 1.196 unless ($_[0]) {
1023 root 1.123 aioreq_pri $pri;
1024     add $grp aio_unlink $src;
1025     }
1026     };
1027     } else {
1028     $grp->result ($_[0]);
1029     }
1030     };
1031 root 1.82
1032 root 1.123 $grp
1033 root 1.82 }
1034    
1035 root 1.209 =item aio_scandir $pathname, $maxreq, $callback->($dirs, $nondirs)
1036 root 1.40
1037 root 1.52 Scans a directory (similar to C<aio_readdir>) but additionally tries to
1038 root 1.76 efficiently separate the entries of directory C<$path> into two sets of
1039     names, directories you can recurse into (directories), and ones you cannot
1040     recurse into (everything else, including symlinks to directories).
1041 root 1.52
1042 root 1.61 C<aio_scandir> is a composite request that creates of many sub requests_
1043     C<$maxreq> specifies the maximum number of outstanding aio requests that
1044     this function generates. If it is C<< <= 0 >>, then a suitable default
1045 root 1.81 will be chosen (currently 4).
1046 root 1.40
1047     On error, the callback is called without arguments, otherwise it receives
1048     two array-refs with path-relative entry names.
1049    
1050     Example:
1051    
1052     aio_scandir $dir, 0, sub {
1053     my ($dirs, $nondirs) = @_;
1054     print "real directories: @$dirs\n";
1055     print "everything else: @$nondirs\n";
1056     };
1057    
1058     Implementation notes.
1059    
1060     The C<aio_readdir> cannot be avoided, but C<stat()>'ing every entry can.
1061    
1062 root 1.149 If readdir returns file type information, then this is used directly to
1063     find directories.
1064    
1065     Otherwise, after reading the directory, the modification time, size etc.
1066     of the directory before and after the readdir is checked, and if they
1067     match (and isn't the current time), the link count will be used to decide
1068     how many entries are directories (if >= 2). Otherwise, no knowledge of the
1069     number of subdirectories will be assumed.
1070    
1071     Then entries will be sorted into likely directories a non-initial dot
1072     currently) and likely non-directories (see C<aio_readdirx>). Then every
1073     entry plus an appended C</.> will be C<stat>'ed, likely directories first,
1074     in order of their inode numbers. If that succeeds, it assumes that the
1075     entry is a directory or a symlink to directory (which will be checked
1076 root 1.207 separately). This is often faster than stat'ing the entry itself because
1077 root 1.52 filesystems might detect the type of the entry without reading the inode
1078 root 1.149 data (e.g. ext2fs filetype feature), even on systems that cannot return
1079     the filetype information on readdir.
1080 root 1.52
1081     If the known number of directories (link count - 2) has been reached, the
1082     rest of the entries is assumed to be non-directories.
1083    
1084     This only works with certainty on POSIX (= UNIX) filesystems, which
1085     fortunately are the vast majority of filesystems around.
1086    
1087     It will also likely work on non-POSIX filesystems with reduced efficiency
1088     as those tend to return 0 or 1 as link counts, which disables the
1089     directory counting heuristic.
1090 root 1.40
1091     =cut
1092    
1093 root 1.100 sub aio_scandir($$;$) {
1094 root 1.123 my ($path, $maxreq, $cb) = @_;
1095    
1096     my $pri = aioreq_pri;
1097 root 1.40
1098 root 1.123 my $grp = aio_group $cb;
1099 root 1.80
1100 root 1.123 $maxreq = 4 if $maxreq <= 0;
1101 root 1.55
1102 root 1.210 # get a wd object
1103 root 1.123 aioreq_pri $pri;
1104 root 1.210 add $grp aio_wd $path, sub {
1105 root 1.212 $_[0]
1106     or return $grp->result ();
1107    
1108 root 1.210 my $wd = [shift, "."];
1109 root 1.40
1110 root 1.210 # stat once
1111 root 1.80 aioreq_pri $pri;
1112 root 1.210 add $grp aio_stat $wd, sub {
1113     return $grp->result () if $_[0];
1114     my $now = time;
1115     my $hash1 = join ":", (stat _)[0,1,3,7,9];
1116 root 1.40
1117 root 1.210 # read the directory entries
1118 root 1.80 aioreq_pri $pri;
1119 root 1.210 add $grp aio_readdirx $wd, READDIR_DIRS_FIRST, sub {
1120     my $entries = shift
1121     or return $grp->result ();
1122    
1123     # stat the dir another time
1124     aioreq_pri $pri;
1125     add $grp aio_stat $wd, sub {
1126     my $hash2 = join ":", (stat _)[0,1,3,7,9];
1127 root 1.95
1128 root 1.210 my $ndirs;
1129 root 1.95
1130 root 1.210 # take the slow route if anything looks fishy
1131     if ($hash1 ne $hash2 or (stat _)[9] == $now) {
1132     $ndirs = -1;
1133     } else {
1134     # if nlink == 2, we are finished
1135     # for non-posix-fs's, we rely on nlink < 2
1136     $ndirs = (stat _)[3] - 2
1137     or return $grp->result ([], $entries);
1138     }
1139 root 1.123
1140 root 1.210 my (@dirs, @nondirs);
1141 root 1.40
1142 root 1.210 my $statgrp = add $grp aio_group sub {
1143     $grp->result (\@dirs, \@nondirs);
1144     };
1145 root 1.40
1146 root 1.210 limit $statgrp $maxreq;
1147     feed $statgrp sub {
1148     return unless @$entries;
1149     my $entry = shift @$entries;
1150    
1151     aioreq_pri $pri;
1152     $wd->[1] = "$entry/.";
1153     add $statgrp aio_stat $wd, sub {
1154     if ($_[0] < 0) {
1155     push @nondirs, $entry;
1156     } else {
1157     # need to check for real directory
1158     aioreq_pri $pri;
1159     $wd->[1] = $entry;
1160     add $statgrp aio_lstat $wd, sub {
1161     if (-d _) {
1162     push @dirs, $entry;
1163    
1164     unless (--$ndirs) {
1165     push @nondirs, @$entries;
1166     feed $statgrp;
1167     }
1168     } else {
1169     push @nondirs, $entry;
1170 root 1.74 }
1171 root 1.40 }
1172     }
1173 root 1.210 };
1174 root 1.74 };
1175 root 1.40 };
1176     };
1177     };
1178 root 1.123 };
1179 root 1.55
1180 root 1.123 $grp
1181 root 1.40 }
1182    
1183 root 1.209 =item aio_rmtree $pathname, $callback->($status)
1184 root 1.99
1185 root 1.100 Delete a directory tree starting (and including) C<$path>, return the
1186 root 1.239 status of the final C<rmdir> only. This is a composite request that
1187 root 1.100 uses C<aio_scandir> to recurse into and rmdir directories, and unlink
1188     everything else.
1189 root 1.99
1190     =cut
1191    
1192     sub aio_rmtree;
1193 root 1.100 sub aio_rmtree($;$) {
1194 root 1.123 my ($path, $cb) = @_;
1195 root 1.99
1196 root 1.123 my $pri = aioreq_pri;
1197     my $grp = aio_group $cb;
1198 root 1.99
1199 root 1.123 aioreq_pri $pri;
1200     add $grp aio_scandir $path, 0, sub {
1201     my ($dirs, $nondirs) = @_;
1202 root 1.99
1203 root 1.123 my $dirgrp = aio_group sub {
1204     add $grp aio_rmdir $path, sub {
1205     $grp->result ($_[0]);
1206 root 1.99 };
1207 root 1.123 };
1208 root 1.99
1209 root 1.123 (aioreq_pri $pri), add $dirgrp aio_rmtree "$path/$_" for @$dirs;
1210     (aioreq_pri $pri), add $dirgrp aio_unlink "$path/$_" for @$nondirs;
1211 root 1.99
1212 root 1.123 add $grp $dirgrp;
1213     };
1214 root 1.99
1215 root 1.123 $grp
1216 root 1.99 }
1217    
1218 root 1.119 =item aio_sync $callback->($status)
1219    
1220     Asynchronously call sync and call the callback when finished.
1221    
1222 root 1.40 =item aio_fsync $fh, $callback->($status)
1223 root 1.1
1224     Asynchronously call fsync on the given filehandle and call the callback
1225     with the fsync result code.
1226    
1227 root 1.40 =item aio_fdatasync $fh, $callback->($status)
1228 root 1.1
1229     Asynchronously call fdatasync on the given filehandle and call the
1230 root 1.26 callback with the fdatasync result code.
1231    
1232     If this call isn't available because your OS lacks it or it couldn't be
1233     detected, it will be emulated by calling C<fsync> instead.
1234 root 1.1
1235 root 1.206 =item aio_syncfs $fh, $callback->($status)
1236    
1237     Asynchronously call the syncfs syscall to sync the filesystem associated
1238     to the given filehandle and call the callback with the syncfs result
1239     code. If syncfs is not available, calls sync(), but returns C<-1> and sets
1240     errno to C<ENOSYS> nevertheless.
1241    
1242 root 1.142 =item aio_sync_file_range $fh, $offset, $nbytes, $flags, $callback->($status)
1243    
1244     Sync the data portion of the file specified by C<$offset> and C<$length>
1245     to disk (but NOT the metadata), by calling the Linux-specific
1246     sync_file_range call. If sync_file_range is not available or it returns
1247     ENOSYS, then fdatasync or fsync is being substituted.
1248    
1249     C<$flags> can be a combination of C<IO::AIO::SYNC_FILE_RANGE_WAIT_BEFORE>,
1250     C<IO::AIO::SYNC_FILE_RANGE_WRITE> and
1251     C<IO::AIO::SYNC_FILE_RANGE_WAIT_AFTER>: refer to the sync_file_range
1252     manpage for details.
1253    
1254 root 1.209 =item aio_pathsync $pathname, $callback->($status)
1255 root 1.120
1256     This request tries to open, fsync and close the given path. This is a
1257 root 1.135 composite request intended to sync directories after directory operations
1258 root 1.120 (E.g. rename). This might not work on all operating systems or have any
1259     specific effect, but usually it makes sure that directory changes get
1260     written to disc. It works for anything that can be opened for read-only,
1261     not just directories.
1262    
1263 root 1.162 Future versions of this function might fall back to other methods when
1264     C<fsync> on the directory fails (such as calling C<sync>).
1265    
1266 root 1.120 Passes C<0> when everything went ok, and C<-1> on error.
1267    
1268     =cut
1269    
1270     sub aio_pathsync($;$) {
1271 root 1.123 my ($path, $cb) = @_;
1272    
1273     my $pri = aioreq_pri;
1274     my $grp = aio_group $cb;
1275 root 1.120
1276 root 1.123 aioreq_pri $pri;
1277     add $grp aio_open $path, O_RDONLY, 0, sub {
1278     my ($fh) = @_;
1279     if ($fh) {
1280     aioreq_pri $pri;
1281     add $grp aio_fsync $fh, sub {
1282     $grp->result ($_[0]);
1283 root 1.120
1284     aioreq_pri $pri;
1285 root 1.123 add $grp aio_close $fh;
1286     };
1287     } else {
1288     $grp->result (-1);
1289     }
1290     };
1291 root 1.120
1292 root 1.123 $grp
1293 root 1.120 }
1294    
1295 root 1.170 =item aio_msync $scalar, $offset = 0, $length = undef, flags = 0, $callback->($status)
1296    
1297     This is a rather advanced IO::AIO call, which only works on mmap(2)ed
1298 root 1.176 scalars (see the C<IO::AIO::mmap> function, although it also works on data
1299     scalars managed by the L<Sys::Mmap> or L<Mmap> modules, note that the
1300     scalar must only be modified in-place while an aio operation is pending on
1301     it).
1302 root 1.170
1303     It calls the C<msync> function of your OS, if available, with the memory
1304     area starting at C<$offset> in the string and ending C<$length> bytes
1305     later. If C<$length> is negative, counts from the end, and if C<$length>
1306     is C<undef>, then it goes till the end of the string. The flags can be
1307     a combination of C<IO::AIO::MS_ASYNC>, C<IO::AIO::MS_INVALIDATE> and
1308     C<IO::AIO::MS_SYNC>.
1309    
1310     =item aio_mtouch $scalar, $offset = 0, $length = undef, flags = 0, $callback->($status)
1311    
1312     This is a rather advanced IO::AIO call, which works best on mmap(2)ed
1313     scalars.
1314    
1315     It touches (reads or writes) all memory pages in the specified
1316 root 1.239 range inside the scalar. All caveats and parameters are the same
1317 root 1.170 as for C<aio_msync>, above, except for flags, which must be either
1318     C<0> (which reads all pages and ensures they are instantiated) or
1319 root 1.239 C<IO::AIO::MT_MODIFY>, which modifies the memory pages (by reading and
1320 root 1.170 writing an octet from it, which dirties the page).
1321    
1322 root 1.182 =item aio_mlock $scalar, $offset = 0, $length = undef, $callback->($status)
1323    
1324     This is a rather advanced IO::AIO call, which works best on mmap(2)ed
1325     scalars.
1326    
1327     It reads in all the pages of the underlying storage into memory (if any)
1328     and locks them, so they are not getting swapped/paged out or removed.
1329    
1330     If C<$length> is undefined, then the scalar will be locked till the end.
1331    
1332     On systems that do not implement C<mlock>, this function returns C<-1>
1333     and sets errno to C<ENOSYS>.
1334    
1335     Note that the corresponding C<munlock> is synchronous and is
1336     documented under L<MISCELLANEOUS FUNCTIONS>.
1337    
1338 root 1.183 Example: open a file, mmap and mlock it - both will be undone when
1339     C<$data> gets destroyed.
1340    
1341     open my $fh, "<", $path or die "$path: $!";
1342     my $data;
1343     IO::AIO::mmap $data, -s $fh, IO::AIO::PROT_READ, IO::AIO::MAP_SHARED, $fh;
1344     aio_mlock $data; # mlock in background
1345    
1346 root 1.182 =item aio_mlockall $flags, $callback->($status)
1347    
1348     Calls the C<mlockall> function with the given C<$flags> (a combination of
1349     C<IO::AIO::MCL_CURRENT> and C<IO::AIO::MCL_FUTURE>).
1350    
1351     On systems that do not implement C<mlockall>, this function returns C<-1>
1352     and sets errno to C<ENOSYS>.
1353    
1354     Note that the corresponding C<munlockall> is synchronous and is
1355     documented under L<MISCELLANEOUS FUNCTIONS>.
1356    
1357 root 1.183 Example: asynchronously lock all current and future pages into memory.
1358    
1359     aio_mlockall IO::AIO::MCL_FUTURE;
1360    
1361 root 1.223 =item aio_fiemap $fh, $start, $length, $flags, $count, $cb->(\@extents)
1362    
1363 root 1.234 Queries the extents of the given file (by calling the Linux C<FIEMAP>
1364     ioctl, see L<http://cvs.schmorp.de/IO-AIO/doc/fiemap.txt> for details). If
1365     the ioctl is not available on your OS, then this request will fail with
1366 root 1.223 C<ENOSYS>.
1367    
1368     C<$start> is the starting offset to query extents for, C<$length> is the
1369     size of the range to query - if it is C<undef>, then the whole file will
1370     be queried.
1371    
1372     C<$flags> is a combination of flags (C<IO::AIO::FIEMAP_FLAG_SYNC> or
1373     C<IO::AIO::FIEMAP_FLAG_XATTR> - C<IO::AIO::FIEMAP_FLAGS_COMPAT> is also
1374     exported), and is normally C<0> or C<IO::AIO::FIEMAP_FLAG_SYNC> to query
1375     the data portion.
1376    
1377     C<$count> is the maximum number of extent records to return. If it is
1378 root 1.232 C<undef>, then IO::AIO queries all extents of the range. As a very special
1379 root 1.223 case, if it is C<0>, then the callback receives the number of extents
1380 root 1.232 instead of the extents themselves (which is unreliable, see below).
1381 root 1.223
1382     If an error occurs, the callback receives no arguments. The special
1383     C<errno> value C<IO::AIO::EBADR> is available to test for flag errors.
1384    
1385     Otherwise, the callback receives an array reference with extent
1386     structures. Each extent structure is an array reference itself, with the
1387     following members:
1388    
1389     [$logical, $physical, $length, $flags]
1390    
1391     Flags is any combination of the following flag values (typically either C<0>
1392 root 1.231 or C<IO::AIO::FIEMAP_EXTENT_LAST> (1)):
1393 root 1.223
1394     C<IO::AIO::FIEMAP_EXTENT_LAST>, C<IO::AIO::FIEMAP_EXTENT_UNKNOWN>,
1395     C<IO::AIO::FIEMAP_EXTENT_DELALLOC>, C<IO::AIO::FIEMAP_EXTENT_ENCODED>,
1396     C<IO::AIO::FIEMAP_EXTENT_DATA_ENCRYPTED>, C<IO::AIO::FIEMAP_EXTENT_NOT_ALIGNED>,
1397     C<IO::AIO::FIEMAP_EXTENT_DATA_INLINE>, C<IO::AIO::FIEMAP_EXTENT_DATA_TAIL>,
1398     C<IO::AIO::FIEMAP_EXTENT_UNWRITTEN>, C<IO::AIO::FIEMAP_EXTENT_MERGED> or
1399     C<IO::AIO::FIEMAP_EXTENT_SHARED>.
1400    
1401 root 1.232 At the time of this writing (Linux 3.2), this requets is unreliable unless
1402     C<$count> is C<undef>, as the kernel has all sorts of bugs preventing
1403     it to return all extents of a range for files with large number of
1404     extents. The code works around all these issues if C<$count> is undef.
1405    
1406 root 1.58 =item aio_group $callback->(...)
1407 root 1.54
1408 root 1.55 This is a very special aio request: Instead of doing something, it is a
1409     container for other aio requests, which is useful if you want to bundle
1410 root 1.71 many requests into a single, composite, request with a definite callback
1411     and the ability to cancel the whole request with its subrequests.
1412 root 1.55
1413     Returns an object of class L<IO::AIO::GRP>. See its documentation below
1414     for more info.
1415    
1416     Example:
1417    
1418     my $grp = aio_group sub {
1419     print "all stats done\n";
1420     };
1421    
1422     add $grp
1423     (aio_stat ...),
1424     (aio_stat ...),
1425     ...;
1426    
1427 root 1.63 =item aio_nop $callback->()
1428    
1429     This is a special request - it does nothing in itself and is only used for
1430     side effects, such as when you want to add a dummy request to a group so
1431     that finishing the requests in the group depends on executing the given
1432     code.
1433    
1434 root 1.64 While this request does nothing, it still goes through the execution
1435     phase and still requires a worker thread. Thus, the callback will not
1436     be executed immediately but only after other requests in the queue have
1437     entered their execution phase. This can be used to measure request
1438     latency.
1439    
1440 root 1.71 =item IO::AIO::aio_busy $fractional_seconds, $callback->() *NOT EXPORTED*
1441 root 1.54
1442     Mainly used for debugging and benchmarking, this aio request puts one of
1443     the request workers to sleep for the given time.
1444    
1445 root 1.56 While it is theoretically handy to have simple I/O scheduling requests
1446 root 1.71 like sleep and file handle readable/writable, the overhead this creates is
1447     immense (it blocks a thread for a long time) so do not use this function
1448     except to put your application under artificial I/O pressure.
1449 root 1.56
1450 root 1.5 =back
1451    
1452 root 1.209
1453     =head2 IO::AIO::WD - multiple working directories
1454    
1455     Your process only has one current working directory, which is used by all
1456     threads. This makes it hard to use relative paths (some other component
1457     could call C<chdir> at any time, and it is hard to control when the path
1458     will be used by IO::AIO).
1459    
1460     One solution for this is to always use absolute paths. This usually works,
1461     but can be quite slow (the kernel has to walk the whole path on every
1462     access), and can also be a hassle to implement.
1463    
1464     Newer POSIX systems have a number of functions (openat, fdopendir,
1465     futimensat and so on) that make it possible to specify working directories
1466     per operation.
1467    
1468     For portability, and because the clowns who "designed", or shall I write,
1469     perpetrated this new interface were obviously half-drunk, this abstraction
1470     cannot be perfect, though.
1471    
1472     IO::AIO allows you to convert directory paths into a so-called IO::AIO::WD
1473     object. This object stores the canonicalised, absolute version of the
1474     path, and on systems that allow it, also a directory file descriptor.
1475    
1476     Everywhere where a pathname is accepted by IO::AIO (e.g. in C<aio_stat>
1477     or C<aio_unlink>), one can specify an array reference with an IO::AIO::WD
1478 root 1.214 object and a pathname instead (or the IO::AIO::WD object alone, which
1479     gets interpreted as C<[$wd, "."]>). If the pathname is absolute, the
1480 root 1.213 IO::AIO::WD object is ignored, otherwise the pathname is resolved relative
1481 root 1.209 to that IO::AIO::WD object.
1482    
1483     For example, to get a wd object for F</etc> and then stat F<passwd>
1484     inside, you would write:
1485    
1486     aio_wd "/etc", sub {
1487     my $etcdir = shift;
1488    
1489     # although $etcdir can be undef on error, there is generally no reason
1490     # to check for errors here, as aio_stat will fail with ENOENT
1491     # when $etcdir is undef.
1492    
1493     aio_stat [$etcdir, "passwd"], sub {
1494     # yay
1495     };
1496     };
1497    
1498 root 1.214 That C<aio_wd> is a request and not a normal function shows that creating
1499     an IO::AIO::WD object is itself a potentially blocking operation, which is
1500     why it is done asynchronously.
1501    
1502     To stat the directory obtained with C<aio_wd> above, one could write
1503     either of the following three request calls:
1504    
1505     aio_lstat "/etc" , sub { ... # pathname as normal string
1506     aio_lstat [$wd, "."], sub { ... # "." relative to $wd (i.e. $wd itself)
1507     aio_lstat $wd , sub { ... # shorthand for the previous
1508 root 1.209
1509     As with normal pathnames, IO::AIO keeps a copy of the working directory
1510     object and the pathname string, so you could write the following without
1511     causing any issues due to C<$path> getting reused:
1512    
1513     my $path = [$wd, undef];
1514    
1515     for my $name (qw(abc def ghi)) {
1516     $path->[1] = $name;
1517     aio_stat $path, sub {
1518     # ...
1519     };
1520     }
1521    
1522     There are some caveats: when directories get renamed (or deleted), the
1523     pathname string doesn't change, so will point to the new directory (or
1524     nowhere at all), while the directory fd, if available on the system,
1525     will still point to the original directory. Most functions accepting a
1526     pathname will use the directory fd on newer systems, and the string on
1527     older systems. Some functions (such as realpath) will always rely on the
1528     string form of the pathname.
1529    
1530 root 1.239 So this functionality is mainly useful to get some protection against
1531 root 1.209 C<chdir>, to easily get an absolute path out of a relative path for future
1532     reference, and to speed up doing many operations in the same directory
1533     (e.g. when stat'ing all files in a directory).
1534    
1535     The following functions implement this working directory abstraction:
1536    
1537     =over 4
1538    
1539     =item aio_wd $pathname, $callback->($wd)
1540    
1541     Asynchonously canonicalise the given pathname and convert it to an
1542     IO::AIO::WD object representing it. If possible and supported on the
1543     system, also open a directory fd to speed up pathname resolution relative
1544     to this working directory.
1545    
1546     If something goes wrong, then C<undef> is passwd to the callback instead
1547     of a working directory object and C<$!> is set appropriately. Since
1548     passing C<undef> as working directory component of a pathname fails the
1549     request with C<ENOENT>, there is often no need for error checking in the
1550     C<aio_wd> callback, as future requests using the value will fail in the
1551     expected way.
1552    
1553     If this call isn't available because your OS lacks it or it couldn't be
1554     detected, it will be emulated by calling C<fsync> instead.
1555    
1556     =item IO::AIO::CWD
1557    
1558     This is a compiletime constant (object) that represents the process
1559     current working directory.
1560    
1561 root 1.239 Specifying this object as working directory object for a pathname is as if
1562     the pathname would be specified directly, without a directory object. For
1563     example, these calls are functionally identical:
1564 root 1.209
1565     aio_stat "somefile", sub { ... };
1566     aio_stat [IO::AIO::CWD, "somefile"], sub { ... };
1567    
1568     =back
1569    
1570 root 1.239 To recover the path associated with an IO::AIO::WD object, you can use
1571     C<aio_realpath>:
1572    
1573     aio_realpath $wd, sub {
1574     warn "path is $_[0]\n";
1575     };
1576    
1577 root 1.209
1578 root 1.53 =head2 IO::AIO::REQ CLASS
1579 root 1.52
1580     All non-aggregate C<aio_*> functions return an object of this class when
1581     called in non-void context.
1582    
1583     =over 4
1584    
1585 root 1.65 =item cancel $req
1586 root 1.52
1587     Cancels the request, if possible. Has the effect of skipping execution
1588     when entering the B<execute> state and skipping calling the callback when
1589     entering the the B<result> state, but will leave the request otherwise
1590 root 1.151 untouched (with the exception of readdir). That means that requests that
1591     currently execute will not be stopped and resources held by the request
1592     will not be freed prematurely.
1593 root 1.52
1594 root 1.65 =item cb $req $callback->(...)
1595    
1596     Replace (or simply set) the callback registered to the request.
1597    
1598 root 1.52 =back
1599    
1600 root 1.55 =head2 IO::AIO::GRP CLASS
1601    
1602     This class is a subclass of L<IO::AIO::REQ>, so all its methods apply to
1603     objects of this class, too.
1604    
1605     A IO::AIO::GRP object is a special request that can contain multiple other
1606     aio requests.
1607    
1608     You create one by calling the C<aio_group> constructing function with a
1609     callback that will be called when all contained requests have entered the
1610     C<done> state:
1611    
1612     my $grp = aio_group sub {
1613     print "all requests are done\n";
1614     };
1615    
1616     You add requests by calling the C<add> method with one or more
1617     C<IO::AIO::REQ> objects:
1618    
1619     $grp->add (aio_unlink "...");
1620    
1621 root 1.58 add $grp aio_stat "...", sub {
1622     $_[0] or return $grp->result ("error");
1623    
1624     # add another request dynamically, if first succeeded
1625     add $grp aio_open "...", sub {
1626     $grp->result ("ok");
1627     };
1628     };
1629 root 1.55
1630     This makes it very easy to create composite requests (see the source of
1631     C<aio_move> for an application) that work and feel like simple requests.
1632    
1633 root 1.62 =over 4
1634    
1635     =item * The IO::AIO::GRP objects will be cleaned up during calls to
1636 root 1.55 C<IO::AIO::poll_cb>, just like any other request.
1637    
1638 root 1.62 =item * They can be canceled like any other request. Canceling will cancel not
1639 root 1.59 only the request itself, but also all requests it contains.
1640 root 1.55
1641 root 1.62 =item * They can also can also be added to other IO::AIO::GRP objects.
1642 root 1.55
1643 root 1.62 =item * You must not add requests to a group from within the group callback (or
1644 root 1.60 any later time).
1645    
1646 root 1.62 =back
1647    
1648 root 1.55 Their lifetime, simplified, looks like this: when they are empty, they
1649     will finish very quickly. If they contain only requests that are in the
1650     C<done> state, they will also finish. Otherwise they will continue to
1651     exist.
1652    
1653 root 1.133 That means after creating a group you have some time to add requests
1654     (precisely before the callback has been invoked, which is only done within
1655     the C<poll_cb>). And in the callbacks of those requests, you can add
1656     further requests to the group. And only when all those requests have
1657     finished will the the group itself finish.
1658 root 1.57
1659 root 1.55 =over 4
1660    
1661 root 1.65 =item add $grp ...
1662    
1663 root 1.55 =item $grp->add (...)
1664    
1665 root 1.57 Add one or more requests to the group. Any type of L<IO::AIO::REQ> can
1666     be added, including other groups, as long as you do not create circular
1667     dependencies.
1668    
1669     Returns all its arguments.
1670 root 1.55
1671 root 1.74 =item $grp->cancel_subs
1672    
1673     Cancel all subrequests and clears any feeder, but not the group request
1674     itself. Useful when you queued a lot of events but got a result early.
1675    
1676 root 1.168 The group request will finish normally (you cannot add requests to the
1677     group).
1678    
1679 root 1.58 =item $grp->result (...)
1680    
1681     Set the result value(s) that will be passed to the group callback when all
1682 root 1.120 subrequests have finished and set the groups errno to the current value
1683 root 1.80 of errno (just like calling C<errno> without an error number). By default,
1684     no argument will be passed and errno is zero.
1685    
1686     =item $grp->errno ([$errno])
1687    
1688     Sets the group errno value to C<$errno>, or the current value of errno
1689     when the argument is missing.
1690    
1691     Every aio request has an associated errno value that is restored when
1692     the callback is invoked. This method lets you change this value from its
1693     default (0).
1694    
1695     Calling C<result> will also set errno, so make sure you either set C<$!>
1696     before the call to C<result>, or call c<errno> after it.
1697 root 1.58
1698 root 1.65 =item feed $grp $callback->($grp)
1699 root 1.60
1700     Sets a feeder/generator on this group: every group can have an attached
1701     generator that generates requests if idle. The idea behind this is that,
1702     although you could just queue as many requests as you want in a group,
1703 root 1.139 this might starve other requests for a potentially long time. For example,
1704 root 1.211 C<aio_scandir> might generate hundreds of thousands of C<aio_stat>
1705     requests, delaying any later requests for a long time.
1706 root 1.60
1707     To avoid this, and allow incremental generation of requests, you can
1708     instead a group and set a feeder on it that generates those requests. The
1709 root 1.68 feed callback will be called whenever there are few enough (see C<limit>,
1710 root 1.60 below) requests active in the group itself and is expected to queue more
1711     requests.
1712    
1713 root 1.68 The feed callback can queue as many requests as it likes (i.e. C<add> does
1714     not impose any limits).
1715 root 1.60
1716 root 1.65 If the feed does not queue more requests when called, it will be
1717 root 1.60 automatically removed from the group.
1718    
1719 root 1.138 If the feed limit is C<0> when this method is called, it will be set to
1720     C<2> automatically.
1721 root 1.60
1722     Example:
1723    
1724     # stat all files in @files, but only ever use four aio requests concurrently:
1725    
1726     my $grp = aio_group sub { print "finished\n" };
1727 root 1.68 limit $grp 4;
1728 root 1.65 feed $grp sub {
1729 root 1.60 my $file = pop @files
1730     or return;
1731    
1732     add $grp aio_stat $file, sub { ... };
1733 root 1.65 };
1734 root 1.60
1735 root 1.68 =item limit $grp $num
1736 root 1.60
1737     Sets the feeder limit for the group: The feeder will be called whenever
1738     the group contains less than this many requests.
1739    
1740     Setting the limit to C<0> will pause the feeding process.
1741    
1742 root 1.138 The default value for the limit is C<0>, but note that setting a feeder
1743     automatically bumps it up to C<2>.
1744    
1745 root 1.55 =back
1746    
1747 root 1.5 =head2 SUPPORT FUNCTIONS
1748    
1749 root 1.86 =head3 EVENT PROCESSING AND EVENT LOOP INTEGRATION
1750    
1751 root 1.5 =over 4
1752    
1753     =item $fileno = IO::AIO::poll_fileno
1754    
1755 root 1.20 Return the I<request result pipe file descriptor>. This filehandle must be
1756 root 1.156 polled for reading by some mechanism outside this module (e.g. EV, Glib,
1757     select and so on, see below or the SYNOPSIS). If the pipe becomes readable
1758     you have to call C<poll_cb> to check the results.
1759 root 1.5
1760     See C<poll_cb> for an example.
1761    
1762     =item IO::AIO::poll_cb
1763    
1764 root 1.240 Process some requests that have reached the result phase (i.e. they have
1765     been executed but the results are not yet reported). You have to call
1766     this "regularly" to finish outstanding requests.
1767    
1768     Returns C<0> if all events could be processed (or there were no
1769     events to process), or C<-1> if it returned earlier for whatever
1770     reason. Returns immediately when no events are outstanding. The amount
1771     of events processed depends on the settings of C<IO::AIO::max_poll_req>,
1772     C<IO::AIO::max_poll_time> and C<IO::AIO::max_outstanding>.
1773    
1774     If not all requests were processed for whatever reason, the poll file
1775     descriptor will still be ready when C<poll_cb> returns, so normally you
1776     don't have to do anything special to have it called later.
1777 root 1.78
1778 root 1.192 Apart from calling C<IO::AIO::poll_cb> when the event filehandle becomes
1779     ready, it can be beneficial to call this function from loops which submit
1780     a lot of requests, to make sure the results get processed when they become
1781     available and not just when the loop is finished and the event loop takes
1782     over again. This function returns very fast when there are no outstanding
1783     requests.
1784    
1785 root 1.20 Example: Install an Event watcher that automatically calls
1786 root 1.156 IO::AIO::poll_cb with high priority (more examples can be found in the
1787     SYNOPSIS section, at the top of this document):
1788 root 1.5
1789     Event->io (fd => IO::AIO::poll_fileno,
1790     poll => 'r', async => 1,
1791     cb => \&IO::AIO::poll_cb);
1792    
1793 root 1.175 =item IO::AIO::poll_wait
1794    
1795 root 1.240 Wait until either at least one request is in the result phase or no
1796     requests are outstanding anymore.
1797    
1798     This is useful if you want to synchronously wait for some requests to
1799     become ready, without actually handling them.
1800 root 1.175
1801     See C<nreqs> for an example.
1802    
1803     =item IO::AIO::poll
1804    
1805     Waits until some requests have been handled.
1806    
1807     Returns the number of requests processed, but is otherwise strictly
1808     equivalent to:
1809    
1810     IO::AIO::poll_wait, IO::AIO::poll_cb
1811    
1812     =item IO::AIO::flush
1813    
1814     Wait till all outstanding AIO requests have been handled.
1815    
1816     Strictly equivalent to:
1817    
1818     IO::AIO::poll_wait, IO::AIO::poll_cb
1819     while IO::AIO::nreqs;
1820    
1821 root 1.86 =item IO::AIO::max_poll_reqs $nreqs
1822    
1823     =item IO::AIO::max_poll_time $seconds
1824    
1825     These set the maximum number of requests (default C<0>, meaning infinity)
1826     that are being processed by C<IO::AIO::poll_cb> in one call, respectively
1827     the maximum amount of time (default C<0>, meaning infinity) spent in
1828     C<IO::AIO::poll_cb> to process requests (more correctly the mininum amount
1829     of time C<poll_cb> is allowed to use).
1830 root 1.78
1831 root 1.89 Setting C<max_poll_time> to a non-zero value creates an overhead of one
1832     syscall per request processed, which is not normally a problem unless your
1833     callbacks are really really fast or your OS is really really slow (I am
1834     not mentioning Solaris here). Using C<max_poll_reqs> incurs no overhead.
1835    
1836 root 1.86 Setting these is useful if you want to ensure some level of
1837     interactiveness when perl is not fast enough to process all requests in
1838     time.
1839 root 1.78
1840 root 1.86 For interactive programs, values such as C<0.01> to C<0.1> should be fine.
1841 root 1.78
1842     Example: Install an Event watcher that automatically calls
1843 root 1.89 IO::AIO::poll_cb with low priority, to ensure that other parts of the
1844 root 1.78 program get the CPU sometimes even under high AIO load.
1845    
1846 root 1.86 # try not to spend much more than 0.1s in poll_cb
1847     IO::AIO::max_poll_time 0.1;
1848    
1849     # use a low priority so other tasks have priority
1850 root 1.78 Event->io (fd => IO::AIO::poll_fileno,
1851     poll => 'r', nice => 1,
1852 root 1.86 cb => &IO::AIO::poll_cb);
1853 root 1.78
1854 root 1.104 =back
1855    
1856 root 1.86 =head3 CONTROLLING THE NUMBER OF THREADS
1857 root 1.13
1858 root 1.105 =over
1859    
1860 root 1.5 =item IO::AIO::min_parallel $nthreads
1861    
1862 root 1.61 Set the minimum number of AIO threads to C<$nthreads>. The current
1863     default is C<8>, which means eight asynchronous operations can execute
1864     concurrently at any one time (the number of outstanding requests,
1865     however, is unlimited).
1866 root 1.5
1867 root 1.34 IO::AIO starts threads only on demand, when an AIO request is queued and
1868 root 1.86 no free thread exists. Please note that queueing up a hundred requests can
1869     create demand for a hundred threads, even if it turns out that everything
1870     is in the cache and could have been processed faster by a single thread.
1871 root 1.34
1872 root 1.61 It is recommended to keep the number of threads relatively low, as some
1873     Linux kernel versions will scale negatively with the number of threads
1874     (higher parallelity => MUCH higher latency). With current Linux 2.6
1875     versions, 4-32 threads should be fine.
1876 root 1.5
1877 root 1.34 Under most circumstances you don't need to call this function, as the
1878     module selects a default that is suitable for low to moderate load.
1879 root 1.5
1880     =item IO::AIO::max_parallel $nthreads
1881    
1882 root 1.34 Sets the maximum number of AIO threads to C<$nthreads>. If more than the
1883     specified number of threads are currently running, this function kills
1884     them. This function blocks until the limit is reached.
1885    
1886     While C<$nthreads> are zero, aio requests get queued but not executed
1887     until the number of threads has been increased again.
1888 root 1.5
1889     This module automatically runs C<max_parallel 0> at program end, to ensure
1890     that all threads are killed and that there are no outstanding requests.
1891    
1892     Under normal circumstances you don't need to call this function.
1893    
1894 root 1.86 =item IO::AIO::max_idle $nthreads
1895    
1896 root 1.188 Limit the number of threads (default: 4) that are allowed to idle
1897     (i.e., threads that did not get a request to process within the idle
1898     timeout (default: 10 seconds). That means if a thread becomes idle while
1899     C<$nthreads> other threads are also idle, it will free its resources and
1900     exit.
1901 root 1.86
1902     This is useful when you allow a large number of threads (e.g. 100 or 1000)
1903     to allow for extremely high load situations, but want to free resources
1904     under normal circumstances (1000 threads can easily consume 30MB of RAM).
1905    
1906     The default is probably ok in most situations, especially if thread
1907     creation is fast. If thread creation is very slow on your system you might
1908     want to use larger values.
1909    
1910 root 1.188 =item IO::AIO::idle_timeout $seconds
1911    
1912     Sets the minimum idle timeout (default 10) after which worker threads are
1913     allowed to exit. SEe C<IO::AIO::max_idle>.
1914    
1915 root 1.123 =item IO::AIO::max_outstanding $maxreqs
1916 root 1.5
1917 root 1.195 Sets the maximum number of outstanding requests to C<$nreqs>. If
1918     you do queue up more than this number of requests, the next call to
1919     C<IO::AIO::poll_cb> (and other functions calling C<poll_cb>, such as
1920     C<IO::AIO::flush> or C<IO::AIO::poll>) will block until the limit is no
1921     longer exceeded.
1922    
1923     In other words, this setting does not enforce a queue limit, but can be
1924     used to make poll functions block if the limit is exceeded.
1925    
1926 root 1.79 This is a very bad function to use in interactive programs because it
1927     blocks, and a bad way to reduce concurrency because it is inexact: Better
1928     use an C<aio_group> together with a feed callback.
1929    
1930 root 1.195 It's main use is in scripts without an event loop - when you want to stat
1931     a lot of files, you can write somehting like this:
1932    
1933     IO::AIO::max_outstanding 32;
1934    
1935     for my $path (...) {
1936     aio_stat $path , ...;
1937     IO::AIO::poll_cb;
1938     }
1939    
1940     IO::AIO::flush;
1941    
1942     The call to C<poll_cb> inside the loop will normally return instantly, but
1943     as soon as more thna C<32> reqeusts are in-flight, it will block until
1944     some requests have been handled. This keeps the loop from pushing a large
1945     number of C<aio_stat> requests onto the queue.
1946    
1947     The default value for C<max_outstanding> is very large, so there is no
1948     practical limit on the number of outstanding requests.
1949 root 1.5
1950 root 1.104 =back
1951    
1952 root 1.86 =head3 STATISTICAL INFORMATION
1953    
1954 root 1.104 =over
1955    
1956 root 1.86 =item IO::AIO::nreqs
1957    
1958     Returns the number of requests currently in the ready, execute or pending
1959     states (i.e. for which their callback has not been invoked yet).
1960    
1961     Example: wait till there are no outstanding requests anymore:
1962    
1963     IO::AIO::poll_wait, IO::AIO::poll_cb
1964     while IO::AIO::nreqs;
1965    
1966     =item IO::AIO::nready
1967    
1968     Returns the number of requests currently in the ready state (not yet
1969     executed).
1970    
1971     =item IO::AIO::npending
1972    
1973     Returns the number of requests currently in the pending state (executed,
1974     but not yet processed by poll_cb).
1975    
1976 root 1.5 =back
1977    
1978 root 1.157 =head3 MISCELLANEOUS FUNCTIONS
1979    
1980     IO::AIO implements some functions that might be useful, but are not
1981     asynchronous.
1982    
1983     =over 4
1984    
1985     =item IO::AIO::sendfile $ofh, $ifh, $offset, $count
1986    
1987     Calls the C<eio_sendfile_sync> function, which is like C<aio_sendfile>,
1988     but is blocking (this makes most sense if you know the input data is
1989     likely cached already and the output filehandle is set to non-blocking
1990     operations).
1991    
1992     Returns the number of bytes copied, or C<-1> on error.
1993    
1994     =item IO::AIO::fadvise $fh, $offset, $len, $advice
1995    
1996 root 1.184 Simply calls the C<posix_fadvise> function (see its
1997 root 1.157 manpage for details). The following advice constants are
1998 root 1.207 available: C<IO::AIO::FADV_NORMAL>, C<IO::AIO::FADV_SEQUENTIAL>,
1999 root 1.157 C<IO::AIO::FADV_RANDOM>, C<IO::AIO::FADV_NOREUSE>,
2000     C<IO::AIO::FADV_WILLNEED>, C<IO::AIO::FADV_DONTNEED>.
2001    
2002     On systems that do not implement C<posix_fadvise>, this function returns
2003     ENOSYS, otherwise the return value of C<posix_fadvise>.
2004    
2005 root 1.184 =item IO::AIO::madvise $scalar, $offset, $len, $advice
2006    
2007     Simply calls the C<posix_madvise> function (see its
2008     manpage for details). The following advice constants are
2009 root 1.207 available: C<IO::AIO::MADV_NORMAL>, C<IO::AIO::MADV_SEQUENTIAL>,
2010 root 1.184 C<IO::AIO::MADV_RANDOM>, C<IO::AIO::MADV_WILLNEED>, C<IO::AIO::MADV_DONTNEED>.
2011    
2012     On systems that do not implement C<posix_madvise>, this function returns
2013     ENOSYS, otherwise the return value of C<posix_madvise>.
2014    
2015     =item IO::AIO::mprotect $scalar, $offset, $len, $protect
2016    
2017     Simply calls the C<mprotect> function on the preferably AIO::mmap'ed
2018     $scalar (see its manpage for details). The following protect
2019 root 1.207 constants are available: C<IO::AIO::PROT_NONE>, C<IO::AIO::PROT_READ>,
2020 root 1.184 C<IO::AIO::PROT_WRITE>, C<IO::AIO::PROT_EXEC>.
2021    
2022     On systems that do not implement C<mprotect>, this function returns
2023     ENOSYS, otherwise the return value of C<mprotect>.
2024    
2025 root 1.176 =item IO::AIO::mmap $scalar, $length, $prot, $flags, $fh[, $offset]
2026    
2027     Memory-maps a file (or anonymous memory range) and attaches it to the
2028 root 1.228 given C<$scalar>, which will act like a string scalar. Returns true on
2029     success, and false otherwise.
2030 root 1.176
2031     The only operations allowed on the scalar are C<substr>/C<vec> that don't
2032     change the string length, and most read-only operations such as copying it
2033     or searching it with regexes and so on.
2034    
2035     Anything else is unsafe and will, at best, result in memory leaks.
2036    
2037     The memory map associated with the C<$scalar> is automatically removed
2038     when the C<$scalar> is destroyed, or when the C<IO::AIO::mmap> or
2039     C<IO::AIO::munmap> functions are called.
2040    
2041     This calls the C<mmap>(2) function internally. See your system's manual
2042     page for details on the C<$length>, C<$prot> and C<$flags> parameters.
2043    
2044     The C<$length> must be larger than zero and smaller than the actual
2045     filesize.
2046    
2047     C<$prot> is a combination of C<IO::AIO::PROT_NONE>, C<IO::AIO::PROT_EXEC>,
2048     C<IO::AIO::PROT_READ> and/or C<IO::AIO::PROT_WRITE>,
2049    
2050     C<$flags> can be a combination of C<IO::AIO::MAP_SHARED> or
2051     C<IO::AIO::MAP_PRIVATE>, or a number of system-specific flags (when
2052     not available, the are defined as 0): C<IO::AIO::MAP_ANONYMOUS>
2053     (which is set to C<MAP_ANON> if your system only provides this
2054     constant), C<IO::AIO::MAP_HUGETLB>, C<IO::AIO::MAP_LOCKED>,
2055     C<IO::AIO::MAP_NORESERVE>, C<IO::AIO::MAP_POPULATE> or
2056     C<IO::AIO::MAP_NONBLOCK>
2057    
2058     If C<$fh> is C<undef>, then a file descriptor of C<-1> is passed.
2059    
2060 root 1.179 C<$offset> is the offset from the start of the file - it generally must be
2061     a multiple of C<IO::AIO::PAGESIZE> and defaults to C<0>.
2062    
2063 root 1.177 Example:
2064    
2065     use Digest::MD5;
2066     use IO::AIO;
2067    
2068     open my $fh, "<verybigfile"
2069     or die "$!";
2070    
2071     IO::AIO::mmap my $data, -s $fh, IO::AIO::PROT_READ, IO::AIO::MAP_SHARED, $fh
2072     or die "verybigfile: $!";
2073    
2074     my $fast_md5 = md5 $data;
2075    
2076 root 1.176 =item IO::AIO::munmap $scalar
2077    
2078     Removes a previous mmap and undefines the C<$scalar>.
2079    
2080 root 1.182 =item IO::AIO::munlock $scalar, $offset = 0, $length = undef
2081 root 1.174
2082 root 1.182 Calls the C<munlock> function, undoing the effects of a previous
2083     C<aio_mlock> call (see its description for details).
2084 root 1.174
2085     =item IO::AIO::munlockall
2086    
2087     Calls the C<munlockall> function.
2088    
2089     On systems that do not implement C<munlockall>, this function returns
2090     ENOSYS, otherwise the return value of C<munlockall>.
2091    
2092 root 1.225 =item IO::AIO::splice $r_fh, $r_off, $w_fh, $w_off, $length, $flags
2093    
2094     Calls the GNU/Linux C<splice(2)> syscall, if available. If C<$r_off> or
2095     C<$w_off> are C<undef>, then C<NULL> is passed for these, otherwise they
2096     should be the file offset.
2097    
2098 root 1.227 C<$r_fh> and C<$w_fh> should not refer to the same file, as splice might
2099     silently corrupt the data in this case.
2100    
2101 root 1.225 The following symbol flag values are available: C<IO::AIO::SPLICE_F_MOVE>,
2102     C<IO::AIO::SPLICE_F_NONBLOCK>, C<IO::AIO::SPLICE_F_MORE> and
2103     C<IO::AIO::SPLICE_F_GIFT>.
2104    
2105     See the C<splice(2)> manpage for details.
2106    
2107     =item IO::AIO::tee $r_fh, $w_fh, $length, $flags
2108    
2109     Calls the GNU/Linux C<tee(2)> syscall, see it's manpage and the
2110     description for C<IO::AIO::splice> above for details.
2111    
2112 root 1.157 =back
2113    
2114 root 1.1 =cut
2115    
2116 root 1.61 min_parallel 8;
2117 root 1.1
2118 root 1.95 END { flush }
2119 root 1.82
2120 root 1.1 1;
2121    
2122 root 1.175 =head1 EVENT LOOP INTEGRATION
2123    
2124     It is recommended to use L<AnyEvent::AIO> to integrate IO::AIO
2125     automatically into many event loops:
2126    
2127     # AnyEvent integration (EV, Event, Glib, Tk, POE, urxvt, pureperl...)
2128     use AnyEvent::AIO;
2129    
2130     You can also integrate IO::AIO manually into many event loops, here are
2131     some examples of how to do this:
2132    
2133     # EV integration
2134     my $aio_w = EV::io IO::AIO::poll_fileno, EV::READ, \&IO::AIO::poll_cb;
2135    
2136     # Event integration
2137     Event->io (fd => IO::AIO::poll_fileno,
2138     poll => 'r',
2139     cb => \&IO::AIO::poll_cb);
2140    
2141     # Glib/Gtk2 integration
2142     add_watch Glib::IO IO::AIO::poll_fileno,
2143     in => sub { IO::AIO::poll_cb; 1 };
2144    
2145     # Tk integration
2146     Tk::Event::IO->fileevent (IO::AIO::poll_fileno, "",
2147     readable => \&IO::AIO::poll_cb);
2148    
2149     # Danga::Socket integration
2150     Danga::Socket->AddOtherFds (IO::AIO::poll_fileno =>
2151     \&IO::AIO::poll_cb);
2152    
2153 root 1.27 =head2 FORK BEHAVIOUR
2154    
2155 root 1.197 Usage of pthreads in a program changes the semantics of fork
2156     considerably. Specifically, only async-safe functions can be called after
2157     fork. Perl doesn't know about this, so in general, you cannot call fork
2158 root 1.204 with defined behaviour in perl if pthreads are involved. IO::AIO uses
2159     pthreads, so this applies, but many other extensions and (for inexplicable
2160     reasons) perl itself often is linked against pthreads, so this limitation
2161     applies to quite a lot of perls.
2162    
2163     This module no longer tries to fight your OS, or POSIX. That means IO::AIO
2164     only works in the process that loaded it. Forking is fully supported, but
2165     using IO::AIO in the child is not.
2166    
2167     You might get around by not I<using> IO::AIO before (or after)
2168     forking. You could also try to call the L<IO::AIO::reinit> function in the
2169     child:
2170    
2171     =over 4
2172    
2173     =item IO::AIO::reinit
2174    
2175 root 1.207 Abandons all current requests and I/O threads and simply reinitialises all
2176     data structures. This is not an operation supported by any standards, but
2177 root 1.204 happens to work on GNU/Linux and some newer BSD systems.
2178    
2179     The only reasonable use for this function is to call it after forking, if
2180     C<IO::AIO> was used in the parent. Calling it while IO::AIO is active in
2181     the process will result in undefined behaviour. Calling it at any time
2182     will also result in any undefined (by POSIX) behaviour.
2183    
2184     =back
2185 root 1.52
2186 root 1.60 =head2 MEMORY USAGE
2187    
2188 root 1.72 Per-request usage:
2189    
2190     Each aio request uses - depending on your architecture - around 100-200
2191     bytes of memory. In addition, stat requests need a stat buffer (possibly
2192     a few hundred bytes), readdir requires a result buffer and so on. Perl
2193     scalars and other data passed into aio requests will also be locked and
2194     will consume memory till the request has entered the done state.
2195 root 1.60
2196 root 1.111 This is not awfully much, so queuing lots of requests is not usually a
2197 root 1.60 problem.
2198    
2199 root 1.72 Per-thread usage:
2200    
2201     In the execution phase, some aio requests require more memory for
2202     temporary buffers, and each thread requires a stack and other data
2203     structures (usually around 16k-128k, depending on the OS).
2204    
2205     =head1 KNOWN BUGS
2206    
2207 root 1.73 Known bugs will be fixed in the next release.
2208 root 1.60
2209 root 1.1 =head1 SEE ALSO
2210    
2211 root 1.125 L<AnyEvent::AIO> for easy integration into event loops, L<Coro::AIO> for a
2212     more natural syntax.
2213 root 1.1
2214     =head1 AUTHOR
2215    
2216     Marc Lehmann <schmorp@schmorp.de>
2217     http://home.schmorp.de/
2218    
2219     =cut
2220