ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/IO-AIO/README
(Generate patch)

Comparing IO-AIO/README (file contents):
Revision 1.17 by root, Mon Jun 26 14:53:55 2006 UTC vs.
Revision 1.36 by root, Sun Jun 7 18:31:18 2009 UTC

… …
3 3
4SYNOPSIS 4SYNOPSIS
5 use IO::AIO; 5 use IO::AIO;
6 6
7 aio_open "/etc/passwd", O_RDONLY, 0, sub { 7 aio_open "/etc/passwd", O_RDONLY, 0, sub {
8 my ($fh) = @_; 8 my $fh = shift
9 or die "/etc/passwd: $!";
9 ... 10 ...
10 }; 11 };
11 12
12 aio_unlink "/tmp/file", sub { }; 13 aio_unlink "/tmp/file", sub { };
13 14
14 aio_read $fh, 30000, 1024, $buffer, 0, sub { 15 aio_read $fh, 30000, 1024, $buffer, 0, sub {
15 $_[0] > 0 or die "read error: $!"; 16 $_[0] > 0 or die "read error: $!";
16 }; 17 };
17 18
18 # AnyEvent 19 # version 2+ has request and group objects
19 open my $fh, "<&=" . IO::AIO::poll_fileno or die "$!"; 20 use IO::AIO 2;
20 my $w = AnyEvent->io (fh => $fh, poll => 'r', cb => sub { IO::AIO::poll_cb });
21 21
22 # Event 22 aioreq_pri 4; # give next request a very high priority
23 my $req = aio_unlink "/tmp/file", sub { };
24 $req->cancel; # cancel request if still in queue
25
26 my $grp = aio_group sub { print "all stats done\n" };
27 add $grp aio_stat "..." for ...;
28
29 # AnyEvent integration (EV, Event, Glib, Tk, POE, urxvt, pureperl...)
30 use AnyEvent::AIO;
31
32 # EV integration
33 my $w = EV::io IO::AIO::poll_fileno, EV::READ, \&IO::AIO::poll_cb;
34
35 # Event integration
23 Event->io (fd => IO::AIO::poll_fileno, 36 Event->io (fd => IO::AIO::poll_fileno,
24 poll => 'r', 37 poll => 'r',
25 cb => \&IO::AIO::poll_cb); 38 cb => \&IO::AIO::poll_cb);
26 39
27 # Glib/Gtk2 40 # Glib/Gtk2 integration
28 add_watch Glib::IO IO::AIO::poll_fileno, 41 add_watch Glib::IO IO::AIO::poll_fileno,
29 in => sub { IO::AIO::poll_cb; 1 }; 42 in => sub { IO::AIO::poll_cb; 1 };
30 43
31 # Tk 44 # Tk integration
32 Tk::Event::IO->fileevent (IO::AIO::poll_fileno, "", 45 Tk::Event::IO->fileevent (IO::AIO::poll_fileno, "",
33 readable => \&IO::AIO::poll_cb); 46 readable => \&IO::AIO::poll_cb);
34 47
35 # Danga::Socket 48 # Danga::Socket integration
36 Danga::Socket->AddOtherFds (IO::AIO::poll_fileno => 49 Danga::Socket->AddOtherFds (IO::AIO::poll_fileno =>
37 \&IO::AIO::poll_cb); 50 \&IO::AIO::poll_cb);
38 51
39DESCRIPTION 52DESCRIPTION
40 This module implements asynchronous I/O using whatever means your 53 This module implements asynchronous I/O using whatever means your
41 operating system supports. 54 operating system supports.
42 55
56 Asynchronous means that operations that can normally block your program
57 (e.g. reading from disk) will be done asynchronously: the operation will
58 still block, but you can do something else in the meantime. This is
59 extremely useful for programs that need to stay interactive even when
60 doing heavy I/O (GUI programs, high performance network servers etc.),
61 but can also be used to easily do operations in parallel that are
62 normally done sequentially, e.g. stat'ing many files, which is much
63 faster on a RAID volume or over NFS when you do a number of stat
64 operations concurrently.
65
66 While most of this works on all types of file descriptors (for example
67 sockets), using these functions on file descriptors that support
68 nonblocking operation (again, sockets, pipes etc.) is very inefficient.
69 Use an event loop for that (such as the Event module): IO::AIO will
70 naturally fit into such an event loop itself.
71
43 Currently, a number of threads are started that execute your read/writes 72 In this version, a number of threads are started that execute your
44 and signal their completion. You don't need thread support in your libc 73 requests and signal their completion. You don't need thread support in
45 or perl, and the threads created by this module will not be visible to 74 perl, and the threads created by this module will not be visible to
46 the pthreads library. In the future, this module might make use of the 75 perl. In the future, this module might make use of the native aio
47 native aio functions available on many operating systems. However, they 76 functions available on many operating systems. However, they are often
48 are often not well-supported (Linux doesn't allow them on normal files 77 not well-supported or restricted (GNU/Linux doesn't allow them on normal
49 currently, for example), and they would only support aio_read and 78 files currently, for example), and they would only support aio_read and
50 aio_write, so the remaining functionality would have to be implemented 79 aio_write, so the remaining functionality would have to be implemented
51 using threads anyway. 80 using threads anyway.
52 81
53 Although the module will work with in the presence of other threads, it 82 Although the module will work in the presence of other (Perl-) threads,
54 is currently not reentrant, so use appropriate locking yourself, always 83 it is currently not reentrant in any way, so use appropriate locking
55 call "poll_cb" from within the same thread, or never call "poll_cb" (or 84 yourself, always call "poll_cb" from within the same thread, or never
56 other "aio_" functions) recursively. 85 call "poll_cb" (or other "aio_" functions) recursively.
86
87 EXAMPLE
88 This is a simple example that uses the Event module and loads
89 /etc/passwd asynchronously:
90
91 use Fcntl;
92 use Event;
93 use IO::AIO;
94
95 # register the IO::AIO callback with Event
96 Event->io (fd => IO::AIO::poll_fileno,
97 poll => 'r',
98 cb => \&IO::AIO::poll_cb);
99
100 # queue the request to open /etc/passwd
101 aio_open "/etc/passwd", O_RDONLY, 0, sub {
102 my $fh = shift
103 or die "error while opening: $!";
104
105 # stat'ing filehandles is generally non-blocking
106 my $size = -s $fh;
107
108 # queue a request to read the file
109 my $contents;
110 aio_read $fh, 0, $size, $contents, 0, sub {
111 $_[0] == $size
112 or die "short read: $!";
113
114 close $fh;
115
116 # file contents now in $contents
117 print $contents;
118
119 # exit event loop and program
120 Event::unloop;
121 };
122 };
123
124 # possibly queue up other requests, or open GUI windows,
125 # check for sockets etc. etc.
126
127 # process events as long as there are some:
128 Event::loop;
129
130REQUEST ANATOMY AND LIFETIME
131 Every "aio_*" function creates a request. which is a C data structure
132 not directly visible to Perl.
133
134 If called in non-void context, every request function returns a Perl
135 object representing the request. In void context, nothing is returned,
136 which saves a bit of memory.
137
138 The perl object is a fairly standard ref-to-hash object. The hash
139 contents are not used by IO::AIO so you are free to store anything you
140 like in it.
141
142 During their existance, aio requests travel through the following
143 states, in order:
144
145 ready
146 Immediately after a request is created it is put into the ready
147 state, waiting for a thread to execute it.
148
149 execute
150 A thread has accepted the request for processing and is currently
151 executing it (e.g. blocking in read).
152
153 pending
154 The request has been executed and is waiting for result processing.
155
156 While request submission and execution is fully asynchronous, result
157 processing is not and relies on the perl interpreter calling
158 "poll_cb" (or another function with the same effect).
159
160 result
161 The request results are processed synchronously by "poll_cb".
162
163 The "poll_cb" function will process all outstanding aio requests by
164 calling their callbacks, freeing memory associated with them and
165 managing any groups they are contained in.
166
167 done
168 Request has reached the end of its lifetime and holds no resources
169 anymore (except possibly for the Perl object, but its connection to
170 the actual aio request is severed and calling its methods will
171 either do nothing or result in a runtime error).
57 172
58FUNCTIONS 173FUNCTIONS
59 AIO FUNCTIONS 174 AIO REQUEST FUNCTIONS
60 All the "aio_*" calls are more or less thin wrappers around the syscall 175 All the "aio_*" calls are more or less thin wrappers around the syscall
61 with the same name (sans "aio_"). The arguments are similar or 176 with the same name (sans "aio_"). The arguments are similar or
62 identical, and they all accept an additional (and optional) $callback 177 identical, and they all accept an additional (and optional) $callback
63 argument which must be a code reference. This code reference will get 178 argument which must be a code reference. This code reference will get
64 called with the syscall return code (e.g. most syscalls return -1 on 179 called with the syscall return code (e.g. most syscalls return -1 on
65 error, unlike perl, which usually delivers "false") as it's sole 180 error, unlike perl, which usually delivers "false") as its sole argument
66 argument when the given syscall has been executed asynchronously. 181 after the given syscall has been executed asynchronously.
67 182
68 All functions expecting a filehandle keep a copy of the filehandle 183 All functions expecting a filehandle keep a copy of the filehandle
69 internally until the request has finished. 184 internally until the request has finished.
70 185
186 All functions return request objects of type IO::AIO::REQ that allow
187 further manipulation of those requests while they are in-flight.
188
71 The pathnames you pass to these routines *must* be absolute and encoded 189 The pathnames you pass to these routines *must* be absolute and encoded
72 in byte form. The reason for the former is that at the time the request 190 as octets. The reason for the former is that at the time the request is
73 is being executed, the current working directory could have changed. 191 being executed, the current working directory could have changed.
74 Alternatively, you can make sure that you never change the current 192 Alternatively, you can make sure that you never change the current
75 working directory. 193 working directory anywhere in the program and then use relative paths.
76 194
77 To encode pathnames to byte form, either make sure you either: a) always 195 To encode pathnames as octets, either make sure you either: a) always
78 pass in filenames you got from outside (command line, readdir etc.), b) 196 pass in filenames you got from outside (command line, readdir etc.)
79 are ASCII or ISO 8859-1, c) use the Encode module and encode your 197 without tinkering, b) are ASCII or ISO 8859-1, c) use the Encode module
80 pathnames to the locale (or other) encoding in effect in the user 198 and encode your pathnames to the locale (or other) encoding in effect in
81 environment, d) use Glib::filename_from_unicode on unicode filenames or 199 the user environment, d) use Glib::filename_from_unicode on unicode
82 e) use something else. 200 filenames or e) use something else to ensure your scalar has the correct
201 contents.
202
203 This works, btw. independent of the internal UTF-8 bit, which IO::AIO
204 handles correctly whether it is set or not.
205
206 $prev_pri = aioreq_pri [$pri]
207 Returns the priority value that would be used for the next request
208 and, if $pri is given, sets the priority for the next aio request.
209
210 The default priority is 0, the minimum and maximum priorities are -4
211 and 4, respectively. Requests with higher priority will be serviced
212 first.
213
214 The priority will be reset to 0 after each call to one of the
215 "aio_*" functions.
216
217 Example: open a file with low priority, then read something from it
218 with higher priority so the read request is serviced before other
219 low priority open requests (potentially spamming the cache):
220
221 aioreq_pri -3;
222 aio_open ..., sub {
223 return unless $_[0];
224
225 aioreq_pri -2;
226 aio_read $_[0], ..., sub {
227 ...
228 };
229 };
230
231 aioreq_nice $pri_adjust
232 Similar to "aioreq_pri", but subtracts the given value from the
233 current priority, so the effect is cumulative.
83 234
84 aio_open $pathname, $flags, $mode, $callback->($fh) 235 aio_open $pathname, $flags, $mode, $callback->($fh)
85 Asynchronously open or create a file and call the callback with a 236 Asynchronously open or create a file and call the callback with a
86 newly created filehandle for the file. 237 newly created filehandle for the file.
87 238
… …
92 They are the same as used by "sysopen". 243 They are the same as used by "sysopen".
93 244
94 Likewise, $mode specifies the mode of the newly created file, if it 245 Likewise, $mode specifies the mode of the newly created file, if it
95 didn't exist and "O_CREAT" has been given, just like perl's 246 didn't exist and "O_CREAT" has been given, just like perl's
96 "sysopen", except that it is mandatory (i.e. use 0 if you don't 247 "sysopen", except that it is mandatory (i.e. use 0 if you don't
97 create new files, and 0666 or 0777 if you do). 248 create new files, and 0666 or 0777 if you do). Note that the $mode
249 will be modified by the umask in effect then the request is being
250 executed, so better never change the umask.
98 251
99 Example: 252 Example:
100 253
101 aio_open "/etc/passwd", O_RDONLY, 0, sub { 254 aio_open "/etc/passwd", O_RDONLY, 0, sub {
102 if ($_[0]) { 255 if ($_[0]) {
… …
107 } 260 }
108 }; 261 };
109 262
110 aio_close $fh, $callback->($status) 263 aio_close $fh, $callback->($status)
111 Asynchronously close a file and call the callback with the result 264 Asynchronously close a file and call the callback with the result
112 code. *WARNING:* although accepted, you should not pass in a perl 265 code.
113 filehandle here, as perl will likely close the file descriptor
114 another time when the filehandle is destroyed. Normally, you can
115 safely call perls "close" or just let filehandles go out of scope.
116 266
117 This is supposed to be a bug in the API, so that might change. It's 267 Unfortunately, you can't do this to perl. Perl *insists* very
118 therefore best to avoid this function. 268 strongly on closing the file descriptor associated with the
269 filehandle itself.
270
271 Therefore, "aio_close" will not close the filehandle - instead it
272 will use dup2 to overwrite the file descriptor with the write-end of
273 a pipe (the pipe fd will be created on demand and will be cached).
274
275 Or in other words: the file descriptor will be closed, but it will
276 not be free for reuse until the perl filehandle is closed.
119 277
120 aio_read $fh,$offset,$length, $data,$dataoffset, $callback->($retval) 278 aio_read $fh,$offset,$length, $data,$dataoffset, $callback->($retval)
121 aio_write $fh,$offset,$length, $data,$dataoffset, $callback->($retval) 279 aio_write $fh,$offset,$length, $data,$dataoffset, $callback->($retval)
122 Reads or writes "length" bytes from the specified "fh" and "offset" 280 Reads or writes $length bytes from or to the specified $fh and
123 into the scalar given by "data" and offset "dataoffset" and calls 281 $offset into the scalar given by $data and offset $dataoffset and
124 the callback without the actual number of bytes read (or -1 on 282 calls the callback without the actual number of bytes read (or -1 on
125 error, just like the syscall). 283 error, just like the syscall).
126 284
285 "aio_read" will, like "sysread", shrink or grow the $data scalar to
286 offset plus the actual number of bytes read.
287
288 If $offset is undefined, then the current file descriptor offset
289 will be used (and updated), otherwise the file descriptor offset
290 will not be changed by these calls.
291
292 If $length is undefined in "aio_write", use the remaining length of
293 $data.
294
295 If $dataoffset is less than zero, it will be counted from the end of
296 $data.
297
127 The $data scalar *MUST NOT* be modified in any way while the request 298 The $data scalar *MUST NOT* be modified in any way while the request
128 is outstanding. Modifying it can result in segfaults or WW3 (if the 299 is outstanding. Modifying it can result in segfaults or World War
129 necessary/optional hardware is installed). 300 III (if the necessary/optional hardware is installed).
130 301
131 Example: Read 15 bytes at offset 7 into scalar $buffer, starting at 302 Example: Read 15 bytes at offset 7 into scalar $buffer, starting at
132 offset 0 within the scalar: 303 offset 0 within the scalar:
133 304
134 aio_read $fh, 7, 15, $buffer, 0, sub { 305 aio_read $fh, 7, 15, $buffer, 0, sub {
135 $_[0] > 0 or die "read error: $!"; 306 $_[0] > 0 or die "read error: $!";
136 print "read $_[0] bytes: <$buffer>\n"; 307 print "read $_[0] bytes: <$buffer>\n";
137 }; 308 };
138
139 aio_move $srcpath, $dstpath, $callback->($status)
140 [EXPERIMENTAL]
141
142 Try to move the *file* (directories not supported as either source
143 or destination) from $srcpath to $dstpath and call the callback with
144 the 0 (error) or -1 ok.
145
146 This is a composite request that tries to rename(2) the file first.
147 If rename files with "EXDEV", it creates the destination file with
148 mode 0200 and copies the contents of the source file into it using
149 "aio_sendfile", followed by restoring atime, mtime, access mode and
150 uid/gid, in that order, and unlinking the $srcpath.
151
152 If an error occurs, the partial destination file will be unlinked,
153 if possible, except when setting atime, mtime, access mode and
154 uid/gid, where errors are being ignored.
155 309
156 aio_sendfile $out_fh, $in_fh, $in_offset, $length, $callback->($retval) 310 aio_sendfile $out_fh, $in_fh, $in_offset, $length, $callback->($retval)
157 Tries to copy $length bytes from $in_fh to $out_fh. It starts 311 Tries to copy $length bytes from $in_fh to $out_fh. It starts
158 reading at byte offset $in_offset, and starts writing at the current 312 reading at byte offset $in_offset, and starts writing at the current
159 file offset of $out_fh. Because of that, it is not safe to issue 313 file offset of $out_fh. Because of that, it is not safe to issue
… …
209 aio_stat "/etc/passwd", sub { 363 aio_stat "/etc/passwd", sub {
210 $_[0] and die "stat failed: $!"; 364 $_[0] and die "stat failed: $!";
211 print "size is ", -s _, "\n"; 365 print "size is ", -s _, "\n";
212 }; 366 };
213 367
368 aio_utime $fh_or_path, $atime, $mtime, $callback->($status)
369 Works like perl's "utime" function (including the special case of
370 $atime and $mtime being undef). Fractional times are supported if
371 the underlying syscalls support them.
372
373 When called with a pathname, uses utimes(2) if available, otherwise
374 utime(2). If called on a file descriptor, uses futimes(2) if
375 available, otherwise returns ENOSYS, so this is not portable.
376
377 Examples:
378
379 # set atime and mtime to current time (basically touch(1)):
380 aio_utime "path", undef, undef;
381 # set atime to current time and mtime to beginning of the epoch:
382 aio_utime "path", time, undef; # undef==0
383
384 aio_chown $fh_or_path, $uid, $gid, $callback->($status)
385 Works like perl's "chown" function, except that "undef" for either
386 $uid or $gid is being interpreted as "do not change" (but -1 can
387 also be used).
388
389 Examples:
390
391 # same as "chown root path" in the shell:
392 aio_chown "path", 0, -1;
393 # same as above:
394 aio_chown "path", 0, undef;
395
396 aio_truncate $fh_or_path, $offset, $callback->($status)
397 Works like truncate(2) or ftruncate(2).
398
399 aio_chmod $fh_or_path, $mode, $callback->($status)
400 Works like perl's "chmod" function.
401
214 aio_unlink $pathname, $callback->($status) 402 aio_unlink $pathname, $callback->($status)
215 Asynchronously unlink (delete) a file and call the callback with the 403 Asynchronously unlink (delete) a file and call the callback with the
216 result code. 404 result code.
405
406 aio_mknod $path, $mode, $dev, $callback->($status)
407 [EXPERIMENTAL]
408
409 Asynchronously create a device node (or fifo). See mknod(2).
410
411 The only (POSIX-) portable way of calling this function is:
412
413 aio_mknod $path, IO::AIO::S_IFIFO | $mode, 0, sub { ...
217 414
218 aio_link $srcpath, $dstpath, $callback->($status) 415 aio_link $srcpath, $dstpath, $callback->($status)
219 Asynchronously create a new link to the existing object at $srcpath 416 Asynchronously create a new link to the existing object at $srcpath
220 at the path $dstpath and call the callback with the result code. 417 at the path $dstpath and call the callback with the result code.
221 418
222 aio_symlink $srcpath, $dstpath, $callback->($status) 419 aio_symlink $srcpath, $dstpath, $callback->($status)
223 Asynchronously create a new symbolic link to the existing object at 420 Asynchronously create a new symbolic link to the existing object at
224 $srcpath at the path $dstpath and call the callback with the result 421 $srcpath at the path $dstpath and call the callback with the result
225 code. 422 code.
226 423
424 aio_readlink $path, $callback->($link)
425 Asynchronously read the symlink specified by $path and pass it to
426 the callback. If an error occurs, nothing or undef gets passed to
427 the callback.
428
227 aio_rename $srcpath, $dstpath, $callback->($status) 429 aio_rename $srcpath, $dstpath, $callback->($status)
228 Asynchronously rename the object at $srcpath to $dstpath, just as 430 Asynchronously rename the object at $srcpath to $dstpath, just as
229 rename(2) and call the callback with the result code. 431 rename(2) and call the callback with the result code.
432
433 aio_mkdir $pathname, $mode, $callback->($status)
434 Asynchronously mkdir (create) a directory and call the callback with
435 the result code. $mode will be modified by the umask at the time the
436 request is executed, so do not change your umask.
230 437
231 aio_rmdir $pathname, $callback->($status) 438 aio_rmdir $pathname, $callback->($status)
232 Asynchronously rmdir (delete) a directory and call the callback with 439 Asynchronously rmdir (delete) a directory and call the callback with
233 the result code. 440 the result code.
234 441
235 aio_readdir $pathname, $callback->($entries) 442 aio_readdir $pathname, $callback->($entries)
236 Unlike the POSIX call of the same name, "aio_readdir" reads an 443 Unlike the POSIX call of the same name, "aio_readdir" reads an
237 entire directory (i.e. opendir + readdir + closedir). The entries 444 entire directory (i.e. opendir + readdir + closedir). The entries
238 will not be sorted, and will NOT include the "." and ".." entries. 445 will not be sorted, and will NOT include the "." and ".." entries.
239 446
240 The callback a single argument which is either "undef" or an 447 The callback is passed a single argument which is either "undef" or
241 array-ref with the filenames. 448 an array-ref with the filenames.
449
450 aio_readdirx $pathname, $flags, $callback->($entries, $flags)
451 Quite similar to "aio_readdir", but the $flags argument allows to
452 tune behaviour and output format. In case of an error, $entries will
453 be "undef".
454
455 The flags are a combination of the following constants, ORed
456 together (the flags will also be passed to the callback, possibly
457 modified):
458
459 IO::AIO::READDIR_DENTS
460 When this flag is off, then the callback gets an arrayref with
461 of names only (as with "aio_readdir"), otherwise it gets an
462 arrayref with "[$name, $type, $inode]" arrayrefs, each
463 describing a single directory entry in more detail.
464
465 $name is the name of the entry.
466
467 $type is one of the "IO::AIO::DT_xxx" constants:
468
469 "IO::AIO::DT_UNKNOWN", "IO::AIO::DT_FIFO", "IO::AIO::DT_CHR",
470 "IO::AIO::DT_DIR", "IO::AIO::DT_BLK", "IO::AIO::DT_REG",
471 "IO::AIO::DT_LNK", "IO::AIO::DT_SOCK", "IO::AIO::DT_WHT".
472
473 "IO::AIO::DT_UNKNOWN" means just that: readdir does not know. If
474 you need to know, you have to run stat yourself. Also, for speed
475 reasons, the $type scalars are read-only: you can not modify
476 them.
477
478 $inode is the inode number (which might not be exact on systems
479 with 64 bit inode numbers and 32 bit perls). On systems that do
480 not deliver the inode information, this will always be zero.
481
482 IO::AIO::READDIR_DIRS_FIRST
483 When this flag is set, then the names will be returned in an
484 order where likely directories come first. This is useful when
485 you need to quickly find directories, or you want to find all
486 directories while avoiding to stat() each entry.
487
488 If the system returns type information in readdir, then this is
489 used to find directories directly. Otherwise, likely directories
490 are files beginning with ".", or otherwise files with no dots,
491 of which files with short names are tried first.
492
493 IO::AIO::READDIR_STAT_ORDER
494 When this flag is set, then the names will be returned in an
495 order suitable for stat()'ing each one. That is, when you plan
496 to stat() all files in the given directory, then the returned
497 order will likely be fastest.
498
499 If both this flag and "IO::AIO::READDIR_DIRS_FIRST" are
500 specified, then the likely dirs come first, resulting in a less
501 optimal stat order.
502
503 IO::AIO::READDIR_FOUND_UNKNOWN
504 This flag should not be set when calling "aio_readdirx".
505 Instead, it is being set by "aio_readdirx", when any of the
506 $type's found were "IO::AIO::DT_UNKNOWN". The absense of this
507 flag therefore indicates that all $type's are known, which can
508 be used to speed up some algorithms.
509
510 aio_load $path, $data, $callback->($status)
511 This is a composite request that tries to fully load the given file
512 into memory. Status is the same as with aio_read.
513
514 aio_copy $srcpath, $dstpath, $callback->($status)
515 Try to copy the *file* (directories not supported as either source
516 or destination) from $srcpath to $dstpath and call the callback with
517 the 0 (error) or -1 ok.
518
519 This is a composite request that creates the destination file with
520 mode 0200 and copies the contents of the source file into it using
521 "aio_sendfile", followed by restoring atime, mtime, access mode and
522 uid/gid, in that order.
523
524 If an error occurs, the partial destination file will be unlinked,
525 if possible, except when setting atime, mtime, access mode and
526 uid/gid, where errors are being ignored.
527
528 aio_move $srcpath, $dstpath, $callback->($status)
529 Try to move the *file* (directories not supported as either source
530 or destination) from $srcpath to $dstpath and call the callback with
531 the 0 (error) or -1 ok.
532
533 This is a composite request that tries to rename(2) the file first;
534 if rename fails with "EXDEV", it copies the file with "aio_copy"
535 and, if that is successful, unlinks the $srcpath.
242 536
243 aio_scandir $path, $maxreq, $callback->($dirs, $nondirs) 537 aio_scandir $path, $maxreq, $callback->($dirs, $nondirs)
244 Scans a directory (similar to "aio_readdir") and tries to separate 538 Scans a directory (similar to "aio_readdir") but additionally tries
245 the entries of directory $path into two sets of names, ones you can 539 to efficiently separate the entries of directory $path into two sets
246 recurse into (directories), and ones you cannot recurse into 540 of names, directories you can recurse into (directories), and ones
247 (everything else). 541 you cannot recurse into (everything else, including symlinks to
542 directories).
248 543
249 "aio_scandir" is a composite request that consists of many 544 "aio_scandir" is a composite request that creates of many sub
250 aio-primitives. $maxreq specifies the maximum number of outstanding 545 requests_ $maxreq specifies the maximum number of outstanding aio
251 aio requests that this function generates. If it is "<= 0", then a 546 requests that this function generates. If it is "<= 0", then a
252 suitable default will be chosen (currently 8). 547 suitable default will be chosen (currently 4).
253 548
254 On error, the callback is called without arguments, otherwise it 549 On error, the callback is called without arguments, otherwise it
255 receives two array-refs with path-relative entry names. 550 receives two array-refs with path-relative entry names.
256 551
257 Example: 552 Example:
… …
265 Implementation notes. 560 Implementation notes.
266 561
267 The "aio_readdir" cannot be avoided, but "stat()"'ing every entry 562 The "aio_readdir" cannot be avoided, but "stat()"'ing every entry
268 can. 563 can.
269 564
565 If readdir returns file type information, then this is used directly
566 to find directories.
567
270 After reading the directory, the modification time, size etc. of the 568 Otherwise, after reading the directory, the modification time, size
271 directory before and after the readdir is checked, and if they 569 etc. of the directory before and after the readdir is checked, and
272 match, the link count will be used to decide how many entries are 570 if they match (and isn't the current time), the link count will be
273 directories (if >= 2). Otherwise, no knowledge of the number of 571 used to decide how many entries are directories (if >= 2).
274 subdirectories will be assumed. 572 Otherwise, no knowledge of the number of subdirectories will be
573 assumed.
275 574
276 Then entires will be sorted into likely directories (everything 575 Then entries will be sorted into likely directories a non-initial
277 without a non-initial dot) and likely non-directories (everything 576 dot currently) and likely non-directories (see "aio_readdirx"). Then
278 else). Then every entry + "/." will be "stat"'ed, likely directories 577 every entry plus an appended "/." will be "stat"'ed, likely
279 first. This is often faster because filesystems might detect the 578 directories first, in order of their inode numbers. If that
579 succeeds, it assumes that the entry is a directory or a symlink to
580 directory (which will be checked seperately). This is often faster
581 than stat'ing the entry itself because filesystems might detect the
280 type of the entry without reading the inode data (e.g. ext2fs 582 type of the entry without reading the inode data (e.g. ext2fs
281 filetype feature). If that succeeds, it assumes that the entry is a 583 filetype feature), even on systems that cannot return the filetype
282 directory or a symlink to directory (which will be checked 584 information on readdir.
283 seperately).
284 585
285 If the known number of directories has been reached, the rest of the 586 If the known number of directories (link count - 2) has been
286 entries is assumed to be non-directories. 587 reached, the rest of the entries is assumed to be non-directories.
588
589 This only works with certainty on POSIX (= UNIX) filesystems, which
590 fortunately are the vast majority of filesystems around.
591
592 It will also likely work on non-POSIX filesystems with reduced
593 efficiency as those tend to return 0 or 1 as link counts, which
594 disables the directory counting heuristic.
595
596 aio_rmtree $path, $callback->($status)
597 Delete a directory tree starting (and including) $path, return the
598 status of the final "rmdir" only. This is a composite request that
599 uses "aio_scandir" to recurse into and rmdir directories, and unlink
600 everything else.
601
602 aio_sync $callback->($status)
603 Asynchronously call sync and call the callback when finished.
287 604
288 aio_fsync $fh, $callback->($status) 605 aio_fsync $fh, $callback->($status)
289 Asynchronously call fsync on the given filehandle and call the 606 Asynchronously call fsync on the given filehandle and call the
290 callback with the fsync result code. 607 callback with the fsync result code.
291 608
… …
294 callback with the fdatasync result code. 611 callback with the fdatasync result code.
295 612
296 If this call isn't available because your OS lacks it or it couldn't 613 If this call isn't available because your OS lacks it or it couldn't
297 be detected, it will be emulated by calling "fsync" instead. 614 be detected, it will be emulated by calling "fsync" instead.
298 615
616 aio_sync_file_range $fh, $offset, $nbytes, $flags, $callback->($status)
617 Sync the data portion of the file specified by $offset and $length
618 to disk (but NOT the metadata), by calling the Linux-specific
619 sync_file_range call. If sync_file_range is not available or it
620 returns ENOSYS, then fdatasync or fsync is being substituted.
621
622 $flags can be a combination of
623 "IO::AIO::SYNC_FILE_RANGE_WAIT_BEFORE",
624 "IO::AIO::SYNC_FILE_RANGE_WRITE" and
625 "IO::AIO::SYNC_FILE_RANGE_WAIT_AFTER": refer to the sync_file_range
626 manpage for details.
627
628 aio_pathsync $path, $callback->($status)
629 This request tries to open, fsync and close the given path. This is
630 a composite request intended to sync directories after directory
631 operations (E.g. rename). This might not work on all operating
632 systems or have any specific effect, but usually it makes sure that
633 directory changes get written to disc. It works for anything that
634 can be opened for read-only, not just directories.
635
636 Passes 0 when everything went ok, and -1 on error.
637
638 aio_group $callback->(...)
639 This is a very special aio request: Instead of doing something, it
640 is a container for other aio requests, which is useful if you want
641 to bundle many requests into a single, composite, request with a
642 definite callback and the ability to cancel the whole request with
643 its subrequests.
644
645 Returns an object of class IO::AIO::GRP. See its documentation below
646 for more info.
647
648 Example:
649
650 my $grp = aio_group sub {
651 print "all stats done\n";
652 };
653
654 add $grp
655 (aio_stat ...),
656 (aio_stat ...),
657 ...;
658
659 aio_nop $callback->()
660 This is a special request - it does nothing in itself and is only
661 used for side effects, such as when you want to add a dummy request
662 to a group so that finishing the requests in the group depends on
663 executing the given code.
664
665 While this request does nothing, it still goes through the execution
666 phase and still requires a worker thread. Thus, the callback will
667 not be executed immediately but only after other requests in the
668 queue have entered their execution phase. This can be used to
669 measure request latency.
670
671 IO::AIO::aio_busy $fractional_seconds, $callback->() *NOT EXPORTED*
672 Mainly used for debugging and benchmarking, this aio request puts
673 one of the request workers to sleep for the given time.
674
675 While it is theoretically handy to have simple I/O scheduling
676 requests like sleep and file handle readable/writable, the overhead
677 this creates is immense (it blocks a thread for a long time) so do
678 not use this function except to put your application under
679 artificial I/O pressure.
680
681 IO::AIO::REQ CLASS
682 All non-aggregate "aio_*" functions return an object of this class when
683 called in non-void context.
684
685 cancel $req
686 Cancels the request, if possible. Has the effect of skipping
687 execution when entering the execute state and skipping calling the
688 callback when entering the the result state, but will leave the
689 request otherwise untouched. That means that requests that currently
690 execute will not be stopped and resources held by the request will
691 not be freed prematurely.
692
693 cb $req $callback->(...)
694 Replace (or simply set) the callback registered to the request.
695
696 IO::AIO::GRP CLASS
697 This class is a subclass of IO::AIO::REQ, so all its methods apply to
698 objects of this class, too.
699
700 A IO::AIO::GRP object is a special request that can contain multiple
701 other aio requests.
702
703 You create one by calling the "aio_group" constructing function with a
704 callback that will be called when all contained requests have entered
705 the "done" state:
706
707 my $grp = aio_group sub {
708 print "all requests are done\n";
709 };
710
711 You add requests by calling the "add" method with one or more
712 "IO::AIO::REQ" objects:
713
714 $grp->add (aio_unlink "...");
715
716 add $grp aio_stat "...", sub {
717 $_[0] or return $grp->result ("error");
718
719 # add another request dynamically, if first succeeded
720 add $grp aio_open "...", sub {
721 $grp->result ("ok");
722 };
723 };
724
725 This makes it very easy to create composite requests (see the source of
726 "aio_move" for an application) that work and feel like simple requests.
727
728 * The IO::AIO::GRP objects will be cleaned up during calls to
729 "IO::AIO::poll_cb", just like any other request.
730
731 * They can be canceled like any other request. Canceling will cancel
732 not only the request itself, but also all requests it contains.
733
734 * They can also can also be added to other IO::AIO::GRP objects.
735
736 * You must not add requests to a group from within the group callback
737 (or any later time).
738
739 Their lifetime, simplified, looks like this: when they are empty, they
740 will finish very quickly. If they contain only requests that are in the
741 "done" state, they will also finish. Otherwise they will continue to
742 exist.
743
744 That means after creating a group you have some time to add requests
745 (precisely before the callback has been invoked, which is only done
746 within the "poll_cb"). And in the callbacks of those requests, you can
747 add further requests to the group. And only when all those requests have
748 finished will the the group itself finish.
749
750 add $grp ...
751 $grp->add (...)
752 Add one or more requests to the group. Any type of IO::AIO::REQ can
753 be added, including other groups, as long as you do not create
754 circular dependencies.
755
756 Returns all its arguments.
757
758 $grp->cancel_subs
759 Cancel all subrequests and clears any feeder, but not the group
760 request itself. Useful when you queued a lot of events but got a
761 result early.
762
763 $grp->result (...)
764 Set the result value(s) that will be passed to the group callback
765 when all subrequests have finished and set the groups errno to the
766 current value of errno (just like calling "errno" without an error
767 number). By default, no argument will be passed and errno is zero.
768
769 $grp->errno ([$errno])
770 Sets the group errno value to $errno, or the current value of errno
771 when the argument is missing.
772
773 Every aio request has an associated errno value that is restored
774 when the callback is invoked. This method lets you change this value
775 from its default (0).
776
777 Calling "result" will also set errno, so make sure you either set $!
778 before the call to "result", or call c<errno> after it.
779
780 feed $grp $callback->($grp)
781 Sets a feeder/generator on this group: every group can have an
782 attached generator that generates requests if idle. The idea behind
783 this is that, although you could just queue as many requests as you
784 want in a group, this might starve other requests for a potentially
785 long time. For example, "aio_scandir" might generate hundreds of
786 thousands "aio_stat" requests, delaying any later requests for a
787 long time.
788
789 To avoid this, and allow incremental generation of requests, you can
790 instead a group and set a feeder on it that generates those
791 requests. The feed callback will be called whenever there are few
792 enough (see "limit", below) requests active in the group itself and
793 is expected to queue more requests.
794
795 The feed callback can queue as many requests as it likes (i.e. "add"
796 does not impose any limits).
797
798 If the feed does not queue more requests when called, it will be
799 automatically removed from the group.
800
801 If the feed limit is 0 when this method is called, it will be set to
802 2 automatically.
803
804 Example:
805
806 # stat all files in @files, but only ever use four aio requests concurrently:
807
808 my $grp = aio_group sub { print "finished\n" };
809 limit $grp 4;
810 feed $grp sub {
811 my $file = pop @files
812 or return;
813
814 add $grp aio_stat $file, sub { ... };
815 };
816
817 limit $grp $num
818 Sets the feeder limit for the group: The feeder will be called
819 whenever the group contains less than this many requests.
820
821 Setting the limit to 0 will pause the feeding process.
822
823 The default value for the limit is 0, but note that setting a feeder
824 automatically bumps it up to 2.
825
299 SUPPORT FUNCTIONS 826 SUPPORT FUNCTIONS
827 EVENT PROCESSING AND EVENT LOOP INTEGRATION
300 $fileno = IO::AIO::poll_fileno 828 $fileno = IO::AIO::poll_fileno
301 Return the *request result pipe file descriptor*. This filehandle 829 Return the *request result pipe file descriptor*. This filehandle
302 must be polled for reading by some mechanism outside this module 830 must be polled for reading by some mechanism outside this module
303 (e.g. Event or select, see below or the SYNOPSIS). If the pipe 831 (e.g. Event or select, see below or the SYNOPSIS). If the pipe
304 becomes readable you have to call "poll_cb" to check the results. 832 becomes readable you have to call "poll_cb" to check the results.
305 833
306 See "poll_cb" for an example. 834 See "poll_cb" for an example.
307 835
308 IO::AIO::poll_cb 836 IO::AIO::poll_cb
309 Process all outstanding events on the result pipe. You have to call 837 Process some outstanding events on the result pipe. You have to call
310 this regularly. Returns the number of events processed. Returns 838 this regularly. Returns 0 if all events could be processed, or -1 if
311 immediately when no events are outstanding. 839 it returned earlier for whatever reason. Returns immediately when no
840 events are outstanding. The amount of events processed depends on
841 the settings of "IO::AIO::max_poll_req" and
842 "IO::AIO::max_poll_time".
843
844 If not all requests were processed for whatever reason, the
845 filehandle will still be ready when "poll_cb" returns, so normally
846 you don't have to do anything special to have it called later.
312 847
313 Example: Install an Event watcher that automatically calls 848 Example: Install an Event watcher that automatically calls
314 IO::AIO::poll_cb with high priority: 849 IO::AIO::poll_cb with high priority:
315 850
316 Event->io (fd => IO::AIO::poll_fileno, 851 Event->io (fd => IO::AIO::poll_fileno,
317 poll => 'r', async => 1, 852 poll => 'r', async => 1,
318 cb => \&IO::AIO::poll_cb); 853 cb => \&IO::AIO::poll_cb);
319 854
855 IO::AIO::max_poll_reqs $nreqs
856 IO::AIO::max_poll_time $seconds
857 These set the maximum number of requests (default 0, meaning
858 infinity) that are being processed by "IO::AIO::poll_cb" in one
859 call, respectively the maximum amount of time (default 0, meaning
860 infinity) spent in "IO::AIO::poll_cb" to process requests (more
861 correctly the mininum amount of time "poll_cb" is allowed to use).
862
863 Setting "max_poll_time" to a non-zero value creates an overhead of
864 one syscall per request processed, which is not normally a problem
865 unless your callbacks are really really fast or your OS is really
866 really slow (I am not mentioning Solaris here). Using
867 "max_poll_reqs" incurs no overhead.
868
869 Setting these is useful if you want to ensure some level of
870 interactiveness when perl is not fast enough to process all requests
871 in time.
872
873 For interactive programs, values such as 0.01 to 0.1 should be fine.
874
875 Example: Install an Event watcher that automatically calls
876 IO::AIO::poll_cb with low priority, to ensure that other parts of
877 the program get the CPU sometimes even under high AIO load.
878
879 # try not to spend much more than 0.1s in poll_cb
880 IO::AIO::max_poll_time 0.1;
881
882 # use a low priority so other tasks have priority
883 Event->io (fd => IO::AIO::poll_fileno,
884 poll => 'r', nice => 1,
885 cb => &IO::AIO::poll_cb);
886
320 IO::AIO::poll_wait 887 IO::AIO::poll_wait
888 If there are any outstanding requests and none of them in the result
321 Wait till the result filehandle becomes ready for reading (simply 889 phase, wait till the result filehandle becomes ready for reading
322 does a "select" on the filehandle. This is useful if you want to 890 (simply does a "select" on the filehandle. This is useful if you
323 synchronously wait for some requests to finish). 891 want to synchronously wait for some requests to finish).
324 892
325 See "nreqs" for an example. 893 See "nreqs" for an example.
326 894
895 IO::AIO::poll
896 Waits until some requests have been handled.
897
898 Returns the number of requests processed, but is otherwise strictly
899 equivalent to:
900
901 IO::AIO::poll_wait, IO::AIO::poll_cb
902
327 IO::AIO::nreqs 903 IO::AIO::flush
328 Returns the number of requests currently outstanding (i.e. for which 904 Wait till all outstanding AIO requests have been handled.
329 their callback has not been invoked yet).
330 905
331 Example: wait till there are no outstanding requests anymore: 906 Strictly equivalent to:
332 907
333 IO::AIO::poll_wait, IO::AIO::poll_cb 908 IO::AIO::poll_wait, IO::AIO::poll_cb
334 while IO::AIO::nreqs; 909 while IO::AIO::nreqs;
335 910
336 IO::AIO::flush 911 CONTROLLING THE NUMBER OF THREADS
337 Wait till all outstanding AIO requests have been handled.
338
339 Strictly equivalent to:
340
341 IO::AIO::poll_wait, IO::AIO::poll_cb
342 while IO::AIO::nreqs;
343
344 IO::AIO::poll
345 Waits until some requests have been handled.
346
347 Strictly equivalent to:
348
349 IO::AIO::poll_wait, IO::AIO::poll_cb
350 if IO::AIO::nreqs;
351
352 IO::AIO::min_parallel $nthreads 912 IO::AIO::min_parallel $nthreads
353 Set the minimum number of AIO threads to $nthreads. The current 913 Set the minimum number of AIO threads to $nthreads. The current
354 default is 4, which means four asynchronous operations can be done 914 default is 8, which means eight asynchronous operations can execute
355 at one time (the number of outstanding operations, however, is 915 concurrently at any one time (the number of outstanding requests,
356 unlimited). 916 however, is unlimited).
357 917
358 IO::AIO starts threads only on demand, when an AIO request is queued 918 IO::AIO starts threads only on demand, when an AIO request is queued
359 and no free thread exists. 919 and no free thread exists. Please note that queueing up a hundred
920 requests can create demand for a hundred threads, even if it turns
921 out that everything is in the cache and could have been processed
922 faster by a single thread.
360 923
361 It is recommended to keep the number of threads low, as some Linux 924 It is recommended to keep the number of threads relatively low, as
362 kernel versions will scale negatively with the number of threads 925 some Linux kernel versions will scale negatively with the number of
363 (higher parallelity => MUCH higher latency). With current Linux 2.6 926 threads (higher parallelity => MUCH higher latency). With current
364 versions, 4-32 threads should be fine. 927 Linux 2.6 versions, 4-32 threads should be fine.
365 928
366 Under most circumstances you don't need to call this function, as 929 Under most circumstances you don't need to call this function, as
367 the module selects a default that is suitable for low to moderate 930 the module selects a default that is suitable for low to moderate
368 load. 931 load.
369 932
… …
379 ensure that all threads are killed and that there are no outstanding 942 ensure that all threads are killed and that there are no outstanding
380 requests. 943 requests.
381 944
382 Under normal circumstances you don't need to call this function. 945 Under normal circumstances you don't need to call this function.
383 946
947 IO::AIO::max_idle $nthreads
948 Limit the number of threads (default: 4) that are allowed to idle
949 (i.e., threads that did not get a request to process within 10
950 seconds). That means if a thread becomes idle while $nthreads other
951 threads are also idle, it will free its resources and exit.
952
953 This is useful when you allow a large number of threads (e.g. 100 or
954 1000) to allow for extremely high load situations, but want to free
955 resources under normal circumstances (1000 threads can easily
956 consume 30MB of RAM).
957
958 The default is probably ok in most situations, especially if thread
959 creation is fast. If thread creation is very slow on your system you
960 might want to use larger values.
961
384 $oldnreqs = IO::AIO::max_outstanding $nreqs 962 IO::AIO::max_outstanding $maxreqs
963 This is a very bad function to use in interactive programs because
964 it blocks, and a bad way to reduce concurrency because it is
965 inexact: Better use an "aio_group" together with a feed callback.
966
385 Sets the maximum number of outstanding requests to $nreqs. If you 967 Sets the maximum number of outstanding requests to $nreqs. If you do
386 try to queue up more than this number of requests, the caller will 968 queue up more than this number of requests, the next call to the
387 block until some requests have been handled. 969 "poll_cb" (and "poll_some" and other functions calling "poll_cb")
970 function will block until the limit is no longer exceeded.
388 971
389 The default is very large, so normally there is no practical limit. 972 The default value is very large, so there is no practical limit on
390 If you queue up many requests in a loop it often improves speed if 973 the number of outstanding requests.
391 you set this to a relatively low number, such as 100.
392 974
393 Under normal circumstances you don't need to call this function. 975 You can still queue as many requests as you want. Therefore,
976 "max_outstanding" is mainly useful in simple scripts (with low
977 values) or as a stop gap to shield against fatal memory overflow
978 (with large values).
979
980 STATISTICAL INFORMATION
981 IO::AIO::nreqs
982 Returns the number of requests currently in the ready, execute or
983 pending states (i.e. for which their callback has not been invoked
984 yet).
985
986 Example: wait till there are no outstanding requests anymore:
987
988 IO::AIO::poll_wait, IO::AIO::poll_cb
989 while IO::AIO::nreqs;
990
991 IO::AIO::nready
992 Returns the number of requests currently in the ready state (not yet
993 executed).
994
995 IO::AIO::npending
996 Returns the number of requests currently in the pending state
997 (executed, but not yet processed by poll_cb).
394 998
395 FORK BEHAVIOUR 999 FORK BEHAVIOUR
1000 This module should do "the right thing" when the process using it forks:
1001
396 Before the fork, IO::AIO enters a quiescent state where no requests can 1002 Before the fork, IO::AIO enters a quiescent state where no requests can
397 be added in other threads and no results will be processed. After the 1003 be added in other threads and no results will be processed. After the
398 fork the parent simply leaves the quiescent state and continues 1004 fork the parent simply leaves the quiescent state and continues
399 request/result processing, while the child clears the request/result 1005 request/result processing, while the child frees the request/result
400 queue (so the requests started before the fork will only be handled in 1006 queue (so that the requests started before the fork will only be handled
401 the parent). Threats will be started on demand until the limit ste in 1007 in the parent). Threads will be started on demand until the limit set in
402 the parent process has been reached again. 1008 the parent process has been reached again.
403 1009
1010 In short: the parent will, after a short pause, continue as if fork had
1011 not been called, while the child will act as if IO::AIO has not been
1012 used yet.
1013
1014 MEMORY USAGE
1015 Per-request usage:
1016
1017 Each aio request uses - depending on your architecture - around 100-200
1018 bytes of memory. In addition, stat requests need a stat buffer (possibly
1019 a few hundred bytes), readdir requires a result buffer and so on. Perl
1020 scalars and other data passed into aio requests will also be locked and
1021 will consume memory till the request has entered the done state.
1022
1023 This is not awfully much, so queuing lots of requests is not usually a
1024 problem.
1025
1026 Per-thread usage:
1027
1028 In the execution phase, some aio requests require more memory for
1029 temporary buffers, and each thread requires a stack and other data
1030 structures (usually around 16k-128k, depending on the OS).
1031
1032KNOWN BUGS
1033 Known bugs will be fixed in the next release.
1034
404SEE ALSO 1035SEE ALSO
405 Coro, Linux::AIO. 1036 AnyEvent::AIO for easy integration into event loops, Coro::AIO for a
1037 more natural syntax.
406 1038
407AUTHOR 1039AUTHOR
408 Marc Lehmann <schmorp@schmorp.de> 1040 Marc Lehmann <schmorp@schmorp.de>
409 http://home.schmorp.de/ 1041 http://home.schmorp.de/
410 1042

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines