ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/IO-AIO/AIO.pm
(Generate patch)

Comparing IO-AIO/AIO.pm (file contents):
Revision 1.105 by root, Sun Mar 25 00:20:27 2007 UTC vs.
Revision 1.115 by root, Mon Sep 24 18:14:00 2007 UTC

62etc.), but can also be used to easily do operations in parallel that are 62etc.), but can also be used to easily do operations in parallel that are
63normally done sequentially, e.g. stat'ing many files, which is much faster 63normally done sequentially, e.g. stat'ing many files, which is much faster
64on a RAID volume or over NFS when you do a number of stat operations 64on a RAID volume or over NFS when you do a number of stat operations
65concurrently. 65concurrently.
66 66
67While most of this works on all types of file descriptors (for example 67While most of this works on all types of file descriptors (for
68sockets), using these functions on file descriptors that support 68example sockets), using these functions on file descriptors that
69nonblocking operation (again, sockets, pipes etc.) is very inefficient or 69support nonblocking operation (again, sockets, pipes etc.) is very
70might not work (aio_read fails on sockets/pipes/fifos). Use an event loop 70inefficient. Use an event loop for that (such as the L<Event|Event>
71for that (such as the L<Event|Event> module): IO::AIO will naturally fit 71module): IO::AIO will naturally fit into such an event loop itself.
72into such an event loop itself.
73 72
74In this version, a number of threads are started that execute your 73In this version, a number of threads are started that execute your
75requests and signal their completion. You don't need thread support 74requests and signal their completion. You don't need thread support
76in perl, and the threads created by this module will not be visible 75in perl, and the threads created by this module will not be visible
77to perl. In the future, this module might make use of the native aio 76to perl. In the future, this module might make use of the native aio
79not well-supported or restricted (GNU/Linux doesn't allow them on normal 78not well-supported or restricted (GNU/Linux doesn't allow them on normal
80files currently, for example), and they would only support aio_read and 79files currently, for example), and they would only support aio_read and
81aio_write, so the remaining functionality would have to be implemented 80aio_write, so the remaining functionality would have to be implemented
82using threads anyway. 81using threads anyway.
83 82
84Although the module will work with in the presence of other (Perl-) 83Although the module will work in the presence of other (Perl-) threads,
85threads, it is currently not reentrant in any way, so use appropriate 84it is currently not reentrant in any way, so use appropriate locking
86locking yourself, always call C<poll_cb> from within the same thread, or 85yourself, always call C<poll_cb> from within the same thread, or never
87never call C<poll_cb> (or other C<aio_> functions) recursively. 86call C<poll_cb> (or other C<aio_> functions) recursively.
88 87
89=head2 EXAMPLE 88=head2 EXAMPLE
90 89
91This is a simple example that uses the Event module and loads 90This is a simple example that uses the Event module and loads
92F</etc/passwd> asynchronously: 91F</etc/passwd> asynchronously:
190use strict 'vars'; 189use strict 'vars';
191 190
192use base 'Exporter'; 191use base 'Exporter';
193 192
194BEGIN { 193BEGIN {
195 our $VERSION = '2.33'; 194 our $VERSION = '2.41';
196 195
197 our @AIO_REQ = qw(aio_sendfile aio_read aio_write aio_open aio_close aio_stat 196 our @AIO_REQ = qw(aio_sendfile aio_read aio_write aio_open aio_close aio_stat
198 aio_lstat aio_unlink aio_rmdir aio_readdir aio_scandir aio_symlink 197 aio_lstat aio_unlink aio_rmdir aio_readdir aio_scandir aio_symlink
199 aio_readlink aio_fsync aio_fdatasync aio_readahead aio_rename aio_link 198 aio_readlink aio_fsync aio_fdatasync aio_readahead aio_rename aio_link
200 aio_move aio_copy aio_group aio_nop aio_mknod aio_load aio_rmtree aio_mkdir); 199 aio_move aio_copy aio_group aio_nop aio_mknod aio_load aio_rmtree aio_mkdir
200 aio_chown aio_chmod aio_utime aio_truncate);
201 our @EXPORT = (@AIO_REQ, qw(aioreq_pri aioreq_nice aio_block)); 201 our @EXPORT = (@AIO_REQ, qw(aioreq_pri aioreq_nice aio_block));
202 our @EXPORT_OK = qw(poll_fileno poll_cb poll_wait flush 202 our @EXPORT_OK = qw(poll_fileno poll_cb poll_wait flush
203 min_parallel max_parallel max_idle 203 min_parallel max_parallel max_idle
204 nreqs nready npending nthreads 204 nreqs nready npending nthreads
205 max_poll_time max_poll_reqs); 205 max_poll_time max_poll_reqs);
271 aio_read $_[0], ..., sub { 271 aio_read $_[0], ..., sub {
272 ... 272 ...
273 }; 273 };
274 }; 274 };
275 275
276
276=item aioreq_nice $pri_adjust 277=item aioreq_nice $pri_adjust
277 278
278Similar to C<aioreq_pri>, but subtracts the given value from the current 279Similar to C<aioreq_pri>, but subtracts the given value from the current
279priority, so the effect is cumulative. 280priority, so the effect is cumulative.
281
280 282
281=item aio_open $pathname, $flags, $mode, $callback->($fh) 283=item aio_open $pathname, $flags, $mode, $callback->($fh)
282 284
283Asynchronously open or create a file and call the callback with a newly 285Asynchronously open or create a file and call the callback with a newly
284created filehandle for the file. 286created filehandle for the file.
305 } else { 307 } else {
306 die "open failed: $!\n"; 308 die "open failed: $!\n";
307 } 309 }
308 }; 310 };
309 311
312
310=item aio_close $fh, $callback->($status) 313=item aio_close $fh, $callback->($status)
311 314
312Asynchronously close a file and call the callback with the result 315Asynchronously close a file and call the callback with the result
313code. I<WARNING:> although accepted, you should not pass in a perl 316code. I<WARNING:> although accepted, you should not pass in a perl
314filehandle here, as perl will likely close the file descriptor another 317filehandle here, as perl will likely close the file descriptor another
316C<close> or just let filehandles go out of scope. 319C<close> or just let filehandles go out of scope.
317 320
318This is supposed to be a bug in the API, so that might change. It's 321This is supposed to be a bug in the API, so that might change. It's
319therefore best to avoid this function. 322therefore best to avoid this function.
320 323
324
321=item aio_read $fh,$offset,$length, $data,$dataoffset, $callback->($retval) 325=item aio_read $fh,$offset,$length, $data,$dataoffset, $callback->($retval)
322 326
323=item aio_write $fh,$offset,$length, $data,$dataoffset, $callback->($retval) 327=item aio_write $fh,$offset,$length, $data,$dataoffset, $callback->($retval)
324 328
325Reads or writes C<length> bytes from the specified C<fh> and C<offset> 329Reads or writes C<$length> bytes from the specified C<$fh> and C<$offset>
326into the scalar given by C<data> and offset C<dataoffset> and calls the 330into the scalar given by C<$data> and offset C<$dataoffset> and calls the
327callback without the actual number of bytes read (or -1 on error, just 331callback without the actual number of bytes read (or -1 on error, just
328like the syscall). 332like the syscall).
329 333
334If C<$offset> is undefined, then the current file descriptor offset will
335be used (and updated), otherwise the file descriptor offset will not be
336changed by these calls.
337
338If C<$length> is undefined in C<aio_write>, use the remaining length of C<$data>.
339
340If C<$dataoffset> is less than zero, it will be counted from the end of
341C<$data>.
342
330The C<$data> scalar I<MUST NOT> be modified in any way while the request 343The C<$data> scalar I<MUST NOT> be modified in any way while the request
331is outstanding. Modifying it can result in segfaults or WW3 (if the 344is outstanding. Modifying it can result in segfaults or World War III (if
332necessary/optional hardware is installed). 345the necessary/optional hardware is installed).
333 346
334Example: Read 15 bytes at offset 7 into scalar C<$buffer>, starting at 347Example: Read 15 bytes at offset 7 into scalar C<$buffer>, starting at
335offset C<0> within the scalar: 348offset C<0> within the scalar:
336 349
337 aio_read $fh, 7, 15, $buffer, 0, sub { 350 aio_read $fh, 7, 15, $buffer, 0, sub {
338 $_[0] > 0 or die "read error: $!"; 351 $_[0] > 0 or die "read error: $!";
339 print "read $_[0] bytes: <$buffer>\n"; 352 print "read $_[0] bytes: <$buffer>\n";
340 }; 353 };
354
341 355
342=item aio_sendfile $out_fh, $in_fh, $in_offset, $length, $callback->($retval) 356=item aio_sendfile $out_fh, $in_fh, $in_offset, $length, $callback->($retval)
343 357
344Tries to copy C<$length> bytes from C<$in_fh> to C<$out_fh>. It starts 358Tries to copy C<$length> bytes from C<$in_fh> to C<$out_fh>. It starts
345reading at byte offset C<$in_offset>, and starts writing at the current 359reading at byte offset C<$in_offset>, and starts writing at the current
359C<$in_fh> than are written, and there is no way to find out how many 373C<$in_fh> than are written, and there is no way to find out how many
360bytes have been read from C<aio_sendfile> alone, as C<aio_sendfile> only 374bytes have been read from C<aio_sendfile> alone, as C<aio_sendfile> only
361provides the number of bytes written to C<$out_fh>. Only if the result 375provides the number of bytes written to C<$out_fh>. Only if the result
362value equals C<$length> one can assume that C<$length> bytes have been 376value equals C<$length> one can assume that C<$length> bytes have been
363read. 377read.
378
364 379
365=item aio_readahead $fh,$offset,$length, $callback->($retval) 380=item aio_readahead $fh,$offset,$length, $callback->($retval)
366 381
367C<aio_readahead> populates the page cache with data from a file so that 382C<aio_readahead> populates the page cache with data from a file so that
368subsequent reads from that file will not block on disk I/O. The C<$offset> 383subsequent reads from that file will not block on disk I/O. The C<$offset>
374file. The current file offset of the file is left unchanged. 389file. The current file offset of the file is left unchanged.
375 390
376If that syscall doesn't exist (likely if your OS isn't Linux) it will be 391If that syscall doesn't exist (likely if your OS isn't Linux) it will be
377emulated by simply reading the data, which would have a similar effect. 392emulated by simply reading the data, which would have a similar effect.
378 393
394
379=item aio_stat $fh_or_path, $callback->($status) 395=item aio_stat $fh_or_path, $callback->($status)
380 396
381=item aio_lstat $fh, $callback->($status) 397=item aio_lstat $fh, $callback->($status)
382 398
383Works like perl's C<stat> or C<lstat> in void context. The callback will 399Works like perl's C<stat> or C<lstat> in void context. The callback will
396 aio_stat "/etc/passwd", sub { 412 aio_stat "/etc/passwd", sub {
397 $_[0] and die "stat failed: $!"; 413 $_[0] and die "stat failed: $!";
398 print "size is ", -s _, "\n"; 414 print "size is ", -s _, "\n";
399 }; 415 };
400 416
417
418=item aio_utime $fh_or_path, $atime, $mtime, $callback->($status)
419
420Works like perl's C<utime> function (including the special case of $atime
421and $mtime being undef). Fractional times are supported if the underlying
422syscalls support them.
423
424When called with a pathname, uses utimes(2) if available, otherwise
425utime(2). If called on a file descriptor, uses futimes(2) if available,
426otherwise returns ENOSYS, so this is not portable.
427
428Examples:
429
430 # set atime and mtime to current time (basically touch(1)):
431 aio_utime "path", undef, undef;
432 # set atime to current time and mtime to beginning of the epoch:
433 aio_utime "path", time, undef; # undef==0
434
435
436=item aio_chown $fh_or_path, $uid, $gid, $callback->($status)
437
438Works like perl's C<chown> function, except that C<undef> for either $uid
439or $gid is being interpreted as "do not change" (but -1 can also be used).
440
441Examples:
442
443 # same as "chown root path" in the shell:
444 aio_chown "path", 0, -1;
445 # same as above:
446 aio_chown "path", 0, undef;
447
448
449=item aio_truncate $fh_or_path, $offset, $callback->($status)
450
451Works like truncate(2) or ftruncate(2).
452
453
454=item aio_chmod $fh_or_path, $mode, $callback->($status)
455
456Works like perl's C<chmod> function.
457
458
401=item aio_unlink $pathname, $callback->($status) 459=item aio_unlink $pathname, $callback->($status)
402 460
403Asynchronously unlink (delete) a file and call the callback with the 461Asynchronously unlink (delete) a file and call the callback with the
404result code. 462result code.
405 463
464
406=item aio_mknod $path, $mode, $dev, $callback->($status) 465=item aio_mknod $path, $mode, $dev, $callback->($status)
407 466
408[EXPERIMENTAL] 467[EXPERIMENTAL]
409 468
410Asynchronously create a device node (or fifo). See mknod(2). 469Asynchronously create a device node (or fifo). See mknod(2).
411 470
412The only (POSIX-) portable way of calling this function is: 471The only (POSIX-) portable way of calling this function is:
413 472
414 aio_mknod $path, IO::AIO::S_IFIFO | $mode, 0, sub { ... 473 aio_mknod $path, IO::AIO::S_IFIFO | $mode, 0, sub { ...
474
415 475
416=item aio_link $srcpath, $dstpath, $callback->($status) 476=item aio_link $srcpath, $dstpath, $callback->($status)
417 477
418Asynchronously create a new link to the existing object at C<$srcpath> at 478Asynchronously create a new link to the existing object at C<$srcpath> at
419the path C<$dstpath> and call the callback with the result code. 479the path C<$dstpath> and call the callback with the result code.
420 480
481
421=item aio_symlink $srcpath, $dstpath, $callback->($status) 482=item aio_symlink $srcpath, $dstpath, $callback->($status)
422 483
423Asynchronously create a new symbolic link to the existing object at C<$srcpath> at 484Asynchronously create a new symbolic link to the existing object at C<$srcpath> at
424the path C<$dstpath> and call the callback with the result code. 485the path C<$dstpath> and call the callback with the result code.
486
425 487
426=item aio_readlink $path, $callback->($link) 488=item aio_readlink $path, $callback->($link)
427 489
428Asynchronously read the symlink specified by C<$path> and pass it to 490Asynchronously read the symlink specified by C<$path> and pass it to
429the callback. If an error occurs, nothing or undef gets passed to the 491the callback. If an error occurs, nothing or undef gets passed to the
430callback. 492callback.
431 493
494
432=item aio_rename $srcpath, $dstpath, $callback->($status) 495=item aio_rename $srcpath, $dstpath, $callback->($status)
433 496
434Asynchronously rename the object at C<$srcpath> to C<$dstpath>, just as 497Asynchronously rename the object at C<$srcpath> to C<$dstpath>, just as
435rename(2) and call the callback with the result code. 498rename(2) and call the callback with the result code.
499
436 500
437=item aio_mkdir $pathname, $mode, $callback->($status) 501=item aio_mkdir $pathname, $mode, $callback->($status)
438 502
439Asynchronously mkdir (create) a directory and call the callback with 503Asynchronously mkdir (create) a directory and call the callback with
440the result code. C<$mode> will be modified by the umask at the time the 504the result code. C<$mode> will be modified by the umask at the time the
441request is executed, so do not change your umask. 505request is executed, so do not change your umask.
442 506
507
443=item aio_rmdir $pathname, $callback->($status) 508=item aio_rmdir $pathname, $callback->($status)
444 509
445Asynchronously rmdir (delete) a directory and call the callback with the 510Asynchronously rmdir (delete) a directory and call the callback with the
446result code. 511result code.
512
447 513
448=item aio_readdir $pathname, $callback->($entries) 514=item aio_readdir $pathname, $callback->($entries)
449 515
450Unlike the POSIX call of the same name, C<aio_readdir> reads an entire 516Unlike the POSIX call of the same name, C<aio_readdir> reads an entire
451directory (i.e. opendir + readdir + closedir). The entries will not be 517directory (i.e. opendir + readdir + closedir). The entries will not be
452sorted, and will B<NOT> include the C<.> and C<..> entries. 518sorted, and will B<NOT> include the C<.> and C<..> entries.
453 519
454The callback a single argument which is either C<undef> or an array-ref 520The callback a single argument which is either C<undef> or an array-ref
455with the filenames. 521with the filenames.
522
456 523
457=item aio_load $path, $data, $callback->($status) 524=item aio_load $path, $data, $callback->($status)
458 525
459This is a composite request that tries to fully load the given file into 526This is a composite request that tries to fully load the given file into
460memory. Status is the same as with aio_read. 527memory. Status is the same as with aio_read.
1140This is a very bad function to use in interactive programs because it 1207This is a very bad function to use in interactive programs because it
1141blocks, and a bad way to reduce concurrency because it is inexact: Better 1208blocks, and a bad way to reduce concurrency because it is inexact: Better
1142use an C<aio_group> together with a feed callback. 1209use an C<aio_group> together with a feed callback.
1143 1210
1144Sets the maximum number of outstanding requests to C<$nreqs>. If you 1211Sets the maximum number of outstanding requests to C<$nreqs>. If you
1145to queue up more than this number of requests, the next call to the 1212do queue up more than this number of requests, the next call to the
1146C<poll_cb> (and C<poll_some> and other functions calling C<poll_cb>) 1213C<poll_cb> (and C<poll_some> and other functions calling C<poll_cb>)
1147function will block until the limit is no longer exceeded. 1214function will block until the limit is no longer exceeded.
1148 1215
1149The default value is very large, so there is no practical limit on the 1216The default value is very large, so there is no practical limit on the
1150number of outstanding requests. 1217number of outstanding requests.
1229bytes of memory. In addition, stat requests need a stat buffer (possibly 1296bytes of memory. In addition, stat requests need a stat buffer (possibly
1230a few hundred bytes), readdir requires a result buffer and so on. Perl 1297a few hundred bytes), readdir requires a result buffer and so on. Perl
1231scalars and other data passed into aio requests will also be locked and 1298scalars and other data passed into aio requests will also be locked and
1232will consume memory till the request has entered the done state. 1299will consume memory till the request has entered the done state.
1233 1300
1234This is now awfully much, so queuing lots of requests is not usually a 1301This is not awfully much, so queuing lots of requests is not usually a
1235problem. 1302problem.
1236 1303
1237Per-thread usage: 1304Per-thread usage:
1238 1305
1239In the execution phase, some aio requests require more memory for 1306In the execution phase, some aio requests require more memory for

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines