ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/IO-AIO/AIO.pm
(Generate patch)

Comparing IO-AIO/AIO.pm (file contents):
Revision 1.89 by root, Sun Oct 29 11:03:18 2006 UTC vs.
Revision 1.116 by root, Wed Oct 3 21:27:51 2007 UTC

… …
5=head1 SYNOPSIS 5=head1 SYNOPSIS
6 6
7 use IO::AIO; 7 use IO::AIO;
8 8
9 aio_open "/etc/passwd", O_RDONLY, 0, sub { 9 aio_open "/etc/passwd", O_RDONLY, 0, sub {
10 my ($fh) = @_; 10 my $fh = shift
11 or die "/etc/passwd: $!";
11 ... 12 ...
12 }; 13 };
13 14
14 aio_unlink "/tmp/file", sub { }; 15 aio_unlink "/tmp/file", sub { };
15 16
… …
61etc.), but can also be used to easily do operations in parallel that are 62etc.), but can also be used to easily do operations in parallel that are
62normally done sequentially, e.g. stat'ing many files, which is much faster 63normally done sequentially, e.g. stat'ing many files, which is much faster
63on a RAID volume or over NFS when you do a number of stat operations 64on a RAID volume or over NFS when you do a number of stat operations
64concurrently. 65concurrently.
65 66
66While this works on all types of file descriptors (for example sockets), 67While most of this works on all types of file descriptors (for
67using these functions on file descriptors that support nonblocking 68example sockets), using these functions on file descriptors that
68operation (again, sockets, pipes etc.) is very inefficient. Use an event 69support nonblocking operation (again, sockets, pipes etc.) is very
69loop for that (such as the L<Event|Event> module): IO::AIO will naturally 70inefficient. Use an event loop for that (such as the L<Event|Event>
70fit into such an event loop itself. 71module): IO::AIO will naturally fit into such an event loop itself.
71 72
72In this version, a number of threads are started that execute your 73In this version, a number of threads are started that execute your
73requests and signal their completion. You don't need thread support 74requests and signal their completion. You don't need thread support
74in perl, and the threads created by this module will not be visible 75in perl, and the threads created by this module will not be visible
75to perl. In the future, this module might make use of the native aio 76to perl. In the future, this module might make use of the native aio
… …
77not well-supported or restricted (GNU/Linux doesn't allow them on normal 78not well-supported or restricted (GNU/Linux doesn't allow them on normal
78files currently, for example), and they would only support aio_read and 79files currently, for example), and they would only support aio_read and
79aio_write, so the remaining functionality would have to be implemented 80aio_write, so the remaining functionality would have to be implemented
80using threads anyway. 81using threads anyway.
81 82
82Although the module will work with in the presence of other (Perl-) 83Although the module will work in the presence of other (Perl-) threads,
83threads, it is currently not reentrant in any way, so use appropriate 84it is currently not reentrant in any way, so use appropriate locking
84locking yourself, always call C<poll_cb> from within the same thread, or 85yourself, always call C<poll_cb> from within the same thread, or never
85never call C<poll_cb> (or other C<aio_> functions) recursively. 86call C<poll_cb> (or other C<aio_> functions) recursively.
86 87
87=head2 EXAMPLE 88=head2 EXAMPLE
88 89
89This is a simple example that uses the Event module and loads 90This is a simple example that uses the Event module and loads
90F</etc/passwd> asynchronously: 91F</etc/passwd> asynchronously:
… …
98 poll => 'r', 99 poll => 'r',
99 cb => \&IO::AIO::poll_cb); 100 cb => \&IO::AIO::poll_cb);
100 101
101 # queue the request to open /etc/passwd 102 # queue the request to open /etc/passwd
102 aio_open "/etc/passwd", O_RDONLY, 0, sub { 103 aio_open "/etc/passwd", O_RDONLY, 0, sub {
103 my $fh = $_[0] 104 my $fh = shift
104 or die "error while opening: $!"; 105 or die "error while opening: $!";
105 106
106 # stat'ing filehandles is generally non-blocking 107 # stat'ing filehandles is generally non-blocking
107 my $size = -s $fh; 108 my $size = -s $fh;
108 109
… …
188use strict 'vars'; 189use strict 'vars';
189 190
190use base 'Exporter'; 191use base 'Exporter';
191 192
192BEGIN { 193BEGIN {
193 our $VERSION = '2.1'; 194 our $VERSION = '2.5';
194 195
195 our @AIO_REQ = qw(aio_sendfile aio_read aio_write aio_open aio_close aio_stat 196 our @AIO_REQ = qw(aio_sendfile aio_read aio_write aio_open aio_close aio_stat
196 aio_lstat aio_unlink aio_rmdir aio_readdir aio_scandir aio_symlink 197 aio_lstat aio_unlink aio_rmdir aio_readdir aio_scandir aio_symlink
197 aio_fsync aio_fdatasync aio_readahead aio_rename aio_link aio_move 198 aio_readlink aio_fsync aio_fdatasync aio_readahead aio_rename aio_link
198 aio_copy aio_group aio_nop aio_mknod); 199 aio_move aio_copy aio_group aio_nop aio_mknod aio_load aio_rmtree aio_mkdir
200 aio_chown aio_chmod aio_utime aio_truncate);
199 our @EXPORT = (@AIO_REQ, qw(aioreq_pri aioreq_nice)); 201 our @EXPORT = (@AIO_REQ, qw(aioreq_pri aioreq_nice aio_block));
200 our @EXPORT_OK = qw(poll_fileno poll_cb poll_wait flush 202 our @EXPORT_OK = qw(poll_fileno poll_cb poll_wait flush
201 min_parallel max_parallel max_idle 203 min_parallel max_parallel max_idle
202 nreqs nready npending nthreads 204 nreqs nready npending nthreads
203 max_poll_time max_poll_reqs); 205 max_poll_time max_poll_reqs);
204 206
… …
269 aio_read $_[0], ..., sub { 271 aio_read $_[0], ..., sub {
270 ... 272 ...
271 }; 273 };
272 }; 274 };
273 275
276
274=item aioreq_nice $pri_adjust 277=item aioreq_nice $pri_adjust
275 278
276Similar to C<aioreq_pri>, but subtracts the given value from the current 279Similar to C<aioreq_pri>, but subtracts the given value from the current
277priority, so the effect is cumulative. 280priority, so the effect is cumulative.
281
278 282
279=item aio_open $pathname, $flags, $mode, $callback->($fh) 283=item aio_open $pathname, $flags, $mode, $callback->($fh)
280 284
281Asynchronously open or create a file and call the callback with a newly 285Asynchronously open or create a file and call the callback with a newly
282created filehandle for the file. 286created filehandle for the file.
… …
288list. They are the same as used by C<sysopen>. 292list. They are the same as used by C<sysopen>.
289 293
290Likewise, C<$mode> specifies the mode of the newly created file, if it 294Likewise, C<$mode> specifies the mode of the newly created file, if it
291didn't exist and C<O_CREAT> has been given, just like perl's C<sysopen>, 295didn't exist and C<O_CREAT> has been given, just like perl's C<sysopen>,
292except that it is mandatory (i.e. use C<0> if you don't create new files, 296except that it is mandatory (i.e. use C<0> if you don't create new files,
293and C<0666> or C<0777> if you do). 297and C<0666> or C<0777> if you do). Note that the C<$mode> will be modified
298by the umask in effect then the request is being executed, so better never
299change the umask.
294 300
295Example: 301Example:
296 302
297 aio_open "/etc/passwd", O_RDONLY, 0, sub { 303 aio_open "/etc/passwd", O_RDONLY, 0, sub {
298 if ($_[0]) { 304 if ($_[0]) {
… …
301 } else { 307 } else {
302 die "open failed: $!\n"; 308 die "open failed: $!\n";
303 } 309 }
304 }; 310 };
305 311
312
306=item aio_close $fh, $callback->($status) 313=item aio_close $fh, $callback->($status)
307 314
308Asynchronously close a file and call the callback with the result 315Asynchronously close a file and call the callback with the result
309code. I<WARNING:> although accepted, you should not pass in a perl 316code.
310filehandle here, as perl will likely close the file descriptor another
311time when the filehandle is destroyed. Normally, you can safely call perls
312C<close> or just let filehandles go out of scope.
313 317
314This is supposed to be a bug in the API, so that might change. It's 318Unlike the other functions operating on files, this function uses the
315therefore best to avoid this function. 319PerlIO layer to close the filehandle. The reason is that the PerlIO API
320insists on closing the underlying fd itself, no matter what, and doesn't
321allow modifications to the fd. Unfortunately, it is not clear that you can
322call PerlIO from different threads (actually, its quite clear that this
323won't work in some cases), so while it likely works perfectly with simple
324file handles (such as the ones created by C<aio_open>) it might fail in
325interesting ways for others.
326
327Having said that, aio_close tries to clean up the filehandle as much as
328possible before handing it to an io thread, and generally does work.
329
316 330
317=item aio_read $fh,$offset,$length, $data,$dataoffset, $callback->($retval) 331=item aio_read $fh,$offset,$length, $data,$dataoffset, $callback->($retval)
318 332
319=item aio_write $fh,$offset,$length, $data,$dataoffset, $callback->($retval) 333=item aio_write $fh,$offset,$length, $data,$dataoffset, $callback->($retval)
320 334
321Reads or writes C<length> bytes from the specified C<fh> and C<offset> 335Reads or writes C<$length> bytes from the specified C<$fh> and C<$offset>
322into the scalar given by C<data> and offset C<dataoffset> and calls the 336into the scalar given by C<$data> and offset C<$dataoffset> and calls the
323callback without the actual number of bytes read (or -1 on error, just 337callback without the actual number of bytes read (or -1 on error, just
324like the syscall). 338like the syscall).
325 339
340If C<$offset> is undefined, then the current file descriptor offset will
341be used (and updated), otherwise the file descriptor offset will not be
342changed by these calls.
343
344If C<$length> is undefined in C<aio_write>, use the remaining length of C<$data>.
345
346If C<$dataoffset> is less than zero, it will be counted from the end of
347C<$data>.
348
326The C<$data> scalar I<MUST NOT> be modified in any way while the request 349The C<$data> scalar I<MUST NOT> be modified in any way while the request
327is outstanding. Modifying it can result in segfaults or WW3 (if the 350is outstanding. Modifying it can result in segfaults or World War III (if
328necessary/optional hardware is installed). 351the necessary/optional hardware is installed).
329 352
330Example: Read 15 bytes at offset 7 into scalar C<$buffer>, starting at 353Example: Read 15 bytes at offset 7 into scalar C<$buffer>, starting at
331offset C<0> within the scalar: 354offset C<0> within the scalar:
332 355
333 aio_read $fh, 7, 15, $buffer, 0, sub { 356 aio_read $fh, 7, 15, $buffer, 0, sub {
334 $_[0] > 0 or die "read error: $!"; 357 $_[0] > 0 or die "read error: $!";
335 print "read $_[0] bytes: <$buffer>\n"; 358 print "read $_[0] bytes: <$buffer>\n";
336 }; 359 };
360
337 361
338=item aio_sendfile $out_fh, $in_fh, $in_offset, $length, $callback->($retval) 362=item aio_sendfile $out_fh, $in_fh, $in_offset, $length, $callback->($retval)
339 363
340Tries to copy C<$length> bytes from C<$in_fh> to C<$out_fh>. It starts 364Tries to copy C<$length> bytes from C<$in_fh> to C<$out_fh>. It starts
341reading at byte offset C<$in_offset>, and starts writing at the current 365reading at byte offset C<$in_offset>, and starts writing at the current
… …
355C<$in_fh> than are written, and there is no way to find out how many 379C<$in_fh> than are written, and there is no way to find out how many
356bytes have been read from C<aio_sendfile> alone, as C<aio_sendfile> only 380bytes have been read from C<aio_sendfile> alone, as C<aio_sendfile> only
357provides the number of bytes written to C<$out_fh>. Only if the result 381provides the number of bytes written to C<$out_fh>. Only if the result
358value equals C<$length> one can assume that C<$length> bytes have been 382value equals C<$length> one can assume that C<$length> bytes have been
359read. 383read.
384
360 385
361=item aio_readahead $fh,$offset,$length, $callback->($retval) 386=item aio_readahead $fh,$offset,$length, $callback->($retval)
362 387
363C<aio_readahead> populates the page cache with data from a file so that 388C<aio_readahead> populates the page cache with data from a file so that
364subsequent reads from that file will not block on disk I/O. The C<$offset> 389subsequent reads from that file will not block on disk I/O. The C<$offset>
… …
370file. The current file offset of the file is left unchanged. 395file. The current file offset of the file is left unchanged.
371 396
372If that syscall doesn't exist (likely if your OS isn't Linux) it will be 397If that syscall doesn't exist (likely if your OS isn't Linux) it will be
373emulated by simply reading the data, which would have a similar effect. 398emulated by simply reading the data, which would have a similar effect.
374 399
400
375=item aio_stat $fh_or_path, $callback->($status) 401=item aio_stat $fh_or_path, $callback->($status)
376 402
377=item aio_lstat $fh, $callback->($status) 403=item aio_lstat $fh, $callback->($status)
378 404
379Works like perl's C<stat> or C<lstat> in void context. The callback will 405Works like perl's C<stat> or C<lstat> in void context. The callback will
… …
392 aio_stat "/etc/passwd", sub { 418 aio_stat "/etc/passwd", sub {
393 $_[0] and die "stat failed: $!"; 419 $_[0] and die "stat failed: $!";
394 print "size is ", -s _, "\n"; 420 print "size is ", -s _, "\n";
395 }; 421 };
396 422
423
424=item aio_utime $fh_or_path, $atime, $mtime, $callback->($status)
425
426Works like perl's C<utime> function (including the special case of $atime
427and $mtime being undef). Fractional times are supported if the underlying
428syscalls support them.
429
430When called with a pathname, uses utimes(2) if available, otherwise
431utime(2). If called on a file descriptor, uses futimes(2) if available,
432otherwise returns ENOSYS, so this is not portable.
433
434Examples:
435
436 # set atime and mtime to current time (basically touch(1)):
437 aio_utime "path", undef, undef;
438 # set atime to current time and mtime to beginning of the epoch:
439 aio_utime "path", time, undef; # undef==0
440
441
442=item aio_chown $fh_or_path, $uid, $gid, $callback->($status)
443
444Works like perl's C<chown> function, except that C<undef> for either $uid
445or $gid is being interpreted as "do not change" (but -1 can also be used).
446
447Examples:
448
449 # same as "chown root path" in the shell:
450 aio_chown "path", 0, -1;
451 # same as above:
452 aio_chown "path", 0, undef;
453
454
455=item aio_truncate $fh_or_path, $offset, $callback->($status)
456
457Works like truncate(2) or ftruncate(2).
458
459
460=item aio_chmod $fh_or_path, $mode, $callback->($status)
461
462Works like perl's C<chmod> function.
463
464
397=item aio_unlink $pathname, $callback->($status) 465=item aio_unlink $pathname, $callback->($status)
398 466
399Asynchronously unlink (delete) a file and call the callback with the 467Asynchronously unlink (delete) a file and call the callback with the
400result code. 468result code.
401 469
470
402=item aio_mknod $path, $mode, $dev, $callback->($status) 471=item aio_mknod $path, $mode, $dev, $callback->($status)
403 472
404[EXPERIMENTAL] 473[EXPERIMENTAL]
405 474
406Asynchronously create a device node (or fifo). See mknod(2). 475Asynchronously create a device node (or fifo). See mknod(2).
407 476
408The only (POSIX-) portable way of calling this function is: 477The only (POSIX-) portable way of calling this function is:
409 478
410 aio_mknod $path, IO::AIO::S_IFIFO | $mode, 0, sub { ... 479 aio_mknod $path, IO::AIO::S_IFIFO | $mode, 0, sub { ...
480
411 481
412=item aio_link $srcpath, $dstpath, $callback->($status) 482=item aio_link $srcpath, $dstpath, $callback->($status)
413 483
414Asynchronously create a new link to the existing object at C<$srcpath> at 484Asynchronously create a new link to the existing object at C<$srcpath> at
415the path C<$dstpath> and call the callback with the result code. 485the path C<$dstpath> and call the callback with the result code.
416 486
487
417=item aio_symlink $srcpath, $dstpath, $callback->($status) 488=item aio_symlink $srcpath, $dstpath, $callback->($status)
418 489
419Asynchronously create a new symbolic link to the existing object at C<$srcpath> at 490Asynchronously create a new symbolic link to the existing object at C<$srcpath> at
420the path C<$dstpath> and call the callback with the result code. 491the path C<$dstpath> and call the callback with the result code.
421 492
493
494=item aio_readlink $path, $callback->($link)
495
496Asynchronously read the symlink specified by C<$path> and pass it to
497the callback. If an error occurs, nothing or undef gets passed to the
498callback.
499
500
422=item aio_rename $srcpath, $dstpath, $callback->($status) 501=item aio_rename $srcpath, $dstpath, $callback->($status)
423 502
424Asynchronously rename the object at C<$srcpath> to C<$dstpath>, just as 503Asynchronously rename the object at C<$srcpath> to C<$dstpath>, just as
425rename(2) and call the callback with the result code. 504rename(2) and call the callback with the result code.
426 505
506
507=item aio_mkdir $pathname, $mode, $callback->($status)
508
509Asynchronously mkdir (create) a directory and call the callback with
510the result code. C<$mode> will be modified by the umask at the time the
511request is executed, so do not change your umask.
512
513
427=item aio_rmdir $pathname, $callback->($status) 514=item aio_rmdir $pathname, $callback->($status)
428 515
429Asynchronously rmdir (delete) a directory and call the callback with the 516Asynchronously rmdir (delete) a directory and call the callback with the
430result code. 517result code.
518
431 519
432=item aio_readdir $pathname, $callback->($entries) 520=item aio_readdir $pathname, $callback->($entries)
433 521
434Unlike the POSIX call of the same name, C<aio_readdir> reads an entire 522Unlike the POSIX call of the same name, C<aio_readdir> reads an entire
435directory (i.e. opendir + readdir + closedir). The entries will not be 523directory (i.e. opendir + readdir + closedir). The entries will not be
436sorted, and will B<NOT> include the C<.> and C<..> entries. 524sorted, and will B<NOT> include the C<.> and C<..> entries.
437 525
438The callback a single argument which is either C<undef> or an array-ref 526The callback a single argument which is either C<undef> or an array-ref
439with the filenames. 527with the filenames.
528
529
530=item aio_load $path, $data, $callback->($status)
531
532This is a composite request that tries to fully load the given file into
533memory. Status is the same as with aio_read.
534
535=cut
536
537sub aio_load($$;$) {
538 aio_block {
539 my ($path, undef, $cb) = @_;
540 my $data = \$_[1];
541
542 my $pri = aioreq_pri;
543 my $grp = aio_group $cb;
544
545 aioreq_pri $pri;
546 add $grp aio_open $path, O_RDONLY, 0, sub {
547 my $fh = shift
548 or return $grp->result (-1);
549
550 aioreq_pri $pri;
551 add $grp aio_read $fh, 0, (-s $fh), $$data, 0, sub {
552 $grp->result ($_[0]);
553 };
554 };
555
556 $grp
557 }
558}
440 559
441=item aio_copy $srcpath, $dstpath, $callback->($status) 560=item aio_copy $srcpath, $dstpath, $callback->($status)
442 561
443Try to copy the I<file> (directories not supported as either source or 562Try to copy the I<file> (directories not supported as either source or
444destination) from C<$srcpath> to C<$dstpath> and call the callback with 563destination) from C<$srcpath> to C<$dstpath> and call the callback with
… …
454errors are being ignored. 573errors are being ignored.
455 574
456=cut 575=cut
457 576
458sub aio_copy($$;$) { 577sub aio_copy($$;$) {
578 aio_block {
459 my ($src, $dst, $cb) = @_; 579 my ($src, $dst, $cb) = @_;
460 580
461 my $pri = aioreq_pri; 581 my $pri = aioreq_pri;
462 my $grp = aio_group $cb; 582 my $grp = aio_group $cb;
463 583
464 aioreq_pri $pri; 584 aioreq_pri $pri;
465 add $grp aio_open $src, O_RDONLY, 0, sub { 585 add $grp aio_open $src, O_RDONLY, 0, sub {
466 if (my $src_fh = $_[0]) { 586 if (my $src_fh = $_[0]) {
467 my @stat = stat $src_fh; 587 my @stat = stat $src_fh;
468 588
469 aioreq_pri $pri; 589 aioreq_pri $pri;
470 add $grp aio_open $dst, O_CREAT | O_WRONLY | O_TRUNC, 0200, sub { 590 add $grp aio_open $dst, O_CREAT | O_WRONLY | O_TRUNC, 0200, sub {
471 if (my $dst_fh = $_[0]) { 591 if (my $dst_fh = $_[0]) {
472 aioreq_pri $pri; 592 aioreq_pri $pri;
473 add $grp aio_sendfile $dst_fh, $src_fh, 0, $stat[7], sub { 593 add $grp aio_sendfile $dst_fh, $src_fh, 0, $stat[7], sub {
474 if ($_[0] == $stat[7]) { 594 if ($_[0] == $stat[7]) {
475 $grp->result (0); 595 $grp->result (0);
476 close $src_fh; 596 close $src_fh;
477 597
478 # those should not normally block. should. should. 598 # those should not normally block. should. should.
479 utime $stat[8], $stat[9], $dst; 599 utime $stat[8], $stat[9], $dst;
480 chmod $stat[2] & 07777, $dst_fh; 600 chmod $stat[2] & 07777, $dst_fh;
481 chown $stat[4], $stat[5], $dst_fh; 601 chown $stat[4], $stat[5], $dst_fh;
482 close $dst_fh; 602 close $dst_fh;
483 } else { 603 } else {
484 $grp->result (-1); 604 $grp->result (-1);
485 close $src_fh; 605 close $src_fh;
486 close $dst_fh; 606 close $dst_fh;
487 607
488 aioreq $pri; 608 aioreq $pri;
489 add $grp aio_unlink $dst; 609 add $grp aio_unlink $dst;
610 }
490 } 611 };
612 } else {
613 $grp->result (-1);
491 }; 614 }
492 } else {
493 $grp->result (-1);
494 } 615 },
616
617 } else {
618 $grp->result (-1);
495 }, 619 }
496
497 } else {
498 $grp->result (-1);
499 } 620 };
621
622 $grp
500 }; 623 }
501
502 $grp
503} 624}
504 625
505=item aio_move $srcpath, $dstpath, $callback->($status) 626=item aio_move $srcpath, $dstpath, $callback->($status)
506 627
507Try to move the I<file> (directories not supported as either source or 628Try to move the I<file> (directories not supported as either source or
… …
513that is successful, unlinking the C<$srcpath>. 634that is successful, unlinking the C<$srcpath>.
514 635
515=cut 636=cut
516 637
517sub aio_move($$;$) { 638sub aio_move($$;$) {
639 aio_block {
518 my ($src, $dst, $cb) = @_; 640 my ($src, $dst, $cb) = @_;
519 641
520 my $pri = aioreq_pri; 642 my $pri = aioreq_pri;
521 my $grp = aio_group $cb; 643 my $grp = aio_group $cb;
522 644
523 aioreq_pri $pri; 645 aioreq_pri $pri;
524 add $grp aio_rename $src, $dst, sub { 646 add $grp aio_rename $src, $dst, sub {
525 if ($_[0] && $! == EXDEV) { 647 if ($_[0] && $! == EXDEV) {
526 aioreq_pri $pri; 648 aioreq_pri $pri;
527 add $grp aio_copy $src, $dst, sub { 649 add $grp aio_copy $src, $dst, sub {
650 $grp->result ($_[0]);
651
652 if (!$_[0]) {
653 aioreq_pri $pri;
654 add $grp aio_unlink $src;
655 }
656 };
657 } else {
528 $grp->result ($_[0]); 658 $grp->result ($_[0]);
529
530 if (!$_[0]) {
531 aioreq_pri $pri;
532 add $grp aio_unlink $src;
533 }
534 }; 659 }
535 } else {
536 $grp->result ($_[0]);
537 } 660 };
661
662 $grp
538 }; 663 }
539
540 $grp
541} 664}
542 665
543=item aio_scandir $path, $maxreq, $callback->($dirs, $nondirs) 666=item aio_scandir $path, $maxreq, $callback->($dirs, $nondirs)
544 667
545Scans a directory (similar to C<aio_readdir>) but additionally tries to 668Scans a directory (similar to C<aio_readdir>) but additionally tries to
… …
592as those tend to return 0 or 1 as link counts, which disables the 715as those tend to return 0 or 1 as link counts, which disables the
593directory counting heuristic. 716directory counting heuristic.
594 717
595=cut 718=cut
596 719
597sub aio_scandir($$$) { 720sub aio_scandir($$;$) {
721 aio_block {
598 my ($path, $maxreq, $cb) = @_; 722 my ($path, $maxreq, $cb) = @_;
599 723
600 my $pri = aioreq_pri; 724 my $pri = aioreq_pri;
601 725
602 my $grp = aio_group $cb; 726 my $grp = aio_group $cb;
603 727
604 $maxreq = 4 if $maxreq <= 0; 728 $maxreq = 4 if $maxreq <= 0;
605 729
606 # stat once 730 # stat once
607 aioreq_pri $pri;
608 add $grp aio_stat $path, sub {
609 return $grp->result () if $_[0];
610 my $now = time;
611 my $hash1 = join ":", (stat _)[0,1,3,7,9];
612
613 # read the directory entries
614 aioreq_pri $pri; 731 aioreq_pri $pri;
615 add $grp aio_readdir $path, sub { 732 add $grp aio_stat $path, sub {
616 my $entries = shift
617 or return $grp->result (); 733 return $grp->result () if $_[0];
734 my $now = time;
735 my $hash1 = join ":", (stat _)[0,1,3,7,9];
618 736
619 # stat the dir another time 737 # read the directory entries
620 aioreq_pri $pri; 738 aioreq_pri $pri;
739 add $grp aio_readdir $path, sub {
740 my $entries = shift
741 or return $grp->result ();
742
743 # stat the dir another time
744 aioreq_pri $pri;
621 add $grp aio_stat $path, sub { 745 add $grp aio_stat $path, sub {
622 my $hash2 = join ":", (stat _)[0,1,3,7,9]; 746 my $hash2 = join ":", (stat _)[0,1,3,7,9];
623 747
624 my $ndirs; 748 my $ndirs;
625 749
626 # take the slow route if anything looks fishy 750 # take the slow route if anything looks fishy
627 if ($hash1 ne $hash2 or (stat _)[9] == $now) { 751 if ($hash1 ne $hash2 or (stat _)[9] == $now) {
628 $ndirs = -1; 752 $ndirs = -1;
629 } else { 753 } else {
630 # if nlink == 2, we are finished 754 # if nlink == 2, we are finished
631 # on non-posix-fs's, we rely on nlink < 2 755 # on non-posix-fs's, we rely on nlink < 2
632 $ndirs = (stat _)[3] - 2 756 $ndirs = (stat _)[3] - 2
633 or return $grp->result ([], $entries); 757 or return $grp->result ([], $entries);
634 } 758 }
635 759
636 # sort into likely dirs and likely nondirs 760 # sort into likely dirs and likely nondirs
637 # dirs == files without ".", short entries first 761 # dirs == files without ".", short entries first
638 $entries = [map $_->[0], 762 $entries = [map $_->[0],
639 sort { $b->[1] cmp $a->[1] } 763 sort { $b->[1] cmp $a->[1] }
640 map [$_, sprintf "%s%04d", (/.\./ ? "1" : "0"), length], 764 map [$_, sprintf "%s%04d", (/.\./ ? "1" : "0"), length],
641 @$entries]; 765 @$entries];
642 766
643 my (@dirs, @nondirs); 767 my (@dirs, @nondirs);
644 768
645 my $statgrp = add $grp aio_group sub { 769 my $statgrp = add $grp aio_group sub {
646 $grp->result (\@dirs, \@nondirs); 770 $grp->result (\@dirs, \@nondirs);
647 }; 771 };
648 772
649 limit $statgrp $maxreq; 773 limit $statgrp $maxreq;
650 feed $statgrp sub { 774 feed $statgrp sub {
651 return unless @$entries; 775 return unless @$entries;
652 my $entry = pop @$entries; 776 my $entry = pop @$entries;
653 777
654 aioreq_pri $pri; 778 aioreq_pri $pri;
655 add $statgrp aio_stat "$path/$entry/.", sub { 779 add $statgrp aio_stat "$path/$entry/.", sub {
656 if ($_[0] < 0) { 780 if ($_[0] < 0) {
657 push @nondirs, $entry; 781 push @nondirs, $entry;
658 } else { 782 } else {
659 # need to check for real directory 783 # need to check for real directory
660 aioreq_pri $pri; 784 aioreq_pri $pri;
661 add $statgrp aio_lstat "$path/$entry", sub { 785 add $statgrp aio_lstat "$path/$entry", sub {
662 if (-d _) { 786 if (-d _) {
663 push @dirs, $entry; 787 push @dirs, $entry;
664 788
665 unless (--$ndirs) { 789 unless (--$ndirs) {
666 push @nondirs, @$entries; 790 push @nondirs, @$entries;
667 feed $statgrp; 791 feed $statgrp;
792 }
793 } else {
794 push @nondirs, $entry;
668 } 795 }
669 } else {
670 push @nondirs, $entry;
671 } 796 }
672 } 797 }
673 } 798 };
674 }; 799 };
675 }; 800 };
676 }; 801 };
677 }; 802 };
803
804 $grp
678 }; 805 }
806}
679 807
808=item aio_rmtree $path, $callback->($status)
809
810Delete a directory tree starting (and including) C<$path>, return the
811status of the final C<rmdir> only. This is a composite request that
812uses C<aio_scandir> to recurse into and rmdir directories, and unlink
813everything else.
814
815=cut
816
817sub aio_rmtree;
818sub aio_rmtree($;$) {
819 aio_block {
820 my ($path, $cb) = @_;
821
822 my $pri = aioreq_pri;
823 my $grp = aio_group $cb;
824
825 aioreq_pri $pri;
826 add $grp aio_scandir $path, 0, sub {
827 my ($dirs, $nondirs) = @_;
828
829 my $dirgrp = aio_group sub {
830 add $grp aio_rmdir $path, sub {
831 $grp->result ($_[0]);
832 };
833 };
834
835 (aioreq_pri $pri), add $dirgrp aio_rmtree "$path/$_" for @$dirs;
836 (aioreq_pri $pri), add $dirgrp aio_unlink "$path/$_" for @$nondirs;
837
838 add $grp $dirgrp;
839 };
840
680 $grp 841 $grp
842 }
681} 843}
682 844
683=item aio_fsync $fh, $callback->($status) 845=item aio_fsync $fh, $callback->($status)
684 846
685Asynchronously call fsync on the given filehandle and call the callback 847Asynchronously call fsync on the given filehandle and call the callback
… …
964 poll => 'r', nice => 1, 1126 poll => 'r', nice => 1,
965 cb => &IO::AIO::poll_cb); 1127 cb => &IO::AIO::poll_cb);
966 1128
967=item IO::AIO::poll_wait 1129=item IO::AIO::poll_wait
968 1130
1131If there are any outstanding requests and none of them in the result
969Wait till the result filehandle becomes ready for reading (simply does a 1132phase, wait till the result filehandle becomes ready for reading (simply
970C<select> on the filehandle. This is useful if you want to synchronously 1133does a C<select> on the filehandle. This is useful if you want to
971wait for some requests to finish). 1134synchronously wait for some requests to finish).
972 1135
973See C<nreqs> for an example. 1136See C<nreqs> for an example.
974 1137
975=item IO::AIO::poll 1138=item IO::AIO::poll
976 1139
977Waits until some requests have been handled. 1140Waits until some requests have been handled.
978 1141
1142Returns the number of requests processed, but is otherwise strictly
979Strictly equivalent to: 1143equivalent to:
980 1144
981 IO::AIO::poll_wait, IO::AIO::poll_cb 1145 IO::AIO::poll_wait, IO::AIO::poll_cb
982 if IO::AIO::nreqs;
983 1146
984=item IO::AIO::flush 1147=item IO::AIO::flush
985 1148
986Wait till all outstanding AIO requests have been handled. 1149Wait till all outstanding AIO requests have been handled.
987 1150
988Strictly equivalent to: 1151Strictly equivalent to:
989 1152
990 IO::AIO::poll_wait, IO::AIO::poll_cb 1153 IO::AIO::poll_wait, IO::AIO::poll_cb
991 while IO::AIO::nreqs; 1154 while IO::AIO::nreqs;
992 1155
1156=back
1157
993=head3 CONTROLLING THE NUMBER OF THREADS 1158=head3 CONTROLLING THE NUMBER OF THREADS
1159
1160=over
994 1161
995=item IO::AIO::min_parallel $nthreads 1162=item IO::AIO::min_parallel $nthreads
996 1163
997Set the minimum number of AIO threads to C<$nthreads>. The current 1164Set the minimum number of AIO threads to C<$nthreads>. The current
998default is C<8>, which means eight asynchronous operations can execute 1165default is C<8>, which means eight asynchronous operations can execute
… …
1046This is a very bad function to use in interactive programs because it 1213This is a very bad function to use in interactive programs because it
1047blocks, and a bad way to reduce concurrency because it is inexact: Better 1214blocks, and a bad way to reduce concurrency because it is inexact: Better
1048use an C<aio_group> together with a feed callback. 1215use an C<aio_group> together with a feed callback.
1049 1216
1050Sets the maximum number of outstanding requests to C<$nreqs>. If you 1217Sets the maximum number of outstanding requests to C<$nreqs>. If you
1051to queue up more than this number of requests, the next call to the 1218do queue up more than this number of requests, the next call to the
1052C<poll_cb> (and C<poll_some> and other functions calling C<poll_cb>) 1219C<poll_cb> (and C<poll_some> and other functions calling C<poll_cb>)
1053function will block until the limit is no longer exceeded. 1220function will block until the limit is no longer exceeded.
1054 1221
1055The default value is very large, so there is no practical limit on the 1222The default value is very large, so there is no practical limit on the
1056number of outstanding requests. 1223number of outstanding requests.
1057 1224
1058You can still queue as many requests as you want. Therefore, 1225You can still queue as many requests as you want. Therefore,
1059C<max_oustsanding> is mainly useful in simple scripts (with low values) or 1226C<max_oustsanding> is mainly useful in simple scripts (with low values) or
1060as a stop gap to shield against fatal memory overflow (with large values). 1227as a stop gap to shield against fatal memory overflow (with large values).
1061 1228
1229=back
1230
1062=head3 STATISTICAL INFORMATION 1231=head3 STATISTICAL INFORMATION
1232
1233=over
1063 1234
1064=item IO::AIO::nreqs 1235=item IO::AIO::nreqs
1065 1236
1066Returns the number of requests currently in the ready, execute or pending 1237Returns the number of requests currently in the ready, execute or pending
1067states (i.e. for which their callback has not been invoked yet). 1238states (i.e. for which their callback has not been invoked yet).
… …
1083 1254
1084=back 1255=back
1085 1256
1086=cut 1257=cut
1087 1258
1088# support function to convert a fd into a perl filehandle
1089sub _fd2fh {
1090 return undef if $_[0] < 0;
1091
1092 # try to generate nice filehandles
1093 my $sym = "IO::AIO::fd#$_[0]";
1094 local *$sym;
1095
1096 open *$sym, "+<&=$_[0]" # usually works under any unix
1097 or open *$sym, "<&=$_[0]" # cygwin needs this
1098 or open *$sym, ">&=$_[0]" # or this
1099 or return undef;
1100
1101 *$sym
1102}
1103
1104min_parallel 8; 1259min_parallel 8;
1105 1260
1106END { 1261END { flush }
1107 min_parallel 1;
1108 flush;
1109};
1110 1262
11111; 12631;
1112 1264
1113=head2 FORK BEHAVIOUR 1265=head2 FORK BEHAVIOUR
1114 1266
… …
1134bytes of memory. In addition, stat requests need a stat buffer (possibly 1286bytes of memory. In addition, stat requests need a stat buffer (possibly
1135a few hundred bytes), readdir requires a result buffer and so on. Perl 1287a few hundred bytes), readdir requires a result buffer and so on. Perl
1136scalars and other data passed into aio requests will also be locked and 1288scalars and other data passed into aio requests will also be locked and
1137will consume memory till the request has entered the done state. 1289will consume memory till the request has entered the done state.
1138 1290
1139This is now awfully much, so queuing lots of requests is not usually a 1291This is not awfully much, so queuing lots of requests is not usually a
1140problem. 1292problem.
1141 1293
1142Per-thread usage: 1294Per-thread usage:
1143 1295
1144In the execution phase, some aio requests require more memory for 1296In the execution phase, some aio requests require more memory for

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines