ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/IO-AIO/AIO.pm
(Generate patch)

Comparing IO-AIO/AIO.pm (file contents):
Revision 1.90 by root, Mon Oct 30 23:30:00 2006 UTC vs.
Revision 1.116 by root, Wed Oct 3 21:27:51 2007 UTC

… …
5=head1 SYNOPSIS 5=head1 SYNOPSIS
6 6
7 use IO::AIO; 7 use IO::AIO;
8 8
9 aio_open "/etc/passwd", O_RDONLY, 0, sub { 9 aio_open "/etc/passwd", O_RDONLY, 0, sub {
10 my ($fh) = @_; 10 my $fh = shift
11 or die "/etc/passwd: $!";
11 ... 12 ...
12 }; 13 };
13 14
14 aio_unlink "/tmp/file", sub { }; 15 aio_unlink "/tmp/file", sub { };
15 16
… …
61etc.), but can also be used to easily do operations in parallel that are 62etc.), but can also be used to easily do operations in parallel that are
62normally done sequentially, e.g. stat'ing many files, which is much faster 63normally done sequentially, e.g. stat'ing many files, which is much faster
63on a RAID volume or over NFS when you do a number of stat operations 64on a RAID volume or over NFS when you do a number of stat operations
64concurrently. 65concurrently.
65 66
66While most of this works on all types of file descriptors (for example 67While most of this works on all types of file descriptors (for
67sockets), using these functions on file descriptors that support 68example sockets), using these functions on file descriptors that
68nonblocking operation (again, sockets, pipes etc.) is very inefficient or 69support nonblocking operation (again, sockets, pipes etc.) is very
69might not work (aio_read fails on sockets/pipes/fifos). Use an event loop 70inefficient. Use an event loop for that (such as the L<Event|Event>
70for that (such as the L<Event|Event> module): IO::AIO will naturally fit 71module): IO::AIO will naturally fit into such an event loop itself.
71into such an event loop itself.
72 72
73In this version, a number of threads are started that execute your 73In this version, a number of threads are started that execute your
74requests and signal their completion. You don't need thread support 74requests and signal their completion. You don't need thread support
75in perl, and the threads created by this module will not be visible 75in perl, and the threads created by this module will not be visible
76to perl. In the future, this module might make use of the native aio 76to perl. In the future, this module might make use of the native aio
… …
78not well-supported or restricted (GNU/Linux doesn't allow them on normal 78not well-supported or restricted (GNU/Linux doesn't allow them on normal
79files currently, for example), and they would only support aio_read and 79files currently, for example), and they would only support aio_read and
80aio_write, so the remaining functionality would have to be implemented 80aio_write, so the remaining functionality would have to be implemented
81using threads anyway. 81using threads anyway.
82 82
83Although the module will work with in the presence of other (Perl-) 83Although the module will work in the presence of other (Perl-) threads,
84threads, it is currently not reentrant in any way, so use appropriate 84it is currently not reentrant in any way, so use appropriate locking
85locking yourself, always call C<poll_cb> from within the same thread, or 85yourself, always call C<poll_cb> from within the same thread, or never
86never call C<poll_cb> (or other C<aio_> functions) recursively. 86call C<poll_cb> (or other C<aio_> functions) recursively.
87 87
88=head2 EXAMPLE 88=head2 EXAMPLE
89 89
90This is a simple example that uses the Event module and loads 90This is a simple example that uses the Event module and loads
91F</etc/passwd> asynchronously: 91F</etc/passwd> asynchronously:
… …
99 poll => 'r', 99 poll => 'r',
100 cb => \&IO::AIO::poll_cb); 100 cb => \&IO::AIO::poll_cb);
101 101
102 # queue the request to open /etc/passwd 102 # queue the request to open /etc/passwd
103 aio_open "/etc/passwd", O_RDONLY, 0, sub { 103 aio_open "/etc/passwd", O_RDONLY, 0, sub {
104 my $fh = $_[0] 104 my $fh = shift
105 or die "error while opening: $!"; 105 or die "error while opening: $!";
106 106
107 # stat'ing filehandles is generally non-blocking 107 # stat'ing filehandles is generally non-blocking
108 my $size = -s $fh; 108 my $size = -s $fh;
109 109
… …
189use strict 'vars'; 189use strict 'vars';
190 190
191use base 'Exporter'; 191use base 'Exporter';
192 192
193BEGIN { 193BEGIN {
194 our $VERSION = '2.1'; 194 our $VERSION = '2.5';
195 195
196 our @AIO_REQ = qw(aio_sendfile aio_read aio_write aio_open aio_close aio_stat 196 our @AIO_REQ = qw(aio_sendfile aio_read aio_write aio_open aio_close aio_stat
197 aio_lstat aio_unlink aio_rmdir aio_readdir aio_scandir aio_symlink 197 aio_lstat aio_unlink aio_rmdir aio_readdir aio_scandir aio_symlink
198 aio_readlink aio_fsync aio_fdatasync aio_readahead aio_rename aio_link 198 aio_readlink aio_fsync aio_fdatasync aio_readahead aio_rename aio_link
199 aio_move aio_copy aio_group aio_nop aio_mknod); 199 aio_move aio_copy aio_group aio_nop aio_mknod aio_load aio_rmtree aio_mkdir
200 aio_chown aio_chmod aio_utime aio_truncate);
200 our @EXPORT = (@AIO_REQ, qw(aioreq_pri aioreq_nice)); 201 our @EXPORT = (@AIO_REQ, qw(aioreq_pri aioreq_nice aio_block));
201 our @EXPORT_OK = qw(poll_fileno poll_cb poll_wait flush 202 our @EXPORT_OK = qw(poll_fileno poll_cb poll_wait flush
202 min_parallel max_parallel max_idle 203 min_parallel max_parallel max_idle
203 nreqs nready npending nthreads 204 nreqs nready npending nthreads
204 max_poll_time max_poll_reqs); 205 max_poll_time max_poll_reqs);
205 206
… …
270 aio_read $_[0], ..., sub { 271 aio_read $_[0], ..., sub {
271 ... 272 ...
272 }; 273 };
273 }; 274 };
274 275
276
275=item aioreq_nice $pri_adjust 277=item aioreq_nice $pri_adjust
276 278
277Similar to C<aioreq_pri>, but subtracts the given value from the current 279Similar to C<aioreq_pri>, but subtracts the given value from the current
278priority, so the effect is cumulative. 280priority, so the effect is cumulative.
281
279 282
280=item aio_open $pathname, $flags, $mode, $callback->($fh) 283=item aio_open $pathname, $flags, $mode, $callback->($fh)
281 284
282Asynchronously open or create a file and call the callback with a newly 285Asynchronously open or create a file and call the callback with a newly
283created filehandle for the file. 286created filehandle for the file.
… …
289list. They are the same as used by C<sysopen>. 292list. They are the same as used by C<sysopen>.
290 293
291Likewise, C<$mode> specifies the mode of the newly created file, if it 294Likewise, C<$mode> specifies the mode of the newly created file, if it
292didn't exist and C<O_CREAT> has been given, just like perl's C<sysopen>, 295didn't exist and C<O_CREAT> has been given, just like perl's C<sysopen>,
293except that it is mandatory (i.e. use C<0> if you don't create new files, 296except that it is mandatory (i.e. use C<0> if you don't create new files,
294and C<0666> or C<0777> if you do). 297and C<0666> or C<0777> if you do). Note that the C<$mode> will be modified
298by the umask in effect then the request is being executed, so better never
299change the umask.
295 300
296Example: 301Example:
297 302
298 aio_open "/etc/passwd", O_RDONLY, 0, sub { 303 aio_open "/etc/passwd", O_RDONLY, 0, sub {
299 if ($_[0]) { 304 if ($_[0]) {
… …
302 } else { 307 } else {
303 die "open failed: $!\n"; 308 die "open failed: $!\n";
304 } 309 }
305 }; 310 };
306 311
312
307=item aio_close $fh, $callback->($status) 313=item aio_close $fh, $callback->($status)
308 314
309Asynchronously close a file and call the callback with the result 315Asynchronously close a file and call the callback with the result
310code. I<WARNING:> although accepted, you should not pass in a perl 316code.
311filehandle here, as perl will likely close the file descriptor another
312time when the filehandle is destroyed. Normally, you can safely call perls
313C<close> or just let filehandles go out of scope.
314 317
315This is supposed to be a bug in the API, so that might change. It's 318Unlike the other functions operating on files, this function uses the
316therefore best to avoid this function. 319PerlIO layer to close the filehandle. The reason is that the PerlIO API
320insists on closing the underlying fd itself, no matter what, and doesn't
321allow modifications to the fd. Unfortunately, it is not clear that you can
322call PerlIO from different threads (actually, its quite clear that this
323won't work in some cases), so while it likely works perfectly with simple
324file handles (such as the ones created by C<aio_open>) it might fail in
325interesting ways for others.
326
327Having said that, aio_close tries to clean up the filehandle as much as
328possible before handing it to an io thread, and generally does work.
329
317 330
318=item aio_read $fh,$offset,$length, $data,$dataoffset, $callback->($retval) 331=item aio_read $fh,$offset,$length, $data,$dataoffset, $callback->($retval)
319 332
320=item aio_write $fh,$offset,$length, $data,$dataoffset, $callback->($retval) 333=item aio_write $fh,$offset,$length, $data,$dataoffset, $callback->($retval)
321 334
322Reads or writes C<length> bytes from the specified C<fh> and C<offset> 335Reads or writes C<$length> bytes from the specified C<$fh> and C<$offset>
323into the scalar given by C<data> and offset C<dataoffset> and calls the 336into the scalar given by C<$data> and offset C<$dataoffset> and calls the
324callback without the actual number of bytes read (or -1 on error, just 337callback without the actual number of bytes read (or -1 on error, just
325like the syscall). 338like the syscall).
326 339
340If C<$offset> is undefined, then the current file descriptor offset will
341be used (and updated), otherwise the file descriptor offset will not be
342changed by these calls.
343
344If C<$length> is undefined in C<aio_write>, use the remaining length of C<$data>.
345
346If C<$dataoffset> is less than zero, it will be counted from the end of
347C<$data>.
348
327The C<$data> scalar I<MUST NOT> be modified in any way while the request 349The C<$data> scalar I<MUST NOT> be modified in any way while the request
328is outstanding. Modifying it can result in segfaults or WW3 (if the 350is outstanding. Modifying it can result in segfaults or World War III (if
329necessary/optional hardware is installed). 351the necessary/optional hardware is installed).
330 352
331Example: Read 15 bytes at offset 7 into scalar C<$buffer>, starting at 353Example: Read 15 bytes at offset 7 into scalar C<$buffer>, starting at
332offset C<0> within the scalar: 354offset C<0> within the scalar:
333 355
334 aio_read $fh, 7, 15, $buffer, 0, sub { 356 aio_read $fh, 7, 15, $buffer, 0, sub {
335 $_[0] > 0 or die "read error: $!"; 357 $_[0] > 0 or die "read error: $!";
336 print "read $_[0] bytes: <$buffer>\n"; 358 print "read $_[0] bytes: <$buffer>\n";
337 }; 359 };
360
338 361
339=item aio_sendfile $out_fh, $in_fh, $in_offset, $length, $callback->($retval) 362=item aio_sendfile $out_fh, $in_fh, $in_offset, $length, $callback->($retval)
340 363
341Tries to copy C<$length> bytes from C<$in_fh> to C<$out_fh>. It starts 364Tries to copy C<$length> bytes from C<$in_fh> to C<$out_fh>. It starts
342reading at byte offset C<$in_offset>, and starts writing at the current 365reading at byte offset C<$in_offset>, and starts writing at the current
… …
356C<$in_fh> than are written, and there is no way to find out how many 379C<$in_fh> than are written, and there is no way to find out how many
357bytes have been read from C<aio_sendfile> alone, as C<aio_sendfile> only 380bytes have been read from C<aio_sendfile> alone, as C<aio_sendfile> only
358provides the number of bytes written to C<$out_fh>. Only if the result 381provides the number of bytes written to C<$out_fh>. Only if the result
359value equals C<$length> one can assume that C<$length> bytes have been 382value equals C<$length> one can assume that C<$length> bytes have been
360read. 383read.
384
361 385
362=item aio_readahead $fh,$offset,$length, $callback->($retval) 386=item aio_readahead $fh,$offset,$length, $callback->($retval)
363 387
364C<aio_readahead> populates the page cache with data from a file so that 388C<aio_readahead> populates the page cache with data from a file so that
365subsequent reads from that file will not block on disk I/O. The C<$offset> 389subsequent reads from that file will not block on disk I/O. The C<$offset>
… …
371file. The current file offset of the file is left unchanged. 395file. The current file offset of the file is left unchanged.
372 396
373If that syscall doesn't exist (likely if your OS isn't Linux) it will be 397If that syscall doesn't exist (likely if your OS isn't Linux) it will be
374emulated by simply reading the data, which would have a similar effect. 398emulated by simply reading the data, which would have a similar effect.
375 399
400
376=item aio_stat $fh_or_path, $callback->($status) 401=item aio_stat $fh_or_path, $callback->($status)
377 402
378=item aio_lstat $fh, $callback->($status) 403=item aio_lstat $fh, $callback->($status)
379 404
380Works like perl's C<stat> or C<lstat> in void context. The callback will 405Works like perl's C<stat> or C<lstat> in void context. The callback will
… …
393 aio_stat "/etc/passwd", sub { 418 aio_stat "/etc/passwd", sub {
394 $_[0] and die "stat failed: $!"; 419 $_[0] and die "stat failed: $!";
395 print "size is ", -s _, "\n"; 420 print "size is ", -s _, "\n";
396 }; 421 };
397 422
423
424=item aio_utime $fh_or_path, $atime, $mtime, $callback->($status)
425
426Works like perl's C<utime> function (including the special case of $atime
427and $mtime being undef). Fractional times are supported if the underlying
428syscalls support them.
429
430When called with a pathname, uses utimes(2) if available, otherwise
431utime(2). If called on a file descriptor, uses futimes(2) if available,
432otherwise returns ENOSYS, so this is not portable.
433
434Examples:
435
436 # set atime and mtime to current time (basically touch(1)):
437 aio_utime "path", undef, undef;
438 # set atime to current time and mtime to beginning of the epoch:
439 aio_utime "path", time, undef; # undef==0
440
441
442=item aio_chown $fh_or_path, $uid, $gid, $callback->($status)
443
444Works like perl's C<chown> function, except that C<undef> for either $uid
445or $gid is being interpreted as "do not change" (but -1 can also be used).
446
447Examples:
448
449 # same as "chown root path" in the shell:
450 aio_chown "path", 0, -1;
451 # same as above:
452 aio_chown "path", 0, undef;
453
454
455=item aio_truncate $fh_or_path, $offset, $callback->($status)
456
457Works like truncate(2) or ftruncate(2).
458
459
460=item aio_chmod $fh_or_path, $mode, $callback->($status)
461
462Works like perl's C<chmod> function.
463
464
398=item aio_unlink $pathname, $callback->($status) 465=item aio_unlink $pathname, $callback->($status)
399 466
400Asynchronously unlink (delete) a file and call the callback with the 467Asynchronously unlink (delete) a file and call the callback with the
401result code. 468result code.
402 469
470
403=item aio_mknod $path, $mode, $dev, $callback->($status) 471=item aio_mknod $path, $mode, $dev, $callback->($status)
404 472
405[EXPERIMENTAL] 473[EXPERIMENTAL]
406 474
407Asynchronously create a device node (or fifo). See mknod(2). 475Asynchronously create a device node (or fifo). See mknod(2).
408 476
409The only (POSIX-) portable way of calling this function is: 477The only (POSIX-) portable way of calling this function is:
410 478
411 aio_mknod $path, IO::AIO::S_IFIFO | $mode, 0, sub { ... 479 aio_mknod $path, IO::AIO::S_IFIFO | $mode, 0, sub { ...
480
412 481
413=item aio_link $srcpath, $dstpath, $callback->($status) 482=item aio_link $srcpath, $dstpath, $callback->($status)
414 483
415Asynchronously create a new link to the existing object at C<$srcpath> at 484Asynchronously create a new link to the existing object at C<$srcpath> at
416the path C<$dstpath> and call the callback with the result code. 485the path C<$dstpath> and call the callback with the result code.
417 486
487
418=item aio_symlink $srcpath, $dstpath, $callback->($status) 488=item aio_symlink $srcpath, $dstpath, $callback->($status)
419 489
420Asynchronously create a new symbolic link to the existing object at C<$srcpath> at 490Asynchronously create a new symbolic link to the existing object at C<$srcpath> at
421the path C<$dstpath> and call the callback with the result code. 491the path C<$dstpath> and call the callback with the result code.
492
422 493
423=item aio_readlink $path, $callback->($link) 494=item aio_readlink $path, $callback->($link)
424 495
425Asynchronously read the symlink specified by C<$path> and pass it to 496Asynchronously read the symlink specified by C<$path> and pass it to
426the callback. If an error occurs, nothing or undef gets passed to the 497the callback. If an error occurs, nothing or undef gets passed to the
427callback. 498callback.
428 499
500
429=item aio_rename $srcpath, $dstpath, $callback->($status) 501=item aio_rename $srcpath, $dstpath, $callback->($status)
430 502
431Asynchronously rename the object at C<$srcpath> to C<$dstpath>, just as 503Asynchronously rename the object at C<$srcpath> to C<$dstpath>, just as
432rename(2) and call the callback with the result code. 504rename(2) and call the callback with the result code.
433 505
506
507=item aio_mkdir $pathname, $mode, $callback->($status)
508
509Asynchronously mkdir (create) a directory and call the callback with
510the result code. C<$mode> will be modified by the umask at the time the
511request is executed, so do not change your umask.
512
513
434=item aio_rmdir $pathname, $callback->($status) 514=item aio_rmdir $pathname, $callback->($status)
435 515
436Asynchronously rmdir (delete) a directory and call the callback with the 516Asynchronously rmdir (delete) a directory and call the callback with the
437result code. 517result code.
518
438 519
439=item aio_readdir $pathname, $callback->($entries) 520=item aio_readdir $pathname, $callback->($entries)
440 521
441Unlike the POSIX call of the same name, C<aio_readdir> reads an entire 522Unlike the POSIX call of the same name, C<aio_readdir> reads an entire
442directory (i.e. opendir + readdir + closedir). The entries will not be 523directory (i.e. opendir + readdir + closedir). The entries will not be
443sorted, and will B<NOT> include the C<.> and C<..> entries. 524sorted, and will B<NOT> include the C<.> and C<..> entries.
444 525
445The callback a single argument which is either C<undef> or an array-ref 526The callback a single argument which is either C<undef> or an array-ref
446with the filenames. 527with the filenames.
528
529
530=item aio_load $path, $data, $callback->($status)
531
532This is a composite request that tries to fully load the given file into
533memory. Status is the same as with aio_read.
534
535=cut
536
537sub aio_load($$;$) {
538 aio_block {
539 my ($path, undef, $cb) = @_;
540 my $data = \$_[1];
541
542 my $pri = aioreq_pri;
543 my $grp = aio_group $cb;
544
545 aioreq_pri $pri;
546 add $grp aio_open $path, O_RDONLY, 0, sub {
547 my $fh = shift
548 or return $grp->result (-1);
549
550 aioreq_pri $pri;
551 add $grp aio_read $fh, 0, (-s $fh), $$data, 0, sub {
552 $grp->result ($_[0]);
553 };
554 };
555
556 $grp
557 }
558}
447 559
448=item aio_copy $srcpath, $dstpath, $callback->($status) 560=item aio_copy $srcpath, $dstpath, $callback->($status)
449 561
450Try to copy the I<file> (directories not supported as either source or 562Try to copy the I<file> (directories not supported as either source or
451destination) from C<$srcpath> to C<$dstpath> and call the callback with 563destination) from C<$srcpath> to C<$dstpath> and call the callback with
… …
461errors are being ignored. 573errors are being ignored.
462 574
463=cut 575=cut
464 576
465sub aio_copy($$;$) { 577sub aio_copy($$;$) {
578 aio_block {
466 my ($src, $dst, $cb) = @_; 579 my ($src, $dst, $cb) = @_;
467 580
468 my $pri = aioreq_pri; 581 my $pri = aioreq_pri;
469 my $grp = aio_group $cb; 582 my $grp = aio_group $cb;
470 583
471 aioreq_pri $pri; 584 aioreq_pri $pri;
472 add $grp aio_open $src, O_RDONLY, 0, sub { 585 add $grp aio_open $src, O_RDONLY, 0, sub {
473 if (my $src_fh = $_[0]) { 586 if (my $src_fh = $_[0]) {
474 my @stat = stat $src_fh; 587 my @stat = stat $src_fh;
475 588
476 aioreq_pri $pri; 589 aioreq_pri $pri;
477 add $grp aio_open $dst, O_CREAT | O_WRONLY | O_TRUNC, 0200, sub { 590 add $grp aio_open $dst, O_CREAT | O_WRONLY | O_TRUNC, 0200, sub {
478 if (my $dst_fh = $_[0]) { 591 if (my $dst_fh = $_[0]) {
479 aioreq_pri $pri; 592 aioreq_pri $pri;
480 add $grp aio_sendfile $dst_fh, $src_fh, 0, $stat[7], sub { 593 add $grp aio_sendfile $dst_fh, $src_fh, 0, $stat[7], sub {
481 if ($_[0] == $stat[7]) { 594 if ($_[0] == $stat[7]) {
482 $grp->result (0); 595 $grp->result (0);
483 close $src_fh; 596 close $src_fh;
484 597
485 # those should not normally block. should. should. 598 # those should not normally block. should. should.
486 utime $stat[8], $stat[9], $dst; 599 utime $stat[8], $stat[9], $dst;
487 chmod $stat[2] & 07777, $dst_fh; 600 chmod $stat[2] & 07777, $dst_fh;
488 chown $stat[4], $stat[5], $dst_fh; 601 chown $stat[4], $stat[5], $dst_fh;
489 close $dst_fh; 602 close $dst_fh;
490 } else { 603 } else {
491 $grp->result (-1); 604 $grp->result (-1);
492 close $src_fh; 605 close $src_fh;
493 close $dst_fh; 606 close $dst_fh;
494 607
495 aioreq $pri; 608 aioreq $pri;
496 add $grp aio_unlink $dst; 609 add $grp aio_unlink $dst;
610 }
497 } 611 };
612 } else {
613 $grp->result (-1);
498 }; 614 }
499 } else {
500 $grp->result (-1);
501 } 615 },
616
617 } else {
618 $grp->result (-1);
502 }, 619 }
503
504 } else {
505 $grp->result (-1);
506 } 620 };
621
622 $grp
507 }; 623 }
508
509 $grp
510} 624}
511 625
512=item aio_move $srcpath, $dstpath, $callback->($status) 626=item aio_move $srcpath, $dstpath, $callback->($status)
513 627
514Try to move the I<file> (directories not supported as either source or 628Try to move the I<file> (directories not supported as either source or
… …
520that is successful, unlinking the C<$srcpath>. 634that is successful, unlinking the C<$srcpath>.
521 635
522=cut 636=cut
523 637
524sub aio_move($$;$) { 638sub aio_move($$;$) {
639 aio_block {
525 my ($src, $dst, $cb) = @_; 640 my ($src, $dst, $cb) = @_;
526 641
527 my $pri = aioreq_pri; 642 my $pri = aioreq_pri;
528 my $grp = aio_group $cb; 643 my $grp = aio_group $cb;
529 644
530 aioreq_pri $pri; 645 aioreq_pri $pri;
531 add $grp aio_rename $src, $dst, sub { 646 add $grp aio_rename $src, $dst, sub {
532 if ($_[0] && $! == EXDEV) { 647 if ($_[0] && $! == EXDEV) {
533 aioreq_pri $pri; 648 aioreq_pri $pri;
534 add $grp aio_copy $src, $dst, sub { 649 add $grp aio_copy $src, $dst, sub {
650 $grp->result ($_[0]);
651
652 if (!$_[0]) {
653 aioreq_pri $pri;
654 add $grp aio_unlink $src;
655 }
656 };
657 } else {
535 $grp->result ($_[0]); 658 $grp->result ($_[0]);
536
537 if (!$_[0]) {
538 aioreq_pri $pri;
539 add $grp aio_unlink $src;
540 }
541 }; 659 }
542 } else {
543 $grp->result ($_[0]);
544 } 660 };
661
662 $grp
545 }; 663 }
546
547 $grp
548} 664}
549 665
550=item aio_scandir $path, $maxreq, $callback->($dirs, $nondirs) 666=item aio_scandir $path, $maxreq, $callback->($dirs, $nondirs)
551 667
552Scans a directory (similar to C<aio_readdir>) but additionally tries to 668Scans a directory (similar to C<aio_readdir>) but additionally tries to
… …
599as those tend to return 0 or 1 as link counts, which disables the 715as those tend to return 0 or 1 as link counts, which disables the
600directory counting heuristic. 716directory counting heuristic.
601 717
602=cut 718=cut
603 719
604sub aio_scandir($$$) { 720sub aio_scandir($$;$) {
721 aio_block {
605 my ($path, $maxreq, $cb) = @_; 722 my ($path, $maxreq, $cb) = @_;
606 723
607 my $pri = aioreq_pri; 724 my $pri = aioreq_pri;
608 725
609 my $grp = aio_group $cb; 726 my $grp = aio_group $cb;
610 727
611 $maxreq = 4 if $maxreq <= 0; 728 $maxreq = 4 if $maxreq <= 0;
612 729
613 # stat once 730 # stat once
614 aioreq_pri $pri;
615 add $grp aio_stat $path, sub {
616 return $grp->result () if $_[0];
617 my $now = time;
618 my $hash1 = join ":", (stat _)[0,1,3,7,9];
619
620 # read the directory entries
621 aioreq_pri $pri; 731 aioreq_pri $pri;
622 add $grp aio_readdir $path, sub { 732 add $grp aio_stat $path, sub {
623 my $entries = shift
624 or return $grp->result (); 733 return $grp->result () if $_[0];
734 my $now = time;
735 my $hash1 = join ":", (stat _)[0,1,3,7,9];
625 736
626 # stat the dir another time 737 # read the directory entries
627 aioreq_pri $pri; 738 aioreq_pri $pri;
739 add $grp aio_readdir $path, sub {
740 my $entries = shift
741 or return $grp->result ();
742
743 # stat the dir another time
744 aioreq_pri $pri;
628 add $grp aio_stat $path, sub { 745 add $grp aio_stat $path, sub {
629 my $hash2 = join ":", (stat _)[0,1,3,7,9]; 746 my $hash2 = join ":", (stat _)[0,1,3,7,9];
630 747
631 my $ndirs; 748 my $ndirs;
632 749
633 # take the slow route if anything looks fishy 750 # take the slow route if anything looks fishy
634 if ($hash1 ne $hash2 or (stat _)[9] == $now) { 751 if ($hash1 ne $hash2 or (stat _)[9] == $now) {
635 $ndirs = -1; 752 $ndirs = -1;
636 } else { 753 } else {
637 # if nlink == 2, we are finished 754 # if nlink == 2, we are finished
638 # on non-posix-fs's, we rely on nlink < 2 755 # on non-posix-fs's, we rely on nlink < 2
639 $ndirs = (stat _)[3] - 2 756 $ndirs = (stat _)[3] - 2
640 or return $grp->result ([], $entries); 757 or return $grp->result ([], $entries);
641 } 758 }
642 759
643 # sort into likely dirs and likely nondirs 760 # sort into likely dirs and likely nondirs
644 # dirs == files without ".", short entries first 761 # dirs == files without ".", short entries first
645 $entries = [map $_->[0], 762 $entries = [map $_->[0],
646 sort { $b->[1] cmp $a->[1] } 763 sort { $b->[1] cmp $a->[1] }
647 map [$_, sprintf "%s%04d", (/.\./ ? "1" : "0"), length], 764 map [$_, sprintf "%s%04d", (/.\./ ? "1" : "0"), length],
648 @$entries]; 765 @$entries];
649 766
650 my (@dirs, @nondirs); 767 my (@dirs, @nondirs);
651 768
652 my $statgrp = add $grp aio_group sub { 769 my $statgrp = add $grp aio_group sub {
653 $grp->result (\@dirs, \@nondirs); 770 $grp->result (\@dirs, \@nondirs);
654 }; 771 };
655 772
656 limit $statgrp $maxreq; 773 limit $statgrp $maxreq;
657 feed $statgrp sub { 774 feed $statgrp sub {
658 return unless @$entries; 775 return unless @$entries;
659 my $entry = pop @$entries; 776 my $entry = pop @$entries;
660 777
661 aioreq_pri $pri; 778 aioreq_pri $pri;
662 add $statgrp aio_stat "$path/$entry/.", sub { 779 add $statgrp aio_stat "$path/$entry/.", sub {
663 if ($_[0] < 0) { 780 if ($_[0] < 0) {
664 push @nondirs, $entry; 781 push @nondirs, $entry;
665 } else { 782 } else {
666 # need to check for real directory 783 # need to check for real directory
667 aioreq_pri $pri; 784 aioreq_pri $pri;
668 add $statgrp aio_lstat "$path/$entry", sub { 785 add $statgrp aio_lstat "$path/$entry", sub {
669 if (-d _) { 786 if (-d _) {
670 push @dirs, $entry; 787 push @dirs, $entry;
671 788
672 unless (--$ndirs) { 789 unless (--$ndirs) {
673 push @nondirs, @$entries; 790 push @nondirs, @$entries;
674 feed $statgrp; 791 feed $statgrp;
792 }
793 } else {
794 push @nondirs, $entry;
675 } 795 }
676 } else {
677 push @nondirs, $entry;
678 } 796 }
679 } 797 }
680 } 798 };
681 }; 799 };
682 }; 800 };
683 }; 801 };
684 }; 802 };
803
804 $grp
685 }; 805 }
806}
686 807
808=item aio_rmtree $path, $callback->($status)
809
810Delete a directory tree starting (and including) C<$path>, return the
811status of the final C<rmdir> only. This is a composite request that
812uses C<aio_scandir> to recurse into and rmdir directories, and unlink
813everything else.
814
815=cut
816
817sub aio_rmtree;
818sub aio_rmtree($;$) {
819 aio_block {
820 my ($path, $cb) = @_;
821
822 my $pri = aioreq_pri;
823 my $grp = aio_group $cb;
824
825 aioreq_pri $pri;
826 add $grp aio_scandir $path, 0, sub {
827 my ($dirs, $nondirs) = @_;
828
829 my $dirgrp = aio_group sub {
830 add $grp aio_rmdir $path, sub {
831 $grp->result ($_[0]);
832 };
833 };
834
835 (aioreq_pri $pri), add $dirgrp aio_rmtree "$path/$_" for @$dirs;
836 (aioreq_pri $pri), add $dirgrp aio_unlink "$path/$_" for @$nondirs;
837
838 add $grp $dirgrp;
839 };
840
687 $grp 841 $grp
842 }
688} 843}
689 844
690=item aio_fsync $fh, $callback->($status) 845=item aio_fsync $fh, $callback->($status)
691 846
692Asynchronously call fsync on the given filehandle and call the callback 847Asynchronously call fsync on the given filehandle and call the callback
… …
971 poll => 'r', nice => 1, 1126 poll => 'r', nice => 1,
972 cb => &IO::AIO::poll_cb); 1127 cb => &IO::AIO::poll_cb);
973 1128
974=item IO::AIO::poll_wait 1129=item IO::AIO::poll_wait
975 1130
1131If there are any outstanding requests and none of them in the result
976Wait till the result filehandle becomes ready for reading (simply does a 1132phase, wait till the result filehandle becomes ready for reading (simply
977C<select> on the filehandle. This is useful if you want to synchronously 1133does a C<select> on the filehandle. This is useful if you want to
978wait for some requests to finish). 1134synchronously wait for some requests to finish).
979 1135
980See C<nreqs> for an example. 1136See C<nreqs> for an example.
981 1137
982=item IO::AIO::poll 1138=item IO::AIO::poll
983 1139
984Waits until some requests have been handled. 1140Waits until some requests have been handled.
985 1141
1142Returns the number of requests processed, but is otherwise strictly
986Strictly equivalent to: 1143equivalent to:
987 1144
988 IO::AIO::poll_wait, IO::AIO::poll_cb 1145 IO::AIO::poll_wait, IO::AIO::poll_cb
989 if IO::AIO::nreqs;
990 1146
991=item IO::AIO::flush 1147=item IO::AIO::flush
992 1148
993Wait till all outstanding AIO requests have been handled. 1149Wait till all outstanding AIO requests have been handled.
994 1150
995Strictly equivalent to: 1151Strictly equivalent to:
996 1152
997 IO::AIO::poll_wait, IO::AIO::poll_cb 1153 IO::AIO::poll_wait, IO::AIO::poll_cb
998 while IO::AIO::nreqs; 1154 while IO::AIO::nreqs;
999 1155
1156=back
1157
1000=head3 CONTROLLING THE NUMBER OF THREADS 1158=head3 CONTROLLING THE NUMBER OF THREADS
1159
1160=over
1001 1161
1002=item IO::AIO::min_parallel $nthreads 1162=item IO::AIO::min_parallel $nthreads
1003 1163
1004Set the minimum number of AIO threads to C<$nthreads>. The current 1164Set the minimum number of AIO threads to C<$nthreads>. The current
1005default is C<8>, which means eight asynchronous operations can execute 1165default is C<8>, which means eight asynchronous operations can execute
… …
1053This is a very bad function to use in interactive programs because it 1213This is a very bad function to use in interactive programs because it
1054blocks, and a bad way to reduce concurrency because it is inexact: Better 1214blocks, and a bad way to reduce concurrency because it is inexact: Better
1055use an C<aio_group> together with a feed callback. 1215use an C<aio_group> together with a feed callback.
1056 1216
1057Sets the maximum number of outstanding requests to C<$nreqs>. If you 1217Sets the maximum number of outstanding requests to C<$nreqs>. If you
1058to queue up more than this number of requests, the next call to the 1218do queue up more than this number of requests, the next call to the
1059C<poll_cb> (and C<poll_some> and other functions calling C<poll_cb>) 1219C<poll_cb> (and C<poll_some> and other functions calling C<poll_cb>)
1060function will block until the limit is no longer exceeded. 1220function will block until the limit is no longer exceeded.
1061 1221
1062The default value is very large, so there is no practical limit on the 1222The default value is very large, so there is no practical limit on the
1063number of outstanding requests. 1223number of outstanding requests.
1064 1224
1065You can still queue as many requests as you want. Therefore, 1225You can still queue as many requests as you want. Therefore,
1066C<max_oustsanding> is mainly useful in simple scripts (with low values) or 1226C<max_oustsanding> is mainly useful in simple scripts (with low values) or
1067as a stop gap to shield against fatal memory overflow (with large values). 1227as a stop gap to shield against fatal memory overflow (with large values).
1068 1228
1229=back
1230
1069=head3 STATISTICAL INFORMATION 1231=head3 STATISTICAL INFORMATION
1232
1233=over
1070 1234
1071=item IO::AIO::nreqs 1235=item IO::AIO::nreqs
1072 1236
1073Returns the number of requests currently in the ready, execute or pending 1237Returns the number of requests currently in the ready, execute or pending
1074states (i.e. for which their callback has not been invoked yet). 1238states (i.e. for which their callback has not been invoked yet).
… …
1090 1254
1091=back 1255=back
1092 1256
1093=cut 1257=cut
1094 1258
1095# support function to convert a fd into a perl filehandle
1096sub _fd2fh {
1097 return undef if $_[0] < 0;
1098
1099 # try to generate nice filehandles
1100 my $sym = "IO::AIO::fd#$_[0]";
1101 local *$sym;
1102
1103 open *$sym, "+<&=$_[0]" # usually works under any unix
1104 or open *$sym, "<&=$_[0]" # cygwin needs this
1105 or open *$sym, ">&=$_[0]" # or this
1106 or return undef;
1107
1108 *$sym
1109}
1110
1111min_parallel 8; 1259min_parallel 8;
1112 1260
1113END { 1261END { flush }
1114 min_parallel 1;
1115 flush;
1116};
1117 1262
11181; 12631;
1119 1264
1120=head2 FORK BEHAVIOUR 1265=head2 FORK BEHAVIOUR
1121 1266
… …
1141bytes of memory. In addition, stat requests need a stat buffer (possibly 1286bytes of memory. In addition, stat requests need a stat buffer (possibly
1142a few hundred bytes), readdir requires a result buffer and so on. Perl 1287a few hundred bytes), readdir requires a result buffer and so on. Perl
1143scalars and other data passed into aio requests will also be locked and 1288scalars and other data passed into aio requests will also be locked and
1144will consume memory till the request has entered the done state. 1289will consume memory till the request has entered the done state.
1145 1290
1146This is now awfully much, so queuing lots of requests is not usually a 1291This is not awfully much, so queuing lots of requests is not usually a
1147problem. 1292problem.
1148 1293
1149Per-thread usage: 1294Per-thread usage:
1150 1295
1151In the execution phase, some aio requests require more memory for 1296In the execution phase, some aio requests require more memory for

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines