[ViewVC] Diff of: cvs/IO-AIO/AIO.pm

Comparing IO-AIO/AIO.pm (file contents):
Revision 1.58 by root, Sun Oct 22 10:33:19 2006 UTC vs.
Revision 1.61 by root, Sun Oct 22 13:52:11 2006 UTC

 Scans a directory (similar to C<aio_readdir>) but additionally tries to
 separate the entries of directory C<$path> into two sets of names, ones
 you can recurse into (directories or links to them), and ones you cannot
 recurse into (everything else).
-C<aio_scandir> is a composite request that consists of many sub
+C<aio_scandir> is a composite request that creates of many sub requests_
-requests. C<$maxreq> specifies the maximum number of outstanding aio
+C<$maxreq> specifies the maximum number of outstanding aio requests that
-requests that this function generates. If it is C<< <= 0 >>, then a
+this function generates. If it is C<< <= 0 >>, then a suitable default
-suitable default will be chosen (currently 8).
+will be chosen (currently 6).
 On error, the callback is called without arguments, otherwise it receives
 two array-refs with path-relative entry names.
 Example:
 sub aio_scandir($$$) {
    my ($path, $maxreq, $cb) = @_;
    my $grp = aio_group $cb;
-   $maxreq = 8 if $maxreq <= 0;
+   $maxreq = 6 if $maxreq <= 0;
    # stat once
    add $grp aio_stat $path, sub {
       return $grp->result () if $_[0];
       my $now = time;
             my (@dirs, @nondirs);
             my ($statcb, $schedcb);
             my $nreq = 0;
+            my $statgrp = add $grp aio_group;
             $schedcb = sub {
                if (@$entries) {
                   if ($nreq < $maxreq) {
                      my $ent = pop @$entries;
                      $nreq++;
-                     add $grp aio_stat "$path/$ent/.", sub { $statcb->($_[0], $ent) };
+                     add $statgrp aio_stat "$path/$ent/.", sub { $statcb->($_[0], $ent) };
                   }
                } elsif (!$nreq) {
                   # finished
+                  $statgrp->cancel;
                   undef $statcb;
                   undef $schedcb;
-                  $grp->result (\@dirs, \@nondirs) if $cb;
+                  $grp->result (\@dirs, \@nondirs);
-                  undef $cb;
                }
             };
             $statcb = sub {
                my ($status, $entry) = @_;
 The IO::AIO::GRP objects will be cleaned up during calls to
 C<IO::AIO::poll_cb>, just like any other request.
 They can be canceled like any other request. Canceling will cancel not
-just the request itself, but also all requests it contains.
+only the request itself, but also all requests it contains.
 They can also can also be added to other IO::AIO::GRP objects.
+You must not add requests to a group from within the group callback (or
+any later time).
 Their lifetime, simplified, looks like this: when they are empty, they
 will finish very quickly. If they contain only requests that are in the
 C<done> state, they will also finish. Otherwise they will continue to
 exist.
 =item $grp->result (...)
 Set the result value(s) that will be passed to the group callback when all
 subrequests have finished. By default, no argument will be passed.
+=item $grp->set_feeder ($callback->($grp))
+[VERY EXPERIMENTAL]
+Sets a feeder/generator on this group: every group can have an attached
+generator that generates requests if idle. The idea behind this is that,
+although you could just queue as many requests as you want in a group,
+this might starve other requests for a potentially long time.  For
+example, C<aio_scandir> might generate hundreds of thousands C<aio_stat>
+requests, delaying any later requests for a long time.
+To avoid this, and allow incremental generation of requests, you can
+instead a group and set a feeder on it that generates those requests. The
+feeder will be called whenever there are few enough (see C<feeder_limit>,
+below) requests active in the group itself and is expected to queue more
+requests.
+The feeder can queue as many requests as it likes (i.e. C<add> does not
+impose any limits).
+If the feeder does not queue more requests when called, it will be
+automatically removed from the group.
+If the feeder limit is C<0>, it will be set to C<2> automatically.
+Example:
+   # stat all files in @files, but only ever use four aio requests concurrently:
+   my $grp = aio_group sub { print "finished\n" };
+   $grp->feeder_limit (4);
+   $grp->set_feeder (sub {
+      my $file = pop @files
+         or return;
+      add $grp aio_stat $file, sub { ... };
+   });
+=item $grp->feeder_limit ($num)
+Sets the feeder limit for the group: The feeder will be called whenever
+the group contains less than this many requests.
+Setting the limit to C<0> will pause the feeding process.
 =back
 =head2 SUPPORT FUNCTIONS
 =over 4
    IO::AIO::poll_wait, IO::AIO::poll_cb
       if IO::AIO::nreqs;
 =item IO::AIO::min_parallel $nthreads
-Set the minimum number of AIO threads to C<$nthreads>. The current default
+Set the minimum number of AIO threads to C<$nthreads>. The current
-is C<4>, which means four asynchronous operations can be done at one time
+default is C<8>, which means eight asynchronous operations can execute
-(the number of outstanding operations, however, is unlimited).
+concurrently at any one time (the number of outstanding requests,
+however, is unlimited).
 IO::AIO starts threads only on demand, when an AIO request is queued and
 no free thread exists.
-It is recommended to keep the number of threads low, as some Linux
+It is recommended to keep the number of threads relatively low, as some
-kernel versions will scale negatively with the number of threads (higher
+Linux kernel versions will scale negatively with the number of threads
-parallelity => MUCH higher latency). With current Linux 2.6 versions, 4-32
+(higher parallelity => MUCH higher latency). With current Linux 2.6
-threads should be fine.
+versions, 4-32 threads should be fine.
 Under most circumstances you don't need to call this function, as the
 module selects a default that is suitable for low to moderate load.
 =item IO::AIO::max_parallel $nthreads
       or return undef;
    *$sym
 }
-min_parallel 4;
+min_parallel 8;
 END {
    max_parallel 0;
 }
 In short: the parent will, after a short pause, continue as if fork had
 not been called, while the child will act as if IO::AIO has not been used
 yet.
+=head2 MEMORY USAGE
+Each aio request uses - depending on your architecture - around 128 bytes
+of memory.  In addition, stat requests need a stat buffer (possibly a few
+hundred bytes). Perl scalars and other data passed into aio requests will
+also be locked.
+This is now awfully much, so queuing lots of requests is not usually a
+problem.
+Each thread needs a stack area which is usually around 16k, sometimes much
+larger, depending on the OS.
 =head1 SEE ALSO
 L<Coro>, L<Linux::AIO> (obsolete).
 =head1 AUTHOR

Diff Legend

-–
+Removed lines
-+
+Added lines
-<
+Changed lines
->
+Changed lines

Comparing IO-AIO/AIO.pm (file contents): Revision 1.58 by root, Sun Oct 22 10:33:19 2006 UTC vs. Revision 1.61 by root, Sun Oct 22 13:52:11 2006 UTC

Diff Legend

Comparing IO-AIO/AIO.pm (file contents):
Revision 1.58 by root, Sun Oct 22 10:33:19 2006 UTC vs.
Revision 1.61 by root, Sun Oct 22 13:52:11 2006 UTC