ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-Fork-Pool/Pool.pm
(Generate patch)

Comparing AnyEvent-Fork-Pool/Pool.pm (file contents):
Revision 1.13 by root, Sun Apr 28 14:27:31 2013 UTC vs.
Revision 1.14 by root, Sun Oct 26 16:22:38 2014 UTC

3AnyEvent::Fork::Pool - simple process pool manager on top of AnyEvent::Fork 3AnyEvent::Fork::Pool - simple process pool manager on top of AnyEvent::Fork
4 4
5=head1 SYNOPSIS 5=head1 SYNOPSIS
6 6
7 use AnyEvent; 7 use AnyEvent;
8 use AnyEvent::Fork;
8 use AnyEvent::Fork::Pool; 9 use AnyEvent::Fork::Pool;
9 # use AnyEvent::Fork is not needed
10 10
11 # all possible parameters shown, with default values 11 # all possible parameters shown, with default values
12 my $pool = AnyEvent::Fork 12 my $pool = AnyEvent::Fork
13 ->new 13 ->new
14 ->require ("MyWorker") 14 ->require ("MyWorker")
86 86
87use Guard (); 87use Guard ();
88use Array::Heap (); 88use Array::Heap ();
89 89
90use AnyEvent; 90use AnyEvent;
91# explicit version on next line, as some cpan-testers test with the 0.1 version,
92# ignoring dependencies, and this line will at least give a clear indication of that.
93use AnyEvent::Fork 0.6; # we don't actually depend on it, this is for convenience
94use AnyEvent::Fork::RPC; 91use AnyEvent::Fork::RPC;
95 92
96# these are used for the first and last argument of events 93# these are used for the first and last argument of events
97# in the hope of not colliding. yes, I don't like it either, 94# in the hope of not colliding. yes, I don't like it either,
98# but didn't come up with an obviously better alternative. 95# but didn't come up with an obviously better alternative.
449 446
450=item $cpus = AnyEvent::Fork::Pool::ncpu [$default_cpus] 447=item $cpus = AnyEvent::Fork::Pool::ncpu [$default_cpus]
451 448
452=item ($cpus, $eus) = AnyEvent::Fork::Pool::ncpu [$default_cpus] 449=item ($cpus, $eus) = AnyEvent::Fork::Pool::ncpu [$default_cpus]
453 450
454Tries to detect the number of CPUs (C<$cpus> often called cpu cores 451Tries to detect the number of CPUs (C<$cpus> often called CPU cores
455nowadays) and execution units (C<$eus>) which include e.g. extra 452nowadays) and execution units (C<$eus>) which include e.g. extra
456hyperthreaded units). When C<$cpus> cannot be determined reliably, 453hyperthreaded units). When C<$cpus> cannot be determined reliably,
457C<$default_cpus> is returned for both values, or C<1> if it is missing. 454C<$default_cpus> is returned for both values, or C<1> if it is missing.
458 455
459For normal CPU bound uses, it is wise to have as many worker processes 456For normal CPU bound uses, it is wise to have as many worker processes
461hyperthreading is usually detrimental to performance, but in those rare 458hyperthreading is usually detrimental to performance, but in those rare
462cases where that really helps it might be beneficial to use more workers 459cases where that really helps it might be beneficial to use more workers
463(C<$eus>). 460(C<$eus>).
464 461
465Currently, F</proc/cpuinfo> is parsed on GNU/Linux systems for both 462Currently, F</proc/cpuinfo> is parsed on GNU/Linux systems for both
466C<$cpus> and C<$eu>, and on {Free,Net,Open}BSD, F<sysctl -n hw.ncpu> is 463C<$cpus> and C<$eus>, and on {Free,Net,Open}BSD, F<sysctl -n hw.ncpu> is
467used for C<$cpus>. 464used for C<$cpus>.
468 465
469Example: create a worker pool with as many workers as cpu cores, or C<2>, 466Example: create a worker pool with as many workers as CPU cores, or C<2>,
470if the actual number could not be determined. 467if the actual number could not be determined.
471 468
472 $fork->AnyEvent::Fork::Pool::run ("myworker::function", 469 $fork->AnyEvent::Fork::Pool::run ("myworker::function",
473 max => (scalar AnyEvent::Fork::Pool::ncpu 2), 470 max => (scalar AnyEvent::Fork::Pool::ncpu 2),
474 ); 471 );
521 518
522=item AnyEvent::Fork::Pool::retire () 519=item AnyEvent::Fork::Pool::retire ()
523 520
524This function sends an event to the parent process to request retirement: 521This function sends an event to the parent process to request retirement:
525the worker is removed from the pool and no new jobs will be sent to it, 522the worker is removed from the pool and no new jobs will be sent to it,
526but it has to handle the jobs that are already queued. 523but it still has to handle the jobs that are already queued.
527 524
528The parentheses are part of the syntax: the function usually isn't defined 525The parentheses are part of the syntax: the function usually isn't defined
529when you compile your code (because that happens I<before> handing the 526when you compile your code (because that happens I<before> handing the
530template process over to C<AnyEvent::Fork::Pool::run>, so you need the 527template process over to C<AnyEvent::Fork::Pool::run>, so you need the
531empty parentheses to tell Perl that the function is indeed a function. 528empty parentheses to tell Perl that the function is indeed a function.
532 529
533Retiring a worker can be useful to gracefully shut it down when the worker 530Retiring a worker can be useful to gracefully shut it down when the worker
534deems this useful. For example, after executing a job, one could check 531deems this useful. For example, after executing a job, it could check the
535the process size or the number of jobs handled so far, and if either is 532process size or the number of jobs handled so far, and if either is too
536too high, the worker could ask to get retired, to avoid memory leaks to 533high, the worker could request to be retired, to avoid memory leaks to
537accumulate. 534accumulate.
538 535
539Example: retire a worker after it has handled roughly 100 requests. 536Example: retire a worker after it has handled roughly 100 requests. It
537doesn't matter whether you retire at the beginning or end of your request,
538as the worker will continue to handle some outstanding requests. Likewise,
539it's ok to call retire multiple times.
540 540
541 my $count = 0; 541 my $count = 0;
542 542
543 sub my::worker { 543 sub my::worker {
544 544
550 550
551=back 551=back
552 552
553=head1 POOL PARAMETERS RECIPES 553=head1 POOL PARAMETERS RECIPES
554 554
555This section describes some recipes for pool paramaters. These are mostly 555This section describes some recipes for pool parameters. These are mostly
556meant for the synchronous RPC backend, as the asynchronous RPC backend 556meant for the synchronous RPC backend, as the asynchronous RPC backend
557changes the rules considerably, making workers themselves responsible for 557changes the rules considerably, making workers themselves responsible for
558their scheduling. 558their scheduling.
559 559
560=over 4 560=over 4
589=item high throughput, I/O bound jobs - set load >= 2, max = 1, or very high 589=item high throughput, I/O bound jobs - set load >= 2, max = 1, or very high
590 590
591When your jobs are I/O bound, using more workers usually boils down to 591When your jobs are I/O bound, using more workers usually boils down to
592higher throughput, depending very much on your actual workload - sometimes 592higher throughput, depending very much on your actual workload - sometimes
593having only one worker is best, for example, when you read or write big 593having only one worker is best, for example, when you read or write big
594files at maixmum speed, as a second worker will increase seek times. 594files at maximum speed, as a second worker will increase seek times.
595 595
596=back 596=back
597 597
598=head1 EXCEPTIONS 598=head1 EXCEPTIONS
599 599
600The same "policy" as with L<AnyEvent::Fork::RPC> applies - exceptins will 600The same "policy" as with L<AnyEvent::Fork::RPC> applies - exceptions
601not be caught, and exceptions in both worker and in callbacks causes 601will not be caught, and exceptions in both worker and in callbacks causes
602undesirable or undefined behaviour. 602undesirable or undefined behaviour.
603 603
604=head1 SEE ALSO 604=head1 SEE ALSO
605 605
606L<AnyEvent::Fork>, to create the processes in the first place. 606L<AnyEvent::Fork>, to create the processes in the first place.

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines