ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-Fork/Fork.pm
(Generate patch)

Comparing AnyEvent-Fork/Fork.pm (file contents):
Revision 1.35 by root, Sat Apr 6 09:39:12 2013 UTC vs.
Revision 1.39 by root, Sat Apr 6 22:39:37 2013 UTC

… …
27 27
28Special care has been taken to make this module useful from other modules, 28Special care has been taken to make this module useful from other modules,
29while still supporting specialised environments such as L<App::Staticperl> 29while still supporting specialised environments such as L<App::Staticperl>
30or L<PAR::Packer>. 30or L<PAR::Packer>.
31 31
32=head1 WHAT THIS MODULE IS NOT 32=head2 WHAT THIS MODULE IS NOT
33 33
34This module only creates processes and lets you pass file handles and 34This module only creates processes and lets you pass file handles and
35strings to it, and run perl code. It does not implement any kind of RPC - 35strings to it, and run perl code. It does not implement any kind of RPC -
36there is no back channel from the process back to you, and there is no RPC 36there is no back channel from the process back to you, and there is no RPC
37or message passing going on. 37or message passing going on.
… …
40in whatever way you like, use some message-passing module such 40in whatever way you like, use some message-passing module such
41as L<AnyEvent::MP>, some pipe such as L<AnyEvent::ZeroMQ>, use 41as L<AnyEvent::MP>, some pipe such as L<AnyEvent::ZeroMQ>, use
42L<AnyEvent::Handle> on both sides to send e.g. JSON or Storable messages, 42L<AnyEvent::Handle> on both sides to send e.g. JSON or Storable messages,
43and so on. 43and so on.
44 44
45=head2 COMPARISON TO OTHER MODULES
46
47There is an abundance of modules on CPAN that do "something fork", such as
48L<Parallel::ForkManager>, L<AnyEvent::ForkManager>, L<AnyEvent::Worker>
49or L<AnyEvent::Subprocess>. There are modules that implement their own
50process management, such as L<AnyEvent::DBI>.
51
52The problems that all these modules try to solve are real, however, none
53of them (from what I have seen) tackle the very real problems of unwanted
54memory sharing, efficiency, not being able to use event processing or
55similar modules in the processes they create.
56
57This module doesn't try to replace any of them - instead it tries to solve
58the problem of creating processes with a minimum of fuss and overhead (and
59also luxury). Ideally, most of these would use AnyEvent::Fork internally,
60except they were written before AnyEvent:Fork was available, so obviously
61had to roll their own.
62
45=head1 PROBLEM STATEMENT 63=head2 PROBLEM STATEMENT
46 64
47There are two traditional ways to implement parallel processing on UNIX 65There are two traditional ways to implement parallel processing on UNIX
48like operating systems - fork and process, and fork+exec and process. They 66like operating systems - fork and process, and fork+exec and process. They
49have different advantages and disadvantages that I describe below, 67have different advantages and disadvantages that I describe below,
50together with how this module tries to mitigate the disadvantages. 68together with how this module tries to mitigate the disadvantages.
… …
780 479 vfork+execs per second, using AnyEvent::Fork->new_exec 798 479 vfork+execs per second, using AnyEvent::Fork->new_exec
781 799
782So how can C<< AnyEvent->new >> be faster than a standard fork, even 800So how can C<< AnyEvent->new >> be faster than a standard fork, even
783though it uses the same operations, but adds a lot of overhead? 801though it uses the same operations, but adds a lot of overhead?
784 802
785The difference is simply the process size: forking the 6MB process takes 803The difference is simply the process size: forking the 5MB process takes
786so much longer than forking the 2.5MB template process that the overhead 804so much longer than forking the 2.5MB template process that the extra
787introduced is canceled out. 805overhead introduced is canceled out.
788 806
789If the benchmark process grows, the normal fork becomes even slower: 807If the benchmark process grows, the normal fork becomes even slower:
790 808
791 1340 new processes, manual fork in a 20MB process 809 1340 new processes, manual fork of a 20MB process
792 731 new processes, manual fork in a 200MB process 810 731 new processes, manual fork of a 200MB process
793 235 new processes, manual fork in a 2000MB process 811 235 new processes, manual fork of a 2000MB process
794 812
795What that means (to me) is that I can use this module without having a 813What that means (to me) is that I can use this module without having a bad
796very bad conscience because of the extra overhead required to start new 814conscience because of the extra overhead required to start new processes.
797processes.
798 815
799=head1 TYPICAL PROBLEMS 816=head1 TYPICAL PROBLEMS
800 817
801This section lists typical problems that remain. I hope by recognising 818This section lists typical problems that remain. I hope by recognising
802them, most can be avoided. 819them, most can be avoided.
803 820
804=over 4 821=over 4
805 822
806=item "leaked" file descriptors for exec'ed processes 823=item leaked file descriptors for exec'ed processes
807 824
808POSIX systems inherit file descriptors by default when exec'ing a new 825POSIX systems inherit file descriptors by default when exec'ing a new
809process. While perl itself laudably sets the close-on-exec flags on new 826process. While perl itself laudably sets the close-on-exec flags on new
810file handles, most C libraries don't care, and even if all cared, it's 827file handles, most C libraries don't care, and even if all cared, it's
811often not possible to set the flag in a race-free manner. 828often not possible to set the flag in a race-free manner.
… …
831libraries or the code that leaks those file descriptors. 848libraries or the code that leaks those file descriptors.
832 849
833Fortunately, most of these leaked descriptors do no harm, other than 850Fortunately, most of these leaked descriptors do no harm, other than
834sitting on some resources. 851sitting on some resources.
835 852
836=item "leaked" file descriptors for fork'ed processes 853=item leaked file descriptors for fork'ed processes
837 854
838Normally, L<AnyEvent::Fork> does start new processes by exec'ing them, 855Normally, L<AnyEvent::Fork> does start new processes by exec'ing them,
839which closes file descriptors not marked for being inherited. 856which closes file descriptors not marked for being inherited.
840 857
841However, L<AnyEvent::Fork::Early> and L<AnyEvent::Fork::Template> offer 858However, L<AnyEvent::Fork::Early> and L<AnyEvent::Fork::Template> offer
… …
850 867
851The solution is to either not load these modules before use'ing 868The solution is to either not load these modules before use'ing
852L<AnyEvent::Fork::Early> or L<AnyEvent::Fork::Template>, or to delay 869L<AnyEvent::Fork::Early> or L<AnyEvent::Fork::Template>, or to delay
853initialising them, for example, by calling C<init Gtk2> manually. 870initialising them, for example, by calling C<init Gtk2> manually.
854 871
855=item exit runs destructors 872=item exiting calls object destructors
856 873
857This only applies to users of Lc<AnyEvent::Fork:Early> and 874This only applies to users of L<AnyEvent::Fork:Early> and
858L<AnyEvent::Fork::Template>. 875L<AnyEvent::Fork::Template>, or when initialiasing code creates objects
876that reference external resources.
859 877
860When a process created by AnyEvent::Fork exits, it might do so by calling 878When a process created by AnyEvent::Fork exits, it might do so by calling
861exit, or simply letting perl reach the end of the program. At which point 879exit, or simply letting perl reach the end of the program. At which point
862Perl runs all destructors. 880Perl runs all destructors.
863 881
… …
882to make it so, mostly due to the bloody broken perl that nobody seems to 900to make it so, mostly due to the bloody broken perl that nobody seems to
883care about. The fork emulation is a bad joke - I have yet to see something 901care about. The fork emulation is a bad joke - I have yet to see something
884useful that you can do with it without running into memory corruption 902useful that you can do with it without running into memory corruption
885issues or other braindamage. Hrrrr. 903issues or other braindamage. Hrrrr.
886 904
887Cygwin perl is not supported at the moment, as it should implement fd 905Cygwin perl is not supported at the moment due to some hilarious
888passing, but doesn't, and rolling my own is hard, as cygwin doesn't 906shortcomings of its API - see L<IO::FDPoll> for more details.
889support enough functionality to do it.
890 907
891=head1 SEE ALSO 908=head1 SEE ALSO
892 909
893L<AnyEvent::Fork::Early> (to avoid executing a perl interpreter), 910L<AnyEvent::Fork::Early> (to avoid executing a perl interpreter),
894L<AnyEvent::Fork::Template> (to create a process by forking the main 911L<AnyEvent::Fork::Template> (to create a process by forking the main

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines