[ViewVC] Diff of: cvs/IO-AIO/AIO.pm

Comparing IO-AIO/AIO.pm (file contents):
Revision 1.67 by root, Tue Oct 24 02:25:16 2006 UTC vs.
Revision 1.74 by root, Tue Oct 24 17:22:17 2006 UTC

…		…
18	};	18	};
19		19
20	# version 2+ has request and group objects	20	# version 2+ has request and group objects
21	use IO::AIO 2;	21	use IO::AIO 2;
22		22
		23	aioreq_pri 4; # give next request a very high priority
23	my $req = aio_unlink "/tmp/file", sub { };	24	my $req = aio_unlink "/tmp/file", sub { };
24	$req->cancel; # cancel request if still in queue	25	$req->cancel; # cancel request if still in queue
25		26
26	my $grp = aio_group sub { print "all stats done\n" };	27	my $grp = aio_group sub { print "all stats done\n" };
27	add $grp aio_stat "..." for ...;	28	add $grp aio_stat "..." for ...;
…		…
50	=head1 DESCRIPTION	51	=head1 DESCRIPTION
51		52
52	This module implements asynchronous I/O using whatever means your	53	This module implements asynchronous I/O using whatever means your
53	operating system supports.	54	operating system supports.
54		55
55	Currently, a number of threads are started that execute your read/writes	56	In this version, a number of threads are started that execute your
56	and signal their completion. You don't need thread support in perl, and	57	requests and signal their completion. You don't need thread support
57	the threads created by this module will not be visible to perl. In the	58	in perl, and the threads created by this module will not be visible
58	future, this module might make use of the native aio functions available	59	to perl. In the future, this module might make use of the native aio
59	on many operating systems. However, they are often not well-supported	60	functions available on many operating systems. However, they are often
60	(Linux doesn't allow them on normal files currently, for example),	61	not well-supported or restricted (Linux doesn't allow them on normal
61	and they would only support aio_read and aio_write, so the remaining	62	files currently, for example), and they would only support aio_read and
62	functionality would have to be implemented using threads anyway.	63	aio_write, so the remaining functionality would have to be implemented
		64	using threads anyway.
63		65
64	Although the module will work with in the presence of other threads, it is	66	Although the module will work with in the presence of other (Perl-)
65	currently not reentrant, so use appropriate locking yourself, always call	67	threads, it is currently not reentrant in any way, so use appropriate
66	C<poll_cb> from within the same thread, or never call C<poll_cb> (or other	68	locking yourself, always call C<poll_cb> from within the same thread, or
67	C<aio_> functions) recursively.	69	never call C<poll_cb> (or other C<aio_> functions) recursively.
		70
		71	=head1 REQUEST ANATOMY AND LIFETIME
		72
		73	Every C<aio_*> function creates a request. which is a C data structure not
		74	directly visible to Perl.
		75
		76	If called in non-void context, every request function returns a Perl
		77	object representing the request. In void context, nothing is returned,
		78	which saves a bit of memory.
		79
		80	The perl object is a fairly standard ref-to-hash object. The hash contents
		81	are not used by IO::AIO so you are free to store anything you like in it.
		82
		83	During their existance, aio requests travel through the following states,
		84	in order:
		85
		86	=over 4
		87
		88	=item ready
		89
		90	Immediately after a request is created it is put into the ready state,
		91	waiting for a thread to execute it.
		92
		93	=item execute
		94
		95	A thread has accepted the request for processing and is currently
		96	executing it (e.g. blocking in read).
		97
		98	=item pending
		99
		100	The request has been executed and is waiting for result processing.
		101
		102	While request submission and execution is fully asynchronous, result
		103	processing is not and relies on the perl interpreter calling C<poll_cb>
		104	(or another function with the same effect).
		105
		106	=item result
		107
		108	The request results are processed synchronously by C<poll_cb>.
		109
		110	The C<poll_cb> function will process all outstanding aio requests by
		111	calling their callbacks, freeing memory associated with them and managing
		112	any groups they are contained in.
		113
		114	=item done
		115
		116	Request has reached the end of its lifetime and holds no resources anymore
		117	(except possibly for the Perl object, but its connection to the actual
		118	aio request is severed and calling its methods will either do nothing or
		119	result in a runtime error).
68		120
69	=cut	121	=cut
70		122
71	package IO::AIO;	123	package IO::AIO;
72		124
…		…
80		132
81	our @AIO_REQ = qw(aio_sendfile aio_read aio_write aio_open aio_close aio_stat	133	our @AIO_REQ = qw(aio_sendfile aio_read aio_write aio_open aio_close aio_stat
82	aio_lstat aio_unlink aio_rmdir aio_readdir aio_scandir aio_symlink	134	aio_lstat aio_unlink aio_rmdir aio_readdir aio_scandir aio_symlink
83	aio_fsync aio_fdatasync aio_readahead aio_rename aio_link aio_move	135	aio_fsync aio_fdatasync aio_readahead aio_rename aio_link aio_move
84	aio_group aio_nop);	136	aio_group aio_nop);
85	our @EXPORT = (@AIO_REQ, qw(aioreq_pri));	137	our @EXPORT = (@AIO_REQ, qw(aioreq_pri aioreq_nice));
86	our @EXPORT_OK = qw(poll_fileno poll_cb poll_wait flush	138	our @EXPORT_OK = qw(poll_fileno poll_cb poll_wait flush
87	min_parallel max_parallel max_outstanding nreqs);	139	min_parallel max_parallel max_outstanding nreqs);
88		140
89	@IO::AIO::GRP::ISA = 'IO::AIO::REQ';	141	@IO::AIO::GRP::ISA = 'IO::AIO::REQ';
90		142
…		…
123	environment, d) use Glib::filename_from_unicode on unicode filenames or e)	175	environment, d) use Glib::filename_from_unicode on unicode filenames or e)
124	use something else.	176	use something else.
125		177
126	=over 4	178	=over 4
127		179
		180	=item aioreq_pri $pri
		181
		182	Sets the priority for the next aio request. The default priority
		183	is C<0>, the minimum and maximum priorities are C<-4> and C<4>,
		184	respectively. Requests with higher priority will be serviced first.
		185
		186	The priority will be reset to C<0> after each call to one of the C<aio_>
		187	functions.
		188
		189	Example: open a file with low priority, then read something from it with
		190	higher priority so the read request is serviced before other low priority
		191	open requests (potentially spamming the cache):
		192
		193	aioreq_pri -3;
		194	aio_open ..., sub {
		195	return unless $_[0];
		196
		197	aioreq_pri -2;
		198	aio_read $_[0], ..., sub {
		199	...
		200	};
		201	};
		202
		203	=item aioreq_nice $pri_adjust
		204
		205	Similar to C<aioreq_pri>, but subtracts the given value from the current
		206	priority, so effects are cumulative.
		207
128	=item aio_open $pathname, $flags, $mode, $callback->($fh)	208	=item aio_open $pathname, $flags, $mode, $callback->($fh)
129		209
130	Asynchronously open or create a file and call the callback with a newly	210	Asynchronously open or create a file and call the callback with a newly
131	created filehandle for the file.	211	created filehandle for the file.
132		212
…		…
183	$_[0] > 0 or die "read error: $!";	263	$_[0] > 0 or die "read error: $!";
184	print "read $_[0] bytes: <$buffer>\n";	264	print "read $_[0] bytes: <$buffer>\n";
185	};	265	};
186		266
187	=item aio_move $srcpath, $dstpath, $callback->($status)	267	=item aio_move $srcpath, $dstpath, $callback->($status)
188
189	[EXPERIMENTAL due to internal aio_group use]
190		268
191	Try to move the I<file> (directories not supported as either source or	269	Try to move the I<file> (directories not supported as either source or
192	destination) from C<$srcpath> to C<$dstpath> and call the callback with	270	destination) from C<$srcpath> to C<$dstpath> and call the callback with
193	the C<0> (error) or C<-1> ok.	271	the C<0> (error) or C<-1> ok.
194		272
…		…
347	The callback a single argument which is either C<undef> or an array-ref	425	The callback a single argument which is either C<undef> or an array-ref
348	with the filenames.	426	with the filenames.
349		427
350	=item aio_scandir $path, $maxreq, $callback->($dirs, $nondirs)	428	=item aio_scandir $path, $maxreq, $callback->($dirs, $nondirs)
351		429
352	[EXPERIMENTAL due to internal aio_group use]
353
354	Scans a directory (similar to C<aio_readdir>) but additionally tries to	430	Scans a directory (similar to C<aio_readdir>) but additionally tries to
355	separate the entries of directory C<$path> into two sets of names, ones	431	separate the entries of directory C<$path> into two sets of names, ones
356	you can recurse into (directories or links to them), and ones you cannot	432	you can recurse into (directories or links to them), and ones you cannot
357	recurse into (everything else).	433	recurse into (everything else).
358		434
…		…
444	map [$_, sprintf "%s%04d", (/.\./ ? "1" : "0"), length],	520	map [$_, sprintf "%s%04d", (/.\./ ? "1" : "0"), length],
445	@$entries];	521	@$entries];
446		522
447	my (@dirs, @nondirs);	523	my (@dirs, @nondirs);
448		524
449	my ($statcb, $schedcb);
450	my $nreq = 0;
451
452	my $statgrp = add $grp aio_group;	525	my $statgrp = add $grp aio_group sub {
		526	$grp->result (\@dirs, \@nondirs);
		527	};
453		528
454	$schedcb = sub {	529	limit $statgrp $maxreq;
455	if (@$entries) {	530	feed $statgrp sub {
456	if ($nreq < $maxreq) {	531	return unless @$entries;
457	my $ent = pop @$entries;	532	my $entry = pop @$entries;
		533
		534	add $statgrp aio_stat "$path/$entry/.", sub {
		535	if ($_[0] < 0) {
		536	push @nondirs, $entry;
		537	} else {
		538	# need to check for real directory
		539	add $statgrp aio_lstat "$path/$entry", sub {
		540	if (-d _) {
		541	push @dirs, $entry;
		542
		543	if (!--$ndirs) {
		544	push @nondirs, @$entries;
		545	$statgrp->cancel_subs;
		546	}
		547	} else {
		548	push @nondirs, $entry;
		549	}
458	$nreq++;	550	}
459	add $statgrp aio_stat "$path/$ent/.", sub { $statcb->($_[0], $ent) };
460	}	551	}
461	} elsif (!$nreq) {
462	# finished
463	$statgrp->cancel;
464	undef $statcb;
465	undef $schedcb;
466	$grp->result (\@dirs, \@nondirs);
467	}	552	};
468	};	553	};
469	$statcb = sub {
470	my ($status, $entry) = @_;
471
472	if ($status < 0) {
473	$nreq--;
474	push @nondirs, $entry;
475	&$schedcb;
476	} else {
477	# need to check for real directory
478	add $grp aio_lstat "$path/$entry", sub {
479	$nreq--;
480
481	if (-d _) {
482	push @dirs, $entry;
483
484	if (!--$ndirs) {
485	push @nondirs, @$entries;
486	$entries = [];
487	}
488	} else {
489	push @nondirs, $entry;
490	}
491
492	&$schedcb;
493	}
494	}
495	};
496
497	&$schedcb while @$entries && $nreq < $maxreq;
498	};	554	};
499	};	555	};
500	};	556	};
501		557
502	$grp	558	$grp
…		…
515	If this call isn't available because your OS lacks it or it couldn't be	571	If this call isn't available because your OS lacks it or it couldn't be
516	detected, it will be emulated by calling C<fsync> instead.	572	detected, it will be emulated by calling C<fsync> instead.
517		573
518	=item aio_group $callback->(...)	574	=item aio_group $callback->(...)
519		575
520	[EXPERIMENTAL]
521
522	This is a very special aio request: Instead of doing something, it is a	576	This is a very special aio request: Instead of doing something, it is a
523	container for other aio requests, which is useful if you want to bundle	577	container for other aio requests, which is useful if you want to bundle
524	many requests into a single, composite, request.	578	many requests into a single, composite, request with a definite callback
		579	and the ability to cancel the whole request with its subrequests.
525		580
526	Returns an object of class L<IO::AIO::GRP>. See its documentation below	581	Returns an object of class L<IO::AIO::GRP>. See its documentation below
527	for more info.	582	for more info.
528		583
529	Example:	584	Example:
…		…
548	phase and still requires a worker thread. Thus, the callback will not	603	phase and still requires a worker thread. Thus, the callback will not
549	be executed immediately but only after other requests in the queue have	604	be executed immediately but only after other requests in the queue have
550	entered their execution phase. This can be used to measure request	605	entered their execution phase. This can be used to measure request
551	latency.	606	latency.
552		607
553	=item IO::AIO::aio_sleep $fractional_seconds, $callback->() NOT EXPORTED	608	=item IO::AIO::aio_busy $fractional_seconds, $callback->() NOT EXPORTED
554		609
555	Mainly used for debugging and benchmarking, this aio request puts one of	610	Mainly used for debugging and benchmarking, this aio request puts one of
556	the request workers to sleep for the given time.	611	the request workers to sleep for the given time.
557		612
558	While it is theoretically handy to have simple I/O scheduling requests	613	While it is theoretically handy to have simple I/O scheduling requests
559	like sleep and file handle readable/writable, the overhead this creates	614	like sleep and file handle readable/writable, the overhead this creates is
560	is immense, so do not use this function except to put your application	615	immense (it blocks a thread for a long time) so do not use this function
561	under artificial I/O pressure.	616	except to put your application under artificial I/O pressure.
562		617
563	=back	618	=back
564		619
565	=head2 IO::AIO::REQ CLASS	620	=head2 IO::AIO::REQ CLASS
566		621
567	All non-aggregate C<aio_*> functions return an object of this class when	622	All non-aggregate C<aio_*> functions return an object of this class when
568	called in non-void context.	623	called in non-void context.
569
570	A request always moves through the following five states in its lifetime,
571	in order: B<ready> (request has been created, but has not been executed
572	yet), B<execute> (request is currently being executed), B<pending>
573	(request has been executed but callback has not been called yet),
574	B<result> (results are being processed synchronously, includes calling the
575	callback) and B<done> (request has reached the end of its lifetime and
576	holds no resources anymore).
577		624
578	=over 4	625	=over 4
579		626
580	=item cancel $req	627	=item cancel $req
581		628
…		…
663	be added, including other groups, as long as you do not create circular	710	be added, including other groups, as long as you do not create circular
664	dependencies.	711	dependencies.
665		712
666	Returns all its arguments.	713	Returns all its arguments.
667		714
		715	=item $grp->cancel_subs
		716
		717	Cancel all subrequests and clears any feeder, but not the group request
		718	itself. Useful when you queued a lot of events but got a result early.
		719
668	=item $grp->result (...)	720	=item $grp->result (...)
669		721
670	Set the result value(s) that will be passed to the group callback when all	722	Set the result value(s) that will be passed to the group callback when all
671	subrequests have finished. By default, no argument will be passed.	723	subrequests have finished. By default, no argument will be passed.
672		724
673	=item feed $grp $callback->($grp)	725	=item feed $grp $callback->($grp)
674
675	[VERY EXPERIMENTAL]
676		726
677	Sets a feeder/generator on this group: every group can have an attached	727	Sets a feeder/generator on this group: every group can have an attached
678	generator that generates requests if idle. The idea behind this is that,	728	generator that generates requests if idle. The idea behind this is that,
679	although you could just queue as many requests as you want in a group,	729	although you could just queue as many requests as you want in a group,
680	this might starve other requests for a potentially long time. For	730	this might starve other requests for a potentially long time. For
681	example, C<aio_scandir> might generate hundreds of thousands C<aio_stat>	731	example, C<aio_scandir> might generate hundreds of thousands C<aio_stat>
682	requests, delaying any later requests for a long time.	732	requests, delaying any later requests for a long time.
683		733
684	To avoid this, and allow incremental generation of requests, you can	734	To avoid this, and allow incremental generation of requests, you can
685	instead a group and set a feeder on it that generates those requests. The	735	instead a group and set a feeder on it that generates those requests. The
686	feed callback will be called whenever there are few enough (see C<feed_limit>,	736	feed callback will be called whenever there are few enough (see C<limit>,
687	below) requests active in the group itself and is expected to queue more	737	below) requests active in the group itself and is expected to queue more
688	requests.	738	requests.
689		739
690	The feed can queue as many requests as it likes (i.e. C<add> does not	740	The feed callback can queue as many requests as it likes (i.e. C<add> does
691	impose any limits).	741	not impose any limits).
692		742
693	If the feed does not queue more requests when called, it will be	743	If the feed does not queue more requests when called, it will be
694	automatically removed from the group.	744	automatically removed from the group.
695		745
696	If the feed limit is C<0>, it will be set to C<2> automatically.	746	If the feed limit is C<0>, it will be set to C<2> automatically.
…		…
698	Example:	748	Example:
699		749
700	# stat all files in @files, but only ever use four aio requests concurrently:	750	# stat all files in @files, but only ever use four aio requests concurrently:
701		751
702	my $grp = aio_group sub { print "finished\n" };	752	my $grp = aio_group sub { print "finished\n" };
703	feed_limit $grp 4;	753	limit $grp 4;
704	feed $grp sub {	754	feed $grp sub {
705	my $file = pop @files	755	my $file = pop @files
706	or return;	756	or return;
707		757
708	add $grp aio_stat $file, sub { ... };	758	add $grp aio_stat $file, sub { ... };
709	};	759	};
710		760
711	=item feed_limit $grp $num	761	=item limit $grp $num
712		762
713	Sets the feeder limit for the group: The feeder will be called whenever	763	Sets the feeder limit for the group: The feeder will be called whenever
714	the group contains less than this many requests.	764	the group contains less than this many requests.
715		765
716	Setting the limit to C<0> will pause the feeding process.	766	Setting the limit to C<0> will pause the feeding process.
…		…
862	This module should do "the right thing" when the process using it forks:	912	This module should do "the right thing" when the process using it forks:
863		913
864	Before the fork, IO::AIO enters a quiescent state where no requests	914	Before the fork, IO::AIO enters a quiescent state where no requests
865	can be added in other threads and no results will be processed. After	915	can be added in other threads and no results will be processed. After
866	the fork the parent simply leaves the quiescent state and continues	916	the fork the parent simply leaves the quiescent state and continues
867	request/result processing, while the child clears the request/result	917	request/result processing, while the child frees the request/result queue
868	queue (so the requests started before the fork will only be handled in	918	(so that the requests started before the fork will only be handled in the
869	the parent). Threads will be started on demand until the limit ste in the	919	parent). Threads will be started on demand until the limit set in the
870	parent process has been reached again.	920	parent process has been reached again.
871		921
872	In short: the parent will, after a short pause, continue as if fork had	922	In short: the parent will, after a short pause, continue as if fork had
873	not been called, while the child will act as if IO::AIO has not been used	923	not been called, while the child will act as if IO::AIO has not been used
874	yet.	924	yet.
875		925
876	=head2 MEMORY USAGE	926	=head2 MEMORY USAGE
877		927
		928	Per-request usage:
		929
878	Each aio request uses - depending on your architecture - around 128 bytes	930	Each aio request uses - depending on your architecture - around 100-200
879	of memory. In addition, stat requests need a stat buffer (possibly a few	931	bytes of memory. In addition, stat requests need a stat buffer (possibly
880	hundred bytes). Perl scalars and other data passed into aio requests will	932	a few hundred bytes), readdir requires a result buffer and so on. Perl
881	also be locked.	933	scalars and other data passed into aio requests will also be locked and
		934	will consume memory till the request has entered the done state.
882		935
883	This is now awfully much, so queuing lots of requests is not usually a	936	This is now awfully much, so queuing lots of requests is not usually a
884	problem.	937	problem.
885		938
886	Each thread needs a stack area which is usually around 16k, sometimes much	939	Per-thread usage:
887	larger, depending on the OS.	940
		941	In the execution phase, some aio requests require more memory for
		942	temporary buffers, and each thread requires a stack and other data
		943	structures (usually around 16k-128k, depending on the OS).
		944
		945	=head1 KNOWN BUGS
		946
		947	Known bugs will be fixed in the next release.
888		948
889	=head1 SEE ALSO	949	=head1 SEE ALSO
890		950
891	L<Coro>, L<Linux::AIO> (obsolete).	951	L<Coro::AIO>.
892		952
893	=head1 AUTHOR	953	=head1 AUTHOR
894		954
895	Marc Lehmann <schmorp@schmorp.de>	955	Marc Lehmann <schmorp@schmorp.de>
896	http://home.schmorp.de/	956	http://home.schmorp.de/

Diff Legend

-–
+Removed lines
-+
+Added lines
-<
+Changed lines
->
+Changed lines

Comparing IO-AIO/AIO.pm (file contents): Revision 1.67 by root, Tue Oct 24 02:25:16 2006 UTC vs. Revision 1.74 by root, Tue Oct 24 17:22:17 2006 UTC

Diff Legend

Comparing IO-AIO/AIO.pm (file contents):
Revision 1.67 by root, Tue Oct 24 02:25:16 2006 UTC vs.
Revision 1.74 by root, Tue Oct 24 17:22:17 2006 UTC