[ViewVC] Diff of: cvs/IO-AIO/AIO.pm

Comparing IO-AIO/AIO.pm (file contents):
Revision 1.82 by root, Fri Oct 27 20:10:06 2006 UTC vs.
Revision 1.105 by root, Sun Mar 25 00:20:27 2007 UTC

…		…
5	=head1 SYNOPSIS	5	=head1 SYNOPSIS
6		6
7	use IO::AIO;	7	use IO::AIO;
8		8
9	aio_open "/etc/passwd", O_RDONLY, 0, sub {	9	aio_open "/etc/passwd", O_RDONLY, 0, sub {
10	my ($fh) = @_;	10	my $fh = shift
		11	or die "/etc/passwd: $!";
11	...	12	...
12	};	13	};
13		14
14	aio_unlink "/tmp/file", sub { };	15	aio_unlink "/tmp/file", sub { };
15		16
…		…
50		51
51	=head1 DESCRIPTION	52	=head1 DESCRIPTION
52		53
53	This module implements asynchronous I/O using whatever means your	54	This module implements asynchronous I/O using whatever means your
54	operating system supports.	55	operating system supports.
		56
		57	Asynchronous means that operations that can normally block your program
		58	(e.g. reading from disk) will be done asynchronously: the operation
		59	will still block, but you can do something else in the meantime. This
		60	is extremely useful for programs that need to stay interactive even
		61	when doing heavy I/O (GUI programs, high performance network servers
		62	etc.), but can also be used to easily do operations in parallel that are
		63	normally done sequentially, e.g. stat'ing many files, which is much faster
		64	on a RAID volume or over NFS when you do a number of stat operations
		65	concurrently.
		66
		67	While most of this works on all types of file descriptors (for example
		68	sockets), using these functions on file descriptors that support
		69	nonblocking operation (again, sockets, pipes etc.) is very inefficient or
		70	might not work (aio_read fails on sockets/pipes/fifos). Use an event loop
		71	for that (such as the L<Event\|Event> module): IO::AIO will naturally fit
		72	into such an event loop itself.
55		73
56	In this version, a number of threads are started that execute your	74	In this version, a number of threads are started that execute your
57	requests and signal their completion. You don't need thread support	75	requests and signal their completion. You don't need thread support
58	in perl, and the threads created by this module will not be visible	76	in perl, and the threads created by this module will not be visible
59	to perl. In the future, this module might make use of the native aio	77	to perl. In the future, this module might make use of the native aio
60	functions available on many operating systems. However, they are often	78	functions available on many operating systems. However, they are often
61	not well-supported or restricted (Linux doesn't allow them on normal	79	not well-supported or restricted (GNU/Linux doesn't allow them on normal
62	files currently, for example), and they would only support aio_read and	80	files currently, for example), and they would only support aio_read and
63	aio_write, so the remaining functionality would have to be implemented	81	aio_write, so the remaining functionality would have to be implemented
64	using threads anyway.	82	using threads anyway.
65		83
66	Although the module will work with in the presence of other (Perl-)	84	Although the module will work with in the presence of other (Perl-)
67	threads, it is currently not reentrant in any way, so use appropriate	85	threads, it is currently not reentrant in any way, so use appropriate
68	locking yourself, always call C<poll_cb> from within the same thread, or	86	locking yourself, always call C<poll_cb> from within the same thread, or
69	never call C<poll_cb> (or other C<aio_> functions) recursively.	87	never call C<poll_cb> (or other C<aio_> functions) recursively.
70		88
		89	=head2 EXAMPLE
		90
		91	This is a simple example that uses the Event module and loads
		92	F</etc/passwd> asynchronously:
		93
		94	use Fcntl;
		95	use Event;
		96	use IO::AIO;
		97
		98	# register the IO::AIO callback with Event
		99	Event->io (fd => IO::AIO::poll_fileno,
		100	poll => 'r',
		101	cb => \&IO::AIO::poll_cb);
		102
		103	# queue the request to open /etc/passwd
		104	aio_open "/etc/passwd", O_RDONLY, 0, sub {
		105	my $fh = shift
		106	or die "error while opening: $!";
		107
		108	# stat'ing filehandles is generally non-blocking
		109	my $size = -s $fh;
		110
		111	# queue a request to read the file
		112	my $contents;
		113	aio_read $fh, 0, $size, $contents, 0, sub {
		114	$_[0] == $size
		115	or die "short read: $!";
		116
		117	close $fh;
		118
		119	# file contents now in $contents
		120	print $contents;
		121
		122	# exit event loop and program
		123	Event::unloop;
		124	};
		125	};
		126
		127	# possibly queue up other requests, or open GUI windows,
		128	# check for sockets etc. etc.
		129
		130	# process events as long as there are some:
		131	Event::loop;
		132
71	=head1 REQUEST ANATOMY AND LIFETIME	133	=head1 REQUEST ANATOMY AND LIFETIME
72		134
73	Every C<aio_*> function creates a request. which is a C data structure not	135	Every C<aio_*> function creates a request. which is a C data structure not
74	directly visible to Perl.	136	directly visible to Perl.
75		137
…		…
116	Request has reached the end of its lifetime and holds no resources anymore	178	Request has reached the end of its lifetime and holds no resources anymore
117	(except possibly for the Perl object, but its connection to the actual	179	(except possibly for the Perl object, but its connection to the actual
118	aio request is severed and calling its methods will either do nothing or	180	aio request is severed and calling its methods will either do nothing or
119	result in a runtime error).	181	result in a runtime error).
120		182
		183	=back
		184
121	=cut	185	=cut
122		186
123	package IO::AIO;	187	package IO::AIO;
124		188
125	no warnings;	189	no warnings;
126	use strict 'vars';	190	use strict 'vars';
127		191
128	use base 'Exporter';	192	use base 'Exporter';
129		193
130	BEGIN {	194	BEGIN {
131	our $VERSION = '2.0';	195	our $VERSION = '2.33';
132		196
133	our @AIO_REQ = qw(aio_sendfile aio_read aio_write aio_open aio_close aio_stat	197	our @AIO_REQ = qw(aio_sendfile aio_read aio_write aio_open aio_close aio_stat
134	aio_lstat aio_unlink aio_rmdir aio_readdir aio_scandir aio_symlink	198	aio_lstat aio_unlink aio_rmdir aio_readdir aio_scandir aio_symlink
135	aio_fsync aio_fdatasync aio_readahead aio_rename aio_link aio_move	199	aio_readlink aio_fsync aio_fdatasync aio_readahead aio_rename aio_link
136	aio_copy aio_group aio_nop aio_mknod);	200	aio_move aio_copy aio_group aio_nop aio_mknod aio_load aio_rmtree aio_mkdir);
137	our @EXPORT = (@AIO_REQ, qw(aioreq_pri aioreq_nice));	201	our @EXPORT = (@AIO_REQ, qw(aioreq_pri aioreq_nice aio_block));
138	our @EXPORT_OK = qw(poll_fileno poll_cb poll_wait flush	202	our @EXPORT_OK = qw(poll_fileno poll_cb poll_wait flush
139	min_parallel max_parallel nreqs nready npending);	203	min_parallel max_parallel max_idle
		204	nreqs nready npending nthreads
		205	max_poll_time max_poll_reqs);
140		206
141	@IO::AIO::GRP::ISA = 'IO::AIO::REQ';	207	@IO::AIO::GRP::ISA = 'IO::AIO::REQ';
142		208
143	require XSLoader;	209	require XSLoader;
144	XSLoader::load ("IO::AIO", $VERSION);	210	XSLoader::load ("IO::AIO", $VERSION);
145	}	211	}
146		212
147	=head1 FUNCTIONS	213	=head1 FUNCTIONS
148		214
149	=head2 AIO FUNCTIONS	215	=head2 AIO REQUEST FUNCTIONS
150		216
151	All the C<aio_*> calls are more or less thin wrappers around the syscall	217	All the C<aio_*> calls are more or less thin wrappers around the syscall
152	with the same name (sans C<aio_>). The arguments are similar or identical,	218	with the same name (sans C<aio_>). The arguments are similar or identical,
153	and they all accept an additional (and optional) C<$callback> argument	219	and they all accept an additional (and optional) C<$callback> argument
154	which must be a code reference. This code reference will get called with	220	which must be a code reference. This code reference will get called with
…		…
157	syscall has been executed asynchronously.	223	syscall has been executed asynchronously.
158		224
159	All functions expecting a filehandle keep a copy of the filehandle	225	All functions expecting a filehandle keep a copy of the filehandle
160	internally until the request has finished.	226	internally until the request has finished.
161		227
162	All requests return objects of type L<IO::AIO::REQ> that allow further	228	All functions return request objects of type L<IO::AIO::REQ> that allow
163	manipulation of those requests while they are in-flight.	229	further manipulation of those requests while they are in-flight.
164		230
165	The pathnames you pass to these routines I<must> be absolute and	231	The pathnames you pass to these routines I<must> be absolute and
166	encoded in byte form. The reason for the former is that at the time the	232	encoded as octets. The reason for the former is that at the time the
167	request is being executed, the current working directory could have	233	request is being executed, the current working directory could have
168	changed. Alternatively, you can make sure that you never change the	234	changed. Alternatively, you can make sure that you never change the
169	current working directory.	235	current working directory anywhere in the program and then use relative
		236	paths.
170		237
171	To encode pathnames to byte form, either make sure you either: a)	238	To encode pathnames as octets, either make sure you either: a) always pass
172	always pass in filenames you got from outside (command line, readdir	239	in filenames you got from outside (command line, readdir etc.) without
173	etc.), b) are ASCII or ISO 8859-1, c) use the Encode module and encode	240	tinkering, b) are ASCII or ISO 8859-1, c) use the Encode module and encode
174	your pathnames to the locale (or other) encoding in effect in the user	241	your pathnames to the locale (or other) encoding in effect in the user
175	environment, d) use Glib::filename_from_unicode on unicode filenames or e)	242	environment, d) use Glib::filename_from_unicode on unicode filenames or e)
176	use something else.	243	use something else to ensure your scalar has the correct contents.
		244
		245	This works, btw. independent of the internal UTF-8 bit, which IO::AIO
		246	handles correctly wether it is set or not.
177		247
178	=over 4	248	=over 4
179		249
180	=item $prev_pri = aioreq_pri [$pri]	250	=item $prev_pri = aioreq_pri [$pri]
181		251
…		…
204	};	274	};
205		275
206	=item aioreq_nice $pri_adjust	276	=item aioreq_nice $pri_adjust
207		277
208	Similar to C<aioreq_pri>, but subtracts the given value from the current	278	Similar to C<aioreq_pri>, but subtracts the given value from the current
209	priority, so effects are cumulative.	279	priority, so the effect is cumulative.
210		280
211	=item aio_open $pathname, $flags, $mode, $callback->($fh)	281	=item aio_open $pathname, $flags, $mode, $callback->($fh)
212		282
213	Asynchronously open or create a file and call the callback with a newly	283	Asynchronously open or create a file and call the callback with a newly
214	created filehandle for the file.	284	created filehandle for the file.
…		…
220	list. They are the same as used by C<sysopen>.	290	list. They are the same as used by C<sysopen>.
221		291
222	Likewise, C<$mode> specifies the mode of the newly created file, if it	292	Likewise, C<$mode> specifies the mode of the newly created file, if it
223	didn't exist and C<O_CREAT> has been given, just like perl's C<sysopen>,	293	didn't exist and C<O_CREAT> has been given, just like perl's C<sysopen>,
224	except that it is mandatory (i.e. use C<0> if you don't create new files,	294	except that it is mandatory (i.e. use C<0> if you don't create new files,
225	and C<0666> or C<0777> if you do).	295	and C<0666> or C<0777> if you do). Note that the C<$mode> will be modified
		296	by the umask in effect then the request is being executed, so better never
		297	change the umask.
226		298
227	Example:	299	Example:
228		300
229	aio_open "/etc/passwd", O_RDONLY, 0, sub {	301	aio_open "/etc/passwd", O_RDONLY, 0, sub {
230	if ($_[0]) {	302	if ($_[0]) {
…		…
331	Asynchronously unlink (delete) a file and call the callback with the	403	Asynchronously unlink (delete) a file and call the callback with the
332	result code.	404	result code.
333		405
334	=item aio_mknod $path, $mode, $dev, $callback->($status)	406	=item aio_mknod $path, $mode, $dev, $callback->($status)
335		407
		408	[EXPERIMENTAL]
		409
336	Asynchronously create a device node (or fifo). See mknod(2): the only	410	Asynchronously create a device node (or fifo). See mknod(2).
337	portable value for C<$mode> is C<S_IFIFO> ored with permissions, and C<0>	411
338	for C<$dev>.	412	The only (POSIX-) portable way of calling this function is:
		413
		414	aio_mknod $path, IO::AIO::S_IFIFO \| $mode, 0, sub { ...
339		415
340	=item aio_link $srcpath, $dstpath, $callback->($status)	416	=item aio_link $srcpath, $dstpath, $callback->($status)
341		417
342	Asynchronously create a new link to the existing object at C<$srcpath> at	418	Asynchronously create a new link to the existing object at C<$srcpath> at
343	the path C<$dstpath> and call the callback with the result code.	419	the path C<$dstpath> and call the callback with the result code.
…		…
345	=item aio_symlink $srcpath, $dstpath, $callback->($status)	421	=item aio_symlink $srcpath, $dstpath, $callback->($status)
346		422
347	Asynchronously create a new symbolic link to the existing object at C<$srcpath> at	423	Asynchronously create a new symbolic link to the existing object at C<$srcpath> at
348	the path C<$dstpath> and call the callback with the result code.	424	the path C<$dstpath> and call the callback with the result code.
349		425
		426	=item aio_readlink $path, $callback->($link)
		427
		428	Asynchronously read the symlink specified by C<$path> and pass it to
		429	the callback. If an error occurs, nothing or undef gets passed to the
		430	callback.
		431
350	=item aio_rename $srcpath, $dstpath, $callback->($status)	432	=item aio_rename $srcpath, $dstpath, $callback->($status)
351		433
352	Asynchronously rename the object at C<$srcpath> to C<$dstpath>, just as	434	Asynchronously rename the object at C<$srcpath> to C<$dstpath>, just as
353	rename(2) and call the callback with the result code.	435	rename(2) and call the callback with the result code.
		436
		437	=item aio_mkdir $pathname, $mode, $callback->($status)
		438
		439	Asynchronously mkdir (create) a directory and call the callback with
		440	the result code. C<$mode> will be modified by the umask at the time the
		441	request is executed, so do not change your umask.
354		442
355	=item aio_rmdir $pathname, $callback->($status)	443	=item aio_rmdir $pathname, $callback->($status)
356		444
357	Asynchronously rmdir (delete) a directory and call the callback with the	445	Asynchronously rmdir (delete) a directory and call the callback with the
358	result code.	446	result code.
…		…
363	directory (i.e. opendir + readdir + closedir). The entries will not be	451	directory (i.e. opendir + readdir + closedir). The entries will not be
364	sorted, and will B<NOT> include the C<.> and C<..> entries.	452	sorted, and will B<NOT> include the C<.> and C<..> entries.
365		453
366	The callback a single argument which is either C<undef> or an array-ref	454	The callback a single argument which is either C<undef> or an array-ref
367	with the filenames.	455	with the filenames.
		456
		457	=item aio_load $path, $data, $callback->($status)
		458
		459	This is a composite request that tries to fully load the given file into
		460	memory. Status is the same as with aio_read.
		461
		462	=cut
		463
		464	sub aio_load($$;$) {
		465	aio_block {
		466	my ($path, undef, $cb) = @_;
		467	my $data = \$_[1];
		468
		469	my $pri = aioreq_pri;
		470	my $grp = aio_group $cb;
		471
		472	aioreq_pri $pri;
		473	add $grp aio_open $path, O_RDONLY, 0, sub {
		474	my $fh = shift
		475	or return $grp->result (-1);
		476
		477	aioreq_pri $pri;
		478	add $grp aio_read $fh, 0, (-s $fh), $$data, 0, sub {
		479	$grp->result ($_[0]);
		480	};
		481	};
		482
		483	$grp
		484	}
		485	}
368		486
369	=item aio_copy $srcpath, $dstpath, $callback->($status)	487	=item aio_copy $srcpath, $dstpath, $callback->($status)
370		488
371	Try to copy the I<file> (directories not supported as either source or	489	Try to copy the I<file> (directories not supported as either source or
372	destination) from C<$srcpath> to C<$dstpath> and call the callback with	490	destination) from C<$srcpath> to C<$dstpath> and call the callback with
…		…
382	errors are being ignored.	500	errors are being ignored.
383		501
384	=cut	502	=cut
385		503
386	sub aio_copy($$;$) {	504	sub aio_copy($$;$) {
		505	aio_block {
387	my ($src, $dst, $cb) = @_;	506	my ($src, $dst, $cb) = @_;
388		507
389	my $pri = aioreq_pri;	508	my $pri = aioreq_pri;
390	my $grp = aio_group $cb;	509	my $grp = aio_group $cb;
391		510
392	aioreq_pri $pri;	511	aioreq_pri $pri;
393	add $grp aio_open $src, O_RDONLY, 0, sub {	512	add $grp aio_open $src, O_RDONLY, 0, sub {
394	if (my $src_fh = $_[0]) {	513	if (my $src_fh = $_[0]) {
395	my @stat = stat $src_fh;	514	my @stat = stat $src_fh;
396		515
397	aioreq_pri $pri;	516	aioreq_pri $pri;
398	add $grp aio_open $dst, O_CREAT \| O_WRONLY \| O_TRUNC, 0200, sub {	517	add $grp aio_open $dst, O_CREAT \| O_WRONLY \| O_TRUNC, 0200, sub {
399	if (my $dst_fh = $_[0]) {	518	if (my $dst_fh = $_[0]) {
400	aioreq_pri $pri;	519	aioreq_pri $pri;
401	add $grp aio_sendfile $dst_fh, $src_fh, 0, $stat[7], sub {	520	add $grp aio_sendfile $dst_fh, $src_fh, 0, $stat[7], sub {
402	if ($_[0] == $stat[7]) {	521	if ($_[0] == $stat[7]) {
403	$grp->result (0);	522	$grp->result (0);
404	close $src_fh;	523	close $src_fh;
405		524
406	# those should not normally block. should. should.	525	# those should not normally block. should. should.
407	utime $stat[8], $stat[9], $dst;	526	utime $stat[8], $stat[9], $dst;
408	chmod $stat[2] & 07777, $dst_fh;	527	chmod $stat[2] & 07777, $dst_fh;
409	chown $stat[4], $stat[5], $dst_fh;	528	chown $stat[4], $stat[5], $dst_fh;
410	close $dst_fh;	529	close $dst_fh;
411	} else {	530	} else {
412	$grp->result (-1);	531	$grp->result (-1);
413	close $src_fh;	532	close $src_fh;
414	close $dst_fh;	533	close $dst_fh;
415		534
416	aioreq $pri;	535	aioreq $pri;
417	add $grp aio_unlink $dst;	536	add $grp aio_unlink $dst;
		537	}
418	}	538	};
		539	} else {
		540	$grp->result (-1);
419	};	541	}
420	} else {
421	$grp->result (-1);
422	}	542	},
		543
		544	} else {
		545	$grp->result (-1);
423	},	546	}
424
425	} else {
426	$grp->result (-1);
427	}	547	};
		548
		549	$grp
428	};	550	}
429
430	$grp
431	}	551	}
432		552
433	=item aio_move $srcpath, $dstpath, $callback->($status)	553	=item aio_move $srcpath, $dstpath, $callback->($status)
434		554
435	Try to move the I<file> (directories not supported as either source or	555	Try to move the I<file> (directories not supported as either source or
…		…
441	that is successful, unlinking the C<$srcpath>.	561	that is successful, unlinking the C<$srcpath>.
442		562
443	=cut	563	=cut
444		564
445	sub aio_move($$;$) {	565	sub aio_move($$;$) {
		566	aio_block {
446	my ($src, $dst, $cb) = @_;	567	my ($src, $dst, $cb) = @_;
447		568
448	my $pri = aioreq_pri;	569	my $pri = aioreq_pri;
449	my $grp = aio_group $cb;	570	my $grp = aio_group $cb;
450		571
451	aioreq_pri $pri;	572	aioreq_pri $pri;
452	add $grp aio_rename $src, $dst, sub {	573	add $grp aio_rename $src, $dst, sub {
453	if ($_[0] && $! == EXDEV) {	574	if ($_[0] && $! == EXDEV) {
454	aioreq_pri $pri;	575	aioreq_pri $pri;
455	add $grp aio_copy $src, $dst, sub {	576	add $grp aio_copy $src, $dst, sub {
		577	$grp->result ($_[0]);
		578
		579	if (!$_[0]) {
		580	aioreq_pri $pri;
		581	add $grp aio_unlink $src;
		582	}
		583	};
		584	} else {
456	$grp->result ($_[0]);	585	$grp->result ($_[0]);
457
458	if (!$_[0]) {
459	aioreq_pri $pri;
460	add $grp aio_unlink $src;
461	}
462	};	586	}
463	} else {
464	$grp->result ($_[0]);
465	}	587	};
		588
		589	$grp
466	};	590	}
467
468	$grp
469	}	591	}
470		592
471	=item aio_scandir $path, $maxreq, $callback->($dirs, $nondirs)	593	=item aio_scandir $path, $maxreq, $callback->($dirs, $nondirs)
472		594
473	Scans a directory (similar to C<aio_readdir>) but additionally tries to	595	Scans a directory (similar to C<aio_readdir>) but additionally tries to
…		…
520	as those tend to return 0 or 1 as link counts, which disables the	642	as those tend to return 0 or 1 as link counts, which disables the
521	directory counting heuristic.	643	directory counting heuristic.
522		644
523	=cut	645	=cut
524		646
525	sub aio_scandir($$$) {	647	sub aio_scandir($$;$) {
		648	aio_block {
526	my ($path, $maxreq, $cb) = @_;	649	my ($path, $maxreq, $cb) = @_;
527		650
528	my $pri = aioreq_pri;	651	my $pri = aioreq_pri;
529		652
530	my $grp = aio_group $cb;	653	my $grp = aio_group $cb;
531		654
532	$maxreq = 4 if $maxreq <= 0;	655	$maxreq = 4 if $maxreq <= 0;
533		656
534	# stat once	657	# stat once
535	aioreq_pri $pri;
536	add $grp aio_stat $path, sub {
537	return $grp->result () if $_[0];
538	my $now = time;
539	my $hash1 = join ":", (stat _)[0,1,3,7,9];
540
541	# read the directory entries
542	aioreq_pri $pri;	658	aioreq_pri $pri;
543	add $grp aio_readdir $path, sub {	659	add $grp aio_stat $path, sub {
544	my $entries = shift
545	or return $grp->result ();	660	return $grp->result () if $_[0];
		661	my $now = time;
		662	my $hash1 = join ":", (stat _)[0,1,3,7,9];
546		663
547	# stat the dir another time	664	# read the directory entries
548	aioreq_pri $pri;	665	aioreq_pri $pri;
		666	add $grp aio_readdir $path, sub {
		667	my $entries = shift
		668	or return $grp->result ();
		669
		670	# stat the dir another time
		671	aioreq_pri $pri;
549	add $grp aio_stat $path, sub {	672	add $grp aio_stat $path, sub {
550	my $hash2 = join ":", (stat _)[0,1,3,7,9];	673	my $hash2 = join ":", (stat _)[0,1,3,7,9];
551		674
552	my $ndirs;	675	my $ndirs;
553		676
554	# take the slow route if anything looks fishy	677	# take the slow route if anything looks fishy
555	if ($hash1 ne $hash2 or (stat _)[9] == $now) {	678	if ($hash1 ne $hash2 or (stat _)[9] == $now) {
556	$ndirs = -1;	679	$ndirs = -1;
557	} else {	680	} else {
558	# if nlink == 2, we are finished	681	# if nlink == 2, we are finished
559	# on non-posix-fs's, we rely on nlink < 2	682	# on non-posix-fs's, we rely on nlink < 2
560	$ndirs = (stat _)[3] - 2	683	$ndirs = (stat _)[3] - 2
561	or return $grp->result ([], $entries);	684	or return $grp->result ([], $entries);
562	}	685	}
563		686
564	# sort into likely dirs and likely nondirs	687	# sort into likely dirs and likely nondirs
565	# dirs == files without ".", short entries first	688	# dirs == files without ".", short entries first
566	$entries = [map $_->[0],	689	$entries = [map $_->[0],
567	sort { $b->[1] cmp $a->[1] }	690	sort { $b->[1] cmp $a->[1] }
568	map [$_, sprintf "%s%04d", (/.\./ ? "1" : "0"), length],	691	map [$_, sprintf "%s%04d", (/.\./ ? "1" : "0"), length],
569	@$entries];	692	@$entries];
570		693
571	my (@dirs, @nondirs);	694	my (@dirs, @nondirs);
572		695
573	my $statgrp = add $grp aio_group sub {	696	my $statgrp = add $grp aio_group sub {
574	$grp->result (\@dirs, \@nondirs);	697	$grp->result (\@dirs, \@nondirs);
575	};	698	};
576		699
577	limit $statgrp $maxreq;	700	limit $statgrp $maxreq;
578	feed $statgrp sub {	701	feed $statgrp sub {
579	return unless @$entries;	702	return unless @$entries;
580	my $entry = pop @$entries;	703	my $entry = pop @$entries;
581		704
582	aioreq_pri $pri;	705	aioreq_pri $pri;
583	add $statgrp aio_stat "$path/$entry/.", sub {	706	add $statgrp aio_stat "$path/$entry/.", sub {
584	if ($_[0] < 0) {	707	if ($_[0] < 0) {
585	push @nondirs, $entry;	708	push @nondirs, $entry;
586	} else {	709	} else {
587	# need to check for real directory	710	# need to check for real directory
588	aioreq_pri $pri;	711	aioreq_pri $pri;
589	add $statgrp aio_lstat "$path/$entry", sub {	712	add $statgrp aio_lstat "$path/$entry", sub {
590	if (-d _) {	713	if (-d _) {
591	push @dirs, $entry;	714	push @dirs, $entry;
592		715
593	unless (--$ndirs) {	716	unless (--$ndirs) {
594	push @nondirs, @$entries;	717	push @nondirs, @$entries;
595	feed $statgrp;	718	feed $statgrp;
		719	}
		720	} else {
		721	push @nondirs, $entry;
596	}	722	}
597	} else {
598	push @nondirs, $entry;
599	}	723	}
600	}	724	}
601	}	725	};
602	};	726	};
603	};	727	};
604	};	728	};
605	};	729	};
		730
		731	$grp
606	};	732	}
		733	}
607		734
		735	=item aio_rmtree $path, $callback->($status)
		736
		737	Delete a directory tree starting (and including) C<$path>, return the
		738	status of the final C<rmdir> only. This is a composite request that
		739	uses C<aio_scandir> to recurse into and rmdir directories, and unlink
		740	everything else.
		741
		742	=cut
		743
		744	sub aio_rmtree;
		745	sub aio_rmtree($;$) {
		746	aio_block {
		747	my ($path, $cb) = @_;
		748
		749	my $pri = aioreq_pri;
		750	my $grp = aio_group $cb;
		751
		752	aioreq_pri $pri;
		753	add $grp aio_scandir $path, 0, sub {
		754	my ($dirs, $nondirs) = @_;
		755
		756	my $dirgrp = aio_group sub {
		757	add $grp aio_rmdir $path, sub {
		758	$grp->result ($_[0]);
		759	};
		760	};
		761
		762	(aioreq_pri $pri), add $dirgrp aio_rmtree "$path/$_" for @$dirs;
		763	(aioreq_pri $pri), add $dirgrp aio_unlink "$path/$_" for @$nondirs;
		764
		765	add $grp $dirgrp;
		766	};
		767
608	$grp	768	$grp
		769	}
609	}	770	}
610		771
611	=item aio_fsync $fh, $callback->($status)	772	=item aio_fsync $fh, $callback->($status)
612		773
613	Asynchronously call fsync on the given filehandle and call the callback	774	Asynchronously call fsync on the given filehandle and call the callback
…		…
827		988
828	=back	989	=back
829		990
830	=head2 SUPPORT FUNCTIONS	991	=head2 SUPPORT FUNCTIONS
831		992
		993	=head3 EVENT PROCESSING AND EVENT LOOP INTEGRATION
		994
832	=over 4	995	=over 4
833		996
834	=item $fileno = IO::AIO::poll_fileno	997	=item $fileno = IO::AIO::poll_fileno
835		998
836	Return the I<request result pipe file descriptor>. This filehandle must be	999	Return the I<request result pipe file descriptor>. This filehandle must be
…		…
840		1003
841	See C<poll_cb> for an example.	1004	See C<poll_cb> for an example.
842		1005
843	=item IO::AIO::poll_cb	1006	=item IO::AIO::poll_cb
844		1007
845	Process all outstanding events on the result pipe. You have to call this	1008	Process some outstanding events on the result pipe. You have to call this
846	regularly. Returns the number of events processed. Returns immediately	1009	regularly. Returns the number of events processed. Returns immediately
847	when no events are outstanding.	1010	when no events are outstanding. The amount of events processed depends on
		1011	the settings of C<IO::AIO::max_poll_req> and C<IO::AIO::max_poll_time>.
848		1012
849	If not all requests were processed for whatever reason, the filehandle	1013	If not all requests were processed for whatever reason, the filehandle
850	will still be ready when C<poll_cb> returns.	1014	will still be ready when C<poll_cb> returns.
851		1015
852	Example: Install an Event watcher that automatically calls	1016	Example: Install an Event watcher that automatically calls
…		…
854		1018
855	Event->io (fd => IO::AIO::poll_fileno,	1019	Event->io (fd => IO::AIO::poll_fileno,
856	poll => 'r', async => 1,	1020	poll => 'r', async => 1,
857	cb => \&IO::AIO::poll_cb);	1021	cb => \&IO::AIO::poll_cb);
858		1022
859	=item IO::AIO::poll_some $max_requests	1023	=item IO::AIO::max_poll_reqs $nreqs
860		1024
861	Similar to C<poll_cb>, but only processes up to C<$max_requests> requests	1025	=item IO::AIO::max_poll_time $seconds
862	at a time.
863		1026
864	Useful if you want to ensure some level of interactiveness when perl is	1027	These set the maximum number of requests (default C<0>, meaning infinity)
865	not fast enough to process all requests in time.	1028	that are being processed by C<IO::AIO::poll_cb> in one call, respectively
		1029	the maximum amount of time (default C<0>, meaning infinity) spent in
		1030	C<IO::AIO::poll_cb> to process requests (more correctly the mininum amount
		1031	of time C<poll_cb> is allowed to use).
		1032
		1033	Setting C<max_poll_time> to a non-zero value creates an overhead of one
		1034	syscall per request processed, which is not normally a problem unless your
		1035	callbacks are really really fast or your OS is really really slow (I am
		1036	not mentioning Solaris here). Using C<max_poll_reqs> incurs no overhead.
		1037
		1038	Setting these is useful if you want to ensure some level of
		1039	interactiveness when perl is not fast enough to process all requests in
		1040	time.
		1041
		1042	For interactive programs, values such as C<0.01> to C<0.1> should be fine.
866		1043
867	Example: Install an Event watcher that automatically calls	1044	Example: Install an Event watcher that automatically calls
868	IO::AIO::poll_some with low priority, to ensure that other parts of the	1045	IO::AIO::poll_cb with low priority, to ensure that other parts of the
869	program get the CPU sometimes even under high AIO load.	1046	program get the CPU sometimes even under high AIO load.
870		1047
		1048	# try not to spend much more than 0.1s in poll_cb
		1049	IO::AIO::max_poll_time 0.1;
		1050
		1051	# use a low priority so other tasks have priority
871	Event->io (fd => IO::AIO::poll_fileno,	1052	Event->io (fd => IO::AIO::poll_fileno,
872	poll => 'r', nice => 1,	1053	poll => 'r', nice => 1,
873	cb => sub { IO::AIO::poll_some 256 });	1054	cb => &IO::AIO::poll_cb);
874		1055
875	=item IO::AIO::poll_wait	1056	=item IO::AIO::poll_wait
876		1057
		1058	If there are any outstanding requests and none of them in the result
877	Wait till the result filehandle becomes ready for reading (simply does a	1059	phase, wait till the result filehandle becomes ready for reading (simply
878	C<select> on the filehandle. This is useful if you want to synchronously wait	1060	does a C<select> on the filehandle. This is useful if you want to
879	for some requests to finish).	1061	synchronously wait for some requests to finish).
880		1062
881	See C<nreqs> for an example.	1063	See C<nreqs> for an example.
882		1064
		1065	=item IO::AIO::poll
		1066
		1067	Waits until some requests have been handled.
		1068
		1069	Returns the number of requests processed, but is otherwise strictly
		1070	equivalent to:
		1071
		1072	IO::AIO::poll_wait, IO::AIO::poll_cb
		1073
883	=item IO::AIO::nreqs	1074	=item IO::AIO::flush
884		1075
885	Returns the number of requests currently in the ready, execute or pending	1076	Wait till all outstanding AIO requests have been handled.
886	states (i.e. for which their callback has not been invoked yet).
887		1077
888	Example: wait till there are no outstanding requests anymore:	1078	Strictly equivalent to:
889		1079
890	IO::AIO::poll_wait, IO::AIO::poll_cb	1080	IO::AIO::poll_wait, IO::AIO::poll_cb
891	while IO::AIO::nreqs;	1081	while IO::AIO::nreqs;
892		1082
893	=item IO::AIO::nready	1083	=back
894		1084
895	Returns the number of requests currently in the ready state (not yet	1085	=head3 CONTROLLING THE NUMBER OF THREADS
896	executed).
897		1086
898	=item IO::AIO::npending	1087	=over
899
900	Returns the number of requests currently in the pending state (executed,
901	but not yet processed by poll_cb).
902
903	=item IO::AIO::flush
904
905	Wait till all outstanding AIO requests have been handled.
906
907	Strictly equivalent to:
908
909	IO::AIO::poll_wait, IO::AIO::poll_cb
910	while IO::AIO::nreqs;
911
912	=item IO::AIO::poll
913
914	Waits until some requests have been handled.
915
916	Strictly equivalent to:
917
918	IO::AIO::poll_wait, IO::AIO::poll_cb
919	if IO::AIO::nreqs;
920		1088
921	=item IO::AIO::min_parallel $nthreads	1089	=item IO::AIO::min_parallel $nthreads
922		1090
923	Set the minimum number of AIO threads to C<$nthreads>. The current	1091	Set the minimum number of AIO threads to C<$nthreads>. The current
924	default is C<8>, which means eight asynchronous operations can execute	1092	default is C<8>, which means eight asynchronous operations can execute
925	concurrently at any one time (the number of outstanding requests,	1093	concurrently at any one time (the number of outstanding requests,
926	however, is unlimited).	1094	however, is unlimited).
927		1095
928	IO::AIO starts threads only on demand, when an AIO request is queued and	1096	IO::AIO starts threads only on demand, when an AIO request is queued and
929	no free thread exists.	1097	no free thread exists. Please note that queueing up a hundred requests can
		1098	create demand for a hundred threads, even if it turns out that everything
		1099	is in the cache and could have been processed faster by a single thread.
930		1100
931	It is recommended to keep the number of threads relatively low, as some	1101	It is recommended to keep the number of threads relatively low, as some
932	Linux kernel versions will scale negatively with the number of threads	1102	Linux kernel versions will scale negatively with the number of threads
933	(higher parallelity => MUCH higher latency). With current Linux 2.6	1103	(higher parallelity => MUCH higher latency). With current Linux 2.6
934	versions, 4-32 threads should be fine.	1104	versions, 4-32 threads should be fine.
…		…
948	This module automatically runs C<max_parallel 0> at program end, to ensure	1118	This module automatically runs C<max_parallel 0> at program end, to ensure
949	that all threads are killed and that there are no outstanding requests.	1119	that all threads are killed and that there are no outstanding requests.
950		1120
951	Under normal circumstances you don't need to call this function.	1121	Under normal circumstances you don't need to call this function.
952		1122
		1123	=item IO::AIO::max_idle $nthreads
		1124
		1125	Limit the number of threads (default: 4) that are allowed to idle (i.e.,
		1126	threads that did not get a request to process within 10 seconds). That
		1127	means if a thread becomes idle while C<$nthreads> other threads are also
		1128	idle, it will free its resources and exit.
		1129
		1130	This is useful when you allow a large number of threads (e.g. 100 or 1000)
		1131	to allow for extremely high load situations, but want to free resources
		1132	under normal circumstances (1000 threads can easily consume 30MB of RAM).
		1133
		1134	The default is probably ok in most situations, especially if thread
		1135	creation is fast. If thread creation is very slow on your system you might
		1136	want to use larger values.
		1137
953	=item $oldmaxreqs = IO::AIO::max_outstanding $maxreqs	1138	=item $oldmaxreqs = IO::AIO::max_outstanding $maxreqs
954		1139
955	This is a very bad function to use in interactive programs because it	1140	This is a very bad function to use in interactive programs because it
956	blocks, and a bad way to reduce concurrency because it is inexact: Better	1141	blocks, and a bad way to reduce concurrency because it is inexact: Better
957	use an C<aio_group> together with a feed callback.	1142	use an C<aio_group> together with a feed callback.
…		…
968	C<max_oustsanding> is mainly useful in simple scripts (with low values) or	1153	C<max_oustsanding> is mainly useful in simple scripts (with low values) or
969	as a stop gap to shield against fatal memory overflow (with large values).	1154	as a stop gap to shield against fatal memory overflow (with large values).
970		1155
971	=back	1156	=back
972		1157
		1158	=head3 STATISTICAL INFORMATION
		1159
		1160	=over
		1161
		1162	=item IO::AIO::nreqs
		1163
		1164	Returns the number of requests currently in the ready, execute or pending
		1165	states (i.e. for which their callback has not been invoked yet).
		1166
		1167	Example: wait till there are no outstanding requests anymore:
		1168
		1169	IO::AIO::poll_wait, IO::AIO::poll_cb
		1170	while IO::AIO::nreqs;
		1171
		1172	=item IO::AIO::nready
		1173
		1174	Returns the number of requests currently in the ready state (not yet
		1175	executed).
		1176
		1177	=item IO::AIO::npending
		1178
		1179	Returns the number of requests currently in the pending state (executed,
		1180	but not yet processed by poll_cb).
		1181
		1182	=back
		1183
973	=cut	1184	=cut
974		1185
975	# support function to convert a fd into a perl filehandle	1186	# support function to convert a fd into a perl filehandle
976	sub _fd2fh {	1187	sub _fd2fh {
977	return undef if $_[0] < 0;	1188	return undef if $_[0] < 0;
…		…
988	*$sym	1199	*$sym
989	}	1200	}
990		1201
991	min_parallel 8;	1202	min_parallel 8;
992		1203
993	END {	1204	END { flush }
994	flush;
995	};
996		1205
997	1;	1206	1;
998		1207
999	=head2 FORK BEHAVIOUR	1208	=head2 FORK BEHAVIOUR
1000		1209

Diff Legend

-–
+Removed lines
-+
+Added lines
-<
+Changed lines
->
+Changed lines

Comparing IO-AIO/AIO.pm (file contents): Revision 1.82 by root, Fri Oct 27 20:10:06 2006 UTC vs. Revision 1.105 by root, Sun Mar 25 00:20:27 2007 UTC

Diff Legend

Comparing IO-AIO/AIO.pm (file contents):
Revision 1.82 by root, Fri Oct 27 20:10:06 2006 UTC vs.
Revision 1.105 by root, Sun Mar 25 00:20:27 2007 UTC