ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/IO-AIO/AIO.pm
(Generate patch)

Comparing IO-AIO/AIO.pm (file contents):
Revision 1.255 by root, Thu Jun 25 15:20:11 2015 UTC vs.
Revision 1.268 by root, Tue May 30 22:45:52 2017 UTC

1=head1 NAME 1=head1 NAME
2 2
3IO::AIO - Asynchronous Input/Output 3IO::AIO - Asynchronous/Advanced Input/Output
4 4
5=head1 SYNOPSIS 5=head1 SYNOPSIS
6 6
7 use IO::AIO; 7 use IO::AIO;
8 8
58not well-supported or restricted (GNU/Linux doesn't allow them on normal 58not well-supported or restricted (GNU/Linux doesn't allow them on normal
59files currently, for example), and they would only support aio_read and 59files currently, for example), and they would only support aio_read and
60aio_write, so the remaining functionality would have to be implemented 60aio_write, so the remaining functionality would have to be implemented
61using threads anyway. 61using threads anyway.
62 62
63In addition to asynchronous I/O, this module also exports some rather
64arcane interfaces, such as C<madvise> or linux's C<splice> system call,
65which is why the C<A> in C<AIO> can also mean I<advanced>.
66
63Although the module will work in the presence of other (Perl-) threads, 67Although the module will work in the presence of other (Perl-) threads,
64it is currently not reentrant in any way, so use appropriate locking 68it is currently not reentrant in any way, so use appropriate locking
65yourself, always call C<poll_cb> from within the same thread, or never 69yourself, always call C<poll_cb> from within the same thread, or never
66call C<poll_cb> (or other C<aio_> functions) recursively. 70call C<poll_cb> (or other C<aio_> functions) recursively.
67 71
94 98
95 # file contents now in $contents 99 # file contents now in $contents
96 print $contents; 100 print $contents;
97 101
98 # exit event loop and program 102 # exit event loop and program
99 EV::unloop; 103 EV::break;
100 }; 104 };
101 }; 105 };
102 106
103 # possibly queue up other requests, or open GUI windows, 107 # possibly queue up other requests, or open GUI windows,
104 # check for sockets etc. etc. 108 # check for sockets etc. etc.
105 109
106 # process events as long as there are some: 110 # process events as long as there are some:
107 EV::loop; 111 EV::run;
108 112
109=head1 REQUEST ANATOMY AND LIFETIME 113=head1 REQUEST ANATOMY AND LIFETIME
110 114
111Every C<aio_*> function creates a request. which is a C data structure not 115Every C<aio_*> function creates a request. which is a C data structure not
112directly visible to Perl. 116directly visible to Perl.
167use common::sense; 171use common::sense;
168 172
169use base 'Exporter'; 173use base 'Exporter';
170 174
171BEGIN { 175BEGIN {
172 our $VERSION = 4.32; 176 our $VERSION = 4.35;
173 177
174 our @AIO_REQ = qw(aio_sendfile aio_seek aio_read aio_write aio_open aio_close 178 our @AIO_REQ = qw(aio_sendfile aio_seek aio_read aio_write aio_open aio_close
175 aio_stat aio_lstat aio_unlink aio_rmdir aio_readdir aio_readdirx 179 aio_stat aio_lstat aio_unlink aio_rmdir aio_readdir aio_readdirx
176 aio_scandir aio_symlink aio_readlink aio_realpath aio_sync 180 aio_scandir aio_symlink aio_readlink aio_realpath aio_fcntl aio_ioctl
177 aio_fsync aio_syncfs aio_fdatasync aio_sync_file_range aio_allocate 181 aio_sync aio_fsync aio_syncfs aio_fdatasync aio_sync_file_range
178 aio_pathsync aio_readahead aio_fiemap 182 aio_pathsync aio_readahead aio_fiemap aio_allocate
179 aio_rename aio_link aio_move aio_copy aio_group 183 aio_rename aio_link aio_move aio_copy aio_group
180 aio_nop aio_mknod aio_load aio_rmtree aio_mkdir aio_chown 184 aio_nop aio_mknod aio_load aio_rmtree aio_mkdir aio_chown
181 aio_chmod aio_utime aio_truncate 185 aio_chmod aio_utime aio_truncate
182 aio_msync aio_mtouch aio_mlock aio_mlockall 186 aio_msync aio_mtouch aio_mlock aio_mlockall
183 aio_statvfs 187 aio_statvfs
240 aio_scandir $pathname, $maxreq, $callback->($dirs, $nondirs) 244 aio_scandir $pathname, $maxreq, $callback->($dirs, $nondirs)
241 aio_load $pathname, $data, $callback->($status) 245 aio_load $pathname, $data, $callback->($status)
242 aio_copy $srcpath, $dstpath, $callback->($status) 246 aio_copy $srcpath, $dstpath, $callback->($status)
243 aio_move $srcpath, $dstpath, $callback->($status) 247 aio_move $srcpath, $dstpath, $callback->($status)
244 aio_rmtree $pathname, $callback->($status) 248 aio_rmtree $pathname, $callback->($status)
249 aio_fcntl $fh, $cmd, $arg, $callback->($status)
250 aio_ioctl $fh, $request, $buf, $callback->($status)
245 aio_sync $callback->($status) 251 aio_sync $callback->($status)
246 aio_syncfs $fh, $callback->($status) 252 aio_syncfs $fh, $callback->($status)
247 aio_fsync $fh, $callback->($status) 253 aio_fsync $fh, $callback->($status)
248 aio_fdatasync $fh, $callback->($status) 254 aio_fdatasync $fh, $callback->($status)
249 aio_sync_file_range $fh, $offset, $nbytes, $flags, $callback->($status) 255 aio_sync_file_range $fh, $offset, $nbytes, $flags, $callback->($status)
250 aio_pathsync $pathname, $callback->($status) 256 aio_pathsync $pathname, $callback->($status)
251 aio_msync $scalar, $offset = 0, $length = undef, flags = 0, $callback->($status) 257 aio_msync $scalar, $offset = 0, $length = undef, flags = MS_SYNC, $callback->($status)
252 aio_mtouch $scalar, $offset = 0, $length = undef, flags = 0, $callback->($status) 258 aio_mtouch $scalar, $offset = 0, $length = undef, flags = 0, $callback->($status)
253 aio_mlock $scalar, $offset = 0, $length = undef, $callback->($status) 259 aio_mlock $scalar, $offset = 0, $length = undef, $callback->($status)
254 aio_mlockall $flags, $callback->($status) 260 aio_mlockall $flags, $callback->($status)
255 aio_group $callback->(...) 261 aio_group $callback->(...)
256 aio_nop $callback->() 262 aio_nop $callback->()
439=item aio_read $fh,$offset,$length, $data,$dataoffset, $callback->($retval) 445=item aio_read $fh,$offset,$length, $data,$dataoffset, $callback->($retval)
440 446
441=item aio_write $fh,$offset,$length, $data,$dataoffset, $callback->($retval) 447=item aio_write $fh,$offset,$length, $data,$dataoffset, $callback->($retval)
442 448
443Reads or writes C<$length> bytes from or to the specified C<$fh> and 449Reads or writes C<$length> bytes from or to the specified C<$fh> and
444C<$offset> into the scalar given by C<$data> and offset C<$dataoffset> 450C<$offset> into the scalar given by C<$data> and offset C<$dataoffset> and
445and calls the callback without the actual number of bytes read (or -1 on 451calls the callback with the actual number of bytes transferred (or -1 on
446error, just like the syscall). 452error, just like the syscall).
447 453
448C<aio_read> will, like C<sysread>, shrink or grow the C<$data> scalar to 454C<aio_read> will, like C<sysread>, shrink or grow the C<$data> scalar to
449offset plus the actual number of bytes read. 455offset plus the actual number of bytes read.
450 456
508As native sendfile syscalls (as practically any non-POSIX interface hacked 514As native sendfile syscalls (as practically any non-POSIX interface hacked
509together in a hurry to improve benchmark numbers) tend to be rather buggy 515together in a hurry to improve benchmark numbers) tend to be rather buggy
510on many systems, this implementation tries to work around some known bugs 516on many systems, this implementation tries to work around some known bugs
511in Linux and FreeBSD kernels (probably others, too), but that might fail, 517in Linux and FreeBSD kernels (probably others, too), but that might fail,
512so you really really should check the return value of C<aio_sendfile> - 518so you really really should check the return value of C<aio_sendfile> -
513fewre bytes than expected might have been transferred. 519fewer bytes than expected might have been transferred.
514 520
515 521
516=item aio_readahead $fh,$offset,$length, $callback->($retval) 522=item aio_readahead $fh,$offset,$length, $callback->($retval)
517 523
518C<aio_readahead> populates the page cache with data from a file so that 524C<aio_readahead> populates the page cache with data from a file so that
522whole pages, so that offset is effectively rounded down to a page boundary 528whole pages, so that offset is effectively rounded down to a page boundary
523and bytes are read up to the next page boundary greater than or equal to 529and bytes are read up to the next page boundary greater than or equal to
524(off-set+length). C<aio_readahead> does not read beyond the end of the 530(off-set+length). C<aio_readahead> does not read beyond the end of the
525file. The current file offset of the file is left unchanged. 531file. The current file offset of the file is left unchanged.
526 532
527If that syscall doesn't exist (likely if your OS isn't Linux) it will be 533If that syscall doesn't exist (likely if your kernel isn't Linux) it will
528emulated by simply reading the data, which would have a similar effect. 534be emulated by simply reading the data, which would have a similar effect.
529 535
530 536
531=item aio_stat $fh_or_path, $callback->($status) 537=item aio_stat $fh_or_path, $callback->($status)
532 538
533=item aio_lstat $fh, $callback->($status) 539=item aio_lstat $fh, $callback->($status)
626 0x00001373 devfs 632 0x00001373 devfs
627 0x00001cd1 devpts 633 0x00001cd1 devpts
628 0x0000f15f ecryptfs 634 0x0000f15f ecryptfs
629 0x00414a53 efs 635 0x00414a53 efs
630 0x0000137d ext 636 0x0000137d ext
631 0x0000ef53 ext2/ext3 637 0x0000ef53 ext2/ext3/ext4
632 0x0000ef51 ext2 638 0x0000ef51 ext2
639 0xf2f52010 f2fs
633 0x00004006 fat 640 0x00004006 fat
634 0x65735546 fuseblk 641 0x65735546 fuseblk
635 0x65735543 fusectl 642 0x65735543 fusectl
636 0x0bad1dea futexfs 643 0x0bad1dea futexfs
637 0x01161970 gfs2 644 0x01161970 gfs2
638 0x47504653 gpfs 645 0x47504653 gpfs
639 0x00004244 hfs 646 0x00004244 hfs
640 0xf995e849 hpfs 647 0xf995e849 hpfs
648 0x00c0ffee hostfs
641 0x958458f6 hugetlbfs 649 0x958458f6 hugetlbfs
642 0x2bad1dea inotifyfs 650 0x2bad1dea inotifyfs
643 0x00009660 isofs 651 0x00009660 isofs
644 0x000072b6 jffs2 652 0x000072b6 jffs2
645 0x3153464a jfs 653 0x3153464a jfs
660 0x00009fa1 openprom 668 0x00009fa1 openprom
661 0x7461636F ocfs2 669 0x7461636F ocfs2
662 0x00009fa0 proc 670 0x00009fa0 proc
663 0x6165676c pstorefs 671 0x6165676c pstorefs
664 0x0000002f qnx4 672 0x0000002f qnx4
673 0x68191122 qnx6
665 0x858458f6 ramfs 674 0x858458f6 ramfs
666 0x52654973 reiserfs 675 0x52654973 reiserfs
667 0x00007275 romfs 676 0x00007275 romfs
668 0x67596969 rpc_pipefs 677 0x67596969 rpc_pipefs
669 0x73636673 securityfs 678 0x73636673 securityfs
1224 }; 1233 };
1225 1234
1226 $grp 1235 $grp
1227} 1236}
1228 1237
1238=item aio_fcntl $fh, $cmd, $arg, $callback->($status)
1239
1240=item aio_ioctl $fh, $request, $buf, $callback->($status)
1241
1242These work just like the C<fcntl> and C<ioctl> built-in functions, except
1243they execute asynchronously and pass the return value to the callback.
1244
1245Both calls can be used for a lot of things, some of which make more sense
1246to run asynchronously in their own thread, while some others make less
1247sense. For example, calls that block waiting for external events, such
1248as locking, will also lock down an I/O thread while it is waiting, which
1249can deadlock the whole I/O system. At the same time, there might be no
1250alternative to using a thread to wait.
1251
1252So in general, you should only use these calls for things that do
1253(filesystem) I/O, not for things that wait for other events (network,
1254other processes), although if you are careful and know what you are doing,
1255you still can.
1256
1257The following constants are available (missing ones are, as usual C<0>):
1258
1259C<FIFREEZE>, C<FITHAW>, C<FITRIM>, C<FICLONE>, C<FICLONERANGE>, C<FIDEDUPERANGE>.
1260
1261C<FS_IOC_GETFLAGS>, C<FS_IOC_SETFLAGS>, C<FS_IOC_GETVERSION>, C<FS_IOC_SETVERSION>,
1262C<FS_IOC_FIEMAP>.
1263
1264C<FS_IOC_FSGETXATTR>, C<FS_IOC_FSSETXATTR>, C<FS_IOC_SET_ENCRYPTION_POLICY>,
1265C<FS_IOC_GET_ENCRYPTION_PWSALT>, C<FS_IOC_GET_ENCRYPTION_POLICY>, C<FS_KEY_DESCRIPTOR_SIZE>.
1266
1267C<FS_SECRM_FL>, C<FS_UNRM_FL>, C<FS_COMPR_FL>, C<FS_SYNC_FL>, C<FS_IMMUTABLE_FL>,
1268C<FS_APPEND_FL>, C<FS_NODUMP_FL>, C<FS_NOATIME_FL>, C<FS_DIRTY_FL>,
1269C<FS_COMPRBLK_FL>, C<FS_NOCOMP_FL>, C<FS_ENCRYPT_FL>, C<FS_BTREE_FL>,
1270C<FS_INDEX_FL>, C<FS_JOURNAL_DATA_FL>, C<FS_NOTAIL_FL>, C<FS_DIRSYNC_FL>, C<FS_TOPDIR_FL>,
1271C<FS_FL_USER_MODIFIABLE>.
1272
1273C<FS_XFLAG_REALTIME>, C<FS_XFLAG_PREALLOC>, C<FS_XFLAG_IMMUTABLE>, C<FS_XFLAG_APPEND>,
1274C<FS_XFLAG_SYNC>, C<FS_XFLAG_NOATIME>, C<FS_XFLAG_NODUMP>, C<FS_XFLAG_RTINHERIT>,
1275C<FS_XFLAG_PROJINHERIT>, C<FS_XFLAG_NOSYMLINKS>, C<FS_XFLAG_EXTSIZE>, C<FS_XFLAG_EXTSZINHERIT>,
1276C<FS_XFLAG_NODEFRAG>, C<FS_XFLAG_FILESTREAM>, C<FS_XFLAG_DAX>, C<FS_XFLAG_HASATTR>,
1277
1229=item aio_sync $callback->($status) 1278=item aio_sync $callback->($status)
1230 1279
1231Asynchronously call sync and call the callback when finished. 1280Asynchronously call sync and call the callback when finished.
1232 1281
1233=item aio_fsync $fh, $callback->($status) 1282=item aio_fsync $fh, $callback->($status)
1301 }; 1350 };
1302 1351
1303 $grp 1352 $grp
1304} 1353}
1305 1354
1306=item aio_msync $scalar, $offset = 0, $length = undef, flags = 0, $callback->($status) 1355=item aio_msync $scalar, $offset = 0, $length = undef, flags = MS_SYNC, $callback->($status)
1307 1356
1308This is a rather advanced IO::AIO call, which only works on mmap(2)ed 1357This is a rather advanced IO::AIO call, which only works on mmap(2)ed
1309scalars (see the C<IO::AIO::mmap> function, although it also works on data 1358scalars (see the C<IO::AIO::mmap> function, although it also works on data
1310scalars managed by the L<Sys::Mmap> or L<Mmap> modules, note that the 1359scalars managed by the L<Sys::Mmap> or L<Mmap> modules, note that the
1311scalar must only be modified in-place while an aio operation is pending on 1360scalar must only be modified in-place while an aio operation is pending on
1313 1362
1314It calls the C<msync> function of your OS, if available, with the memory 1363It calls the C<msync> function of your OS, if available, with the memory
1315area starting at C<$offset> in the string and ending C<$length> bytes 1364area starting at C<$offset> in the string and ending C<$length> bytes
1316later. If C<$length> is negative, counts from the end, and if C<$length> 1365later. If C<$length> is negative, counts from the end, and if C<$length>
1317is C<undef>, then it goes till the end of the string. The flags can be 1366is C<undef>, then it goes till the end of the string. The flags can be
1318a combination of C<IO::AIO::MS_ASYNC>, C<IO::AIO::MS_INVALIDATE> and 1367either C<IO::AIO::MS_ASYNC> or C<IO::AIO::MS_SYNC>, plus an optional
1319C<IO::AIO::MS_SYNC>. 1368C<IO::AIO::MS_INVALIDATE>.
1320 1369
1321=item aio_mtouch $scalar, $offset = 0, $length = undef, flags = 0, $callback->($status) 1370=item aio_mtouch $scalar, $offset = 0, $length = undef, flags = 0, $callback->($status)
1322 1371
1323This is a rather advanced IO::AIO call, which works best on mmap(2)ed 1372This is a rather advanced IO::AIO call, which works best on mmap(2)ed
1324scalars. 1373scalars.
2017=item IO::AIO::madvise $scalar, $offset, $len, $advice 2066=item IO::AIO::madvise $scalar, $offset, $len, $advice
2018 2067
2019Simply calls the C<posix_madvise> function (see its 2068Simply calls the C<posix_madvise> function (see its
2020manpage for details). The following advice constants are 2069manpage for details). The following advice constants are
2021available: C<IO::AIO::MADV_NORMAL>, C<IO::AIO::MADV_SEQUENTIAL>, 2070available: C<IO::AIO::MADV_NORMAL>, C<IO::AIO::MADV_SEQUENTIAL>,
2022C<IO::AIO::MADV_RANDOM>, C<IO::AIO::MADV_WILLNEED>, C<IO::AIO::MADV_DONTNEED>. 2071C<IO::AIO::MADV_RANDOM>, C<IO::AIO::MADV_WILLNEED>, C<IO::AIO::MADV_DONTNEED>,
2072C<IO::AIO::MADV_FREE>.
2023 2073
2024On systems that do not implement C<posix_madvise>, this function returns 2074On systems that do not implement C<posix_madvise>, this function returns
2025ENOSYS, otherwise the return value of C<posix_madvise>. 2075ENOSYS, otherwise the return value of C<posix_madvise>.
2026 2076
2027=item IO::AIO::mprotect $scalar, $offset, $len, $protect 2077=item IO::AIO::mprotect $scalar, $offset, $len, $protect
2038 2088
2039Memory-maps a file (or anonymous memory range) and attaches it to the 2089Memory-maps a file (or anonymous memory range) and attaches it to the
2040given C<$scalar>, which will act like a string scalar. Returns true on 2090given C<$scalar>, which will act like a string scalar. Returns true on
2041success, and false otherwise. 2091success, and false otherwise.
2042 2092
2093The scalar must exist, but its contents do not matter - this means you
2094cannot use a nonexistant array or hash element. When in doubt, C<undef>
2095the scalar first.
2096
2043The only operations allowed on the scalar are C<substr>/C<vec> that don't 2097The only operations allowed on the mmapped scalar are C<substr>/C<vec>,
2044change the string length, and most read-only operations such as copying it 2098which don't change the string length, and most read-only operations such
2045or searching it with regexes and so on. 2099as copying it or searching it with regexes and so on.
2046 2100
2047Anything else is unsafe and will, at best, result in memory leaks. 2101Anything else is unsafe and will, at best, result in memory leaks.
2048 2102
2049The memory map associated with the C<$scalar> is automatically removed 2103The memory map associated with the C<$scalar> is automatically removed
2050when the C<$scalar> is destroyed, or when the C<IO::AIO::mmap> or 2104when the C<$scalar> is undef'd or destroyed, or when the C<IO::AIO::mmap>
2051C<IO::AIO::munmap> functions are called. 2105or C<IO::AIO::munmap> functions are called on it.
2052 2106
2053This calls the C<mmap>(2) function internally. See your system's manual 2107This calls the C<mmap>(2) function internally. See your system's manual
2054page for details on the C<$length>, C<$prot> and C<$flags> parameters. 2108page for details on the C<$length>, C<$prot> and C<$flags> parameters.
2055 2109
2056The C<$length> must be larger than zero and smaller than the actual 2110The C<$length> must be larger than zero and smaller than the actual
2057filesize. 2111filesize.
2058 2112
2059C<$prot> is a combination of C<IO::AIO::PROT_NONE>, C<IO::AIO::PROT_EXEC>, 2113C<$prot> is a combination of C<IO::AIO::PROT_NONE>, C<IO::AIO::PROT_EXEC>,
2060C<IO::AIO::PROT_READ> and/or C<IO::AIO::PROT_WRITE>, 2114C<IO::AIO::PROT_READ> and/or C<IO::AIO::PROT_WRITE>,
2061 2115
2062C<$flags> can be a combination of C<IO::AIO::MAP_SHARED> or 2116C<$flags> can be a combination of
2063C<IO::AIO::MAP_PRIVATE>, or a number of system-specific flags (when 2117C<IO::AIO::MAP_SHARED> or
2064not available, the are defined as 0): C<IO::AIO::MAP_ANONYMOUS> 2118C<IO::AIO::MAP_PRIVATE>,
2119or a number of system-specific flags (when not available, the are C<0>):
2065(which is set to C<MAP_ANON> if your system only provides this 2120C<IO::AIO::MAP_ANONYMOUS> (which is set to C<MAP_ANON> if your system only provides this constant),
2066constant), C<IO::AIO::MAP_HUGETLB>, C<IO::AIO::MAP_LOCKED>, 2121C<IO::AIO::MAP_LOCKED>,
2067C<IO::AIO::MAP_NORESERVE>, C<IO::AIO::MAP_POPULATE> or 2122C<IO::AIO::MAP_NORESERVE>,
2123C<IO::AIO::MAP_POPULATE>,
2068C<IO::AIO::MAP_NONBLOCK> 2124C<IO::AIO::MAP_NONBLOCK>,
2125C<IO::AIO::MAP_FIXED>,
2126C<IO::AIO::MAP_GROWSDOWN>,
2127C<IO::AIO::MAP_32BIT>,
2128C<IO::AIO::MAP_HUGETLB> or
2129C<IO::AIO::MAP_STACK>.
2069 2130
2070If C<$fh> is C<undef>, then a file descriptor of C<-1> is passed. 2131If C<$fh> is C<undef>, then a file descriptor of C<-1> is passed.
2071 2132
2072C<$offset> is the offset from the start of the file - it generally must be 2133C<$offset> is the offset from the start of the file - it generally must be
2073a multiple of C<IO::AIO::PAGESIZE> and defaults to C<0>. 2134a multiple of C<IO::AIO::PAGESIZE> and defaults to C<0>.

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines