ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/libeio/eio.pod
(Generate patch)

Comparing libeio/eio.pod (file contents):
Revision 1.10 by root, Sun Jun 5 23:22:04 2011 UTC vs.
Revision 1.24 by sf-exg, Thu Jul 14 08:51:29 2011 UTC

45Unlike the name component C<stamp> might indicate, it is also used for 45Unlike the name component C<stamp> might indicate, it is also used for
46time differences throughout libeio. 46time differences throughout libeio.
47 47
48=head2 FORK SUPPORT 48=head2 FORK SUPPORT
49 49
50Calling C<fork ()> is fully supported by this module. It is implemented in these steps: 50Calling C<fork ()> is fully supported by this module - but you must not
51rely on this. It is currently implemented in these steps:
51 52
52 1. wait till all requests in "execute" state have been handled 53 1. wait till all requests in "execute" state have been handled
53 (basically requests that are already handed over to the kernel). 54 (basically requests that are already handed over to the kernel).
54 2. fork 55 2. fork
55 3. in the parent, continue business as usual, done 56 3. in the parent, continue business as usual, done
56 4. in the child, destroy all ready and pending requests and free the 57 4. in the child, destroy all ready and pending requests and free the
57 memory used by the worker threads. This gives you a fully empty 58 memory used by the worker threads. This gives you a fully empty
58 libeio queue. 59 libeio queue.
59 60
60Note, however, since libeio does use threads, thr above guarantee doesn't 61Note, however, since libeio does use threads, the above guarantee doesn't
61cover your libc, for example, malloc and other libc functions are not 62cover your libc, for example, malloc and other libc functions are not
62fork-safe, so there is very little you can do after a fork, and in fatc, 63fork-safe, so there is very little you can do after a fork, and in fact,
63the above might crash, and thus change. 64the above might crash, and thus change.
64 65
65=head1 INITIALISATION/INTEGRATION 66=head1 INITIALISATION/INTEGRATION
66 67
67Before you can call any eio functions you first have to initialise the 68Before you can call any eio functions you first have to initialise the
124=back 125=back
125 126
126For libev, you would typically use an C<ev_async> watcher: the 127For libev, you would typically use an C<ev_async> watcher: the
127C<want_poll> callback would invoke C<ev_async_send> to wake up the event 128C<want_poll> callback would invoke C<ev_async_send> to wake up the event
128loop. Inside the callback set for the watcher, one would call C<eio_poll 129loop. Inside the callback set for the watcher, one would call C<eio_poll
129()> (followed by C<ev_async_send> again if C<eio_poll> indicates that not 130()>.
130all requests have been handled yet). The race is taken care of because 131
131libev resets/rearms the async watcher before calling your callback, 132If C<eio_poll ()> is configured to not handle all results in one go
132and therefore, before calling C<eio_poll>. This might result in (some) 133(i.e. it returns C<-1>) then you should start an idle watcher that calls
133spurious wake-ups, but is generally harmless. 134C<eio_poll> until it returns something C<!= -1>.
135
136A full-featured connector between libeio and libev would look as follows
137(if C<eio_poll> is handling all requests, it can of course be simplified a
138lot by removing the idle watcher logic):
139
140 static struct ev_loop *loop;
141 static ev_idle repeat_watcher;
142 static ev_async ready_watcher;
143
144 /* idle watcher callback, only used when eio_poll */
145 /* didn't handle all results in one call */
146 static void
147 repeat (EV_P_ ev_idle *w, int revents)
148 {
149 if (eio_poll () != -1)
150 ev_idle_stop (EV_A_ w);
151 }
152
153 /* eio has some results, process them */
154 static void
155 ready (EV_P_ ev_async *w, int revents)
156 {
157 if (eio_poll () == -1)
158 ev_idle_start (EV_A_ &repeat_watcher);
159 }
160
161 /* wake up the event loop */
162 static void
163 want_poll (void)
164 {
165 ev_async_send (loop, &ready_watcher)
166 }
167
168 void
169 my_init_eio ()
170 {
171 loop = EV_DEFAULT;
172
173 ev_idle_init (&repeat_watcher, repeat);
174 ev_async_init (&ready_watcher, ready);
175 ev_async_start (loop &watcher);
176
177 eio_init (want_poll, 0);
178 }
134 179
135For most other event loops, you would typically use a pipe - the event 180For most other event loops, you would typically use a pipe - the event
136loop should be told to wait for read readiness on the read end. In 181loop should be told to wait for read readiness on the read end. In
137C<want_poll> you would write a single byte, in C<done_poll> you would try 182C<want_poll> you would write a single byte, in C<done_poll> you would try
138to read that byte, and in the callback for the read end, you would call 183to read that byte, and in the callback for the read end, you would call
139C<eio_poll>. The race is avoided here because the event loop should invoke 184C<eio_poll>.
140your callback again and again until the byte has been read (as the pipe 185
141read callback does not read it, only C<done_poll>). 186You don't have to take special care in the case C<eio_poll> doesn't handle
187all requests, as the done callback will not be invoked, so the event loop
188will still signal readiness for the pipe until I<all> results have been
189processed.
142 190
143 191
144=head1 HIGH LEVEL REQUEST API 192=head1 HIGH LEVEL REQUEST API
145 193
146Libeio has both a high-level API, which consists of calling a request 194Libeio has both a high-level API, which consists of calling a request
153 201
154You submit a request by calling the relevant C<eio_TYPE> function with the 202You submit a request by calling the relevant C<eio_TYPE> function with the
155required parameters, a callback of type C<int (*eio_cb)(eio_req *req)> 203required parameters, a callback of type C<int (*eio_cb)(eio_req *req)>
156(called C<eio_cb> below) and a freely usable C<void *data> argument. 204(called C<eio_cb> below) and a freely usable C<void *data> argument.
157 205
158The return value will either be 0 206The return value will either be 0, in case something went really wrong
207(which can basically only happen on very fatal errors, such as C<malloc>
208returning 0, which is rather unlikely), or a pointer to the newly-created
209and submitted C<eio_req *>.
159 210
160The callback will be called with an C<eio_req *> which contains the 211The callback will be called with an C<eio_req *> which contains the
161results of the request. The members you can access inside that structure 212results of the request. The members you can access inside that structure
162vary from request to request, except for: 213vary from request to request, except for:
163 214
210 } 261 }
211 262
212 /* the first three arguments are passed to open(2) */ 263 /* the first three arguments are passed to open(2) */
213 /* the remaining are priority, callback and data */ 264 /* the remaining are priority, callback and data */
214 if (!eio_open ("/etc/passwd", O_RDONLY, 0, 0, file_open_done, 0)) 265 if (!eio_open ("/etc/passwd", O_RDONLY, 0, 0, file_open_done, 0))
215 abort (); /* something ent wrong, we will all die!!! */ 266 abort (); /* something went wrong, we will all die!!! */
216 267
217Note that you additionally need to call C<eio_poll> when the C<want_cb> 268Note that you additionally need to call C<eio_poll> when the C<want_cb>
218indicates that requests are ready to be processed. 269indicates that requests are ready to be processed.
270
271=head2 CANCELLING REQUESTS
272
273Sometimes the need for a request goes away before the request is
274finished. In that case, one can cancel the request by a call to
275C<eio_cancel>:
276
277=over 4
278
279=item eio_cancel (eio_req *req)
280
281Cancel the request (and all its subrequests). If the request is currently
282executing it might still continue to execute, and in other cases it might
283still take a while till the request is cancelled.
284
285Even if cancelled, the finish callback will still be invoked - the
286callbacks of all cancellable requests need to check whether the request
287has been cancelled by calling C<EIO_CANCELLED (req)>:
288
289 static int
290 my_eio_cb (eio_req *req)
291 {
292 if (EIO_CANCELLED (req))
293 return 0;
294 }
295
296In addition, cancelled requests will I<either> have C<< req->result >>
297set to C<-1> and C<errno> to C<ECANCELED>, or I<otherwise> they were
298successfully executed, despite being cancelled (e.g. when they have
299already been executed at the time they were cancelled).
300
301C<EIO_CANCELLED> is still true for requests that have successfully
302executed, as long as C<eio_cancel> was called on them at some point.
303
304=back
219 305
220=head2 AVAILABLE REQUESTS 306=head2 AVAILABLE REQUESTS
221 307
222The following request functions are available. I<All> of them return the 308The following request functions are available. I<All> of them return the
223C<eio_req *> on success and C<0> on failure, and I<all> of them have the 309C<eio_req *> on success and C<0> on failure, and I<all> of them have the
226custom data value as C<data>. 312custom data value as C<data>.
227 313
228=head3 POSIX API WRAPPERS 314=head3 POSIX API WRAPPERS
229 315
230These requests simply wrap the POSIX call of the same name, with the same 316These requests simply wrap the POSIX call of the same name, with the same
231arguments. If a function is not implemented by the OS and cnanot be emulated 317arguments. If a function is not implemented by the OS and cannot be emulated
232in some way, then all of these return C<-1> and set C<errorno> to C<ENOSYS>. 318in some way, then all of these return C<-1> and set C<errorno> to C<ENOSYS>.
233 319
234=over 4 320=over 4
235 321
236=item eio_open (const char *path, int flags, mode_t mode, int pri, eio_cb cb, void *data) 322=item eio_open (const char *path, int flags, mode_t mode, int pri, eio_cb cb, void *data)
317 char *target = strndup ((char *)req->ptr2, req->result); 403 char *target = strndup ((char *)req->ptr2, req->result);
318 404
319 free (target); 405 free (target);
320 } 406 }
321 407
408=item eio_realpath (const char *path, int pri, eio_cb cb, void *data)
409
410Similar to the realpath libc function, but unlike that one, C<<
411req->result >> is C<-1> on failure. On success, the result is the length
412of the returned path in C<ptr2> (which is I<NOT> 0-terminated) - this is
413similar to readlink.
414
322=item eio_stat (const char *path, int pri, eio_cb cb, void *data) 415=item eio_stat (const char *path, int pri, eio_cb cb, void *data)
323 416
324=item eio_lstat (const char *path, int pri, eio_cb cb, void *data) 417=item eio_lstat (const char *path, int pri, eio_cb cb, void *data)
325 418
326=item eio_fstat (int fd, int pri, eio_cb cb, void *data) 419=item eio_fstat (int fd, int pri, eio_cb cb, void *data)
327 420
328Stats a file - if C<< req->result >> indicates success, then you can 421Stats a file - if C<< req->result >> indicates success, then you can
329access the C<struct stat>-like structure via C<< req->ptr2 >>: 422access the C<struct stat>-like structure via C<< req->ptr2 >>:
330 423
331 EIO_STRUCT_STAT *statdata = (EIO_STRUCT_STAT *)req->ptr2; 424 EIO_STRUCT_STAT *statdata = (EIO_STRUCT_STAT *)req->ptr2;
332 425
333=item eio_statvfs (const char *path, int pri, eio_cb cb, void *data) 426=item eio_statvfs (const char *path, int pri, eio_cb cb, void *data)
334 427
335=item eio_fstatvfs (int fd, int pri, eio_cb cb, void *data) 428=item eio_fstatvfs (int fd, int pri, eio_cb cb, void *data)
336 429
337Stats a filesystem - if C<< req->result >> indicates success, then you can 430Stats a filesystem - if C<< req->result >> indicates success, then you can
338access the C<struct statvfs>-like structure via C<< req->ptr2 >>: 431access the C<struct statvfs>-like structure via C<< req->ptr2 >>:
339 432
340 EIO_STRUCT_STATVFS *statdata = (EIO_STRUCT_STATVFS *)req->ptr2; 433 EIO_STRUCT_STATVFS *statdata = (EIO_STRUCT_STATVFS *)req->ptr2;
341 434
342=back 435=back
343 436
344=head3 READING DIRECTORIES 437=head3 READING DIRECTORIES
345 438
346Reading directories sounds simple, but can be rather demanding, especially 439Reading directories sounds simple, but can be rather demanding, especially
347if you want to do stuff such as traversing a diretcory hierarchy or 440if you want to do stuff such as traversing a directory hierarchy or
348processing all files in a directory. Libeio can assist thess complex tasks 441processing all files in a directory. Libeio can assist these complex tasks
349with it's C<eio_readdir> call. 442with it's C<eio_readdir> call.
350 443
351=over 4 444=over 4
352 445
353=item eio_readdir (const char *path, int flags, int pri, eio_cb cb, void *data) 446=item eio_readdir (const char *path, int flags, int pri, eio_cb cb, void *data)
385 478
386If this flag is specified, then, in addition to the names in C<ptr2>, 479If this flag is specified, then, in addition to the names in C<ptr2>,
387also an array of C<struct eio_dirent> is returned, in C<ptr1>. A C<struct 480also an array of C<struct eio_dirent> is returned, in C<ptr1>. A C<struct
388eio_dirent> looks like this: 481eio_dirent> looks like this:
389 482
390 struct eio_dirent 483 struct eio_dirent
391 { 484 {
392 int nameofs; /* offset of null-terminated name string in (char *)req->ptr2 */ 485 int nameofs; /* offset of null-terminated name string in (char *)req->ptr2 */
393 unsigned short namelen; /* size of filename without trailing 0 */ 486 unsigned short namelen; /* size of filename without trailing 0 */
394 unsigned char type; /* one of EIO_DT_* */ 487 unsigned char type; /* one of EIO_DT_* */
395 signed char score; /* internal use */ 488 signed char score; /* internal use */
396 ino_t inode; /* the inode number, if available, otherwise unspecified */ 489 ino_t inode; /* the inode number, if available, otherwise unspecified */
397 }; 490 };
398 491
399The only members you normally would access are C<nameofs>, which is the 492The only members you normally would access are C<nameofs>, which is the
400byte-offset from C<ptr2> to the start of the name, C<namelen> and C<type>. 493byte-offset from C<ptr2> to the start of the name, C<namelen> and C<type>.
401 494
402C<type> can be one of: 495C<type> can be one of:
445When this flag is specified, then the names will be returned in an order 538When this flag is specified, then the names will be returned in an order
446suitable for stat()'ing each one. That is, when you plan to stat() 539suitable for stat()'ing each one. That is, when you plan to stat()
447all files in the given directory, then the returned order will likely 540all files in the given directory, then the returned order will likely
448be fastest. 541be fastest.
449 542
450If both this flag and C<EIO_READDIR_DIRS_FIRST> are specified, then 543If both this flag and C<EIO_READDIR_DIRS_FIRST> are specified, then the
451the likely dirs come first, resulting in a less optimal stat order. 544likely directories come first, resulting in a less optimal stat order.
452 545
453=item EIO_READDIR_FOUND_UNKNOWN 546=item EIO_READDIR_FOUND_UNKNOWN
454 547
455This flag should not be specified when calling C<eio_readdir>. Instead, 548This flag should not be specified when calling C<eio_readdir>. Instead,
456it is being set by C<eio_readdir> (you can access the C<flags> via C<< 549it is being set by C<eio_readdir> (you can access the C<flags> via C<<
457req->int1 >>, when any of the C<type>'s found were C<EIO_DT_UNKNOWN>. The 550req->int1 >>, when any of the C<type>'s found were C<EIO_DT_UNKNOWN>. The
458absense of this flag therefore indicates that all C<type>'s are known, 551absence of this flag therefore indicates that all C<type>'s are known,
459which can be used to speed up some algorithms. 552which can be used to speed up some algorithms.
460 553
461A typical use case would be to identify all subdirectories within a 554A typical use case would be to identify all subdirectories within a
462directory - you would ask C<eio_readdir> for C<EIO_READDIR_DIRS_FIRST>. If 555directory - you would ask C<eio_readdir> for C<EIO_READDIR_DIRS_FIRST>. If
463then this flag is I<NOT> set, then all the entries at the beginning of the 556then this flag is I<NOT> set, then all the entries at the beginning of the
501as calling C<fdatasync>. 594as calling C<fdatasync>.
502 595
503Flags can be any combination of C<EIO_SYNC_FILE_RANGE_WAIT_BEFORE>, 596Flags can be any combination of C<EIO_SYNC_FILE_RANGE_WAIT_BEFORE>,
504C<EIO_SYNC_FILE_RANGE_WRITE> and C<EIO_SYNC_FILE_RANGE_WAIT_AFTER>. 597C<EIO_SYNC_FILE_RANGE_WRITE> and C<EIO_SYNC_FILE_RANGE_WAIT_AFTER>.
505 598
599=item eio_fallocate (int fd, int mode, off_t offset, off_t len, int pri, eio_cb cb, void *data)
600
601Calls C<fallocate> (note: I<NOT> C<posix_fallocate>!). If the syscall is
602missing, then it returns failure and sets C<errno> to C<ENOSYS>.
603
604The C<mode> argument can be C<0> (for behaviour similar to
605C<posix_fallocate>), or C<EIO_FALLOC_FL_KEEP_SIZE>, which keeps the size
606of the file unchanged (but still preallocates space beyond end of file).
607
506=back 608=back
507 609
508=head3 LIBEIO-SPECIFIC REQUESTS 610=head3 LIBEIO-SPECIFIC REQUESTS
509 611
510These requests are specific to libeio and do not correspond to any OS call. 612These requests are specific to libeio and do not correspond to any OS call.
551 653
552 eio_custom (my_open, 0, my_open_done, "/etc/passwd"); 654 eio_custom (my_open, 0, my_open_done, "/etc/passwd");
553 655
554=item eio_busy (eio_tstamp delay, int pri, eio_cb cb, void *data) 656=item eio_busy (eio_tstamp delay, int pri, eio_cb cb, void *data)
555 657
556This is a a request that takes C<delay> seconds to execute, but otherwise 658This is a request that takes C<delay> seconds to execute, but otherwise
557does nothing - it simply puts one of the worker threads to sleep for this 659does nothing - it simply puts one of the worker threads to sleep for this
558long. 660long.
559 661
560This request can be used to artificially increase load, e.g. for debugging 662This request can be used to artificially increase load, e.g. for debugging
561or benchmarking reasons. 663or benchmarking reasons.
568 670
569=back 671=back
570 672
571=head3 GROUPING AND LIMITING REQUESTS 673=head3 GROUPING AND LIMITING REQUESTS
572 674
675There is one more rather special request, C<eio_grp>. It is a very special
676aio request: Instead of doing something, it is a container for other eio
677requests.
678
679There are two primary use cases for this: a) bundle many requests into a
680single, composite, request with a definite callback and the ability to
681cancel the whole request with its subrequests and b) limiting the number
682of "active" requests.
683
684Further below you will find more discussion of these topics - first
685follows the reference section detailing the request generator and other
686methods.
687
688=over 4
689
690=item eio_req *grp = eio_grp (eio_cb cb, void *data)
691
692Creates, submits and returns a group request. Note that it doesn't have a
693priority, unlike all other requests.
694
695=item eio_grp_add (eio_req *grp, eio_req *req)
696
697Adds a request to the request group.
698
699=item eio_grp_cancel (eio_req *grp)
700
701Cancels all requests I<in> the group, but I<not> the group request
702itself. You can cancel the group request I<and> all subrequests via a
703normal C<eio_cancel> call.
704
705=back
706
707=head4 GROUP REQUEST LIFETIME
708
709Left alone, a group request will instantly move to the pending state and
710will be finished at the next call of C<eio_poll>.
711
712The usefulness stems from the fact that, if a subrequest is added to a
713group I<before> a call to C<eio_poll>, via C<eio_grp_add>, then the group
714will not finish until all the subrequests have finished.
715
716So the usage cycle of a group request is like this: after it is created,
717you normally instantly add a subrequest. If none is added, the group
718request will finish on it's own. As long as subrequests are added before
719the group request is finished it will be kept from finishing, that is the
720callbacks of any subrequests can, in turn, add more requests to the group,
721and as long as any requests are active, the group request itself will not
722finish.
723
724=head4 CREATING COMPOSITE REQUESTS
725
726Imagine you wanted to create an C<eio_load> request that opens a file,
727reads it and closes it. This means it has to execute at least three eio
728requests, but for various reasons it might be nice if that request looked
729like any other eio request.
730
731This can be done with groups:
732
733=over 4
734
735=item 1) create the request object
736
737Create a group that contains all further requests. This is the request you
738can return as "the load request".
739
740=item 2) open the file, maybe
741
742Next, open the file with C<eio_open> and add the request to the group
743request and you are finished setting up the request.
744
745If, for some reason, you cannot C<eio_open> (path is a null ptr?) you
746can set C<< grp->result >> to C<-1> to signal an error and let the group
747request finish on its own.
748
749=item 3) open callback adds more requests
750
751In the open callback, if the open was not successful, copy C<<
752req->errorno >> to C<< grp->errorno >> and set C<< grp->errorno >> to
753C<-1> to signal an error.
754
755Otherwise, malloc some memory or so and issue a read request, adding the
756read request to the group.
757
758=item 4) continue issuing requests till finished
759
760In the real callback, check for errors and possibly continue with
761C<eio_close> or any other eio request in the same way.
762
763As soon as no new requests are added the group request will finish. Make
764sure you I<always> set C<< grp->result >> to some sensible value.
765
766=back
767
768=head4 REQUEST LIMITING
769
770
573#TODO 771#TODO
574 772
575/*****************************************************************************/
576/* groups */
577
578eio_req *eio_grp (eio_cb cb, void *data);
579void eio_grp_feed (eio_req *grp, void (*feed)(eio_req *req), int limit);
580void eio_grp_limit (eio_req *grp, int limit); 773void eio_grp_limit (eio_req *grp, int limit);
581void eio_grp_add (eio_req *grp, eio_req *req);
582void eio_grp_cancel (eio_req *grp); /* cancels all sub requests but not the group */
583 774
584 775
585=back 776=back
586 777
587 778
593=head1 ANATOMY AND LIFETIME OF AN EIO REQUEST 784=head1 ANATOMY AND LIFETIME OF AN EIO REQUEST
594 785
595A request is represented by a structure of type C<eio_req>. To initialise 786A request is represented by a structure of type C<eio_req>. To initialise
596it, clear it to all zero bytes: 787it, clear it to all zero bytes:
597 788
598 eio_req req; 789 eio_req req;
599 790
600 memset (&req, 0, sizeof (req)); 791 memset (&req, 0, sizeof (req));
601 792
602A more common way to initialise a new C<eio_req> is to use C<calloc>: 793A more common way to initialise a new C<eio_req> is to use C<calloc>:
603 794
604 eio_req *req = calloc (1, sizeof (*req)); 795 eio_req *req = calloc (1, sizeof (*req));
605 796
606In either case, libeio neither allocates, initialises or frees the 797In either case, libeio neither allocates, initialises or frees the
607C<eio_req> structure for you - it merely uses it. 798C<eio_req> structure for you - it merely uses it.
608 799
609zero 800zero
627for example, in interactive programs, you might want to limit this time to 818for example, in interactive programs, you might want to limit this time to
628C<0.01> seconds or so. 819C<0.01> seconds or so.
629 820
630Note that: 821Note that:
631 822
823=over 4
824
632a) libeio doesn't know how long your request callbacks take, so the time 825=item a) libeio doesn't know how long your request callbacks take, so the
633spent in C<eio_poll> is up to one callback invocation longer then this 826time spent in C<eio_poll> is up to one callback invocation longer then
634interval. 827this interval.
635 828
636b) this is implemented by calling C<gettimeofday> after each request, 829=item b) this is implemented by calling C<gettimeofday> after each
637which can be costly. 830request, which can be costly.
638 831
639c) at least one request will be handled. 832=item c) at least one request will be handled.
833
834=back
640 835
641=item eio_set_max_poll_reqs (unsigned int nreqs) 836=item eio_set_max_poll_reqs (unsigned int nreqs)
642 837
643When C<nreqs> is non-zero, then C<eio_poll> will not handle more than 838When C<nreqs> is non-zero, then C<eio_poll> will not handle more than
644C<nreqs> requests per invocation. This is a less costly way to limit the 839C<nreqs> requests per invocation. This is a less costly way to limit the

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines