ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-HTTP/HTTP.pm
(Generate patch)

Comparing AnyEvent-HTTP/HTTP.pm (file contents):
Revision 1.41 by root, Sun Jul 5 23:50:59 2009 UTC vs.
Revision 1.44 by root, Tue Jul 7 00:15:32 2009 UTC

… …
41use strict; 41use strict;
42no warnings; 42no warnings;
43 43
44use Errno (); 44use Errno ();
45 45
46use AnyEvent 4.452 (); 46use AnyEvent 4.8 ();
47use AnyEvent::Util (); 47use AnyEvent::Util ();
48use AnyEvent::Socket (); 48use AnyEvent::Socket ();
49use AnyEvent::Handle (); 49use AnyEvent::Handle ();
50 50
51use base Exporter::; 51use base Exporter::;
52 52
53our $VERSION = '1.12'; 53our $VERSION = '1.4';
54 54
55our @EXPORT = qw(http_get http_post http_head http_request); 55our @EXPORT = qw(http_get http_post http_head http_request);
56 56
57our $USERAGENT = "Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)"; 57our $USERAGENT = "Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)";
58our $MAX_RECURSE = 10; 58our $MAX_RECURSE = 10;
59our $MAX_PERSISTENT = 8; 59our $MAX_PERSISTENT = 8;
60our $PERSISTENT_TIMEOUT = 2; 60our $PERSISTENT_TIMEOUT = 2;
61our $TIMEOUT = 300; 61our $TIMEOUT = 300;
62 62
63# changing these is evil 63# changing these is evil
64our $MAX_PERSISTENT_PER_HOST = 2; 64our $MAX_PERSISTENT_PER_HOST = 0;
65our $MAX_PER_HOST = 4; 65our $MAX_PER_HOST = 4;
66 66
67our $PROXY; 67our $PROXY;
68our $ACTIVE = 0; 68our $ACTIVE = 0;
69 69
… …
94When called in void context, nothing is returned. In other contexts, 94When called in void context, nothing is returned. In other contexts,
95C<http_request> returns a "cancellation guard" - you have to keep the 95C<http_request> returns a "cancellation guard" - you have to keep the
96object at least alive until the callback get called. If the object gets 96object at least alive until the callback get called. If the object gets
97destroyed before the callbakc is called, the request will be cancelled. 97destroyed before the callbakc is called, the request will be cancelled.
98 98
99The callback will be called with the response data as first argument 99The callback will be called with the response body data as first argument
100(or C<undef> if it wasn't available due to errors), and a hash-ref with 100(or C<undef> if an error occured), and a hash-ref with response headers as
101response headers as second argument. 101second argument.
102 102
103All the headers in that hash are lowercased. In addition to the response 103All the headers in that hash are lowercased. In addition to the response
104headers, the "pseudo-headers" C<HTTPVersion>, C<Status> and C<Reason> 104headers, the "pseudo-headers" C<HTTPVersion>, C<Status> and C<Reason>
105contain the three parts of the HTTP Status-Line of the same name. The 105contain the three parts of the HTTP Status-Line of the same name. The
106pseudo-header C<URL> contains the original URL (which can differ from the 106pseudo-header C<URL> contains the original URL (which can differ from the
… …
186verification) TLS context. 186verification) TLS context.
187 187
188The default for this option is C<low>, which could be interpreted as "give 188The default for this option is C<low>, which could be interpreted as "give
189me the page, no matter what". 189me the page, no matter what".
190 190
191=item on_header => $callback->($hdr) 191=item on_header => $callback->($headers)
192 192
193When specified, this callback will be called with the header hash as soon 193When specified, this callback will be called with the header hash as soon
194as headers have been successfully received from the remote server (not on 194as headers have been successfully received from the remote server (not on
195locally-generated errors). 195locally-generated errors).
196 196
… …
200 200
201This callback is useful, among other things, to quickly reject unwanted 201This callback is useful, among other things, to quickly reject unwanted
202content, which, if it is supposed to be rare, can be faster than first 202content, which, if it is supposed to be rare, can be faster than first
203doing a C<HEAD> request. 203doing a C<HEAD> request.
204 204
205Example: cancel the request unless the content-type is "text/html".
206
207 on_header => sub {
208 $_[0]{"content-type"} =~ /^text\/html\s*(?:;|$)/
209 },
210
205=item on_body => $callback->($data, $hdr) 211=item on_body => $callback->($partial_body, $headers)
206 212
207When specified, all body data will be "filtered" through this callback. 213When specified, all body data will be passed to this callback instead of
214to the completion callback. The completion callback will get the empty
215string instead of the body data.
208 216
209The callback will incrementally receive body data, and is supposed to 217It has to return either true (in which case AnyEvent::HTTP will continue),
210return it or a modified version of it (empty strings are valid returns). 218or false, in which case AnyEvent::HTTP will cancel the download (and call
219the completion callback with an error code of C<598>).
211 220
212If the callback returns C<undef>, then the request will be cancelled. 221This callback is useful when the data is too large to be held in memory
213 222(so the callback writes it to a file) or when only some information should
214This callback is useful when you want to do some processing on the data, 223be extracted, or when the body should be processed incrementally.
215or the data is too large to be held in memory (so the callback writes it
216to a file and returns the empty string) and so on.
217 224
218It is usually preferred over doing your own body handling via 225It is usually preferred over doing your own body handling via
219C<want_body_handle>. 226C<want_body_handle>.
220 227
221=item want_body_handle => $enable 228=item want_body_handle => $enable
… …
347 $uauthority =~ /^(?: .*\@ )? ([^\@:]+) (?: : (\d+) )?$/x 354 $uauthority =~ /^(?: .*\@ )? ([^\@:]+) (?: : (\d+) )?$/x
348 or return $cb->(undef, { Status => 599, Reason => "Unparsable URL", URL => $url }); 355 or return $cb->(undef, { Status => 599, Reason => "Unparsable URL", URL => $url });
349 356
350 my $uhost = $1; 357 my $uhost = $1;
351 $uport = $2 if defined $2; 358 $uport = $2 if defined $2;
359
360 $hdr{host} = defined $2 ? "$uhost:$2" : "$uhost";
352 361
353 $uhost =~ s/^\[(.*)\]$/$1/; 362 $uhost =~ s/^\[(.*)\]$/$1/;
354 $upath .= "?$query" if length $query; 363 $upath .= "?$query" if length $query;
355 364
356 $upath =~ s%^/?%/%; 365 $upath =~ s%^/?%/%;
… …
399 } 408 }
400 409
401 $hdr{"user-agent"} ||= $USERAGENT; 410 $hdr{"user-agent"} ||= $USERAGENT;
402 $hdr{referer} ||= "$uscheme://$uauthority$upath"; # leave out fragment and query string, just a heuristic 411 $hdr{referer} ||= "$uscheme://$uauthority$upath"; # leave out fragment and query string, just a heuristic
403 412
404 $hdr{host} = "$uhost:$uport";
405 $hdr{"content-length"} = length $arg{body}; 413 $hdr{"content-length"} = length $arg{body};
406 414
407 my %state = (connect_guard => 1); 415 my %state = (connect_guard => 1);
408 416
409 _get_slot $uhost, sub { 417 _get_slot $uhost, sub {
… …
411 419
412 return unless $state{connect_guard}; 420 return unless $state{connect_guard};
413 421
414 $state{connect_guard} = AnyEvent::Socket::tcp_connect $rhost, $rport, sub { 422 $state{connect_guard} = AnyEvent::Socket::tcp_connect $rhost, $rport, sub {
415 $state{fh} = shift 423 $state{fh} = shift
424 or do {
425 my $err = "$!";
426 %state = ();
416 or return (%state = (), $cb->(undef, { Status => 599, Reason => "$!", URL => $url })); 427 return $cb->(undef, { Status => 599, Reason => $err, URL => $url });
428 };
429
417 pop; # free memory, save a tree 430 pop; # free memory, save a tree
418 431
419 return unless delete $state{connect_guard}; 432 return unless delete $state{connect_guard};
420 433
421 # get handle 434 # get handle
… …
483 # things seen, not parsed: 496 # things seen, not parsed:
484 # p3pP="NON CUR OTPi OUR NOR UNI" 497 # p3pP="NON CUR OTPi OUR NOR UNI"
485 498
486 $hdr{lc $1} .= ",$2" 499 $hdr{lc $1} .= ",$2"
487 while /\G 500 while /\G
488 ([^:\000-\037]+): 501 ([^:\000-\037]*):
489 [\011\040]* 502 [\011\040]*
490 ((?: [^\012]+ | \012[\011\040] )*) 503 ((?: [^\012]+ | \012[\011\040] )*)
491 \012 504 \012
492 /gxc; 505 /gxc;
493 506
… …
597 my $len = $hdr{"content-length"}; 610 my $len = $hdr{"content-length"};
598 611
599 if (!$redirect && $arg{on_header} && !$arg{on_header}(\%hdr)) { 612 if (!$redirect && $arg{on_header} && !$arg{on_header}(\%hdr)) {
600 $finish->(undef, { Status => 598, Reason => "Request cancelled by on_header", URL => $url }); 613 $finish->(undef, { Status => 598, Reason => "Request cancelled by on_header", URL => $url });
601 } elsif ( 614 } elsif (
602 $hdr{Status} =~ /^(?:1..|204|304)$/ 615 $hdr{Status} =~ /^(?:1..|[23]04)$/
603 or $method eq "HEAD" 616 or $method eq "HEAD"
604 or (defined $len && !$len) 617 or (defined $len && !$len)
605 ) { 618 ) {
606 # no body 619 # no body
607 $finish->("", \%hdr); 620 $finish->("", \%hdr);
… …
647 $finish->((substr delete $_[0]{rbuf}, 0, $len, ""), \%hdr) 660 $finish->((substr delete $_[0]{rbuf}, 0, $len, ""), \%hdr)
648 if $len <= length $_[0]{rbuf}; 661 if $len <= length $_[0]{rbuf};
649 }); 662 });
650 } else { 663 } else {
651 $_[0]->on_error (sub { 664 $_[0]->on_error (sub {
652 $! == Errno::EPIPE 665 $! == Errno::EPIPE || !$!
653 ? $finish->(delete $_[0]{rbuf}, \%hdr) 666 ? $finish->(delete $_[0]{rbuf}, \%hdr)
654 : $finish->(undef, { Status => 599, Reason => $_[2], URL => $url }); 667 : $finish->(undef, { Status => 599, Reason => $_[2], URL => $url });
655 }); 668 });
656 $_[0]->on_read (sub { }); 669 $_[0]->on_read (sub { });
657 } 670 }
… …
725=item $AnyEvent::HTTP::USERAGENT 738=item $AnyEvent::HTTP::USERAGENT
726 739
727The default value for the C<User-Agent> header (the default is 740The default value for the C<User-Agent> header (the default is
728C<Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)>). 741C<Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)>).
729 742
730=item $AnyEvent::HTTP::MAX_PERSISTENT 743=item $AnyEvent::HTTP::MAX_PER_HOST
731 744
732The maximum number of persistent connections to keep open (default: 8). 745The maximum number of concurrent conenctions to the same host (identified
746by the hostname). If the limit is exceeded, then the additional requests
747are queued until previous connections are closed.
733 748
734Not implemented currently. 749The default value for this is C<4>, and it is highly advisable to not
735 750increase it.
736=item $AnyEvent::HTTP::PERSISTENT_TIMEOUT
737
738The maximum time to cache a persistent connection, in seconds (default: 2).
739
740Not implemented currently.
741 751
742=item $AnyEvent::HTTP::ACTIVE 752=item $AnyEvent::HTTP::ACTIVE
743 753
744The number of active connections. This is not the number of currently 754The number of active connections. This is not the number of currently
745running requests, but the number of currently open and non-idle TCP 755running requests, but the number of currently open and non-idle TCP

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines