ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-HTTP/HTTP.pm
(Generate patch)

Comparing AnyEvent-HTTP/HTTP.pm (file contents):
Revision 1.41 by root, Sun Jul 5 23:50:59 2009 UTC vs.
Revision 1.50 by root, Tue Aug 11 13:20:42 2009 UTC

… …
41use strict; 41use strict;
42no warnings; 42no warnings;
43 43
44use Errno (); 44use Errno ();
45 45
46use AnyEvent 4.452 (); 46use AnyEvent 4.8 ();
47use AnyEvent::Util (); 47use AnyEvent::Util ();
48use AnyEvent::Socket (); 48use AnyEvent::Socket ();
49use AnyEvent::Handle (); 49use AnyEvent::Handle ();
50 50
51use base Exporter::; 51use base Exporter::;
52 52
53our $VERSION = '1.12'; 53our $VERSION = '1.42';
54 54
55our @EXPORT = qw(http_get http_post http_head http_request); 55our @EXPORT = qw(http_get http_post http_head http_request);
56 56
57our $USERAGENT = "Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)"; 57our $USERAGENT = "Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)";
58our $MAX_RECURSE = 10; 58our $MAX_RECURSE = 10;
59our $MAX_PERSISTENT = 8; 59our $MAX_PERSISTENT = 8;
60our $PERSISTENT_TIMEOUT = 2; 60our $PERSISTENT_TIMEOUT = 2;
61our $TIMEOUT = 300; 61our $TIMEOUT = 300;
62 62
63# changing these is evil 63# changing these is evil
64our $MAX_PERSISTENT_PER_HOST = 2; 64our $MAX_PERSISTENT_PER_HOST = 0;
65our $MAX_PER_HOST = 4; 65our $MAX_PER_HOST = 4;
66 66
67our $PROXY; 67our $PROXY;
68our $ACTIVE = 0; 68our $ACTIVE = 0;
69 69
… …
94When called in void context, nothing is returned. In other contexts, 94When called in void context, nothing is returned. In other contexts,
95C<http_request> returns a "cancellation guard" - you have to keep the 95C<http_request> returns a "cancellation guard" - you have to keep the
96object at least alive until the callback get called. If the object gets 96object at least alive until the callback get called. If the object gets
97destroyed before the callbakc is called, the request will be cancelled. 97destroyed before the callbakc is called, the request will be cancelled.
98 98
99The callback will be called with the response data as first argument 99The callback will be called with the response body data as first argument
100(or C<undef> if it wasn't available due to errors), and a hash-ref with 100(or C<undef> if an error occured), and a hash-ref with response headers as
101response headers as second argument. 101second argument.
102 102
103All the headers in that hash are lowercased. In addition to the response 103All the headers in that hash are lowercased. In addition to the response
104headers, the "pseudo-headers" C<HTTPVersion>, C<Status> and C<Reason> 104headers, the "pseudo-headers" C<HTTPVersion>, C<Status> and C<Reason>
105contain the three parts of the HTTP Status-Line of the same name. The 105contain the three parts of the HTTP Status-Line of the same name. The
106pseudo-header C<URL> contains the original URL (which can differ from the 106pseudo-header C<URL> contains the original URL (which can differ from the
… …
138 138
139=item headers => hashref 139=item headers => hashref
140 140
141The request headers to use. Currently, C<http_request> may provide its 141The request headers to use. Currently, C<http_request> may provide its
142own C<Host:>, C<Content-Length:>, C<Connection:> and C<Cookie:> headers 142own C<Host:>, C<Content-Length:>, C<Connection:> and C<Cookie:> headers
143and will provide defaults for C<User-Agent:> and C<Referer:>. 143and will provide defaults for C<User-Agent:> and C<Referer:> (this can be
144suppressed by using C<undef> for these headers in which case they won't be
145sent at all).
144 146
145=item timeout => $seconds 147=item timeout => $seconds
146 148
147The time-out to use for various stages - each connect attempt will reset 149The time-out to use for various stages - each connect attempt will reset
148the timeout, as will read or write activity. Default timeout is 5 minutes. 150the timeout, as will read or write activity. Default timeout is 5 minutes.
… …
150=item proxy => [$host, $port[, $scheme]] or undef 152=item proxy => [$host, $port[, $scheme]] or undef
151 153
152Use the given http proxy for all requests. If not specified, then the 154Use the given http proxy for all requests. If not specified, then the
153default proxy (as specified by C<$ENV{http_proxy}>) is used. 155default proxy (as specified by C<$ENV{http_proxy}>) is used.
154 156
155C<$scheme> must be either missing or C<http> for HTTP, or C<https> for 157C<$scheme> must be either missing, C<http> for HTTP or C<https> for
156HTTPS. 158HTTPS.
157 159
158=item body => $string 160=item body => $string
159 161
160The request body, usually empty. Will be-sent as-is (future versions of 162The request body, usually empty. Will be-sent as-is (future versions of
… …
186verification) TLS context. 188verification) TLS context.
187 189
188The default for this option is C<low>, which could be interpreted as "give 190The default for this option is C<low>, which could be interpreted as "give
189me the page, no matter what". 191me the page, no matter what".
190 192
191=item on_header => $callback->($hdr) 193=item on_header => $callback->($headers)
192 194
193When specified, this callback will be called with the header hash as soon 195When specified, this callback will be called with the header hash as soon
194as headers have been successfully received from the remote server (not on 196as headers have been successfully received from the remote server (not on
195locally-generated errors). 197locally-generated errors).
196 198
… …
200 202
201This callback is useful, among other things, to quickly reject unwanted 203This callback is useful, among other things, to quickly reject unwanted
202content, which, if it is supposed to be rare, can be faster than first 204content, which, if it is supposed to be rare, can be faster than first
203doing a C<HEAD> request. 205doing a C<HEAD> request.
204 206
207Example: cancel the request unless the content-type is "text/html".
208
209 on_header => sub {
210 $_[0]{"content-type"} =~ /^text\/html\s*(?:;|$)/
211 },
212
205=item on_body => $callback->($data, $hdr) 213=item on_body => $callback->($partial_body, $headers)
206 214
207When specified, all body data will be "filtered" through this callback. 215When specified, all body data will be passed to this callback instead of
216to the completion callback. The completion callback will get the empty
217string instead of the body data.
208 218
209The callback will incrementally receive body data, and is supposed to 219It has to return either true (in which case AnyEvent::HTTP will continue),
210return it or a modified version of it (empty strings are valid returns). 220or false, in which case AnyEvent::HTTP will cancel the download (and call
221the completion callback with an error code of C<598>).
211 222
212If the callback returns C<undef>, then the request will be cancelled. 223This callback is useful when the data is too large to be held in memory
213 224(so the callback writes it to a file) or when only some information should
214This callback is useful when you want to do some processing on the data, 225be extracted, or when the body should be processed incrementally.
215or the data is too large to be held in memory (so the callback writes it
216to a file and returns the empty string) and so on.
217 226
218It is usually preferred over doing your own body handling via 227It is usually preferred over doing your own body handling via
219C<want_body_handle>. 228C<want_body_handle>, but in case of streaming APIs, where HTTP is
229only used to create a connection, C<want_body_handle> is the better
230alternative, as it allows you to install your own event handler, reducing
231resource usage.
220 232
221=item want_body_handle => $enable 233=item want_body_handle => $enable
222 234
223When enabled (default is disabled), the behaviour of AnyEvent::HTTP 235When enabled (default is disabled), the behaviour of AnyEvent::HTTP
224changes considerably: after parsing the headers, and instead of 236changes considerably: after parsing the headers, and instead of
… …
236This is useful with some push-type services, where, after the initial 248This is useful with some push-type services, where, after the initial
237headers, an interactive protocol is used (typical example would be the 249headers, an interactive protocol is used (typical example would be the
238push-style twitter API which starts a JSON/XML stream). 250push-style twitter API which starts a JSON/XML stream).
239 251
240If you think you need this, first have a look at C<on_body>, to see if 252If you think you need this, first have a look at C<on_body>, to see if
241that doesn'T solve your problem in a better way. 253that doesn't solve your problem in a better way.
242 254
243=back 255=back
244 256
245Example: make a simple HTTP GET request for http://www.nethype.de/ 257Example: make a simple HTTP GET request for http://www.nethype.de/
246 258
… …
302 push @{ $CO_SLOT{$_[0]}[1] }, $_[1]; 314 push @{ $CO_SLOT{$_[0]}[1] }, $_[1];
303 315
304 _slot_schedule $_[0]; 316 _slot_schedule $_[0];
305} 317}
306 318
307our $qr_nl = qr<\015?\012>; 319our $qr_nl = qr{\015?\012};
308our $qr_nlnl = qr<\015?\012\015?\012>; 320our $qr_nlnl = qr{(?<![^\012])\015?\012};
309 321
310our $TLS_CTX_LOW = { cache => 1, sslv2 => 1 }; 322our $TLS_CTX_LOW = { cache => 1, sslv2 => 1 };
311our $TLS_CTX_HIGH = { cache => 1, verify => 1, verify_peername => "https" }; 323our $TLS_CTX_HIGH = { cache => 1, verify => 1, verify_peername => "https" };
312 324
313sub http_request($$@) { 325sub http_request($$@) {
… …
347 $uauthority =~ /^(?: .*\@ )? ([^\@:]+) (?: : (\d+) )?$/x 359 $uauthority =~ /^(?: .*\@ )? ([^\@:]+) (?: : (\d+) )?$/x
348 or return $cb->(undef, { Status => 599, Reason => "Unparsable URL", URL => $url }); 360 or return $cb->(undef, { Status => 599, Reason => "Unparsable URL", URL => $url });
349 361
350 my $uhost = $1; 362 my $uhost = $1;
351 $uport = $2 if defined $2; 363 $uport = $2 if defined $2;
364
365 $hdr{host} = defined $2 ? "$uhost:$2" : "$uhost";
352 366
353 $uhost =~ s/^\[(.*)\]$/$1/; 367 $uhost =~ s/^\[(.*)\]$/$1/;
354 $upath .= "?$query" if length $query; 368 $upath .= "?$query" if length $query;
355 369
356 $upath =~ s%^/?%/%; 370 $upath =~ s%^/?%/%;
… …
389 my ($rhost, $rport, $rscheme, $rpath); # request host, port, path 403 my ($rhost, $rport, $rscheme, $rpath); # request host, port, path
390 404
391 if ($proxy) { 405 if ($proxy) {
392 ($rpath, $rhost, $rport, $rscheme) = ($url, @$proxy); 406 ($rpath, $rhost, $rport, $rscheme) = ($url, @$proxy);
393 407
408 $rscheme = "http" unless defined $rscheme;
409
394 # don't support https requests over https-proxy transport, 410 # don't support https requests over https-proxy transport,
395 # can't be done with tls as spec'ed, unless you double-encrypt. 411 # can't be done with tls as spec'ed, unless you double-encrypt.
396 $rscheme = "http" if $uscheme eq "https" && $rscheme eq "https"; 412 $rscheme = "http" if $uscheme eq "https" && $rscheme eq "https";
397 } else { 413 } else {
398 ($rhost, $rport, $rscheme, $rpath) = ($uhost, $uport, $uscheme, $upath); 414 ($rhost, $rport, $rscheme, $rpath) = ($uhost, $uport, $uscheme, $upath);
399 } 415 }
400 416
401 $hdr{"user-agent"} ||= $USERAGENT; 417 # leave out fragment and query string, just a heuristic
402 $hdr{referer} ||= "$uscheme://$uauthority$upath"; # leave out fragment and query string, just a heuristic 418 $hdr{referer} ||= "$uscheme://$uauthority$upath" unless exists $hdr{referer};
419 $hdr{"user-agent"} ||= $USERAGENT unless exists $hdr{"user-agent"};
403 420
404 $hdr{host} = "$uhost:$uport";
405 $hdr{"content-length"} = length $arg{body}; 421 $hdr{"content-length"} = length $arg{body};
406 422
407 my %state = (connect_guard => 1); 423 my %state = (connect_guard => 1);
408 424
409 _get_slot $uhost, sub { 425 _get_slot $uhost, sub {
… …
411 427
412 return unless $state{connect_guard}; 428 return unless $state{connect_guard};
413 429
414 $state{connect_guard} = AnyEvent::Socket::tcp_connect $rhost, $rport, sub { 430 $state{connect_guard} = AnyEvent::Socket::tcp_connect $rhost, $rport, sub {
415 $state{fh} = shift 431 $state{fh} = shift
432 or do {
433 my $err = "$!";
434 %state = ();
416 or return (%state = (), $cb->(undef, { Status => 599, Reason => "$!", URL => $url })); 435 return $cb->(undef, { Status => 599, Reason => $err, URL => $url });
436 };
437
417 pop; # free memory, save a tree 438 pop; # free memory, save a tree
418 439
419 return unless delete $state{connect_guard}; 440 return unless delete $state{connect_guard};
420 441
421 # get handle 442 # get handle
… …
454 $state{handle}->starttls ("connect") if $uscheme eq "https" && !exists $state{handle}{tls}; 475 $state{handle}->starttls ("connect") if $uscheme eq "https" && !exists $state{handle}{tls};
455 476
456 # send request 477 # send request
457 $state{handle}->push_write ( 478 $state{handle}->push_write (
458 "$method $rpath HTTP/1.0\015\012" 479 "$method $rpath HTTP/1.0\015\012"
459 . (join "", map "\u$_: $hdr{$_}\015\012", keys %hdr) 480 . (join "", map "\u$_: $hdr{$_}\015\012", grep defined $hdr{$_}, keys %hdr)
460 . "\015\012" 481 . "\015\012"
461 . (delete $arg{body}) 482 . (delete $arg{body})
462 ); 483 );
463 484
464 %hdr = (); # reduce memory usage, save a kitten 485 %hdr = (); # reduce memory usage, save a kitten
… …
475 URL => ",$url" 496 URL => ",$url"
476 ); 497 );
477 498
478 # headers, could be optimized a bit 499 # headers, could be optimized a bit
479 $state{handle}->unshift_read (line => $qr_nlnl, sub { 500 $state{handle}->unshift_read (line => $qr_nlnl, sub {
480 for ("$_[1]\012") { 501 for ("$_[1]") {
481 y/\015//d; # weed out any \015, as they show up in the weirdest of places. 502 y/\015//d; # weed out any \015, as they show up in the weirdest of places.
482 503
483 # things seen, not parsed: 504 # things seen, not parsed:
484 # p3pP="NON CUR OTPi OUR NOR UNI" 505 # p3pP="NON CUR OTPi OUR NOR UNI"
485 506
486 $hdr{lc $1} .= ",$2" 507 $hdr{lc $1} .= ",$2"
487 while /\G 508 while /\G
488 ([^:\000-\037]+): 509 ([^:\000-\037]*):
489 [\011\040]* 510 [\011\040]*
490 ((?: [^\012]+ | \012[\011\040] )*) 511 ((?: [^\012]+ | \012[\011\040] )*)
491 \012 512 \012
492 /gxc; 513 /gxc;
493 514
… …
582 603
583 redo if /\G\s*,/gc; 604 redo if /\G\s*,/gc;
584 } 605 }
585 } 606 }
586 607
587 if ($redirect) { 608 if ($redirect && exists $hdr{location}) {
588 # we ignore any errors, as it is very common to receive 609 # we ignore any errors, as it is very common to receive
589 # Content-Length != 0 but no actual body 610 # Content-Length != 0 but no actual body
590 # we also access %hdr, as $_[1] might be an erro 611 # we also access %hdr, as $_[1] might be an erro
591 http_request ($method => $hdr{location}, %arg, recurse => $recurse - 1, $cb); 612 http_request ($method => $hdr{location}, %arg, recurse => $recurse - 1, $cb);
592 } else { 613 } else {
… …
597 my $len = $hdr{"content-length"}; 618 my $len = $hdr{"content-length"};
598 619
599 if (!$redirect && $arg{on_header} && !$arg{on_header}(\%hdr)) { 620 if (!$redirect && $arg{on_header} && !$arg{on_header}(\%hdr)) {
600 $finish->(undef, { Status => 598, Reason => "Request cancelled by on_header", URL => $url }); 621 $finish->(undef, { Status => 598, Reason => "Request cancelled by on_header", URL => $url });
601 } elsif ( 622 } elsif (
602 $hdr{Status} =~ /^(?:1..|204|304)$/ 623 $hdr{Status} =~ /^(?:1..|[23]04)$/
603 or $method eq "HEAD" 624 or $method eq "HEAD"
604 or (defined $len && !$len) 625 or (defined $len && !$len)
605 ) { 626 ) {
606 # no body 627 # no body
607 $finish->("", \%hdr); 628 $finish->("", \%hdr);
… …
647 $finish->((substr delete $_[0]{rbuf}, 0, $len, ""), \%hdr) 668 $finish->((substr delete $_[0]{rbuf}, 0, $len, ""), \%hdr)
648 if $len <= length $_[0]{rbuf}; 669 if $len <= length $_[0]{rbuf};
649 }); 670 });
650 } else { 671 } else {
651 $_[0]->on_error (sub { 672 $_[0]->on_error (sub {
652 $! == Errno::EPIPE 673 $! == Errno::EPIPE || !$!
653 ? $finish->(delete $_[0]{rbuf}, \%hdr) 674 ? $finish->(delete $_[0]{rbuf}, \%hdr)
654 : $finish->(undef, { Status => 599, Reason => $_[2], URL => $url }); 675 : $finish->(undef, { Status => 599, Reason => $_[2], URL => $url });
655 }); 676 });
656 $_[0]->on_read (sub { }); 677 $_[0]->on_read (sub { });
657 } 678 }
… …
681 }); 702 });
682 } else { 703 } else {
683 &$handle_actual_request; 704 &$handle_actual_request;
684 } 705 }
685 706
686 }, sub { 707 }, $arg{on_prepare} || sub { $timeout };
687 $timeout
688 };
689 }; 708 };
690 709
691 defined wantarray && AnyEvent::Util::guard { %state = () } 710 defined wantarray && AnyEvent::Util::guard { %state = () }
692} 711}
693 712
… …
725=item $AnyEvent::HTTP::USERAGENT 744=item $AnyEvent::HTTP::USERAGENT
726 745
727The default value for the C<User-Agent> header (the default is 746The default value for the C<User-Agent> header (the default is
728C<Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)>). 747C<Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)>).
729 748
730=item $AnyEvent::HTTP::MAX_PERSISTENT 749=item $AnyEvent::HTTP::MAX_PER_HOST
731 750
732The maximum number of persistent connections to keep open (default: 8). 751The maximum number of concurrent connections to the same host (identified
752by the hostname). If the limit is exceeded, then the additional requests
753are queued until previous connections are closed.
733 754
734Not implemented currently. 755The default value for this is C<4>, and it is highly advisable to not
735 756increase it.
736=item $AnyEvent::HTTP::PERSISTENT_TIMEOUT
737
738The maximum time to cache a persistent connection, in seconds (default: 2).
739
740Not implemented currently.
741 757
742=item $AnyEvent::HTTP::ACTIVE 758=item $AnyEvent::HTTP::ACTIVE
743 759
744The number of active connections. This is not the number of currently 760The number of active connections. This is not the number of currently
745running requests, but the number of currently open and non-idle TCP 761running requests, but the number of currently open and non-idle TCP

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines