ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-HTTP/HTTP.pm
(Generate patch)

Comparing AnyEvent-HTTP/HTTP.pm (file contents):
Revision 1.43 by root, Mon Jul 6 03:03:12 2009 UTC vs.
Revision 1.68 by root, Fri Dec 31 19:22:18 2010 UTC

41use strict; 41use strict;
42no warnings; 42no warnings;
43 43
44use Errno (); 44use Errno ();
45 45
46use AnyEvent 4.452 (); 46use AnyEvent 5.0 ();
47use AnyEvent::Util (); 47use AnyEvent::Util ();
48use AnyEvent::Socket ();
49use AnyEvent::Handle (); 48use AnyEvent::Handle ();
50 49
51use base Exporter::; 50use base Exporter::;
52 51
53our $VERSION = '1.12'; 52our $VERSION = '1.5';
54 53
55our @EXPORT = qw(http_get http_post http_head http_request); 54our @EXPORT = qw(http_get http_post http_head http_request);
56 55
57our $USERAGENT = "Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)"; 56our $USERAGENT = "Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)";
58our $MAX_RECURSE = 10; 57our $MAX_RECURSE = 10;
92must be an absolute http or https URL. 91must be an absolute http or https URL.
93 92
94When called in void context, nothing is returned. In other contexts, 93When called in void context, nothing is returned. In other contexts,
95C<http_request> returns a "cancellation guard" - you have to keep the 94C<http_request> returns a "cancellation guard" - you have to keep the
96object at least alive until the callback get called. If the object gets 95object at least alive until the callback get called. If the object gets
97destroyed before the callbakc is called, the request will be cancelled. 96destroyed before the callback is called, the request will be cancelled.
98 97
99The callback will be called with the response body data as first argument 98The callback will be called with the response body data as first argument
100(or C<undef> if an error occured), and a hash-ref with response headers as 99(or C<undef> if an error occured), and a hash-ref with response headers
101second argument. 100(and trailers) as second argument.
102 101
103All the headers in that hash are lowercased. In addition to the response 102All the headers in that hash are lowercased. In addition to the response
104headers, the "pseudo-headers" C<HTTPVersion>, C<Status> and C<Reason> 103headers, the "pseudo-headers" (uppercase to avoid clashing with possible
104response headers) C<HTTPVersion>, C<Status> and C<Reason> contain the
105contain the three parts of the HTTP Status-Line of the same name. The 105three parts of the HTTP Status-Line of the same name. If an error occurs
106during the body phase of a request, then the original C<Status> and
107C<Reason> values from the header are available as C<OrigStatus> and
108C<OrigReason>.
109
106pseudo-header C<URL> contains the original URL (which can differ from the 110The pseudo-header C<URL> contains the actual URL (which can differ from
107requested URL when following redirects). 111the requested URL when following redirects - for example, you might get
112an error that your URL scheme is not supported even though your URL is a
113valid http URL because it redirected to an ftp URL, in which case you can
114look at the URL pseudo header).
115
116The pseudo-header C<Redirect> only exists when the request was a result
117of an internal redirect. In that case it is an array reference with
118the C<($data, $headers)> from the redirect response. Note that this
119response could in turn be the result of a redirect itself, and C<<
120$headers->{Redirect}[1]{Redirect} >> will then contain the original
121response, and so on.
108 122
109If the server sends a header multiple times, then their contents will be 123If the server sends a header multiple times, then their contents will be
110joined together with a comma (C<,>), as per the HTTP spec. 124joined together with a comma (C<,>), as per the HTTP spec.
111 125
112If an internal error occurs, such as not being able to resolve a hostname, 126If an internal error occurs, such as not being able to resolve a hostname,
136Whether to recurse requests or not, e.g. on redirects, authentication 150Whether to recurse requests or not, e.g. on redirects, authentication
137retries and so on, and how often to do so. 151retries and so on, and how often to do so.
138 152
139=item headers => hashref 153=item headers => hashref
140 154
141The request headers to use. Currently, C<http_request> may provide its 155The request headers to use. Currently, C<http_request> may provide its own
142own C<Host:>, C<Content-Length:>, C<Connection:> and C<Cookie:> headers 156C<Host:>, C<Content-Length:>, C<Connection:> and C<Cookie:> headers and
143and will provide defaults for C<User-Agent:> and C<Referer:>. 157will provide defaults for C<TE:>, C<Referer:> and C<User-Agent:> (this can
158be suppressed by using C<undef> for these headers in which case they won't
159be sent at all).
144 160
145=item timeout => $seconds 161=item timeout => $seconds
146 162
147The time-out to use for various stages - each connect attempt will reset 163The time-out to use for various stages - each connect attempt will reset
148the timeout, as will read or write activity. Default timeout is 5 minutes. 164the timeout, as will read or write activity, i.e. this is not an overall
165timeout.
166
167Default timeout is 5 minutes.
149 168
150=item proxy => [$host, $port[, $scheme]] or undef 169=item proxy => [$host, $port[, $scheme]] or undef
151 170
152Use the given http proxy for all requests. If not specified, then the 171Use the given http proxy for all requests. If not specified, then the
153default proxy (as specified by C<$ENV{http_proxy}>) is used. 172default proxy (as specified by C<$ENV{http_proxy}>) is used.
154 173
155C<$scheme> must be either missing or C<http> for HTTP, or C<https> for 174C<$scheme> must be either missing, C<http> for HTTP or C<https> for
156HTTPS. 175HTTPS.
157 176
158=item body => $string 177=item body => $string
159 178
160The request body, usually empty. Will be-sent as-is (future versions of 179The request body, usually empty. Will be sent as-is (future versions of
161this module might offer more options). 180this module might offer more options).
162 181
163=item cookie_jar => $hash_ref 182=item cookie_jar => $hash_ref
164 183
165Passing this parameter enables (simplified) cookie-processing, loosely 184Passing this parameter enables (simplified) cookie-processing, loosely
186verification) TLS context. 205verification) TLS context.
187 206
188The default for this option is C<low>, which could be interpreted as "give 207The default for this option is C<low>, which could be interpreted as "give
189me the page, no matter what". 208me the page, no matter what".
190 209
210=item on_prepare => $callback->($fh)
211
212In rare cases you need to "tune" the socket before it is used to
213connect (for exmaple, to bind it on a given IP address). This parameter
214overrides the prepare callback passed to C<AnyEvent::Socket::tcp_connect>
215and behaves exactly the same way (e.g. it has to provide a
216timeout). See the description for the C<$prepare_cb> argument of
217C<AnyEvent::Socket::tcp_connect> for details.
218
219=item tcp_connect => $callback->($host, $service, $connect_cb, $prepare_cb)
220
221In even rarer cases you want total control over how AnyEvent::HTTP
222establishes connections. Normally it uses L<AnyEvent::Socket::tcp_connect>
223to do this, but you can provide your own C<tcp_connect> function -
224obviously, it has to follow the same calling conventions, except that it
225may always return a connection guard object.
226
227There are probably lots of weird uses for this function, starting from
228tracing the hosts C<http_request> actually tries to connect, to (inexact
229but fast) host => IP address caching or even socks protocol support.
230
191=item on_header => $callback->($headers) 231=item on_header => $callback->($headers)
192 232
193When specified, this callback will be called with the header hash as soon 233When specified, this callback will be called with the header hash as soon
194as headers have been successfully received from the remote server (not on 234as headers have been successfully received from the remote server (not on
195locally-generated errors). 235locally-generated errors).
200 240
201This callback is useful, among other things, to quickly reject unwanted 241This callback is useful, among other things, to quickly reject unwanted
202content, which, if it is supposed to be rare, can be faster than first 242content, which, if it is supposed to be rare, can be faster than first
203doing a C<HEAD> request. 243doing a C<HEAD> request.
204 244
245The downside is that cancelling the request makes it impossible to re-use
246the connection. Also, the C<on_header> callback will not receive any
247trailer (headers sent after the response body).
248
205Example: cancel the request unless the content-type is "text/html". 249Example: cancel the request unless the content-type is "text/html".
206 250
207 on_header => sub { 251 on_header => sub {
208 $_[0]{"content-type"} =~ /^text\/html\s*(?:;|$)/ 252 $_[0]{"content-type"} =~ /^text\/html\s*(?:;|$)/
209 }, 253 },
216 260
217It has to return either true (in which case AnyEvent::HTTP will continue), 261It has to return either true (in which case AnyEvent::HTTP will continue),
218or false, in which case AnyEvent::HTTP will cancel the download (and call 262or false, in which case AnyEvent::HTTP will cancel the download (and call
219the completion callback with an error code of C<598>). 263the completion callback with an error code of C<598>).
220 264
265The downside to cancelling the request is that it makes it impossible to
266re-use the connection.
267
221This callback is useful when the data is too large to be held in memory 268This callback is useful when the data is too large to be held in memory
222(so the callback writes it to a file) or when only some information should 269(so the callback writes it to a file) or when only some information should
223be extracted, or when the body should be processed incrementally. 270be extracted, or when the body should be processed incrementally.
224 271
225It is usually preferred over doing your own body handling via 272It is usually preferred over doing your own body handling via
226C<want_body_handle>. 273C<want_body_handle>, but in case of streaming APIs, where HTTP is
274only used to create a connection, C<want_body_handle> is the better
275alternative, as it allows you to install your own event handler, reducing
276resource usage.
227 277
228=item want_body_handle => $enable 278=item want_body_handle => $enable
229 279
230When enabled (default is disabled), the behaviour of AnyEvent::HTTP 280When enabled (default is disabled), the behaviour of AnyEvent::HTTP
231changes considerably: after parsing the headers, and instead of 281changes considerably: after parsing the headers, and instead of
243This is useful with some push-type services, where, after the initial 293This is useful with some push-type services, where, after the initial
244headers, an interactive protocol is used (typical example would be the 294headers, an interactive protocol is used (typical example would be the
245push-style twitter API which starts a JSON/XML stream). 295push-style twitter API which starts a JSON/XML stream).
246 296
247If you think you need this, first have a look at C<on_body>, to see if 297If you think you need this, first have a look at C<on_body>, to see if
248that doesn'T solve your problem in a better way. 298that doesn't solve your problem in a better way.
249 299
250=back 300=back
251 301
252Example: make a simple HTTP GET request for http://www.nethype.de/ 302Example: do a simple HTTP GET request for http://www.nethype.de/ and print
303the response body.
253 304
254 http_request GET => "http://www.nethype.de/", sub { 305 http_request GET => "http://www.nethype.de/", sub {
255 my ($body, $hdr) = @_; 306 my ($body, $hdr) = @_;
256 print "$body\n"; 307 print "$body\n";
257 }; 308 };
258 309
259Example: make a HTTP HEAD request on https://www.google.com/, use a 310Example: do a HTTP HEAD request on https://www.google.com/, use a
260timeout of 30 seconds. 311timeout of 30 seconds.
261 312
262 http_request 313 http_request
263 GET => "https://www.google.com", 314 GET => "https://www.google.com",
264 timeout => 30, 315 timeout => 30,
267 use Data::Dumper; 318 use Data::Dumper;
268 print Dumper $hdr; 319 print Dumper $hdr;
269 } 320 }
270 ; 321 ;
271 322
272Example: make another simple HTTP GET request, but immediately try to 323Example: do another simple HTTP GET request, but immediately try to
273cancel it. 324cancel it.
274 325
275 my $request = http_request GET => "http://www.nethype.de/", sub { 326 my $request = http_request GET => "http://www.nethype.de/", sub {
276 my ($body, $hdr) = @_; 327 my ($body, $hdr) = @_;
277 print "$body\n"; 328 print "$body\n";
309 push @{ $CO_SLOT{$_[0]}[1] }, $_[1]; 360 push @{ $CO_SLOT{$_[0]}[1] }, $_[1];
310 361
311 _slot_schedule $_[0]; 362 _slot_schedule $_[0];
312} 363}
313 364
314our $qr_nl = qr<\015?\012>; 365# continue to parse $_ for headers and place them into the arg
315our $qr_nlnl = qr<\015?\012\015?\012>; 366sub parse_hdr() {
367 my %hdr;
368
369 # things seen, not parsed:
370 # p3pP="NON CUR OTPi OUR NOR UNI"
371
372 $hdr{lc $1} .= ",$2"
373 while /\G
374 ([^:\000-\037]*):
375 [\011\040]*
376 ((?: [^\012]+ | \012[\011\040] )*)
377 \012
378 /gxc;
379
380 /\G$/
381 or return;
382
383 # remove the "," prefix we added to all headers above
384 substr $_, 0, 1, ""
385 for values %hdr;
386
387 \%hdr
388}
389
390our $qr_nlnl = qr{(?<![^\012])\015?\012};
316 391
317our $TLS_CTX_LOW = { cache => 1, sslv2 => 1 }; 392our $TLS_CTX_LOW = { cache => 1, sslv2 => 1 };
318our $TLS_CTX_HIGH = { cache => 1, verify => 1, verify_peername => "https" }; 393our $TLS_CTX_HIGH = { cache => 1, verify => 1, verify_peername => "https" };
319 394
320sub http_request($$@) { 395sub http_request($$@) {
332 while (my ($k, $v) = each %$hdr) { 407 while (my ($k, $v) = each %$hdr) {
333 $hdr{lc $k} = $v; 408 $hdr{lc $k} = $v;
334 } 409 }
335 } 410 }
336 411
412 # pseudo headers for all subsequent responses
413 my @pseudo = (URL => $url);
414 push @pseudo, Redirect => delete $arg{Redirect} if exists $arg{Redirect};
415
337 my $recurse = exists $arg{recurse} ? delete $arg{recurse} : $MAX_RECURSE; 416 my $recurse = exists $arg{recurse} ? delete $arg{recurse} : $MAX_RECURSE;
338 417
339 return $cb->(undef, { Status => 599, Reason => "Too many redirections", URL => $url }) 418 return $cb->(undef, { @pseudo, Status => 599, Reason => "Too many redirections" })
340 if $recurse < 0; 419 if $recurse < 0;
341 420
342 my $proxy = $arg{proxy} || $PROXY; 421 my $proxy = $arg{proxy} || $PROXY;
343 my $timeout = $arg{timeout} || $TIMEOUT; 422 my $timeout = $arg{timeout} || $TIMEOUT;
344 423
345 my ($uscheme, $uauthority, $upath, $query, $fragment) = 424 my ($uscheme, $uauthority, $upath, $query, $fragment) =
346 $url =~ m|(?:([^:/?#]+):)?(?://([^/?#]*))?([^?#]*)(?:\?([^#]*))?(?:#(.*))?|; 425 $url =~ m|(?:([^:/?#]+):)?(?://([^/?#]*))?([^?#]*)(?:(\?[^#]*))?(?:#(.*))?|;
347 426
348 $uscheme = lc $uscheme; 427 $uscheme = lc $uscheme;
349 428
350 my $uport = $uscheme eq "http" ? 80 429 my $uport = $uscheme eq "http" ? 80
351 : $uscheme eq "https" ? 443 430 : $uscheme eq "https" ? 443
352 : return $cb->(undef, { Status => 599, Reason => "Only http and https URL schemes supported", URL => $url }); 431 : return $cb->(undef, { @pseudo, Status => 599, Reason => "Only http and https URL schemes supported" });
353 432
354 $uauthority =~ /^(?: .*\@ )? ([^\@:]+) (?: : (\d+) )?$/x 433 $uauthority =~ /^(?: .*\@ )? ([^\@:]+) (?: : (\d+) )?$/x
355 or return $cb->(undef, { Status => 599, Reason => "Unparsable URL", URL => $url }); 434 or return $cb->(undef, { @pseudo, Status => 599, Reason => "Unparsable URL" });
356 435
357 my $uhost = $1; 436 my $uhost = $1;
358 $uport = $2 if defined $2; 437 $uport = $2 if defined $2;
359 438
360 $hdr{host} = defined $2 ? "$uhost:$2" : "$uhost"; 439 $hdr{host} = defined $2 ? "$uhost:$2" : "$uhost"
440 unless exists $hdr{host};
361 441
362 $uhost =~ s/^\[(.*)\]$/$1/; 442 $uhost =~ s/^\[(.*)\]$/$1/;
363 $upath .= "?$query" if length $query; 443 $upath .= $query if length $query;
364 444
365 $upath =~ s%^/?%/%; 445 $upath =~ s%^/?%/%;
366 446
367 # cookie processing 447 # cookie processing
368 if (my $jar = $arg{cookie_jar}) { 448 if (my $jar = $arg{cookie_jar}) {
398 my ($rhost, $rport, $rscheme, $rpath); # request host, port, path 478 my ($rhost, $rport, $rscheme, $rpath); # request host, port, path
399 479
400 if ($proxy) { 480 if ($proxy) {
401 ($rpath, $rhost, $rport, $rscheme) = ($url, @$proxy); 481 ($rpath, $rhost, $rport, $rscheme) = ($url, @$proxy);
402 482
483 $rscheme = "http" unless defined $rscheme;
484
403 # don't support https requests over https-proxy transport, 485 # don't support https requests over https-proxy transport,
404 # can't be done with tls as spec'ed, unless you double-encrypt. 486 # can't be done with tls as spec'ed, unless you double-encrypt.
405 $rscheme = "http" if $uscheme eq "https" && $rscheme eq "https"; 487 $rscheme = "http" if $uscheme eq "https" && $rscheme eq "https";
406 } else { 488 } else {
407 ($rhost, $rport, $rscheme, $rpath) = ($uhost, $uport, $uscheme, $upath); 489 ($rhost, $rport, $rscheme, $rpath) = ($uhost, $uport, $uscheme, $upath);
408 } 490 }
409 491
410 $hdr{"user-agent"} ||= $USERAGENT; 492 # leave out fragment and query string, just a heuristic
411 $hdr{referer} ||= "$uscheme://$uauthority$upath"; # leave out fragment and query string, just a heuristic 493 $hdr{referer} = "$uscheme://$uauthority$upath" unless exists $hdr{referer};
494 $hdr{"user-agent"} = $USERAGENT unless exists $hdr{"user-agent"};
412 495
413 $hdr{"content-length"} = length $arg{body}; 496 $hdr{"content-length"} = length $arg{body}
497 if length $arg{body} || $method ne "GET";
498
499 $hdr{connection} = "close TE"; #1.1
500 $hdr{te} = "trailers" unless exists $hdr{te}; #1.1
414 501
415 my %state = (connect_guard => 1); 502 my %state = (connect_guard => 1);
416 503
417 _get_slot $uhost, sub { 504 _get_slot $uhost, sub {
418 $state{slot_guard} = shift; 505 $state{slot_guard} = shift;
419 506
420 return unless $state{connect_guard}; 507 return unless $state{connect_guard};
421 508
422 $state{connect_guard} = AnyEvent::Socket::tcp_connect $rhost, $rport, sub { 509 my $connect_cb = sub {
423 $state{fh} = shift 510 $state{fh} = shift
424 or return (%state = (), $cb->(undef, { Status => 599, Reason => "$!", URL => $url })); 511 or do {
512 my $err = "$!";
513 %state = ();
514 return $cb->(undef, { @pseudo, Status => 599, Reason => $err });
515 };
516
425 pop; # free memory, save a tree 517 pop; # free memory, save a tree
426 518
427 return unless delete $state{connect_guard}; 519 return unless delete $state{connect_guard};
428 520
429 # get handle 521 # get handle
430 $state{handle} = new AnyEvent::Handle 522 $state{handle} = new AnyEvent::Handle
431 fh => $state{fh}, 523 fh => $state{fh},
524 peername => $rhost,
525 tls_ctx => $arg{tls_ctx},
526 # these need to be reconfigured on keepalive handles
432 timeout => $timeout, 527 timeout => $timeout,
433 peername => $rhost, 528 on_error => sub {
434 tls_ctx => $arg{tls_ctx}; 529 %state = ();
530 $cb->(undef, { @pseudo, Status => 599, Reason => $_[2] });
531 },
532 on_eof => sub {
533 %state = ();
534 $cb->(undef, { @pseudo, Status => 599, Reason => "Unexpected end-of-file" });
535 },
536 ;
435 537
436 # limit the number of persistent connections 538 # limit the number of persistent connections
437 # keepalive not yet supported 539 # keepalive not yet supported
438 if ($KA_COUNT{$_[1]} < $MAX_PERSISTENT_PER_HOST) { 540# if ($KA_COUNT{$_[1]} < $MAX_PERSISTENT_PER_HOST) {
439 ++$KA_COUNT{$_[1]}; 541# ++$KA_COUNT{$_[1]};
440 $state{handle}{ka_count_guard} = AnyEvent::Util::guard { 542# $state{handle}{ka_count_guard} = AnyEvent::Util::guard {
441 --$KA_COUNT{$_[1]} 543# --$KA_COUNT{$_[1]}
442 }; 544# };
443 $hdr{connection} = "keep-alive"; 545# $hdr{connection} = "keep-alive";
444 } else {
445 delete $hdr{connection};
446 } 546# }
447
448 # (re-)configure handle
449 $state{handle}->on_error (sub {
450 %state = ();
451 $cb->(undef, { Status => 599, Reason => $_[2], URL => $url });
452 });
453 $state{handle}->on_eof (sub {
454 %state = ();
455 $cb->(undef, { Status => 599, Reason => "Unexpected end-of-file", URL => $url });
456 });
457 547
458 $state{handle}->starttls ("connect") if $rscheme eq "https"; 548 $state{handle}->starttls ("connect") if $rscheme eq "https";
459 549
460 # handle actual, non-tunneled, request 550 # handle actual, non-tunneled, request
461 my $handle_actual_request = sub { 551 my $handle_actual_request = sub {
462 $state{handle}->starttls ("connect") if $uscheme eq "https" && !exists $state{handle}{tls}; 552 $state{handle}->starttls ("connect") if $uscheme eq "https" && !exists $state{handle}{tls};
463 553
464 # send request 554 # send request
465 $state{handle}->push_write ( 555 $state{handle}->push_write (
466 "$method $rpath HTTP/1.0\015\012" 556 "$method $rpath HTTP/1.1\015\012"
467 . (join "", map "\u$_: $hdr{$_}\015\012", keys %hdr) 557 . (join "", map "\u$_: $hdr{$_}\015\012", grep defined $hdr{$_}, keys %hdr)
468 . "\015\012" 558 . "\015\012"
469 . (delete $arg{body}) 559 . (delete $arg{body})
470 ); 560 );
471 561
562 # return if error occured during push_write()
563 return unless %state;
564
472 %hdr = (); # reduce memory usage, save a kitten 565 %hdr = (); # reduce memory usage, save a kitten, also make it possible to re-use
473 566
474 # status line 567 # status line and headers
475 $state{handle}->push_read (line => $qr_nl, sub { 568 $state{read_response} = sub {
476 $_[1] =~ /^HTTP\/([0-9\.]+) \s+ ([0-9]{3}) (?: \s+ ([^\015\012]*) )?/ix
477 or return (%state = (), $cb->(undef, { Status => 599, Reason => "Invalid server response ($_[1])", URL => $url }));
478
479 my %hdr = ( # response headers
480 HTTPVersion => ",$1",
481 Status => ",$2",
482 Reason => ",$3",
483 URL => ",$url"
484 );
485
486 # headers, could be optimized a bit
487 $state{handle}->unshift_read (line => $qr_nlnl, sub {
488 for ("$_[1]\012") { 569 for ("$_[1]") {
489 y/\015//d; # weed out any \015, as they show up in the weirdest of places. 570 y/\015//d; # weed out any \015, as they show up in the weirdest of places.
490 571
491 # things seen, not parsed: 572 /^HTTP\/([0-9\.]+) \s+ ([0-9]{3}) (?: \s+ ([^\012]*) )? \012/igxc
492 # p3pP="NON CUR OTPi OUR NOR UNI" 573 or return (%state = (), $cb->(undef, { @pseudo, Status => 599, Reason => "Invalid server response" }));
493 574
494 $hdr{lc $1} .= ",$2" 575 # 100 Continue handling
495 while /\G 576 # should not happen as we don't send expect: 100-continue,
496 ([^:\000-\037]*): 577 # but we handle it just in case.
497 [\011\040]* 578 # since we send the request body regardless, if we get an error
498 ((?: [^\012]+ | \012[\011\040] )*) 579 # we are out of-sync, which we currently do NOT handle correctly.
499 \012 580 return $state{handle}->push_read (line => $qr_nlnl, $state{read_response})
500 /gxc; 581 if $2 eq 100;
501 582
583 push @pseudo,
584 HTTPVersion => $1,
585 Status => $2,
586 Reason => $3,
502 /\G$/ 587 ;
588
589 my $hdr = parse_hdr
503 or return (%state = (), $cb->(undef, { Status => 599, Reason => "Garbled response headers", URL => $url })); 590 or return (%state = (), $cb->(undef, { @pseudo, Status => 599, Reason => "Garbled response headers" }));
591
592 %hdr = (%$hdr, @pseudo);
593 }
594
595 # redirect handling
596 # microsoft and other shitheads don't give a shit for following standards,
597 # try to support some common forms of broken Location headers.
598 if ($hdr{location} !~ /^(?: $ | [^:\/?\#]+ : )/x) {
599 $hdr{location} =~ s/^\.\/+//;
600
601 my $url = "$rscheme://$uhost:$uport";
602
603 unless ($hdr{location} =~ s/^\///) {
604 $url .= $upath;
605 $url =~ s/\/[^\/]*$//;
504 } 606 }
505 607
506 substr $_, 0, 1, ""
507 for values %hdr;
508
509 # redirect handling
510 # microsoft and other shitheads don't give a shit for following standards,
511 # try to support some common forms of broken Location headers.
512 if ($hdr{location} !~ /^(?: $ | [^:\/?\#]+ : )/x) {
513 $hdr{location} =~ s/^\.\/+//;
514
515 my $url = "$rscheme://$uhost:$uport";
516
517 unless ($hdr{location} =~ s/^\///) {
518 $url .= $upath;
519 $url =~ s/\/[^\/]*$//;
520 }
521
522 $hdr{location} = "$url/$hdr{location}"; 608 $hdr{location} = "$url/$hdr{location}";
609 }
610
611 my $redirect;
612
613 if ($recurse) {
614 my $status = $hdr{Status};
615
616 # industry standard is to redirect POST as GET for
617 # 301, 302 and 303, in contrast to http/1.0 and 1.1.
618 # also, the UA should ask the user for 301 and 307 and POST,
619 # industry standard seems to be to simply follow.
620 # we go with the industry standard.
621 if ($status == 301 or $status == 302 or $status == 303) {
622 # HTTP/1.1 is unclear on how to mutate the method
623 $method = "GET" unless $method eq "HEAD";
624 $redirect = 1;
625 } elsif ($status == 307) {
626 $redirect = 1;
523 } 627 }
628 }
524 629
630 my $finish = sub { # ($data, $err_status, $err_reason[, $keepalive])
525 my $redirect; 631 my $keepalive = pop;
526 632
633 $state{handle}->destroy if $state{handle};
634 %state = ();
635
636 if (defined $_[1]) {
637 $hdr{OrigStatus} = $hdr{Status}; $hdr{Status} = $_[1];
638 $hdr{OrigReason} = $hdr{Reason}; $hdr{Reason} = $_[2];
639 }
640
641 # set-cookie processing
527 if ($recurse) { 642 if ($arg{cookie_jar}) {
528 if ($hdr{Status} =~ /^30[12]$/ && $method ne "POST") { 643 for ($hdr{"set-cookie"}) {
529 # apparently, mozilla et al. just change POST to GET here 644 # parse NAME=VALUE
530 # more research is needed before we do the same 645 my @kv;
646
647 while (/\G\s* ([^=;,[:space:]]+) \s*=\s* (?: "((?:[^\\"]+|\\.)*)" | ([^=;,[:space:]]*) )/gcxs) {
531 $redirect = 1; 648 my $name = $1;
532 } elsif ($hdr{Status} == 303) { 649 my $value = $3;
533 # even http/1.1 is unclear on how to mutate the method 650
534 $method = "GET" unless $method eq "HEAD"; 651 unless ($value) {
652 $value = $2;
653 $value =~ s/\\(.)/$1/gs;
654 }
655
656 push @kv, $name => $value;
657
658 last unless /\G\s*;/gc;
659 }
660
661 last unless @kv;
662
663 my $name = shift @kv;
664 my %kv = (value => shift @kv, @kv);
665
666 my $cdom;
667 my $cpath = (delete $kv{path}) || "/";
668
669 if (exists $kv{domain}) {
670 $cdom = delete $kv{domain};
671
672 $cdom =~ s/^\.?/./; # make sure it starts with a "."
673
674 next if $cdom =~ /\.$/;
675
676 # this is not rfc-like and not netscape-like. go figure.
677 my $ndots = $cdom =~ y/.//;
678 next if $ndots < ($cdom =~ /\.[^.][^.]\.[^.][^.]$/ ? 3 : 2);
679 } else {
680 $cdom = $uhost;
681 }
682
535 $redirect = 1; 683 # store it
536 } elsif ($hdr{Status} == 307 && $method =~ /^(?:GET|HEAD)$/) { 684 $arg{cookie_jar}{version} = 1;
537 $redirect = 1; 685 $arg{cookie_jar}{$cdom}{$cpath}{$name} = \%kv;
686
687 redo if /\G\s*,/gc;
538 } 688 }
539 } 689 }
540 690
541 my $finish = sub { 691 if ($redirect && exists $hdr{location}) {
542 $state{handle}->destroy if $state{handle};
543 %state = ();
544
545 # set-cookie processing
546 if ($arg{cookie_jar}) {
547 for ($_[1]{"set-cookie"}) {
548 # parse NAME=VALUE
549 my @kv;
550
551 while (/\G\s* ([^=;,[:space:]]+) \s*=\s* (?: "((?:[^\\"]+|\\.)*)" | ([^=;,[:space:]]*) )/gcxs) {
552 my $name = $1;
553 my $value = $3;
554
555 unless ($value) {
556 $value = $2;
557 $value =~ s/\\(.)/$1/gs;
558 }
559
560 push @kv, $name => $value;
561
562 last unless /\G\s*;/gc;
563 }
564
565 last unless @kv;
566
567 my $name = shift @kv;
568 my %kv = (value => shift @kv, @kv);
569
570 my $cdom;
571 my $cpath = (delete $kv{path}) || "/";
572
573 if (exists $kv{domain}) {
574 $cdom = delete $kv{domain};
575
576 $cdom =~ s/^\.?/./; # make sure it starts with a "."
577
578 next if $cdom =~ /\.$/;
579
580 # this is not rfc-like and not netscape-like. go figure.
581 my $ndots = $cdom =~ y/.//;
582 next if $ndots < ($cdom =~ /\.[^.][^.]\.[^.][^.]$/ ? 3 : 2);
583 } else {
584 $cdom = $uhost;
585 }
586
587 # store it
588 $arg{cookie_jar}{version} = 1;
589 $arg{cookie_jar}{$cdom}{$cpath}{$name} = \%kv;
590
591 redo if /\G\s*,/gc;
592 }
593 }
594
595 if ($redirect) {
596 # we ignore any errors, as it is very common to receive 692 # we ignore any errors, as it is very common to receive
597 # Content-Length != 0 but no actual body 693 # Content-Length != 0 but no actual body
598 # we also access %hdr, as $_[1] might be an erro 694 # we also access %hdr, as $_[1] might be an erro
599 http_request ($method => $hdr{location}, %arg, recurse => $recurse - 1, $cb); 695 http_request (
600 } else { 696 $method => $hdr{location},
601 $cb->($_[0], $_[1]);
602 } 697 %arg,
698 recurse => $recurse - 1,
699 Redirect => [$_[0], \%hdr],
603 }; 700 $cb);
604
605 my $len = $hdr{"content-length"};
606
607 if (!$redirect && $arg{on_header} && !$arg{on_header}(\%hdr)) {
608 $finish->(undef, { Status => 598, Reason => "Request cancelled by on_header", URL => $url });
609 } elsif (
610 $hdr{Status} =~ /^(?:1..|[23]04)$/
611 or $method eq "HEAD"
612 or (defined $len && !$len)
613 ) {
614 # no body
615 $finish->("", \%hdr);
616 } else { 701 } else {
702 $cb->($_[0], \%hdr);
703 }
704 };
705
706 my $len = $hdr{"content-length"};
707
708 if (!$redirect && $arg{on_header} && !$arg{on_header}(\%hdr)) {
709 $finish->(undef, 598 => "Request cancelled by on_header");
710 } elsif (
711 $hdr{Status} =~ /^(?:1..|204|205|304)$/
712 or $method eq "HEAD"
713 or (defined $len && !$len)
714 ) {
715 # no body
716 $finish->("", undef, undef, 1);
717 } else {
617 # body handling, four different code paths 718 # body handling, many different code paths
618 # for want_body_handle, on_body (2x), normal (2x) 719 # - no body expected
619 # we might read too much here, but it does not matter yet (no pers. connections) 720 # - want_body_handle
721 # - te chunked
722 # - 2x length known (with or without on_body)
723 # - 2x length not known (with or without on_body)
620 if (!$redirect && $arg{want_body_handle}) { 724 if (!$redirect && $arg{want_body_handle}) {
621 $_[0]->on_eof (undef); 725 $_[0]->on_eof (undef);
622 $_[0]->on_error (undef); 726 $_[0]->on_error (undef);
623 $_[0]->on_read (undef); 727 $_[0]->on_read (undef);
624 728
625 $finish->(delete $state{handle}, \%hdr); 729 $finish->(delete $state{handle});
626 730
627 } elsif ($arg{on_body}) { 731 } elsif ($hdr{"transfer-encoding"} =~ /\bchunked\b/i) {
732 my $cl = 0;
733 my $body = undef;
734 my $on_body = $arg{on_body} || sub { $body .= shift; 1 };
735
628 $_[0]->on_error (sub { $finish->(undef, { Status => 599, Reason => $_[2], URL => $url }) }); 736 $_[0]->on_error (sub { $finish->(undef, 599 => $_[2]) });
737
738 my $read_chunk; $read_chunk = sub {
739 $_[1] =~ /^([0-9a-fA-F]+)/
740 or $finish->(undef, 599 => "Garbled chunked transfer encoding");
741
742 my $len = hex $1;
743
629 if ($len) { 744 if ($len) {
630 $_[0]->on_eof (undef); 745 $cl += $len;
746
747 $_[0]->push_read (chunk => $len, sub {
748 $on_body->($_[1], \%hdr)
749 or return $finish->(undef, 598 => "Request cancelled by on_body");
750
631 $_[0]->on_read (sub { 751 $_[0]->push_read (line => sub {
632 $len -= length $_[0]{rbuf}; 752 length $_[1]
633 753 and return $finish->(undef, 599 => "Garbled chunked transfer encoding");
634 $arg{on_body}(delete $_[0]{rbuf}, \%hdr) 754 $_[0]->push_read (line => $read_chunk);
635 or $finish->(undef, { Status => 598, Reason => "Request cancelled by on_body", URL => $url });
636
637 $len > 0 755 });
638 or $finish->("", \%hdr);
639 }); 756 });
640 } else { 757 } else {
641 $_[0]->on_eof (sub { 758 $hdr{"content-length"} ||= $cl;
642 $finish->("", \%hdr); 759
760 $_[0]->push_read (line => $qr_nlnl, sub {
761 if (length $_[1]) {
762 for ("$_[1]") {
763 y/\015//d; # weed out any \015, as they show up in the weirdest of places.
764
765 my $hdr = parse_hdr
766 or return $finish->(undef, 599 => "Garbled response trailers");
767
768 %hdr = (%hdr, %$hdr);
769 }
643 }); 770 }
644 $_[0]->on_read (sub { 771
645 $arg{on_body}(delete $_[0]{rbuf}, \%hdr) 772 $finish->($body, undef, undef, 1);
646 or $finish->(undef, { Status => 598, Reason => "Request cancelled by on_body", URL => $url });
647 }); 773 });
648 } 774 }
775 };
776
777 $_[0]->push_read (line => $read_chunk);
778
779 } elsif ($arg{on_body}) {
780 $_[0]->on_error (sub { $finish->(undef, 599 => $_[2]) });
781
782 if ($len) {
783 $_[0]->on_read (sub {
784 $len -= length $_[0]{rbuf};
785
786 $arg{on_body}(delete $_[0]{rbuf}, \%hdr)
787 or return $finish->(undef, 598 => "Request cancelled by on_body");
788
789 $len > 0
790 or $finish->("", undef, undef, 1);
791 });
649 } else { 792 } else {
650 $_[0]->on_eof (undef); 793 $_[0]->on_eof (sub {
651 794 $finish->("");
652 if ($len) { 795 });
653 $_[0]->on_error (sub { $finish->(undef, { Status => 599, Reason => $_[2], URL => $url }) });
654 $_[0]->on_read (sub { 796 $_[0]->on_read (sub {
797 $arg{on_body}(delete $_[0]{rbuf}, \%hdr)
798 or $finish->(undef, 598 => "Request cancelled by on_body");
799 });
800 }
801 } else {
802 $_[0]->on_eof (undef);
803
804 if ($len) {
805 $_[0]->on_error (sub { $finish->(undef, 599 => $_[2]) });
806 $_[0]->on_read (sub {
655 $finish->((substr delete $_[0]{rbuf}, 0, $len, ""), \%hdr) 807 $finish->((substr delete $_[0]{rbuf}, 0, $len, ""), undef, undef, 1)
656 if $len <= length $_[0]{rbuf}; 808 if $len <= length $_[0]{rbuf};
657 }); 809 });
658 } else { 810 } else {
659 $_[0]->on_error (sub { 811 $_[0]->on_error (sub {
660 $! == Errno::EPIPE || !$! 812 ($! == Errno::EPIPE || !$!)
661 ? $finish->(delete $_[0]{rbuf}, \%hdr) 813 ? $finish->(delete $_[0]{rbuf})
662 : $finish->(undef, { Status => 599, Reason => $_[2], URL => $url }); 814 : $finish->(undef, 599 => $_[2]);
663 }); 815 });
664 $_[0]->on_read (sub { }); 816 $_[0]->on_read (sub { });
665 }
666 } 817 }
667 } 818 }
668 }); 819 }
669 }); 820 };
821
822 $state{handle}->push_read (line => $qr_nlnl, $state{read_response});
670 }; 823 };
671 824
672 # now handle proxy-CONNECT method 825 # now handle proxy-CONNECT method
673 if ($proxy && $uscheme eq "https") { 826 if ($proxy && $uscheme eq "https") {
674 # oh dear, we have to wrap it into a connect request 827 # oh dear, we have to wrap it into a connect request
675 828
676 # maybe re-use $uauthority with patched port? 829 # maybe re-use $uauthority with patched port?
677 $state{handle}->push_write ("CONNECT $uhost:$uport HTTP/1.0\015\012Host: $uhost\015\012\015\012"); 830 $state{handle}->push_write ("CONNECT $uhost:$uport HTTP/1.0\015\012Host: $uhost\015\012\015\012");
678 $state{handle}->push_read (line => $qr_nlnl, sub { 831 $state{handle}->push_read (line => $qr_nlnl, sub {
679 $_[1] =~ /^HTTP\/([0-9\.]+) \s+ ([0-9]{3}) (?: \s+ ([^\015\012]*) )?/ix 832 $_[1] =~ /^HTTP\/([0-9\.]+) \s+ ([0-9]{3}) (?: \s+ ([^\015\012]*) )?/ix
680 or return (%state = (), $cb->(undef, { Status => 599, Reason => "Invalid proxy connect response ($_[1])", URL => $url })); 833 or return (%state = (), $cb->(undef, { @pseudo, Status => 599, Reason => "Invalid proxy connect response ($_[1])" }));
681 834
682 if ($2 == 200) { 835 if ($2 == 200) {
683 $rpath = $upath; 836 $rpath = $upath;
684 &$handle_actual_request; 837 &$handle_actual_request;
685 } else { 838 } else {
686 %state = (); 839 %state = ();
687 $cb->(undef, { Status => $2, Reason => $3, URL => $url }); 840 $cb->(undef, { @pseudo, Status => $2, Reason => $3 });
688 } 841 }
689 }); 842 });
690 } else { 843 } else {
691 &$handle_actual_request; 844 &$handle_actual_request;
692 } 845 }
693
694 }, sub {
695 $timeout
696 }; 846 };
847
848 my $tcp_connect = $arg{tcp_connect}
849 || do { require AnyEvent::Socket; \&AnyEvent::Socket::tcp_connect };
850
851 $state{connect_guard} = $tcp_connect->($rhost, $rport, $connect_cb, $arg{on_prepare} || sub { $timeout });
852
697 }; 853 };
698 854
699 defined wantarray && AnyEvent::Util::guard { %state = () } 855 defined wantarray && AnyEvent::Util::guard { %state = () }
700} 856}
701 857
715 &http_request 871 &http_request
716} 872}
717 873
718=back 874=back
719 875
876=head2 DNS CACHING
877
878AnyEvent::HTTP uses the AnyEvent::Socket::tcp_connect function for
879the actual connection, which in turn uses AnyEvent::DNS to resolve
880hostnames. The latter is a simple stub resolver and does no caching
881on its own. If you want DNS caching, you currently have to provide
882your own default resolver (by storing a suitable resolver object in
883C<$AnyEvent::DNS::RESOLVER>).
884
720=head2 GLOBAL FUNCTIONS AND VARIABLES 885=head2 GLOBAL FUNCTIONS AND VARIABLES
721 886
722=over 4 887=over 4
723 888
724=item AnyEvent::HTTP::set_proxy "proxy-url" 889=item AnyEvent::HTTP::set_proxy "proxy-url"
725 890
726Sets the default proxy server to use. The proxy-url must begin with a 891Sets the default proxy server to use. The proxy-url must begin with a
727string of the form C<http://host:port> (optionally C<https:...>). 892string of the form C<http://host:port> (optionally C<https:...>), croaks
893otherwise.
894
895To clear an already-set proxy, use C<undef>.
896
897=item $date = AnyEvent::HTTP::format_date $timestamp
898
899Takes a POSIX timestamp (seconds since the epoch) and formats it as a HTTP
900Date (RFC 2616).
901
902=item $timestamp = AnyEvent::HTTP::parse_date $date
903
904Takes a HTTP Date (RFC 2616) and returns the corresponding POSIX
905timestamp, or C<undef> if the date cannot be parsed.
728 906
729=item $AnyEvent::HTTP::MAX_RECURSE 907=item $AnyEvent::HTTP::MAX_RECURSE
730 908
731The default value for the C<recurse> request parameter (default: C<10>). 909The default value for the C<recurse> request parameter (default: C<10>).
732 910
735The default value for the C<User-Agent> header (the default is 913The default value for the C<User-Agent> header (the default is
736C<Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)>). 914C<Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)>).
737 915
738=item $AnyEvent::HTTP::MAX_PER_HOST 916=item $AnyEvent::HTTP::MAX_PER_HOST
739 917
740The maximum number of concurrent conenctions to the same host (identified 918The maximum number of concurrent connections to the same host (identified
741by the hostname). If the limit is exceeded, then the additional requests 919by the hostname). If the limit is exceeded, then the additional requests
742are queued until previous connections are closed. 920are queued until previous connections are closed.
743 921
744The default value for this is C<4>, and it is highly advisable to not 922The default value for this is C<4>, and it is highly advisable to not
745increase it. 923increase it.
752 930
753=back 931=back
754 932
755=cut 933=cut
756 934
935our @month = qw(Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec);
936our @weekday = qw(Sun Mon Tue Wed Thu Fri Sat);
937
938sub format_date($) {
939 my ($time) = @_;
940
941 # RFC 822/1123 format
942 my ($S, $M, $H, $mday, $mon, $year, $wday, $yday, undef) = gmtime $time;
943
944 sprintf "%s, %02d %s %04d %02d:%02d:%02d GMT",
945 $weekday[$wday], $mday, $month[$mon], $year + 1900,
946 $H, $M, $S;
947}
948
949sub parse_date($) {
950 my ($date) = @_;
951
952 my ($d, $m, $y, $H, $M, $S);
953
954 if ($date =~ /^[A-Z][a-z][a-z], ([0-9][0-9]) ([A-Z][a-z][a-z]) ([0-9][0-9][0-9][0-9]) ([0-9][0-9]):([0-9][0-9]):([0-9][0-9]) GMT$/) {
955 # RFC 822/1123, required by RFC 2616
956 ($d, $m, $y, $H, $M, $S) = ($1, $2, $3, $4, $5, $6);
957
958 } elsif ($date =~ /^[A-Z][a-z]+, ([0-9][0-9])-([A-Z][a-z][a-z])-([0-9][0-9]) ([0-9][0-9]):([0-9][0-9]):([0-9][0-9]) GMT$/) {
959 # RFC 850
960 ($d, $m, $y, $H, $M, $S) = ($1, $2, $3 < 69 ? $3 + 2000 : $3 + 1900, $4, $5, $6);
961
962 } elsif ($date =~ /^[A-Z][a-z][a-z] ([A-Z][a-z][a-z]) ([0-9 ][0-9]) ([0-9][0-9]):([0-9][0-9]):([0-9][0-9]) ([0-9][0-9][0-9][0-9])$/) {
963 # ISO C's asctime
964 ($d, $m, $y, $H, $M, $S) = ($2, $1, $6, $3, $4, $5);
965 }
966 # other formats fail in the loop below
967
968 for (0..11) {
969 if ($m eq $month[$_]) {
970 require Time::Local;
971 return Time::Local::timegm ($S, $M, $H, $d, $_, $y);
972 }
973 }
974
975 undef
976}
977
757sub set_proxy($) { 978sub set_proxy($) {
979 if (length $_[0]) {
758 $PROXY = [$2, $3 || 3128, $1] if $_[0] =~ m%^(https?):// ([^:/]+) (?: : (\d*) )?%ix; 980 $_[0] =~ m%^(https?):// ([^:/]+) (?: : (\d*) )?%ix
981 or Carp::croak "$_[0]: invalid proxy URL";
982 $PROXY = [$2, $3 || 3128, $1]
983 } else {
984 undef $PROXY;
985 }
759} 986}
760 987
761# initialise proxy from environment 988# initialise proxy from environment
989eval {
762set_proxy $ENV{http_proxy}; 990 set_proxy $ENV{http_proxy};
991};
992
993=head2 SOCKS PROXIES
994
995Socks proxies are not directly supported by AnyEvent::HTTP. You can
996compile your perl to support socks, or use an external program such as
997F<socksify> (dante) or F<tsocks> to make your program use a socks proxy
998transparently.
999
1000Alternatively, for AnyEvent::HTTP only, you can use your own
1001C<tcp_connect> function that does the proxy handshake - here is an example
1002that works with socks4a proxies:
1003
1004 use Errno;
1005 use AnyEvent::Util;
1006 use AnyEvent::Socket;
1007 use AnyEvent::Handle;
1008
1009 # host, port and username of/for your socks4a proxy
1010 my $socks_host = "10.0.0.23";
1011 my $socks_port = 9050;
1012 my $socks_user = "";
1013
1014 sub socks4a_connect {
1015 my ($host, $port, $connect_cb, $prepare_cb) = @_;
1016
1017 my $hdl = new AnyEvent::Handle
1018 connect => [$socks_host, $socks_port],
1019 on_prepare => sub { $prepare_cb->($_[0]{fh}) },
1020 on_error => sub { $connect_cb->() },
1021 ;
1022
1023 $hdl->push_write (pack "CCnNZ*Z*", 4, 1, $port, 1, $socks_user, $host);
1024
1025 $hdl->push_read (chunk => 8, sub {
1026 my ($hdl, $chunk) = @_;
1027 my ($status, $port, $ipn) = unpack "xCna4", $chunk;
1028
1029 if ($status == 0x5a) {
1030 $connect_cb->($hdl->{fh}, (format_address $ipn) . ":$port");
1031 } else {
1032 $! = Errno::ENXIO; $connect_cb->();
1033 }
1034 });
1035
1036 $hdl
1037 }
1038
1039Use C<socks4a_connect> instead of C<tcp_connect> when doing C<http_request>s,
1040possibly after switching off other proxy types:
1041
1042 AnyEvent::HTTP::set_proxy undef; # usually you do not want other proxies
1043
1044 http_get 'http://www.google.com', tcp_connect => \&socks4a_connect, sub {
1045 my ($data, $headers) = @_;
1046 ...
1047 };
763 1048
764=head1 SEE ALSO 1049=head1 SEE ALSO
765 1050
766L<AnyEvent>. 1051L<AnyEvent>.
767 1052

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines