ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-HTTP/HTTP.pm
(Generate patch)

Comparing AnyEvent-HTTP/HTTP.pm (file contents):
Revision 1.62 by root, Thu Dec 30 04:30:24 2010 UTC vs.
Revision 1.69 by root, Fri Dec 31 19:32:47 2010 UTC

… …
47use AnyEvent::Util (); 47use AnyEvent::Util ();
48use AnyEvent::Handle (); 48use AnyEvent::Handle ();
49 49
50use base Exporter::; 50use base Exporter::;
51 51
52our $VERSION = '1.46'; 52our $VERSION = '1.5';
53 53
54our @EXPORT = qw(http_get http_post http_head http_request); 54our @EXPORT = qw(http_get http_post http_head http_request);
55 55
56our $USERAGENT = "Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)"; 56our $USERAGENT = "Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)";
57our $MAX_RECURSE = 10; 57our $MAX_RECURSE = 10;
… …
94C<http_request> returns a "cancellation guard" - you have to keep the 94C<http_request> returns a "cancellation guard" - you have to keep the
95object at least alive until the callback get called. If the object gets 95object at least alive until the callback get called. If the object gets
96destroyed before the callback is called, the request will be cancelled. 96destroyed before the callback is called, the request will be cancelled.
97 97
98The callback will be called with the response body data as first argument 98The callback will be called with the response body data as first argument
99(or C<undef> if an error occured), and a hash-ref with response headers as 99(or C<undef> if an error occured), and a hash-ref with response headers
100second argument. 100(and trailers) as second argument.
101 101
102All the headers in that hash are lowercased. In addition to the response 102All the headers in that hash are lowercased. In addition to the response
103headers, the "pseudo-headers" (uppercase to avoid clashing with possible 103headers, the "pseudo-headers" (uppercase to avoid clashing with possible
104response headers) C<HTTPVersion>, C<Status> and C<Reason> contain the 104response headers) C<HTTPVersion>, C<Status> and C<Reason> contain the
105three parts of the HTTP Status-Line of the same name. 105three parts of the HTTP Status-Line of the same name. If an error occurs
106during the body phase of a request, then the original C<Status> and
107C<Reason> values from the header are available as C<OrigStatus> and
108C<OrigReason>.
106 109
107The pseudo-header C<URL> contains the actual URL (which can differ from 110The pseudo-header C<URL> contains the actual URL (which can differ from
108the requested URL when following redirects - for example, you might get 111the requested URL when following redirects - for example, you might get
109an error that your URL scheme is not supported even though your URL is a 112an error that your URL scheme is not supported even though your URL is a
110valid http URL because it redirected to an ftp URL, in which case you can 113valid http URL because it redirected to an ftp URL, in which case you can
… …
147Whether to recurse requests or not, e.g. on redirects, authentication 150Whether to recurse requests or not, e.g. on redirects, authentication
148retries and so on, and how often to do so. 151retries and so on, and how often to do so.
149 152
150=item headers => hashref 153=item headers => hashref
151 154
152The request headers to use, with the header name (I<MUST be in lowercase>) 155The request headers to use. Currently, C<http_request> may provide its own
153as key and header value as hash value. 156C<Host:>, C<Content-Length:>, C<Connection:> and C<Cookie:> headers and
154 157will provide defaults at least for C<TE:>, C<Referer:> and C<User-Agent:>
155Currently, http_request> may provide its own C<host>, C<content-length>, 158(this can be suppressed by using C<undef> for these headers in which case
156C<connection> and C<cookie> headers and will provide defaults for 159they won't be sent at all).
157C<user-agent> and C<referer> (this can be suppressed by using a value of
158C<undef> for these headers in which case they won't be sent at all).
159 160
160=item timeout => $seconds 161=item timeout => $seconds
161 162
162The time-out to use for various stages - each connect attempt will reset 163The time-out to use for various stages - each connect attempt will reset
163the timeout, as will read or write activity, i.e. this is not an overall 164the timeout, as will read or write activity, i.e. this is not an overall
… …
173C<$scheme> must be either missing, C<http> for HTTP or C<https> for 174C<$scheme> must be either missing, C<http> for HTTP or C<https> for
174HTTPS. 175HTTPS.
175 176
176=item body => $string 177=item body => $string
177 178
178The request body, usually empty. Will be-sent as-is (future versions of 179The request body, usually empty. Will be sent as-is (future versions of
179this module might offer more options). 180this module might offer more options).
180 181
181=item cookie_jar => $hash_ref 182=item cookie_jar => $hash_ref
182 183
183Passing this parameter enables (simplified) cookie-processing, loosely 184Passing this parameter enables (simplified) cookie-processing, loosely
… …
191Note that this cookie implementation is not of very high quality, nor 192Note that this cookie implementation is not of very high quality, nor
192meant to be complete. If you want complete cookie management you have to 193meant to be complete. If you want complete cookie management you have to
193do that on your own. C<cookie_jar> is meant as a quick fix to get some 194do that on your own. C<cookie_jar> is meant as a quick fix to get some
194cookie-using sites working. Cookies are a privacy disaster, do not use 195cookie-using sites working. Cookies are a privacy disaster, do not use
195them unless required to. 196them unless required to.
197
198When cookie processing is enabled, the C<Cookie:> and C<Set-Cookie:>
199headers will be ste and handled by this module, otherwise they will be
200left untouched.
196 201
197=item tls_ctx => $scheme | $tls_ctx 202=item tls_ctx => $scheme | $tls_ctx
198 203
199Specifies the AnyEvent::TLS context to be used for https connections. This 204Specifies the AnyEvent::TLS context to be used for https connections. This
200parameter follows the same rules as the C<tls_ctx> parameter to 205parameter follows the same rules as the C<tls_ctx> parameter to
… …
239 244
240This callback is useful, among other things, to quickly reject unwanted 245This callback is useful, among other things, to quickly reject unwanted
241content, which, if it is supposed to be rare, can be faster than first 246content, which, if it is supposed to be rare, can be faster than first
242doing a C<HEAD> request. 247doing a C<HEAD> request.
243 248
249The downside is that cancelling the request makes it impossible to re-use
250the connection. Also, the C<on_header> callback will not receive any
251trailer (headers sent after the response body).
252
244Example: cancel the request unless the content-type is "text/html". 253Example: cancel the request unless the content-type is "text/html".
245 254
246 on_header => sub { 255 on_header => sub {
247 $_[0]{"content-type"} =~ /^text\/html\s*(?:;|$)/ 256 $_[0]{"content-type"} =~ /^text\/html\s*(?:;|$)/
248 }, 257 },
… …
254string instead of the body data. 263string instead of the body data.
255 264
256It has to return either true (in which case AnyEvent::HTTP will continue), 265It has to return either true (in which case AnyEvent::HTTP will continue),
257or false, in which case AnyEvent::HTTP will cancel the download (and call 266or false, in which case AnyEvent::HTTP will cancel the download (and call
258the completion callback with an error code of C<598>). 267the completion callback with an error code of C<598>).
268
269The downside to cancelling the request is that it makes it impossible to
270re-use the connection.
259 271
260This callback is useful when the data is too large to be held in memory 272This callback is useful when the data is too large to be held in memory
261(so the callback writes it to a file) or when only some information should 273(so the callback writes it to a file) or when only some information should
262be extracted, or when the body should be processed incrementally. 274be extracted, or when the body should be processed incrementally.
263 275
… …
289If you think you need this, first have a look at C<on_body>, to see if 301If you think you need this, first have a look at C<on_body>, to see if
290that doesn't solve your problem in a better way. 302that doesn't solve your problem in a better way.
291 303
292=back 304=back
293 305
294Example: make a simple HTTP GET request for http://www.nethype.de/ 306Example: do a simple HTTP GET request for http://www.nethype.de/ and print
307the response body.
295 308
296 http_request GET => "http://www.nethype.de/", sub { 309 http_request GET => "http://www.nethype.de/", sub {
297 my ($body, $hdr) = @_; 310 my ($body, $hdr) = @_;
298 print "$body\n"; 311 print "$body\n";
299 }; 312 };
300 313
301Example: make a HTTP HEAD request on https://www.google.com/, use a 314Example: do a HTTP HEAD request on https://www.google.com/, use a
302timeout of 30 seconds. 315timeout of 30 seconds.
303 316
304 http_request 317 http_request
305 GET => "https://www.google.com", 318 GET => "https://www.google.com",
306 timeout => 30, 319 timeout => 30,
… …
309 use Data::Dumper; 322 use Data::Dumper;
310 print Dumper $hdr; 323 print Dumper $hdr;
311 } 324 }
312 ; 325 ;
313 326
314Example: make another simple HTTP GET request, but immediately try to 327Example: do another simple HTTP GET request, but immediately try to
315cancel it. 328cancel it.
316 329
317 my $request = http_request GET => "http://www.nethype.de/", sub { 330 my $request = http_request GET => "http://www.nethype.de/", sub {
318 my ($body, $hdr) = @_; 331 my ($body, $hdr) = @_;
319 print "$body\n"; 332 print "$body\n";
… …
351 push @{ $CO_SLOT{$_[0]}[1] }, $_[1]; 364 push @{ $CO_SLOT{$_[0]}[1] }, $_[1];
352 365
353 _slot_schedule $_[0]; 366 _slot_schedule $_[0];
354} 367}
355 368
369# continue to parse $_ for headers and place them into the arg
370sub parse_hdr() {
371 my %hdr;
372
373 # things seen, not parsed:
374 # p3pP="NON CUR OTPi OUR NOR UNI"
375
376 $hdr{lc $1} .= ",$2"
377 while /\G
378 ([^:\000-\037]*):
379 [\011\040]*
380 ((?: [^\012]+ | \012[\011\040] )*)
381 \012
382 /gxc;
383
384 /\G$/
385 or return;
386
387 # remove the "," prefix we added to all headers above
388 substr $_, 0, 1, ""
389 for values %hdr;
390
391 \%hdr
392}
393
356our $qr_nlnl = qr{(?<![^\012])\015?\012}; 394our $qr_nlnl = qr{(?<![^\012])\015?\012};
357 395
358our $TLS_CTX_LOW = { cache => 1, sslv2 => 1 }; 396our $TLS_CTX_LOW = { cache => 1, sslv2 => 1 };
359our $TLS_CTX_HIGH = { cache => 1, verify => 1, verify_peername => "https" }; 397our $TLS_CTX_HIGH = { cache => 1, verify => 1, verify_peername => "https" };
360 398
… …
379 my @pseudo = (URL => $url); 417 my @pseudo = (URL => $url);
380 push @pseudo, Redirect => delete $arg{Redirect} if exists $arg{Redirect}; 418 push @pseudo, Redirect => delete $arg{Redirect} if exists $arg{Redirect};
381 419
382 my $recurse = exists $arg{recurse} ? delete $arg{recurse} : $MAX_RECURSE; 420 my $recurse = exists $arg{recurse} ? delete $arg{recurse} : $MAX_RECURSE;
383 421
384 return $cb->(undef, { Status => 599, Reason => "Too many redirections", @pseudo }) 422 return $cb->(undef, { @pseudo, Status => 599, Reason => "Too many redirections" })
385 if $recurse < 0; 423 if $recurse < 0;
386 424
387 my $proxy = $arg{proxy} || $PROXY; 425 my $proxy = $arg{proxy} || $PROXY;
388 my $timeout = $arg{timeout} || $TIMEOUT; 426 my $timeout = $arg{timeout} || $TIMEOUT;
389 427
… …
392 430
393 $uscheme = lc $uscheme; 431 $uscheme = lc $uscheme;
394 432
395 my $uport = $uscheme eq "http" ? 80 433 my $uport = $uscheme eq "http" ? 80
396 : $uscheme eq "https" ? 443 434 : $uscheme eq "https" ? 443
397 : return $cb->(undef, { Status => 599, Reason => "Only http and https URL schemes supported", @pseudo }); 435 : return $cb->(undef, { @pseudo, Status => 599, Reason => "Only http and https URL schemes supported" });
398 436
399 $uauthority =~ /^(?: .*\@ )? ([^\@:]+) (?: : (\d+) )?$/x 437 $uauthority =~ /^(?: .*\@ )? ([^\@:]+) (?: : (\d+) )?$/x
400 or return $cb->(undef, { Status => 599, Reason => "Unparsable URL", @pseudo }); 438 or return $cb->(undef, { @pseudo, Status => 599, Reason => "Unparsable URL" });
401 439
402 my $uhost = $1; 440 my $uhost = $1;
403 $uport = $2 if defined $2; 441 $uport = $2 if defined $2;
404 442
405 $hdr{host} = defined $2 ? "$uhost:$2" : "$uhost" 443 $hdr{host} = defined $2 ? "$uhost:$2" : "$uhost"
… …
454 } else { 492 } else {
455 ($rhost, $rport, $rscheme, $rpath) = ($uhost, $uport, $uscheme, $upath); 493 ($rhost, $rport, $rscheme, $rpath) = ($uhost, $uport, $uscheme, $upath);
456 } 494 }
457 495
458 # leave out fragment and query string, just a heuristic 496 # leave out fragment and query string, just a heuristic
459 $hdr{referer} ||= "$uscheme://$uauthority$upath" unless exists $hdr{referer}; 497 $hdr{referer} = "$uscheme://$uauthority$upath" unless exists $hdr{referer};
460 $hdr{"user-agent"} ||= $USERAGENT unless exists $hdr{"user-agent"}; 498 $hdr{"user-agent"} = $USERAGENT unless exists $hdr{"user-agent"};
461 499
462 $hdr{"content-length"} = length $arg{body} 500 $hdr{"content-length"} = length $arg{body}
463 if length $arg{body} || $method ne "GET"; 501 if length $arg{body} || $method ne "GET";
464 502
503 $hdr{connection} = "close TE"; #1.1
504 $hdr{te} = "trailers" unless exists $hdr{te}; #1.1
505
465 my %state = (connect_guard => 1); 506 my %state = (connect_guard => 1);
466 507
467 _get_slot $uhost, sub { 508 _get_slot $uhost, sub {
468 $state{slot_guard} = shift; 509 $state{slot_guard} = shift;
469 510
470 return unless $state{connect_guard}; 511 return unless $state{connect_guard};
471 512
472 my $tcp_connect = $arg{tcp_connect} 513 my $connect_cb = sub {
473 || do { require AnyEvent::Socket; \&AnyEvent::Socket::tcp_connect };
474
475 $state{connect_guard} = $tcp_connect->(
476 $rhost,
477 $rport,
478 sub {
479 $state{fh} = shift 514 $state{fh} = shift
480 or do { 515 or do {
481 my $err = "$!"; 516 my $err = "$!";
482 %state = (); 517 %state = ();
483 return $cb->(undef, { Status => 599, Reason => $err, @pseudo }); 518 return $cb->(undef, { @pseudo, Status => 599, Reason => $err });
484 }; 519 };
485 520
486 pop; # free memory, save a tree 521 pop; # free memory, save a tree
487 522
488 return unless delete $state{connect_guard}; 523 return unless delete $state{connect_guard};
489 524
490 # get handle 525 # get handle
491 $state{handle} = new AnyEvent::Handle 526 $state{handle} = new AnyEvent::Handle
492 fh => $state{fh}, 527 fh => $state{fh},
493 peername => $rhost, 528 peername => $rhost,
494 tls_ctx => $arg{tls_ctx}, 529 tls_ctx => $arg{tls_ctx},
495 # these need to be reconfigured on keepalive handles 530 # these need to be reconfigured on keepalive handles
496 timeout => $timeout, 531 timeout => $timeout,
497 on_error => sub { 532 on_error => sub {
498 %state = (); 533 %state = ();
499 $cb->(undef, { Status => 599, Reason => $_[2], @pseudo }); 534 $cb->(undef, { @pseudo, Status => 599, Reason => $_[2] });
500 }, 535 },
501 on_eof => sub { 536 on_eof => sub {
502 %state = (); 537 %state = ();
503 $cb->(undef, { Status => 599, Reason => "Unexpected end-of-file", @pseudo }); 538 $cb->(undef, { @pseudo, Status => 599, Reason => "Unexpected end-of-file" });
504 }, 539 },
505 ; 540 ;
506 541
507 # limit the number of persistent connections 542 # limit the number of persistent connections
508 # keepalive not yet supported 543 # keepalive not yet supported
509# if ($KA_COUNT{$_[1]} < $MAX_PERSISTENT_PER_HOST) { 544# if ($KA_COUNT{$_[1]} < $MAX_PERSISTENT_PER_HOST) {
510# ++$KA_COUNT{$_[1]}; 545# ++$KA_COUNT{$_[1]};
511# $state{handle}{ka_count_guard} = AnyEvent::Util::guard { 546# $state{handle}{ka_count_guard} = AnyEvent::Util::guard {
512# --$KA_COUNT{$_[1]} 547# --$KA_COUNT{$_[1]}
513# }; 548# };
514# $hdr{connection} = "keep-alive"; 549# $hdr{connection} = "keep-alive";
515# } else {
516 delete $hdr{connection};
517# } 550# }
518 551
519 $state{handle}->starttls ("connect") if $rscheme eq "https"; 552 $state{handle}->starttls ("connect") if $rscheme eq "https";
520 553
521 # handle actual, non-tunneled, request 554 # handle actual, non-tunneled, request
522 my $handle_actual_request = sub { 555 my $handle_actual_request = sub {
523 $state{handle}->starttls ("connect") if $uscheme eq "https" && !exists $state{handle}{tls}; 556 $state{handle}->starttls ("connect") if $uscheme eq "https" && !exists $state{handle}{tls};
524 557
525 # send request 558 # send request
526 $state{handle}->push_write ( 559 $state{handle}->push_write (
527 "$method $rpath HTTP/1.0\015\012" 560 "$method $rpath HTTP/1.1\015\012"
528 . (join "", map "\u$_: $hdr{$_}\015\012", grep defined $hdr{$_}, keys %hdr) 561 . (join "", map "\u$_: $hdr{$_}\015\012", grep defined $hdr{$_}, keys %hdr)
529 . "\015\012" 562 . "\015\012"
530 . (delete $arg{body}) 563 . (delete $arg{body})
531 ); 564 );
532 565
533 # return if error occured during push_write() 566 # return if error occured during push_write()
534 return unless %state; 567 return unless %state;
535 568
536 %hdr = (); # reduce memory usage, save a kitten, also make it possible to re-use 569 %hdr = (); # reduce memory usage, save a kitten, also make it possible to re-use
537 570
538 # status line and headers 571 # status line and headers
539 $state{handle}->push_read (line => $qr_nlnl, sub { 572 $state{read_response} = sub {
540 for ("$_[1]") { 573 for ("$_[1]") {
541 y/\015//d; # weed out any \015, as they show up in the weirdest of places. 574 y/\015//d; # weed out any \015, as they show up in the weirdest of places.
542 575
543 /^HTTP\/([0-9\.]+) \s+ ([0-9]{3}) (?: \s+ ([^\015\012]*) )? \015?\012/igxc 576 /^HTTP\/([0-9\.]+) \s+ ([0-9]{3}) (?: \s+ ([^\012]*) )? \012/igxc
544 or return (%state = (), $cb->(undef, { Status => 599, Reason => "Invalid server response", @pseudo })); 577 or return (%state = (), $cb->(undef, { @pseudo, Status => 599, Reason => "Invalid server response" }));
545 578
579 # 100 Continue handling
580 # should not happen as we don't send expect: 100-continue,
581 # but we handle it just in case.
582 # since we send the request body regardless, if we get an error
583 # we are out of-sync, which we currently do NOT handle correctly.
584 return $state{handle}->push_read (line => $qr_nlnl, $state{read_response})
585 if $2 eq 100;
586
546 push @pseudo, 587 push @pseudo,
547 HTTPVersion => $1, 588 HTTPVersion => $1,
548 Status => $2, 589 Status => $2,
549 Reason => $3, 590 Reason => $3,
550 ; 591 ;
551 592
552 # things seen, not parsed: 593 my $hdr = parse_hdr
553 # p3pP="NON CUR OTPi OUR NOR UNI"
554
555 $hdr{lc $1} .= ",$2"
556 while /\G
557 ([^:\000-\037]*):
558 [\011\040]*
559 ((?: [^\012]+ | \012[\011\040] )*)
560 \012
561 /gxc;
562
563 /\G$/
564 or return (%state = (), $cb->(undef, { Status => 599, Reason => "Garbled response headers", @pseudo })); 594 or return (%state = (), $cb->(undef, { @pseudo, Status => 599, Reason => "Garbled response headers" }));
595
596 %hdr = (%$hdr, @pseudo);
597 }
598
599 # redirect handling
600 # microsoft and other shitheads don't give a shit for following standards,
601 # try to support some common forms of broken Location headers.
602 if ($hdr{location} !~ /^(?: $ | [^:\/?\#]+ : )/x) {
603 $hdr{location} =~ s/^\.\/+//;
604
605 my $url = "$rscheme://$uhost:$uport";
606
607 unless ($hdr{location} =~ s/^\///) {
608 $url .= $upath;
609 $url =~ s/\/[^\/]*$//;
565 } 610 }
566 611
567 # remove the "," prefix we added to all headers above
568 substr $_, 0, 1, ""
569 for values %hdr;
570
571 # patch in all pseudo headers
572 %hdr = (%hdr, @pseudo);
573
574 # redirect handling
575 # microsoft and other shitheads don't give a shit for following standards,
576 # try to support some common forms of broken Location headers.
577 if ($hdr{location} !~ /^(?: $ | [^:\/?\#]+ : )/x) {
578 $hdr{location} =~ s/^\.\/+//;
579
580 my $url = "$rscheme://$uhost:$uport";
581
582 unless ($hdr{location} =~ s/^\///) {
583 $url .= $upath;
584 $url =~ s/\/[^\/]*$//;
585 }
586
587 $hdr{location} = "$url/$hdr{location}"; 612 $hdr{location} = "$url/$hdr{location}";
613 }
614
615 my $redirect;
616
617 if ($recurse) {
618 my $status = $hdr{Status};
619
620 # industry standard is to redirect POST as GET for
621 # 301, 302 and 303, in contrast to http/1.0 and 1.1.
622 # also, the UA should ask the user for 301 and 307 and POST,
623 # industry standard seems to be to simply follow.
624 # we go with the industry standard.
625 if ($status == 301 or $status == 302 or $status == 303) {
626 # HTTP/1.1 is unclear on how to mutate the method
627 $method = "GET" unless $method eq "HEAD";
628 $redirect = 1;
629 } elsif ($status == 307) {
630 $redirect = 1;
588 } 631 }
632 }
589 633
634 my $finish = sub { # ($data, $err_status, $err_reason[, $keepalive])
590 my $redirect; 635 my $keepalive = pop;
591 636
637 $state{handle}->destroy if $state{handle};
638 %state = ();
639
640 if (defined $_[1]) {
641 $hdr{OrigStatus} = $hdr{Status}; $hdr{Status} = $_[1];
642 $hdr{OrigReason} = $hdr{Reason}; $hdr{Reason} = $_[2];
643 }
644
645 # set-cookie processing
592 if ($recurse) { 646 if ($arg{cookie_jar}) {
593 my $status = $hdr{Status}; 647 for ($hdr{"set-cookie"}) {
648 # parse NAME=VALUE
649 my @kv;
594 650
595 # industry standard is to redirect POST as GET for 651 while (/\G\s* ([^=;,[:space:]]+) \s*=\s* (?: "((?:[^\\"]+|\\.)*)" | ([^=;,[:space:]]*) )/gcxs) {
596 # 301, 302 and 303, in contrast to http/1.0 and 1.1.
597 # also, the UA should ask the user for 301 and 307 and POST,
598 # industry standard seems to be to simply follow.
599 # we go with the industry standard.
600 if ($status == 301 or $status == 302 or $status == 303) {
601 # HTTP/1.1 is unclear on how to mutate the method
602 $method = "GET" unless $method eq "HEAD";
603 $redirect = 1; 652 my $name = $1;
604 } elsif ($status == 307) { 653 my $value = $3;
654
655 unless ($value) {
656 $value = $2;
657 $value =~ s/\\(.)/$1/gs;
658 }
659
660 push @kv, $name => $value;
661
662 last unless /\G\s*;/gc;
663 }
664
665 last unless @kv;
666
667 my $name = shift @kv;
668 my %kv = (value => shift @kv, @kv);
669
670 my $cdom;
671 my $cpath = (delete $kv{path}) || "/";
672
673 if (exists $kv{domain}) {
674 $cdom = delete $kv{domain};
675
676 $cdom =~ s/^\.?/./; # make sure it starts with a "."
677
678 next if $cdom =~ /\.$/;
679
680 # this is not rfc-like and not netscape-like. go figure.
681 my $ndots = $cdom =~ y/.//;
682 next if $ndots < ($cdom =~ /\.[^.][^.]\.[^.][^.]$/ ? 3 : 2);
683 } else {
684 $cdom = $uhost;
685 }
686
605 $redirect = 1; 687 # store it
688 $arg{cookie_jar}{version} = 1;
689 $arg{cookie_jar}{$cdom}{$cpath}{$name} = \%kv;
690
691 redo if /\G\s*,/gc;
606 } 692 }
607 } 693 }
608 694
609 my $finish = sub {
610 $state{handle}->destroy if $state{handle};
611 %state = ();
612
613 # set-cookie processing
614 if ($arg{cookie_jar}) {
615 for ($_[1]{"set-cookie"}) {
616 # parse NAME=VALUE
617 my @kv;
618
619 while (/\G\s* ([^=;,[:space:]]+) \s*=\s* (?: "((?:[^\\"]+|\\.)*)" | ([^=;,[:space:]]*) )/gcxs) {
620 my $name = $1;
621 my $value = $3;
622
623 unless ($value) {
624 $value = $2;
625 $value =~ s/\\(.)/$1/gs;
626 }
627
628 push @kv, $name => $value;
629
630 last unless /\G\s*;/gc;
631 }
632
633 last unless @kv;
634
635 my $name = shift @kv;
636 my %kv = (value => shift @kv, @kv);
637
638 my $cdom;
639 my $cpath = (delete $kv{path}) || "/";
640
641 if (exists $kv{domain}) {
642 $cdom = delete $kv{domain};
643
644 $cdom =~ s/^\.?/./; # make sure it starts with a "."
645
646 next if $cdom =~ /\.$/;
647
648 # this is not rfc-like and not netscape-like. go figure.
649 my $ndots = $cdom =~ y/.//;
650 next if $ndots < ($cdom =~ /\.[^.][^.]\.[^.][^.]$/ ? 3 : 2);
651 } else {
652 $cdom = $uhost;
653 }
654
655 # store it
656 $arg{cookie_jar}{version} = 1;
657 $arg{cookie_jar}{$cdom}{$cpath}{$name} = \%kv;
658
659 redo if /\G\s*,/gc;
660 }
661 }
662
663 if ($redirect && exists $hdr{location}) { 695 if ($redirect && exists $hdr{location}) {
664 # we ignore any errors, as it is very common to receive 696 # we ignore any errors, as it is very common to receive
665 # Content-Length != 0 but no actual body 697 # Content-Length != 0 but no actual body
666 # we also access %hdr, as $_[1] might be an erro 698 # we also access %hdr, as $_[1] might be an erro
667 http_request ( 699 http_request (
668 $method => $hdr{location}, 700 $method => $hdr{location},
669 %arg, 701 %arg,
670 recurse => $recurse - 1, 702 recurse => $recurse - 1,
671 Redirect => \@_, 703 Redirect => [$_[0], \%hdr],
672 $cb); 704 $cb);
673 } else {
674 $cb->($_[0], $_[1]);
675 }
676 };
677
678 my $len = $hdr{"content-length"};
679
680 if (!$redirect && $arg{on_header} && !$arg{on_header}(\%hdr)) {
681 $finish->(undef, { Status => 598, Reason => "Request cancelled by on_header", @pseudo });
682 } elsif (
683 $hdr{Status} =~ /^(?:1..|[23]04)$/
684 or $method eq "HEAD"
685 or (defined $len && !$len)
686 ) {
687 # no body
688 $finish->("", \%hdr);
689 } else { 705 } else {
706 $cb->($_[0], \%hdr);
707 }
708 };
709
710 my $len = $hdr{"content-length"};
711
712 if (!$redirect && $arg{on_header} && !$arg{on_header}(\%hdr)) {
713 $finish->(undef, 598 => "Request cancelled by on_header");
714 } elsif (
715 $hdr{Status} =~ /^(?:1..|204|205|304)$/
716 or $method eq "HEAD"
717 or (defined $len && !$len)
718 ) {
719 # no body
720 $finish->("", undef, undef, 1);
721 } else {
690 # body handling, four different code paths 722 # body handling, many different code paths
691 # for want_body_handle, on_body (2x), normal (2x) 723 # - no body expected
692 # we might read too much here, but it does not matter yet (no pers. connections) 724 # - want_body_handle
725 # - te chunked
726 # - 2x length known (with or without on_body)
727 # - 2x length not known (with or without on_body)
693 if (!$redirect && $arg{want_body_handle}) { 728 if (!$redirect && $arg{want_body_handle}) {
694 $_[0]->on_eof (undef); 729 $_[0]->on_eof (undef);
695 $_[0]->on_error (undef); 730 $_[0]->on_error (undef);
696 $_[0]->on_read (undef); 731 $_[0]->on_read (undef);
697 732
698 $finish->(delete $state{handle}, \%hdr); 733 $finish->(delete $state{handle});
699 734
700 } elsif ($arg{on_body}) { 735 } elsif ($hdr{"transfer-encoding"} =~ /\bchunked\b/i) {
736 my $cl = 0;
737 my $body = undef;
738 my $on_body = $arg{on_body} || sub { $body .= shift; 1 };
739
701 $_[0]->on_error (sub { $finish->(undef, { Status => 599, Reason => $_[2], @pseudo }) }); 740 $_[0]->on_error (sub { $finish->(undef, 599 => $_[2]) });
741
742 my $read_chunk; $read_chunk = sub {
743 $_[1] =~ /^([0-9a-fA-F]+)/
744 or $finish->(undef, 599 => "Garbled chunked transfer encoding");
745
746 my $len = hex $1;
747
702 if ($len) { 748 if ($len) {
703 $_[0]->on_eof (undef); 749 $cl += $len;
750
751 $_[0]->push_read (chunk => $len, sub {
752 $on_body->($_[1], \%hdr)
753 or return $finish->(undef, 598 => "Request cancelled by on_body");
754
704 $_[0]->on_read (sub { 755 $_[0]->push_read (line => sub {
705 $len -= length $_[0]{rbuf}; 756 length $_[1]
706 757 and return $finish->(undef, 599 => "Garbled chunked transfer encoding");
707 $arg{on_body}(delete $_[0]{rbuf}, \%hdr) 758 $_[0]->push_read (line => $read_chunk);
708 or $finish->(undef, { Status => 598, Reason => "Request cancelled by on_body", @pseudo });
709
710 $len > 0 759 });
711 or $finish->("", \%hdr);
712 }); 760 });
713 } else { 761 } else {
714 $_[0]->on_eof (sub { 762 $hdr{"content-length"} ||= $cl;
715 $finish->("", \%hdr); 763
764 $_[0]->push_read (line => $qr_nlnl, sub {
765 if (length $_[1]) {
766 for ("$_[1]") {
767 y/\015//d; # weed out any \015, as they show up in the weirdest of places.
768
769 my $hdr = parse_hdr
770 or return $finish->(undef, 599 => "Garbled response trailers");
771
772 %hdr = (%hdr, %$hdr);
773 }
716 }); 774 }
717 $_[0]->on_read (sub { 775
718 $arg{on_body}(delete $_[0]{rbuf}, \%hdr) 776 $finish->($body, undef, undef, 1);
719 or $finish->(undef, { Status => 598, Reason => "Request cancelled by on_body", @pseudo });
720 }); 777 });
721 } 778 }
779 };
780
781 $_[0]->push_read (line => $read_chunk);
782
783 } elsif ($arg{on_body}) {
784 $_[0]->on_error (sub { $finish->(undef, 599 => $_[2]) });
785
786 if ($len) {
787 $_[0]->on_read (sub {
788 $len -= length $_[0]{rbuf};
789
790 $arg{on_body}(delete $_[0]{rbuf}, \%hdr)
791 or return $finish->(undef, 598 => "Request cancelled by on_body");
792
793 $len > 0
794 or $finish->("", undef, undef, 1);
795 });
722 } else { 796 } else {
723 $_[0]->on_eof (undef); 797 $_[0]->on_eof (sub {
724 798 $finish->("");
725 if ($len) { 799 });
726 $_[0]->on_error (sub { $finish->(undef, { Status => 599, Reason => $_[2], @pseudo }) });
727 $_[0]->on_read (sub { 800 $_[0]->on_read (sub {
801 $arg{on_body}(delete $_[0]{rbuf}, \%hdr)
802 or $finish->(undef, 598 => "Request cancelled by on_body");
803 });
804 }
805 } else {
806 $_[0]->on_eof (undef);
807
808 if ($len) {
809 $_[0]->on_error (sub { $finish->(undef, 599 => $_[2]) });
810 $_[0]->on_read (sub {
728 $finish->((substr delete $_[0]{rbuf}, 0, $len, ""), \%hdr) 811 $finish->((substr delete $_[0]{rbuf}, 0, $len, ""), undef, undef, 1)
729 if $len <= length $_[0]{rbuf}; 812 if $len <= length $_[0]{rbuf};
730 }); 813 });
731 } else { 814 } else {
732 $_[0]->on_error (sub { 815 $_[0]->on_error (sub {
733 ($! == Errno::EPIPE || !$!) 816 ($! == Errno::EPIPE || !$!)
734 ? $finish->(delete $_[0]{rbuf}, \%hdr) 817 ? $finish->(delete $_[0]{rbuf})
735 : $finish->(undef, { Status => 599, Reason => $_[2], @pseudo }); 818 : $finish->(undef, 599 => $_[2]);
736 }); 819 });
737 $_[0]->on_read (sub { }); 820 $_[0]->on_read (sub { });
738 }
739 } 821 }
740 } 822 }
741 }); 823 }
742 }; 824 };
743 825
826 $state{handle}->push_read (line => $qr_nlnl, $state{read_response});
827 };
828
744 # now handle proxy-CONNECT method 829 # now handle proxy-CONNECT method
745 if ($proxy && $uscheme eq "https") { 830 if ($proxy && $uscheme eq "https") {
746 # oh dear, we have to wrap it into a connect request 831 # oh dear, we have to wrap it into a connect request
747 832
748 # maybe re-use $uauthority with patched port? 833 # maybe re-use $uauthority with patched port?
749 $state{handle}->push_write ("CONNECT $uhost:$uport HTTP/1.0\015\012Host: $uhost\015\012\015\012"); 834 $state{handle}->push_write ("CONNECT $uhost:$uport HTTP/1.0\015\012Host: $uhost\015\012\015\012");
750 $state{handle}->push_read (line => $qr_nlnl, sub { 835 $state{handle}->push_read (line => $qr_nlnl, sub {
751 $_[1] =~ /^HTTP\/([0-9\.]+) \s+ ([0-9]{3}) (?: \s+ ([^\015\012]*) )?/ix 836 $_[1] =~ /^HTTP\/([0-9\.]+) \s+ ([0-9]{3}) (?: \s+ ([^\015\012]*) )?/ix
752 or return (%state = (), $cb->(undef, { Status => 599, Reason => "Invalid proxy connect response ($_[1])", @pseudo })); 837 or return (%state = (), $cb->(undef, { @pseudo, Status => 599, Reason => "Invalid proxy connect response ($_[1])" }));
753 838
754 if ($2 == 200) { 839 if ($2 == 200) {
755 $rpath = $upath; 840 $rpath = $upath;
756 &$handle_actual_request; 841 &$handle_actual_request;
757 } else { 842 } else {
758 %state = (); 843 %state = ();
759 $cb->(undef, { Status => $2, Reason => $3, @pseudo }); 844 $cb->(undef, { @pseudo, Status => $2, Reason => $3 });
760 }
761 }); 845 }
846 });
762 } else { 847 } else {
763 &$handle_actual_request; 848 &$handle_actual_request;
764 }
765
766 }, 849 }
767 $arg{on_prepare} || sub { $timeout }
768 ); 850 };
851
852 my $tcp_connect = $arg{tcp_connect}
853 || do { require AnyEvent::Socket; \&AnyEvent::Socket::tcp_connect };
854
855 $state{connect_guard} = $tcp_connect->($rhost, $rport, $connect_cb, $arg{on_prepare} || sub { $timeout });
856
769 }; 857 };
770 858
771 defined wantarray && AnyEvent::Util::guard { %state = () } 859 defined wantarray && AnyEvent::Util::guard { %state = () }
772} 860}
773 861

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines