ViewVC Help
View File | Revision Log | Show Annotations | Download File
/cvs/AnyEvent-HTTP/HTTP.pm
(Generate patch)

Comparing AnyEvent-HTTP/HTTP.pm (file contents):
Revision 1.65 by root, Fri Dec 31 03:47:32 2010 UTC vs.
Revision 1.75 by root, Sat Jan 1 00:08:51 2011 UTC

… …
94C<http_request> returns a "cancellation guard" - you have to keep the 94C<http_request> returns a "cancellation guard" - you have to keep the
95object at least alive until the callback get called. If the object gets 95object at least alive until the callback get called. If the object gets
96destroyed before the callback is called, the request will be cancelled. 96destroyed before the callback is called, the request will be cancelled.
97 97
98The callback will be called with the response body data as first argument 98The callback will be called with the response body data as first argument
99(or C<undef> if an error occured), and a hash-ref with response headers as 99(or C<undef> if an error occured), and a hash-ref with response headers
100second argument. 100(and trailers) as second argument.
101 101
102All the headers in that hash are lowercased. In addition to the response 102All the headers in that hash are lowercased. In addition to the response
103headers, the "pseudo-headers" (uppercase to avoid clashing with possible 103headers, the "pseudo-headers" (uppercase to avoid clashing with possible
104response headers) C<HTTPVersion>, C<Status> and C<Reason> contain the 104response headers) C<HTTPVersion>, C<Status> and C<Reason> contain the
105three parts of the HTTP Status-Line of the same name. If an error occurs 105three parts of the HTTP Status-Line of the same name. If an error occurs
… …
150Whether to recurse requests or not, e.g. on redirects, authentication 150Whether to recurse requests or not, e.g. on redirects, authentication
151retries and so on, and how often to do so. 151retries and so on, and how often to do so.
152 152
153=item headers => hashref 153=item headers => hashref
154 154
155The request headers to use. Currently, C<http_request> may provide its 155The request headers to use. Currently, C<http_request> may provide its own
156own C<Host:>, C<Content-Length:>, C<Connection:> and C<Cookie:> headers 156C<Host:>, C<Content-Length:>, C<Connection:> and C<Cookie:> headers and
157and will provide defaults for C<User-Agent:> and C<Referer:> (this can be 157will provide defaults at least for C<TE:>, C<Referer:> and C<User-Agent:>
158suppressed by using C<undef> for these headers in which case they won't be 158(this can be suppressed by using C<undef> for these headers in which case
159sent at all). 159they won't be sent at all).
160 160
161=item timeout => $seconds 161=item timeout => $seconds
162 162
163The time-out to use for various stages - each connect attempt will reset 163The time-out to use for various stages - each connect attempt will reset
164the timeout, as will read or write activity, i.e. this is not an overall 164the timeout, as will read or write activity, i.e. this is not an overall
… …
174C<$scheme> must be either missing, C<http> for HTTP or C<https> for 174C<$scheme> must be either missing, C<http> for HTTP or C<https> for
175HTTPS. 175HTTPS.
176 176
177=item body => $string 177=item body => $string
178 178
179The request body, usually empty. Will be-sent as-is (future versions of 179The request body, usually empty. Will be sent as-is (future versions of
180this module might offer more options). 180this module might offer more options).
181 181
182=item cookie_jar => $hash_ref 182=item cookie_jar => $hash_ref
183 183
184Passing this parameter enables (simplified) cookie-processing, loosely 184Passing this parameter enables (simplified) cookie-processing, loosely
185based on the original netscape specification. 185based on the original netscape specification.
186 186
187The C<$hash_ref> must be an (initially empty) hash reference which will 187The C<$hash_ref> must be an (initially empty) hash reference which will
188get updated automatically. It is possible to save the cookie_jar to 188get updated automatically. It is possible to save the cookie jar to
189persistent storage with something like JSON or Storable, but this is not 189persistent storage with something like JSON or Storable, but this is not
190recommended, as expiry times are currently being ignored. 190recommended, as session-only cookies might survive longer than expected.
191 191
192Note that this cookie implementation is not of very high quality, nor 192Note that this cookie implementation is not meant to be complete. If
193meant to be complete. If you want complete cookie management you have to 193you want complete cookie management you have to do that on your
194do that on your own. C<cookie_jar> is meant as a quick fix to get some 194own. C<cookie_jar> is meant as a quick fix to get some cookie-using sites
195cookie-using sites working. Cookies are a privacy disaster, do not use 195working. Cookies are a privacy disaster, do not use them unless required
196them unless required to. 196to.
197
198When cookie processing is enabled, the C<Cookie:> and C<Set-Cookie:>
199headers will be set and handled by this module, otherwise they will be
200left untouched.
197 201
198=item tls_ctx => $scheme | $tls_ctx 202=item tls_ctx => $scheme | $tls_ctx
199 203
200Specifies the AnyEvent::TLS context to be used for https connections. This 204Specifies the AnyEvent::TLS context to be used for https connections. This
201parameter follows the same rules as the C<tls_ctx> parameter to 205parameter follows the same rules as the C<tls_ctx> parameter to
… …
240 244
241This callback is useful, among other things, to quickly reject unwanted 245This callback is useful, among other things, to quickly reject unwanted
242content, which, if it is supposed to be rare, can be faster than first 246content, which, if it is supposed to be rare, can be faster than first
243doing a C<HEAD> request. 247doing a C<HEAD> request.
244 248
249The downside is that cancelling the request makes it impossible to re-use
250the connection. Also, the C<on_header> callback will not receive any
251trailer (headers sent after the response body).
252
245Example: cancel the request unless the content-type is "text/html". 253Example: cancel the request unless the content-type is "text/html".
246 254
247 on_header => sub { 255 on_header => sub {
248 $_[0]{"content-type"} =~ /^text\/html\s*(?:;|$)/ 256 $_[0]{"content-type"} =~ /^text\/html\s*(?:;|$)/
249 }, 257 },
… …
255string instead of the body data. 263string instead of the body data.
256 264
257It has to return either true (in which case AnyEvent::HTTP will continue), 265It has to return either true (in which case AnyEvent::HTTP will continue),
258or false, in which case AnyEvent::HTTP will cancel the download (and call 266or false, in which case AnyEvent::HTTP will cancel the download (and call
259the completion callback with an error code of C<598>). 267the completion callback with an error code of C<598>).
268
269The downside to cancelling the request is that it makes it impossible to
270re-use the connection.
260 271
261This callback is useful when the data is too large to be held in memory 272This callback is useful when the data is too large to be held in memory
262(so the callback writes it to a file) or when only some information should 273(so the callback writes it to a file) or when only some information should
263be extracted, or when the body should be processed incrementally. 274be extracted, or when the body should be processed incrementally.
264 275
… …
290If you think you need this, first have a look at C<on_body>, to see if 301If you think you need this, first have a look at C<on_body>, to see if
291that doesn't solve your problem in a better way. 302that doesn't solve your problem in a better way.
292 303
293=back 304=back
294 305
295Example: make a simple HTTP GET request for http://www.nethype.de/ 306Example: do a simple HTTP GET request for http://www.nethype.de/ and print
307the response body.
296 308
297 http_request GET => "http://www.nethype.de/", sub { 309 http_request GET => "http://www.nethype.de/", sub {
298 my ($body, $hdr) = @_; 310 my ($body, $hdr) = @_;
299 print "$body\n"; 311 print "$body\n";
300 }; 312 };
301 313
302Example: make a HTTP HEAD request on https://www.google.com/, use a 314Example: do a HTTP HEAD request on https://www.google.com/, use a
303timeout of 30 seconds. 315timeout of 30 seconds.
304 316
305 http_request 317 http_request
306 GET => "https://www.google.com", 318 GET => "https://www.google.com",
307 timeout => 30, 319 timeout => 30,
… …
310 use Data::Dumper; 322 use Data::Dumper;
311 print Dumper $hdr; 323 print Dumper $hdr;
312 } 324 }
313 ; 325 ;
314 326
315Example: make another simple HTTP GET request, but immediately try to 327Example: do another simple HTTP GET request, but immediately try to
316cancel it. 328cancel it.
317 329
318 my $request = http_request GET => "http://www.nethype.de/", sub { 330 my $request = http_request GET => "http://www.nethype.de/", sub {
319 my ($body, $hdr) = @_; 331 my ($body, $hdr) = @_;
320 print "$body\n"; 332 print "$body\n";
… …
352 push @{ $CO_SLOT{$_[0]}[1] }, $_[1]; 364 push @{ $CO_SLOT{$_[0]}[1] }, $_[1];
353 365
354 _slot_schedule $_[0]; 366 _slot_schedule $_[0];
355} 367}
356 368
369# extract cookies from jar
370sub cookie_jar_extract($$$$) {
371 my ($jar, $uscheme, $uhost, $upath) = @_;
372
373 %$jar = () if $jar->{version} != 1;
374
375 my @cookies;
376
377 while (my ($chost, $paths) = each %$jar) {
378 next unless ref $paths;
379
380 if ($chost =~ /^\./) {
381 next unless $chost eq substr $uhost, -length $chost;
382 } elsif ($chost =~ /\./) {
383 next unless $chost eq $uhost;
384 } else {
385 next;
386 }
387
388 while (my ($cpath, $cookies) = each %$paths) {
389 next unless $cpath eq substr $upath, 0, length $cpath;
390
391 while (my ($cookie, $kv) = each %$cookies) {
392 next if $uscheme ne "https" && exists $kv->{secure};
393
394 if (exists $kv->{expires}) {
395 if (AE::now > parse_date ($kv->{expires})) {
396 delete $cookies->{$cookie};
397 next;
398 }
399 }
400
401 my $value = $kv->{value};
402
403 if ($value =~ /[=;,[:space:]]/) {
404 $value =~ s/([\\"])/\\$1/g;
405 $value = "\"$value\"";
406 }
407
408 push @cookies, "$cookie=$value";
409 }
410 }
411 }
412
413 \@cookies
414}
415
416# parse set_cookie header into jar
417sub cookie_jar_set_cookie($$$) {
418 my ($jar, $set_cookie, $uhost) = @_;
419
420 for ($set_cookie) {
421 # parse NAME=VALUE
422 my @kv;
423
424 while (
425 m{
426 \G\s*
427 (?:
428 expires \s*=\s* ([A-Z][a-z][a-z],\ [^,;]+)
429 | ([^=;,[:space:]]+) \s*=\s* (?: "((?:[^\\"]+|\\.)*)" | ([^=;,[:space:]]*) )
430 )
431 }gcxsi
432 ) {
433 my $name = $2;
434 my $value = $4;
435
436 unless (defined $name) {
437 # expires
438 $name = "expires";
439 $value = $1;
440 } elsif (!defined $value) {
441 # quoted
442 $value = $3;
443 $value =~ s/\\(.)/$1/gs;
444 }
445
446 push @kv, lc $name, $value;
447
448 last unless /\G\s*;/gc;
449 }
450
451 last unless @kv;
452
453 my $name = shift @kv;
454 my %kv = (value => shift @kv, @kv);
455
456 $kv{expires} ||= format_date (AE::now + $kv{"max-age"})
457 if exists $kv{"max-age"};
458
459 my $cdom;
460 my $cpath = (delete $kv{path}) || "/";
461
462 if (exists $kv{domain}) {
463 $cdom = delete $kv{domain};
464
465 $cdom =~ s/^\.?/./; # make sure it starts with a "."
466
467 next if $cdom =~ /\.$/;
468
469 # this is not rfc-like and not netscape-like. go figure.
470 my $ndots = $cdom =~ y/.//;
471 next if $ndots < ($cdom =~ /\.[^.][^.]\.[^.][^.]$/ ? 3 : 2);
472 } else {
473 $cdom = $uhost;
474 }
475
476 # store it
477 $jar->{version} = 1;
478 $jar->{$cdom}{$cpath}{$name} = \%kv;
479
480 redo if /\G\s*,/gc;
481 }
482}
483
484# continue to parse $_ for headers and place them into the arg
485sub parse_hdr() {
486 my %hdr;
487
488 # things seen, not parsed:
489 # p3pP="NON CUR OTPi OUR NOR UNI"
490
491 $hdr{lc $1} .= ",$2"
492 while /\G
493 ([^:\000-\037]*):
494 [\011\040]*
495 ((?: [^\012]+ | \012[\011\040] )*)
496 \012
497 /gxc;
498
499 /\G$/
500 or return;
501
502 # remove the "," prefix we added to all headers above
503 substr $_, 0, 1, ""
504 for values %hdr;
505
506 \%hdr
507}
508
357our $qr_nlnl = qr{(?<![^\012])\015?\012}; 509our $qr_nlnl = qr{(?<![^\012])\015?\012};
358 510
359our $TLS_CTX_LOW = { cache => 1, sslv2 => 1 }; 511our $TLS_CTX_LOW = { cache => 1, sslv2 => 1 };
360our $TLS_CTX_HIGH = { cache => 1, verify => 1, verify_peername => "https" }; 512our $TLS_CTX_HIGH = { cache => 1, verify => 1, verify_peername => "https" };
361 513
… …
411 563
412 $upath =~ s%^/?%/%; 564 $upath =~ s%^/?%/%;
413 565
414 # cookie processing 566 # cookie processing
415 if (my $jar = $arg{cookie_jar}) { 567 if (my $jar = $arg{cookie_jar}) {
416 %$jar = () if $jar->{version} != 1; 568 my $cookies = cookie_jar_extract $jar, $uscheme, $uhost, $upath;
417 569
418 my @cookie;
419
420 while (my ($chost, $v) = each %$jar) {
421 if ($chost =~ /^\./) {
422 next unless $chost eq substr $uhost, -length $chost;
423 } elsif ($chost =~ /\./) {
424 next unless $chost eq $uhost;
425 } else {
426 next;
427 }
428
429 while (my ($cpath, $v) = each %$v) {
430 next unless $cpath eq substr $upath, 0, length $cpath;
431
432 while (my ($k, $v) = each %$v) {
433 next if $uscheme ne "https" && exists $v->{secure};
434 my $value = $v->{value};
435 $value =~ s/([\\"])/\\$1/g;
436 push @cookie, "$k=\"$value\"";
437 }
438 }
439 }
440
441 $hdr{cookie} = join "; ", @cookie 570 $hdr{cookie} = join "; ", @$cookies
442 if @cookie; 571 if @$cookies;
443 } 572 }
444 573
445 my ($rhost, $rport, $rscheme, $rpath); # request host, port, path 574 my ($rhost, $rport, $rscheme, $rpath); # request host, port, path
446 575
447 if ($proxy) { 576 if ($proxy) {
… …
455 } else { 584 } else {
456 ($rhost, $rport, $rscheme, $rpath) = ($uhost, $uport, $uscheme, $upath); 585 ($rhost, $rport, $rscheme, $rpath) = ($uhost, $uport, $uscheme, $upath);
457 } 586 }
458 587
459 # leave out fragment and query string, just a heuristic 588 # leave out fragment and query string, just a heuristic
460 $hdr{referer} ||= "$uscheme://$uauthority$upath" unless exists $hdr{referer}; 589 $hdr{referer} = "$uscheme://$uauthority$upath" unless exists $hdr{referer};
461 $hdr{"user-agent"} ||= $USERAGENT unless exists $hdr{"user-agent"}; 590 $hdr{"user-agent"} = $USERAGENT unless exists $hdr{"user-agent"};
462 591
463 $hdr{"content-length"} = length $arg{body} 592 $hdr{"content-length"} = length $arg{body}
464 if length $arg{body} || $method ne "GET"; 593 if length $arg{body} || $method ne "GET";
594
595 $hdr{connection} = "close TE"; #1.1
596 $hdr{te} = "trailers" unless exists $hdr{te}; #1.1
465 597
466 my %state = (connect_guard => 1); 598 my %state = (connect_guard => 1);
467 599
468 _get_slot $uhost, sub { 600 _get_slot $uhost, sub {
469 $state{slot_guard} = shift; 601 $state{slot_guard} = shift;
… …
475 or do { 607 or do {
476 my $err = "$!"; 608 my $err = "$!";
477 %state = (); 609 %state = ();
478 return $cb->(undef, { @pseudo, Status => 599, Reason => $err }); 610 return $cb->(undef, { @pseudo, Status => 599, Reason => $err });
479 }; 611 };
480
481 pop; # free memory, save a tree
482 612
483 return unless delete $state{connect_guard}; 613 return unless delete $state{connect_guard};
484 614
485 # get handle 615 # get handle
486 $state{handle} = new AnyEvent::Handle 616 $state{handle} = new AnyEvent::Handle
… …
505# ++$KA_COUNT{$_[1]}; 635# ++$KA_COUNT{$_[1]};
506# $state{handle}{ka_count_guard} = AnyEvent::Util::guard { 636# $state{handle}{ka_count_guard} = AnyEvent::Util::guard {
507# --$KA_COUNT{$_[1]} 637# --$KA_COUNT{$_[1]}
508# }; 638# };
509# $hdr{connection} = "keep-alive"; 639# $hdr{connection} = "keep-alive";
510# } else {
511 delete $hdr{connection};
512# } 640# }
513 641
514 $state{handle}->starttls ("connect") if $rscheme eq "https"; 642 $state{handle}->starttls ("connect") if $rscheme eq "https";
515 643
516 # handle actual, non-tunneled, request 644 # handle actual, non-tunneled, request
517 my $handle_actual_request = sub { 645 my $handle_actual_request = sub {
518 $state{handle}->starttls ("connect") if $uscheme eq "https" && !exists $state{handle}{tls}; 646 $state{handle}->starttls ("connect") if $uscheme eq "https" && !exists $state{handle}{tls};
519 647
520 # send request 648 # send request
521 $state{handle}->push_write ( 649 $state{handle}->push_write (
522 "$method $rpath HTTP/1.0\015\012" 650 "$method $rpath HTTP/1.1\015\012"
523 . (join "", map "\u$_: $hdr{$_}\015\012", grep defined $hdr{$_}, keys %hdr) 651 . (join "", map "\u$_: $hdr{$_}\015\012", grep defined $hdr{$_}, keys %hdr)
524 . "\015\012" 652 . "\015\012"
525 . (delete $arg{body}) 653 . (delete $arg{body})
526 ); 654 );
527 655
… …
529 return unless %state; 657 return unless %state;
530 658
531 %hdr = (); # reduce memory usage, save a kitten, also make it possible to re-use 659 %hdr = (); # reduce memory usage, save a kitten, also make it possible to re-use
532 660
533 # status line and headers 661 # status line and headers
534 $state{handle}->push_read (line => $qr_nlnl, sub { 662 $state{read_response} = sub {
535 my $keepalive = pop;
536
537 for ("$_[1]") { 663 for ("$_[1]") {
538 y/\015//d; # weed out any \015, as they show up in the weirdest of places. 664 y/\015//d; # weed out any \015, as they show up in the weirdest of places.
539 665
540 /^HTTP\/([0-9\.]+) \s+ ([0-9]{3}) (?: \s+ ([^\015\012]*) )? \015?\012/igxc 666 /^HTTP\/([0-9\.]+) \s+ ([0-9]{3}) (?: \s+ ([^\012]*) )? \012/igxc
541 or return (%state = (), $cb->(undef, { @pseudo, Status => 599, Reason => "Invalid server response" })); 667 or return (%state = (), $cb->(undef, { @pseudo, Status => 599, Reason => "Invalid server response" }));
668
669 # 100 Continue handling
670 # should not happen as we don't send expect: 100-continue,
671 # but we handle it just in case.
672 # since we send the request body regardless, if we get an error
673 # we are out of-sync, which we currently do NOT handle correctly.
674 return $state{handle}->push_read (line => $qr_nlnl, $state{read_response})
675 if $2 eq 100;
542 676
543 push @pseudo, 677 push @pseudo,
544 HTTPVersion => $1, 678 HTTPVersion => $1,
545 Status => $2, 679 Status => $2,
546 Reason => $3, 680 Reason => $3,
547 ; 681 ;
548 682
549 # things seen, not parsed: 683 my $hdr = parse_hdr
550 # p3pP="NON CUR OTPi OUR NOR UNI"
551
552 $hdr{lc $1} .= ",$2"
553 while /\G
554 ([^:\000-\037]*):
555 [\011\040]*
556 ((?: [^\012]+ | \012[\011\040] )*)
557 \012
558 /gxc;
559
560 /\G$/
561 or return (%state = (), $cb->(undef, { @pseudo, Status => 599, Reason => "Garbled response headers" })); 684 or return (%state = (), $cb->(undef, { @pseudo, Status => 599, Reason => "Garbled response headers" }));
685
686 %hdr = (%$hdr, @pseudo);
562 } 687 }
563
564 # remove the "," prefix we added to all headers above
565 substr $_, 0, 1, ""
566 for values %hdr;
567
568 # patch in all pseudo headers
569 %hdr = (%hdr, @pseudo);
570 688
571 # redirect handling 689 # redirect handling
572 # microsoft and other shitheads don't give a shit for following standards, 690 # microsoft and other shitheads don't give a shit for following standards,
573 # try to support some common forms of broken Location headers. 691 # try to support some common forms of broken Location headers.
574 if ($hdr{location} !~ /^(?: $ | [^:\/?\#]+ : )/x) { 692 if ($hdr{location} !~ /^(?: $ | [^:\/?\#]+ : )/x) {
… …
602 $redirect = 1; 720 $redirect = 1;
603 } 721 }
604 } 722 }
605 723
606 my $finish = sub { # ($data, $err_status, $err_reason[, $keepalive]) 724 my $finish = sub { # ($data, $err_status, $err_reason[, $keepalive])
725 my $may_keep_alive = $_[3];
726
607 $state{handle}->destroy if $state{handle}; 727 $state{handle}->destroy if $state{handle};
608 %state = (); 728 %state = ();
609 729
610 if (defined $_[1]) { 730 if (defined $_[1]) {
611 $hdr{OrigStatus} = $hdr{Status}; $hdr{Status} = $_[1]; 731 $hdr{OrigStatus} = $hdr{Status}; $hdr{Status} = $_[1];
612 $hdr{OrigReason} = $hdr{Reason}; $hdr{Reason} = $_[2]; 732 $hdr{OrigReason} = $hdr{Reason}; $hdr{Reason} = $_[2];
613 } 733 }
614 734
615 # set-cookie processing 735 # set-cookie processing
616 if ($arg{cookie_jar}) { 736 if ($arg{cookie_jar}) {
617 for ($hdr{"set-cookie"}) { 737 cookie_jar_set_cookie $arg{cookie_jar}, $hdr{"set-cookie"}, $uhost;
618 # parse NAME=VALUE
619 my @kv;
620
621 while (/\G\s* ([^=;,[:space:]]+) \s*=\s* (?: "((?:[^\\"]+|\\.)*)" | ([^=;,[:space:]]*) )/gcxs) {
622 my $name = $1;
623 my $value = $3;
624
625 unless ($value) {
626 $value = $2;
627 $value =~ s/\\(.)/$1/gs;
628 }
629
630 push @kv, $name => $value;
631
632 last unless /\G\s*;/gc;
633 }
634
635 last unless @kv;
636
637 my $name = shift @kv;
638 my %kv = (value => shift @kv, @kv);
639
640 my $cdom;
641 my $cpath = (delete $kv{path}) || "/";
642
643 if (exists $kv{domain}) {
644 $cdom = delete $kv{domain};
645
646 $cdom =~ s/^\.?/./; # make sure it starts with a "."
647
648 next if $cdom =~ /\.$/;
649
650 # this is not rfc-like and not netscape-like. go figure.
651 my $ndots = $cdom =~ y/.//;
652 next if $ndots < ($cdom =~ /\.[^.][^.]\.[^.][^.]$/ ? 3 : 2);
653 } else {
654 $cdom = $uhost;
655 }
656
657 # store it
658 $arg{cookie_jar}{version} = 1;
659 $arg{cookie_jar}{$cdom}{$cpath}{$name} = \%kv;
660
661 redo if /\G\s*,/gc;
662 }
663 } 738 }
664 739
665 if ($redirect && exists $hdr{location}) { 740 if ($redirect && exists $hdr{location}) {
666 # we ignore any errors, as it is very common to receive 741 # we ignore any errors, as it is very common to receive
667 # Content-Length != 0 but no actual body 742 # Content-Length != 0 but no actual body
… …
687 or (defined $len && !$len) 762 or (defined $len && !$len)
688 ) { 763 ) {
689 # no body 764 # no body
690 $finish->("", undef, undef, 1); 765 $finish->("", undef, undef, 1);
691 } else { 766 } else {
692 # body handling, four different code paths 767 # body handling, many different code paths
693 # for want_body_handle, on_body (2x), normal (2x) 768 # - no body expected
694 # we might read too much here, but it does not matter yet (no pipelining) 769 # - want_body_handle
770 # - te chunked
771 # - 2x length known (with or without on_body)
772 # - 2x length not known (with or without on_body)
695 if (!$redirect && $arg{want_body_handle}) { 773 if (!$redirect && $arg{want_body_handle}) {
696 $_[0]->on_eof (undef); 774 $_[0]->on_eof (undef);
697 $_[0]->on_error (undef); 775 $_[0]->on_error (undef);
698 $_[0]->on_read (undef); 776 $_[0]->on_read (undef);
699 777
700 $finish->(delete $state{handle}); 778 $finish->(delete $state{handle});
701 779
780 } elsif ($hdr{"transfer-encoding"} =~ /\bchunked\b/i) {
781 my $cl = 0;
782 my $body = undef;
783 my $on_body = $arg{on_body} || sub { $body .= shift; 1 };
784
785 $_[0]->on_error (sub { $finish->(undef, 599 => $_[2]) });
786
787 my $read_chunk; $read_chunk = sub {
788 $_[1] =~ /^([0-9a-fA-F]+)/
789 or $finish->(undef, 599 => "Garbled chunked transfer encoding");
790
791 my $len = hex $1;
792
793 if ($len) {
794 $cl += $len;
795
796 $_[0]->push_read (chunk => $len, sub {
797 $on_body->($_[1], \%hdr)
798 or return $finish->(undef, 598 => "Request cancelled by on_body");
799
800 $_[0]->push_read (line => sub {
801 length $_[1]
802 and return $finish->(undef, 599 => "Garbled chunked transfer encoding");
803 $_[0]->push_read (line => $read_chunk);
804 });
805 });
806 } else {
807 $hdr{"content-length"} ||= $cl;
808
809 $_[0]->push_read (line => $qr_nlnl, sub {
810 if (length $_[1]) {
811 for ("$_[1]") {
812 y/\015//d; # weed out any \015, as they show up in the weirdest of places.
813
814 my $hdr = parse_hdr
815 or return $finish->(undef, 599 => "Garbled response trailers");
816
817 %hdr = (%hdr, %$hdr);
818 }
819 }
820
821 $finish->($body, undef, undef, 1);
822 });
823 }
824 };
825
826 $_[0]->push_read (line => $read_chunk);
827
702 } elsif ($arg{on_body}) { 828 } elsif ($arg{on_body}) {
703 $_[0]->on_error (sub { $finish->(undef, 599 => $_[2]) }); 829 $_[0]->on_error (sub { $finish->(undef, 599 => $_[2]) });
830
704 if ($len) { 831 if ($len) {
705 $_[0]->on_read (sub { 832 $_[0]->on_read (sub {
706 $len -= length $_[0]{rbuf}; 833 $len -= length $_[0]{rbuf};
707 834
708 $arg{on_body}(delete $_[0]{rbuf}, \%hdr) 835 $arg{on_body}(delete $_[0]{rbuf}, \%hdr)
709 or $finish->(undef, 598 => "Request cancelled by on_body"); 836 or return $finish->(undef, 598 => "Request cancelled by on_body");
710 837
711 $len > 0 838 $len > 0
712 or $finish->("", undef, undef, 1); 839 or $finish->("", undef, undef, 1);
713 }); 840 });
714 } else { 841 } else {
… …
737 }); 864 });
738 $_[0]->on_read (sub { }); 865 $_[0]->on_read (sub { });
739 } 866 }
740 } 867 }
741 } 868 }
742 }); 869 };
870
871 $state{handle}->push_read (line => $qr_nlnl, $state{read_response});
743 }; 872 };
744 873
745 # now handle proxy-CONNECT method 874 # now handle proxy-CONNECT method
746 if ($proxy && $uscheme eq "https") { 875 if ($proxy && $uscheme eq "https") {
747 # oh dear, we have to wrap it into a connect request 876 # oh dear, we have to wrap it into a connect request
… …
819Takes a POSIX timestamp (seconds since the epoch) and formats it as a HTTP 948Takes a POSIX timestamp (seconds since the epoch) and formats it as a HTTP
820Date (RFC 2616). 949Date (RFC 2616).
821 950
822=item $timestamp = AnyEvent::HTTP::parse_date $date 951=item $timestamp = AnyEvent::HTTP::parse_date $date
823 952
824Takes a HTTP Date (RFC 2616) and returns the corresponding POSIX 953Takes a HTTP Date (RFC 2616) or a Cookie date (netscape cookie spec) and
825timestamp, or C<undef> if the date cannot be parsed. 954returns the corresponding POSIX timestamp, or C<undef> if the date cannot
955be parsed.
826 956
827=item $AnyEvent::HTTP::MAX_RECURSE 957=item $AnyEvent::HTTP::MAX_RECURSE
828 958
829The default value for the C<recurse> request parameter (default: C<10>). 959The default value for the C<recurse> request parameter (default: C<10>).
830 960
… …
869sub parse_date($) { 999sub parse_date($) {
870 my ($date) = @_; 1000 my ($date) = @_;
871 1001
872 my ($d, $m, $y, $H, $M, $S); 1002 my ($d, $m, $y, $H, $M, $S);
873 1003
874 if ($date =~ /^[A-Z][a-z][a-z], ([0-9][0-9]) ([A-Z][a-z][a-z]) ([0-9][0-9][0-9][0-9]) ([0-9][0-9]):([0-9][0-9]):([0-9][0-9]) GMT$/) { 1004 if ($date =~ /^[A-Z][a-z][a-z], ([0-9][0-9])[\- ]([A-Z][a-z][a-z])[\- ]([0-9][0-9][0-9][0-9]) ([0-9][0-9]):([0-9][0-9]):([0-9][0-9]) GMT$/) {
875 # RFC 822/1123, required by RFC 2616 1005 # RFC 822/1123, required by RFC 2616 (with " ")
1006 # cookie dates (with "-")
1007
876 ($d, $m, $y, $H, $M, $S) = ($1, $2, $3, $4, $5, $6); 1008 ($d, $m, $y, $H, $M, $S) = ($1, $2, $3, $4, $5, $6);
877 1009
878 } elsif ($date =~ /^[A-Z][a-z]+, ([0-9][0-9])-([A-Z][a-z][a-z])-([0-9][0-9]) ([0-9][0-9]):([0-9][0-9]):([0-9][0-9]) GMT$/) { 1010 } elsif ($date =~ /^[A-Z][a-z]+, ([0-9][0-9])-([A-Z][a-z][a-z])-([0-9][0-9]) ([0-9][0-9]):([0-9][0-9]):([0-9][0-9]) GMT$/) {
879 # RFC 850 1011 # RFC 850
880 ($d, $m, $y, $H, $M, $S) = ($1, $2, $3 < 69 ? $3 + 2000 : $3 + 1900, $4, $5, $6); 1012 ($d, $m, $y, $H, $M, $S) = ($1, $2, $3 < 69 ? $3 + 2000 : $3 + 1900, $4, $5, $6);

Diff Legend

– Removed lines
+ Added lines
< Changed lines
> Changed lines