[ViewVC] Diff of: cvs/AnyEvent-HTTP/HTTP.pm

Comparing AnyEvent-HTTP/HTTP.pm (file contents):
Revision 1.120 by root, Sun Jun 8 23:36:36 2014 UTC vs.
Revision 1.136 by root, Wed Oct 16 01:20:02 2019 UTC

 use AnyEvent::Util ();
 use AnyEvent::Handle ();
 use base Exporter::;
-our $VERSION = 2.21;
+our $VERSION = 2.24;
 our @EXPORT = qw(http_get http_post http_head http_request);
 our $USERAGENT          = "Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)";
 our $MAX_RECURSE        =  10;
 C<$scheme> must be either missing or must be C<http> for HTTP.
 If not specified, then the default proxy is used (see
 C<AnyEvent::HTTP::set_proxy>).
+Currently, if your proxy requires authorization, you have to specify an
+appropriate "Proxy-Authorization" header in every request.
+Note that this module will prefer an existing persistent connection,
+even if that connection was made using another proxy. If you need to
+ensure that a new connection is made in this case, you can either force
+C<persistent> to false or e.g. use the proxy address in your C<sessionid>.
 =item body => $string
 The request body, usually empty. Will be sent as-is (future versions of
 this module might offer more options).
 The default for this option is C<low>, which could be interpreted as "give
 me the page, no matter what".
 See also the C<sessionid> parameter.
-=item session => $string
+=item sessionid => $string
-The module might reuse connections to the same host internally. Sometimes
+The module might reuse connections to the same host internally (regardless
-(e.g. when using TLS), you do not want to reuse connections from other
+of other settings, such as C<tcp_connect> or C<proxy>). Sometimes (e.g.
+when using TLS or a specfic proxy), you do not want to reuse connections
-sessions. This can be achieved by setting this parameter to some unique
+from other sessions. This can be achieved by setting this parameter to
-ID (such as the address of an object storing your state data, or the TLS
+some unique ID (such as the address of an object storing your state data
-context) - only connections using the same unique ID will be reused.
+or the TLS context, or the proxy IP) - only connections using the same
+unique ID will be reused.
 =item on_prepare => $callback->($fh)
 In rare cases you need to "tune" the socket before it is used to
 connect (for example, to bind it on a given IP address). This parameter
 In even rarer cases you want total control over how AnyEvent::HTTP
 establishes connections. Normally it uses L<AnyEvent::Socket::tcp_connect>
 to do this, but you can provide your own C<tcp_connect> function -
 obviously, it has to follow the same calling conventions, except that it
 may always return a connection guard object.
+The connections made by this hook will be treated as equivalent to
+connections made the built-in way, specifically, they will be put into
+and taken from the persistent connection cache. If your C<$tcp_connect>
+function is incompatible with this kind of re-use, consider switching off
+C<persistent> connections and/or providing a C<sessionid> identifier.
 There are probably lots of weird uses for this function, starting from
 tracing the hosts C<http_request> actually tries to connect, to (inexact
 but fast) host => IP address caching or even socks protocol support.
 =item persistent => $boolean
 Try to create/reuse a persistent connection. When this flag is set
 (default: true for idempotent requests, false for all others), then
 C<http_request> tries to re-use an existing (previously-created)
-persistent connection to the host and, failing that, tries to create a new
+persistent connection to same host (i.e. identical URL scheme, hostname,
-one.
+port and sessionid) and, failing that, tries to create a new one.
 Requests failing in certain ways will be automatically retried once, which
 is dangerous for non-idempotent requests, which is why it defaults to off
 for them. The reason for this is because the bozos who designed HTTP/1.1
 made it impossible to distinguish between a fatal error and a normal
 connection timeout, so you never know whether there was a problem with
 your request or not.
 When reusing an existent connection, many parameters (such as TLS context)
-will be ignored. See the C<session> parameter for a workaround.
+will be ignored. See the C<sessionid> parameter for a workaround.
 =item keepalive => $boolean
 Only used when C<persistent> is also true. This parameter decides whether
 C<http_request> tries to handshake a HTTP/1.0-style keep-alive connection
 # expire cookies
 sub cookie_jar_expire($;$) {
    my ($jar, $session_end) = @_;
-   %$jar = () if $jar->{version} != 1;
+   %$jar = () if $jar->{version} != 2;
    my $anow = AE::now;
    while (my ($chost, $paths) = each %$jar) {
       next unless ref $paths;
 # extract cookies from jar
 sub cookie_jar_extract($$$$) {
    my ($jar, $scheme, $host, $path) = @_;
-   %$jar = () if $jar->{version} != 1;
+   %$jar = () if $jar->{version} != 2;
+   $host = AnyEvent::Util::idn_to_ascii $host
+      if $host =~ /[^\x00-\x7f]/;
    my @cookies;
    while (my ($chost, $paths) = each %$jar) {
       next unless ref $paths;
-      if ($chost =~ /^\./) {
+      # exact match or suffix including . match
-         next unless $chost eq substr $host, -length $chost;
+      $chost eq $host or ".$chost" eq substr $host, -1 - length $chost
-      } elsif ($chost =~ /\./) {
-         next unless $chost eq $host;
-      } else {
-         next;
+         or next;
-      }
       while (my ($cpath, $cookies) = each %$paths) {
          next unless $cpath eq substr $path, 0, length $cpath;
          while (my ($cookie, $kv) = each %$cookies) {
 }
 # parse set_cookie header into jar
 sub cookie_jar_set_cookie($$$$) {
    my ($jar, $set_cookie, $host, $date) = @_;
+   %$jar = () if $jar->{version} != 2;
    my $anow = int AE::now;
    my $snow; # server-now
    for ($set_cookie) {
       my $cdom;
       my $cpath = (delete $kv{path}) || "/";
       if (exists $kv{domain}) {
-         $cdom = delete $kv{domain};
+         $cdom = $kv{domain};
          $cdom =~ s/^\.?/./; # make sure it starts with a "."
          next if $cdom =~ /\.$/;
          # this is not rfc-like and not netscape-like. go figure.
          my $ndots = $cdom =~ y/.//;
          next if $ndots < ($cdom =~ /\.[^.][^.]\.[^.][^.]$/ ? 3 : 2);
+         $cdom = substr $cdom, 1; # remove initial .
       } else {
          $cdom = $host;
       }
       # store it
-      $jar->{version} = 1;
+      $jar->{version} = 2;
       $jar->{lc $cdom}{$cpath}{$name} = \%kv;
       redo if /\G\s*,/gc;
    }
 }
    my $uport = $uscheme eq "http"  ?  80
              : $uscheme eq "https" ? 443
              : return $cb->(undef, { @pseudo, Status => 599, Reason => "Only http and https URL schemes supported" });
-   $uauthority =~ /^(?: .*\@ )? ([^\@:]+) (?: : (\d+) )?$/x
+   $uauthority =~ /^(?: .*\@ )? ([^\@]+?) (?: : (\d+) )?$/x
       or return $cb->(undef, { @pseudo, Status => 599, Reason => "Unparsable URL" });
    my $uhost = lc $1;
    $uport = $2 if defined $2;
          # we give our best and fall back to URI if available.
          if (exists $hdr{location}) {
             my $loc = $hdr{location};
             if ($loc =~ m%^//%) { # //
-               $loc = "$rscheme:$loc";
+               $loc = "$uscheme:$loc";
             } elsif ($loc eq "") {
                $loc = $url;
             } elsif ($loc !~ /^(?: $ | [^:\/?\#]+ : )/x) { # anything "simple"
                $loc =~ s/^\.\/+//;
                if ($loc !~ m%^[.?#]%) {
-                  my $prefix = "$rscheme://$uhost:$uport";
+                  my $prefix = "$uscheme://$uauthority";
                   unless ($loc =~ s/^\///) {
                      $prefix .= $upath;
                      $prefix =~ s/\/[^\/]*$//;
                   }
             # industry standard is to redirect POST as GET for
             # 301, 302 and 303, in contrast to HTTP/1.0 and 1.1.
             # also, the UA should ask the user for 301 and 307 and POST,
             # industry standard seems to be to simply follow.
             # we go with the industry standard. 308 is defined
-            # by rfc7238
+            # by rfc7538
             if ($status == 301 or $status == 302 or $status == 303) {
+               $redirect = 1;
                # HTTP/1.1 is unclear on how to mutate the method
-               $method = "GET" unless $method eq "HEAD";
+               unless ($method eq "HEAD") {
-               $redirect = 1;
+                  $method = "GET";
+                  delete $arg{body};
+               }
             } elsif ($status == 307 or $status == 308) {
                $redirect = 1;
             }
          }
             $finish->(delete $state{handle});
          } elsif ($chunked) {
             my $cl = 0;
             my $body = "";
-            my $on_body = $arg{on_body} || sub { $body .= shift; 1 };
+            my $on_body = (!$redirect && $arg{on_body}) || sub { $body .= shift; 1 };
             $state{read_chunk} = sub {
                $_[1] =~ /^([0-9a-fA-F]+)/
                   or return $finish->(undef, $ae_error => "Garbled chunked transfer encoding");
                }
             };
             $_[0]->push_read (line => $state{read_chunk});
-         } elsif ($arg{on_body}) {
+         } elsif (!$redirect && $arg{on_body}) {
             if (defined $len) {
                $_[0]->on_read (sub {
                   $len -= length $_[0]{rbuf};
                   $arg{on_body}(delete $_[0]{rbuf}, \%hdr)
             _destroy_state %state;
             %state = ();
             $state{recurse} =
                http_request (
-                  $method   => $url,
+                  $method    => $url,
                   %arg,
-                  recurse   => $recurse - 1,
+                  recurse    => $recurse - 1,
-                  keepalive => 0,
+                  persistent => 0,
                   sub {
                      %state = ();
                      &$cb
                   }
                );
       # now handle proxy-CONNECT method
       if ($proxy && $uscheme eq "https") {
          # oh dear, we have to wrap it into a connect request
+         my $auth = exists $hdr{"proxy-authorization"}
+            ? "proxy-authorization: " . (delete $hdr{"proxy-authorization"}) . "\015\012"
+            : "";
          # maybe re-use $uauthority with patched port?
-         $state{handle}->push_write ("CONNECT $uhost:$uport HTTP/1.0\015\012\015\012");
+         $state{handle}->push_write ("CONNECT $uhost:$uport HTTP/1.0\015\012$auth\015\012");
          $state{handle}->push_read (line => $qr_nlnl, sub {
             $_[1] =~ /^HTTP\/([0-9\.]+) \s+ ([0-9]{3}) (?: \s+ ([^\015\012]*) )?/ix
                or return _error %state, $cb, { @pseudo, Status => 599, Reason => "Invalid proxy connect response ($_[1])" };
             if ($2 == 200) {
             } else {
                _error %state, $cb, { @pseudo, Status => $2, Reason => $3 };
             }
          });
       } else {
+         delete $hdr{"proxy-authorization"} unless $proxy;
          $handle_actual_request->();
       }
    };
    _get_slot $uhost, sub {
       # on a keepalive request (in theory, this should be a separate config option).
       if ($persistent && $KA_CACHE{$ka_key}) {
          $was_persistent = 1;
          $state{handle} = ka_fetch $ka_key;
-         $state{handle}->destroyed
+#         $state{handle}->destroyed
-            and die "AnyEvent::HTTP: unexpectedly got a destructed handle (1), please report.";#d#
+#            and die "AnyEvent::HTTP: unexpectedly got a destructed handle (1), please report.";#d#
          $prepare_handle->();
-         $state{handle}->destroyed
+#         $state{handle}->destroyed
-            and die "AnyEvent::HTTP: unexpectedly got a destructed handle (2), please report.";#d#
+#            and die "AnyEvent::HTTP: unexpectedly got a destructed handle (2), please report.";#d#
          $handle_actual_request->();
       } else {
          my $tcp_connect = $arg{tcp_connect}
                            || do { require AnyEvent::Socket; \&AnyEvent::Socket::tcp_connect };
 save cookies to disk, and you should call this function after loading them
 again. If you have a long-running program you can additionally call this
 function from time to time.
 A cookie jar is initially an empty hash-reference that is managed by this
-module. It's format is subject to change, but currently it is like this:
+module. Its format is subject to change, but currently it is as follows:
-The key C<version> has to contain C<1>, otherwise the hash gets
+The key C<version> has to contain C<2>, otherwise the hash gets
-emptied. All other keys are hostnames or IP addresses pointing to
+cleared. All other keys are hostnames or IP addresses pointing to
 hash-references. The key for these inner hash references is the
 server path for which this cookie is meant, and the values are again
 hash-references. Each key of those hash-references is a cookie name, and
 the value, you guessed it, is another hash-reference, this time with the
 key-value pairs from the cookie, except for C<expires> and C<max-age>,
 Here is an example of a cookie jar with a single cookie, so you have a
 chance of understanding the above paragraph:
    {
-      version    => 1,
+      version    => 2,
       "10.0.0.1" => {
          "/" => {
             "mythweb_id" => {
               _expires => 1293917923,
               value    => "ooRung9dThee3ooyXooM1Ohm",
 C<Mozilla/5.0 (compatible; U; AnyEvent-HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)>).
 =item $AnyEvent::HTTP::MAX_PER_HOST
 The maximum number of concurrent connections to the same host (identified
-by the hostname). If the limit is exceeded, then the additional requests
+by the hostname). If the limit is exceeded, then additional requests
 are queued until previous connections are closed. Both persistent and
 non-persistent connections are counted in this limit.
 The default value for this is C<4>, and it is highly advisable to not
 increase it much.
          or die "$file: $!";
       my %hdr;
       my $ofs = 0;
-      warn stat $fh;
-      warn -s _;
       if (stat $fh and -s _) {
          $ofs = -s _;
          warn "-s is ", $ofs;
          $hdr{"if-unmodified-since"} = AnyEvent::HTTP::format_date +(stat _)[9];
          $hdr{"range"} = "bytes=$ofs-";
             my (undef, $hdr) = @_;
             my $status = $hdr->{Status};
             if (my $time = AnyEvent::HTTP::parse_date $hdr->{"last-modified"}) {
-               utime $fh, $time, $time;
+               utime $time, $time, $fh;
             }
             if ($status == 200 || $status == 206 || $status == 416) {
                # download ok || resume ok || file already fully downloaded
                $cb->(1, $hdr);

Diff Legend

-–
+Removed lines
-+
+Added lines
-<
+Changed lines
->
+Changed lines

Comparing AnyEvent-HTTP/HTTP.pm (file contents): Revision 1.120 by root, Sun Jun 8 23:36:36 2014 UTC vs. Revision 1.136 by root, Wed Oct 16 01:20:02 2019 UTC

Diff Legend

Comparing AnyEvent-HTTP/HTTP.pm (file contents):
Revision 1.120 by root, Sun Jun 8 23:36:36 2014 UTC vs.
Revision 1.136 by root, Wed Oct 16 01:20:02 2019 UTC