AnyEvent-HTTP/HTTP.pm

=head1 NAME

AnyEvent::HTTP - simple but non-blocking HTTP/HTTPS client

=head1 SYNOPSIS

   use AnyEvent::HTTP;

   http_get "http://www.nethype.de/", sub { print $_[1] };

   # ... do something else here

=head1 DESCRIPTION

This module is an L<AnyEvent> user, you need to make sure that you use and
run a supported event loop.

This module implements a simple, stateless and non-blocking HTTP
client. It supports GET, POST and other request methods, cookies and more,
all on a very low level. It can follow redirects supports proxies and
automatically limits the number of connections to the values specified in
the RFC.

It should generally be a "good client" that is enough for most HTTP
tasks. Simple tasks should be simple, but complex tasks should still be
possible as the user retains control over request and response headers.

The caller is responsible for authentication management, cookies (if
the simplistic implementation in this module doesn't suffice), referer
and other high-level protocol details for which this module offers only
limited support.

=head2 METHODS

=over 4

=cut

package AnyEvent::HTTP;

use strict;
no warnings;

use Carp;

use AnyEvent ();
use AnyEvent::Util ();
use AnyEvent::Socket ();
use AnyEvent::Handle ();

use base Exporter::;

our $VERSION = '1.05';

our @EXPORT = qw(http_get http_post http_head http_request);

our $USERAGENT          = "Mozilla/5.0 (compatible; AnyEvent::HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)";
our $MAX_RECURSE        =  10;
our $MAX_PERSISTENT     =   8;
our $PERSISTENT_TIMEOUT =   2;
our $TIMEOUT            = 300;

# changing these is evil
our $MAX_PERSISTENT_PER_HOST = 2;
our $MAX_PER_HOST       = 4;

our $PROXY;
our $ACTIVE = 0;

my %KA_COUNT; # number of open keep-alive connections per host
my %CO_SLOT;  # number of open connections, and wait queue, per host

=item http_get $url, key => value..., $cb->($data, $headers)

Executes an HTTP-GET request. See the http_request function for details on
additional parameters and the return value.

=item http_head $url, key => value..., $cb->($data, $headers)

Executes an HTTP-HEAD request. See the http_request function for details
on additional parameters and the return value.

=item http_post $url, $body, key => value..., $cb->($data, $headers)

Executes an HTTP-POST request with a request body of C<$body>. See the
http_request function for details on additional parameters and the return
value.

=item http_request $method => $url, key => value..., $cb->($data, $headers)

Executes a HTTP request of type C<$method> (e.g. C<GET>, C<POST>). The URL
must be an absolute http or https URL.

When called in void context, nothing is returned. In other contexts,
C<http_request> returns a "cancellation guard" - you have to keep the
object at least alive until the callback get called. If the object gets
destroyed before the callbakc is called, the request will be cancelled.

The callback will be called with the response data as first argument
(or C<undef> if it wasn't available due to errors), and a hash-ref with
response headers as second argument.

All the headers in that hash are lowercased. In addition to the response
headers, the "pseudo-headers" C<HTTPVersion>, C<Status> and C<Reason>
contain the three parts of the HTTP Status-Line of the same name. The
pseudo-header C<URL> contains the original URL (which can differ from the
requested URL when following redirects).

If the server sends a header multiple times, then their contents will be
joined together with a comma (C<,>), as per the HTTP spec.

If an internal error occurs, such as not being able to resolve a hostname,
then C<$data> will be C<undef>, C<< $headers->{Status} >> will be C<599>
and the C<Reason> pseudo-header will contain an error message.

A typical callback might look like this:

   sub {
      my ($body, $hdr) = @_;

      if ($hdr->{Status} =~ /^2/) {
         ... everything should be ok
      } else {
         print "error, $hdr->{Status} $hdr->{Reason}\n";
      }
   }

Additional parameters are key-value pairs, and are fully optional. They
include:

=over 4

=item recurse => $count (default: $MAX_RECURSE)

Whether to recurse requests or not, e.g. on redirects, authentication
retries and so on, and how often to do so.

=item headers => hashref

The request headers to use. Currently, C<http_request> may provide its
own C<Host:>, C<Content-Length:>, C<Connection:> and C<Cookie:> headers
and will provide defaults for C<User-Agent:> and C<Referer:>.

=item timeout => $seconds

The time-out to use for various stages - each connect attempt will reset
the timeout, as will read or write activity. Default timeout is 5 minutes.

=item proxy => [$host, $port[, $scheme]] or undef

Use the given http proxy for all requests. If not specified, then the
default proxy (as specified by C<$ENV{http_proxy}>) is used.

C<$scheme> must be either missing or C<http> for HTTP, or C<https> for
HTTPS.

=item body => $string

The request body, usually empty. Will be-sent as-is (future versions of
this module might offer more options).

=item cookie_jar => $hash_ref

Passing this parameter enables (simplified) cookie-processing, loosely
based on the original netscape specification.

The C<$hash_ref> must be an (initially empty) hash reference which will
get updated automatically. It is possible to save the cookie_jar to
persistent storage with something like JSON or Storable, but this is not
recommended, as expire times are currently being ignored.

Note that this cookie implementation is not of very high quality, nor
meant to be complete. If you want complete cookie management you have to
do that on your own. C<cookie_jar> is meant as a quick fix to get some
cookie-using sites working. Cookies are a privacy disaster, do not use
them unless required to.

=back

Example: make a simple HTTP GET request for http://www.nethype.de/

   http_request GET => "http://www.nethype.de/", sub {
      my ($body, $hdr) = @_;
      print "$body\n";
   };

Example: make a HTTP HEAD request on https://www.google.com/, use a
timeout of 30 seconds.

   http_request
      GET     => "https://www.google.com",
      timeout => 30,
      sub {
         my ($body, $hdr) = @_;
         use Data::Dumper;
         print Dumper $hdr;
      }
   ;

Example: make another simple HTTP GET request, but immediately try to
cancel it.

   my $request = http_request GET => "http://www.nethype.de/", sub {
      my ($body, $hdr) = @_;
      print "$body\n";
   };

   undef $request;

=cut

sub _slot_schedule;
sub _slot_schedule($) {
   my $host = shift;

   while ($CO_SLOT{$host}[0] < $MAX_PER_HOST) {
      if (my $cb = shift @{ $CO_SLOT{$host}[1] }) {
         # somebody wants that slot
         ++$CO_SLOT{$host}[0];
         ++$ACTIVE;

         $cb->(AnyEvent::Util::guard {
            --$ACTIVE;
            --$CO_SLOT{$host}[0];
            _slot_schedule $host;
         });
      } else {
         # nobody wants the slot, maybe we can forget about it
         delete $CO_SLOT{$host} unless $CO_SLOT{$host}[0];
         last;
      }
   }
}

# wait for a free slot on host, call callback
sub _get_slot($$) {
   push @{ $CO_SLOT{$_[0]}[1] }, $_[1];

   _slot_schedule $_[0];
}

sub http_request($$@) {
   my $cb = pop;
   my ($method, $url, %arg) = @_;

   my %hdr;

   $method = uc $method;

   if (my $hdr = $arg{headers}) {
      while (my ($k, $v) = each %$hdr) {
         $hdr{lc $k} = $v;
      }
   }

   my $recurse = exists $arg{recurse} ? delete $arg{recurse} : $MAX_RECURSE;

   return $cb->(undef, { Status => 599, Reason => "recursion limit reached", URL => $url })
      if $recurse < 0;

   my $proxy   = $arg{proxy}   || $PROXY;
   my $timeout = $arg{timeout} || $TIMEOUT;

   $hdr{"user-agent"} ||= $USERAGENT;

   my ($uscheme, $uauthority, $upath, $query, $fragment) =
      $url =~ m|(?:([^:/?#]+):)?(?://([^/?#]*))?([^?#]*)(?:\?([^#]*))?(?:#(.*))?|;

   $uscheme = lc $uscheme;

   my $uport = $uscheme eq "http"  ?  80
             : $uscheme eq "https" ? 443
             : return $cb->(undef, { Status => 599, Reason => "only http and https URL schemes supported", URL => $url });

   $hdr{referer} ||= "$uscheme://$uauthority$upath"; # leave out fragment and query string, just a heuristic

   $uauthority =~ /^(?: .*\@ )? ([^\@:]+) (?: : (\d+) )?$/x
      or return $cb->(undef, { Status => 599, Reason => "unparsable URL", URL => $url });

   my $uhost = $1;
   $uport = $2 if defined $2;

   $uhost =~ s/^\[(.*)\]$/$1/;
   $upath .= "?$query" if length $query;

   $upath =~ s%^/?%/%;

   # cookie processing
   if (my $jar = $arg{cookie_jar}) {
      %$jar = () if $jar->{version} != 1;
 
      my @cookie;
 
      while (my ($chost, $v) = each %$jar) {
         if ($chost =~ /^\./) {
            next unless $chost eq substr $uhost, -length $chost;
         } elsif ($chost =~ /\./) {
            next unless $chost eq $uhost;
         } else {
            next;
         }
 
         while (my ($cpath, $v) = each %$v) {
            next unless $cpath eq substr $upath, 0, length $cpath;
 
            while (my ($k, $v) = each %$v) {
               next if $uscheme ne "https" && exists $v->{secure};
               my $value = $v->{value};
               $value =~ s/([\\"])/\\$1/g;
               push @cookie, "$k=\"$value\"";
            }
         }
      }
 
      $hdr{cookie} = join "; ", @cookie
         if @cookie;
   }

   my ($rhost, $rport, $rscheme, $rpath); # request host, port, path

   if ($proxy) {
      ($rhost, $rport, $rscheme, $rpath) = (@$proxy, $url);

      # don't support https requests over https-proxy transport,
      # can't be done with tls as spec'ed.
      $rscheme = "http" if $uscheme eq "https" && $rscheme eq "https";
   } else {
      ($rhost, $rport, $rscheme, $rpath) = ($uhost, $uport, $uscheme, $upath);
   }

   $hdr{host} = $uhost;
   $hdr{"content-length"} = length $arg{body};

   my %state = (connect_guard => 1);

   _get_slot $uhost, sub {
      $state{slot_guard} = shift;

      return unless $state{connect_guard};

      $state{connect_guard} = AnyEvent::Socket::tcp_connect $rhost, $rport, sub {
         $state{fh} = shift
            or return $cb->(undef, { Status => 599, Reason => "$!", URL => $url });

         delete $state{connect_guard}; # reduce memory usage, save a tree

         # get handle
         $state{handle} = new AnyEvent::Handle
            fh => $state{fh};

         # limit the number of persistent connections
         if ($KA_COUNT{$_[1]} < $MAX_PERSISTENT_PER_HOST) {
            ++$KA_COUNT{$_[1]};
            $state{handle}{ka_count_guard} = AnyEvent::Util::guard { --$KA_COUNT{$_[1]} };
            $hdr{connection} = "keep-alive";
            delete $hdr{connection}; # keep-alive not yet supported
         } else {
            delete $hdr{connection};
         }

         # (re-)configure handle
         $state{handle}->timeout ($timeout);
         $state{handle}->on_error (sub {
            my $errno = "$!";
            %state = ();
            $cb->(undef, { Status => 599, Reason => $errno, URL => $url });
         });
         $state{handle}->on_eof (sub {
            %state = ();
            $cb->(undef, { Status => 599, Reason => "unexpected end-of-file", URL => $url });
         });

         $state{handle}->starttls ("connect") if $rscheme eq "https";

         # handle actual, non-tunneled, request
         my $handle_actual_request = sub {
#            $state{handle}->starttls ("connect") if $uscheme eq "https";

            # send request
            $state{handle}->push_write (
               "$method $rpath HTTP/1.0\015\012"
               . (join "", map "\u$_: $hdr{$_}\015\012", keys %hdr)
               . "\015\012"
               . (delete $arg{body})
            );

            %hdr = (); # reduce memory usage, save a kitten

            # status line
            $state{handle}->push_read (line => qr/\015?\012/, sub {
               $_[1] =~ /^HTTP\/([0-9\.]+) \s+ ([0-9]{3}) (?: \s+ ([^\015\012]*) )?/ix
                  or return (%state = (), $cb->(undef, { Status => 599, Reason => "invalid server response ($_[1])", URL => $url }));

               my %hdr = ( # response headers
                  HTTPVersion => ",$1",
                  Status      => ",$2",
                  Reason      => ",$3",
                  URL         => ",$url"
               );

               # headers, could be optimized a bit
               $state{handle}->unshift_read (line => qr/\015?\012\015?\012/, sub {
                  for ("$_[1]\012") {
                     y/\015//d; # weed out any \015, as they show up in the weirdest of places.

                     # we support spaces in field names, as lotus domino
                     # creates them (actually spaces around seperators
                     # are strictly allowed in http, they are a security issue).
                     $hdr{lc $1} .= ",$2"
                        while /\G
                              ([^:\000-\037]+):
                              [\011\040]*
                              ((?: [^\012]+ | \012[\011\040] )*)
                              \012
                           /gxc;

                     /\G$/
                       or return (%state = (), $cb->(undef, { Status => 599, Reason => "garbled response headers", URL => $url }));
                  }

                  substr $_, 0, 1, ""
                     for values %hdr;

                  my $finish = sub {
                     # TODO: use destroy method, when/if available
                     #$state{handle}->destroy;
                     $state{handle}->on_eof   (undef);
                     $state{handle}->on_error (undef);
                     %state = ();

                     # set-cookie processing
                     if ($arg{cookie_jar}) {
                        for ($hdr{"set-cookie"}) {
                           # parse NAME=VALUE
                           my @kv;

                           while (/\G\s* ([^=;,[:space:]]+) \s*=\s* (?: "((?:[^\\"]+|\\.)*)" | ([^=;,[:space:]]*) )/gcxs) {
                              my $name = $1;
                              my $value = $3;

                              unless ($value) {
                                 $value = $2;
                                 $value =~ s/\\(.)/$1/gs;
                              }

                              push @kv, $name => $value;

                              last unless /\G\s*;/gc;
                           }

                           last unless @kv;

                           my $name = shift @kv;
                           my %kv = (value => shift @kv, @kv);

                           my $cdom;
                           my $cpath = (delete $kv{path}) || "/";

                           if (exists $kv{domain}) {
                              $cdom = delete $kv{domain};
       
                              $cdom =~ s/^\.?/./; # make sure it starts with a "."

                              next if $cdom =~ /\.$/;
          
                              # this is not rfc-like and not netscape-like. go figure.
                              my $ndots = $cdom =~ y/.//;
                              next if $ndots < ($cdom =~ /\.[^.][^.]\.[^.][^.]$/ ? 3 : 2);
                           } else {
                              $cdom = $uhost;
                           }
       
                           # store it
                           $arg{cookie_jar}{version} = 1;
                           $arg{cookie_jar}{$cdom}{$cpath}{$name} = \%kv;

                           redo if /\G\s*,/gc;
                        }
                     }

                     # microsoft and other shitheads don't give a shit for following standards,
                     # try to support some common forms of broken Location headers.
                     if ($_[1]{location} !~ /^(?: $ | [^:\/?\#]+ : )/x) {
                        $_[1]{location} =~ s/^\.\/+//;

                        my $url = "$rscheme://$uhost:$uport";

                        unless ($_[1]{location} =~ s/^\///) {
                           $url .= $upath;
                           $url =~ s/\/[^\/]*$//;
                        }

                        $_[1]{location} = "$url/$_[1]{location}";
                     }

                     if ($_[1]{Status} =~ /^30[12]$/ && $recurse && $method ne "POST") {
                        # apparently, mozilla et al. just change POST to GET here
                        # more research is needed before we do the same
                        http_request ($method, $_[1]{location}, %arg, recurse => $recurse - 1, $cb);
                     } elsif ($_[1]{Status} == 303 && $recurse) {
                        # even http/1.1 is unclear on how to mutate the method
                        $method = "GET" unless $method eq "HEAD";
                        http_request ($method => $_[1]{location}, %arg, recurse => $recurse - 1, $cb);
                     } elsif ($_[1]{Status} == 307 && $recurse && $method =~ /^(?:GET|HEAD)$/) {
                        http_request ($method => $_[1]{location}, %arg, recurse => $recurse - 1, $cb);
                     } else {
                        $cb->($_[0], $_[1]);
                     }
                  };

                  if ($hdr{Status} =~ /^(?:1..|204|304)$/ or $method eq "HEAD") {
                     $finish->(undef, \%hdr);
                  } else {
                     if (exists $hdr{"content-length"}) {
                        $_[0]->unshift_read (chunk => $hdr{"content-length"}, sub {
                           # could cache persistent connection now
                           if ($hdr{connection} =~ /\bkeep-alive\b/i) {
                              # but we don't, due to misdesigns, this is annoyingly complex
                           };

                           $finish->($_[1], \%hdr);
                        });
                     } else {
                        # too bad, need to read until we get an error or EOF,
                        # no way to detect winged data.
                        $_[0]->on_error (sub {
                           $finish->($_[0]{rbuf}, \%hdr);
                        });
                        $_[0]->on_eof (undef);
                        $_[0]->on_read (sub { });
                     }
                  }
               });
            });
         };

         # now handle proxy-CONNECT method
         if ($proxy && $uscheme eq "https") {
            # oh dear, we have to wrap it into a connect request

            # maybe re-use $uauthority with patched port?
            $state{handle}->push_write ("CONNECT $uhost:$uport HTTP/1.0\015\012Host: $uhost\015\012\015\012");
            $state{handle}->push_read (line => qr/\015?\012\015?\012/, sub {
               $_[1] =~ /^HTTP\/([0-9\.]+) \s+ ([0-9]{3}) (?: \s+ ([^\015\012]*) )?/ix
                  or return (%state = (), $cb->(undef, { Status => 599, Reason => "invalid proxy connect response ($_[1])", URL => $url }));

               if ($2 == 200) {
                  $rpath = $upath;
                  &$handle_actual_request;
               } else {
                  %state = ();
                  $cb->(undef, { Status => $2, Reason => $3, URL => $url });
               }
            });
         } else {
            &$handle_actual_request;
         }

      }, sub {
         $timeout
      };
   };

   defined wantarray && AnyEvent::Util::guard { %state = () }
}

sub http_get($@) {
   unshift @_, "GET";
   &http_request
}

sub http_head($@) {
   unshift @_, "HEAD";
   &http_request
}

sub http_post($$@) {
   my $url = shift;
   unshift @_, "POST", $url, "body";
   &http_request
}

=back

=head2 GLOBAL FUNCTIONS AND VARIABLES

=over 4

=item AnyEvent::HTTP::set_proxy "proxy-url"

Sets the default proxy server to use. The proxy-url must begin with a
string of the form C<http://host:port> (optionally C<https:...>).

=item $AnyEvent::HTTP::MAX_RECURSE

The default value for the C<recurse> request parameter (default: C<10>).

=item $AnyEvent::HTTP::USERAGENT

The default value for the C<User-Agent> header (the default is
C<Mozilla/5.0 (compatible; AnyEvent::HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)>).

=item $AnyEvent::HTTP::MAX_PERSISTENT

The maximum number of persistent connections to keep open (default: 8).

Not implemented currently.

=item $AnyEvent::HTTP::PERSISTENT_TIMEOUT

The maximum time to cache a persistent connection, in seconds (default: 2).

Not implemented currently.

=item $AnyEvent::HTTP::ACTIVE

The number of active connections. This is not the number of currently
running requests, but the number of currently open and non-idle TCP
connections. This number of can be useful for load-leveling.

=back

=cut

sub set_proxy($) {
   $PROXY = [$2, $3 || 3128, $1] if $_[0] =~ m%^(https?):// ([^:/]+) (?: : (\d*) )?%ix;
}

# initialise proxy from environment
set_proxy $ENV{http_proxy};

=head1 SEE ALSO

L<AnyEvent>.

=head1 AUTHOR

   Marc Lehmann <schmorp@schmorp.de>
   http://home.schmorp.de/

=cut

1

Revision:	1.32
Committed:	Fri Oct 24 01:27:29 2008 UTC (15 years, 6 months ago) by root
Branch:	MAIN
Changes since 1.31:	+2 -2 lines
Log Message:	* empty log message *
#	User	Rev	Content
1	root	1.1	=head1 NAME
2
3			AnyEvent::HTTP - simple but non-blocking HTTP/HTTPS client
4
5			=head1 SYNOPSIS
6
7			use AnyEvent::HTTP;
8
9	root	1.17	http_get "http://www.nethype.de/", sub { print $_[1] };
10
11			# ... do something else here
12
13	root	1.1	=head1 DESCRIPTION
14
15			This module is an L<AnyEvent> user, you need to make sure that you use and
16			run a supported event loop.
17
18	root	1.11	This module implements a simple, stateless and non-blocking HTTP
19			client. It supports GET, POST and other request methods, cookies and more,
20			all on a very low level. It can follow redirects supports proxies and
21			automatically limits the number of connections to the values specified in
22			the RFC.
23
24			It should generally be a "good client" that is enough for most HTTP
25			tasks. Simple tasks should be simple, but complex tasks should still be
26			possible as the user retains control over request and response headers.
27
28			The caller is responsible for authentication management, cookies (if
29			the simplistic implementation in this module doesn't suffice), referer
30			and other high-level protocol details for which this module offers only
31			limited support.
32
33	root	1.1	=head2 METHODS
34
35			=over 4
36
37			=cut
38
39			package AnyEvent::HTTP;
40
41			use strict;
42			no warnings;
43
44			use Carp;
45
46			use AnyEvent ();
47			use AnyEvent::Util ();
48			use AnyEvent::Socket ();
49			use AnyEvent::Handle ();
50
51			use base Exporter::;
52
53	root	1.28	our $VERSION = '1.05';
54	root	1.1
55	root	1.17	our @EXPORT = qw(http_get http_post http_head http_request);
56	root	1.1
57			our $USERAGENT = "Mozilla/5.0 (compatible; AnyEvent::HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)";
58	root	1.3	our $MAX_RECURSE = 10;
59	root	1.2	our $MAX_PERSISTENT = 8;
60			our $PERSISTENT_TIMEOUT = 2;
61			our $TIMEOUT = 300;
62	root	1.1
63			# changing these is evil
64			our $MAX_PERSISTENT_PER_HOST = 2;
65	root	1.11	our $MAX_PER_HOST = 4;
66	root	1.1
67	root	1.2	our $PROXY;
68	root	1.14	our $ACTIVE = 0;
69	root	1.2
70	root	1.1	my %KA_COUNT; # number of open keep-alive connections per host
71	root	1.11	my %CO_SLOT; # number of open connections, and wait queue, per host
72	root	1.1
73			=item http_get $url, key => value..., $cb->($data, $headers)
74
75			Executes an HTTP-GET request. See the http_request function for details on
76	root	1.29	additional parameters and the return value.
77	root	1.1
78	root	1.5	=item http_head $url, key => value..., $cb->($data, $headers)
79
80	root	1.29	Executes an HTTP-HEAD request. See the http_request function for details
81			on additional parameters and the return value.
82	root	1.5
83			=item http_post $url, $body, key => value..., $cb->($data, $headers)
84	root	1.3
85	root	1.26	Executes an HTTP-POST request with a request body of C<$body>. See the
86	root	1.29	http_request function for details on additional parameters and the return
87			value.
88	root	1.3
89	root	1.1	=item http_request $method => $url, key => value..., $cb->($data, $headers)
90
91			Executes a HTTP request of type C<$method> (e.g. C<GET>, C<POST>). The URL
92			must be an absolute http or https URL.
93
94	root	1.29	When called in void context, nothing is returned. In other contexts,
95			C<http_request> returns a "cancellation guard" - you have to keep the
96			object at least alive until the callback get called. If the object gets
97			destroyed before the callbakc is called, the request will be cancelled.
98
99	root	1.2	The callback will be called with the response data as first argument
100			(or C<undef> if it wasn't available due to errors), and a hash-ref with
101			response headers as second argument.
102
103	root	1.7	All the headers in that hash are lowercased. In addition to the response
104	root	1.20	headers, the "pseudo-headers" C<HTTPVersion>, C<Status> and C<Reason>
105			contain the three parts of the HTTP Status-Line of the same name. The
106			pseudo-header C<URL> contains the original URL (which can differ from the
107			requested URL when following redirects).
108
109	root	1.32	If the server sends a header multiple times, then their contents will be
110			joined together with a comma (C<,>), as per the HTTP spec.
111	root	1.2
112			If an internal error occurs, such as not being able to resolve a hostname,
113			then C<$data> will be C<undef>, C<< $headers->{Status} >> will be C<599>
114			and the C<Reason> pseudo-header will contain an error message.
115
116	root	1.6	A typical callback might look like this:
117
118			sub {
119			my ($body, $hdr) = @_;
120
121			if ($hdr->{Status} =~ /^2/) {
122			... everything should be ok
123			} else {
124			print "error, $hdr->{Status} $hdr->{Reason}\n";
125			}
126			}
127
128	root	1.1	Additional parameters are key-value pairs, and are fully optional. They
129			include:
130
131			=over 4
132
133	root	1.3	=item recurse => $count (default: $MAX_RECURSE)
134	root	1.1
135			Whether to recurse requests or not, e.g. on redirects, authentication
136	root	1.3	retries and so on, and how often to do so.
137	root	1.1
138			=item headers => hashref
139
140	root	1.12	The request headers to use. Currently, C<http_request> may provide its
141			own C<Host:>, C<Content-Length:>, C<Connection:> and C<Cookie:> headers
142			and will provide defaults for C<User-Agent:> and C<Referer:>.
143	root	1.1
144			=item timeout => $seconds
145
146			The time-out to use for various stages - each connect attempt will reset
147	root	1.2	the timeout, as will read or write activity. Default timeout is 5 minutes.
148
149			=item proxy => [$host, $port[, $scheme]] or undef
150
151			Use the given http proxy for all requests. If not specified, then the
152			default proxy (as specified by C<$ENV{http_proxy}>) is used.
153
154			C<$scheme> must be either missing or C<http> for HTTP, or C<https> for
155			HTTPS.
156	root	1.1
157	root	1.3	=item body => $string
158
159			The request body, usually empty. Will be-sent as-is (future versions of
160			this module might offer more options).
161
162	root	1.10	=item cookie_jar => $hash_ref
163
164			Passing this parameter enables (simplified) cookie-processing, loosely
165			based on the original netscape specification.
166
167			The C<$hash_ref> must be an (initially empty) hash reference which will
168			get updated automatically. It is possible to save the cookie_jar to
169			persistent storage with something like JSON or Storable, but this is not
170			recommended, as expire times are currently being ignored.
171
172			Note that this cookie implementation is not of very high quality, nor
173			meant to be complete. If you want complete cookie management you have to
174			do that on your own. C<cookie_jar> is meant as a quick fix to get some
175			cookie-using sites working. Cookies are a privacy disaster, do not use
176			them unless required to.
177
178	root	1.1	=back
179
180	root	1.9	Example: make a simple HTTP GET request for http://www.nethype.de/
181
182			http_request GET => "http://www.nethype.de/", sub {
183			my ($body, $hdr) = @_;
184			print "$body\n";
185			};
186
187			Example: make a HTTP HEAD request on https://www.google.com/, use a
188			timeout of 30 seconds.
189
190			http_request
191			GET => "https://www.google.com",
192			timeout => 30,
193			sub {
194			my ($body, $hdr) = @_;
195			use Data::Dumper;
196			print Dumper $hdr;
197			}
198			;
199	root	1.1
200	root	1.29	Example: make another simple HTTP GET request, but immediately try to
201			cancel it.
202
203			my $request = http_request GET => "http://www.nethype.de/", sub {
204			my ($body, $hdr) = @_;
205			print "$body\n";
206			};
207
208			undef $request;
209
210	root	1.1	=cut
211
212	root	1.12	sub _slot_schedule;
213	root	1.11	sub _slot_schedule($) {
214			my $host = shift;
215
216			while ($CO_SLOT{$host}[0] < $MAX_PER_HOST) {
217			if (my $cb = shift @{ $CO_SLOT{$host}[1] }) {
218	root	1.12	# somebody wants that slot
219	root	1.11	++$CO_SLOT{$host}[0];
220	root	1.14	++$ACTIVE;
221	root	1.11
222			$cb->(AnyEvent::Util::guard {
223	root	1.14	--$ACTIVE;
224	root	1.11	--$CO_SLOT{$host}[0];
225			_slot_schedule $host;
226			});
227			} else {
228			# nobody wants the slot, maybe we can forget about it
229			delete $CO_SLOT{$host} unless $CO_SLOT{$host}[0];
230			last;
231			}
232			}
233			}
234
235			# wait for a free slot on host, call callback
236			sub _get_slot($$) {
237			push @{ $CO_SLOT{$_[0]}[1] }, $_[1];
238
239			_slot_schedule $_[0];
240			}
241
242	elmex	1.15	sub http_request($$@) {
243	root	1.1	my $cb = pop;
244			my ($method, $url, %arg) = @_;
245
246			my %hdr;
247
248	root	1.3	$method = uc $method;
249
250	root	1.8	if (my $hdr = $arg{headers}) {
251	root	1.1	while (my ($k, $v) = each %$hdr) {
252			$hdr{lc $k} = $v;
253			}
254			}
255
256	root	1.23	my $recurse = exists $arg{recurse} ? delete $arg{recurse} : $MAX_RECURSE;
257	root	1.8
258	elmex	1.19	return $cb->(undef, { Status => 599, Reason => "recursion limit reached", URL => $url })
259	root	1.8	if $recurse < 0;
260
261	root	1.2	my $proxy = $arg{proxy} \|\| $PROXY;
262	root	1.1	my $timeout = $arg{timeout} \|\| $TIMEOUT;
263
264			$hdr{"user-agent"} \|\|= $USERAGENT;
265
266	root	1.31	my ($uscheme, $uauthority, $upath, $query, $fragment) =
267	root	1.10	$url =~ m\|(?:([^:/?#]+):)?(?://([^/?#]))?([^?#])(?:\?([^#]))?(?:#(.))?\|;
268	root	1.2
269	root	1.31	$uscheme = lc $uscheme;
270	root	1.1
271	root	1.31	my $uport = $uscheme eq "http" ? 80
272			: $uscheme eq "https" ? 443
273	elmex	1.19	: return $cb->(undef, { Status => 599, Reason => "only http and https URL schemes supported", URL => $url });
274	root	1.10
275	root	1.31	$hdr{referer} \|\|= "$uscheme://$uauthority$upath"; # leave out fragment and query string, just a heuristic
276	root	1.13
277	root	1.31	$uauthority =~ /^(?: .*\@ )? ([^\@:]+) (?: : (\d+) )?$/x
278	elmex	1.19	or return $cb->(undef, { Status => 599, Reason => "unparsable URL", URL => $url });
279	root	1.10
280			my $uhost = $1;
281			$uport = $2 if defined $2;
282
283			$uhost =~ s/^\[(.*)\]$/$1/;
284			$upath .= "?$query" if length $query;
285
286			$upath =~ s%^/?%/%;
287
288			# cookie processing
289			if (my $jar = $arg{cookie_jar}) {
290	root	1.31	%$jar = () if $jar->{version} != 1;
291	root	1.10
292			my @cookie;
293
294			while (my ($chost, $v) = each %$jar) {
295	root	1.30	if ($chost =~ /^\./) {
296			next unless $chost eq substr $uhost, -length $chost;
297			} elsif ($chost =~ /\./) {
298			next unless $chost eq $uhost;
299			} else {
300			next;
301			}
302	root	1.10
303			while (my ($cpath, $v) = each %$v) {
304			next unless $cpath eq substr $upath, 0, length $cpath;
305
306			while (my ($k, $v) = each %$v) {
307	root	1.31	next if $uscheme ne "https" && exists $v->{secure};
308			my $value = $v->{value};
309			$value =~ s/([\\"])/\\$1/g;
310			push @cookie, "$k=\"$value\"";
311	root	1.10	}
312			}
313			}
314
315			$hdr{cookie} = join "; ", @cookie
316			if @cookie;
317			}
318	root	1.1
319	root	1.31	my ($rhost, $rport, $rscheme, $rpath); # request host, port, path
320	root	1.2
321	root	1.10	if ($proxy) {
322	root	1.31	($rhost, $rport, $rscheme, $rpath) = (@$proxy, $url);
323
324			# don't support https requests over https-proxy transport,
325			# can't be done with tls as spec'ed.
326			$rscheme = "http" if $uscheme eq "https" && $rscheme eq "https";
327	root	1.10	} else {
328	root	1.31	($rhost, $rport, $rscheme, $rpath) = ($uhost, $uport, $uscheme, $upath);
329	root	1.2	}
330
331	root	1.31	$hdr{host} = $uhost;
332	root	1.10	$hdr{"content-length"} = length $arg{body};
333	root	1.1
334	root	1.11	my %state = (connect_guard => 1);
335
336			_get_slot $uhost, sub {
337			$state{slot_guard} = shift;
338	root	1.1
339	root	1.11	return unless $state{connect_guard};
340	root	1.1
341	root	1.11	$state{connect_guard} = AnyEvent::Socket::tcp_connect $rhost, $rport, sub {
342			$state{fh} = shift
343	elmex	1.19	or return $cb->(undef, { Status => 599, Reason => "$!", URL => $url });
344	root	1.11
345			delete $state{connect_guard}; # reduce memory usage, save a tree
346
347			# get handle
348			$state{handle} = new AnyEvent::Handle
349	root	1.31	fh => $state{fh};
350	root	1.11
351			# limit the number of persistent connections
352			if ($KA_COUNT{$_[1]} < $MAX_PERSISTENT_PER_HOST) {
353			++$KA_COUNT{$_[1]};
354			$state{handle}{ka_count_guard} = AnyEvent::Util::guard { --$KA_COUNT{$_[1]} };
355			$hdr{connection} = "keep-alive";
356			delete $hdr{connection}; # keep-alive not yet supported
357			} else {
358			delete $hdr{connection};
359			}
360	root	1.1
361	root	1.11	# (re-)configure handle
362			$state{handle}->timeout ($timeout);
363			$state{handle}->on_error (sub {
364	root	1.14	my $errno = "$!";
365	root	1.11	%state = ();
366	elmex	1.19	$cb->(undef, { Status => 599, Reason => $errno, URL => $url });
367	root	1.11	});
368			$state{handle}->on_eof (sub {
369			%state = ();
370	elmex	1.19	$cb->(undef, { Status => 599, Reason => "unexpected end-of-file", URL => $url });
371	root	1.11	});
372	root	1.1
373	root	1.31	$state{handle}->starttls ("connect") if $rscheme eq "https";
374
375			# handle actual, non-tunneled, request
376			my $handle_actual_request = sub {
377			# $state{handle}->starttls ("connect") if $uscheme eq "https";
378
379			# send request
380			$state{handle}->push_write (
381			"$method $rpath HTTP/1.0\015\012"
382			. (join "", map "\u$_: $hdr{$_}\015\012", keys %hdr)
383			. "\015\012"
384			. (delete $arg{body})
385	root	1.11	);
386
387	root	1.31	%hdr = (); # reduce memory usage, save a kitten
388	root	1.10
389	root	1.31	# status line
390			$state{handle}->push_read (line => qr/\015?\012/, sub {
391			$_[1] =~ /^HTTP\/([0-9\.]+) \s+ ([0-9]{3}) (?: \s+ ([^\015\012]*) )?/ix
392			or return (%state = (), $cb->(undef, { Status => 599, Reason => "invalid server response ($_[1])", URL => $url }));
393
394			my %hdr = ( # response headers
395			HTTPVersion => ",$1",
396			Status => ",$2",
397			Reason => ",$3",
398			URL => ",$url"
399			);
400
401			# headers, could be optimized a bit
402			$state{handle}->unshift_read (line => qr/\015?\012\015?\012/, sub {
403			for ("$_[1]\012") {
404			y/\015//d; # weed out any \015, as they show up in the weirdest of places.
405
406			# we support spaces in field names, as lotus domino
407			# creates them (actually spaces around seperators
408			# are strictly allowed in http, they are a security issue).
409			$hdr{lc $1} .= ",$2"
410			while /\G
411			([^:\000-\037]+):
412			[\011\040]*
413			((?: [^\012]+ \| \012[\011\040] )*)
414			\012
415			/gxc;
416
417			/\G$/
418			or return (%state = (), $cb->(undef, { Status => 599, Reason => "garbled response headers", URL => $url }));
419			}
420
421			substr $_, 0, 1, ""
422			for values %hdr;
423
424			my $finish = sub {
425			# TODO: use destroy method, when/if available
426			#$state{handle}->destroy;
427			$state{handle}->on_eof (undef);
428			$state{handle}->on_error (undef);
429			%state = ();
430
431			# set-cookie processing
432			if ($arg{cookie_jar}) {
433			for ($hdr{"set-cookie"}) {
434			# parse NAME=VALUE
435			my @kv;
436
437			while (/\G\s* ([^=;,[:space:]]+) \s=\s (?: "((?:[^\\"]+\|\\.))" \| ([^=;,[:space:]]) )/gcxs) {
438			my $name = $1;
439			my $value = $3;
440
441			unless ($value) {
442			$value = $2;
443			$value =~ s/\\(.)/$1/gs;
444			}
445
446			push @kv, $name => $value;
447
448			last unless /\G\s*;/gc;
449			}
450
451			last unless @kv;
452	root	1.10
453	root	1.31	my $name = shift @kv;
454			my %kv = (value => shift @kv, @kv);
455	root	1.11
456	root	1.31	my $cdom;
457			my $cpath = (delete $kv{path}) \|\| "/";
458	root	1.10
459	root	1.31	if (exists $kv{domain}) {
460			$cdom = delete $kv{domain};
461
462			$cdom =~ s/^\.?/./; # make sure it starts with a "."
463	root	1.11
464	root	1.31	next if $cdom =~ /\.$/;
465
466			# this is not rfc-like and not netscape-like. go figure.
467			my $ndots = $cdom =~ y/.//;
468			next if $ndots < ($cdom =~ /\.[^.][^.]\.[^.][^.]$/ ? 3 : 2);
469			} else {
470			$cdom = $uhost;
471			}
472	root	1.30
473	root	1.31	# store it
474			$arg{cookie_jar}{version} = 1;
475			$arg{cookie_jar}{$cdom}{$cpath}{$name} = \%kv;
476
477			redo if /\G\s*,/gc;
478	root	1.30	}
479	root	1.11	}
480	root	1.8
481	root	1.31	# microsoft and other shitheads don't give a shit for following standards,
482			# try to support some common forms of broken Location headers.
483			if ($_[1]{location} !~ /^(?: $ \| [^:\/?\#]+ : )/x) {
484			$_[1]{location} =~ s/^\.\/+//;
485
486			my $url = "$rscheme://$uhost:$uport";
487
488			unless ($_[1]{location} =~ s/^\///) {
489			$url .= $upath;
490			$url =~ s/\/[^\/]*$//;
491			}
492
493			$_[1]{location} = "$url/$_[1]{location}";
494	root	1.25	}
495
496	root	1.31	if ($_[1]{Status} =~ /^30[12]$/ && $recurse && $method ne "POST") {
497			# apparently, mozilla et al. just change POST to GET here
498			# more research is needed before we do the same
499			http_request ($method, $_[1]{location}, %arg, recurse => $recurse - 1, $cb);
500			} elsif ($_[1]{Status} == 303 && $recurse) {
501			# even http/1.1 is unclear on how to mutate the method
502			$method = "GET" unless $method eq "HEAD";
503			http_request ($method => $_[1]{location}, %arg, recurse => $recurse - 1, $cb);
504			} elsif ($_[1]{Status} == 307 && $recurse && $method =~ /^(?:GET\|HEAD)$/) {
505			http_request ($method => $_[1]{location}, %arg, recurse => $recurse - 1, $cb);
506			} else {
507			$cb->($_[0], $_[1]);
508			}
509			};
510	root	1.24
511	root	1.31	if ($hdr{Status} =~ /^(?:1..\|204\|304)$/ or $method eq "HEAD") {
512			$finish->(undef, \%hdr);
513	root	1.11	} else {
514	root	1.31	if (exists $hdr{"content-length"}) {
515			$_[0]->unshift_read (chunk => $hdr{"content-length"}, sub {
516			# could cache persistent connection now
517			if ($hdr{connection} =~ /\bkeep-alive\b/i) {
518			# but we don't, due to misdesigns, this is annoyingly complex
519			};
520
521			$finish->($_[1], \%hdr);
522			});
523			} else {
524			# too bad, need to read until we get an error or EOF,
525			# no way to detect winged data.
526			$_[0]->on_error (sub {
527			$finish->($_[0]{rbuf}, \%hdr);
528			});
529			$_[0]->on_eof (undef);
530			$_[0]->on_read (sub { });
531			}
532	root	1.11	}
533	root	1.31	});
534			});
535			};
536	root	1.3
537	root	1.31	# now handle proxy-CONNECT method
538			if ($proxy && $uscheme eq "https") {
539			# oh dear, we have to wrap it into a connect request
540
541			# maybe re-use $uauthority with patched port?
542			$state{handle}->push_write ("CONNECT $uhost:$uport HTTP/1.0\015\012Host: $uhost\015\012\015\012");
543			$state{handle}->push_read (line => qr/\015?\012\015?\012/, sub {
544			$_[1] =~ /^HTTP\/([0-9\.]+) \s+ ([0-9]{3}) (?: \s+ ([^\015\012]*) )?/ix
545			or return (%state = (), $cb->(undef, { Status => 599, Reason => "invalid proxy connect response ($_[1])", URL => $url }));
546
547			if ($2 == 200) {
548			$rpath = $upath;
549			&$handle_actual_request;
550	root	1.3	} else {
551	root	1.31	%state = ();
552			$cb->(undef, { Status => $2, Reason => $3, URL => $url });
553	root	1.3	}
554	root	1.11	});
555	root	1.31	} else {
556			&$handle_actual_request;
557			}
558
559	root	1.11	}, sub {
560			$timeout
561			};
562	root	1.1	};
563
564			defined wantarray && AnyEvent::Util::guard { %state = () }
565			}
566
567	elmex	1.15	sub http_get($@) {
568	root	1.1	unshift @_, "GET";
569			&http_request
570			}
571
572	elmex	1.15	sub http_head($@) {
573	root	1.4	unshift @_, "HEAD";
574			&http_request
575			}
576
577	elmex	1.15	sub http_post($$@) {
578	root	1.22	my $url = shift;
579			unshift @_, "POST", $url, "body";
580	root	1.3	&http_request
581			}
582
583	root	1.9	=back
584
585	root	1.2	=head2 GLOBAL FUNCTIONS AND VARIABLES
586	root	1.1
587			=over 4
588
589	root	1.2	=item AnyEvent::HTTP::set_proxy "proxy-url"
590
591			Sets the default proxy server to use. The proxy-url must begin with a
592			string of the form C<http://host:port> (optionally C<https:...>).
593
594	root	1.3	=item $AnyEvent::HTTP::MAX_RECURSE
595	root	1.1
596	root	1.3	The default value for the C<recurse> request parameter (default: C<10>).
597	root	1.1
598			=item $AnyEvent::HTTP::USERAGENT
599
600			The default value for the C<User-Agent> header (the default is
601			C<Mozilla/5.0 (compatible; AnyEvent::HTTP/$VERSION; +http://software.schmorp.de/pkg/AnyEvent)>).
602
603			=item $AnyEvent::HTTP::MAX_PERSISTENT
604
605			The maximum number of persistent connections to keep open (default: 8).
606
607	root	1.3	Not implemented currently.
608
609	root	1.1	=item $AnyEvent::HTTP::PERSISTENT_TIMEOUT
610
611	root	1.2	The maximum time to cache a persistent connection, in seconds (default: 2).
612	root	1.1
613	root	1.3	Not implemented currently.
614
615	root	1.14	=item $AnyEvent::HTTP::ACTIVE
616
617			The number of active connections. This is not the number of currently
618			running requests, but the number of currently open and non-idle TCP
619			connections. This number of can be useful for load-leveling.
620
621	root	1.1	=back
622
623			=cut
624
625	root	1.2	sub set_proxy($) {
626			$PROXY = [$2, $3 \|\| 3128, $1] if $_[0] =~ m%^(https?):// ([^:/]+) (?: : (\d*) )?%ix;
627			}
628
629			# initialise proxy from environment
630			set_proxy $ENV{http_proxy};
631
632	root	1.1	=head1 SEE ALSO
633
634			L<AnyEvent>.
635
636			=head1 AUTHOR
637
638	root	1.18	Marc Lehmann <schmorp@schmorp.de>
639			http://home.schmorp.de/
640	root	1.1
641			=cut
642
643			1
644